Ethereum in BigQuery:  how we built this dataset

Ethereum in BigQuery: how we built this dataset

2018-09-15 admin ofollow,noindex" target="_blank">GoogleCloud No comments

In this blog post, we’ll share more on how we built the BigQuery Ethereum Public Dataset that contains the Ethereum blockchain data. This includes the primary data structures—blocks, transactions—as well as high-value data derivatives—token transfers, smart contract method descriptions.

Leveraging the power of BigQuery allows you to access the Ethereum blockchain via SQL and find meaningful insights rapidly. This data can also be easily exported to CSV, Avro, or JSON files and used for further analysis using graph databases, visualization tools, and machine learning frameworks.

We focused on simplicity, consistency, and reliability when working on this project. Some interesting challenges and important design decisions will be covered in the following sections. We’ll also demonstrate some powerful BigQuery features that allow disassembling all Ethereum smart contracts in seconds.

Our architecture

Stateless Workflow

The workflow starts with export_blocks_and_transactions task where a command from Ethereum ETL named   get_block_range_for_date.py is invoked. Here is an example of usage:

$ python get_block_range_for_date.py –date 2018-01-01

> 4832686,4838611

It returns a block range given a date (alternatively, get_block_range_for_timestamps.py returns a block range that covers some starting and ending timestamps). This command uses Ethereum C#eth_getblockbynumber" rel="nofollow,noindex" target="_blank">JSON RPC API to probe the first and the last blocks, then narrows the bounds recursively using linear interpolation until the required blocks are found. This algorithm is called Interpolation Search and takes O(log(log(n)) time (no typo there) on uniformly distributed data. We implemented an improved version, which handles the worst case by probing estimation points differently , depending on whether the graph is convex or concave.

This handy command eliminates the need to store the latest synchronised block, making the workflow stateless. You can simply schedule the export tasks daily or hourly and get the block range to export based on the execution timestamp.

Batching JSON RPC requests

Both go-ethereum and parity-ethereum support batch JSON RPC requests. They make the exporting process orders of magnitude faster by reducing the number of round trips to Ethereum nodes. Unfortunately, batch requests are not supported by most Ethereum API clients as this feature requires language-specific architecture and needs to be thought through in the early stages of code design.

That’s why we implemented custom providers that support batch requests, which gave us an almost 10-fold boost in performance. This is the command that exports blocks in batches of 100 by default:

$ python export_blocks_and_transactions.py --start-block $START_BLOCK --end-block $START_BLOCK --blocks-output blocks.csv --transactions-output transactions.csv

Below is the schema for the blocks table in BigQuery:

• wait_* tasks wait for an export file on a particular day.

• load_* tasks load the data from a Cloud Storage bucket to BigQuery tables.

• enrich_* tasks join multiple tables and cast columns to different types.

• verify_* tasks run SQLs that verify the consistency of the data.

Atomicity and idempotence

We wanted to make sure all operations in the workflow are atomic and idempotent , as these properties are necessary to guarantee the consistency of the data. Idempotency means you can run an operation multiple times and the result will be as if you ran it just once. It allows Airflow to retry the tasks without keeping or checking any state.

All export tasks are idempotent because the output files will be overwritten in the Cloud Storage bucket, after multiple runs. BigQuery load and query operations are atomic and idempotent, which means there will be no case in which query results are incorrect, even during the process of loading new data.

The idempotence restriction is the reason we don’t append tables with new data, but instead reload all CSV files starting from the genesis block; otherwise we’d end up with duplicate records if a task is retried. Fortunately, load operations are free in BigQuery. A more efficient implementation would be to replace a single partition; we’ll implement this approach at a future date when we add streaming support.

Streaming

We don’t use BigQuery streaming because ensuring consistency in this case requires additional setup . Without it you can end up having duplicate or missing rows in your tables. Another obstacle to ensuring consistency is chain reorganizations in Ethereum. This is a situation when a temporary fork happens in the blockchain, which could make blocks previously streamed to BigQuery stale.

We plan to use streaming alongside daily tasks, which will load and replace past-day partitions (approach called Lambda_architecture" rel="nofollow,noindex" target="_blank">Lambda Architecture ). This will make the data eventually consistent — only the last day may have duplicates, missing rows, and stale blocks, while all the previous days will be strongly consistent. The feature in BigQuery that allows this is the ability to load and query a single partition . For example the following command loads only data for the date June 23, 2018:

$ bq load --destination_table ethereum_blockchain.blocks$20180623 gs://<bucket>/*.csv

The result of ethereumetl_load_dag is enriched tables in the BigQuery dataset, which you can query in the console or via API.

Query

You can access the dataset here . You’ll find all the tables, their schema, and you’ll be able to run queries. Below is the query to retrieve transaction volume in Ether:

You can even disassemble all contracts in the dataset and see which opcodes are most frequent. The query that demonstrates this, as well as the ability to work with arrays and structs in BigQuery, can be found here .

You can try to do many other things with UDFs, for instance parse transaction inputs, parse event logs, or even run EVM! (Just kidding about that last one.) Check out other interesting queries and implementation details in my blog .

Future plans

The next big feature we are going to work on is support for message calls (a.k.a. internal transactions), which will uncover many interesting use cases such as querying all addresses that ever existed, along with their balances and querying all contracts. Another big feature is the streaming support , which will allow analysis of live Ethereum data.

You can find the list of open issues on Github: ethereum-etl , ethereum-etl-airflow . Also be sure to check out limitations before you start using the dataset.

Everyone reading this article is encouraged to contribute to the project — your contribution will be available in the public dataset. Feel free to contact us here if you would like to contribute or have any questions.