> For the complete documentation index, see [llms.txt](https://docs.envio.dev/llms.txt). # Getting Started with HyperSync :::tip Easiest way to get started: the Query Builder The **[HyperSync Query Builder](https://builder.hypersync.xyz)** lets you construct and run queries directly in your browser, with no install and no code. It's the fastest way to get familiar with HyperSync and see what's possible. **[Open builder.hypersync.xyz →](https://builder.hypersync.xyz)** ::: HyperSync is Envio's high-performance blockchain data engine that provides up to 2000x faster access to blockchain data compared to traditional RPC endpoints. This guide will help you understand how to effectively use HyperSync in your applications. ## Quick Start Video Watch this quick tutorial to see HyperSync in action: [Video](https://www.youtube.com/watch?v=S9Z6XkY3aP8) ## Core Concepts HyperSync revolves around two main concepts: 1. **Queries** - Define what blockchain data you want to retrieve 2. **Output Configuration** - Specify how you want that data formatted and delivered Think of queries as your data filter and the output configuration as your data processor. ## Building Effective Queries Queries are the heart of working with HyperSync. They allow you to filter for specific blocks, logs, transactions, and traces. ### Query Structure A basic HyperSync query contains: ```python query = hypersync.Query( from_block=12345678, # Required: Starting block number to_block=12345778, # Optional: Ending block number field_selection=field_selection, # Required: What fields to return logs=[log_selection], # Optional: Filter for specific logs transactions=[tx_selection], # Optional: Filter for specific transactions traces=[trace_selection], # Optional: Filter for specific traces include_all_blocks=False, # Optional: Include blocks with no matches max_num_blocks=1000, # Optional: Soft cap on blocks returned per response max_num_transactions=5000, # Optional: Soft cap on transactions returned per response max_num_logs=5000, # Optional: Soft cap on logs returned per response max_num_traces=5000 # Optional: Soft cap on traces returned per response ) ``` `to_block` is exclusive: the query covers `from_block` up to, but not including, `to_block`. ### Field Selection Field selection allows you to specify exactly which data fields you want to retrieve. This improves performance by only fetching what you need: ```python field_selection = hypersync.FieldSelection( # Block fields you want to retrieve block=[ BlockField.NUMBER, BlockField.TIMESTAMP, BlockField.HASH ], # Transaction fields you want to retrieve transaction=[ TransactionField.HASH, TransactionField.FROM, TransactionField.TO, TransactionField.VALUE ], # Log fields you want to retrieve log=[ LogField.ADDRESS, LogField.TOPIC0, LogField.TOPIC1, LogField.TOPIC2, LogField.TOPIC3, LogField.DATA, LogField.TRANSACTION_HASH ], # Trace fields you want to retrieve (if applicable) trace=[ TraceField.FROM, TraceField.TO, TraceField.VALUE ] ) ``` ### Filtering for Specific Data For most use cases, you'll want to filter for specific logs, transactions, or traces: #### Log Selection Example ```python # Filter for Transfer events from USDC contract log_selection = hypersync.LogSelection( address=["0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48"], # USDC contract topics=[ ["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"] # Transfer event signature ] ) ``` #### Transaction Selection Example ```python # Filter for transactions to the Uniswap V3 router tx_selection = hypersync.TransactionSelection( to=["0xE592427A0AEce92De3Edee1F18E0157C05861564"] # Uniswap V3 Router ) ``` ## Processing the Results HyperSync provides multiple ways to process query results: ### Stream to Parquet Files Parquet is the recommended format for large data sets: ```python # Configure output format parquet_config = hypersync.StreamConfig( hex_output=hypersync.HexOutput.PREFIXED, event_signature="Transfer(address indexed from, address indexed to, uint256 value)" ) # Stream results to a Parquet file await client.collect_parquet("data_directory", query, parquet_config) ``` `event_signature` is accepted by the parquet and arrow methods, `collect_parquet` (which writes an extra `decoded_logs.parquet` alongside the raw output), `collect_arrow` and `stream_arrow`. The methods returning plain Python objects reject it: `collect`, `collect_events`, `stream` and `stream_events` all raise `config.event_signature can't be passed to simple type function`. With those, decode explicitly instead, as shown below. ### Process Data in Memory For immediate processing, stream the results and decode the logs yourself: ```python # Build a decoder for the events you care about decoder = hypersync.Decoder([ "Transfer(address indexed from, address indexed to, uint256 value)" ]) stream = await client.stream(query, hypersync.StreamConfig()) while True: res = await stream.recv() if res is None: # reached the end of the requested range break for log in await decoder.decode_logs(res.data.logs): if log is None: # a log that doesn't match the signature continue # Indexed parameters land in `indexed`, the rest in `body` print(f"Transfer from {log.indexed[0].val} to {log.indexed[1].val} of {log.body[0].val}") ``` ## Tips and Best Practices ### Performance Optimization - **Use Appropriate Batch Sizes**: Adjust batch size based on your chain and use case: ```python config = hypersync.StreamConfig( batch_size=1000000, # Initial block range for the first wave of requests concurrency=10, # Use 10 concurrent workers ) ``` Leave `hex_output` unset with `stream` and `collect`. Any value other than `NO_ENCODE` makes them raise `config.hex_output must be HexOutput::NoEncode for simple type functions`; it only applies to `collect_parquet`, `collect_arrow` and `stream_arrow`. See [Stream Config & Tuning](https://docs.envio.dev/docs/HyperSync/stream-config-tuning.md) for the other options. - **Enable Trace Logs**: Set `RUST_LOG=trace` to see detailed progress: ```bash export RUST_LOG=trace ``` - **Paginate Large Queries**: A single `get` request runs for at most about 10 seconds on the server, so it may return only part of the range. The `collect` and `stream` methods handle this for you. If you call `get` directly, use `next_block` to continue from where it stopped. `archive_height` is the latest block the server has, so add 1 to compare it with the exclusive `end_block`. ```python query.from_block = start_block query.to_block = end_block while True: res = await client.get(query) # process res.data here if res.next_block >= min(end_block, res.archive_height + 1): break query.from_block = res.next_block ``` ### Network-Specific Considerations - **High-Volume Networks**: For networks like Ethereum Mainnet, use smaller block ranges or more specific filters - **Low-Volume Networks**: For smaller chains, you can process the entire chain in one query ## Complete Example Here's a complete example that fetches all USDC Transfer events: ```python import hypersync from hypersync import ( LogSelection, LogField, BlockField, FieldSelection, TransactionField, HexOutput ) import asyncio async def collect_usdc_transfers(): # Initialize client client = hypersync.HypersyncClient( hypersync.ClientConfig( url="https://eth.hypersync.xyz", api_token="your-token-here", # Get from https://docs.envio.dev/docs/HyperSync/api-tokens ) ) # Define field selection field_selection = hypersync.FieldSelection( block=[BlockField.NUMBER, BlockField.TIMESTAMP], transaction=[TransactionField.HASH], log=[ LogField.ADDRESS, LogField.TOPIC0, LogField.TOPIC1, LogField.TOPIC2, LogField.DATA, ] ) # Define query for USDC transfers query = hypersync.Query( from_block=12000000, to_block=12100000, field_selection=field_selection, logs=[ LogSelection( address=["0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48"], # USDC contract topics=[ ["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"] # Transfer signature ] ) ] ) # Configure output config = hypersync.StreamConfig( hex_output=HexOutput.PREFIXED, event_signature="Transfer(address indexed from, address indexed to, uint256 value)" ) # Collect data to a Parquet file await client.collect_parquet("usdc_transfers", query, config) print(f"Processed blocks {query.from_block} to {query.to_block}") asyncio.run(collect_usdc_transfers()) ``` ## Decoding Event Logs When working with blockchain data, event logs contain encoded data that needs to be properly decoded to extract meaningful information. HyperSync provides powerful decoding capabilities to simplify this process. ### Understanding Log Structure Event logs in Ethereum have the following structure: - **Address**: The contract that emitted the event - **Topic0**: The event signature hash (keccak256 of the event signature) - **Topics 1-3**: Indexed parameters (up to 3) - **Data**: Non-indexed parameters packed together ### Using the Decoder HyperSync's client libraries include a `Decoder` class that can parse these raw logs into structured data: ```javascript // Create a decoder with event signatures const decoder = Decoder.fromSignatures([ "Transfer(address indexed from, address indexed to, uint256 amount)", "Approval(address indexed owner, address indexed spender, uint256 amount)", ]); // Decode logs const decodedLogs = await decoder.decodeLogs(logs); ``` ### Single vs. Multiple Event Types HyperSync provides flexibility to decode different types of event logs: - **Single Event Type**: For processing one type of event (e.g., only Swap events) - See complete example: [run-decoder.js](https://github.com/enviodev/hypersync-quickstart/blob/main/run-decoder.js) - **Multiple Event Types**: For processing different events from the same contract (e.g., Transfer and Approval) - See complete example: [run-decoder-multi.js](https://github.com/enviodev/hypersync-quickstart/blob/main/run-decoder-multi.js) ### Working with Decoded Data After decoding, you can access the log parameters in a structured way: - **Indexed parameters**: Available in `decodedLog.indexed` array - **Non-indexed parameters**: Available in `decodedLog.body` array Each parameter object has a single field, `val`, which holds the decoded value. For example, to access parameters from a Transfer event: ```javascript // Access indexed parameters (from, to) const from = decodedLog.indexed[0]?.val.toString(); const to = decodedLog.indexed[1]?.val.toString(); // Access non-indexed parameters (amount) const amount = decodedLog.body[0]?.val.toString(); ``` ### Benefits of Using the Decoder - **Type Safety**: Values are properly converted to their corresponding types - **Simplified Access**: Direct access to decoded values by position - **Batch Processing**: Decode multiple logs with a single call - **Multiple Event Support**: Handle different event types in the same processing pipeline ## Next Steps Now that you understand the basics of using HyperSync: - Browse the [Python Client](https://docs.envio.dev/docs/HyperSync/hypersync-clients.md) or other language-specific clients - Learn about [advanced query options](https://docs.envio.dev/docs/HyperSync/hypersync-query.md) - Tune throughput, and read your rate-limit quota from code, in [Stream Config & Tuning](https://docs.envio.dev/docs/HyperSync/stream-config-tuning.md) - See [example queries for common use cases](https://docs.envio.dev/docs/HyperSync/hypersync-curl-examples.md) - [Get your API token](https://docs.envio.dev/docs/HyperSync/api-tokens.md) to start building For detailed API references and examples in other languages, check our [client documentation](https://docs.envio.dev/docs/HyperSync/hypersync-clients.md).