# Envio: Full Documentation for LLMs
> Every page of docs.envio.dev concatenated as markdown, with per-page source URLs, for direct ingestion into LLM context windows. Pair with https://docs.envio.dev/llms.txt for the navigational index.
# API Tokens
> Learn how to generate and use HyperSync API tokens for secure access.
import Tabs from "@theme/Tabs";
import TabItem from "@theme/TabItem";
# API Tokens for HyperSync
## Overview
API tokens provide authenticated access to HyperSync services, enabling enhanced capabilities and usage tracking.
HyperSync implements **rate limits for requests without API tokens**. API tokens will be required from **3 November 2025**. Indexers deployed to Envio Cloud will have special access to HyperSync that does not require a custom API token.
## Table of Contents
- [Generating API Tokens](#generating-api-tokens)
- [Implementation Guide](#implementation-guide)
- [Security Best Practices](#security-best-practices)
## Generating API Tokens
You can generate API tokens through the Envio Dashboard:
1. Visit [https://envio.dev/app/api-tokens](https://envio.dev/app/api-tokens)
2. Sign in to your account (or create one if you don't have one)
3. Follow the prompts to create a new token
4. Copy and securely store your token
## Implementation Guide
To use an API token, pass it as a `bearer_token` when creating a HyperSync client:
```typescript
const client = new HypersyncClient({
url: "https://eth.hypersync.xyz",
apiToken: process.env.ENVIO_API_TOKEN!,
});
```
```python
client = hypersync.HypersyncClient(hypersync.ClientConfig(
url="https://eth.hypersync.xyz",
bearer_token=os.environ.get("ENVIO_API_TOKEN")
))
```
```rust
let client = Client::new(ClientConfig {
api_token: std::env::var("ENVIO_API_TOKEN").expect("ENVIO_API_TOKEN must be set"),
..Default::default()
}).unwrap()
```
## Understanding Usage
To understand your current month's usage, visit [https://envio.dev/app/api-tokens](https://envio.dev/app/api-tokens). Usage is composed of two main components:
- **Number of Requests**: The total count of API requests made.
- **Credits**: A comprehensive calculation that takes into account multiple factors including data bandwidth, disk read operations, and other resource utilization metrics. This provides the most accurate representation of actual service usage. We're happy to provide more detailed breakdowns of the credit calculation upon request.
## Security Best Practices
When working with API tokens:
- **Never commit tokens to git repositories**
- **Use environment variables** to store tokens instead of hardcoding
- **Add token files like `.env` to your `.gitignore**
- **Rotate tokens periodically** for enhanced security
- **Limit token sharing** to only those who require access
```bash
# Example .env file
ENVIO_API_TOKEN=your_secret_token_here
```
This approach keeps your tokens secure while making them available to your application at runtime.
---
# HyperFuel Query
> Explore all fields and parameters for HyperFuel queries.
# Query Structure
This section is dedicated to giving an exhaustive list of all the fields and query parameters of a HyperFuel query. HyperFuel is extremely powerful but learning how to craft queries can take some practice. It is recommended to look at the examples and reference this page. HyperFuel query structure is the same across clients.
# Top-level query structure
Illustrated as json
```json
{
// The block to start the query from
"from_block": Number,
// The block to end the query at. If not specified, the query will go until the
// end of data. Exclusive, the returned range will be [from_block..to_block).
//
// The query will return before it reaches this target block if it hits the time limit
// configured on the server. The user should continue their query by putting the
// next_block field in the response into from_block field of their next query. This implements
// pagination.
"to_block": Number, // Optional, defaults to latest block
// List of receipt selections, the query will return receipts that match any of these selections.
// All selections have an OR relationship with each other.
"receipts": [{ReceiptSelection}], // Optional
// List of input selections, the query will return inputs that match any of these selections.
// All selections have an OR relationship with each other.
"inputs": [{InputSelection}], // Optional
// List of output selections, the query will return outputs that match any of these selections.
// All selections have an OR relationship with each other.
"outputs": [{OutputSelection}], // Optional
// Whether to include all blocks regardless of whether they match a receipt, input, or output selection. Normally
// The server will return only the blocks that are related to the receipts, inputs, or outputs in the response. But if this
// is set to true, the server will return data for all blocks in the requested range [from_block, to_block).
"include_all_blocks": bool, // Optional, defaults to false
// The user selects which fields they want returned. Requesting fewer fields will improve
// query execution time and reduce the payload size so the user should always use a minimal number of fields.
"field_selection": {FieldSelection},
// Maximum number of blocks that should be returned, the server might return more blocks than this number but
//It won't overshoot by too much.
"max_num_blocks": Number, // Optional, defaults to no maximum
}
```
# ReceiptSelection
The query takes an array of ReceiptSelection objects and returns receipts that match any of the selections. All fields are optional.
Below is an exhaustive list of all fields in a ReceiptSelection JSON object. Reference the [Fuel docs on receipts](https://docs.fuel.network/docs/specs/abi/receipts/#receipts) for field explanations.
```json
{
// address that emitted the receipt
"root_contract_id": [String],
// The recipient address
"to_address": [String],
// The asset id of the coins transferred.
"asset_id": [String],
// the type of receipt
// 0 = Call
// 1 = Return,
// 2 = ReturnData,
// 3 = Panic,
// 4 = Revert,
// 5 = Log,
// 6 = LogData,
// 7 = Transfer,
// 8 = TransferOut,
// 9 = ScriptResult,
// 10 = MessageOut,
// 11 = Mint,
// 12 = Burn,
"receipt_type": [Number],
// The address of the message sender.
"sender": [String],
// The address of the message recipient.
"recipient": [String],
// The contract id of the current context is in an internal context. null otherwise
"contract_id": [String],
// receipt register values.
"ra": [Number],
"rb": [Number],
"rc": [Number],
"rd": [Number],
// the status of the transaction that the receipt originated from
// 1 = Success
// 3 = Failure
"tx_status": [Number],
// the type of the transaction that the receipt originated from
// 0 = script
// 1 = create
// 2 = mint
// 3 = upgrade
// 4 = upload
"tx_type": [Number]
}
```
# InputSelection
The query takes an array of InputSelection objects and returns inputs that match any of the selections. All fields are optional.
Below is an exhaustive list of all fields in an InputSelection JSON object. Reference the [Fuel docs on inputs](https://docs.fuel.network/docs/specs/tx-format/input/#input) for field explanations.
```json
{
// The owning address or predicate root.
"owner": [String],
// The asset ID of the coins.
"asset_id": [String],
// The input contract.
"contract": [String],
// The sender address of the message.
"sender": [String],
// The recipient address of the message.
"recipient": [String],
// The type of input
// 0 = InputCoin,
// 1 = InputContract,
// 2 = InputMessage,
"input_type": [Number],
// the status of the transaction that the input originated from
// 1 = Success
// 3 = Failure
"tx_status": [Number],
// the type of the transaction that the input originated from
// 0 = script
// 1 = create
// 2 = mint
// 3 = upgrade
// 4 = upload
"tx_type": [Number]
}
```
# OutputSelection
The query takes an array of OutputSelection objects and returns outputs that match any of the selections. All fields are optional.
Below is an exhaustive list of all fields in an OutputSelection JSON object. Reference the [Fuel docs on outputs](https://docs.fuel.network/docs/specs/tx-format/output/#output) for field explanations.
```json
{
// The address the coins were sent to.
"to": [String],
// The asset id for the coins sent.
"asset_id": [String],
// The contract that was created.
"contract": [String],
// the type of output
// 0 = CoinOutput,
// 1 = ContractOutput,
// 2 = ChangeOutput,
// 3 = VariableOutput,
// 4 = ContractCreated,
"output_type": [Number],
// the status of the transaction that the input originated from
// 1 = Success
// 3 = Failure
"tx_status": [Number],
// the type of the transaction that the input originated from
// 0 = script
// 1 = create
// 2 = mint
// 3 = upgrade
// 4 = upload
"tx_type": [Number]
}
```
# FieldSelection
The query takes a FieldSelection JSON object where the user specifies what they want returned from data matched by their `ReceiptSelection`, `OutputSelection`, and `InputSelection`. There is no `BlockSelection` or `TransactionSelection` because the query returns all blocks and transactions that include the data you specified in your `ReceiptSelection`, `OutputSelection`, or `InputSelection`.
For best performance, select a minimal amount of fields.
*Important note:* all fields draw inspiration from Fuel's [graphql schema](https://docs.fuel.network/docs/graphql/reference/). Mainly Blocks, Transactions, Receipts, Inputs, and Outputs. Enums of each type (ex: Receipt has 12 different types, two of which are Log and LogData, Input has 3: InputCoin, InputContract, InputMessage, and Output has 5: CoinOutput, ContractOutput, ChangeOutput, VariableOutput, ContractCreated) are flattened into the parent type. This is why multiple fields on any returned Receipt, Input, or Output might be null; it's not a field on all possible enums of that type, so null is inserted.
All fields are optional. Below is an exhaustive list of all fields in a FieldSelection JSON object.
```json
{
"block": [
"id",
"da_height",
"consensus_parameters_version",
"state_transition_bytecode_version",
"transactions_count",
"message_receipt_count",
"transactions_root",
"message_outbox_root",
"event_inbox_root",
"height",
"prev_root",
"time",
"application_hash"
],
"transaction": [
"block_height",
"id",
"input_asset_ids",
"input_contracts",
"input_contract_utxo_id",
"input_contract_balance_root",
"input_contract_state_root",
"input_contract_tx_pointer_tx_index",
"input_contract",
"policies_tip",
"policies_witness_limit",
"policies_maturity",
"policies_max_fee",
"script_gas_limit",
"maturity",
"mint_amount",
"mint_asset_id",
"mint_gas_price",
"tx_pointer_block_height",
"tx_pointer_tx_index",
"tx_type",
"output_contract_input_index",
"output_contract_balance_root",
"output_contract_state_root",
"witnesses",
"receipts_root",
"status",
"time",
"reason",
"script",
"script_data",
"bytecode_witness_index",
"bytecode_root",
"subsection_index",
"subsections_number",
"proof_set",
"consensus_parameters_upgrade_purpose_witness_index",
"consensus_parameters_upgrade_purpose_checksum",
"state_transition_upgrade_purpose_root",
"salt"
],
"receipt": [
"receipt_index",
"root_contract_id",
"tx_id",
"tx_status",
"tx_type",
"block_height",
"pc",
"is",
"to",
"to_address",
"amount",
"asset_id",
"gas",
"param1",
"param2",
"val",
"ptr",
"digest",
"reason",
"ra",
"rb",
"rc",
"rd",
"len",
"receipt_type",
"result",
"gas_used",
"data",
"sender",
"recipient",
"nonce",
"contract_id",
"sub_id"
],
"input": [
"tx_id",
"tx_status",
"tx_type",
"block_height",
"input_type",
"utxo_id",
"owner",
"amount",
"asset_id",
"tx_pointer_block_height",
"tx_pointer_tx_index",
"witness_index",
"predicate_gas_used",
"predicate",
"predicate_data",
"balance_root",
"state_root",
"contract",
"sender",
"recipient",
"nonce",
"data"
],
"output": [
"tx_id",
"tx_status",
"tx_type",
"block_height",
"output_type",
"to",
"amount",
"asset_id",
"input_index",
"balance_root",
"state_root",
"contract",
]
}
```
---
# HyperFuel
> HyperFuel gives you a high performance way to query and sync Fuel network data using HyperSync. Access real-time and historical data with flexible queries across contracts, events and state.
[HyperSync](https://docs.envio.dev/docs/HyperSync/overview) is a high-performance data node and accelerated data query layer that powers Envio’s Indexing framework, [HyperIndex](https://docs.envio.dev/docs/HyperIndex/overview), for up to 1000x faster data retrieval than standard RPC methods.
HyperFuel is HyperSync adapted for the [Fuel Network](https://fuel.network/) and is exposed as a low-level API for developers and data analysts to create flexible, high-speed queries for all fuel data.
Users can interact with the HyperFuel in Rust, Python, NodeJS clients, or directly via the JSON API to extract data into parquet files, arrow format, or as typed data. Client examples are listed furhter below.
Using HyperFuel, application developers can easily sync and search large datasets in a few minutes. HyperFuel is an ideal solution for indexers, block explorers, data analysts, bridges, and other applications or use cases focused on performance.
You can integrate with HyperFuel using any of our clients:
- Rust: https://github.com/enviodev/hyperfuel-client-rust
- Python: https://github.com/enviodev/hyperfuel-client-python
- Nodejs: https://github.com/enviodev/hyperfuel-client-node
- JSON API: https://github.com/enviodev/hyperfuel-json-api
:::info
HyperFuel supports Fuel mainnet and testnet:
Mainnet: https://fuel.hypersync.xyz
Testnet: https://fuel-testnet.hypersync.xyz
:::
:::note
HyperFuel requires an API token. Generate one from the [Envio Dashboard](https://envio.dev/app/api-tokens) and pass it to the client when you initialize it. For example, the Rust client below reads it from the `ENVIO_API_TOKEN` environment variable and passes it via the `api_token` config field. See [API Tokens](/docs/HyperSync/api-tokens) for more details.
:::
## Example usage
Below is an example of a Hyperfuel query in each of our clients searching the first 1,300,000 blocks for all `input` objects of a specific `asset-id`. This example returns 10,543 inputs in around 100ms - not including latency.
## Rust ([repo](https://github.com/enviodev/hyperfuel-client-rust/tree/main/examples))
```rust
use hyperfuel_client::{Client, ClientConfig};
use hyperfuel_net_types::Query;
use url::Url;
#[tokio::main]
async fn main() {
let client_config = ClientConfig {
url: Some(Url::parse("https://fuel-testnet.hypersync.xyz").unwrap()),
api_token: std::env::var("ENVIO_API_TOKEN")
.expect("ENVIO_API_TOKEN env var is required, get a token from https://envio.dev/app/api-tokens"),
..Default::default()
};
let client = Client::new(client_config).unwrap();
// Construct query in json. Can also construct it as a typed struct (see predicate-root example)
let query: Query = serde_json::from_value(serde_json::json!({
// start query from block 0
"from_block": 0,
// if to_block is not set, query runs to the end of the chain
"to_block": 1300000,
// load inputs that have `asset_id` = 0x2a0d0ed9d2217ec7f32dcd9a1902ce2a66d68437aeff84e3a3cc8bebee0d2eea
"inputs": [
{
"asset_id": ["0x2a0d0ed9d2217ec7f32dcd9a1902ce2a66d68437aeff84e3a3cc8bebee0d2eea"]
}
],
// fields we want returned from loaded inputs
"field_selection": {
"input": [
"tx_id",
"block_height",
"input_type",
"utxo_id",
"owner",
"amount",
"asset_id"
]
}
}))
.unwrap();
let res = client.get_selected_data(&query).await.unwrap();
println!("inputs: {:?}", res.data.inputs);
}
```
## Python ([repo](https://github.com/enviodev/hyperfuel-client-python/blob/main/examples/asset-id.py))
```python
import hyperfuel
from hyperfuel import InputField
import asyncio
async def main():
client = hyperfuel.HyperfuelClient()
query = hyperfuel.Query(
# start query from block 0
from_block=0,
# if to_block is not set, query runs to the end of the chain
to_block = 1300000,
# load inputs that have `asset_id` = 0x2a0d0ed9d2217ec7f32dcd9a1902ce2a66d68437aeff84e3a3cc8bebee0d2eea
inputs=[
hyperfuel.InputSelection(
asset_id=["0x2a0d0ed9d2217ec7f32dcd9a1902ce2a66d68437aeff84e3a3cc8bebee0d2eea"]
)
],
# what data we want returned from the inputs we loaded
field_selection=hyperfuel.FieldSelection(
input=[
InputField.TX_ID,
InputField.BLOCK_HEIGHT,
InputField.INPUT_TYPE,
InputField.UTXO_ID,
InputField.OWNER,
InputField.AMOUNT,
InputField.ASSET_ID,
]
)
)
res = await client.get_selected_data(query)
print("inputs: " + str(res.data.inputs))
asyncio.run(main())
```
## Node Js ([repo](https://github.com/enviodev/hyperfuel-client-node/tree/main/examples/asset-id))
```js
import { HyperfuelClient, Query } from "@envio-dev/hyperfuel-client";
async function main() {
const client = HyperfuelClient.new({
url: "https://fuel-testnet.hypersync.xyz",
});
const query: Query = {
// start query from block 0
fromBlock: 0,
// if to_block is not set, query runs to the end of the chain
toBlock: 1300000,
// load inputs that have `asset_id` = 0x2a0d0ed9d2217ec7f32dcd9a1902ce2a66d68437aeff84e3a3cc8bebee0d2eea
inputs: [
{
assetId: [
"0x2a0d0ed9d2217ec7f32dcd9a1902ce2a66d68437aeff84e3a3cc8bebee0d2eea",
],
},
],
// fields we want returned from loaded inputs
fieldSelection: {
input: [
"tx_id",
"block_height",
"input_type",
"utxo_id",
"owner",
"amount",
"asset_id",
],
},
};
const res = await client.getSelectedData(query);
console.log(`inputs: ${JSON.stringify(res.data.inputs)}`);
}
main();
```
## Json Api ([repo](https://github.com/enviodev/hyperfuel-json-api/tree/main/asset-id-query-example))
```bash
curl --request POST \
--url https://fuel-testnet.hypersync.xyz/query \
--header 'Content-Type: application/json' \
--data '{
"from_block": 0,
"to_block": 1300000,
"inputs": [
{
"asset_id": ["0x2a0d0ed9d2217ec7f32dcd9a1902ce2a66d68437aeff84e3a3cc8bebee0d2eea"]
}
],
"field_selection": {
"input": [
"tx_id",
"block_height",
"input_type",
"utxo_id",
"owner",
"amount",
"asset_id"
]
}
}'
```
---
# HyperSync Clients
> Explore HyperSync clients for fast blockchain data access in Node.js, Python, Rust, and Go.
# HyperSync Client Libraries
HyperSync provides powerful client libraries that enable you to integrate high-performance blockchain data access into your applications. These libraries handle the communication with HyperSync servers, data serialization/deserialization, and provide convenient APIs for querying blockchain data.
## Quick Links
| Client | Resources |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Node.js** | [📝 API Docs](https://enviodev.github.io/hypersync-client-node/) · [📦 NPM](https://www.npmjs.com/package/@envio-dev/hypersync-client) · [💻 GitHub](https://github.com/enviodev/hypersync-client-node) · [🧪 Examples](https://github.com/enviodev/hypersync-client-node/tree/main/examples) |
| **Python** | [📦 PyPI](https://pypi.org/project/hypersync/) · [💻 GitHub](https://github.com/enviodev/hypersync-client-python) · [🧪 Examples](https://github.com/enviodev/hypersync-client-python/tree/main/examples) |
| **Rust** | [📦 Crates.io](https://crates.io/crates/hypersync-client) · [📝 API Docs](https://docs.rs/hypersync-client/latest/hypersync_client/) · [💻 GitHub](https://github.com/enviodev/hypersync-client-rust) · [🧪 Examples](https://github.com/enviodev/hypersync-client-rust/tree/main/examples) |
| **Go** _(community)_ | [💻 GitHub](https://github.com/enviodev/hypersync-client-go) · [🧪 Examples](https://github.com/enviodev/hypersync-client-go/tree/main/examples) |
| **API Tokens** | [🔑 Get Tokens](./api-tokens.mdx) |
## Client Overview
All HyperSync clients share these key features:
- **High Performance**: Built on a common Rust foundation for maximum efficiency
- **Optimized Transport**: Uses binary formats to minimize bandwidth and maximize throughput
- **Consistent Experience**: Similar APIs across all language implementations
- **Automatic Pagination**: Handles large data sets efficiently
- **Event Decoding**: Parses binary event data into structured formats
Choose the client that best matches your application's technology stack:
| Feature | Node.js | Python | Rust | Go |
| ----------------- | ------------- | ------------------ | ------------- | ------------- |
| Async Support | ✅ | ✅ | ✅ | ✅ |
| Typing | TypeScript | Type Hints | Native | Native |
| Data Formats | JSON, Parquet | JSON, Parquet, CSV | JSON, Parquet | JSON, Parquet |
| Memory Efficiency | Good | Better | Best | Better |
| Installation | npm | pip | cargo | go get |
## Node.js Client
:::info Resources
- [📝 API Documentation](https://enviodev.github.io/hypersync-client-node/)
- [📦 NPM Package](https://www.npmjs.com/package/@envio-dev/hypersync-client)
- [💻 GitHub Repository](https://github.com/enviodev/hypersync-client-node)
- [🧪 Example Projects](https://github.com/enviodev/hypersync-client-node/tree/main/examples)
:::
The Node.js client provides a TypeScript-first experience for JavaScript developers.
### Installation
```bash
# Using npm
npm install @envio-dev/hypersync-client
# Using yarn
yarn add @envio-dev/hypersync-client
# Using pnpm
pnpm add @envio-dev/hypersync-client
```
## Python Client
:::info Resources
- [📦 PyPI Package](https://pypi.org/project/hypersync/)
- [💻 GitHub Repository](https://github.com/enviodev/hypersync-client-python)
- [🧪 Example Projects](https://github.com/enviodev/hypersync-client-python/tree/main/examples)
:::
The Python client provides a Pythonic interface with full type hinting support.
### Installation
```bash
pip install hypersync
```
## Rust Client
:::info Resources
- [📦 Crates.io Package](https://crates.io/crates/hypersync-client)
- [📝 API Documentation](https://docs.rs/hypersync-client/latest/hypersync_client/)
- [💻 GitHub Repository](https://github.com/enviodev/hypersync-client-rust)
- [🧪 Example Projects](https://github.com/enviodev/hypersync-client-rust/tree/main/examples)
:::
The Rust client provides the most efficient and direct access to HyperSync, with all the safety and performance benefits of Rust.
### Installation
Add the following to your `Cargo.toml`:
```toml
[dependencies]
hypersync-client = "0.1"
tokio = { version = "1", features = ["full"] }
```
## Go Client
:::info Resources
- [💻 GitHub Repository](https://github.com/enviodev/hypersync-client-go)
- [🧪 Example Projects](https://github.com/enviodev/hypersync-client-go/tree/main/examples)
:::
:::caution Community Maintained
The Go client is community maintained and marked as work-in-progress. For production use, you may want to test thoroughly or consider the officially supported clients.
:::
The Go client provides a native Go interface for accessing HyperSync, with support for streaming and decoding blockchain data.
### Installation
```bash
go get github.com/enviodev/hypersync-client-go
```
## Using API Tokens
:::info Get Your API Token
You'll need an API token to use any HyperSync client. [Get your token here](./api-tokens.mdx).
:::
All HyperSync clients require an API token for authentication. Tokens are used to manage access and usage limits.
To get an API token:
1. Visit [Envio](https://envio.dev)
2. Register or sign in with your github account
3. Navigate to the API Tokens section
4. Create a new token
For detailed instructions, see our [API Tokens](./api-tokens.mdx) guide.
## Client Selection Guide
Choose the client that best fits your use case:
:::tip Node.js Client
**Choose when**: You're building JavaScript/TypeScript applications or if your team is most comfortable with the JavaScript ecosystem.
:::
:::tip Python Client
**Choose when**: You're doing data science work, need integration with pandas/numpy, or if your team prefers Python's simplicity.
:::
:::tip Rust Client
**Choose when**: You need maximum performance, are doing systems programming, or building performance-critical applications.
:::
:::tip Go Client
**Choose when**: You're working in a Go ecosystem and want native integration with Go applications. Note that this client is community maintained.
:::
## Additional Resources
- [📚 HyperSync Usage Guide](./hypersync-usage)
- [📝 Query Reference](./hypersync-query)
- [🧪 cURL Examples](./hypersync-curl-examples)
- [📊 Supported Networks](./hypersync-supported-networks)
- [🧱 HyperIndex blockchain indexer](/docs/HyperIndex/overview), the full framework with schema, handlers, and a hosted GraphQL API, powered by HyperSync
## Support
Need help getting started or have questions about our clients? Connect with our community:
- [Discord Community](https://discord.gg/envio)
- [GitHub Issues](https://github.com/enviodev)
- [Documentation Feedback](https://github.com/enviodev/docs/issues)
```
```
---
# Using curl with HyperSync
> Explore how to quickly fetch blockchain data using curl commands with HyperSync.
# Using curl with HyperSync
This guide demonstrates how to interact with HyperSync using direct HTTP requests via curl. These examples provide a quick way to explore HyperSync functionality without installing client libraries.
:::info Recommended Approach
We highly recommend trying these curl examples as they're super quick and easy to run directly in your terminal. It's one of the fastest ways to experience HyperSync's performance firsthand and see just how quickly you can retrieve blockchain data without any setup overhead. Simply copy, paste, and be amazed by the response speed!
While curl requests are technically slower than our client libraries (since they use HTTP rather than binary data transfer protocols), they're still impressively fast and provide an excellent demonstration of HyperSync's capabilities without any installation requirements.
:::
## Table of Contents
- [Curl vs. Client Libraries](#curl-vs-client-libraries)
- [When to Use curl (JSON API)](#when-to-use-curl-json-api)
- [When to Use Client Libraries](#when-to-use-client-libraries)
- [Common Use Cases](#common-use-cases)
- [Get All ERC-20 Transfers for an Address](#get-all-erc-20-transfers-for-an-address)
- [Get All Logs for a Smart Contract](#get-all-logs-for-a-smart-contract)
- [Get Blob Data for the Optimism Chain](#get-blob-data-for-the-optimism-chain)
- [Get Mint USDC Events](#get-mint-usdc-events)
- [Get All Transactions for an Address](#get-all-transactions-for-an-address)
- [Get Successful or Failed Transactions](#get-successful-or-failed-transactions)
## Curl vs. Client Libraries
When deciding whether to use curl commands or client libraries, consider the following comparison:
### When to Use curl (JSON API)
- **Quick Prototyping**: Test endpoints and explore data structure without setup
- **Simple Scripts**: Perfect for shell scripts and automation
- **Language Independence**: When working with languages without HyperSync client libraries
- **API Exploration**: When learning the HyperSync API capabilities
### When to Use Client Libraries
- **Production Applications**: For stable, maintained codebases
- **Complex Data Processing**: When working with large datasets or complex workflows
- **Performance**: Client libraries offer automatic compression and pagination
- **Error Handling**: Built-in retry mechanisms and better error reporting
- **Data Formats**: Support for efficient formats like Apache Arrow
## Common Use Cases
### Get All ERC-20 Transfers for an Address
This example filters for all ERC-20 transfer events involving a specific address, either as sender or recipient. Feel free to swap your address into the example.
**What this query does:**
- Filters logs for the Transfer event signature (topic0)
- Matches when the address appears in either topic1 (sender) or topic2 (recipient)
- Also includes direct transactions to/from the address
```bash
curl --request POST \
--url https://eth.hypersync.xyz/query \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer YOUR_API_KEY' \
--data '{
"from_block": 0,
"logs": [
{
"topics": [
[
"0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"
],
[],
[
"0x0000000000000000000000001e037f97d730Cc881e77F01E409D828b0bb14de0"
]
]
},
{
"topics": [
[
"0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"
],
[
"0x0000000000000000000000001e037f97d730Cc881e77F01E409D828b0bb14de0"
],
[]
]
}
],
"transactions": [
{
"from": [
"0x1e037f97d730Cc881e77F01E409D828b0bb14de0"
]
},
{
"to": [
"0x1e037f97d730Cc881e77F01E409D828b0bb14de0"
]
}
],
"field_selection": {
"block": [
"number",
"timestamp",
"hash"
],
"log": [
"block_number",
"log_index",
"transaction_index",
"data",
"address",
"topic0",
"topic1",
"topic2",
"topic3"
],
"transaction": [
"block_number",
"transaction_index",
"hash",
"from",
"to",
"value",
"input"
]
}
}'
```
### Get All Logs for a Smart Contract
This example retrieves all event logs emitted by a specific contract (USDC in this case).
**Key points:**
- Sets `from_block: 0` to scan from the beginning of the chain
- Uses `next_block` in the response for pagination to fetch subsequent data
- Includes relevant block, log, and transaction fields
```bash
curl --request POST \
--url https://eth.hypersync.xyz/query \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer YOUR_API_KEY' \
--data '{
"from_block": 0,
"logs": [
{
"address": ["0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48"]
}
],
"field_selection": {
"block": [
"number",
"timestamp",
"hash"
],
"log": [
"block_number",
"log_index",
"transaction_index",
"data",
"address",
"topic0",
"topic1",
"topic2",
"topic3"
],
"transaction": [
"block_number",
"transaction_index",
"hash",
"from",
"to",
"value",
"input"
]
}
}'
```
### Get Blob Data for the Optimism Chain
This example finds blob transactions used by the Optimism chain for data availability.
**Key points:**
- Starts at a relatively recent block (20,000,000)
- Filters for transactions from the Optimism sequencer address
- Specifically looks for type 3 (blob) transactions
- Results can be used to retrieve the actual blob data from Ethereum
```bash
curl --request POST \
--url https://eth.hypersync.xyz/query \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer YOUR_API_KEY' \
--data '{
"from_block": 20000000,
"transactions": [
{
"from": ["0x6887246668a3b87F54DeB3b94Ba47a6f63F32985"],
"to": ["0xFF00000000000000000000000000000000000010"],
"type": [3]
}
],
"field_selection": {
"block": [
"number",
"timestamp",
"hash"
],
"transaction": [
"block_number",
"transaction_index",
"hash",
"from",
"to",
"type"
]
}
}'
```
### Get Mint USDC Events
This example identifies USDC token minting events.
**How it works:**
- Filters for the USDC contract address
- Looks for Transfer events (topic0)
- Specifically matches when topic1 (from address) is the zero address, indicating a mint
- Returns detailed information about each mint event
```bash
curl --request POST \
--url https://eth.hypersync.xyz/query \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer YOUR_API_KEY' \
--data '{
"from_block": 0,
"logs": [
{
"address": ["0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48"],
"topics": [
[
"0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"
],
[
"0x0000000000000000000000000000000000000000000000000000000000000000"
],
[]
]
}
],
"field_selection": {
"block": [
"number",
"timestamp",
"hash"
],
"log": [
"block_number",
"log_index",
"transaction_index",
"data",
"address",
"topic0",
"topic1",
"topic2",
"topic3"
],
"transaction": [
"block_number",
"transaction_index",
"hash",
"from",
"to",
"value",
"input"
]
}
}'
```
### Get All Transactions for an Address
This example retrieves all transactions where a specific address is either the sender or receiver.
**Implementation notes:**
- Starts from a specific block (15,362,000) for efficiency
- Uses two transaction filters in an OR relationship
- Only includes essential fields in the response
- Multiple queries may be needed for complete history
```bash
curl --request POST \
--url https://eth.hypersync.xyz/query \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer YOUR_API_KEY' \
--data '{
"from_block": 15362000,
"transactions": [
{
"from": ["0xdb255746609baadd67ef44fc15b5e1d04befbca7"]
},
{
"to": ["0xdb255746609baadd67ef44fc15b5e1d04befbca7"]
}
],
"field_selection": {
"block": [
"number",
"timestamp",
"hash"
],
"transaction": [
"block_number",
"transaction_index",
"hash",
"from",
"to"
]
}
}'
```
### Get Successful or Failed Transactions
This example shows how to filter transactions based on their status (successful or failed) for recent blocks.
**How it works:**
1. First, query the current chain height
2. Calculate a starting point (current height minus 10)
3. Query transactions with status=1 (successful) or status=0 (failed)
```bash
# Get current height and calculate starting block
height=$((`curl https://eth.hypersync.xyz/height | jq .height` - 10))
# Query successful transactions (change status to 0 for failed transactions)
curl --request POST \
--url https://eth.hypersync.xyz/query \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer YOUR_API_KEY' \
--data "{
\"from_block\": ${height},
\"transactions\": [
{
\"status\": 1
}
],
\"field_selection\": {
\"block\": [
\"number\",
\"timestamp\",
\"hash\"
],
\"transaction\": [
\"block_number\",
\"transaction_index\",
\"hash\",
\"from\",
\"to\"
]
}
}"
```
---
# Preset Queries
> Explore ready-to-use HyperSync query presets to fetch blocks, transactions, and logs efficiently.
# HyperSync Preset Queries
HyperSync's client libraries include helper functions that build common queries. These presets are useful when you need raw blockchain objects without crafting a query manually.
Each preset returns a `Query` object so you can pass it directly to `client.get`, `client.stream`, or `client.collect`.
## Available Presets
### `preset_query_blocks_and_transactions(from_block, to_block)`
Returns every block and all associated transactions within the supplied block range.
```python
import hypersync
import asyncio
async def main():
client = hypersync.HypersyncClient(hypersync.ClientConfig())
query = hypersync.preset_query_blocks_and_transactions(17_000_000, 17_000_050)
result = await client.get(query)
print(f"Query returned {len(result.data.blocks)} blocks and {len(result.data.transactions)} transactions")
asyncio.run(main())
```
### `preset_query_blocks_and_transaction_hashes(from_block, to_block)`
Returns each block in the range along with only the transaction hashes.
### `preset_query_get_logs(addresses, from_block, to_block)`
Fetches all logs emitted by the provided contract addresses in the given block range.
```python
logs_res = await client.get(
hypersync.preset_query_get_logs(["0xYourContract"], 17_000_000, 17_000_050)
)
```
### `preset_query_logs(from_block, to_block)`
Fetches every log across the specified blocks.
### `preset_query_logs_of_event(event_signature, from_block, to_block)`
Fetches logs for the specified event signature over the block range.
Client libraries for other languages expose the same presets under similar names. See the [Python](https://github.com/enviodev/hypersync-client-python) and [Node.js](https://github.com/enviodev/hypersync-client-node) example repositories for more details.
Use these helpers whenever you need a quick query without specifying field selections or joins manually.
---
# HyperSync Query
> Explore HyperSync queries to efficiently retrieve, filter, and join blockchain data.
# HyperSync Query
This guide explains how to structure queries for HyperSync to efficiently retrieve blockchain data. You'll learn both the basics and advanced techniques to make the most of HyperSync's powerful querying capabilities.
:::note HyperFuel Limitations
Not all features implemented in HyperSync are available in HyperFuel (the Fuel implementation of HyperSync). For example, as of this writing, stream and collect functions aren't implemented in the Fuel client.
:::
## Client Examples
HyperSync offers client libraries in multiple languages, each with its own comprehensive examples. Instead of providing generic examples here, we recommend exploring the language-specific examples:
| Client | Example Links |
| ----------- | -------------------------------------------------------------------------------------------- |
| **Node.js** | [Example Repository](https://github.com/enviodev/hypersync-client-node/tree/main/examples) |
| **Python** | [Example Repository](https://github.com/enviodev/hypersync-client-python/tree/main/examples) |
| **Rust** | [Example Repository](https://github.com/enviodev/hypersync-client-rust/tree/main/examples) |
| **Go** | [Example Repository](https://github.com/enviodev/hypersync-client-go/tree/main/examples) |
Additionally, we maintain a comprehensive collection of real-world examples covering various use cases across different languages:
- [**30 HyperSync Examples**](https://github.com/enviodev/30-hypersync-examples) - A diverse collection of practical examples demonstrating HyperSync's capabilities in Python, JavaScript, TypeScript, Rust, and more.
For more details on client libraries, see the [HyperSync Clients](./hypersync-clients) documentation.
:::tip Visual Query Builder
Need help building queries? Try our **[Intuitive Query Builder](http://builder.hypersync.xyz)** to construct queries visually and see the results in real-time.
:::
:::tip Developer Tip
Set the `RUST_LOG` environment variable to `trace` for more detailed logs when using client libraries.
:::
## Table of Contents
1. [Understanding HyperSync Queries](#understanding-hypersync-queries)
2. [Query Execution Process](#query-execution-process)
3. [Query Structure Reference](#query-structure-reference)
4. [Data Schema](#data-schema)
5. [Response Structure](#response-structure)
6. [Stream and Collect Functions](#stream-and-collect-functions)
7. [Working with Join Modes](#working-with-join-modes)
8. [Best Practices](#best-practices)
## Understanding HyperSync Queries
A HyperSync query defines what blockchain data you want to retrieve and how you want it returned. Unlike regular RPC calls, HyperSync queries offer:
- **Flexible filtering** across logs, transactions, traces, and blocks
- **Field selection** to retrieve only the data you need
- **Automatic pagination** to handle large result sets
- **Join capabilities** that link related blockchain data together
### Core Concepts
- **Selections**: Define criteria for filtering blockchain data (logs, transactions, traces)
- **Field Selection**: Specify which fields to include in the response
- **Limits**: Control query execution time and response size
- **Joins**: Determine how related data is connected in the response
## Query Execution Process
### How Data is Organized
HyperSync organizes blockchain data into groups of contiguous blocks. When executing a query:
1. The server identifies which block group contains the starting block
2. It processes data groups sequentially until it hits a limit
3. Results are returned along with a `next_block` value for pagination
### Query Limits
HyperSync enforces several types of limits to ensure efficient query execution:
| Limit Type | Description | Behavior |
| ----------------- | ------------------------------------------------ | --------------------------------------------------------- |
| **Time** | Server-configured maximum execution time | May slightly exceed limit to complete current block group |
| **Response Size** | Maximum data returned | May slightly exceed limit to complete current block group |
| **to_block** | User-specified ending block (exclusive) | Never exceeds this limit |
| **max*num*\*** | User-specified maximum number of results by type | May slightly exceed limit to complete current block group |
### Execution Steps
1. Server receives query and identifies the starting block group
2. It scans each block group, applying selection criteria
3. It joins related data according to the specified join mode
4. When a limit is reached, it finishes processing the current block group
5. It returns results with pagination information
### Understanding Pagination
HyperSync uses a time-based pagination model that differs from traditional RPC calls:
- By default, HyperSync has a **5-second query execution limit**
- Within this time window, it processes as many blocks as possible
- For example, starting with `from_block: 0` might progress to block 10 million in a single request
- Each response includes a `next_block` value indicating where to resume for the next query
- This differs from RPC calls where you typically specify fixed block ranges (e.g., 0-1000)
#### Understanding nextBlock
`nextBlock` is the block number immediately after the last block included in the response. Use it as the `fromBlock` of your next query if you want to continue scanning. Resuming from `nextBlock` gives you a continuous, non-overlapping scan—no gaps, no duplicates.
**Usage pattern:** Call `get` or `getEvents`, process the page, then if `nextBlock` is less than your desired end (`toBlock` or `archiveHeight`), set `fromBlock = nextBlock` and repeat:
```javascript
let query = { fromBlock: 0, logs: [...], fieldSelection: {...} };
while (true) {
const res = await client.get(query);
// Process res.data...
const targetEnd = query.toBlock ?? res.archiveHeight;
if (res.nextBlock >= targetEnd) break;
query = { ...query, fromBlock: res.nextBlock };
}
```
For most use cases, the `stream` function handles pagination automatically, making it the recommended approach for processing large ranges of blocks.
### Reverse Search
HyperSync supports searching from the head of the chain backwards, which is useful for:
- Block explorers showing the most recent activity
- UIs displaying latest transactions for a user
- Any use case where recent data is more relevant
To use reverse search, add the `reverse: true` parameter to your stream call:
```javascript
// Example of reverse search to get recent transactions
const receiver = await client.stream(query, { reverse: true });
let count = 0;
while (true) {
let res = await receiver.recv();
if (res === null) {
break;
}
for (const tx of res.data.transactions) {
console.log(JSON.stringify(tx, null, 2));
}
count += res.data.transactions.length;
if (count >= 20) {
break;
}
}
```
With reverse search, HyperSync starts from the latest block and works backwards, allowing you to efficiently access the most recent blockchain data first.
## Query Structure Reference
A complete HyperSync query can include the following components:
### Core Query Parameters
```rust
struct Query {
/// The block to start the query from
from_block: u64,
/// The block to end the query at (exclusive)
/// If not specified, the query runs until the end of available data
to_block: Optional,
/// Log selection criteria (OR relationship between selections)
logs: Array,
/// Transaction selection criteria (OR relationship between selections)
transactions: Array,
/// Trace selection criteria (OR relationship between selections)
traces: Array,
/// Whether to include all blocks in the requested range
/// Default: only return blocks related to matched transactions/logs
include_all_blocks: bool,
/// Fields to include in the response
field_selection: FieldSelection,
/// Maximum results limits (approximate)
max_num_blocks: Optional,
max_num_transactions: Optional,
max_num_logs: Optional,
max_num_traces: Optional,
/// Data relationship model (Default, JoinAll, or JoinNothing)
join_mode: JoinMode,
}
```
### Selection Types
#### Log Selection
```rust
struct LogSelection {
/// Contract addresses to match (empty = match all)
address: Array,
/// Topics to match by position (empty = match all)
/// Each array element corresponds to a topic position (0-3)
/// Within each position, any matching value will satisfy the condition
topics: Array>,
}
```
#### Transaction Selection
```rust
struct TransactionSelection {
/// Sender addresses (empty = match all)
/// Has AND relationship with 'to' field
from: Array,
/// Recipient addresses (empty = match all)
/// Has AND relationship with 'from' field
to: Array,
/// Method signatures to match (first 4 bytes of input)
sighash: Array,
/// Transaction status to match (1 = success, 0 = failure)
status: Optional,
/// Transaction types to match (e.g., 0 = legacy, 2 = EIP-1559)
type: Array,
/// Created contract addresses to match
contract_address: Array,
}
```
#### Block Selection
```rust
struct BlockSelection {
/// Block hashes to match (empty = match all)
hash: Array,
/// Miner/validator addresses to match (empty = match all)
miner: Array,
}
```
#### Trace Selection
```rust
struct TraceSelection {
/// Sender addresses (empty = match all)
/// Has AND relationship with 'to' field
from: Array,
/// Recipient addresses (empty = match all)
/// Has AND relationship with 'from' field
to: Array,
/// Created contract addresses to match
address: Array,
/// Call types to match (e.g., "call", "delegatecall")
call_type: Array,
/// Reward types to match (e.g., "block", "uncle")
reward_type: Array,
/// Trace types to match (e.g., "call", "create", "suicide", "reward")
kind: Array,
/// Method signatures to match (first 4 bytes of input)
sighash: Array,
}
```
#### Field Selection
```rust
struct FieldSelection {
/// Block fields to include in response
block: Array,
/// Transaction fields to include in response
transaction: Array,
/// Log fields to include in response
log: Array,
/// Trace fields to include in response
trace: Array,
}
```
## Data Schema
HyperSync organizes blockchain data into four main tables. Below are the available fields for each table.
:::info Field Naming
When specifying fields in your query, always use snake_case names (e.g., `block_number`, not `blockNumber`).
:::
### Block Fields
```python
class BlockField(StrEnum):
# Fields present on all EVM chains
NUMBER = 'number' # Block number
HASH = 'hash' # Block hash
PARENT_HASH = 'parent_hash' # Parent block hash
SHA3_UNCLES = 'sha3_uncles' # SHA3 of uncles data
LOGS_BLOOM = 'logs_bloom' # Bloom filter for logs
TRANSACTIONS_ROOT = 'transactions_root' # Root of transaction trie
STATE_ROOT = 'state_root' # Root of state trie
RECEIPTS_ROOT = 'receipts_root' # Root of receipts trie
MINER = 'miner' # Miner/validator address
EXTRA_DATA = 'extra_data' # Extra data field
SIZE = 'size' # Block size in bytes
GAS_LIMIT = 'gas_limit' # Block gas limit
GAS_USED = 'gas_used' # Total gas used in block
TIMESTAMP = 'timestamp' # Block timestamp (Unix time)
# Optional fields — not present on all EVM chains (may be null)
NONCE = 'nonce' # Block nonce (absent on some L2s)
DIFFICULTY = 'difficulty' # Block difficulty (PoW chains only)
TOTAL_DIFFICULTY = 'total_difficulty' # Total chain difficulty (PoW chains only)
UNCLES = 'uncles' # Uncle block hashes (absent on some L2s)
MIX_HASH = 'mix_hash' # Mix hash (absent on some L2s)
BASE_FEE_PER_GAS = 'base_fee_per_gas' # EIP-1559 base fee (post-London chains only)
BLOB_GAS_USED = 'blob_gas_used' # Total blob gas used (EIP-4844 chains only)
EXCESS_BLOB_GAS = 'excess_blob_gas' # Excess blob gas (EIP-4844 chains only)
PARENT_BEACON_BLOCK_ROOT = 'parent_beacon_block_root' # Parent beacon block root (EIP-4844 chains only)
WITHDRAWALS_ROOT = 'withdrawals_root' # Root of withdrawals trie (post-Shanghai chains only)
WITHDRAWALS = 'withdrawals' # Validator withdrawals (post-Shanghai chains only)
L1_BLOCK_NUMBER = 'l1_block_number' # L1 block number (Arbitrum only)
SEND_COUNT = 'send_count' # Send count (Arbitrum only)
SEND_ROOT = 'send_root' # Send root (Arbitrum only)
```
### Transaction Fields
```python
class TransactionField(StrEnum):
# Block-related fields
BLOCK_HASH = 'block_hash' # The Keccak 256-bit hash of the block
BLOCK_NUMBER = 'block_number' # Block number containing the transaction
# Transaction identifiers
HASH = 'hash' # Transaction hash (keccak hash of RLP encoded signed transaction)
TRANSACTION_INDEX = 'transaction_index' # Index of the transaction in the block
# Transaction participants
FROM = 'from' # 160-bit address of the sender
TO = 'to' # 160-bit address of the recipient (null for contract creation)
# Gas information
GAS = 'gas' # Gas limit set by sender
GAS_PRICE = 'gas_price' # Wei paid per unit of gas
GAS_USED = 'gas_used' # Actual gas used by the transaction
CUMULATIVE_GAS_USED = 'cumulative_gas_used' # Total gas used in the block up to this transaction
EFFECTIVE_GAS_PRICE = 'effective_gas_price' # Sum of base fee and tip paid per unit of gas
# EIP-1559 fields
MAX_PRIORITY_FEE_PER_GAS = 'max_priority_fee_per_gas' # Max priority fee (a.k.a. GasTipCap)
MAX_FEE_PER_GAS = 'max_fee_per_gas' # Max fee per gas (a.k.a. GasFeeCap)
# Transaction data
INPUT = 'input' # Transaction input data or contract initialization code
VALUE = 'value' # Amount of ETH transferred in wei
NONCE = 'nonce' # Number of transactions sent by the sender
# Signature fields
V = 'v' # Replay protection value (based on chain_id)
R = 'r' # The R field of the signature
S = 's' # The S field of the signature
Y_PARITY = 'y_parity' # Signature Y parity
CHAIN_ID = 'chain_id' # Chain ID for replay protection (EIP-155)
# Contract-related fields
CONTRACT_ADDRESS = 'contract_address' # Address of created contract (for contract creation txs)
# Transaction result fields
STATUS = 'status' # Success (1) or failure (0)
LOGS_BLOOM = 'logs_bloom' # Bloom filter for logs produced by this transaction
ROOT = 'root' # State root (pre-Byzantium)
# EIP-2930 fields
ACCESS_LIST = 'access_list' # List of addresses and storage keys to pre-warm
# EIP-4844 (blob transactions) fields
MAX_FEE_PER_BLOB_GAS = 'max_fee_per_blob_gas' # Max fee per data gas (blob fee cap)
BLOB_VERSIONED_HASHES = 'blob_versioned_hashes' # List of blob versioned hashes
# Transaction type
KIND = 'type' # Transaction type (0=legacy, 1=EIP-2930, 2=EIP-1559, 3=EIP-4844, 4=EIP-7702) # note - in old versions of the clients this was called 'kind', in newer versions its called 'type'
# L2-specific fields (for rollups)
L1_FEE = 'l1_fee' # Fee for L1 data (L1GasPrice × L1GasUsed)
L1_GAS_PRICE = 'l1_gas_price' # Gas price on L1
L1_GAS_USED = 'l1_gas_used' # Amount of gas consumed on L1
L1_FEE_SCALAR = 'l1_fee_scalar' # Multiplier for L1 fee calculation
GAS_USED_FOR_L1 = 'gas_used_for_l1' # Gas spent on L1 calldata in L2 gas units
```
### Log Fields
```python
class LogField(StrEnum):
# Log identification
LOG_INDEX = 'log_index' # Index of the log in the block
TRANSACTION_INDEX = 'transaction_index' # Index of the transaction in the block
# Transaction information
TRANSACTION_HASH = 'transaction_hash' # Hash of the transaction that created this log
# Block information
BLOCK_HASH = 'block_hash' # Hash of the block containing this log
BLOCK_NUMBER = 'block_number' # Block number containing this log
# Log content
ADDRESS = 'address' # Contract address that emitted the event
DATA = 'data' # Non-indexed data from the event
# Topics (indexed parameters)
TOPIC0 = 'topic0' # Event signature hash
TOPIC1 = 'topic1' # First indexed parameter
TOPIC2 = 'topic2' # Second indexed parameter
TOPIC3 = 'topic3' # Third indexed parameter
# Reorg information
REMOVED = 'removed' # True if log was removed due to chain reorganization
```
### Trace Fields
```python
class TraceField(StrEnum):
# Trace identification
TRANSACTION_HASH = 'transaction_hash' # Hash of the transaction
TRANSACTION_POSITION = 'transaction_position' # Index of the transaction in the block
SUBTRACES = 'subtraces' # Number of sub-traces created during execution
TRACE_ADDRESS = 'trace_address' # Array indicating position in the trace tree
# Block information
BLOCK_HASH = 'block_hash' # Hash of the block containing this trace
BLOCK_NUMBER = 'block_number' # Block number containing this trace
# Transaction participants
FROM = 'from' # Address of the sender
TO = 'to' # Address of the recipient (null for contract creation)
# Value and gas
VALUE = 'value' # ETH value transferred (in wei)
GAS = 'gas' # Gas limit
GAS_USED = 'gas_used' # Gas actually used
# Call data
INPUT = 'input' # Call data for function calls
INIT = 'init' # Initialization code for contract creation
OUTPUT = 'output' # Return data from the call
# Contract information
ADDRESS = 'address' # Contract address (for creation/destruction)
CODE = 'code' # Contract code
# Trace types and categorization
TYPE = 'type' # Trace type (call, create, suicide, reward)
CALL_TYPE = 'call_type' # Call type (call, delegatecall, staticcall, etc.)
REWARD_TYPE = 'reward_type' # Reward type (block, uncle)
# Other actors
AUTHOR = 'author' # Address of receiver for reward transactions
# Result information
ERROR = 'error' # Error message if failed
```
For a complete list of all available fields, refer to the [HyperSync API Reference](https://docs.envio.dev/docs/HyperSync/hypersync-query).
## Response Structure
When you execute a HyperSync query, the response includes both metadata and the requested data:
```rust
struct QueryResponse {
/// Current height of the blockchain in HyperSync
archive_height: Optional,
/// Block number immediately after the last block included in this response.
/// Use as from_block in your next query for pagination.
next_block: u64,
/// Query execution time in milliseconds
total_execution_time: u64,
/// The actual blockchain data matching your query
data: ResponseData,
/// Information to help handle chain reorganizations
rollback_guard: Optional,
}
```
The `next_block` value tells you where to resume scanning. See [Understanding nextBlock](#understanding-nextblock) for a clear definition and usage pattern.
### Rollback Guard
The optional `rollback_guard` lets you detect chain reorganizations (reorgs) between successive queries, so you can re-fetch any data that has become stale.
```rust
struct RollbackGuard {
/// Last block scanned in this query
block_number: u64,
/// Timestamp of the last block scanned
timestamp: i64,
/// Hash of the last block scanned
hash: Hash,
/// First block scanned in this query
first_block_number: u64,
/// Parent hash of the first block scanned
first_parent_hash: Hash,
}
```
The guard is `Optional`: it is present whenever the response covers blocks near the chain tip (where reorgs can still happen) and absent for queries that return no data.
#### How HyperSync handles reorgs internally
As HyperSync ingests new blocks it checks each block's `parent_hash` against the previous block's `hash`. When a mismatch is detected, HyperSync re-syncs the affected blocks and continues serving the canonical chain.
A single query response is always internally consistent: you will never receive a mix of blocks from different forks. The rollback guard exists to detect reorgs that happen _between_ successive queries, where data you fetched earlier may now be stale.
#### Detecting a reorg
After each query, store the guard's `block_number` and `hash`. On the next query, compare:
- `previous response.hash` (last block you saw)
- `next response.first_parent_hash` (parent of the first block in the new batch)
If they match, the chain is intact. If they differ, a reorg occurred somewhere between the two queries.
```
Query N: rollback_guard.hash = 0xABC... (stored)
Query N+1: rollback_guard.first_parent_hash = 0xABC... match -> no reorg
= 0xDEF... mismatch -> reorg
```
#### Recovering from a reorg
The guard tells you _that_ a reorg happened but not how deep. To find the depth, keep enough history to cover your chain's reorg threshold (for example, 200 blocks for Polygon) and walk backwards: re-fetch each stored block's hash and compare. The first block whose hash still matches is the last canonical block; rewind your downstream state to there and resume querying.
```python
history = [] # list of (block_number, hash)
while True:
res = client.get(query)
guard = res.rollback_guard
if guard is None:
process(res.data)
query.from_block = res.next_block
continue
if history and guard.first_parent_hash != history[-1][1]:
# Walk back to find the last block still on chain.
while history:
block_num, stored_hash = history[-1]
if client.get_block_hash(block_num) == stored_hash:
break
history.pop()
rewind_to = history[-1][0] + 1 if history else query.from_block
rollback_state_to(rewind_to)
query.from_block = rewind_to
continue
process(res.data)
history.append((guard.block_number, guard.hash))
cutoff = guard.block_number - REORG_THRESHOLD
history = [(b, h) for b, h in history if b >= cutoff]
query.from_block = res.next_block
```
:::tip Consider HyperIndex
[HyperIndex](/docs/HyperIndex/overview) handles all of this for you: it tracks recent block hashes, locates the reorg point, and rolls back database state automatically. See [Reorgs Support](/docs/HyperIndex/reorgs-support) for details.
:::
## Stream and Collect Functions
For continuous data processing or building data pipelines, client libraries provide `stream` and `collect` functions that wrap the base query functionality.
:::caution Tip of Chain Warning
These functions are not designed for use at the blockchain tip where rollbacks may occur. For real-time data near the chain tip, implement a custom loop using the `get` functions and handle rollbacks manually.
:::
### Stream Function
The `stream` function:
- Runs multiple queries concurrently
- Returns a stream handle that yields results as they're available
- Optimizes performance through pipelined decoding/decompression
- Continues until reaching either `to_block` or the chain height at stream start
### Collect Functions
The `collect` functions:
- Call `stream` internally and aggregate results
- Offer different output formats (JSON, Parquet)
- Handle data that may not fit in memory
:::warning Resource Management
Always call `close()` on stream handles when finished to prevent resource leaks, especially if creating multiple streams.
:::
## Working with Join Modes
HyperSync "joins" connect related blockchain data automatically. Unlike SQL joins that combine rows from different tables, HyperSync joins determine which related records to include in the response.
### Default Join Mode (logs → transactions → traces → blocks)
With the default join mode:
1. When you query logs, you automatically get their associated transactions
2. Those transactions' traces are also included
3. The blocks containing these transactions are included
```
┌───────┐ ┌───────────────┐ ┌───────┐ ┌───────┐
│ Logs │ ──> │ Transactions │ ──> │ Traces│ ──> │ Blocks│
└───────┘ └───────────────┘ └───────┘ └───────┘
```
### JoinAll Mode
JoinAll creates a more comprehensive network of related data:
```
┌─────────────────────────────┐
│ │
▼ │
┌───────┐ <──> ┌───────────────┐ <──> ┌───────┐ <──> ┌───────┐
│ Logs │ │ Transactions │ │ Traces│ │ Blocks│
└───────┘ └───────────────┘ └───────┘ └───────┘
```
For example, if you query a trace:
1. You get the transaction that created it
2. You get ALL logs from that transaction (not just the ones matching your criteria)
3. You get ALL traces from that transaction
4. You get the block containing the transaction
### JoinNothing Mode
JoinNothing is the most restrictive:
```
┌───────┐ ┌───────────────┐ ┌───────┐ ┌───────┐
│ Logs │ │ Transactions │ │ Traces│ │ Blocks│
└───────┘ └───────────────┘ └───────┘ └───────┘
```
Only data directly matching your selection criteria is returned, with no related records included.
## Best Practices
To get the most out of HyperSync queries:
1. **Minimize field selection** - Only request fields you actually need to improve performance
2. **Use appropriate limits** - Set `max_num_*` parameters to control response size
3. **Choose the right join mode** - Use `JoinNothing` for minimal data, `JoinAll` for complete context
4. **Process in chunks** - For large datasets, use pagination or the `stream` function
5. **Consider Parquet** - For analytical workloads, use `collect_parquet` for efficient storage
6. **Handle chain tip carefully** - Near the chain tip, implement custom rollback handling
---
# HyperSync Supported Networks
> See all networks currently supported by HyperSync, including reliability notes and the criteria we use when adding new chains.
:::note
We are rapidly adding new supported networks. If you don't see your network here or would like us to add a network to HyperSync, pop us a message in our [Discord](https://discord.gg/envio).
:::
:::info
The Tier is the level of support (and therefore reliability) based on the infrastructure running the chain. We are actively working to make the tier distinctions more clear and transparent to our users.
Currently, tiers relate to various service quality aspects including:
- Allocated resources and compute power
- Query processing speed
- Infrastructure redundancy
- Backup frequency and retention
- Multi-region availability
- Priority for upgrades and new features
- SLA guarantees
While detailed tier specifications are still being finalized, we're committed to providing transparent service level information in the near future.
If you are a network operator or user and would like improved service support or to discuss upgrading a chain's level of support, please reach out to us in [Discord](https://discord.gg/envio).
:::
| Network Name | Network ID | HyperSync URL | HyperRPC URL |
| ------------------------- | --------------- | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| Ab | 36888 | https://ab.hypersync.xyz or https://36888.hypersync.xyz | https://ab.rpc.hypersync.xyz or https://36888.rpc.hypersync.xyz |
| Abstract | 2741 | https://abstract.hypersync.xyz or https://2741.hypersync.xyz | https://abstract.rpc.hypersync.xyz or https://2741.rpc.hypersync.xyz |
| Arbitrum | 42161 | https://arbitrum.hypersync.xyz or https://42161.hypersync.xyz | https://arbitrum.rpc.hypersync.xyz or https://42161.rpc.hypersync.xyz |
| Arbitrum Nova | 42170 | https://arbitrum-nova.hypersync.xyz or https://42170.hypersync.xyz | https://arbitrum-nova.rpc.hypersync.xyz or https://42170.rpc.hypersync.xyz |
| Arbitrum Sepolia | 421614 | https://arbitrum-sepolia.hypersync.xyz or https://421614.hypersync.xyz | https://arbitrum-sepolia.rpc.hypersync.xyz or https://421614.rpc.hypersync.xyz |
| Arc Testnet | 5042002 | https://arc-testnet.hypersync.xyz or https://5042002.hypersync.xyz | https://arc-testnet.rpc.hypersync.xyz or https://5042002.rpc.hypersync.xyz |
| Aurora | 1313161554 | https://aurora.hypersync.xyz or https://1313161554.hypersync.xyz | https://aurora.rpc.hypersync.xyz or https://1313161554.rpc.hypersync.xyz |
| Avalanche | 43114 | https://avalanche.hypersync.xyz or https://43114.hypersync.xyz | https://avalanche.rpc.hypersync.xyz or https://43114.rpc.hypersync.xyz |
| Base | 8453 | https://base.hypersync.xyz or https://8453.hypersync.xyz | https://base.rpc.hypersync.xyz or https://8453.rpc.hypersync.xyz |
| Base Sepolia | 84532 | https://base-sepolia.hypersync.xyz or https://84532.hypersync.xyz | https://base-sepolia.rpc.hypersync.xyz or https://84532.rpc.hypersync.xyz |
| Base Traces* | 8453 | https://base-traces.hypersync.xyz or https://8453-traces.hypersync.xyz | https://base-traces.rpc.hypersync.xyz or https://8453-traces.rpc.hypersync.xyz |
| Berachain | 80094 | https://berachain.hypersync.xyz or https://80094.hypersync.xyz | https://berachain.rpc.hypersync.xyz or https://80094.rpc.hypersync.xyz |
| Blast | 81457 | https://blast.hypersync.xyz or https://81457.hypersync.xyz | https://blast.rpc.hypersync.xyz or https://81457.rpc.hypersync.xyz |
| Boba | 288 | https://boba.hypersync.xyz or https://288.hypersync.xyz | https://boba.rpc.hypersync.xyz or https://288.rpc.hypersync.xyz |
| Bsc | 56 | https://bsc.hypersync.xyz or https://56.hypersync.xyz | https://bsc.rpc.hypersync.xyz or https://56.rpc.hypersync.xyz |
| Bsc Testnet | 97 | https://bsc-testnet.hypersync.xyz or https://97.hypersync.xyz | https://bsc-testnet.rpc.hypersync.xyz or https://97.rpc.hypersync.xyz |
| Celo | 42220 | https://celo.hypersync.xyz or https://42220.hypersync.xyz | https://celo.rpc.hypersync.xyz or https://42220.rpc.hypersync.xyz |
| Chiliz | 88888 | https://chiliz.hypersync.xyz or https://88888.hypersync.xyz | https://chiliz.rpc.hypersync.xyz or https://88888.rpc.hypersync.xyz |
| Citrea | 4114 | https://citrea.hypersync.xyz or https://4114.hypersync.xyz | https://citrea.rpc.hypersync.xyz or https://4114.rpc.hypersync.xyz |
| Citrea Testnet | 5115 | https://citrea-testnet.hypersync.xyz or https://5115.hypersync.xyz | https://citrea-testnet.rpc.hypersync.xyz or https://5115.rpc.hypersync.xyz |
| Curtis | 33111 | https://curtis.hypersync.xyz or https://33111.hypersync.xyz | https://curtis.rpc.hypersync.xyz or https://33111.rpc.hypersync.xyz |
| Cyber | 7560 | https://cyber.hypersync.xyz or https://7560.hypersync.xyz | https://cyber.rpc.hypersync.xyz or https://7560.rpc.hypersync.xyz |
| Eth Traces | 1 | https://eth-traces.hypersync.xyz or https://1-traces.hypersync.xyz | https://eth-traces.rpc.hypersync.xyz or https://1-traces.rpc.hypersync.xyz |
| Ethereum Mainnet | 1 | https://eth.hypersync.xyz or https://1.hypersync.xyz | https://eth.rpc.hypersync.xyz or https://1.rpc.hypersync.xyz |
| Etherlink | 42793 | https://etherlink.hypersync.xyz or https://42793.hypersync.xyz | https://etherlink.rpc.hypersync.xyz or https://42793.rpc.hypersync.xyz |
| Fantom | 250 | https://fantom.hypersync.xyz or https://250.hypersync.xyz | https://fantom.rpc.hypersync.xyz or https://250.rpc.hypersync.xyz |
| Flare | 14 | https://flare.hypersync.xyz or https://14.hypersync.xyz | https://flare.rpc.hypersync.xyz or https://14.rpc.hypersync.xyz |
| Fraxtal | 252 | https://fraxtal.hypersync.xyz or https://252.hypersync.xyz | https://fraxtal.rpc.hypersync.xyz or https://252.rpc.hypersync.xyz |
| Fuji | 43113 | https://fuji.hypersync.xyz or https://43113.hypersync.xyz | https://fuji.rpc.hypersync.xyz or https://43113.rpc.hypersync.xyz |
| Gnosis | 100 | https://gnosis.hypersync.xyz or https://100.hypersync.xyz | https://gnosis.rpc.hypersync.xyz or https://100.rpc.hypersync.xyz |
| Gnosis Chiado | 10200 | https://gnosis-chiado.hypersync.xyz or https://10200.hypersync.xyz | https://gnosis-chiado.rpc.hypersync.xyz or https://10200.rpc.hypersync.xyz |
| Harmony Shard 0 | 1666600000 | https://harmony-shard-0.hypersync.xyz or https://1666600000.hypersync.xyz | https://harmony-shard-0.rpc.hypersync.xyz or https://1666600000.rpc.hypersync.xyz |
| Hoodi | 560048 | https://hoodi.hypersync.xyz or https://560048.hypersync.xyz | https://hoodi.rpc.hypersync.xyz or https://560048.rpc.hypersync.xyz |
| Hyperliquid | 999 | https://hyperliquid.hypersync.xyz or https://999.hypersync.xyz | https://hyperliquid.rpc.hypersync.xyz or https://999.rpc.hypersync.xyz |
| Injective* | 1776 | https://injective.hypersync.xyz or https://1776.hypersync.xyz | https://injective.rpc.hypersync.xyz or https://1776.rpc.hypersync.xyz |
| Ink | 57073 | https://ink.hypersync.xyz or https://57073.hypersync.xyz | https://ink.rpc.hypersync.xyz or https://57073.rpc.hypersync.xyz |
| Katana | 747474 | https://katana.hypersync.xyz or https://747474.hypersync.xyz | https://katana.rpc.hypersync.xyz or https://747474.rpc.hypersync.xyz |
| Kroma | 255 | https://kroma.hypersync.xyz or https://255.hypersync.xyz | https://kroma.rpc.hypersync.xyz or https://255.rpc.hypersync.xyz |
| Linea | 59144 | https://linea.hypersync.xyz or https://59144.hypersync.xyz | https://linea.rpc.hypersync.xyz or https://59144.rpc.hypersync.xyz |
| Lisk | 1135 | https://lisk.hypersync.xyz or https://1135.hypersync.xyz | https://lisk.rpc.hypersync.xyz or https://1135.rpc.hypersync.xyz |
| Lukso | 42 | https://lukso.hypersync.xyz or https://42.hypersync.xyz | https://lukso.rpc.hypersync.xyz or https://42.rpc.hypersync.xyz |
| Lukso Testnet | 4201 | https://lukso-testnet.hypersync.xyz or https://4201.hypersync.xyz | https://lukso-testnet.rpc.hypersync.xyz or https://4201.rpc.hypersync.xyz |
| Manta | 169 | https://manta.hypersync.xyz or https://169.hypersync.xyz | https://manta.rpc.hypersync.xyz or https://169.rpc.hypersync.xyz |
| Mantle | 5000 | https://mantle.hypersync.xyz or https://5000.hypersync.xyz | https://mantle.rpc.hypersync.xyz or https://5000.rpc.hypersync.xyz |
| Megaeth | 4326 | https://megaeth.hypersync.xyz or https://4326.hypersync.xyz | https://megaeth.rpc.hypersync.xyz or https://4326.rpc.hypersync.xyz |
| Megaeth Testnet2 | 6343 | https://megaeth-testnet2.hypersync.xyz or https://6343.hypersync.xyz | https://megaeth-testnet2.rpc.hypersync.xyz or https://6343.rpc.hypersync.xyz |
| Merlin | 4200 | https://merlin.hypersync.xyz or https://4200.hypersync.xyz | https://merlin.rpc.hypersync.xyz or https://4200.rpc.hypersync.xyz |
| Metall2 | 1750 | https://metall2.hypersync.xyz or https://1750.hypersync.xyz | https://metall2.rpc.hypersync.xyz or https://1750.rpc.hypersync.xyz |
| Mode | 34443 | https://mode.hypersync.xyz or https://34443.hypersync.xyz | https://mode.rpc.hypersync.xyz or https://34443.rpc.hypersync.xyz |
| Monad | 143 | https://monad.hypersync.xyz or https://143.hypersync.xyz | https://monad.rpc.hypersync.xyz or https://143.rpc.hypersync.xyz |
| Monad Testnet | 10143 | https://monad-testnet.hypersync.xyz or https://10143.hypersync.xyz | https://monad-testnet.rpc.hypersync.xyz or https://10143.rpc.hypersync.xyz |
| Moonbeam | 1284 | https://moonbeam.hypersync.xyz or https://1284.hypersync.xyz | https://moonbeam.rpc.hypersync.xyz or https://1284.rpc.hypersync.xyz |
| Morph | 2818 | https://morph.hypersync.xyz or https://2818.hypersync.xyz | https://morph.rpc.hypersync.xyz or https://2818.rpc.hypersync.xyz |
| Opbnb | 204 | https://opbnb.hypersync.xyz or https://204.hypersync.xyz | https://opbnb.rpc.hypersync.xyz or https://204.rpc.hypersync.xyz |
| Optimism | 10 | https://optimism.hypersync.xyz or https://10.hypersync.xyz | https://optimism.rpc.hypersync.xyz or https://10.rpc.hypersync.xyz |
| Optimism Sepolia | 11155420 | https://optimism-sepolia.hypersync.xyz or https://11155420.hypersync.xyz | https://optimism-sepolia.rpc.hypersync.xyz or https://11155420.rpc.hypersync.xyz |
| Plasma | 9745 | https://plasma.hypersync.xyz or https://9745.hypersync.xyz | https://plasma.rpc.hypersync.xyz or https://9745.rpc.hypersync.xyz |
| Plume | 98866 | https://plume.hypersync.xyz or https://98866.hypersync.xyz | https://plume.rpc.hypersync.xyz or https://98866.rpc.hypersync.xyz |
| Polygon | 137 | https://polygon.hypersync.xyz or https://137.hypersync.xyz | https://polygon.rpc.hypersync.xyz or https://137.rpc.hypersync.xyz |
| Polygon Amoy | 80002 | https://polygon-amoy.hypersync.xyz or https://80002.hypersync.xyz | https://polygon-amoy.rpc.hypersync.xyz or https://80002.rpc.hypersync.xyz |
| Polygon zkEVM | 1101 | https://polygon-zkevm.hypersync.xyz or https://1101.hypersync.xyz | https://polygon-zkevm.rpc.hypersync.xyz or https://1101.rpc.hypersync.xyz |
| Robinhood | 4663 | https://robinhood.hypersync.xyz or https://4663.hypersync.xyz | https://robinhood.rpc.hypersync.xyz or https://4663.rpc.hypersync.xyz |
| Rootstock | 30 | https://rootstock.hypersync.xyz or https://30.hypersync.xyz | https://rootstock.rpc.hypersync.xyz or https://30.rpc.hypersync.xyz |
| Scroll | 534352 | https://scroll.hypersync.xyz or https://534352.hypersync.xyz | https://scroll.rpc.hypersync.xyz or https://534352.rpc.hypersync.xyz |
| Sei* | 1329 | https://sei.hypersync.xyz or https://1329.hypersync.xyz | https://sei.rpc.hypersync.xyz or https://1329.rpc.hypersync.xyz |
| Sei Testnet* | 1328 | https://sei-testnet.hypersync.xyz or https://1328.hypersync.xyz | https://sei-testnet.rpc.hypersync.xyz or https://1328.rpc.hypersync.xyz |
| Sepolia | 11155111 | https://sepolia.hypersync.xyz or https://11155111.hypersync.xyz | https://sepolia.rpc.hypersync.xyz or https://11155111.rpc.hypersync.xyz |
| Shimmer Evm | 148 | https://shimmer-evm.hypersync.xyz or https://148.hypersync.xyz | https://shimmer-evm.rpc.hypersync.xyz or https://148.rpc.hypersync.xyz |
| Soneium | 1868 | https://soneium.hypersync.xyz or https://1868.hypersync.xyz | https://soneium.rpc.hypersync.xyz or https://1868.rpc.hypersync.xyz |
| Sonic | 146 | https://sonic.hypersync.xyz or https://146.hypersync.xyz | https://sonic.rpc.hypersync.xyz or https://146.rpc.hypersync.xyz |
| Sonic Testnet | 14601 | https://sonic-testnet.hypersync.xyz or https://14601.hypersync.xyz | https://sonic-testnet.rpc.hypersync.xyz or https://14601.rpc.hypersync.xyz |
| Sophon | 50104 | https://sophon.hypersync.xyz or https://50104.hypersync.xyz | https://sophon.rpc.hypersync.xyz or https://50104.rpc.hypersync.xyz |
| Stable | 988 | Access on request — [contact us](https://discord.gg/envio) | Access on request — [contact us](https://discord.gg/envio) |
| Superseed | 5330 | https://superseed.hypersync.xyz or https://5330.hypersync.xyz | https://superseed.rpc.hypersync.xyz or https://5330.rpc.hypersync.xyz |
| Swell | 1923 | https://swell.hypersync.xyz or https://1923.hypersync.xyz | https://swell.rpc.hypersync.xyz or https://1923.rpc.hypersync.xyz |
| Tempo | 4217 | https://tempo.hypersync.xyz or https://4217.hypersync.xyz | https://tempo.rpc.hypersync.xyz or https://4217.rpc.hypersync.xyz |
| Tron | 728126428 | https://tron.hypersync.xyz or https://728126428.hypersync.xyz | https://tron.rpc.hypersync.xyz or https://728126428.rpc.hypersync.xyz |
| Unichain | 130 | https://unichain.hypersync.xyz or https://130.hypersync.xyz | https://unichain.rpc.hypersync.xyz or https://130.rpc.hypersync.xyz |
| Worldchain | 480 | https://worldchain.hypersync.xyz or https://480.hypersync.xyz | https://worldchain.rpc.hypersync.xyz or https://480.rpc.hypersync.xyz |
| XDC | 50 | Access on request — [contact us](https://discord.gg/envio) | Access on request — [contact us](https://discord.gg/envio) |
| XDC Testnet | 51 | Access on request — [contact us](https://discord.gg/envio) | Access on request — [contact us](https://discord.gg/envio) |
| Zeta | 7000 | https://zeta.hypersync.xyz or https://7000.hypersync.xyz | https://zeta.rpc.hypersync.xyz or https://7000.rpc.hypersync.xyz |
| Zircuit | 48900 | Access on request — [contact us](https://discord.gg/envio) | Access on request — [contact us](https://discord.gg/envio) |
| ZKsync | 324 | https://zksync.hypersync.xyz or https://324.hypersync.xyz | https://zksync.rpc.hypersync.xyz or https://324.rpc.hypersync.xyz |
| Zora | 7777777 | https://zora.hypersync.xyz or https://7777777.hypersync.xyz | https://zora.rpc.hypersync.xyz or https://7777777.rpc.hypersync.xyz |
**Notes:**
- **Base Traces***: Start block: 39000000 (earlier blocks available on request)
- **Injective***: Start block: 129846180 (non-evm before that)
- **Sei***: Start block: 79123881 (non-evm before that)
- **Sei Testnet***: Start block: 186100000 (non-evm before that)
---
---
# Using HyperSync
> Learn how to fetch, filter, and decode blockchain data using HyperSync.
# Getting Started with HyperSync
:::tip Easiest way to get started: the Query Builder
The **[HyperSync Query Builder](https://builder.hypersync.xyz)** lets you construct and run queries directly in your browser — no install, no code required. It's the fastest way to get familiar with HyperSync and see what's possible.
**[Open builder.hypersync.xyz →](https://builder.hypersync.xyz)**
:::
HyperSync is Envio's high-performance blockchain data engine that provides up to 2000x faster access to blockchain data compared to traditional RPC endpoints. This guide will help you understand how to effectively use HyperSync in your applications.
## Quick Start Video
Watch this quick tutorial to see HyperSync in action:
## Core Concepts
HyperSync revolves around two main concepts:
1. **Queries** - Define what blockchain data you want to retrieve
2. **Output Configuration** - Specify how you want that data formatted and delivered
Think of queries as your data filter and the output configuration as your data processor.
## Building Effective Queries
Queries are the heart of working with HyperSync. They allow you to filter for specific blocks, logs, transactions, and traces.
### Query Structure
A basic HyperSync query contains:
```python
query = hypersync.Query(
from_block=12345678, # Required: Starting block number
to_block=12345778, # Optional: Ending block number
field_selection=field_selection, # Required: What fields to return
logs=[log_selection], # Optional: Filter for specific logs
transactions=[tx_selection], # Optional: Filter for specific transactions
traces=[trace_selection], # Optional: Filter for specific traces
include_all_blocks=False, # Optional: Include blocks with no matches
max_num_blocks=1000, # Optional: Limit number of blocks processed
max_num_transactions=5000, # Optional: Limit number of transactions processed
max_num_logs=5000, # Optional: Limit number of logs processed
max_num_traces=5000 # Optional: Limit number of traces processed
)
```
### Field Selection
Field selection allows you to specify exactly which data fields you want to retrieve. This improves performance by only fetching what you need:
```python
field_selection = hypersync.FieldSelection(
# Block fields you want to retrieve
block=[
BlockField.NUMBER,
BlockField.TIMESTAMP,
BlockField.HASH
],
# Transaction fields you want to retrieve
transaction=[
TransactionField.HASH,
TransactionField.FROM,
TransactionField.TO,
TransactionField.VALUE
],
# Log fields you want to retrieve
log=[
LogField.ADDRESS,
LogField.TOPIC0,
LogField.TOPIC1,
LogField.TOPIC2,
LogField.TOPIC3,
LogField.DATA,
LogField.TRANSACTION_HASH
],
# Trace fields you want to retrieve (if applicable)
trace=[
TraceField.ACTION_FROM,
TraceField.ACTION_TO,
TraceField.ACTION_VALUE
]
)
```
### Filtering for Specific Data
For most use cases, you'll want to filter for specific logs, transactions, or traces:
#### Log Selection Example
```python
# Filter for Transfer events from USDC contract
log_selection = hypersync.LogSelection(
address=["0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48"], # USDC contract
topics=[
["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"] # Transfer event signature
]
)
```
#### Transaction Selection Example
```python
# Filter for transactions to the Uniswap V3 router
tx_selection = hypersync.TransactionSelection(
to=["0xE592427A0AEce92De3Edee1F18E0157C05861564"] # Uniswap V3 Router
)
```
## Processing the Results
HyperSync provides multiple ways to process query results:
### Stream to Parquet Files
Parquet is the recommended format for large data sets:
```python
# Configure output format
config = hypersync.StreamConfig(
hex_output=hypersync.HexOutput.PREFIXED,
event_signature="Transfer(address indexed from, address indexed to, uint256 value)"
)
# Stream results to a Parquet file
await client.collect_parquet("data_directory", query, config)
```
### Stream to JSON Files
For smaller datasets or debugging:
```python
# Stream results to JSON
await client.collect_json("output.json", query, config)
```
### Process Data in Memory
For immediate processing:
```python
# Process data directly
async for result in client.stream(query, config):
for log in result.logs:
# Process each log
print(f"Transfer from {log.event_params['from']} to {log.event_params['to']}")
```
## Tips and Best Practices
### Performance Optimization
- **Use Appropriate Batch Sizes**: Adjust batch size based on your chain and use case:
```python
config = hypersync.ParquetConfig(
path="data",
hex_output=hypersync.HexOutput.PREFIXED,
batch_size=1000000, # Process 1M blocks at a time
concurrency=10, # Use 10 concurrent workers
)
```
- **Enable Trace Logs**: Set `RUST_LOG=trace` to see detailed progress:
```bash
export RUST_LOG=trace
```
- **Paginate Large Queries**: HyperSync requests have a 5-second time limit. For large data sets, paginate results:
```python
current_block = start_block
while current_block < end_block:
query.from_block = current_block
query.to_block = min(current_block + 1000000, end_block)
result = await client.collect_parquet("data", query, config)
current_block = result.end_block + 1
```
### Network-Specific Considerations
- **High-Volume Networks**: For networks like Ethereum Mainnet, use smaller block ranges or more specific filters
- **Low-Volume Networks**: For smaller chains, you can process the entire chain in one query
## Complete Example
Here's a complete example that fetches all USDC Transfer events:
```python
import hypersync
from hypersync import (
LogSelection,
LogField,
BlockField,
FieldSelection,
TransactionField,
HexOutput
)
import asyncio
async def collect_usdc_transfers():
# Initialize client
client = hypersync.HypersyncClient(
hypersync.ClientConfig(
url="https://eth.hypersync.xyz",
bearer_token="your-token-here", # Get from https://docs.envio.dev/docs/HyperSync/api-tokens
)
)
# Define field selection
field_selection = hypersync.FieldSelection(
block=[BlockField.NUMBER, BlockField.TIMESTAMP],
transaction=[TransactionField.HASH],
log=[
LogField.ADDRESS,
LogField.TOPIC0,
LogField.TOPIC1,
LogField.TOPIC2,
LogField.DATA,
]
)
# Define query for USDC transfers
query = hypersync.Query(
from_block=12000000,
to_block=12100000,
field_selection=field_selection,
logs=[
LogSelection(
address=["0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48"], # USDC contract
topics=[
["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"] # Transfer signature
]
)
]
)
# Configure output
config = hypersync.StreamConfig(
hex_output=HexOutput.PREFIXED,
event_signature="Transfer(address indexed from, address indexed to, uint256 value)"
)
# Collect data to a Parquet file
result = await client.collect_parquet("usdc_transfers", query, config)
print(f"Processed blocks {query.from_block} to {result.end_block}")
asyncio.run(collect_usdc_transfers())
```
## Decoding Event Logs
When working with blockchain data, event logs contain encoded data that needs to be properly decoded to extract meaningful information. HyperSync provides powerful decoding capabilities to simplify this process.
### Understanding Log Structure
Event logs in Ethereum have the following structure:
- **Address**: The contract that emitted the event
- **Topic0**: The event signature hash (keccak256 of the event signature)
- **Topics 1-3**: Indexed parameters (up to 3)
- **Data**: Non-indexed parameters packed together
### Using the Decoder
HyperSync's client libraries include a `Decoder` class that can parse these raw logs into structured data:
```javascript
// Create a decoder with event signatures
const decoder = Decoder.fromSignatures([
"Transfer(address indexed from, address indexed to, uint256 amount)",
"Approval(address indexed owner, address indexed spender, uint256 amount)",
]);
// Decode logs
const decodedLogs = await decoder.decodeLogs(logs);
```
### Single vs. Multiple Event Types
HyperSync provides flexibility to decode different types of event logs:
- **Single Event Type**: For processing one type of event (e.g., only Swap events)
- See complete example: [run-decoder.js](https://github.com/enviodev/hypersync-quickstart/blob/main/run-decoder.js)
- **Multiple Event Types**: For processing different events from the same contract (e.g., Transfer and Approval)
- See complete example: [run-decoder-multi.js](https://github.com/enviodev/hypersync-quickstart/blob/main/run-decoder-multi.js)
### Working with Decoded Data
After decoding, you can access the log parameters in a structured way:
- **Indexed parameters**: Available in `decodedLog.indexed` array
- **Non-indexed parameters**: Available in `decodedLog.body` array
Each parameter object contains:
- **name**: The parameter name from the signature
- **type**: The Solidity type
- **val**: The actual value
For example, to access parameters from a Transfer event:
```javascript
// Access indexed parameters (from, to)
const from = decodedLog.indexed[0]?.val.toString();
const to = decodedLog.indexed[1]?.val.toString();
// Access non-indexed parameters (amount)
const amount = decodedLog.body[0]?.val.toString();
```
### Benefits of Using the Decoder
- **Type Safety**: Values are properly converted to their corresponding types
- **Simplified Access**: Direct access to named parameters
- **Batch Processing**: Decode multiple logs with a single call
- **Multiple Event Support**: Handle different event types in the same processing pipeline
## Next Steps
Now that you understand the basics of using HyperSync:
- Browse the [Python Client](./hypersync-clients.md) or other language-specific clients
- Learn about [advanced query options](./hypersync-query.md)
- See [example queries for common use cases](./hypersync-curl-examples.md)
- [Get your API token](./api-tokens.mdx) to start building
For detailed API references and examples in other languages, check our [client documentation](./hypersync-clients.md).
---
# HyperSync: Ultra-Fast Blockchain Data API
> HyperSync is Envio's ultra-fast blockchain data API, a Rust-built alternative to RPC with flexible queries across dozens of EVM chains and Fuel.
# HyperSync: Ultra-Fast & Flexible Data API
## What is HyperSync?
HyperSync is a purpose-built, high-performance data retrieval layer that gives developers unprecedented access to blockchain data. Built from the ground up in Rust, HyperSync serves as an alternative to traditional JSON-RPC endpoints, offering dramatically faster queries and more flexible data access patterns.
:::info HyperSync & HyperIndex
**HyperSync** is Envio's high-performance blockchain data engine that serves as a direct replacement for traditional RPC endpoints, delivering up to 2000x faster data access.
**HyperIndex** is built on top of HyperSync, providing a complete indexing framework with schema management, event handling, and GraphQL APIs.
Use HyperSync directly when you need raw blockchain data at maximum speed, or use HyperIndex when you need a full-featured indexing solution.
:::
## The Problem HyperSync Solves
Traditional blockchain data access through JSON-RPC faces several limitations:
- **Speed constraints**: Retrieving large amounts of historical data can take days
- **Query flexibility**: Complex data analysis requires many separate calls
- **Cost inefficiency**: Expensive for data-intensive applications
## Key Benefits
- **Exceptional Performance**: Retrieve and process blockchain data up to 1000x faster than traditional RPC methods
- **Comprehensive Coverage**: Access data across [ EVM chains](/docs/HyperSync/hypersync-supported-networks) and Fuel, with new networks added regularly
- **Flexible Query Capabilities**: Filter, select, and process exactly the data you need with powerful query options
- **Cost Efficiency**: Dramatically reduce infrastructure costs for data-intensive applications
- **Simple Integration**: Client libraries available for Python, Rust, Node.js, and Go
## Performance Benchmarks
HyperSync delivers transformative performance compared to traditional methods:
| Task | Traditional RPC | HyperSync | Improvement |
| ------------------------------------------------ | --------------- | --------- | ------------- |
| Scan Arbitrum blockchain for sparse log data | Hours/Days | 2 seconds | ~2000x faster |
| Fetch all Uniswap v3 PoolCreated events ethereum | Hours | Seconds | ~500x faster |
## Use Cases
HyperSync powers a wide range of blockchain applications, enabling developers to build tools that would be impractical with traditional data access methods:
### General Applications
- **Blockchain Indexers**: Build high-performance data indexers with minimal infrastructure
- **Data Analytics**: Perform complex on-chain analysis in seconds instead of days
- **Block Explorers**: Create responsive explorers with comprehensive data access
- **Monitoring Tools**: Track blockchain activity with near real-time updates
- **Cross-chain Applications**: Access unified data across multiple networks
- **ETL Pipelines**: Create pipelines to extract and save data fast
### Powered by HyperSync
#### [HyperIndex](https://docs.envio.dev/docs/HyperIndex/overview)
- **100x faster blockchain indexing** across EVM chains and Fuel
- **Powers 100 plus applications** like v4.xyz analytics
#### [ChainDensity.xyz](https://chaindensity.xyz)
- **Fast transaction/event density analysis** for any address
- **Generates insights in seconds** that would take hours with traditional methods
#### [Scope.sh](https://scope.sh)
- **Ultra-fast Account Abstraction (AA) focused block explorer**
- **Fast historical data retrieval** with minimal latency
#### [LogTUI](https://www.npmjs.com/package/logtui)
- **Terminal-based UI** for finding all historical blockchain events
- **Built-in presets** for 20+ protocols (Uniswap, Chainlink, Aave, ENS, etc.)
- **Try it**: `pnpx logtui aave arbitrum` to track Aave events on Arbitrum in your terminal
## See HyperSync in Action
## Next Steps
- **[Try the Quick Start Guide](/docs/HyperSync/hypersync-quickstart)** to get up and running in minutes
- **[Build queries visually](http://builder.hypersync.xyz)** with our Intuitive Query Builder
- [Get an API Token](/docs/HyperSync/api-tokens) to access HyperSync services
- [View Supported Networks](/docs/HyperSync/hypersync-supported-networks) to see available chains
- [Check Client Documentation](/docs/HyperSync/hypersync-clients) for language-specific guides
- [Join our Discord](https://discord.gg/envio) for support and updates
:::note
Our documentation is continuously improving! If you have questions or need assistance, please reach out in our [Discord community](https://discord.gg/envio).
:::
---
# HyperSync Quickstart
> Get started quickly with HyperSync to stream and filter blockchain events.
:::tip Easiest way to get started: the Query Builder
The **[HyperSync Query Builder](https://builder.hypersync.xyz)** lets you construct and run queries directly in your browser — no install, no code required. It's the fastest way to get familiar with HyperSync and see what's possible.
**[Open builder.hypersync.xyz →](https://builder.hypersync.xyz)**
:::
Get up and running with HyperSync in minutes. This guide will help you start accessing blockchain data at unprecedented speeds with minimal setup. For a conceptual overview before you dive in, see [What is HyperSync?](/blog/what-is-hypersync).
## Quickest Start: Try LogTUI
Want to see HyperSync in action with zero setup? Try LogTUI, a terminal-based blockchain event viewer:
```bash
# Monitor Aave events on Arbitrum (no installation needed)
pnpx logtui aave arbitrum
```
## Clone the Quickstart Repository
The fastest way to get started is to clone our minimal example repository:
```bash
git clone https://github.com/enviodev/hypersync-quickstart.git
cd hypersync-quickstart
```
This repository contains everything you need to start streaming blockchain data using HyperSync.
## Install Dependencies
```bash
# Using pnpm (recommended)
pnpm install
```
## Choose Your Adventure
The repository includes three different script options, all of which retrieve Uniswap V3 events from Ethereum mainnet:
```bash
# Run minimal version (recommended for beginners)
node run-simple.js
# Run full version with progress bar
node run.js
# Run version with terminal UI
node run-tui.js
```
That's it! You're now streaming data directly from Ethereum mainnet through HyperSync! (TUI version below)
## Understanding the Code
Let's look at the core concepts in the example code:
### 1. Initialize the Client
```javascript
import { HypersyncClient } from "@envio-dev/hypersync-client";
// Initialize Hypersync client
const client = new HypersyncClient({
url: "https://eth.hypersync.xyz", // Change this URL for different networks
apiToken: process.env.ENVIO_API_TOKEN!,
});
```
> **Note:** To connect to different networks, see the [Supported Networks](/docs/HyperSync/hypersync-supported-networks) page for a complete list of available URLs.
### 2. Build Your Query
The heart of HyperSync is the query object, which defines what data you want to retrieve:
```javascript
let query = {
fromBlock: 0, // Start block (0 = genesis)
logs: [
// Filter for specific events
{
topics: [topic0_list], // Event signatures we're interested in
},
],
fieldSelection: {
// Only return fields we need
log: [
"Data",
"Address",
"Topic0",
"Topic1",
"Topic2",
"Topic3",
],
},
};
```
### 3. Stream and Process Results
```javascript
// Start streaming events
const stream = await client.stream(query, {});
while (true) {
const res = await stream.recv();
// Process results
if (res.data && res.data.logs) {
// Do something with the logs
totalEvents += res.data.logs.length;
}
// Update starting block for next batch
if (res.nextBlock) {
query.fromBlock = res.nextBlock;
}
}
```
## Key Concepts for Building Queries
### Filtering Data
HyperSync lets you filter blockchain data in several ways:
- **Log filters**: Find specific events by contract address and event signature
- **Transaction filters**: Filter by sender/receiver addresses, method signatures, etc.
- **Trace filters**: Access internal transactions and state changes (only supported on select networks like Ethereum Mainnet)
- **Block filters**: Get data from specific block ranges
### Field Selection
One of HyperSync's most powerful features is the ability to retrieve only the fields you need:
```javascript
fieldSelection: {
// Block fields
block: ["Number", "Timestamp"],
// Log fields
log: ["Address", "Topic0", "Data"],
// Transaction fields
transaction: ["From", "To", "Value"],
}
```
This selective approach dramatically reduces unnecessary data transfer and improves performance.
### Join Modes
HyperSync allows you to control how related data is joined:
- **JoinNothing**: Return only exact matches
- **JoinAll**: Return matches plus all related objects
- **JoinTransactions**: Return matches plus their transactions
- **Default**: Return a reasonable set of related objects
## Examples
### Finding Uniswap V3 Events
This example (from the quickstart repo) streams all Uniswap V3 events from the beginning of Ethereum:
```javascript
import { keccak256, toHex } from "viem";
import { HypersyncClient } from "@envio-dev/hypersync-client";
// Define Uniswap V3 event signatures
const event_signatures = [
"PoolCreated(address,address,uint24,int24,address)",
"Burn(address,int24,int24,uint128,uint256,uint256)",
"Initialize(uint160,int24)",
"Mint(address,address,int24,int24,uint128,uint256,uint256)",
"Swap(address,address,int256,int256,uint160,uint128,int24)",
];
// Create topic0 hashes from event signatures
const topic0_list = event_signatures.map((sig) => keccak256(toHex(sig)));
// Initialize Hypersync client
const client = new HypersyncClient({
url: "https://eth.hypersync.xyz",
apiToken: process.env.ENVIO_API_TOKEN!,
});
// Define query for Uniswap V3 events
let query = {
fromBlock: 0,
logs: [
{
topics: [topic0_list],
},
],
fieldSelection: {
log: [
"Data",
"Address",
"Topic0",
"Topic1",
"Topic2",
"Topic3",
],
},
};
const main = async () => {
console.log("Starting Uniswap V3 event scan...");
const stream = await client.stream(query, {});
// Process stream...
};
main();
```
## Supported Networks
HyperSync supports EVM-compatible networks. You can change networks by simply changing the client URL:
```javascript
// Ethereum Mainnet
const client = new HypersyncClient({
url: "https://eth.hypersync.xyz",
apiToken: process.env.ENVIO_API_TOKEN!,
});
// Arbitrum
const client = new HypersyncClient({
url: "https://arbitrum.hypersync.xyz",
apiToken: process.env.ENVIO_API_TOKEN!,
});
// Base
const client = new HypersyncClient({
url: "https://base.hypersync.xyz",
apiToken: process.env.ENVIO_API_TOKEN!,
});
```
See the [Supported Networks](/docs/HyperSync/hypersync-supported-networks) page for a complete list.
## Using LogTUI
This quickstart repository powers [LogTUI](https://www.npmjs.com/package/logtui), a terminal-based blockchain event viewer built on HyperSync. LogTUI lets you monitor events from popular protocols across multiple chains with zero configuration.
Try it with a single command:
```bash
# Monitor Uniswap events on unichain
pnpx logtui uniswap-v4 unichain
# Monitor Aave events on Arbitrum
pnpx logtui aave arbitrum
# See all available options
pnpx logtui --help
```
LogTUI supports scanning historically for any events across all networks supported by HyperSync.
## Next Steps
You're now ready to build with HyperSync! Here are some resources for diving deeper:
- [Client Libraries](/docs/HyperSync/hypersync-clients) - Explore language-specific clients
- [Query Reference](/docs/HyperSync/hypersync-query) - Learn advanced query techniques
- **[Build queries visually](http://builder.hypersync.xyz)** - Use our Intuitive Query Builder
- [curl Examples](/docs/HyperSync/hypersync-curl-examples) - Test queries directly in your terminal
- [Complete Getting Started Guide](/docs/HyperSync/hypersync-usage) - More comprehensive guidance
## API Token
An API token is required to use HyperSync. [Get an API token](/docs/HyperSync/api-tokens) and set it as an environment variable:
```bash
export ENVIO_API_TOKEN="your-api-token-here"
```
Congratulations! You've taken your first steps with HyperSync, bringing ultra-fast blockchain data access to your applications. Happy building!
---
# Solana curl Examples
> Copy-paste curl examples for the Solana HyperSync API against real protocols.
# Solana curl Examples
:::info Early access
Solana HyperSync is **early** but the query shape used in these examples is stable enough to build against. Only **recent** slots are retained — the floor is a **rolling window** (see [Overview](./solana)); use `GET /height` instead of hard-coding how far back you can query. Working on something specific? [Ping us on Discord](https://discord.gg/envio) — we can often suggest a tighter query for your use case.
:::
Copy-paste examples against **`https://solana.hypersync.xyz`**. Use the same [API token](/docs/HyperSync/api-tokens) as EVM HyperSync: pass `Authorization: Bearer ` on **`POST /query`** (and on Arrow). `GET /health`, `GET /height`, and `GET /height/sse` are typically usable without a token, but follow whatever your deployment returns.
Curl is great for testing; for production, prefer one of our clients. The [Rust client](https://github.com/enviodev/hypersync-client-solana) is the most complete today (and uses Arrow for faster decoding); TypeScript and Python clients are in progress — tell us on Discord which one would unblock you and we'll prioritize accordingly.
```bash
export URL=https://solana.hypersync.xyz
export TOKEN="your-api-token"
# JSON POST helper (adds auth + content-type)
curl_query() {
curl -sS "$URL/query" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d "$1"
}
```
Discriminator filters accept hex **with or without** a `0x` prefix (`03` and `0x03` are the same). Pipe responses through `jq` or `python3 -m json.tool` for readability.
## Quick checks
```bash
curl -sS "$URL/health"
curl -sS "$URL/height"
```
### Head slot (SSE)
`curl -N` disables buffering so lines arrive as the server pushes them:
```bash
curl -sSN -H "Accept: text/event-stream" "$URL/height/sse"
```
## Orca Whirlpool (`swap` discriminator)
8-byte Anchor discriminator. Example response shape (truncated):
```bash
curl_query '{
"from_slot": 391800000,
"to_slot": 391800100,
"field_selection": {
"instruction": ["slot", "transaction_index", "program_id", "accounts", "data", "d8"],
"transaction": ["slot", "signatures", "fee_payer", "success", "fee"]
},
"instructions": [{
"program_id": ["whirLbMiicVdio4qvUfM5KAg6Ct8VwpYzGff3uctyCc"],
"d8": ["0xf8c69e91e17587c8"]
}]
}' | jq '{next_slot, sample_instruction: .instructions[0], sample_tx: .transactions[0]}'
```
## SPL Token `Transfer` (`d1`)
1-byte discriminator: `0x03` = `Transfer` (hex with or without `0x`).
```bash
curl_query '{
"from_slot": 391800000,
"to_slot": 391800100,
"field_selection": {
"instruction": ["slot", "program_id", "accounts", "data", "d1"],
"token_balance": ["slot", "transaction_index", "account", "mint", "owner", "pre_amount", "post_amount"]
},
"instructions": [{
"program_id": ["TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA"],
"d1": ["0x03"]
}]
}'
```
## Jupiter **or** Orca (program-only OR)
Each object in `instructions` is OR-ed. This is the “match by program id only” pattern (no discriminator).
```bash
curl_query '{
"from_slot": 391800000,
"to_slot": 391800100,
"field_selection": {
"instruction": ["slot", "program_id", "data", "d8"]
},
"instructions": [
{ "program_id": ["JUP6LkbZbjS1jKKwapdHNy74zcZ3tLUZoi5QNyVTaV4"] },
{ "program_id": ["whirLbMiicVdio4qvUfM5KAg6Ct8VwpYzGff3uctyCc"] }
]
}'
```
## Transactions by fee payer
```bash
curl_query '{
"from_slot": 391800000,
"to_slot": 391800100,
"field_selection": {
"transaction": ["slot", "signatures", "fee_payer", "success", "fee", "compute_units_consumed"],
"instruction": ["slot", "program_id", "data", "accounts"]
},
"transactions": [{
"fee_payer": ["MfDuWeqSHEqTFVYZ7LoexgAK9dxk7cy4DFJWjWMGVWa"]
}]
}'
```
## Pump.fun bonding-curve trades (account index)
`a2` matches the **third account** in the instruction's account metas (`a0` = first). For Pump.fun's buy/sell instructions, the mint is **account index 2 per that program's IDL**—not a Solana-wide rule.
```bash
curl_query '{
"from_slot": 391800000,
"to_slot": 391800100,
"field_selection": {
"instruction": ["slot", "program_id", "accounts", "data", "d8", "a2"],
"transaction": ["slot", "fee_payer", "success"]
},
"instructions": [{
"program_id": ["6EF8rrecthR5Dkzon8Nwu78hRvfCKubJ14M5uBEwF6P"]
}]
}'
```
## Raydium AMM logs
```bash
curl_query '{
"from_slot": 391800000,
"to_slot": 391800100,
"field_selection": {
"log": ["slot", "program_id", "kind", "message"],
"transaction": ["slot", "fee_payer", "success"]
},
"logs": [{
"program_id": ["675kPX9MHTjS2zt1qfr1NYHuzeLXfQM9H24wFSUt1Mp8"]
}]
}'
```
## Paginating a bounded scan
Use the same termination rule as [Query & Response](./solana-query#pagination): stop when `next_slot >= to_slot`, or when `next_slot` does not advance (stuck at head).
```bash
FROM=391800000
TO=391801000
SLOT=$FROM
while [ "$SLOT" -lt "$TO" ]; do
RESP=$(curl_query "{
\"from_slot\": $SLOT,
\"to_slot\": $TO,
\"field_selection\": { \"instruction\": [\"slot\", \"program_id\", \"d8\"] },
\"instructions\": [{ \"program_id\": [\"whirLbMiicVdio4qvUfM5KAg6Ct8VwpYzGff3uctyCc\"] }]
}")
echo "$RESP" | jq '.instructions | length, .next_slot'
NEXT=$(echo "$RESP" | jq -r .next_slot)
if [ "$NEXT" -ge "$TO" ] || [ "$NEXT" -le "$SLOT" ]; then
break
fi
SLOT=$NEXT
done
```
---
# Solana Query & Response
> Solana HyperSync query schema, filters, field selection, pagination, and reorg handling.
# Query & Response
:::info Early access
Solana HyperSync is **early** — the query shape, filters, and tables described below are the ones we expect to keep, and they're stable enough to build against today. See [What's stable vs. what's still evolving](./solana#whats-stable-vs-whats-still-evolving) for the current line between the two, and [reach out on Discord](https://discord.gg/envio) if you're planning a real workload — we can usually save you time by suggesting the right query shape for your use case.
Data is kept in a **rolling retention window**; the floor moves forward over time. Use `GET /height` instead of hard-coding how far back you can query.
:::
A query selects a slot range, optional filters on instructions / transactions / logs, and the columns you want. The server returns matched rows plus a `next_slot` cursor.
Some slots have **no block**; the `blocks` array can be sparse across the requested slot range.
## Query shape
```json
{
"from_slot": 391800000,
"to_slot": 391800100,
"include_all_blocks": false,
"instructions": [ ... ],
"transactions": [ ... ],
"logs": [ ... ],
"field_selection": { ... }
}
```
- `from_slot` is **inclusive**, `to_slot` is **exclusive**. Omit `to_slot` to run toward the current head.
- Within one selection object, all set fields are **AND**-ed.
- Multiple objects in `instructions`, `transactions`, or `logs` are **OR**-ed.
- If **`instructions`**, **`transactions`**, and **`logs`** are all absent or empty and **`include_all_blocks`** is false, you get **no matching rows** (empty tables). Set `include_all_blocks: true` to pull block headers across a range without program filters.
## Filters
### InstructionSelection
| Field | Description |
|---|---|
| `program_id` | Match program (base58 pubkeys). |
| `d1` / `d2` / `d4` / `d8` | First _N_ bytes of instruction data, as hex. **`0x` prefix is optional** (`"0x03"` and `"03"` are equivalent). |
| `a0` - `a9` | Account pubkey at that **index in the instruction's account metas** (`a0` = first account, `a2` = third). Which account is "the mint", "the pool", etc. is **defined by the program's IDL / instruction layout**, not by Solana globally. |
| `is_inner` | `true` = inner only, `false` = outer only, **omitted** = both. |
**`instruction_address`:** This array encodes **where** the instruction sits in the transaction: outer-only indices use one element, e.g. `[2]` = third top-level instruction; inner instructions append an index, e.g. `[2, 0]` = first inner instruction inside that outer instruction.
### TransactionSelection
| Field | Description |
|---|---|
| `fee_payer` | Match fee payer pubkey. |
| `success` | `true` = succeeded only, `false` = failed only, **omitted** = both (same pattern as `is_inner`). |
### LogSelection
| Field | Description |
|---|---|
| `program_id` | Match log emitter program. |
| `kind` | Parsed log line category (see below). |
#### Log `kind` values
These mirror the usual Solana runtime log line shapes (see the [transactions](https://solana.com/docs/core/transactions) docs and your RPC `logsSubscribe` / meta log output for raw strings).
| `kind` | Typical meaning |
|---|---|
| `invoke` | `Program invoke ` |
| `success` | `Program success` |
| `failure` | `Program failed: ...` |
| `log` | `Program log: ...` |
| `data` | `Program data: ` |
| `other` | Anything else the parser did not classify (full text still in `message`) |
## Field selection
Use `field_selection` to choose columns per logical table. Omit a table key to receive **all** columns for that table (when rows are returned).
```json
{
"field_selection": {
"block": ["slot", "blockhash", "block_time"],
"instruction": ["slot", "program_id", "data", "d8"],
"transaction": ["slot", "fee_payer", "success"]
}
}
```
### Available fields (by table)
| Table | Fields |
|---|---|
| `block` | `slot`, `blockhash`, `parent_slot`, `parent_blockhash`, `block_time`, `block_height` |
| `transaction` | `slot`, `transaction_index`, `signatures`, `fee_payer`, `success`, `err`, `fee`, `compute_units_consumed`, `account_keys`, `recent_blockhash`, `version`, `loaded_addresses_writable`, `loaded_addresses_readonly` |
| `instruction` | `slot`, `transaction_index`, `instruction_address`, `program_id`, `accounts`, `data`, `d1`, `d2`, `d4`, `d8`, `a0`-`a9`, `is_inner`, `is_committed` |
| `log` | `slot`, `transaction_index`, `instruction_address`, `program_id`, `kind`, `message` |
| `balance` | `slot`, `transaction_index`, `account`, `pre`, `post` |
| `token_balance` | `slot`, `transaction_index`, `account`, `mint`, `owner`, `pre_amount`, `post_amount` |
| `reward` | `slot`, `pubkey`, `lamports`, `post_balance`, `reward_type`, `commission` |
**`is_committed` (instruction):** Whether this instruction row is part of the **executed** instruction trace for the landed transaction (as opposed to being present only for structural / edge cases). Always interpret next to `transaction.success` and `transaction.err`: failed transactions can still include instructions up to the failure point.
## Join behavior
The server automatically joins related rows based on which tables you include in `field_selection`. For example, if your query filters on `instructions` and you also select `transaction` fields, the server returns the parent transaction for each matched instruction — no extra flags needed.
:::note Join modes not yet available
Solana HyperSync currently operates on a single default join mode. More granular control — for example fetching only the directly matched rows with no joins, or fetching all rows belonging to matched transactions — is planned but not yet exposed.
If you have specific join or filtering requirements that the current API cannot satisfy, we would love to hear about your use case. Reach out on [Discord](https://discord.gg/envio) or open an issue on [GitHub](https://github.com/enviodev/hypersync-client-solana/issues).
:::
## Limits (optional)
Advanced knobs (defaults are usually fine):
| Field | Role |
|---|---|
| `max_num_blocks` | Cap rows returned per table (approximate server-side bound). |
| `max_num_transactions` | Same, for `transactions`. |
| `max_num_instructions` | Same, for `instructions`. |
| `max_num_logs` | Same, for `logs`. |
## Response
Top-level keys include `next_slot`, `total_execution_time_ms`, optional `rollback_guard`, and one array per table when present: `blocks`, `transactions`, `instructions`, `logs`, `balances`, `token_balances`, `rewards`—each holds **row objects** shaped by your `field_selection`.
### Example fragment (illustrative)
```json
{
"next_slot": 391800050,
"total_execution_time_ms": 12,
"rollback_guard": null,
"blocks": [
{
"slot": 391800000,
"blockhash": "8dK...",
"block_time": 1731000123
}
],
"instructions": [
{
"slot": 391800000,
"program_id": "TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA",
"accounts": ["7xK...", "9mY...", "TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA"],
"data": ""
}
],
"transactions": []
}
```
- Pubkeys in filters and in many columns are **base58** strings.
- Instruction `data` is an **encoded payload** (treat as opaque unless you decode it); for filtering, prefer the hex **discriminator** fields (`d1` / `d2` / `d4` / `d8`) in the query body.
- On failure, `transaction.err` carries the chain's error structure (object or string depending on field selection).
### Pagination
Use the response's `next_slot` as the **next** request's `from_slot`.
**Bounded scan** (you set `to_slot`): repeat while `next_slot < to_slot`. When `next_slot >= to_slot`, the range `[from_slot, to_slot)` is exhausted.
**Unbounded / to head** (no `to_slot`, or you stop at live head): repeat while `next_slot` **strictly increases** between requests. If `next_slot` is not greater than the previous `from_slot`, you have caught up to the server's head or hit a limit—stop or backoff.
The server may stop early after a time or size budget; a single response can cover more or fewer slots than requested depending on filter density.
### Reorg detection (`rollback_guard`)
`rollback_guard` is **`null`** when the response does **not** overlap the unfinalized / risky tip region. When it is **present**, the fields tie the returned batch to a specific head blockhash so you can detect shallow reorgs between paginated calls.
**Algorithm (defensive):**
1. Let `G` be `rollback_guard` from response _n_. If `G` is null, skip reorg checks for that page (data is from finalized-safe depth).
2. When `G` is present, record `G.blockhash` and `G.first_previous_blockhash` together with the span of slots you believe you have ingested from that page.
3. On response _n+1_, if both pages have a non-null `rollback_guard`, compare response _n_'s **`rollback_guard.blockhash`** to response _n+1_'s **`rollback_guard.first_previous_blockhash`**. They should chain the same parent hash across the gap you queried; if they **differ**, a **reorg** occurred between the two calls—re-sync from a finalized slot or from the parent you still trust.
4. If a reorg **deeper** than your last page (your highest ingested slot is no longer on the winning fork), matching on parent blockhash alone may not fire until you overlap the new tip again—**never assume slot numbers alone are stable identifiers**; always reconcile with `blockhash` / `parent_blockhash` when consuming near-head data.
Example `rollback_guard` payload (field names only—values are illustrative):
```json
{
"slot_number": 391800099,
"timestamp": 1731000000,
"blockhash": "8dK...",
"first_slot_number": 391800000,
"first_previous_blockhash": "3nF..."
}
```
## Authentication
Same **Bearer token** model as EVM HyperSync. See [API tokens](/docs/HyperSync/api-tokens).
---
# Solana HyperSync
> HyperSync for Solana - ultra-fast queries over Solana blocks, transactions, instructions, and logs.
# Solana HyperSync
:::info Early access — built in the open
Solana HyperSync is **early**. The core query path (slots, transactions, instructions, logs, balances, token balances, rewards) is live and ready to test against real workloads — and we're actively shaping it with the teams using it. If you're evaluating it for a real project, **please [say hi on Discord](https://discord.gg/envio)** before you build a lot on top of it: we can tell you which parts are stable, which parts are still moving, and often suggest a better data path for your specific use case (NFTs, AMMs, token flows, wallet activity, custom programs, etc.).
**Rolling retention window.** Only the most recent chain data is retained — not a fixed history from one slot forever. The current retention floor is roughly slot `391791680`; as new slots are indexed, older slots fall off. Use `GET https://solana.hypersync.xyz/height` for the current synced head and **do not hard-code** historical lower bounds. Need a deeper window for backfill? Tell us — we're prioritizing this based on real use cases.
:::
HyperSync for Solana exposes **`https://solana.hypersync.xyz`**: one JSON (or Arrow) API over slots, transactions, instructions, logs, balances, token balances, and rewards. Use the [Rust client](https://github.com/enviodev/hypersync-client-solana) or any HTTP client (for example `curl`). Details: [Query & Response](./solana-query), [curl Examples](./solana-curl-examples).
**Slots vs blocks:** Some slots have **no block** (skipped leader, etc.). A query over `[from_slot, to_slot)` can return **fewer block rows** than the slot span implies; that is normal, not a bug.
## Differences vs EVM HyperSync
| Concept | EVM | Solana |
|---|---|---|
| Unit of progress | `block` | `slot` |
| Range bounds | `from_block` / `to_block` | `from_slot` / `to_slot` |
| Primary filter | `logs`, `transactions`, `traces` | `instructions`, `transactions`, `logs` |
| Match key | event topic + address | program ID + discriminator + account positions |
| Logs | Contract events (topics + structured log data) | Program output lines (free-form strings; filter by emitter `program_id` and parsed `kind`) |
| Pagination | `next_block` | `next_slot` |
## Endpoints
| Path | Description |
|---|---|
| `POST /query` | JSON query, JSON response. |
| `POST /query/arrow` | Same JSON query; response is **Apache Arrow IPC** (stream-encoded record batches—typically smaller and faster to decode than JSON). |
| `GET /height` | Current synced slot (JSON). Example: `curl https://solana.hypersync.xyz/height`. |
| `GET /height/sse` | Server-sent events stream of the head slot (see [curl Examples](./solana-curl-examples#head-slot-sse)). |
| `GET /health` | Health check. |
| `POST /`, `POST /rpc` | **Experimental** Solana JSON-RPC-compatible facade for tooling that already speaks JSON-RPC; coverage may be incomplete—prefer `POST /query` for indexing. |
## Minimal first query
```bash
curl -sS "https://solana.hypersync.xyz/query" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"from_slot": 391800000,
"to_slot": 391800010,
"field_selection": { "instruction": ["slot", "program_id", "d8"] },
"instructions": [
{ "program_id": ["6EF8rrecthR5Dkzon8Nwu78hRvfCKubJ14M5uBEwF6P"] }
]
}'
```
Expect JSON with `instructions` (and any joined tables you asked for), `next_slot`, optional `rollback_guard`, and other keys empty or omitted. [API tokens](/docs/HyperSync/api-tokens) are the same as for EVM HyperSync (`Authorization: Bearer`).
## What's stable vs. what's still evolving
We want you to be able to build against this without guessing what will move under you. As of today:
**Stable enough to build on**
- The **endpoint** (`https://solana.hypersync.xyz`) and **bearer-token auth** model.
- The **request shape** for `POST /query`: `from_slot` / `to_slot`, the `instructions` / `transactions` / `logs` selection arrays, `field_selection` projection, and the AND-within-object / OR-across-objects semantics.
- The **core filter primitives**: `program_id`, discriminator filters (`d1` / `d2` / `d4` / `d8`), account-position filters (`a0`–`a9`), `is_inner`, `success`, `fee_payer`, log `kind`.
- The **table model**: `block`, `transaction`, `instruction`, `log`, `balance`, `token_balance`, `reward`, with the fields listed in [Query & Response](./solana-query#available-fields-by-table).
- **Pagination** via `next_slot` and **reorg detection** via `rollback_guard`.
**Still evolving — check in if you depend on these**
- The **historical retention floor** (rolling window today; we're prioritizing deeper backfill based on demand).
- Decoded / higher-level helpers built on top of the raw tables (IDL-aware decoding, common-program shortcuts).
- The **JSON-RPC-compatible facade** (`POST /` / `POST /rpc`) — useful for tools that already speak Solana JSON-RPC, but coverage is incomplete; prefer `POST /query` for indexing.
- Client libraries beyond the [Rust client](https://github.com/enviodev/hypersync-client-solana) (TypeScript / Python clients are in progress).
If a piece you need is in the second list, the fastest path is to tell us — most of the roadmap here is being driven by the use cases people bring us.
## Working with us
Solana HyperSync is the right time to be a design partner: the foundation is live, the abstractions on top are being shaped now, and your use case can influence what gets prioritized.
- **[Join us on Discord](https://discord.gg/envio)** — fastest way to reach the team building this.
- Have a specific Solana indexing problem (NFTs, AMM trades, token flows, wallet activity, a custom program)? Share a sample transaction signature or program ID and we'll map it to a concrete query path.
- Hitting a missing field, a too-shallow retention window, or a filter you wish existed? File it on [GitHub](https://github.com/enviodev/hypersync-client-solana/issues) or tell us on Discord — early feedback shapes what ships next.
---
# Stream Config & Tuning
> Configure and tune the HyperSync streaming engine (concurrency, response size targets, batch sizes) and find the best config for your query with the built-in observer.
# Stream Config & Tuning
When you `stream` (or `collect`) with a HyperSync client, the engine fans your block
range out across many concurrent HTTP requests, sizes each request automatically, and
delivers results to you **in block order**. `StreamConfig` controls that engine.
:::tip TL;DR
**The defaults are good for most workloads, so you usually don't need to touch this.**
Reach for tuning when you want to squeeze more throughput out of a specific workload, or to
bound memory. The fastest way to find a good config for *your* query is the
[`tune_stream`](#find-the-best-config-for-your-query) tool, which sweeps configs against your
query and prints a comparison table.
:::
## Quick decision guide
| Your situation | Do this |
|---|---|
| Just getting started | Use the defaults. |
| Want maximum throughput on a busy contract (lots of logs) | Raise `concurrency` (for example 20). |
| Scanning a wide range for a rare event | Keep `concurrency` moderate (around 10); a larger `batch_size` helps. |
| Pulling full blocks + all transactions | Defaults are fine; the adaptive buffer handles the large responses. |
| Memory constrained | Set `max_buffered_bytes` to a fixed cap. |
| Hitting API rate limits | The client waits out limits and retries automatically. For higher limits, [upgrade your plan](./api-tokens.mdx). |
| Not sure | Run [`tune_stream`](#find-the-best-config-for-your-query) against your query. |
## How the engine works (the 30-second model)
Two ideas make every knob obvious:
1. **One HTTP request is one unit of work.** The engine sizes each request to land near
`response_bytes_target` bytes, based on the byte-density it has measured so far. If the
server returns less than requested (a "truncation"), the leftover range becomes a gap that
any free worker backfills in parallel.
2. **Delivery is in block order.** Results are buffered and handed to you contiguously, so
the stream yields one response per HTTP response in ascending (or `reverse`) block order.
So `concurrency` sets how many requests run in parallel, and `response_bytes_target` sets how
big each response is. Because truncations are backfilled automatically, the engine is
forgiving: an over-estimated request size corrects itself rather than failing.
## Configuration options
Field names below are shown in `snake_case` (Rust, Python). **Node and TypeScript use
`camelCase`** (for example `response_bytes_target` becomes `responseBytesTarget`).
| Option | Default | What it does |
|---|---|---|
| `concurrency` | `10` | Number of requests in flight, and your main throughput knob. `0` is an error, `1` streams sequentially, `2` or more uses the parallel scheduler. |
| `response_bytes_target` | `400_000` | Target size in bytes for each response. Each request is sized to aim here. Raise it for fewer, larger responses. |
| `batch_size` | `1_000` | Initial block range for the first wave of requests, before any density has been measured. Also the fallback. |
| `min_batch_size` | `200` | Lower clamp on the projected block count, to avoid tiny ranges. |
| `max_batch_size` | _unset_ | Optional hard cap on blocks per request. Unset means no cap (over-shoot self-corrects via backfill). Set it only if you specifically want to bound blocks per request. |
| `max_buffered_bytes` | _unset_ | Cap on bytes of fetched-but-undelivered data held for re-ordering (consumer backpressure). Unset means adaptive (it grows with the largest response seen) so byte-heavy pulls stay pipelined. Set a fixed value to bound memory. |
| `reverse` | `false` | Stream from the top of the range downward. |
| `max_num_blocks`, `max_num_transactions`, `max_num_logs`, `max_num_traces` | _unset_ | Stop the stream once this many of an entity have been delivered. |
| `column_mapping`, `event_signature`, `hex_output` | none | Output shaping (decoding, hex formatting). Not performance knobs. |
:::info Breaking change in v1.3.0 / streaming v2
`response_bytes_floor` and `response_bytes_ceiling` were replaced by a single
`response_bytes_target`. `max_batch_size` became optional (unset means no cap).
`max_buffered_bytes` was added. If you set the old fields, switch to `response_bytes_target`.
:::
## Tuning recipes
These are good starting points. Confirm against your own query with
[`tune_stream`](#find-the-best-config-for-your-query).
### Dense: busy contracts and all-logs
Lots of matching data per block. Throughput scales with parallelism.
```python
# Python
config = hypersync.StreamConfig(concurrency=20, response_bytes_target=400_000)
```
```typescript
// Node / TypeScript
const config = { concurrency: 20, responseBytesTarget: 400_000 };
```
```rust
// Rust has a ready-made preset
let config = StreamConfig::dense();
```
### Sparse: rare events over a wide range
Most blocks match nothing. Raising concurrency past the default tends to just fragment the
empty range into more, smaller requests without adding throughput, so keep it moderate. A
larger `batch_size` lets the first wave cover more ground.
```python
config = hypersync.StreamConfig(concurrency=10, batch_size=20_000)
```
```typescript
const config = { concurrency: 10, batchSize: 20_000 };
```
```rust
let config = StreamConfig::sparse();
```
### Archival: full blocks and all transactions
Each response is many megabytes, so the run is bound by the re-order buffer rather than
concurrency. Leave `max_buffered_bytes` unset so the adaptive buffer keeps the pipeline full.
```python
config = hypersync.StreamConfig(concurrency=12)
```
```typescript
const config = { concurrency: 12 };
```
```rust
let config = StreamConfig::archival();
```
## Find the best config for your query
Rather than guess, measure. The Rust client ships a standalone **`tune_stream`** example
that runs your query under a grid of configs and prints a comparison table: throughput,
request count, truncation rate, and how close responses land to the target. It takes a query
as JSON, so it works for any query regardless of which client language you use.
### 1. Save your query as JSON
```json title="query.json"
{
"from_block": 18000000,
"to_block": 18100000,
"logs": [
{ "topics": [["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"]] }
],
"field_selection": { "log": ["block_number", "log_index", "data", "topic0"] }
}
```
### 2. Run the sweep
```bash
git clone https://github.com/enviodev/hypersync-client-rust
cd hypersync-client-rust
export ENVIO_API_TOKEN=
export CHAIN_ID=1 # 1 = Ethereum mainnet
# Sweep a grid of configs and print a comparison table
cargo run -p tune_stream -- query.json
# Or a single detailed run with the default config
cargo run -p tune_stream -- query.json --single
```
The table shows, per config: requests, truncation %, blocks/s, MB/s, the mean
size-vs-target ratio, and the observed buffer and in-flight counts. Pick the config with the
best throughput for your workload.
### 3. Apply the winning config
Copy `response_bytes_target`, `concurrency`, and any other fields from the best row into your
client's `StreamConfig`.
### Rust: attach an observer in your own code
Rust users can read the same metrics live, without the example, via the observer API:
```rust
use std::sync::Arc;
use hypersync_client::{Client, StreamConfig, StreamMetrics, StreamObserver};
let metrics = Arc::new(StreamMetrics::new());
let observer: Arc = metrics.clone();
let mut rx = client
.stream_arrow_with_observer(query, StreamConfig::default(), observer)
.await?;
while let Some(res) = rx.recv().await {
let _ = res?; // consume the stream
}
let summary = metrics.summary();
println!(
"requests={} truncated={:.0}% blocks/s={:.0} mean size ratio={:.2}",
summary.num_requests,
summary.truncation_rate * 100.0,
summary.blocks_per_sec,
summary.mean_size_ratio,
);
```
The plain `stream` and `stream_arrow` methods do no metrics work; `RequestStats` are only
built when you attach an observer. (Surfacing this handle in the Node and Python clients is a
fast-follow; until then, use `tune_stream` for those languages.)
## Reading the metrics
When tuning, the two numbers worth watching are:
- **Mean size ratio**, which is response size divided by `response_bytes_target`. Around
`1.0` means responses are landing on target. Well below `1.0` means the server is returning
smaller responses than your target, which is perfectly normal for selective queries or
minimal field selections; raising `response_bytes_target` won't change it, and the way to
go faster is usually more `concurrency`.
- **Blocks per second**, which is the throughput you are tuning for.
Truncation is shown too, but you can usually ignore it. Some truncation is normal and
harmless because the leftover range is backfilled automatically, and a sparse query that
selects only a few fields will often show small, frequently-truncated responses while still
streaming quickly. It is only worth a look if throughput is poor.
## Rate limits
The clients handle rate limits for you: when the server signals a limit (HTTP 429) the client
waits for the window to reset and retries, so a stream slows down rather than failing. If you
want more headroom or higher throughput, [upgrade your plan](./api-tokens.mdx).
If you need the client to make fewer requests per unit time (for example to fit a fixed
request budget), `concurrency` is the lever. It trades throughput for request volume: fewer
requests run in parallel, so fewer go out per unit time. Setting `concurrency = 1` streams
sequentially, one request at a time.
## Full default reference
```text
concurrency = 10
response_bytes_target = 400_000
batch_size = 1_000
min_batch_size = 200
max_batch_size = unset (no cap)
max_buffered_bytes = unset (adaptive)
reverse = false
max_num_blocks / transactions / logs / traces = unset
```
---
# Analyzing All Transactions To and From an Address
> Explore all transactions, token balances, and approvals for any EVM address with HyperSync.
# Analyzing All Transactions To and From an Address
## Introduction
Understanding all transactions to and from an address is an interesting use case. Traditionally extracting this information would be very difficult with an RPC. In this tutorial, we'll introduce you to the [evm-address-summary](https://github.com/enviodev/evm-address-summary) tool, which uses HyperSync to efficiently extract all transactions associated with a specific address.
## About evm-address-summary
The [evm-address-summary](https://github.com/enviodev/evm-address-summary) repository contains a collection of scripts designed to get activity related to an address. These scripts leverage HyperSync's efficient data access to make complex address analysis simple and quick.
**GitHub Repository**: [https://github.com/enviodev/evm-address-summary](https://github.com/enviodev/evm-address-summary)
## Available Scripts
The repository offers several specialized scripts:
1. **All Transfers**: This script scans the entire blockchain (from block 0 to the present) and retrieves all relevant transactions for the given address. It iterates through these transactions and sums up their values to calculate aggregates for each token.
2. **NFT Holders**: This script scans the entire blockchain and retrieves all token transfer events for an ERC721 address. It records all the owners of these tokens and how many tokens they have traded in the past.
3. **ERC20 Transfers and Approvals**: This script scans the blockchain and retrieves all ERC20 transfer and approval events for the given address.
It calculates the following:
- **Token balances**: Summing up all incoming and outgoing transfers for each token
- **Token transaction counts**: Counting the number of incoming and outgoing transactions for each token
- **Approvals**: Tracking approvals for each token, including the spender and approved amount
## Quick Start Guide
### Prerequisites
- [Node.js](https://nodejs.org/) (v16 or newer)
- [pnpm](https://pnpm.io/installation) (recommended)
- Git
### Basic Setup
1. **Clone the Repository**
```bash
git clone https://github.com/enviodev/evm-address-summary.git
cd evm-address-summary
```
2. **Install Dependencies**
```bash
pnpm install
```
3. **Run a Script** (example with all-transfers)
```bash
pnpm run all-transfers 0xYourAddressHere
```
For complete details on all available scripts, their usage, and example outputs, refer to the [project README](https://github.com/enviodev/evm-address-summary#readme).
## Customizing Network Endpoints
The scripts work with any network supported by HyperSync. To change networks, edit the `hyperSyncEndpoint` in the appropriate config file:
```typescript
// For Ethereum Mainnet
export const hyperSyncEndpoint = "https://eth.hypersync.xyz";
```
For a complete list of supported networks, see our [HyperSync Supported Networks](/docs/HyperSync/hypersync-supported-networks) documentation.
## Practical Use Cases
One powerful application is measuring value at risk for any address, similar to [revoke.cash](https://revoke.cash). You can quickly scan an address to find all approvals and transfers to easily determine any outstanding approvals on any token. This helps identify potential security risks from forgotten token approvals.
Other use cases include:
- Portfolio tracking and analysis
- Auditing transaction history
- Research on token holder behavior
- Monitoring NFT ownership changes
## Next Steps
- Check out the [evm-address-summary repository](https://github.com/enviodev/evm-address-summary) for full documentation
- Explore the source code to understand how HyperSync is used for data retrieval
- Try modifying the scripts for your specific use cases
- Learn more about [HyperSync's capabilities](/docs/HyperSync/hypersync-query) for blockchain data analysis
For any questions or support, join our [Discord community](https://discord.gg/envio) or create an issue on the GitHub repository.
---
# Configuration Schema Reference
Static, deep-linkable reference for the V3 `config.yaml` schema.
> Tip: Use the Table of Contents to jump to a field or definition.
## Top-level Properties
- [name](#name) (required)
- [description](#description)
- [schema](#schema)
- [handlers](#handlers)
- [full_batch_size](#fullbatchsize)
- [storage](#storage)
- [ecosystem](#ecosystem)
- [contracts](#contracts)
- [chains](#chains) (required)
- [rollback_on_reorg](#rollbackonreorg)
- [save_full_history](#savefullhistory)
- [field_selection](#fieldselection)
- [raw_events](#rawevents)
- [address_format](#addressformat)
### name {#name}
Name of the project
- **type**: `string`
Example (config.yaml):
```yaml
name: MyIndexer
```
### description {#description}
Description of the project
- **type**: `string | null`
Example (config.yaml):
```yaml
description: Greeter indexer
```
### schema {#schema}
Custom path to schema.graphql file
- **type**: `string | null`
Example (config.yaml):
```yaml
schema: ./schema.graphql
```
### handlers {#handlers}
Optional relative path to handlers directory for auto-loading. Defaults to 'src/handlers' if not specified.
- **type**: `string | null`
### full_batch_size {#fullbatchsize}
Target number of events to be processed per batch. Set it to smaller number if you have many Effect API calls which are slow to resolve and can't be batched. (Default: 5000)
- **type**: `integer | null`
- **bounds**: min: 0, format: `uint64`
Example (config.yaml):
```yaml
full_batch_size: 5000
```
### storage {#storage}
Storage backends the indexer writes data to. Defaults to Postgres when omitted. Set `clickhouse: true` to additionally sync the indexed data to ClickHouse. Mark a backend with `default: true` to store entities that don't have an @storage directive in the schema, e.g. `clickhouse: {default: true}`.
- **type**: `anyOf(object | null)`
Variants:
- `1`: [StorageConfig](#def-storageconfig)
- `2`: `null`
Example (config.yaml):
```yaml
storage:
postgres:
default: true
column_name_format: snake_case
clickhouse: true
```
### ecosystem {#ecosystem}
Ecosystem of the project.
- **type**: `anyOf(enum (1 values) | null)`
Variants:
- `1`: [EcosystemTag](#def-ecosystemtag)
- `2`: `null`
Example (config.yaml):
```yaml
ecosystem: evm
```
### contracts {#contracts}
Global contract definitions that must contain all definitions except addresses. You can share a single handler/abi/event definitions for contracts across multiple chains.
- **type**: `array | null`
Example (config.yaml):
```yaml
contracts:
- name: Greeter
events:
- event: "NewGreeting(address user, string greeting)"
```
### chains {#chains}
Configuration of the blockchain chains that the project is deployed on.
- **type**: `array>`
- **items**: `object`
- **items ref**: [Chain](#def-chain)
Example (config.yaml):
```yaml
chains:
- id: 1
start_block: 0
contracts:
- name: Greeter
address: "0x9D02A17dE4E68545d3a58D3a20BbBE0399E05c9c"
```
### rollback_on_reorg {#rollbackonreorg}
A flag to indicate if the indexer should rollback to the last known valid block on a reorg. This currently incurs a performance hit on historical sync and is recommended to turn this off while developing (default: true)
- **type**: `boolean | null`
Example (config.yaml):
```yaml
rollback_on_reorg: true
```
### save_full_history {#savefullhistory}
A flag to indicate if the indexer should save the full history of events. This is useful for debugging but will increase the size of the database (default: false)
- **type**: `boolean | null`
Example (config.yaml):
```yaml
save_full_history: false
```
### field_selection {#fieldselection}
Select the block and transaction fields to include in all events globally
- **type**: `anyOf(object | null)`
Variants:
- `1`: [FieldSelection](#def-fieldselection)
- `2`: `null`
Example (config.yaml):
```yaml
field_selection:
transaction_fields:
- hash
block_fields:
- miner
```
### raw_events {#rawevents}
If true, the indexer will store the raw event data in the database. This is useful for debugging, but will increase the size of the database and the amount of time it takes to process events (default: false)
- **type**: `boolean | null`
Example (config.yaml):
```yaml
raw_events: true
```
### address_format {#addressformat}
Address format for Ethereum addresses: 'checksum' or 'lowercase' (default: checksum)
- **type**: `anyOf(enum (2 values) | null)`
Variants:
- `1`: [AddressFormat](#def-addressformat)
- `2`: `null`
## Definitions
### StorageConfig {#def-storageconfig}
- **type**: `object`
Properties:
- `postgres`: `anyOf(boolean | null | object)` – Whether to use Postgres as a storage backend (default: true). Accepts a boolean or an options object (the object form implies the backend is enabled).
- `clickhouse`: `anyOf(boolean | null | object)` – Whether to additionally sync the indexed data to ClickHouse. Requires Postgres to be enabled (default: false). Accepts a boolean or an options object (the object form implies the backend is enabled).
Example (config.yaml):
```yaml
storage:
postgres:
# Entities without an @storage directive land here
default: true
# Columns become snake_case in the database, while GraphQL and
# handler types keep the schema.graphql casing
column_name_format: snake_case
clickhouse:
default: false
```
### EcosystemTag {#def-ecosystemtag}
- **type**: `enum (1 values)`
- **allowed**: `evm`
Example (config.yaml):
```yaml
ecosystem: evm
```
### GlobalContract {#def-globalcontract}
- **type**: `object`
- **required**: `name`, `events`
Properties:
- `name`: `string` – A unique project-wide name for this contract (no spaces)
- `abi_file_path`: `string | null` – Relative path (from config) to a json abi. If this is used then each configured event should simply be referenced by its name
- `handler`: `string | null` – Optional relative path to a file where handlers are registered for the given contract. If not provided, handlers can be auto-loaded from src directory.
- `events`: `array>` – A list of events that should be indexed on this contract
Example (config.yaml):
```yaml
contracts:
- name: Greeter
events:
- event: "NewGreeting(address user, string greeting)"
```
### EventConfig {#def-eventconfig}
- **type**: `object`
- **required**: `event`
Properties:
- `event`: `string` – The human readable signature of an event 'eg. Transfer(address indexed from, address indexed to, uint256 value)' OR a reference to the name of an event in a json ABI file defined in your contract config. A provided signature will take precedence over what is defined in the json ABI
- `name`: `string | null` – Name of the event in the HyperIndex generated code. When ommitted, the event field will be used. Should be unique per contract
- `field_selection`: `anyOf(object | null)` – Select the block and transaction fields to include in the specific event
Example (config.yaml):
```yaml
contracts:
- name: Greeter
events:
- event: "Assigned(address indexed recipientId, uint256 amount, address token)"
name: Assigned
field_selection:
transaction_fields:
- transactionIndex
```
### FieldSelection {#def-fieldselection}
- **type**: `object`
Properties:
- `transaction_fields`: `array | null` – The transaction fields to include in the event, or in all events if applied globally
- Available values:
`transactionIndex`, `hash`, `from`, `to`, `gas`, `gasPrice`, `maxPriorityFeePerGas`, `maxFeePerGas`, `cumulativeGasUsed`, `effectiveGasPrice`, `gasUsed`, `input`, `nonce`, `value`, `v`, `r`, `s`, `contractAddress`, `logsBloom`, `root`, `status`, `yParity`, `accessList`, `maxFeePerBlobGas`, `blobVersionedHashes`, `type`, `l1Fee`, `l1GasPrice`, `l1GasUsed`, `l1FeeScalar`, `gasUsedForL1`, `authorizationList`
- `block_fields`: `array | null` – The block fields to include in the event, or in all events if applied globally
- Available values:
`parentHash`, `nonce`, `sha3Uncles`, `logsBloom`, `transactionsRoot`, `stateRoot`, `receiptsRoot`, `miner`, `difficulty`, `totalDifficulty`, `extraData`, `size`, `gasLimit`, `gasUsed`, `uncles`, `baseFeePerGas`, `blobGasUsed`, `excessBlobGas`, `parentBeaconBlockRoot`, `withdrawalsRoot`, `l1BlockNumber`, `sendCount`, `sendRoot`, `mixHash`
Example (config.yaml):
```yaml
events:
- event: "Assigned(address indexed user, uint256 amount)"
# can be within an event as shown here, or globally for all events
field_selection:
transaction_fields:
- transactionIndex
block_fields:
- miner
```
### Chain {#def-chain}
- **type**: `object`
- **required**: `id`, `start_block`
Properties:
- `id`: `integer` – The public blockchain chain ID.
- `skip`: `boolean | null` – Excludes the chain from indexing and migrations. Code generation is unaffected. For testing, prefer using a test framework instead.
- `rpc`: `anyOf(anyOf(string | object | array>) | null)` – RPC configuration for your indexer. If not specified otherwise, for chains supported by HyperSync, RPC serves as a fallback for added reliability. For others, it acts as the primary data-source. HyperSync offers significant performance improvements, up to a 1000x faster than traditional RPC.
- `hypersync_config`: `anyOf(object | null)` – Optional HyperSync Config for additional fine-tuning
- `max_reorg_depth`: `integer | null` – The number of blocks from the head that the indexer should account for in case of reorgs.
- `block_lag`: `integer | null` – The number of blocks behind the chain head that the indexer should lag. Useful for avoiding reorg issues by indexing slightly behind the tip.
- `start_block`: `integer` – The block at which the indexer should start ingesting data
- `end_block`: `integer | null` – The block at which the indexer should terminate.
- `contracts`: `array | null` – All the contracts that should be indexed on the given chain
Example (config.yaml):
```yaml
chains:
- id: 1
start_block: 0
end_block: 19000000
contracts:
- name: Greeter
address: "0x1111111111111111111111111111111111111111"
# Excluded from indexing and migrations, but still code-generated
- id: 137
skip: true
start_block: 0
```
### RpcSelection {#def-rpcselection}
- **type**: `anyOf(string | object | array>)`
Variants:
- `1`: `string`
- `2`: [Rpc](#def-rpc)
- `3`: `array>`
### Rpc {#def-rpc}
- **type**: `object`
- **required**: `url`
Properties:
- `url`: `string` – The RPC endpoint URL.
- `for`: `anyOf(oneOf(const sync | const fallback | const realtime) | null)` – Determines if this RPC is for historical sync, real-time chain indexing, or as a fallback. If not specified, defaults to "fallback" when HyperSync is available for the chain, or "sync" otherwise.
- `ws`: `string | null` – Optional WebSocket endpoint URL (wss:// or ws://) for real-time block header notifications via eth_subscribe("newHeads"). Provides lower latency than HTTP polling for detecting new blocks.
- `headers`: `object | null` – Optional HTTP headers sent with every request to this RPC endpoint, e.g. an Authorization bearer token for gated endpoints. Values support `${ENV_VAR}` interpolation.
- `initial_block_interval`: `integer | null` – The starting interval in range of blocks per query
- `backoff_multiplicative`: `number | null` – After an RPC error, how much to scale back the number of blocks requested at once
- `acceleration_additive`: `integer | null` – Without RPC errors or timeouts, how much to increase the number of blocks requested by for the next batch
- `interval_ceiling`: `integer | null` – Do not further increase the block interval past this limit
- `backoff_millis`: `integer | null` – After an error, how long to wait before retrying
- `fallback_stall_timeout`: `integer | null` – If a fallback RPC is provided, the amount of time in ms to wait before kicking off the next provider
- `query_timeout_millis`: `integer | null` – How long to wait before cancelling an RPC request
- `polling_interval`: `integer | null` – How frequently (in milliseconds) to check for new blocks in realtime. Default is 1000ms. Note: Setting this higher than block time does not reduce RPC usage as every block is still fetched to check for reorgs.
Example (config.yaml):
```yaml
chains:
- id: 1
rpc:
- url: https://eth.llamarpc.com
for: sync
headers:
Authorization: "Bearer ${RPC_API_KEY}"
- url: wss://eth.llamarpc.com
for: realtime
- url: https://fallback.example.com
for: fallback
```
### For {#def-for}
- **type**: `oneOf(const sync | const fallback | const realtime)`
Variants:
- `1`: `const sync`
- `2`: `const fallback`
- `3`: `const realtime`
### HypersyncConfig {#def-hypersyncconfig}
- **type**: `object`
- **required**: `url`
Properties:
- `url`: `string` – URL of the HyperSync endpoint (default: The most performant HyperSync endpoint for the network)
Example (config.yaml):
```yaml
chains:
- id: 1
hypersync_config:
url: https://eth.hypersync.xyz
```
### ChainContract {#def-chaincontract}
- **type**: `object`
- **required**: `name`
Properties:
- `name`: `string` – A unique project-wide name for this contract if events and handler are defined OR a reference to the name of contract defined globally at the top level
- `address`: `anyOf(anyOf(string | integer) | array)` – A single address or a list of addresses to be indexed. This can be left as null in the case where this contracts addresses will be registered dynamically.
- `start_block`: `integer | null` – The block at which the indexer should start ingesting data for this specific contract. If not specified, uses the chain start_block. Can be greater than the chain start_block for more specific indexing.
- `abi_file_path`: `string | null` – Relative path (from config) to a json abi. If this is used then each configured event should simply be referenced by its name
- `handler`: `string | null` – Optional relative path to a file where handlers are registered for the given contract. If not provided, handlers can be auto-loaded from src directory.
- `events`: `array>` – A list of events that should be indexed on this contract
Example (config.yaml):
```yaml
chains:
- id: 1
start_block: 0
contracts:
- name: Greeter
address:
- "0x1111111111111111111111111111111111111111"
events:
- event: Transfer(address indexed from, address indexed to, uint256 value)
```
### Addresses {#def-addresses}
- **type**: `anyOf(anyOf(string | integer) | array)`
Variants:
- `1`: `anyOf(string | integer)`
- `2`: `array`
Example (config.yaml):
```yaml
chains:
- id: 1
contracts:
- name: Greeter
address:
- "0x1111111111111111111111111111111111111111"
- "0x2222222222222222222222222222222222222222"
```
### AddressFormat {#def-addressformat}
- **type**: `enum (2 values)`
- **allowed**: `checksum`, `lowercase`
## Removed in V3
The following V2 options have been removed and are no longer accepted in `config.yaml`:
- `output` — generated types are always emitted to `.envio/`.
- `unordered_multichain_mode` — unordered is now the only mode. The V2 `multichain: ordered` opt-in has also been removed.
- `event_decoder` — the Rust-based decoder is the only implementation.
- `loaders` — Preload Optimization is now always on.
- `preload_handlers` — now always enabled.
- `preRegisterDynamicContracts` — no longer needed.
- `rpc_config` — replaced by `rpc` (see above).
- `networks` — renamed to `chains`.
- `confirmed_block_threshold` — renamed to `max_reorg_depth`.
---
# Loading Dynamic Contracts / Factories
> Learn how to track and index dynamically created contracts from factory contracts.
## Introduction
Many blockchain systems use factory patterns where new contracts are created dynamically. Common examples include:
- DEXes like Uniswap where each trading pair creates a new contract
- NFT platforms that deploy new collection contracts
- Lending protocols that create new markets as isolated contracts
When indexing these systems, you need a way to discover and track these dynamically created contracts. Envio provides powerful tools to handle this use case.
### Contract Registration Handler
Instead of a template based approach, we've introduced a `contractRegister` handler that can be added to any event.
This allows you to easily:
- Register contracts from any event handler.
- Use conditions and any logic you want to register contracts.
- Have nested factories which are registered by other factories.
```typescript
import { indexer } from "envio";
indexer.contractRegister(
{ contract: "", event: "" },
({ event, context }) => {
context.chain..add();
},
);
```
## Example: NFT Factory Pattern
Let's look at a complete example using an NFT factory pattern.
### Scenario
- `NftFactory` contract creates new `SimpleNft` contracts
- We want to index events from all NFTs created by this factory
- Each time a new NFT is created, the factory emits a `SimpleNftCreated` event
### 1. Configure Your Contracts in config.yaml
```yaml
name: nftindexer
description: NFT Factory
chains:
- id: 1337
start_block: 0
contracts:
- name: NftFactory
abi_file_path: abis/NftFactory.json
address: "0x4675a6B115329294e0518A2B7cC12B70987895C4" # Factory address is known
events:
- event: SimpleNftCreated (string name, string symbol, uint256 maxSupply, address contractAddress)
- name: SimpleNft
abi_file_path: abis/SimpleNft.json
# No address field - we'll discover these addresses from events
events:
- event: Transfer (address from, address to, uint256 tokenId)
```
Note that:
- The `NftFactory` contract has a known address specified in the config
- The `SimpleNft` contract has no address, as we'll register instances dynamically
### 2. Create the Contract Registration Handler
In your `src/handlers/.ts` file:
```typescript
import { indexer } from "envio";
// Register SimpleNft contracts whenever they're created by the factory
indexer.contractRegister(
{ contract: "NftFactory", event: "SimpleNftCreated" },
({ event, context }) => {
// Register the new NFT contract using its address from the event
context.chain.SimpleNft.add(event.params.contractAddress);
context.log.info(
`Registered new SimpleNft at ${event.params.contractAddress}`
);
},
);
// Handle Transfer events from all SimpleNft contracts
indexer.onEvent(
{ contract: "SimpleNft", event: "Transfer" },
async ({ event, context }) => {
// Your event handling logic here
context.log.info(
`NFT Transfer at ${event.srcAddress} - Token ID: ${event.params.tokenId}`
);
// Example: Store transfer information in the database
// ...
},
);
```
## Async Contract Register
As of version `2.21`, you can use async contract registration.
This is a unique feature of Envio that allows you to perform an external call to determine the address of the contract to register.
```typescript
import { indexer } from "envio";
indexer.contractRegister(
{ contract: "NftFactory", event: "SimpleNftCreated" },
async ({ event, context }) => {
const version = await getContractVersion(event.params.contractAddress);
if (version === "v2") {
context.chain.SimpleNftV2.add(event.params.contractAddress);
} else {
context.chain.SimpleNft.add(event.params.contractAddress);
}
},
);
```
## Coming from TheGraph?
If you're migrating from a subgraph that uses **data source templates** (`DataSource.create()`), the equivalent in Envio is the `contractRegister` handler.
| TheGraph | Envio (HyperIndex) |
|---|---|
| Define a template in `subgraph.yaml` | Define the contract in `config.yaml` without an `address` |
| Call `MyTemplate.create(address)` in a mapping | Call `context.chain.MyContract.add(address)` in a `contractRegister` handler |
| Templates are triggered from other mappings | `contractRegister` runs before the event handler, on any event |
The key difference is that Envio's `contractRegister` is more flexible — you can add conditional logic, perform async calls, and register contracts from any event, not just from a dedicated factory mapping.
For a step-by-step migration guide, see [Migrating from a Subgraph](/docs/HyperIndex/migration-guide).
## When to Use Dynamic Contract Registration
Use dynamic contract registration when:
- Your system includes factory contracts that deploy new contracts over time
- You want to index events from all instances of a particular contract type
- The addresses of these contracts aren't known at the time you create your indexer
## Important Notes
- **Block Coverage**: When a dynamic contract is registered, Envio will index all events from that contract in the same block where it was created, even if those events happened in transactions before the registration event. This is particularly useful for contracts that emit events during their construction.
- **Triggering Event**: Any event can trigger a registration — it doesn't have to be the one that created the contract. You might register a token when you see it added to a registry, or when it's first used, rather than at deployment.
## Scaling to Very Large Factories
There is no practical ceiling on how many addresses a factory can register. Before v3.5, HyperIndex started to struggle at around 8 million addresses. That limit is gone — indexers with billions of registered addresses are supported.
You don't need to configure anything for this. Once a contract accumulates enough addresses, HyperIndex automatically changes how it filters events for that contract: instead of asking the data source for a specific address list, it fetches events more broadly and filters them locally. The switch is per contract, so one large factory doesn't affect how your other contracts are indexed.
The trade-off is that a very large factory fetches more data than its handlers ultimately use, since the filtering happens after the fetch rather than before it. That's what keeps indexing fast at this scale.
To see how many addresses a chain has registered, watch `envio_indexing_addresses` on the [metrics endpoint](/docs/HyperIndex/observability#available-metrics).
## Debugging Tips
- Use logging in your `contractRegister` function to confirm contracts are being registered.
- If you're not seeing events from your dynamic contracts, verify they're being properly registered in database.
For more information on writing event handlers, see the [Event Handlers Guide](../Guides/event-handlers.mdx).
---
# Effect API
> Learn how to use the Effect API for external calls in handlers.
import Video from "@site/src/components/Video";
The Effect API provides a powerful and convenient way to perform external calls from your handlers. It's especially effective when used with [Preload Optimization](/docs/HyperIndex/preload-optimization):
- **Automatic batching**: Calls of the same kind are automatically batched together
- **Intelligent memoization**: Calls are memoized, so you don't need to worry about the handler function being called multiple times
- **Deduplication**: Calls with the same arguments are deduplicated to prevent overfetching
- **Persistence**: Built-in support for result persistence for indexer reruns (opt-in via `cache: true`)
- **Future enhancements**: We're working on automatic retry logic and enhanced caching workflows 🏗️
To use the Effect API, you first need to define an effect using `createEffect` function from the `envio` package:
```typescript
import { createEffect, S } from "envio";
export const getMetadata = createEffect(
{
name: "getMetadata",
input: S.string,
output: {
description: S.string,
value: S.bigint,
},
rateLimit: {
calls: 5,
per: "second",
},
cache: true,
},
async ({ input, context }) => {
const response = await fetch(`https://api.example.com/metadata/${input}`);
const data = await response.json();
context.log.info(`Fetched metadata for ${input}`);
return {
description: data.description,
value: data.value,
};
}
);
```
The first argument is an options object that describes the effect:
- `name` (required) - the name of the effect used for debugging and logging
- `input` (required) - the input type of the effect
- `output` (required) - the output type of the effect
- `rateLimit` (required) - the maximum calls allowed per timeframe, or `false` to disable
- `cache` (optional) - save effect results in the database to prevent duplicate calls
- `crossChain` (optional) - whether the cache and rate limit are shared across all chains (default: `true`). See [Per-Chain Effects](#per-chain-effects)
The second argument is a function that will be called with the effect's input.
> **Note:** For type definitions, you should use `S` from the `envio` package, which uses [Sury](https://github.com/DZakh/sury) library under the hood.
After defining an effect, you can use `context.effect` to call it from your handler or another effect.
The `context.effect` function accepts an effect as the first argument and the effect's input as the second argument:
```typescript
import { indexer } from "envio";
indexer.onEvent(
{ contract: "ERC20", event: "Transfer" },
async ({ event, context }) => {
const metadata = await context.effect(getMetadata, event.params.from);
// Process the event with the metadata
},
);
```
### Reading On-Chain State (eth_call)
The Effect API is how you perform `eth_call`-style reads from your handlers — for example, reading a token balance, fetching a contract's name, or querying any view function at a specific block.
### Viem Transport Batching
You can use `viem` or any other blockchain client inside your effect functions. When doing so, it's highly recommended to enable the `batch` option to group all effect calls into fewer RPC requests:
```typescript
// Create a public client to interact with the blockchain
const client = createPublicClient({
chain: mainnet,
// Enable batching to group calls into fewer RPC requests
transport: http(rpcUrl, { batch: true }),
});
// Get the contract instance for your contract
const lbtcContract = getContract({
abi: erc20Abi,
address: "0x8236a87084f8B84306f72007F36F2618A5634494",
client: client,
});
// Effect to get the balance of a specific address at a specific block
export const getBalance = createEffect(
{
name: "getBalance",
input: {
address: S.string,
blockNumber: S.optional(S.bigint),
},
output: S.bigint,
rateLimit: {
calls: 5,
per: "second",
},
cache: true,
},
async ({ input, context }) => {
try {
// If blockNumber is provided, use it to get balance at that specific block
const options = input.blockNumber
? { blockNumber: input.blockNumber }
: undefined;
const balance = await lbtcContract.read.balanceOf(
[input.address as `0x${string}`],
options
);
return balance;
} catch (error) {
context.log.error(`Error getting balance for ${input.address}: ${error}`);
// Return 0 on error to prevent processing failures
return BigInt(0);
}
}
);
```
### Persistence
By default, effect results are not persisted in the database. This means if the effect with the same input is called again, the function will be executed the second time.
To persist effect results, you can set the `cache` option to `true` when creating the effect. This will save the effect results in the database and reuse them in future indexer runs. You can also override caching for a specific call by setting `context.cache = false`, which prevents storing results for that execution, especially useful when handling failed responses.
Example setting cache to false with context.cache:
```typescript
export const getBalance = createEffect(
{
// effect options
cache: true,
},
async ({ input, context }) => {
try {
// your effect logic
} catch (_) {
// Don't cache failed response
context.cache = false;
return undefined;
}
}
);
```
Every effect cache creates a new table in the database `envio_effect_${effectName}`. You can see it and query in Hasura console with admin secret.
Also, use our [Development Console](https://envio.dev/console) to track the cache size and see number of calls which didn't hit the cache.
### Reuse Effect Cache on Indexer Reruns
To prevent invalid data we don't keep the effect cache on indexer reruns. But you can explicitly configure cache, which should be preloaded when the indexer is rerun.
Open [Development Console](https://envio.dev/console) of the running indexer which accumulated the cache. You'll be able to see the `Sync Cache` button right at the `Effects` section. Clicking the button will load the cache from the indexer database to the `.envio/cache` directory in your indexer project.
When the indexer is rerun by using `envio dev` or `envio start -r` call, the initial cache will be loaded from the `.envio/cache` directory and used for the indexer run.
> **Note:** This doesn't support rollbacks on reorgs. The support for reorgs will be added in the future.
### Cache on Envio Cloud
Envio Cloud provides built-in cache management for Effect API results, allowing you to save and restore caches directly from the dashboard without committing files to your repository.
**Key Features:**
- **Save Cache**: Capture effect data from any deployment with one click via Quick Actions
- **Cache Settings**: Manage caches in Settings > Cache - enable/disable caching and select which cache to use
- **Automatic Restore**: New deployments automatically preload effect data from your selected cache
This eliminates the need to commit `.envio/cache` to your repository and removes file size limitations.
For detailed instructions, see the [Effect API Cache documentation](/docs/HyperIndex/hosted-service-features#effect-api-cache).
### Rate Limit
The `rateLimit` option controls how frequently an effect can run within a given timeframe. You can set it to `false` to disable rate limiting or define a custom limit such as calls per second, minute, or a duration in milliseconds.
```typescript
// Effect to get the balance of a specific address at a specific block
export const getBalance = createEffect(
{
name: "getBalance",
input: {
address: S.string,
blockNumber: S.optional(S.bigint),
},
output: S.bigint,
// rateLimit: false, // you can set rateLimit to false if needed
rateLimit: {
calls: 5,
per: "second", // also supports "minute" or a duration in milliseconds
},
cache: true,
},
async ({ input, context }) => {
// your effect logic
}
);
```
Watch the following video to learn more about createEffect and other updates introduced in [v2.32.0](https://github.com/enviodev/hyperindex/releases/tag/v2.32.0).
### Per-Chain Effects
By default an effect is **cross-chain**: one cache and one rate-limit budget are shared by every chain. That's correct when the input fully identifies the result — an IPFS hash means the same thing whichever chain asked for it.
It's wrong when the same input means different things per chain. A token address on Ethereum and the same address on Base are different contracts, so a shared cache would serve one chain's result to the other.
Since v3.3, set `crossChain: false` to scope an effect to the chain that called it:
```typescript
import { createEffect, S } from "envio";
export const getTokenName = createEffect(
{
name: "getTokenName",
input: S.string,
output: S.string,
// Each chain gets its own 5 calls/second budget
rateLimit: {
calls: 5,
per: "second",
},
cache: true,
crossChain: false,
},
async ({ input, context }) => {
// context.chain is only available on `crossChain: false` effects
const client = clientsByChainId[context.chain.id];
return await client.readContract({
address: input,
abi: erc20Abi,
functionName: "name",
});
},
);
```
Setting `crossChain: false` changes three things:
- **Cache** — entries are keyed per chain, so two chains calling with the same input each get their own result.
- **Rate limit** — each chain gets its own budget rather than competing for a shared one. An effect limited to 5 calls/second across 3 chains now allows 15 calls/second in total.
- **`context.chain`** — the handler gains `context.chain.id`, the chain the effect was called on. Accessing `context.chain` on a cross-chain effect throws.
:::warning
`crossChain` is part of the effect's cache identity. Changing it on an effect with `cache: true` invalidates the existing cached results, and they will be refetched on the next run.
:::
### Sending Notifications (Webhooks)
You can use the Effect API to send push notifications or webhook calls when specific events occur. This is useful for alerting systems, Discord/Slack bots, or triggering downstream workflows.
```typescript
import { createEffect, S } from "envio";
export const sendWebhook = createEffect(
{
name: "sendWebhook",
input: {
event: S.string,
data: S.string,
},
output: S.boolean,
rateLimit: {
calls: 10,
per: "second",
},
// Don't cache webhook calls - we want them to fire every time
cache: false,
},
async ({ input, context }) => {
try {
await fetch("https://your-webhook-url.com/notify", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ event: input.event, data: input.data }),
});
return true;
} catch (error) {
context.log.error(`Webhook failed: ${error}`);
return false;
}
}
);
```
Then call it from your handler:
```typescript
import { indexer } from "envio";
indexer.onEvent(
{ contract: "MyContract", event: "LargeTransfer" },
async ({ event, context }) => {
await context.effect(sendWebhook, {
event: "large_transfer",
data: JSON.stringify({
from: event.params.from,
to: event.params.to,
amount: event.params.value.toString(),
}),
});
},
);
```
:::warning
Webhook effects will fire on every indexer re-run unless you set `cache: true`. If you cache them, the webhook will only fire once per unique input. Consider which behavior is appropriate for your use case.
:::
### Migrate from Experimental
If you're migrating from `experimental_createEffect` to `createEffect`, remove the `experimental_` prefix and add the `rateLimit` option, which is now required. In `experimental_createEffect`, the `rateLimit` option was optional and defaulted to `false`.
```diff typescript
- export const getBalance = experimental_createEffect(
+ export const getBalance = createEffect(
{
name: "getBalance",
input: {
address: S.string,
blockNumber: S.optional(S.bigint),
},
output: S.bigint,
+ rateLimit: {
+ calls: 5,
+ per: "second",
+ },
cache: true,
},
async ({ input, context }) => {
// your effect logic
}
);
```
---
# Generated Indexing Files
> Learn how generated files handle type-safe data access, event processing, and runtime indexing.
# Understanding Generated Indexing Files
## Overview
In V3, the local `generated` package is gone. Code generation now writes a single ambient declaration file at `.envio/types.d.ts` (git-ignored) and wires it into your project through a small `envio-env.d.ts` file at the project root. Everything you used to import from `generated` is now exported from the `envio` package.
These generated declarations form the type-level backbone of your blockchain indexer, translating your configuration, schema, and event handlers into the strongly-typed runtime values exposed by `envio`.
> **Important:** The contents of `.envio/` should never be manually edited. Any changes will be overwritten the next time code generation runs.
## What V3 Emits
| File / location | Purpose |
| ----------------------------- | ------------------------------------------------------------------------------------------------ |
| `.envio/types.d.ts` | Ambient TypeScript declarations describing your contracts, events, entities, enums, and chains. |
| `envio-env.d.ts` (root) | Tiny shim that references `.envio/types.d.ts` so the compiler picks it up. |
| `.envio/cache/` (optional) | Local cache of [Effect API](/docs/HyperIndex/effect-api) results, populated via the dev console. |
The `generated/` directory used by V2 (with ReScript sources, JS shims, and a per-project `package.json`) is no longer produced.
## Purpose of Generated Files
Generated files serve several critical functions:
1. **Type-Safe Data Access** - They provide strongly-typed interfaces to interact with your defined entities through `envio`.
2. **Event Processing** - They describe each contract's events so `indexer.onEvent({ contract, event }, ...)` is fully type-checked.
3. **Database Interactions** - They generate the entity types and helper signatures used by `context.` and `indexer.`.
4. **Runtime Orchestration** - They feed into the `indexer` value (chains, contracts, entities) that orchestrates indexing.
## Real-World Example: Uniswap V4 Indexer
Let's examine how specific elements from a real Uniswap V4 indexer translate into generated declarations.
### From Schema to Generated Types
For a schema entity like this:
```graphql
type Pool {
id: ID!
chainId: BigInt!
currency0: String!
currency1: String!
fee: BigInt!
tickSpacing: BigInt!
hooks: String!
numberOfSwaps: BigInt! @index
createdAtTimestamp: BigInt!
createdAtBlockNumber: BigInt!
}
```
The codegen process emits a TypeScript type you can import from `envio`:
```typescript
import type { Entity } from "envio";
// Equivalent to importing the `Pool` named type directly.
type Pool = Entity<"Pool">;
// Shape of the generated entity:
// {
// id: string;
// chainId: bigint;
// currency0: string;
// currency1: string;
// fee: bigint;
// tickSpacing: bigint;
// hooks: string;
// numberOfSwaps: bigint;
// createdAtTimestamp: bigint;
// createdAtBlockNumber: bigint;
// }
```
You read and write `Pool` entities through the type-safe `context` and `indexer` APIs:
```typescript
// Inside a handler
const pool = await context.Pool.get(id);
context.Pool.set({ id, chainId, currency0, currency1, fee, tickSpacing, hooks, numberOfSwaps, createdAtTimestamp, createdAtBlockNumber });
// Inside a test or script
await indexer.Pool.set({ /* ... */ });
const stored = await indexer.Pool.getOrThrow(id);
```
### From Config to Generated Event Handlers
Given a contract event in `config.yaml`:
```yaml
contracts:
- name: PoolManager
events:
- event: Swap(bytes32 indexed id, address indexed sender, int128 amount0, int128 amount1, uint160 sqrtPriceX96, uint128 liquidity, int24 tick, uint24 fee)
```
Codegen widens the `indexer.onEvent` overloads so that the following call is fully typed end-to-end (event params, return type, `context.chain`, etc.):
```typescript
import { indexer } from "envio";
import type { EvmEvent } from "envio";
indexer.onEvent(
{ contract: "PoolManager", event: "Swap" },
async ({ event, context }) => {
const { id, sender, amount0, amount1, sqrtPriceX96, liquidity, tick, fee } = event.params;
// ...
},
);
// The full event payload type is also available as a generic:
type PoolManagerSwapEvent = EvmEvent<"PoolManager", "Swap">;
type PoolManagerSwapBlock = EvmEvent<"PoolManager", "Swap">["block"];
```
### From Multi-Chain Config to Generated Chain Handlers
Your config has multiple chains:
```yaml
chains:
- id: 1 # Ethereum Mainnet
# ...
- id: 10 # Optimism
# ...
- id: 42161 # Arbitrum
# ...
```
Codegen turns the chain set into a literal `ChainId` union and exposes per-chain helpers under `indexer.chains`:
```typescript
import { indexer } from "envio";
import type { ChainId } from "envio";
// ChainId is `1 | 10 | 42161`
const mainnet = indexer.chains[1];
const optimism = indexer.chains[10];
const arbitrum = indexer.chains[42161];
// `chain.id` is also typed inside handler/where callbacks:
indexer.onBlock(
{
name: "Heartbeat",
where: ({ chain }) => {
// chain.id is narrowed to the configured ChainId union
return chain.id === 1;
},
},
async ({ block, context }) => {
// context.chain.id is typed as well
},
);
```
`getGeneratedByChainId(...)` from V2 has been replaced by `indexer.chains[chainId]`.
## When to Run Code Generation
You should run code generation using the Envio CLI whenever you:
```bash
pnpm envio codegen
```
Codegen should be run after:
1. Modifying your `config.yaml` file
2. Changing your GraphQL schema
3. Adding or updating event handlers
4. Switching to a new contract or ABI
5. After pulling changes from version control
> Note: changes to handler files in V3 no longer trigger automatic codegen on `pnpm dev`.
## Troubleshooting Generation Errors
When code generation fails, the errors typically point to issues in your setup files. Here are common error patterns and their solutions:
### Configuration Errors
Error messages containing `Config validation failed` typically mean there's an issue in your `config.yaml` file:
- Check for syntax errors in YAML formatting
- Verify that all required fields are present
- Ensure contract addresses are in the correct format
- Confirm that referenced chains are valid
For example, if you see an error about invalid chain IDs, check that all chain IDs in your config are valid:
```yaml
chains:
- id: 1 # Valid Ethereum mainnet
- id: 10 # Valid Optimism
- id: 999 # Might be invalid if this chain ID isn't recognized
```
### Schema Errors
Errors mentioning `Schema parsing error` point to issues in your GraphQL schema:
- Check for invalid GraphQL syntax
- Ensure entity names match those referenced in handlers
- Verify that relationships between entities are properly defined
- Check for unsupported types or directives
For example, if you're using the `@index` directive as in your `Pool` entity's `numberOfSwaps` field, make sure it's correctly placed:
```graphql
type Pool {
id: ID!
numberOfSwaps: BigInt! @index # Correct placement of @index directive
}
```
### Handler Errors
If you see `Handler validation failed` errors:
- Check that handler function signatures match expected patterns
- Ensure all referenced entities exist in your schema
- Verify proper import syntax for entities and contract events (everything comes from `envio`)
## Relationship with Setup Files
The generated declarations directly reflect the structure defined in your setup files:
- **config.yaml** → Determines which chains, contracts, and events are indexed
- **schema.graphql** → Defines the entities and relationships that are generated
- **handlers in `src/handlers/`** → Provide the business logic that the generated types describe
## Best Practices
1. **Never modify generated files directly** - Always change the source files
2. **Run codegen before starting your indexer** - Ensure all declarations are up to date
3. **Check error messages carefully** - They often pinpoint issues in your setup files
4. **Commit `envio-env.d.ts` but ignore `.envio/`** - The shim is part of the project; the generated artifacts are not.
## Summary
Generated declarations form the critical bridge between your indexing specifications and the actual runtime execution. While you shouldn't modify them directly, understanding their structure and purpose can help you debug issues and optimize your indexing process.
If you encounter persistent errors related to generated files, ensure your configuration, schema, and handlers follow Envio's best practices, or contact support for assistance.
---
---
# HyperSync as Data Source
> Learn how HyperSync is used in HyperIndex to fetch raw blockchain data.
# Using HyperSync as Your Indexing Data Source
> **"Beam me up, Scotty!"** 🖖 — Just like the Star Trek transporter, HyperSync delivers your blockchain data at warp speed.
## What is HyperSync?
HyperSync is a purpose built data-node that helps powers the exceptional performance of HyperIndex. It's a specialized data source optimized for indexing that provides:
- **2000x faster sync speeds** compared to traditional RPC methods
- **Cost-effective data retrieval** with optimized resource usage
- **Flexibility** with the ability to fetch multiple data points in a single round trip with more complex filtering
## How HyperSync Powers Your Indexers
### The Performance Advantage
Traditional blockchain indexing relies on RPC (Remote Procedure Call) endpoints to query blockchain data. While functional, RPCs become highly inefficient when:
- Indexing millions of events
- Processing historical blockchain data
- Extracting data across multiple networks
- Working with thousands of contracts
HyperSync addresses these limitations by providing a streamlined data access layer that dramatically reduces sync times from days to minutes.
### Default Enablement
**HyperSync is used by default** as the data source for all HyperIndex chains. This means:
- No additional configuration is required to benefit from its speed
- No need to worry about RPC rate limiting
- No management of multiple RPC providers
- No costs for external RPC services
:::info
Starting in V3, HyperSync requires an API token. Create a free token at [envio.dev/app/api-tokens](https://envio.dev/app/api-tokens) and expose it to your indexer as `ENVIO_API_TOKEN`:
```bash
export ENVIO_API_TOKEN=your_token_here
```
:::
## Using HyperSync in Your Projects
### Configuration
To use HyperSync (the default), simply don't set an RPC for historical sync in your config. HyperIndex will automatically use HyperSync for supported chains:
```yaml
name: Greeter
description: Greeter indexer
chains:
- id: 137 # Polygon
start_block: 0 # With HyperSync, you can use 0 regardless of contract deployment time
contracts:
- name: PolygonGreeter
abi_file_path: abis/greeter-abi.json
address: "0x9D02A17dE4E68545d3a58D3a20BbBE0399E05c9c"
events:
- event: NewGreeting
- event: ClearGreeting
```
### Smart Block Detection
When using HyperSync, you can specify `start_block: 0` in your configuration. HyperSync will automatically:
1. Detect the first block where your contract was deployed
2. Begin indexing from that block
3. Skip unnecessary processing of earlier blocks
This feature eliminates the need to manually determine the deployment block of your contract, saving setup time and reducing configuration errors.
## Hosting and Support
HyperSync is maintained and hosted by Envio for all supported networks. We handle the infrastructure, allowing you to focus on building your indexer logic.
### Supported Chains
HyperSync supports numerous EVM chains including Ethereum, Unichain, Arbitrum, Optimism, and more. For a complete and up-to-date list of supported chains, see the [HyperSync Supported Networks](/docs/HyperSync/hypersync-supported-networks) documentation.
### Alternative Data Sources
HyperSync data source is vendorlock-free. While HyperSync is **recommended for optimal performance**, you can always switch to RPCs without the need to change your indexer code. For information on configuring RPC-based indexing, visit the [RPC Data Source](/docs/HyperIndex/Advanced/rpc-sync.md) documentation.
### Improving Resilience with RPC fallback
For production deployments, it’s recommended to use HyperSync as the primary data source and have RPCs as fallback to improve reliability.You can read more about it in the [RPC Fallback](/docs/HyperIndex/rpc-sync#improving-resilience-with-rpc-fallback) section.
## Performance Comparison
| Metric | Traditional RPC | HyperSync |
| ------------------ | --------------------- | ---------------------------- |
| Indexing 1M Events | Hours to days | Minutes |
| Resource Usage | High | Optimized |
| Network Calls | Many individual calls | Batched for efficiency |
| Rate Limiting | Common issue | Not applicable |
| Cost | Pay per API call | Included with Envio Cloud |
## Summary
HyperSync provides a significant competitive advantage for Envio indexers by dramatically reducing sync times, lowering costs, and simplifying configuration. By using HyperSync as your default data source, you'll experience:
- Faster indexing performance
- Support for previously impossible indexing cases
- Enhanced reliability
- Reduced operational complexity
To learn more about HyperSync's underlying technology, visit the [HyperSync documentation](/docs/HyperSync/overview).
---
# Loaders
> Historical reference for the V2 Loaders API, replaced by built-in Preload Optimization in V3.
# Loaders Optimization (Removed in V3)
:::warning
The `handlerWithLoader` API and the `loaders` flag in `config.yaml` were removed in HyperIndex V3. [Preload Optimization](/docs/HyperIndex/preload-optimization) is now always on — there is no flag to enable or disable it. This page is kept for historical context and to help V2 projects migrate.
:::
## What Were Loaders?
Loaders were a feature in early V2 versions of HyperIndex that significantly improved database access performance for event handlers.
They worked by implementing the [Preload Optimization](/docs/HyperIndex/preload-optimization) - loading required data upfront before processing events.
The preloaded data would then be available to event handlers through a `loaderReturn` object, eliminating the need for individual database queries during event processing.
In V2, handlers with loaders didn't have a Preload Phase and always ran once. In V3, every handler runs through the Preload Phase automatically, so the dedicated `handlerWithLoader` API and the `loaders:` config flag have both been removed.
The V2 shape looked like this (no longer accepted in V3):
```typescript
// V2 only — removed in V3
ContractName.EventName.handlerWithLoader({
// The loader function runs before event processing starts
loader: async ({ event, context }) => {
// Load all required data from the database
// Return the data needed for event processing
return {}; // This will be available in the handler as loaderReturn
},
// The handler function processes each event with pre-loaded data
handler: async ({ event, context, loaderReturn }) => {
// Process the event using the data returned by the loader
},
});
```
## How It Works in V3
In V3 the optimization is built in. See [Preload Optimization - How It Works?](/docs/HyperIndex/preload-optimization#how-it-works) for the full mechanics. The two-phase execution is identical to what loaders provided, just without a separate API.
For example, this is how a V2 loader is rewritten as a regular V3 handler — the only thing that changed is that the loader code now lives inline at the top of the handler:
```typescript
// V2 — removed
ERC20.Transfer.handlerWithLoader({
loader: async ({ event, context }) => {
// Load sender and receiver accounts efficiently
const sender = await context.Account.get(event.params.from);
const receiver = await context.Account.get(event.params.to);
// Return the loaded data to the handler
return {
sender,
receiver,
};
},
handler: async ({ event, context, loaderReturn }) => {
const { sender, receiver } = loaderReturn;
// Process the transfer with the pre-loaded data
// No database lookups needed here!
},
});
// V3 — Preload Optimization is always on
import { indexer } from "envio";
indexer.onEvent(
{ contract: "ERC20", event: "Transfer" },
async ({ event, context }) => {
// Load sender and receiver accounts efficiently
const sender = await context.Account.get(event.params.from);
const receiver = await context.Account.get(event.params.to);
// To imitate the behavior of the loader,
// we can use `context.isPreload` to make next code run only once.
// Note: This is not required, but might be useful for CPU-intensive operations.
if (context.isPreload) {
return;
}
// Process the transfer with the pre-loaded data
},
);
```
If your project still has `loaders: true` or `preload_handlers: true` in `config.yaml`, remove both fields — V3 will reject them.
---
# MCP Server
> Connect AI coding assistants to Envio documentation using the Model Context Protocol (MCP) server
# MCP Server
Envio provides a Model Context Protocol (MCP) server that lets AI coding assistants search and retrieve documentation directly. This means tools like Claude Code, Cursor, and other MCP-compatible clients can access Envio docs without you needing to copy-paste context manually.
## Endpoint
```
https://docs.envio.dev/mcp
```
## Available Tools
The MCP server exposes two tools:
| Tool | Description |
|------|-------------|
| `docs_search` | Full-text search across all documentation. Returns matching pages with titles, URLs, and content snippets. |
| `docs_fetch` | Retrieves the full content of a documentation page as markdown. |
## Envio CLI Tools
Starting from Envio `3.1`, the same documentation search and retrieval is also available directly through the Envio CLI — no MCP server setup required. AI agents can call these tools out of the box:
| Tool | Description |
|------|-------------|
| `envio tools search-docs ` | Full-text search across all documentation, equivalent to the `docs_search` MCP tool. |
| `envio tools fetch-docs ` | Retrieves the full content of a documentation page as markdown, equivalent to the `docs_fetch` MCP tool. |
## Setup
### Claude Code
```bash
claude mcp add --transport http envio-docs https://docs.envio.dev/mcp
```
### Cursor / VS Code
Add the following to your MCP configuration (`.cursor/mcp.json` or VS Code MCP settings):
```json
{
"mcpServers": {
"envio-docs": {
"url": "https://docs.envio.dev/mcp"
}
}
}
```
### Other MCP Clients
Point any MCP-compatible client to the endpoint URL above using the **Streamable HTTP** transport.
---
# Multichain Indexing
> Learn how to index and process events across multiple chains with HyperIndex.
# Understanding Multichain Indexing
For a conceptual overview of what multichain indexing is and when to use it, see [What is multichain indexing?](/blog/what-is-multi-chain-indexing). This page covers the HyperIndex configuration and patterns.
Multichain indexing allows you to monitor and process events from contracts deployed across multiple blockchain networks within a single indexer instance. This capability is essential for applications that:
- Track the same contract deployed across multiple networks
- Need to aggregate data from different chains into a unified view
- Monitor cross-chain interactions or state
## How It Works
With multichain indexing, events from contracts deployed on multiple chains can be used to create and update entities defined in your schema file. Your blockchain indexer will process events from all configured networks, maintaining proper synchronization across chains.
## Configuration Requirements
To implement multichain indexing, you need to:
1. Populate the `chains` section in your `config.yaml` file for each chain
2. Specify contracts to index from each chain
3. Create event handlers for the specified contracts
## Real-World Example: Uniswap V4 Multichain Indexer
For a comprehensive, production-ready example of multichain indexing, we recommend exploring our [Uniswap V4 Multichain Indexer](/docs/HyperIndex/example-uniswap-v4-multi-chain-indexer). This official reference implementation:
- Indexes Uniswap V4 deployments across 10 different blockchain chains
- Powers the official [v4.xyz](https://v4.xyz) interface with real-time data
- Demonstrates best practices for high-performance multichain indexing
- Provides a complete, production-grade implementation you can study and adapt
The Uniswap V4 indexer showcases how to effectively structure a multichain indexer for a complex DeFi protocol, handling high volumes of data across multiple networks while maintaining performance and reliability.
## Config File Structure for Multichain Indexing
The `config.yaml` file for multichain indexing contains three key sections:
1. **Global contract definitions** - Define contracts, ABIs, and events once
2. **Chain-specific configurations** - Specify chain IDs and starting blocks
3. **Contract instances** - Reference global contracts with chain-specific addresses
```yaml
# Example structure (simplified)
contracts:
- name: ExampleContract
abi_file_path: ./abis/example-abi.json
events:
- event: ExampleEvent
chains:
- id: 1 # Ethereum Mainnet
start_block: 0
contracts:
- name: ExampleContract
address: "0x1234..."
- id: 137 # Polygon
start_block: 0
contracts:
- name: ExampleContract
address: "0x5678..."
```
### Key Configuration Concepts
- The global `contracts` section defines the contract interface, ABI, handlers, and events once
- The `chains` section lists each blockchain chain you want to index
- Each chain entry references the global contract and provides the chain-specific address
- This structure allows you to reuse the same handler functions and event definitions across chains
> 📢 **Best Practice**: When developing multichain indexers, append the chain ID to entity IDs to avoid collisions. For example: `user-1` for Ethereum and `user-137` for Polygon.
## Multichain Event Ordering
In V3 the indexer always processes multichain events in **unordered** mode. Events from different chains are processed as soon as they're available, without waiting for the other chains, which keeps latency low.
- Events are still processed in order within each individual chain.
- Events across different chains may be processed out of order.
- Processing happens as soon as events are emitted, so you don't wait for the slowest chain's block time.
This is ideal when:
- Operations on your entities are [commutative](./terminology#commutative-property) (order doesn't matter).
- Entities from different chains never interact with each other.
- Processing speed matters more than guaranteed cross-chain ordering.
The V2 `unordered_multichain_mode` option, the `multichain: ordered` opt-in, and the `UNORDERED_MULTICHAIN_MODE` / `UNSTABLE__TEMP_UNORDERED_HEAD_MODE` environment variables have all been removed in V3 — there is nothing to configure.
## Ordered Multichain Mode
HyperIndex V3 doesn't offer an ordered multichain mode, and it's a deliberate choice rather than a feature gap. Ordered mode worked by pausing every chain until events from the slowest chain had caught up, so the moment one chain hiccuped (RPC rate limits, a slow block, a brief reorg) every other chain stalled with it. In practice that meant a multichain indexer was only ever as reliable and as fast as its worst-performing chain, and even healthy chains paid for that coupling with significantly higher latency at the head.
You almost always get better reliability and lower latency by keeping every chain unordered and modeling the cross-chain relationship in your schema instead. Each chain writes a small temporary entity when its side of a cross-chain interaction happens, and the second chain (whichever arrives last) reads those temporary entities and finalizes the unified entity. Because each chain progresses independently, none of them are blocked on each other.
A typical pattern for an `A → B` cross-chain message:
```graphql
type CrossChainMessage {
id: ID! # The shared cross-chain message id (e.g. nonce + originChainId)
sourceChainId: Int
sourceTxHash: String
destinationChainId: Int
destinationTxHash: String
status: String! # "sent" | "delivered"
}
```
```typescript
import { indexer } from "envio";
// Chain A: the message was emitted. Create or update the entity with the
// "sent" side of the data. The destination handler may run before or after.
indexer.onEvent(
{ contract: "Bridge", event: "MessageSent" },
async ({ event, context }) => {
const id = `${event.params.originChainId}-${event.params.nonce}`;
const existing = await context.CrossChainMessage.get(id);
context.CrossChainMessage.set({
id,
sourceChainId: event.chainId,
sourceTxHash: event.transaction.hash,
destinationChainId: existing?.destinationChainId,
destinationTxHash: existing?.destinationTxHash,
status: existing?.destinationTxHash ? "delivered" : "sent",
});
},
);
// Chain B: the message was delivered. Read the (maybe-already-existing)
// entity and fill in the destination side, regardless of which side arrived
// first.
indexer.onEvent(
{ contract: "Bridge", event: "MessageDelivered" },
async ({ event, context }) => {
const id = `${event.params.originChainId}-${event.params.nonce}`;
const existing = await context.CrossChainMessage.get(id);
context.CrossChainMessage.set({
id,
sourceChainId: existing?.sourceChainId,
sourceTxHash: existing?.sourceTxHash,
destinationChainId: event.chainId,
destinationTxHash: event.transaction.hash,
status: existing?.sourceTxHash ? "delivered" : "sent",
});
},
);
```
The same shape works for any "rendezvous" where two chains contribute parts of one logical record (bridges, cross-chain governance, multichain user profiles, etc.). The "temporary" entity is just a regular entity that gets progressively completed as each chain's events arrive — there's no special API to learn, and the indexer stays fast and resilient because no chain ever waits on another.
If you have a use case that genuinely cannot be expressed this way, reach out on [Discord](https://discord.gg/envio) — we'd like to hear it.
## Best Practices for Multichain Indexing
### 1. Entity ID Namespacing
Always namespace your entity IDs with the chain ID to prevent collisions between chains. This ensures that entities from different chains remain distinct.
### 2. Error Handling
Implement robust error handling for chain-specific issues. A failure on one chain shouldn't prevent indexing from continuing on other chains.
### 3. Testing
- Test your indexer with realistic scenarios across all chains
- Use testnet deployments for initial validation
- Verify entity updates work correctly across chains
### 4. Performance Considerations
- Consider your indexing frequency based on the block times of each chain.
- Monitor resource usage, as indexing multiple chains increases load.
- Adding more chains does **not** linearly degrade performance — chains are indexed in parallel.
### 5. Adding a New Chain to an Existing Indexer
To add a new chain to a running indexer:
1. Add the new chain entry to your `config.yaml` with the appropriate `start_block` and contract addresses
2. Push the updated code to your deployment branch (for Envio Cloud) or restart locally with `pnpm envio dev -r`
On **Envio Cloud**, this creates a new deployment that re-indexes all chains (including the new one). Your previous deployment continues serving queries with zero downtime until the new deployment is fully synced. See the [deployment guide](../Hosted_Service/hosted-service-deployment.md#updating-your-deployment) for details.
**Locally**, adding a new chain requires a restart and will re-index all chains from their respective start blocks. Note that in V3, `envio dev` no longer auto-resets the database — pass `-r` (or `--restart`) explicitly when you want a fresh sync. `envio start` is now production-only.
## Troubleshooting Common Issues
1. **Entity Conflicts**: If you see unexpected entity updates, verify that your entity IDs are properly namespaced with chain IDs.
2. **Memory Usage**: If your indexer uses excessive memory, consider optimizing your entity structure and implementing pagination in your queries.
## Next Steps
- Explore our [Uniswap V4 Multichain Indexer](/docs/HyperIndex/example-uniswap-v4-multi-chain-indexer) for a complete implementation
- Review [performance optimization techniques](/docs/HyperIndex/performance) for your indexer
---
# Observability
> Monitor a HyperIndex indexer in development and production — Terminal UI, logs, Prometheus metrics, health checks, indexing status via the _meta query, the Dev Console, and Envio Cloud monitoring.
# Observability
HyperIndex gives you several complementary ways to understand what your indexer
is doing — while you develop locally and once it's running in production:
- **[Terminal UI](#terminal-ui)** — a live dashboard in your terminal.
- **[Logs](#logs)** — structured logs from the runtime and your handlers.
- **[Metrics](#metrics)** — Prometheus metrics at the `/metrics` endpoint.
- **[Health checks](#health-checks)** — a `/healthz` endpoint for liveness probes.
- **[Indexing status](#indexing-status)** — per-chain progress via the `_meta` GraphQL query.
- **[Dev Console](#dev-console)** — a web UI for debugging local development.
- **[Envio Cloud](#envio-cloud)** — managed dashboards, alerts, and metrics for [hosted indexers](/docs/HyperIndex/hosted-service).
:::info Endpoints & port
When the indexer starts it serves an HTTP server (default port **`9898`**) that
exposes `/metrics`, `/metrics/runtime`, and `/healthz`. Change the port with
`ENVIO_INDEXER_PORT`:
```bash
ENVIO_INDEXER_PORT=9899 envio start
```
:::
## Terminal UI
Running `envio dev` (or `envio start`) launches a live **Terminal UI (TUI)** that
shows the state of your indexer at a glance:
- Per-chain sync status and block numbers
- Events processed and indexing progress
- The GraphQL (Hasura) endpoint, and the [Dev Console](#dev-console) URL when running `envio dev`
- HyperSync rate-limit information when you're being throttled
### Disabling the TUI
The TUI is enabled by default in interactive terminals. It is **automatically
disabled** when the process is non-interactive — when `stdout` is not a TTY, in
CI (`CI` is set), under AI coding agents, or when `TERM=dumb`.
To disable it explicitly (for example, to capture plain logs), set `ENVIO_TUI`
or pass the CLI flag:
```bash
export ENVIO_TUI=false # or: envio dev --tui-off
```
:::tip Keep the TUI while capturing logs
The TUI can sometimes obscure errors. To keep the dashboard **and** write full
logs to a file at the same time:
```bash
export LOG_STRATEGY="both-prettyconsole"
export LOG_FILE="./debug.log"
```
:::
## Logs
HyperIndex uses [pino](https://github.com/pinojs/pino/) for high-performance
structured logging. Logs come from two sources: the indexer runtime, and your
own handler code.
### Log levels
The runtime supports the following levels, in ascending order of severity.
Levels prefixed with `u` are **user-level** logs emitted from your handlers and
loaders via the `context.log` API.
| Level | Description |
|-------|-------------|
| `trace` | Most verbose; detailed tracing information |
| `debug` | Debugging information for developers |
| `info` | General information about system operation (**default**) |
| `udebug` | User-level debug logs |
| `uinfo` | User-level info logs |
| `uwarn` | User-level warning logs |
| `uerror` | User-level error logs |
| `warn` | System warnings |
| `error` | System errors |
| `fatal` | Critical errors causing shutdown |
Configure levels with environment variables:
```bash
LOG_LEVEL="info" # Level for console output (default: info)
FILE_LOG_LEVEL="trace" # Level for file output (default: trace)
```
Use `LOG_LEVEL="silent"` to suppress console logs entirely.
### Log format
The default format is human-readable with color-coded levels. Switch formats
with `LOG_STRATEGY` — the [ECS](https://www.elastic.co/guide/en/ecs/current/index.html)
formats are convenient for shipping logs to the Elastic Stack (Kibana):
```bash
LOG_STRATEGY="console-pretty" # Default: human-readable, colored console logs
LOG_STRATEGY="console-raw" # Raw pino JSON to console
LOG_STRATEGY="ecs-file" # ECS-formatted JSON to a file
LOG_STRATEGY="ecs-console" # ECS-formatted JSON to console
LOG_STRATEGY="file-only" # pino JSON to a file (most efficient)
LOG_STRATEGY="both-prettyconsole" # Pretty console + pino JSON to a file
# Where file-based strategies write to (default: logs/envio.log)
LOG_FILE="./logs/envio.log"
```
### User logs
Inside your handlers, use the logging methods on the `context` object. They map
to the `u*` levels above:
```typescript
// Inside your handler
context.log.debug(`Processing event in block: ${event.block.number}`);
context.log.info(`Handled transfer of ${event.params.value}`);
context.log.warn(`Unexpected state for ${event.params.from}`);
context.log.error(`Failed to process event: ${event.transaction.hash}`);
// Pass an Error as the second argument to capture a stack trace:
context.log.error("Failed to process event", new Error("boom"));
// Or pass an object for structured logging:
context.log.info("Processing blockchain event", {
blockNumber: event.block.number,
contract: "ERC20",
data: { from: event.params.from, to: event.params.to },
});
```
For production analytics, pair the ECS log strategies with a tool like Kibana to
build dashboards and alerts from your logs.
:::tip Performance
Logging isn't free — excessive logging (especially per-event `debug`/`trace`
logs on high-throughput chains) can slow indexing down. Keep only the logs that
serve a real observability need, and lean on [metrics](#metrics) rather than logs
for high-frequency signals.
:::
## Metrics
HyperIndex exposes [Prometheus](https://prometheus.io/) metrics so you can plug
into your existing monitoring stack (Prometheus, Grafana, Datadog, etc.).
:::info Stability
As of **v3.0.0** the `/metrics` endpoint is official: metric names follow
Prometheus conventions, time is measured in seconds, and the metric surface is
covered by semver and documented here. The one exception to the seconds
convention is `envio_progress_latency`, which is reported in milliseconds.
:::
### The `/metrics` endpoint
The running indexer serves Prometheus metrics at:
```
http://localhost:9898/metrics
```
A separate `/metrics/runtime` endpoint exposes Node.js process metrics (CPU,
memory, garbage collection, event-loop lag) from a dedicated registry, isolated
from the index metrics:
```
http://localhost:9898/metrics/runtime
```
### Fetching metrics from the CLI
You don't need to `curl` the endpoint by hand — the CLI fetches metrics from a
locally running indexer for you:
```bash
envio metrics # raw Prometheus metrics from /metrics
envio metrics runtime # runtime metrics from /metrics/runtime
```
The command resolves the port automatically from `ENVIO_INDEXER_PORT` in your
shell session or the `.env` file at your project root, falling back to `9898`.
### Available metrics
Every metric is prefixed with `envio_`. Counters and gauges are labeled where
noted (e.g. by `chainId`, `contract`, `event`).
#### Progress & sync status
| Metric | Type | Description |
|--------|------|-------------|
| `envio_progress_block` | gauge | Latest block number processed and stored in the database. Labeled by `chainId`. |
| `envio_progress_events` | gauge | Number of events processed and reflected in the database. Labeled by `chainId`. |
| `envio_progress_ready` | gauge | Whether the chain is fully synced to the head (`1` = synced). Labeled by `chainId`. |
| `envio_progress_latency` | gauge | Milliseconds between the latest processed event being created on chain and being written to storage. Labeled by `chainId`. |
#### Event processing
| Metric | Type | Description |
|--------|------|-------------|
| `envio_processing_seconds` | counter | Cumulative time spent executing event handlers during batch processing. |
| `envio_processing_handler_seconds` | counter | Cumulative time spent inside individual event handler executions. Labeled by `contract` and `event`. |
| `envio_processing_handler_total` | counter | Total number of individual event handler executions. Labeled by `contract` and `event`. |
| `envio_processing_max_batch_size` | gauge | Maximum number of items to process in a single batch. |
| `envio_processing_stalled_on_fetch_seconds` | counter | Time the indexer had nothing to process while waiting for events to be fetched. Waiting at the chain head for new blocks is not counted. |
| `envio_processing_stalled_on_storage_write_seconds` | counter | Time the indexer paused processing because too many changes were still waiting to be written. |
#### Entity preloading
| Metric | Type | Description |
|--------|------|-------------|
| `envio_preload_seconds` | counter | Cumulative time spent preloading entities during batch processing. |
| `envio_preload_handler_seconds` | counter | Wall-clock time spent inside individual preload handler executions. Labeled by `contract` and `event`. |
| `envio_preload_handler_seconds_total` | counter | Cumulative time spent in preload handlers (can exceed wall-clock time due to parallel execution). Labeled by `contract` and `event`. |
| `envio_preload_handler_total` | counter | Total number of individual preload handler executions. Labeled by `contract` and `event`. |
#### Storage
| Metric | Type | Description |
|--------|------|-------------|
| `envio_storage_write_seconds` | counter | Cumulative time spent writing batch data to storage. |
| `envio_storage_write_total` | counter | Total number of batch writes to storage. |
| `envio_storage_load_seconds` | counter | Time spent loading data from storage. Labeled by `operation`. |
| `envio_storage_load_seconds_total` | counter | Cumulative time spent loading data from storage during indexing. Labeled by `operation`. |
| `envio_storage_load_total` | counter | Number of successful storage load operations. Labeled by `operation`. |
| `envio_storage_load_size` | counter | Cumulative number of records loaded from storage. Labeled by `operation`. |
| `envio_storage_load_where_size` | counter | Cumulative number of filter conditions (`where` items) used in storage load operations. Labeled by `operation`. |
#### Data source & fetching
| Metric | Type | Description |
|--------|------|-------------|
| `envio_fetching_block_range_seconds` | counter | Cumulative time spent fetching block ranges. Labeled by `chainId`. |
| `envio_fetching_block_range_total` | counter | Total number of block range fetch operations. Labeled by `chainId`. |
| `envio_fetching_block_range_events_total` | counter | Cumulative number of events fetched across all block range operations. Labeled by `chainId`. |
| `envio_fetching_block_range_size` | counter | Cumulative number of blocks covered across all fetch operations. Labeled by `chainId`. |
| `envio_fetching_block_range_parse_seconds` | counter | Cumulative time spent parsing block range fetch responses. Labeled by `chainId`. |
| `envio_source_request_total` | counter | Number of requests made to data sources. Labeled by `source`, `chainId`, and `method`. |
| `envio_source_request_seconds_total` | counter | Cumulative time spent on data source requests. Labeled by `source`, `chainId`, and `method`. |
| `envio_source_known_height` | gauge | Latest known block number reported by the data source. Labeled by `source` and `chainId`. |
#### Indexing pipeline
| Metric | Type | Description |
|--------|------|-------------|
| `envio_indexing_known_height` | gauge | Latest known block number reported by the active indexing source. Labeled by `chainId`. |
| `envio_indexing_concurrency` | gauge | Number of executing concurrent queries to the chain data source. Labeled by `chainId`. |
| `envio_indexing_buffer_size` | gauge | Current number of items in the indexing buffer. Labeled by `chainId`. |
| `envio_indexing_buffer_block` | gauge | Highest block number fully fetched by the indexer. Labeled by `chainId`. |
| `envio_indexing_idle_seconds` | counter | Time the indexer source syncing has been idle. A high value may indicate a bottleneck. Labeled by `chainId`. |
| `envio_indexing_partitions` | gauge | Number of partitions used to split fetching logic. Labeled by `chainId`. |
| `envio_indexing_addresses` | gauge | Number of addresses indexed on chain (static and dynamic). Labeled by `chainId`. |
| `envio_indexing_target_buffer_size` | gauge | Indexer-wide target buffer size shared across all chains. The queue may exceed it, but the indexer always tries to keep the buffer filled up to this target. |
| `envio_indexing_end_block` | gauge | The block number to stop indexing at (inclusive). Labeled by `chainId`. |
| `envio_indexing_source_querying_seconds` | counter | Time spent performing queries to the chain data source. Labeled by `chainId`. |
| `envio_indexing_source_waiting_seconds` | counter | Time the indexer has been waiting for new blocks. Labeled by `chainId`. |
#### Reorgs & rollbacks
| Metric | Type | Description |
|--------|------|-------------|
| `envio_reorg_detected_total` | counter | Total number of reorgs detected. |
| `envio_reorg_detected_block` | gauge | Block number where a reorg was last detected. |
| `envio_reorg_threshold` | gauge | Whether indexing is currently within the reorg threshold. |
| `envio_rollback_enabled` | gauge | Whether rollback on reorg is enabled. |
| `envio_rollback_total` | counter | Number of successful rollbacks on reorg. |
| `envio_rollback_seconds` | counter | Total time spent on rollbacks. |
| `envio_rollback_events` | counter | Number of events rolled back on reorg. |
| `envio_rollback_target_block` | gauge | Block number the last reorg was rolled back to. Labeled by `chainId`. |
| `envio_rollback_history_prune_total` | counter | Number of successful entity history prunes. Labeled by `entity`. |
| `envio_rollback_history_prune_seconds` | counter | Total time spent pruning entity history outside the reorg threshold. Labeled by `entity`. |
#### Effect API
Metrics for the [Effect API](/docs/HyperIndex/effect-api).
| Metric | Type | Description |
|--------|------|-------------|
| `envio_effect_call_seconds` | counter | Processing time taken to call the Effect function. Labeled by `effect`. |
| `envio_effect_call_seconds_total` | counter | Cumulative time spent calling the Effect function during indexing. Labeled by `effect`. |
| `envio_effect_call_total` | counter | Cumulative number of resolved Effect function calls. Labeled by `effect`. |
| `envio_effect_active_calls` | gauge | Number of Effect function calls currently running. Labeled by `effect`. |
| `envio_effect_cache` | gauge | Number of items in the effect cache. Labeled by `effect`. |
| `envio_effect_cache_invalidations` | counter | Number of effect cache invalidations. Labeled by `effect`. |
| `envio_effect_queue` | gauge | Number of effect calls waiting in the rate-limit queue. |
| `envio_effect_queue_wait_seconds` | counter | Time spent waiting in the rate-limit queue. Labeled by `effect`. |
#### System info
| Metric | Type | Description |
|--------|------|-------------|
| `envio_info` | gauge | Information about the indexer. Labeled by `version`. |
| `envio_process_start_time_seconds` | gauge | Start time of the process since the Unix epoch, in seconds. |
| `envio_process_metric_time_seconds` | gauge | The time these metrics were collected. Use it to tell how fresh a snapshot is, or to measure rates between two snapshots. |
| `envio_process_elapsed_seconds` | gauge | How long the indexer has been running. Divide a cumulative seconds metric by this to get the share of the run it took. |
### Scraping with Prometheus
Add a scrape job to your `prometheus.yml`:
```yaml
scrape_configs:
- job_name: "envio-indexer"
metrics_path: "/metrics"
static_configs:
- targets: ["localhost:9898"]
```
### Key metrics to watch
- **`envio_progress_ready`** — confirm your indexer is caught up to the chain head.
- **`envio_progress_block`** — track indexing progress over time.
- **`envio_processing_handler_seconds`** — identify slow event handlers by contract and event.
- **`envio_indexing_idle_seconds`** — a high value may indicate the data source sync is a bottleneck.
- **`envio_reorg_detected_total`** — track how frequently chain reorganizations occur.
### Finding the bottleneck
A slow indexer is usually waiting on one of two things: fetching events, or writing them. Since v3.5, two counters attribute that wait directly, so you no longer have to infer it from handler timings:
- **`envio_processing_stalled_on_fetch_seconds`** — the indexer had nothing to process because events hadn't arrived yet. A high rate means **fetching** is the bottleneck: check data-source latency and whether it can take more concurrency. Time spent waiting at the chain head for new blocks is deliberately excluded, so a fully synced indexer doesn't look stalled.
- **`envio_processing_stalled_on_storage_write_seconds`** — the indexer paused because too many changes were still queued for writing. A high rate means **storage writes** are the bottleneck: cross-check `envio_storage_write_seconds` and your [database performance](/docs/HyperIndex/database-performance-optimization).
Both are cumulative counters, so read them as a rate. To turn one into a share of total runtime, divide by `envio_process_elapsed_seconds`:
```promql
# Fraction of the last 5 minutes spent stalled on fetches (0 to 1)
rate(envio_processing_stalled_on_fetch_seconds[5m])
# Same, but across the whole run so far
envio_processing_stalled_on_fetch_seconds / envio_process_elapsed_seconds
```
The same division works for any cumulative seconds metric — `envio_processing_seconds / envio_process_elapsed_seconds` gives the share of the run spent inside event handlers. If you scrape snapshots by hand rather than with Prometheus, `envio_process_metric_time_seconds` tells you when each was collected.
## Health checks
The indexer exposes a `/healthz` endpoint that returns HTTP `200` once the
service is up. It's designed as a machine-readable liveness probe — for example,
a Kubernetes `livenessProbe`:
```
GET http://localhost:9898/healthz → 200 OK
```
```yaml
livenessProbe:
httpGet:
path: /healthz
port: 9898
initialDelaySeconds: 10
periodSeconds: 10
```
## Indexing status
To track per-chain indexing progress from your application, query the official
`_meta` field on the GraphQL API. This is the most reliable way to know whether a
chain has caught up before you read indexed data — useful for health checks,
custom dashboards, and waiting for a write to be indexed before refetching.
```graphql
{
_meta {
chainId
progressBlock
eventsProcessed
bufferBlock
firstEventBlock
sourceBlock
readyAt
isReady
startBlock
endBlock
}
}
```
Result:
```json
{
"data": {
"_meta": [
{
"chainId": 1,
"progressBlock": 22817138,
"eventsProcessed": 2380000,
"bufferBlock": 22820499,
"firstEventBlock": 21688545,
"sourceBlock": 23368264,
"readyAt": null,
"isReady": false,
"startBlock": 0,
"endBlock": null
},
{
"chainId": 10,
"progressBlock": 137848820,
"eventsProcessed": 2455000,
"bufferBlock": 137873621,
"firstEventBlock": 130990676,
"sourceBlock": 141168975,
"readyAt": null,
"isReady": false,
"startBlock": 0,
"endBlock": null
}
]
}
}
```
### Metadata fields
#### Configuration fields
These fields are populated on indexer startup and don't change during the
indexer process.
- `chainId` — Chain ID the metadata belongs to. Results are sorted by `chainId` in ascending order. Use `_meta(where: { chainId: { _eq: 1 } })` to get the metadata for a specific chain.
- `startBlock` — Start block number from `config.yaml`.
- `endBlock` — End block number from `config.yaml`.
#### Transactional fields
These fields are updated in the batch write transaction, and are guaranteed to
be written to the database at the same time. This means `progressBlock` and
`eventsProcessed` increase at the same moment the data for the processed events
is written to the database and becomes available for querying.
- `progressBlock` — Block number fully processed and written to the DB.
- `eventsProcessed` — Number of processed events written to the DB (reorg resistant).
#### Throttled fields
These fields are updated outside of the batch transaction and throttled to avoid
performance overhead. There might be a small delay between the event processing
and the metadata update.
- `bufferBlock` — Block number of the latest event ready for processing.
- `firstEventBlock` — Block number of the first processed event for the chain.
- `sourceBlock` — The latest known block number of the actively used data source (the chain head).
- `readyAt` — Timestamp when the chain finished historical sync or reached its end block.
- `isReady` — Whether the chain finished historical sync or reached its end block.
## Dev Console
When you run `envio dev`, HyperIndex enables the **Dev Console** — a web UI for
debugging your indexer during local development, available at
[envio.dev/console](https://envio.dev/console). The TUI also prints the link on
startup.
The Dev Console connects to your locally running indexer, so your data never
leaves your machine. It's available in development mode only — `envio start`
(production) does not expose console state.
Use it to inspect indexing state and to manage the [Effect API](/docs/HyperIndex/effect-api)
cache while iterating.
:::note Local GraphQL (Hasura)
`envio dev` also starts a local [Hasura](/docs/HyperIndex/navigating-hasura)
console at [http://localhost:8080](http://localhost:8080) for exploring your
indexed data. The default admin secret is `testing`. Disable Hasura with
`ENVIO_HASURA=false`.
:::
## Envio Cloud
If you deploy to [Envio Cloud](/docs/HyperIndex/hosted-service), observability is
fully managed for you — no need to run Prometheus or ship logs yourself:
- **[Real-time dashboard](/docs/HyperIndex/hosted-service-monitoring#dashboard-overview)** — sync status, events processed, and per-network progress bars.
- **[Logs](/docs/HyperIndex/hosted-service-monitoring#logging)** — live and historical logs with level filtering, integrated and configured by Envio.
- **[Built-in alerts](/docs/HyperIndex/hosted-service-monitoring#built-in-alerts)** — get notified via Discord, Slack, Telegram, Email, or a generic webhook when your endpoint goes down or your indexer stops processing.
Envio Cloud also exposes the same **Prometheus metrics** described above, so you
can scrape them into your own Grafana or Datadog. On Cloud the endpoint is served
under your deployment's URL:
```
/hyperindex/metrics
```
:::info Version requirement
Prometheus metrics on Envio Cloud require indexers deployed with version
**3.0.0** or higher.
:::
See [Monitoring Your Indexer](/docs/HyperIndex/hosted-service-monitoring) and the
[Envio Cloud features](/docs/HyperIndex/hosted-service-features) page for details,
and [Envio Cloud CLI](/docs/HyperIndex/envio-cloud-cli) for monitoring deployments
from the command line.
---
# Benchmarking Your Indexer
> Learn how to benchmark your indexer to identify bottlenecks and optimize performance.
# Benchmarking Your Indexer Performance
## Why Benchmark Your Indexer?
Benchmarking is a critical tool for understanding and optimizing your indexer's performance. By collecting and analyzing performance metrics, you can:
- Identify bottlenecks in your indexing pipeline
- Determine if performance issues are due to data fetching, processing, or database operations
- Measure the impact of code optimizations
- Set realistic expectations for indexing speed
- Plan infrastructure requirements for production deployments
## Running Benchmarks
### Capturing Benchmark Data
To collect performance metrics while your indexer is running:
```bash
pnpm envio start --bench
```
> **Note:** Benchmarking adds some memory and processing overhead. It should not be enabled in production environments, as it holds benchmark data points in memory and periodically writes them to disk.
### Viewing Benchmark Results
After running your indexer with benchmarking enabled, you can generate a performance summary:
```bash
pnpm envio benchmark-summary
```
This command processes the collected benchmark data and displays a comprehensive performance report.
## Understanding Benchmark Output
The benchmark output is divided into several sections, each providing insights into different aspects of your indexer's performance:
### Time Breakdown
```
Time breakdown
┌─────────────────────────────────────────┬─────────┐
│ (index) │ seconds │
├─────────────────────────────────────────┼─────────┤
│ Total Runtime │ 45 │
│ Total Time Fetching Chain 1 Partition 0 │ 44 │
│ Total Time Processing │ 9 │
└─────────────────────────────────────────┴─────────┘
```
**What This Tells You:**
- **Total Runtime**: Overall time the indexer has been running
- **Total Time Fetching**: Time spent retrieving data from the blockchain
- **Total Time Processing**: Time spent in event handlers and database operations
**How to Interpret:**
- If fetching time dominates (as in this example), your bottleneck is data retrieval, not processing
- If processing time is high relative to fetching, your handlers may need optimization
- Note that fetching and processing can overlap, so the sum may exceed total runtime
### General Performance Metrics
```
General
┌─────────────────────┬─────────┐
│ (index) │ Values │
├─────────────────────┼─────────┤
│ batch sizes sum │ 158205 │
│ total runtime (sec) │ 45.801 │
│ events per second │ 3454.18 │
└─────────────────────┴─────────┘
```
**What This Tells You:**
- **Batch Sizes Sum**: Total number of events processed
- **Total Runtime**: Precise runtime in seconds
- **Events Per Second**: Overall processing throughput
**How to Interpret:**
- Events per second is your key performance indicator
- Over 10,000 events/second represents excellent performance
- 1,000-5,000 events/second indicates good performance
- Under 500 events/second may indicate optimization opportunities
### Block Fetching Performance
```
BlockRangeFetched Summary for Chain 1 Root Register
┌───────────────────────────┬────┬───────────┬────────────┬──────┬──────────┬──────────┐
│ (index) │ n │ mean │ std-dev │ min │ max │ sum │
├───────────────────────────┼────┼───────────┼────────────┼──────┼──────────┼──────────┤
│ Total Time Elapsed (ms) │ 12 │ 3675.17 │ 1147.69 │ 2329 │ 5972 │ 44102 │
│ Parsing Time Elapsed (ms) │ 12 │ 142.17 │ 40.15 │ 80 │ 235 │ 1706 │
│ Page Fetch Time (ms) │ 12 │ 3481.58 │ 1042.93 │ 2249 │ 5737 │ 41779 │
│ Num Events │ 12 │ 13183.75 │ 3858.92 │ 7579 │ 22426 │ 158205 │
│ Block Range Size │ 12 │ 906593.17 │ 3006127.15 │ 149 │ 10876789 │ 10879118 │
└───────────────────────────┴────┴───────────┴────────────┴──────┴──────────┴──────────┘
```
**What This Tells You:**
- **Total Time Elapsed**: Time spent fetching and parsing each batch of blocks
- **Parsing Time**: Time spent decoding and preparing event data
- **Page Fetch Time**: Time spent retrieving data from the blockchain
- **Num Events**: Number of events in each batch
- **Block Range Size**: Number of blocks in each fetch operation
**How to Interpret:**
- Compare Page Fetch Time to Total Time to see if data retrieval is your bottleneck
- Large standard deviations (std-dev) indicate inconsistent performance
- If Block Range Size varies significantly, your indexer may be adjusting batch sizes dynamically
### Event Processing Performance
```
EventProcessing Summary
┌─────────────────────────────────┬────┬─────────┬─────────┬─────┬──────┬────────┐
│ (index) │ n │ mean │ std-dev │ min │ max │ sum │
├─────────────────────────────────┼────┼─────────┼─────────┼─────┼──────┼────────┤
│ Batch Size │ 38 │ 4163.29 │ 1424.85 │ 89 │ 5000 │ 158205 │
│ Contract Register Duration (ms) │ 38 │ 0.11 │ 0.38 │ 0 │ 2 │ 4 │
│ Load Duration (ms) │ 38 │ 80.79 │ 32.58 │ 5 │ 149 │ 3070 │
│ Handler Duration (ms) │ 38 │ 22.18 │ 9.07 │ 0 │ 47 │ 843 │
│ DB Write Duration (ms) │ 38 │ 135.92 │ 52.09 │ 8 │ 220 │ 5165 │
│ Total Time Elapsed (ms) │ 38 │ 239 │ 83.24 │ 13 │ 370 │ 9082 │
└─────────────────────────────────┴────┴─────────┴─────────┴─────┴──────┴────────┘
```
**What This Tells You:**
- **Batch Size**: Number of events in each processing batch
- **Contract Register Duration**: Time spent preparing contract data
- **Load Duration**: Time spent loading entities from the database
- **Handler Duration**: Time spent executing your event handler logic
- **DB Write Duration**: Time spent writing updated entities to the database
- **Total Time Elapsed**: Overall time for the processing phase
**How to Interpret:**
- Compare Load, Handler, and DB Write durations to identify bottlenecks
- In this example, DB Write (135ms) and Load (80ms) operations dominate processing time
- If Load Duration is high, consider implementing [entity loaders](/docs/HyperIndex/loaders)
- If DB Write Duration is high, check if you're updating too many entities per event
### Per-Handler Performance
```
Handlers Per Event
┌─────────────────────────────┬────────┬────────┬─────────┬────────┬────────┬──────────┐
│ (index) │ n │ mean │ std-dev │ min │ max │ sum │
├─────────────────────────────┼────────┼────────┼─────────┼────────┼────────┼──────────┤
│ ERC20 Transfer Handler (ms) │ 158205 │ 0.0021 │ 0.0364 │ 0.0007 │ 4.6752 │ 329.7264 │
└─────────────────────────────┴────────┴────────┴─────────┴────────┴────────┴──────────┘
```
**What This Tells You:**
- Detailed timing for each specific event handler
- Shows average and total execution time across all events
**How to Interpret:**
- Compare different handlers to identify which ones are most expensive
- Look for handlers with high maximum values (max column), which may indicate inconsistent performance
- Handlers averaging above 1ms per event may benefit from optimization
## Interpreting Results and Taking Action
### Identifying Your Bottleneck
Based on the benchmark data, determine your primary performance bottleneck:
1. **Data Fetching Bottleneck**
- **Symptoms**: Most time spent in "Total Time Fetching"
- **Solutions**:
- Use [HyperSync](/docs/HyperIndex/hypersync) if available for your network
- If using RPC, consider a more performant provider
- Adjust block batch sizes in your configuration
2. **Data Loading Bottleneck**
- **Symptoms**: High "Load Duration" in Event Processing
- **Solutions**:
- Implement [entity loaders](/docs/HyperIndex/loaders) to batch database operations
- Add appropriate [database indices](/docs/HyperIndex/database-performance-optimization) for frequently queried fields
- Optimize your entity relationships to reduce join complexity
3. **Handler Logic Bottleneck**
- **Symptoms**: High "Handler Duration" relative to other metrics
- **Solutions**:
- Simplify complex calculations in your handlers
- Move complex operations to a post-processing step
- Consider caching frequently accessed values
4. **Database Write Bottleneck**
- **Symptoms**: High "DB Write Duration"
- **Solutions**:
- Reduce the number of entities updated per event
- Batch related updates where possible
- Check if you're updating the same entity multiple times unnecessarily
### Benchmarking Best Practices
1. **Benchmark Before and After Optimizations**
- Run benchmarks before making changes to establish a baseline
- Run again after each optimization to measure impact
2. **Focus on the Largest Bottleneck First**
- Prioritize optimizations based on where time is being spent
- Small improvements to the critical path yield the greatest results
3. **Watch for Memory Usage**
- Monitor memory consumption alongside performance metrics
- High memory usage can lead to degraded performance over time
4. **Consider Real-World Conditions**
- Test with realistic data volumes and event patterns
- Include periods of high activity in your benchmark tests
## Advanced Performance Tuning
For cases where standard optimizations aren't sufficient:
1. **Custom Database Indices**
- Create indices tailored to your specific query patterns
- Add composite indices for multi-field filters
2. **Handler Specialization**
- Create specialized handlers for high-volume events
- Simplify logic for the most common paths
3. **Speak to the Envio Team**
- We can help!
By regularly benchmarking your indexer and methodically addressing performance bottlenecks, you can achieve significant improvements in indexing speed and efficiency.
---
# Database Performance Optimization
> Learn how to optimize HyperIndex databases with indices and schema design for faster queries.
# Optimizing Database Performance in HyperIndex
## Introduction
As your indexed data grows, database performance becomes critical to maintaining responsive queries and efficient operations. This guide explains how to optimize your HyperIndex database through strategic indexing and schema design to ensure your applications remain fast even as data volume increases.
## Understanding Database Indices
Database indices are special data structures that improve the speed of data retrieval operations. Think of them like the index at the back of a book — rather than scanning every page (row) to find what you're looking for, indices allow the database to quickly locate the relevant data.
### Why Indices Matter
Without proper indices, your database must perform "full table scans" when searching for data, examining every row to find matches. As your data grows, this becomes increasingly inefficient:
| Data Size | Without Indices | With Proper Indices |
| ------------------ | --------------- | ------------------- |
| 1,000 records | ~10ms | ~1ms |
| 100,000 records | ~500ms | ~2ms |
| 1,000,000+ records | 5+ seconds | ~5ms |
_Note: Actual performance varies based on hardware, query complexity, and database load._
## Creating Custom Indices in Your Schema
HyperIndex provides several ways to define indices in your GraphQL schema, giving you control over database performance.
### Single-Column Indices
The simplest form of indexing is on individual fields using the `@index` directive:
```graphql
type Transaction {
id: ID!
userAddress: String! @index
tokenAddress: String! @index
amount: BigInt!
timestamp: BigInt! @index
}
```
In this example:
- Queries filtering on `userAddress` (e.g., finding all transactions for a user)
- Queries filtering on `tokenAddress` (e.g., finding all transactions for a token)
- Queries filtering on `timestamp` (e.g., finding transactions in a date range)
All become significantly faster because the database can use the indices to quickly locate matching records.
### Composite Indices for Multi-Field Queries
When you frequently query using multiple fields together, composite indices provide better performance:
```graphql
type Transfer @index(fields: ["from", "to", "tokenId"]) {
id: ID!
from: String! @index
to: String! @index
tokenId: BigInt!
value: BigInt!
timestamp: BigInt!
}
```
This creates:
1. Individual indices on `from` and `to` fields
2. A composite index on the combination of `from`, `to`, and `tokenId`
Composite indices are particularly valuable for complex queries that filter on multiple columns simultaneously, such as "find all transfers from address X to address Y for token Z."
### Automatic Indices
HyperIndex automatically creates indices for:
- All `ID` fields
- All fields marked with `@derivedFrom`
There's no need to manually add indices for these fields.
## Deferred Index Creation
Indices make writes slower, and a backfill is nothing but writes. Since v3.5, HyperIndex builds the indices your schema declares in one pass *after* the backfill completes, rather than up front. Some users see a **2.5x backfill speedup** from this alone.
Nothing to configure. Two things change in what you'll observe:
- Querying the database directly mid-backfill is slower, because the declared indices don't exist yet.
- Near the end of the backfill, the indexer pauses to create them and logs that it's doing so. On a large database this takes a while — it's progress, not a hang.
### Indices created on demand
A [`getWhere`](/docs/HyperIndex/event-handlers#retrieving-entities-by-field) call on a field with no index creates one the first time a handler asks for it. That means `getWhere` works on **any** entity field, and you don't need to declare `@index` for the fields your handlers filter on — HyperIndex works out which indices your handlers need.
`@index` is for the queries **you** serve. Declare it on the fields your GraphQL consumers filter and sort by, since HyperIndex can't know those from your handler code:
```graphql
type Transfer {
id: ID!
userAddress: String! @index
timestamp: BigInt!
}
```
## Strategic Indexing: When to Use Each Type
### When to Use Single-Column Indices
Use single-column indices when:
- You frequently filter by a specific field
- You sort results by a specific field
- The field has high "cardinality" (many different values)
**Example use case**: Indexing `userAddress` in a transaction table when users frequently look up their transaction history.
### When to Use Composite Indices
Use composite indices when:
- You frequently query using multiple fields together
- Your queries consistently filter on the same combination of fields
- You need to optimize complex queries with multiple conditions
**Example use case**: Indexing `(tokenAddress, timestamp)` together when users frequently view token transaction history within specific time ranges.
## Performance Tradeoffs
While indices improve query performance, they come with tradeoffs:
### Write Performance Impact
Each index requires additional updates when data is inserted or modified:
- **No indices**: Fastest write performance, but slow reads
- **Few targeted indices**: Slight write slowdown (5-10%), much faster reads
- **Many indices**: Noticeable write slowdown (15%+), fastest possible reads
For most applications, the read performance benefits outweigh the write performance costs, especially since blockchain data is primarily read-intensive.
### Storage Considerations
Indices increase database storage requirements:
- Each index typically requires 2-10 bytes per row
- For large datasets (millions of records), this can add up
- Consider storage requirements when designing indices for very large tables
## Practical Examples
### Optimizing Token Transfer Queries
Consider a token transfer entity:
```graphql
type TokenTransfer {
id: ID!
token: Token! @index
from: String! @index
to: String! @index
amount: BigInt!
blockNumber: BigInt! @index
timestamp: BigInt! @index
}
```
With this schema, the following queries will be optimized:
- Find all transfers for a specific token
- Find all transfers from a specific address
- Find all transfers to a specific address
- Find transfers within a specific block range
- Find transfers within a specific time range
### Optimizing Complex NFT Marketplace Queries
For an NFT marketplace with listings and sales:
```graphql
type NFTListing
@index(fields: ["collection", "status", "price"])
@index(fields: ["seller", "status"]) {
id: ID!
collection: String! @index
tokenId: BigInt!
seller: String! @index
price: BigInt!
status: String! @index # "active", "sold", "cancelled"
createdAt: BigInt! @index
}
```
This schema efficiently supports:
- Finding all active listings for a collection, sorted by price
- Finding all listings by a specific seller with a specific status
- Finding recently created listings across all collections
## Optimizing GraphQL Queries
Beyond schema design, how you write your GraphQL queries affects performance:
### Fetch Only What You Need
Request only the fields you actually need:
```graphql
# Good
query {
tokenTransfers(where: { token: { _eq: "0x123" } }, limit: 10) {
id
amount
}
}
# Bad - fetches unnecessary fields
query {
tokenTransfers(where: { token: { _eq: "0x123" } }, limit: 10) {
id
amount
from
to
timestamp
blockNumber
transactionHash
# other fields you don't need
}
}
```
### Use Pagination for Large Result Sets
Always paginate large result sets:
```graphql
query {
tokenTransfers(
where: { token: { _eq: "0x123" } }
limit: 20
offset: 40 # Skip first 40 results (page 3 with 20 items per page)
) {
id
amount
}
}
```
### Use Timestamps for Efficient Polling
When building applications that poll for updates, use timestamps to fetch only new data:
```graphql
query getUpdatedTransfers($lastFetched: BigInt!) {
tokenTransfers(where: { timestamp: { _gt: $lastFetched } }) {
id
from
to
amount
}
}
```
## Diagnosing Slow Queries
If your GraphQL queries are returning slowly, follow this workflow:
### 1. Check if you have indices on filtered fields
The most common cause of slow queries is missing indices. If you're filtering or sorting by a field that doesn't have `@index`, add it:
```graphql
# Before - slow queries on userAddress
type Transaction {
id: ID!
userAddress: String!
}
# After - fast queries on userAddress
type Transaction {
id: ID!
userAddress: String! @index
}
```
After adding indices, you'll need to redeploy (on Envio Cloud) or restart locally for the schema changes to take effect.
### 2. Reduce result set size
Large unbounded queries are a common cause of slow responses. Always use `limit` and check that you're not requesting more data than needed.
## Summary
Proper database indexing is essential for maintaining performance as your indexed data grows. By strategically placing indices on frequently queried fields and field combinations, you can ensure fast query responses even with large datasets.
**Key takeaways:**
- Use `@index` for frequently filtered or sorted individual fields
- Use composite indices for multi-field query patterns
- Consider performance tradeoffs for write-heavy applications
- Design your schema and queries with performance in mind from the start
- Always use pagination for large result sets
---
# Historical Sync
> Learn how to optimize historical sync with HyperIndex for fast and efficient data retrieval.
# Historical Sync
Historical sync is an important aspect of maintaining and populating your database. Here's an overview of how historical sync works with HyperIndex:
## Optimized HyperSync Backend Nodes
- **Fast Retrieval**: Historical sync is fast when retrieving data from our optimized HyperSync backend nodes. These nodes are specifically designed to handle large amounts of historical data efficiently.
## RPC Endpoint Limitations
- **Slow Performance**: Unfortunately, historical sync for chains using RPC endpoints is not fast. The process is slower due to the limitations of RPC endpoints.
- **No Local Caching**: We do not have any form of local caching at the moment. This feature was disabled in a previous release due to stability concerns.
## Preload Optimization
- **Always on in V3**: Preload Optimization is enabled automatically in HyperIndex V3, with no flag to turn it on or off. It speeds up historical sync by batching the entity reads in your handlers into a small number of database queries, and by running external calls in parallel through the Effect API.
- **Biggest gains**: The benefit is largest when your handlers read and update the same entities frequently, or when they make external calls. For details on how it works and how to write handlers that take advantage of it, see [Preload Optimization](/docs/HyperIndex/preload-optimization).
By understanding these factors, you can better optimize the performance of your historical sync processes.
---
# Performance Optimization
> Learn how to optimize HyperIndex performance for syncing, indexing, and querying data.
## Quick Navigation
- [Database Performance Optimization](./database-performance-optimization)
- [Historical Sync](./historical-sync)
- [Latency at the Head](./latency-at-head)
- [Benchmarking](benchmarking)
There are different types of performance that you may be concerned about when indexing using HyperIndex. There is the speed of historical sync (populating the db from scratch), the latency of indexing new blocks/transactions, and the speed of querying the data once it is indexed. This page will cover some of the ways you can optimize the performance of your HyperIndex setup.
---
# Understanding Chain Head Latency
> Learn how Envio keeps blockchain indexers updated with low latency and reliable multi-chain sync.
# Understanding Chain Head Latency
Maintaining low latency at the chain head is crucial for ensuring timely data updates in your indexed data. This page explains how HyperSync handles this important aspect of blockchain indexing.
## HyperSync Block Retrieval
- **Efficient Processing**: We pull new blocks from HyperSync using a highly efficient process, ensuring your indexer stays up-to-date with minimal delay.
- **Reliable Operation**: This process typically runs smoothly without significant issues.
- **Redundancy Plans**: We're developing a system to sync new blocks from both RPC and HyperSync simultaneously, improving robustness if one source experiences issues.
## Network-Specific Performance
### Optimized Major Networks
- **Priority Networks**: We've dedicated significant resources to maintaining extremely low latency on popular networks including Ethereum, Optimism, and Arbitrum.
- **User Experience**: Users should experience seamless, near real-time data updates on these networks.
### Smaller Chain Networks
- **Standard Performance**: On smaller chains, latency might be slightly higher as these networks have received less optimization.
- **Improvement Process**: Your feedback helps us prioritize which chains to optimize next. Please let us know in Discord if low latency on specific smaller chains is important for your use case.
## Special Configuration Options
### Multi-Chain Indexing
- **Unordered Multi-Chain Mode**: For applications indexing multiple chains, our [unordered multi-chain mode](./multichain-indexing#multichain-event-ordering) allows each chain to continue syncing independently.
- **Resilient Design**: With this configuration, even if one chain experiences latency, your other chains will continue syncing normally.
### Chain Reorganization Support
- **Reorg Handling**: Our reorg support system ensures data consistency even when chains reorganize.
- **Documentation**: Contact our team on Discord if you have concerns about reorg support while we finalize documentation.
## Envio Cloud Performance
Envio Cloud offers reliable performance with ongoing improvements:
- **Continuous Enhancement**: We're actively improving sync and build times on Envio Cloud.
- **Relative Performance**: Currently, indexers may run slightly slower on Envio Cloud compared to high-performance local machines.
- **Enterprise Solutions**: For applications requiring exceptional performance, contact us on Discord to discuss our enterprise hosting plans.
By leveraging these features and providing feedback on your specific needs, you can help us continually improve the HyperIndex head latency performance.
---
# Preload Optimization
> Learn how preload optimization improves event handlers with batched reads and parallel external calls.
# Preload Optimization
> **Important!** Preload optimization makes your handlers run **twice**.
In HyperIndex V3, preload optimization is **always on** — there is no flag to enable or disable it.
This optimization enables HyperIndex to efficiently preload entities used by handlers through batched database queries, while ensuring events are processed synchronously in their original order. When combined with the [Effect API](/docs/HyperIndex/effect-api) for external calls, this feature delivers performance improvements of multiple orders of magnitude compared to other indexing solutions.
## Configure
Nothing to configure. Previously, V2 required the `preload_handlers: true` flag in [`config.yaml`](/docs/HyperIndex/configuration-file). In V3 the flag has been removed and the optimization is always active. If your project still has `preload_handlers:` in `config.yaml`, delete it — V3 will reject the field.
## Why Preload?
To ensure reliable data, HyperIndex guarantees that all events will be processed in the same order as they occurred on-chain.
This guarantee is crucial as it allows you to build indexers that depend on the sequential order of events.
However, this leads to a challenge: Handlers must run one at a time, sequentially for each event. Any asynchronous operations will block the entire process.
To solve this, we introduced Preload Optimization.
It combines in-memory storage, batching, deduplication, and the [Effect API](/docs/HyperIndex/effect-api) to parallelize asynchronous operations across batches of events.
## How It Works?
With Preload Optimization handlers run twice per event:
1. **First Run (Preload Phase)**: All event handlers run concurrently for the whole batch of events. During the phase all DB write operations are skipped and only DB read operations and external calls are performed.
2. **Second Run (Processing Phase)**: Each event handler runs sequentially in the on-chain order. During the phase it'll get the data from the in-memory store, reflecting changes made by previously processed events.
This double execution pattern ensures that entities created by earlier events in the batch are available to later events.
### The Database I/O Problem
Consider this common pattern of getting entities in event handlers:
```typescript
import { indexer } from "envio";
indexer.onEvent(
{ contract: "ERC20", event: "Transfer" },
async ({ event, context }) => {
const sender = await context.Account.get(event.params.from);
const receiver = await context.Account.get(event.params.to);
// Process the transfer...
},
);
```
**Without Preload Optimization:** If you're processing 5,000 transfer events, each with unique `from` and `to` addresses, this results in **10,000 total database roundtrips**—one for each sender and receiver lookup (2 per event × 5,000 events). This creates a significant bottleneck that slows down your entire indexing process.
**With Preload Optimization:** During the Preload Phase, all 5,000 events are processed in parallel. HyperIndex batches database reads that occur simultaneously into single database queries - one query for sender lookups and one for receiver lookups. The loaded accounts are cached in memory. After the Preload Phase completes, the second processing phase begins. This phase runs handlers sequentially in on-chain order, but instead of making database calls, it retrieves the data from the in-memory cache.
For our example of 5,000 transfer events, this optimization reduces database roundtrips from 10,000 calls to just 2!
#### Optimizing for Concurrency
You can further optimize performance by requesting multiple entities concurrently:
```typescript
import { indexer } from "envio";
indexer.onEvent(
{ contract: "ERC20", event: "Transfer" },
async ({ event, context }) => {
// Request sender and receiver concurrently for maximum efficiency
const [sender, receiver] = await Promise.all([
context.Account.get(event.params.from),
context.Account.get(event.params.to),
]);
// Process the transfer...
},
);
```
This approach can reduce the database roundtrips to just 1 for the entire batch of events!
### The External Calls Problem
Let's say you want to populate your indexer with offchain data:
```typescript
import { indexer } from "envio";
indexer.onEvent(
{ contract: "ERC20", event: "Transfer" },
async ({ event, context }) => {
// Without Preload: Blocking external calls
const metadata = await fetch(
`https://api.example.com/metadata/${event.params.from}`
);
// Process the transfer...
},
);
```
**Without Preload Optimization:** If you're processing 5,000 transfer events, each with an external call, this results in **5,000 sequential external calls**—each waiting for the previous one to complete. This can turn a fast indexing process into a slow, sequential crawl.
**With Preload Optimization:** Since handlers run **twice** for each event, making direct external calls can be problematic. The [Effect API](/docs/HyperIndex/effect-api) provides a solution. During the Preload Phase, it batches all external calls and runs them in parallel. Then during the Processing Phase, it runs the handlers sequentially, retrieving the already requested data from the in-memory store.
```typescript
import { S, createEffect, indexer } from "envio";
const fetchMetadata = createEffect(
{
name: "fetchMetadata",
input: {
from: S.string,
},
output: {
decimals: S.number,
symbol: S.string,
},
rateLimit: {
calls: 5,
per: "second",
},
},
async ({ input }) => {
const metadata = await fetch(
`https://api.example.com/metadata/${input.from}`
);
return metadata;
}
);
indexer.onEvent(
{ contract: "ERC20", event: "Transfer" },
async ({ event, context }) => {
// With Preload: Performs the call in parallel
const metadata = await context.effect(fetchMetadata, {
from: event.params.from,
});
// Process the transfer...
},
);
```
Assuming an average call takes 200ms, this optimization reduces the total processing time for 5,000 events from ~16 minutes to ~200 milliseconds - making it 5,000 times faster!
Learn more about the [Effect API](/docs/HyperIndex/effect-api) in our dedicated guide.
### Preload Phase Behavior
The Preload Phase is a special phase that runs before the actual event processing. It's designed to preload data that will be used during event processing.
Key characteristics of the Preload Phase:
- It runs in parallel for all events in the batch
- Exceptions won't crash the indexer but will silently abort the Preload Phase for that specific event
- All storage updates are ignored
- All `context.log` calls are ignored
During the second run (Processing Phase), all operations become fully enabled:
- Exceptions will crash the indexer if not handled
- Entity setting operations will persist to the database
- Logging will output to the console
This two-phase design allows the Preload Phase to optimistically attempt loading data that may not exist yet, while ensuring data consistency during the Processing Phase when all operations are executed normally.
If you're using an earlier version of `envio`, we strongly recommend upgrading to the latest version using `pnpm install envio@latest` to benefit from this improved Preload Phase behavior.
## Double-Run Footgun
As mentioned above, the Preload Phase gives a lot of benefits for the event processing, but also it means that you must be aware of its table run nature:
- Never call `fetch` or other external calls directly in the handler.
- Use the [Effect API](/docs/HyperIndex/effect-api) instead.
- Or use `context.isPreload` to guarantee that the code will run once.
Due to the optimistic nature of the Preload Phase, the Effect API may occasionally execute with stale data, leading to redundant external calls. If you need to ensure that external calls are made with the most up-to-date data, you can use the `context.isPreload` check to restrict execution to only the processing phase.
> Note: This will disable the Preload Optimization for the external calls.
```typescript
import { indexer } from "envio";
indexer.onEvent(
{ contract: "ERC20", event: "Transfer" },
async ({ event, context }) => {
const sender = await context.Account.get(event.params.from);
if (context.isPreload) {
return;
}
const metadata = await fetch(
`https://api.example.com/metadata/${sender.metadataId}`
);
},
);
```
## Best Practices
- Use `Promise.all` to load multiple entities concurrently for better performance
- Place database reads and external calls at the beginning of your handler to maximize the benefits of Preload Optimization
- Consider using `context.isPreload` to exit early from the Preload Phase after loading required data
## Migrating from Loaders
The Preload Optimization for handlers was born from a concept we had before called [Loaders](/docs/HyperIndex/loaders). The `handlerWithLoader` API has been removed in V3 — move the loader code into the handler and rely on the always-on Preload Phase.
```typescript
// V2 — removed in V3
ERC20.Transfer.handlerWithLoader({
loader: async ({ event, context }) => {
// Load sender and receiver accounts efficiently
const sender = await context.Account.get(event.params.from);
const receiver = await context.Account.get(event.params.to);
// Return the loaded data to the handler
return {
sender,
receiver,
};
},
handler: async ({ event, context, loaderReturn }) => {
const { sender, receiver } = loaderReturn;
// Process the transfer with the pre-loaded data
// No database lookups needed here!
},
});
// V3
import { indexer } from "envio";
indexer.onEvent(
{ contract: "ERC20", event: "Transfer" },
async ({ event, context }) => {
// Load sender and receiver accounts efficiently
const sender = await context.Account.get(event.params.from);
const receiver = await context.Account.get(event.params.to);
// To imitate the behavior of the loader,
// we can use `context.isPreload` to make next code run only once.
// Note: This is not required, but might be useful for CPU-intensive operations.
if (context.isPreload) {
return;
}
// Process the transfer with the pre-loaded data
},
);
```
---
# Query Conversion Guide
> Learn how to convert queries from TheGraph's custom GraphQL syntax to Envio's standard GraphQL syntax.
# Query Conversion
Envio uses standard GraphQL query language, while TheGraph uses a custom GraphQL syntax. While the queries are very similar, there are some important differences to be aware of when migrating.
This guide covers all the differences between TheGraph's query syntax and Envio's query syntax, with examples for each conversion rule.
## Converter Tool
We've built a [query converter tool](https://github.com/enviodev/subgraph-to-hyperindex-query-converter) that automatically converts TheGraph queries to Envio's GraphQL syntax. You can:
- **Convert and execute**: Provide your Envio GraphQL endpoint and a query written in TheGraph syntax. The tool will convert it, execute it against your endpoint, and return the results
- **Convert only**: Use the tool to convert queries and view the converted output without executing them. To convert queries without executing them, add `/debug` to the converter endpoint and send your query. The response will contain only the converted query without actually running it.
**Repository**: [subgraph-to-hyperindex-query-converter](https://github.com/enviodev/subgraph-to-hyperindex-query-converter)
:::info Availability
The query converter tool is only available to users on paid tiers.
:::
:::info Setup Assistance
If you'd like to use the query converter tool for your indexer, please reach out to our team and we can add an instance of the converter tool to be deployed with your indexer.
:::
:::tip Best Practice
**We strongly recommend converting your queries to use Envio's standard GraphQL syntax** rather than relying on the query converter tool. This ensures better performance, maintainability, and avoids potential conversion limitations. The converter tool is primarily intended for backwards compatibility in cases where you might have external-facing APIs which depend on TheGraph's query syntax.
:::
:::warning Beta Status
This converter tool is still very much in beta. We're actively working on it and discovering new query conversions that need to be handled.
**If you encounter any conversion failures or incorrect conversions, please [file a GitHub issue](https://github.com/enviodev/subgraph-to-hyperindex-query-converter/issues) in the repository so we can address it.**
:::
### Converter Tool Limitations
The query converter tool has several limitations to be aware of:
- **Rate limits**: The rate limits of your associated tier will still apply when using the converter tool.
- **Beta status and incomplete coverage**: Since the tool is in beta there is a possibility we have missed some conversion patterns. Some queries may still fail to convert correctly or may not be handled at all. If you encounter any conversion failures or incorrect conversions, please [file a GitHub issue](https://github.com/enviodev/subgraph-to-hyperindex-query-converter/issues) in the repository so we can address it.
- **WebSocket subscriptions**: WebSocket subscriptions are not supported. The converter only works with standard HTTP GraphQL queries.
- **Directives not supported**: GraphQL directives (such as `@skip`, `@include`, etc.) are not supported because the converter uses simple string parsing rather than a full GraphQL parser. Directives are ignored or may break parsing.
- **Variables in orderBy/orderDirection**: Variables used in `orderBy` and `orderDirection` parameters are ignored because HyperIndex requires literal field names in `order_by` clauses, and variable values are unknown at conversion time. Queries using variables for ordering will return unordered results.
- **Array filter operators**: The `_containsAny` and `_containsAll` filters are TheGraph-specific array operators that don't have direct Hasura equivalents. The converter explicitly rejects these and returns an `UnsupportedFilter` error. Use `_in` for array matching instead.
- **Meta queries**: Meta queries only support `_meta { block { number } }` because HyperIndex exposes block information differently. Other `_meta` fields (hash, timestamp, deployment, hasIndexingErrors) are not available in HyperIndex's schema and will return a `ComplexMetaQuery` error.
- **Introspection queries**: Introspection queries only work if they use the operation name `"IntrospectionQuery"`. Other introspection queries (like querying `__schema` directly) will fail because they go through normal conversion which doesn't understand introspection syntax.
For production applications, we recommend migrating your queries to Envio's standard GraphQL syntax to avoid these limitations.
---
### 1. Entity Name Conversion
**Rule**: TheGraph uses pluralized entity names (e.g., `pools`, `factories`, `tokens`), while Envio uses the entity name as-is from the schema (singular, PascalCase). When converting, plural entity names are automatically singularized and capitalized to match Envio's schema.
**Example**:
```graphql
# TheGraph
query {
pools { id }
factories { id }
tokens { id }
}
# Envio
query {
Pool { id }
Factory { id }
Token { id }
}
```
Single entity queries use `EntityName_by_pk` (by primary key) in Envio. Alternatively, you could use a `where` clause with the primary key field:
```graphql
# TheGraph
query {
pool(id: "0x123") { value }
}
# Envio
query {
Pool_by_pk(id: "0x123") { value }
}
```
Or using a `where` clause:
```graphql
# Envio
query {
Pool(where: {id: {_eq: "0x123"}}) { value }
}
```
### 2. Pagination Parameters
#### First → Limit
**Rule**: The `first` parameter is converted to `limit`.
**Example**:
```graphql
# TheGraph
query {
pools(first: 10) { id }
}
# Envio
query {
Pool(limit: 10) { id }
}
```
#### Skip → Offset
**Rule**: The `skip` parameter is converted to `offset`.
**Example**:
```graphql
# TheGraph
query {
pools(skip: 20) { id }
}
# Envio
query {
Pool(offset: 20) { id }
}
```
### 3. Ordering Parameters
#### OrderBy and OrderDirection → Order_by
**Rule**: The `orderBy` and `orderDirection` parameters are combined into a single `order_by: {field: direction}` clause.
**Example**:
```graphql
# TheGraph
query {
pools(orderBy: name, orderDirection: desc) { id name }
# Use orderDirection: asc for ascending order
}
# Envio
query {
Pool(order_by: {name: desc}) { id name }
# Use asc for ascending order, e.g., order_by: {name: asc}
}
```
### 4. Filter Operators
#### Equality Filter
**Rule**: Simple equality filters are converted to `where: {field: {_eq: value}}` format.
**Example**:
```graphql
# TheGraph
query {
pools(name: "test") { id name }
}
# Envio
query {
Pool(where: {name: {_eq: "test"}}) { id name }
}
```
#### Comparison Filters
**Rule**: Comparison filters (`_not`, `_gt`, `_gte`, `_lt`, `_lte`, `_in`, `_not_in`) are converted to their Hasura equivalents.
**Examples**:
```graphql
# TheGraph
query {
pools(id_not: "0x123") { id }
pools(amount_gt: 100) { id amount }
pools(amount_gte: 100) { id amount }
pools(timestamp_lt: 1650000000) { id timestamp }
pools(timestamp_lte: 1650000000) { id timestamp }
pools(id_in: ["1", "2", "3"]) { id name }
pools(id_not_in: ["1", "2", "3"]) { id name }
}
# Envio
query {
Pool(where: {id: {_neq: "0x123"}}) { id }
Pool(where: {amount: {_gt: 100}}) { id amount }
Pool(where: {amount: {_gte: 100}}) { id amount }
Pool(where: {timestamp: {_lt: 1650000000}}) { id timestamp }
Pool(where: {timestamp: {_lte: 1650000000}}) { id timestamp }
Pool(where: {id: {_in: ["1", "2", "3"]}}) { id name }
Pool(where: {id: {_nin: ["1", "2", "3"]}}) { id name }
}
```
#### String Filters
**Rule**: String filters (`_contains`, `_starts_with`, `_ends_with`, and their `_not` and `_nocase` variants) are converted to `_ilike` with appropriate wildcards. The `%` symbol represents any text at that position in the pattern.
**Examples**:
```graphql
# TheGraph
query {
pools(name_contains: "test") { id name }
pools(name_not_contains: "test") { id name }
pools(symbol_starts_with: "ABC") { id symbol }
pools(symbol_ends_with: "XYZ") { id symbol }
pools(name_not_starts_with: "A") { id name }
pools(name_not_ends_with: "x") { id name }
pools(name_contains_nocase: "test") { id name }
pools(name_starts_with_nocase: "test") { id name }
pools(name_ends_with_nocase: "test") { id name }
}
# Envio
query {
Pool(where: {name: {_ilike: "%test%"}}) { id name }
Pool(where: {_not: {name: {_ilike: "%test%"}}}) { id name }
Pool(where: {symbol: {_ilike: "ABC%"}}) { id symbol }
Pool(where: {symbol: {_ilike: "%XYZ"}}) { id symbol }
Pool(where: {_not: {name: {_ilike: "A%"}}}) { id name }
Pool(where: {_not: {name: {_ilike: "%x"}}}) { id name }
Pool(where: {name: {_ilike: "%test%"}}) { id name }
Pool(where: {name: {_ilike: "test%"}}) { id name }
Pool(where: {name: {_ilike: "%test"}}) { id name }
}
```
### 5. Variable Type Conversions
#### ID → String
**Rule**: Variable types `ID` and `ID!` are converted to `String` and `String!` respectively.
**Example**:
```graphql
# TheGraph
query getPoolValue($id: ID!) {
pool(id: $id) { value }
}
# Envio
query getPoolValue($id: String!) {
Pool_by_pk(id: $id) { value }
}
```
#### Bytes → String
**Rule**: Variable types `Bytes` and `Bytes!` are converted to `String` and `String!` respectively.
**Example**:
```graphql
# TheGraph
query getTokens($id: Bytes) {
tokens(where: { id: $id }) { id timestamp }
}
# Envio
query getTokens($id: String) {
Token(where: {id: {_eq: $id}}) { id timestamp }
}
```
#### BigInt → numeric
**Rule**: Variable types `BigInt` and `BigInt!` are converted to `numeric` and `numeric!` respectively.
**Example**:
```graphql
# TheGraph
query GetTokens($amount: BigInt) {
tokens(where: { amount: $amount }) { id amount }
}
# Envio
query GetTokens($amount: numeric) {
Token(where: {amount: {_eq: $amount}}) { id amount }
}
```
#### BigDecimal → numeric
**Rule**: Variable types `BigDecimal` and `BigDecimal!` are converted to `numeric` and `numeric!` respectively.
**Example**:
```graphql
# TheGraph
query GetValue($value: BigDecimal!) {
pools(where: { value: $value }) { id }
}
# Envio
query GetValue($value: numeric!) {
Pool(where: {value: {_eq: $value}}) { id }
}
```
---
## Summary Table
| Category | TheGraph | Envio | Example |
|----------|-----------|-------|---------|
| **Entity Names** | Plural camelCase | Singular PascalCase (as-is from schema) | `pools` → `Pool` |
| **Pagination** | `first`, `skip` | `limit`, `offset` | `first: 10, skip: 20` → `limit: 10, offset: 20` |
| **Ordering** | `orderBy`, `orderDirection` | `order_by: {field: direction}` | `orderBy: name, orderDirection: desc` → `order_by: {name: desc}` |
| **Equality Filter** | `field: value` | `field: {_eq: value}` | `name: "test"` → `name: {_eq: "test"}` |
| **Comparison Filters** | `field_gt`, `field_gte`, etc. | `field: {_gt: value}`, etc. | `amount_gt: 100` → `amount: {_gt: 100}` |
| **String Filters** | `_contains`, `_starts_with`, etc. | `_ilike` with `%` wildcards | `name_contains: "test"` → `name: {_ilike: "%test%"}` |
| **Variable Types** | `ID`, `Bytes`, `BigInt`, `BigDecimal` | `String`, `numeric` | `$id: ID!` → `$id: String!` |
---
## Getting Help
If you encounter any issues with query conversion or have questions:
- **Converter Issues**: File a [GitHub issue](https://github.com/enviodev/subgraph-to-hyperindex-query-converter/issues) for the converter tool
- **General Questions**: Join our [Discord community](https://discord.gg/envio) for support
---
# Chain Reorganization Support
> Learn how to handle chain reorgs and keep your indexed data consistent.
# Understanding and Handling Chain Reorganizations
## What Are Chain Reorganizations?
Chain reorganizations (reorgs) occur when the blockchain temporarily forks and then resolves to a single chain, causing some previously confirmed blocks to be replaced by different blocks. This is a normal part of blockchain consensus mechanisms, especially in proof-of-work chains.
When a reorg happens:
- Transactions that were previously considered confirmed may be dropped
- New transactions may be added to the blockchain
- The order of transactions might change
For indexers, this presents a challenge: data that was previously indexed may no longer be valid, requiring a rollback and reprocessing of the affected blocks.
## Automatic Reorg Handling in HyperIndex
HyperIndex includes built-in support for handling chain reorganizations, ensuring your indexed data remains consistent with the blockchain's canonical state. This feature is **enabled by default** to protect your data integrity.
## Configuration Options
### Enabling or Disabling Reorg Support
You can control reorg handling through the `rollback_on_reorg` flag in your `config.yaml` file:
```yaml
# Enable reorg handling (default)
rollback_on_reorg: true
chains:
# chain configurations...
# OR
# Disable reorg handling (not recommended for production)
rollback_on_reorg: false
chains:
# chain configurations...
```
### Configuring Confirmation Thresholds
You can customize the number of blocks required before considering a block "confirmed" and no longer subject to reorgs:
```yaml
rollback_on_reorg: true
chains:
- id: 137 # Polygon
max_reorg_depth: 150
- id: 1 # Ethereum
# Using default threshold
```
The `max_reorg_depth` field (renamed from V2's `confirmed_block_threshold`) defines how many blocks below the chain head are considered safe from reorganizations. Any reorg deeper than this threshold won't trigger a rollback in your indexer.
## Default Confirmation Thresholds
Currently, all chains default to a threshold of **200 blocks**. In future releases, these thresholds will be tailored per chain based on their specific characteristics and historical reorg depths.
| Chain Type | Default Threshold | Notes |
| ---------- | ----------------- | ----------------------------------------------- |
| All Chains | 200 blocks | Will be customized per chain in future releases |
## Technical Details and Limitations
### Guaranteed Detection
Reorg detection is guaranteed when using [HyperSync](/docs/HyperIndex/Advanced/hypersync.md) as your data source. HyperSync's architecture ensures that any reorganization in the blockchain will be properly detected and handled.
### RPC Limitations
When using a [custom RPC endpoint](/docs/HyperIndex/Advanced/rpc-sync.md) as your data source, there are some edge cases where reorgs might go undetected, depending on the RPC provider's implementation and your indexing pattern.
### Scope of Rollbacks
During a reorg-triggered rollback:
✅ **What is rolled back:**
- All entities defined in your schema
- All data that your handlers read or write to the database
❌ **What is not rolled back:**
- Side effects in your handler code (API calls, external services)
- Custom caching mechanisms outside of HyperIndex
- Logs or external files written by your handlers
## Best Practices
1. **Keep reorg support enabled** for production indexers
2. **Use HyperSync** when possible for guaranteed reorg detection
3. **Avoid external side effects** in your handlers that cannot be rolled back
4. **Consider higher thresholds** for high-value applications or chains with historically deep reorgs
## Example Configuration
Here's a complete example showing reorg handling configuration for multiple chains:
```yaml
rollback_on_reorg: true
chains:
- id: 1 # Ethereum Mainnet
max_reorg_depth: 250 # Higher threshold for Ethereum
# other chain config...
- id: 137 # Polygon
max_reorg_depth: 150 # Lower threshold for Polygon
# other chain config...
- id: 42161 # Arbitrum One
# Using default threshold (200)
# other chain config...
```
By properly configuring reorg support, you ensure that your indexed data remains consistent with the blockchain, even when the chain reorganizes.
## Using HyperSync Directly? Handle Reorgs with the Rollback Guard
If you use [HyperSync](/docs/HyperSync/hypersync-query#rollback-guard) directly, without HyperIndex, you have to handle reorg detection and rollback yourself using the **rollback guard** returned on each query response.
HyperSync validates block parent hashes internally and re-syncs when it detects a fork, so it always serves canonical chain data. Data you have already fetched can still go stale after a reorg, though. To detect that, compare the `first_parent_hash` of the current response against the `hash` you stored from the previous response. If they differ, a reorg has occurred and you need to re-fetch the affected range.
For full details, including a pseudocode example, see the [HyperSync Rollback Guard documentation](/docs/HyperSync/hypersync-query#rollback-guard).
:::tip
HyperIndex automates all of this: it fetches recent block hashes to pinpoint exactly where a reorg occurred and automatically rolls back database state. Unless you need the full flexibility of raw HyperSync, HyperIndex saves significant implementation effort.
:::
---
---
# RPC as Data Source
> Learn how to index any EVM blockchain using RPC and optimize performance with HyperIndex
# Using RPC as Your Indexing Data Source
HyperIndex supports indexing any EVM blockchain using RPC (Remote Procedure Call) as the data source. This page explains when and how to use RPC for your indexing needs.
## When to Use RPC
While [HyperSync](/docs/HyperIndex/hypersync) is the recommended and default data source for optimal performance, there are scenarios where you might need to use RPC instead:
1. **Unsupported Chains**: When indexing a blockchain that isn't yet supported by HyperSync
2. **Custom Requirements**: When you need specific RPC functionality not available in HyperSync
3. **Private Chains**: When working with private or development EVM chains
> **Note**: For chains that HyperSync supports, we strongly recommend using HyperSync rather than RPC. HyperSync provides significantly faster indexing performance (up to 100x) and doesn't require managing RPC endpoints or worrying about rate limits.
## Configuring RPC in Your Indexer
### Basic Configuration
In V3 the V2 `rpc_config` field has been replaced with `rpc`, which accepts a single URL, a single `Rpc` object, or a list of `Rpc` objects. Each entry can declare what it's `for`: `sync` (historical), `realtime` (head, including WebSocket URLs), or `fallback`.
To use RPC as the primary historical data source, add an `rpc` entry with `for: sync` to your chain configuration in `config.yaml`:
```yaml
chains:
- id: 1 # Ethereum Mainnet
rpc:
- url: https://eth-mainnet.your-rpc-provider.com # Your RPC endpoint
for: sync
start_block: 15000000
contracts:
- name: MyContract
address: "0x1234..."
# Additional contract configuration...
```
The presence of an RPC marked `for: sync` tells HyperIndex to use RPC instead of HyperSync for historical sync on this chain. You can also add a `for: realtime` WebSocket endpoint to follow the head:
```yaml
chains:
- id: 1
rpc:
- url: https://eth-mainnet.your-rpc-provider.com
for: sync
- url: wss://eth-mainnet.your-rpc-provider.com
for: realtime
```
### Advanced RPC Configuration
For more control over how your indexer interacts with the RPC endpoint, you can configure additional parameters:
```yaml
chains:
- id: 1
rpc:
- url: https://eth-mainnet.your-rpc-provider.com
for: sync
initial_block_interval: 10000 # Initial number of blocks to fetch in each request
backoff_multiplicative: 0.8 # Factor to scale back block request size after errors
acceleration_additive: 2000 # How many more blocks to request when successful
interval_ceiling: 10000 # Maximum blocks to request in a single call
backoff_millis: 5000 # Milliseconds to wait after an error
query_timeout_millis: 20000 # Milliseconds before timing out a request
start_block: 15000000
# Additional chain configuration...
```
### Configuration Parameters Explained
| Parameter | Description | Recommended Value |
| ------------------------ | ------------------------------------------ | ------------------- |
| `url` | Your RPC endpoint URL | Depends on provider |
| `initial_block_interval` | Starting block batch size | 1,000 - 10,000 |
| `backoff_multiplicative` | How much to reduce batch size after errors | 0.5 - 0.9 |
| `acceleration_additive` | How much to increase batch size on success | 500 - 2,000 |
| `interval_ceiling` | Maximum blocks per request | 5,000 - 10,000 |
| `backoff_millis` | Wait time after errors (ms) | 1,000 - 10,000 |
| `query_timeout_millis` | Request timeout (ms) | 10,000 - 30,000 |
The optimal values depend on your RPC provider's performance and limits, as well as the complexity of your contracts and the data being indexed.
### Custom HTTP Headers
Some providers gate their endpoints behind an `Authorization` header or a custom API-key header rather than putting the key in the URL. Since v3.3, `headers` sets arbitrary HTTP headers on every request to an RPC endpoint:
```yaml
chains:
- id: 1
rpc:
- url: https://eth-mainnet.your-rpc-provider.com
for: sync
headers:
Authorization: "Bearer ${RPC_API_KEY}"
X-Custom-Header: "my-value"
start_block: 15000000
```
Header values support `${ENV_VAR}` interpolation, so keep credentials in [environment variables](/docs/HyperIndex/environment-variables) rather than committing them to `config.yaml`.
### Event Filtering on RPC
RPC sources support the same [`where` filtering](/docs/HyperIndex/wildcard-indexing#topic-filtering) as HyperSync, and since v3.3 that includes:
- **`OR` conditions** — passing an array to `params` matches an event if any entry in the array matches. See [Multiple Filters](/docs/HyperIndex/wildcard-indexing#multiple-filters).
- **Multiple wildcard events** — an RPC-backed indexer can register more than one [wildcard](/docs/HyperIndex/wildcard-indexing) event. Earlier versions allowed only one.
## RPC Best Practices
### Selecting an RPC Provider
When choosing an RPC provider, consider:
- **Rate limits**: Most providers have limits on requests per second/minute
- **Node performance**: Some providers offer faster nodes for premium tiers
- **Archive nodes**: Required if you need historical state (e.g., balances at past blocks)
- **Geographic location**: Choose nodes closest to your indexer deployment
### Performance Optimization
To get the best performance when using RPC:
1. **Start from a recent block** if possible, rather than indexing from genesis
2. **Tune batch parameters** based on your provider's capabilities
3. **Use a paid service** for better reliability and higher rate limits
4. **Consider multiple fallback RPCs** for redundancy
### Improving resilience with RPC fallback
HyperIndex allows you to configure additional RPC providers as fallback data sources. This redundancy is **recommended** for production deployments to ensure continuous operation of your indexer. If HyperSync experiences any interruption, your indexer will automatically switch to the fallback RPC provider.
Adding an RPC fallback provides these benefits:
- **High availability**: Your indexer continues to function even during temporary HyperSync outages
- **Automatic failover**: The system detects issues and switches to fallback RPC without manual intervention
- **Operational control**: You can specify which RPC providers to use as fallbacks based on your requirements
Configure a fallback RPC by adding the `rpc` field to your chain configuration:
```diff
name: Greeter
description: Greeter indexer
chains:
- id: 137 # Polygon
+ # Short and simple
+ rpc: https://polygon.your-rpc-provider.com?API_KEY={ENVIO_POLYGON_API_KEY}
+ # Or provide multiple RPC endpoints with more flexibility
+ rpc:
+ - url: https://polygon.your-rpc-provider.com?API_KEY={ENVIO_POLYGON_API_KEY}
+ for: fallback
+ - url: https://polygon.your-free-rpc-provider.com
+ for: fallback
+ initial_block_interval: 1000
start_block: 0 # With HyperSync, you can use 0 regardless of contract deployment time
contracts:
- name: PolygonGreeter
abi_file_path: abis/greeter-abi.json
address: 0x9D02A17dE4E68545d3a58D3a20BbBE0399E05c9c
events:
- event: NewGreeting
- event: ClearGreeting
```
:::info
The fallback RPC is activated only when a primary data source doesn't receive a new block for more than 20 seconds.
:::
## Enhanced RPC with eRPC
For more robust RPC usage, you can implement [eRPC](https://github.com/erpc/erpc) - a fault-tolerant EVM RPC proxy with advanced features like caching and failover.
### What eRPC Provides
- **Permanent caching**: Stores historical responses to reduce redundant requests
- **Auto failover**: Automatically switches between multiple RPC providers
- **Re-org awareness**: Properly handles blockchain reorganizations
- **Auto-batching**: Optimizes requests to minimize network overhead
- **Load balancing**: Distributes requests across multiple providers
### Setting Up eRPC
1. **Create your eRPC configuration file** (`erpc.yaml`):
```yaml
logLevel: debug
projects:
- id: main
upstreams:
# Add HyperRPC as primary source
- endpoint: evm+envio://rpc.hypersync.xyz
# Add fallback RPC endpoints
- endpoint: https://eth-mainnet-provider1.com
- endpoint: https://eth-mainnet-provider2.com
- endpoint: https://eth-mainnet-provider3.com
```
2. **Run eRPC using Docker**:
```bash
docker run -v $(pwd)/erpc.yaml:/root/erpc.yaml -p 4000:4000 -p 4001:4001 ghcr.io/erpc/erpc:latest
```
Or add it to your existing Docker Compose setup:
```yaml
services:
# Your existing services...
erpc:
image: ghcr.io/erpc/erpc:latest
platform: linux/amd64
volumes:
- "${PWD}/erpc.yaml:/root/erpc.yaml"
ports:
- 4000:4000
- 4001:4001
restart: always
```
3. **Configure HyperIndex to use eRPC** in your `config.yaml`:
```yaml
chains:
- id: 1
rpc:
- url: http://erpc:4000/main/evm/1 # eRPC endpoint for Ethereum Mainnet
for: sync
start_block: 15000000
# Additional chain configuration...
```
For more detailed configuration options, refer to the [eRPC documentation](https://docs.erpc.cloud/config/example).
## Comparing HyperSync and RPC
| Feature | HyperSync | RPC |
| --------------- | ------------------------------------------------------------------ | ---------------------------- |
| Speed | 10-100x faster | Baseline |
| Configuration | Minimal | Requires tuning |
| Rate Limits | None | Depends on provider |
| Cost | Included with Envio Cloud | Pay per request/subscription |
| Chain Support | [Supported chains](/docs/HyperSync/hypersync-supported-networks) | Any EVM chain |
| Maintenance | Managed by Envio | Self-managed |
## Summary
While RPC provides the flexibility to index any EVM blockchain, it comes with performance limitations and configuration complexity. For supported networks, we recommend using HyperSync as your data source for optimal performance.
If you must use RPC:
- Choose a reliable provider
- Configure your indexer for optimal performance
- Consider implementing eRPC for enhanced reliability and performance
- Start from recent blocks when possible to reduce indexing time
For any questions about using RPC with HyperIndex, please contact the Envio team.
---
# Terminology & Key Concepts
> Explore key terms for fast reference and blockchain indexer development.
# HyperIndex Terminology & Key Concepts
This comprehensive glossary explains the key terms and concepts used throughout the Envio documentation and ecosystem. Terms are organized by category for easier reference.
## Table of Contents
- [Blockchain Fundamentals](#blockchain-fundamentals)
- [Smart Contract Concepts](#smart-contract-concepts)
- [Indexing & Data](#indexing--data)
- [Development Tools](#development-tools)
- [Programming Languages](#programming-languages)
- [Envio Platform](#envio-platform)
- [Mathematical Concepts](#mathematical-concepts)
## Blockchain Fundamentals
### Address
A unique identifier representing an account or entity within a blockchain network. Addresses are typically represented as hexadecimal strings (e.g., `0x1234...abcd`) and used to send, receive, or interact with blockchain resources.
### Block
A collection of data containing a set of transactions that are bundled together and added to the blockchain. Blocks are linked together chronologically to form the blockchain.
### EVM
**Ethereum Virtual Machine (EVM)** is a runtime environment that executes smart contracts on the Ethereum blockchain. It provides a sandboxed and deterministic execution environment for smart contract code.
### EVM Compatible
The ability for a blockchain to run the EVM and execute Ethereum smart contracts. In the context of Envio, it's the ability to deploy a unified API to retrieve data from multiple EVM-compatible blockchains (e.g., Ethereum, BSC, Arbitrum, Polygon, Avalanche, Optimism, Fantom, Cronos, etc.).
### Node
A device or computer that participates in a blockchain network, maintaining a copy of the blockchain and validating transactions.
### Transaction
An action or set of actions recorded on the blockchain, typically involving the transfer of assets, execution of smart contracts, or other network interactions. Once confirmed, transactions become a permanent part of the blockchain.
## Smart Contract Concepts
### Event
A specific occurrence or action within a blockchain system that is specified in smart contracts and used to emit data from the blockchain. Smart contracts can emit events to essentially communicate that something has happened on the blockchain.
Web applications or any kind of application (e.g., mobile app, backend job, etc.) can listen to events and take actions when they occur. Events are typically data that are not stored on-chain as it would be considerably more expensive to store.
**Example:**
Declaring an event:
```solidity
event Deposit(address indexed _from, bytes32 indexed _id, uint _value);
```
Emitting an event:
```solidity
emit Deposit(msg.sender, _id, msg.value);
```
### Event Handler
A function that listens for a specific event from a smart contract and either updates or inserts new data into your Envio API. Event handlers define the business logic for processing blockchain events.
### Smart Contract
A self-executing program with the terms of an agreement directly written into code that runs on the blockchain. Smart contracts are not controlled by a user but are deployed to the network and run as programmed. User accounts can interact with smart contracts by submitting transactions that execute defined functions.
### Tokens
Digital representations of assets or utilities within a blockchain system that follow a specific standard. Common token standards include:
- **ERC-20**: Standard for fungible tokens (identical and interchangeable)
- **ERC-721**: Standard for non-fungible tokens (unique and non-interchangeable)
- **ERC-1155**: Multi-token standard supporting both fungible and non-fungible tokens
## Indexing & Data
### API
**Application Programming Interface** is a set of protocols and tools for building software applications. APIs define how different software components should interact with each other.
### Endpoint
A URL that can be used to query an Envio custom API. Endpoints provide a structured way to request specific data from the indexer.
### GraphQL
A query language for interacting with APIs, commonly used in blockchain systems for retrieving specific data from blockchain platforms. As an alternative to REST, GraphQL lets developers construct requests that pull data from multiple data sources in a single API call.
### GraphQL API
The data presentation part of an Envio indexer. Typically, it's a GraphQL API auto-generated from the schema file, allowing flexible and efficient data queries.
### Blockchain Indexer
A specialized database management system (DBMS) that indexes and organizes blockchain data, making it easier for developers to efficiently query, retrieve, and utilize on-chain data.
Web2 apps usually rely on indexers like Google to pre-sort information into indices for data retrieval and filtering. In blockchain and Web3, applications need blockchain indexers to achieve similar data retrieval capabilities.
### Query
A request for data. In the context of Envio, a query is a request for data from an Envio API that will be answered by an Envio Indexer.
### Schema File
A file that defines entities based on events emitted from smart contracts and specifies the data types for these entities. The schema serves as the blueprint for your indexed data structure.
## Development Tools
### Codegen
The process of automatically generating code based on a given input. In blockchain development, codegen is often used for generating client libraries, interfaces, or type-safe data access layers from schemas or specifications.
### Envio CLI
A command line interface tool for building and deploying Envio indexers. The CLI provides commands for initializing, developing, and managing your indexer projects.
### SDK
**Software Development Kit** is a collection of tools, libraries, and documentation that facilitates the development of applications for a specific platform or system.
## Programming Languages
### JavaScript
A high-level, interpreted programming language primarily used for client-side scripting in web browsers. It is the de facto language for web development, enabling developers to create interactive and dynamic web applications.
### ReScript
A robustly typed language that compiles to efficient and human-readable JavaScript. ReScript aims to bring the power and expressiveness of functional programming to JavaScript development. It offers seamless integration with JavaScript and provides features like static typing, pattern matching, and immutable data structures.
### TypeScript
A superset of JavaScript that adds static typing and other advanced features to the language. It compiles down to plain JavaScript, making it compatible with existing JavaScript codebases. TypeScript helps developers catch errors during development by providing type-checking and improved tooling support. It enhances JavaScript by adding features like interfaces, classes, modules, and generics.
## Envio Platform
### Envio Cloud
A managed service platform for building, hosting, and querying Envio's Indexers with guaranteed uptime and performance service level agreements. Envio Cloud removes the operational burden of running blockchain indexers.
### Ploffen
Ploffen (meaning "Pop" in Dutch) is a fun game based on an ERC20 token contract, where users can deposit a game token (i.e., make a contribution) into a savings pool.
The last user to add a contribution to the savings pool has a chance of winning the entire pool if no other user deposits a contribution within 1 hour of the previous contribution. For example, if 30 persons play the game, and each person contributes a small amount, the last person can win the _total contributions_ made by all 30 persons in the savings pool.
The Ploffen project demonstrates a Hardhat framework example. It includes a sample contract, a test for that contract, a deployment script, and the Envio integration to index emitted events from the Ploffen smart contract.
## Mathematical Concepts
### Commutative Property
A fundamental property of certain binary operations in mathematics. An operation is said to be commutative if the order in which you apply the operation to two operands does not affect the result. In other words, for a commutative operation:
a + b = b + a
**Examples of commutative operations:**
1. **Addition**: 2 + 3 = 3 + 2
2. **Multiplication**: 2 _ 3 = 3 _ 2
**Examples of non-commutative operations:**
1. **Subtraction**: 5 - 3 ≠ 3 - 5
2. **Division**: 8 / 4 ≠ 4 / 8
3. **String Concatenation**: "Hello" + "World" ≠ "World" + "Hello"
The commutative property is a property of the operation itself, not necessarily the numbers involved. If an operation is commutative, you can switch the order of the operands without changing the result.
---
# Using WebSockets with GraphQL
> Learn how to use WebSocket connections with your HyperIndex GraphQL endpoint
# Using WebSockets with GraphQL
> **⚠️ Important**: WebSocket support is available but should be used at your own risk on plans other than dedicated.
## Overview
By default, HyperIndex provides GraphQL endpoints over HTTP/HTTPS. However, you can also connect to your GraphQL endpoint using WebSocket connections (WSS/WS) for real-time subscriptions and persistent connections.
## WebSocket Support Status
WebSocket connections are available with different levels of support depending on your hosting plan:
- **Dedicated**: WebSockets are fully supported. The dedicated package includes additional infrastructure such as connection pooling which enables proper WebSocket support.
- **Other Plans**: WebSocket support is available but should be used at your own risk. These plans do not have the same infrastructure support for WebSockets as dedicated. We don't recommend relying on WebSockets for more than 10 concurrent connections.
## How to Use WebSockets
To use WebSockets with your HyperIndex GraphQL endpoint, simply swap the protocol in your endpoint URL:
### Protocol Mapping
- **HTTPS → WSS**: Change `https://` to `wss://` in your GraphQL endpoint URL
- **HTTP → WS**: Change `http://` to `ws://` in your GraphQL endpoint URL
### Example
If your GraphQL endpoint is:
```
https://indexer.hyperindex.xyz/123abcd/graphql
```
You can connect via WebSocket using:
```
wss://indexer.hyperindex.xyz/123abcd/graphql
```
Similarly, for HTTP endpoints:
```
http://localhost:8080/v1/graphql
```
Becomes:
```
ws://localhost:8080/v1/graphql
```
## Usage Example
Here's a TypeScript example using the `graphql-ws` library to subscribe to real-time updates from your indexer:
```typescript
import { createClient } from 'graphql-ws';
// Define the entity you are subscribing to (must match your schema definition)
interface Swap {
id: string;
}
interface SubscriptionData {
data?: {
Swap?: Swap[];
};
}
const client = createClient({
url: 'ws://localhost:8080/v1/graphql',
});
console.log('Connecting to WebSocket...');
client.subscribe(
{
query: `
subscription {
Swap(order_by: { id: desc }, limit: 10) {
id
}
}
`,
},
{
next: (data) => {
console.log('\n📥 New event received:');
console.log(JSON.stringify(data, null, 2));
},
error: (err) => {
console.error('❌ Error:', err);
},
complete: () => {
console.log('✅ Subscription completed');
},
}
);
```
### Installation
To use this example, install the required dependencies:
```bash
npm install graphql-ws ws
# or
yarn add graphql-ws ws
```
## Getting Support
For any questions about WebSocket usage, contact the Envio team via [Telegram](https://t.me/envio) or [Discord](https://discord.gg/envio).
---
# Wildcard Indexing & Topic Filtering
> Learn how to index events using wildcard indexing feature and topic based filtering.
Wildcard indexing is a feature that allows you to index all events matching a specified event signature without requiring the contract address from which the event was emitted. This is useful in cases such as indexing contracts deployed through factories, where the factory contract does not emit any events upon contract creation. It also enables indexing events from all contracts implementing a standard (e.g. all ERC20 transfers).
:::note
Wildcard Indexing is supported on [HyperSync](/docs/HyperIndex/hypersync), [HyperFuel](/docs/HyperIndex/fuel), and [RPC](/docs/HyperIndex/rpc-sync) data sources. Since v3.3, an RPC-backed indexer can register **multiple** wildcard events — earlier versions allowed only one.
:::
## Index all ERC20 transfers
As an example, let's say we want to index all ERC20 `Transfer` events. Start with a [`config.yaml`](/docs/HyperIndex/configuration-file) file:
```yaml
name: transefer-indexer
chains:
- id: 1
start_block: 0
contracts:
- name: ERC20
events:
- event: Transfer(address indexed from, address indexed to, uint256 value)
```
Let's also define some entities in [`schema.graphql`](/docs/HyperIndex/schema) file, so our handlers can store the processed data:
```graphql
type Transfer {
id: ID!
from: String!
to: String!
}
```
And the last bit is to register an event handler in the `src/handlers`. Note how we pass the `wildcard: true` option to enable wildcard indexing:
```typescript
import { indexer } from "envio";
indexer.onEvent(
{ contract: "ERC20", event: "Transfer", wildcard: true },
async ({ event, context }) => {
context.Transfer.set({
id: `${event.chainId}_${event.block.number}_${event.logIndex}`,
from: event.params.from,
to: event.params.to,
});
},
);
```
After running your indexer with `pnpm dev` you will have all ERC20 `Transfer` events indexed, regardless of the contract address from which the event was emitted.
## Topic Filtering
Indexing all ERC20 `Transfer` events can be noisy. Use Topic Filtering to keep only the events you need.
When registering an event handler or a contract registration handler, provide the `where` option (formerly `eventFilters` in V2). Filter by each `indexed` event parameter by returning `{ params: [...] }`.
Let's say you only want to index `Mint` events where the `from` address is equal to `ZERO_ADDRESS`:
```typescript
import { indexer } from "envio";
const ZERO_ADDRESS = "0x0000000000000000000000000000000000000000";
indexer.onEvent(
{
contract: "ERC20",
event: "Transfer",
wildcard: true,
where: () => ({ params: [{ from: ZERO_ADDRESS }] }),
},
async ({ event, context }) => {
//... your handler logic
},
);
```
## Multiple Filters
If you want to index both `Mint` and `Burn` events you can provide multiple filters as an array, and an event is indexed if it matches any one of them. Within a single filter, all parameters must match. Also, every parameter can accept an array to filter by multiple possible values. We'll use it to filter by a group of whitelisted addresses in the example below:
```typescript
import { indexer } from "envio";
const ZERO_ADDRESS = "0x0000000000000000000000000000000000000000";
const WHITELISTED_ADDRESSES = [
"0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266",
"0x70997970C51812dc3A010C7d01b50e0d17dc79C8",
"0x3C44CdDdB6a900fa2b585dd299e03d12FA4293BC",
];
indexer.onEvent(
{
contract: "ERC20",
event: "Transfer",
wildcard: true,
where: () => ({
params: [
{ from: ZERO_ADDRESS, to: WHITELISTED_ADDRESSES },
{ from: WHITELISTED_ADDRESSES, to: ZERO_ADDRESS },
],
}),
},
async ({ event, context }) => {
//... your handler logic
},
);
```
:::note
On [RPC data sources](/docs/HyperIndex/rpc-sync), passing an array requires v3.3 or later. HyperSync supports it on all V3 versions.
:::
## Different Filters per Chain
For [Multichain Indexers](/docs/HyperIndex/multichain-indexing) the `where` callback receives `{ chain }` and you can read `chain.id` to filter by different values per chain:
```typescript
import { indexer } from "envio";
const ZERO_ADDRESS = "0x0000000000000000000000000000000000000000";
const WHITELISTED_ADDRESSES = {
1: ["0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266"],
137: [
"0x70997970C51812dc3A010C7d01b50e0d17dc79C8",
"0x3C44CdDdB6a900fa2b585dd299e03d12FA4293BC",
],
};
indexer.onEvent(
{
contract: "ERC20",
event: "Transfer",
wildcard: true,
where: ({ chain }) => ({
params: [
{ from: ZERO_ADDRESS, to: WHITELISTED_ADDRESSES[chain.id] },
{ from: WHITELISTED_ADDRESSES[chain.id], to: ZERO_ADDRESS },
],
}),
},
async ({ event, context }) => {
//... your handler logic
},
);
```
## Index all ERC20 transfers to your Contract
Besides `chain.id` you can also read the contract's configured (and dynamically registered) addresses from `chain..addresses`.
For example, if you have a `Safe` contract, you can index all ERC20 transfers sent specifically to/from your `Safe` contracts. The `where` callback can read `chain.Safe.addresses`, so we need to define the `Transfer` event on the `Safe` contract:
```yaml
name: locker
chains:
- id: 1
start_block: 0
contracts:
- name: Safe
events:
- event: Transfer(address indexed from, address indexed to, uint256 value)
address:
- "0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266"
- "0x70997970C51812dc3A010C7d01b50e0d17dc79C8"
- "0x3C44CdDdB6a900fa2b585dd299e03d12FA4293BC"
```
```typescript
import { indexer } from "envio";
indexer.onEvent(
{
contract: "Safe",
event: "Transfer",
wildcard: true,
where: ({ chain }) => ({
params: [
{ from: chain.Safe.addresses },
{ to: chain.Safe.addresses },
],
}),
},
async ({ event, context }) => {},
);
```
This example is not much different from using a `WHITELISTED_ADDRESSES` constant, but this becomes much more powerful when the `Safe` contract addresses are [registered dynamically](/docs/HyperIndex/dynamic-contracts) by a factory contract:
```yaml
name: locker
chains:
- id: 1
start_block: 0
contracts:
- name: SafeRegistry
events:
- event: NewSafe(address safe)
address:
- "0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266"
- name: Safe
events:
- event: Transfer(address indexed from, address indexed to, uint256 value)
```
```typescript
import { indexer } from "envio";
indexer.contractRegister(
{ contract: "SafeRegistry", event: "NewSafe" },
async ({ event, context }) => {
context.chain.Safe.add(event.params.safe);
},
);
indexer.onEvent(
{
contract: "Safe",
event: "Transfer",
wildcard: true,
where: ({ chain }) => ({
params: [
{ from: chain.Safe.addresses },
{ to: chain.Safe.addresses },
],
}),
},
async ({ event, context }) => {},
);
```
### Assert ERC20 Transfers in Handler
After you got all ERC20 Transfers relevant to your contracts, you can additionally filter them in the handler. For example, to get only `USDC` transfers:
```typescript
import { indexer } from "envio";
const USDC_ADDRESS = {
84532: "0x036CbD53842c5426634e7929541eC2318f3dCF7e",
11155111: "0x1c7D4B196Cb0C7B01d743Fbc6116a902379C7238",
};
indexer.onEvent(
{
contract: "Safe",
event: "Transfer",
wildcard: true,
where: ({ chain }) => ({
params: [
{ from: chain.Safe.addresses },
{ to: chain.Safe.addresses },
],
}),
},
async ({ event, context }) => {
// Filter and store only the USDC transfers that involve a Safe address
if (event.srcAddress === USDC_ADDRESS[event.chainId]) {
context.Transfer.set({
id: `${event.chainId}_${event.block.number}_${event.logIndex}`,
from: event.params.from,
to: event.params.to,
});
}
},
);
```
## Contract Register Example
The same `where` option can be applied to `indexer.contractRegister`. Here is an example where we only register Uniswap pools that contain DAI token:
```typescript
import { indexer } from "envio";
const DAI_ADDRESS = "0x6B175474E89094C44Da98b954EedeAC495271d0F";
indexer.contractRegister(
{
contract: "UniV3Factory",
event: "PoolCreated",
where: () => ({
params: [{ token0: DAI_ADDRESS }, { token1: DAI_ADDRESS }],
}),
},
async ({ event, context }) => {
const poolAddress = event.params.pool;
context.chain.UniV3Pool.add(poolAddress);
},
);
```
:::note
Wildcard registrations are matched by event signature. If the same signature is declared on more than one contract in your `config.yaml`, set `wildcard: true` on just one of them.
:::
---
# HyperIndex Benchmarks
> Discover HyperIndex benchmarks and see why it's the fastest blockchain data indexer.
import Head from '@docusaurus/Head';
{/* Dataset structured data: describes this benchmark page for search + answer
engines and points them at the open benchmark data. Metadata only — it does
not restate the figures in the table below. */}
# HyperIndex Performance Benchmarks
## Overview
HyperIndex delivers industry-leading performance for blockchain data indexing. Independent benchmarks have consistently shown Envio's HyperIndex to be the fastest blockchain indexing solution available, with dramatic performance advantages over competitive offerings.
## Recent Independent Benchmarks
The most comprehensive and up-to-date benchmarks were conducted by Sentio in April 2025 and are available in the [sentio-benchmark repository](https://github.com/enviodev/sentio-benchmark). These benchmarks compare Envio's HyperIndex against other popular blockchain indexers across multiple real-world scenarios:
### Key Performance Highlights
| Case | Description | Envio | Nearest Competitor | The Graph | Ponder |
| ------------------------------ | ------------------------------------------- | ------ | --------------------------- | ------------------- | -------------------- |
| LBTC Token Transfers | Event handling, No RPC calls, Write-only | 3m | 8m - 2.6x slower (Sentio) | 3h9m - 3780x slower | 1h40m - 2000x slower |
| LBTC Token with RPC calls | Event handling, RPC calls, Read-after-write | 1m | 6m - 6x slower (Sentio) | 1h3m - 63x slower | 45m - 45x slower |
| Ethereum Block Processing | 100K blocks with Metadata extraction | 7.9s | 1m - 7.5x slower (Subsquid) | 10m - 75x slower | 33m - 250x slower |
| Ethereum Transaction Gas Usage | Transaction handling, Gas calculations | 1m 26s | 7m - 4.8x slower (Subsquid) | N/A | 33m - 23x slower |
| Uniswap V2 Swap Trace Analysis | Transaction trace handling, Swap decoding | 41s | 2m - 3x slower (Subsquid) | 8m - 11x slower | N/A |
| Uniswap V2 Factory | Event handling, Pair and swap analysis | 8s | 2m - 15x slower (Subsquid) | 19m - 142x slower | 21m - 157x slower |
The independent benchmark results demonstrate that HyperIndex consistently outperforms all competitors across every tested scenario. This includes the most realistic real-world indexing scenario LBTC Token with RPC calls - where HyperIndex was up to 6x faster than the nearest competitor and over 63x faster than The Graph.
For a wider, benchmark-backed comparison across the rest of the category, see [Best Blockchain Indexers in 2026](/blog/best-blockchain-indexers-2026), which covers Envio, The Graph, Goldsky, SubQuery, Subsquid, Ormi, and Ponder side by side.
## Historical Benchmarking Results
Our internal benchmarking from October 2023 showed similar performance advantages. When indexing the Uniswap V3 ETH-USDC pool contract on Ethereum Mainnet, HyperIndex achieved:
- 2.1x faster indexing than the nearest competitor
- Over 100x faster indexing than some popular alternatives
You can read the full details in our [Indexer Benchmarking Results blog post](/blog/indexer-benchmarking-results).
## Verify For Yourself
We encourage developers to run their own benchmarks. You can also use the templates provided in the [Open Indexer Benchmark repository](https://github.com/enviodev/open-indexer-benchmark).
---
# HyperIndex Quickstart
> Learn to quickly autogenerate and configure a HyperIndex indexer for any smart contract.
The **Quickstart** enables you to instantly autogenerate a powerful blockchain indexer and start querying blockchain data in minutes. This is the fastest and easiest way to begin using HyperIndex.
**Example:** Autogenerate an indexer for the Eigenlayer contract and index its entire history in less than 5 minutes by simply running `pnpx envio init` and providing the contract address from [Etherscan](https://etherscan.io/address/0x858646372cc42e1a627fce94aa7a7033e7cf075a).
---
## Prerequisites
- **[Node.js](https://nodejs.org/en/download/current)** _(v22 or newer recommended)_
- **[pnpm](https://pnpm.io/installation)** _(recommended but not required)_
- **[Docker Desktop](https://www.docker.com/products/docker-desktop/)** _(required to run the Envio indexer locally)_
> **Note:** Docker is only required if you plan to run your indexer locally. You can skip installing Docker if you'll only be using Envio Cloud.
### Additionally for Windows Users:
- **[WSL](https://learn.microsoft.com/en-us/windows/wsl/install)** _Windows Subsystem for Linux_
---
## Getting Started
Run the following command to initialize your blockchain indexer:
```bash
pnpx envio init
```
You'll then follow interactive prompts to customize your indexer.
---
### Video Tutorials
---
## Indexer Initialization Options
During initialization, you'll be presented with two options:
- **Contract Import** _(recommended for existing smart contracts)_
- **Template**
Choose the **Contract Import** option to auto-generate indexers directly from smart contracts.
```bash
? Choose an initialization option
Template
> Contract Import
[↑↓ to move, enter to select]
```
---
## Contract Import Methods
There are two convenient methods to import your contract:
- **Block Explorer** _(verified contracts on supported explorers like Etherscan and Blockscout)_
- **Local ABI** _(custom or unverified contracts)_
### 1. Block Explorer Import
This method uses a verified contract's address from a supported blockchain explorer (Etherscan, Routescan, etc.) to automatically fetch the ABI.
#### Steps:
**a. Select the blockchain**
```bash
? Which blockchain would you like to import a contract from?
> ethereum-mainnet
goerli
optimism
base
bsc
gnosis
polygon
[↑↓ to move, enter to select]
```
:::note
HyperIndex supports all EVM-compatible chains. If your desired chain is not listed, you can import via the local ABI method or manually adjust the `config.yaml` file after initialization.
:::
**b. Enter the contract address**
```bash
? What is the address of the contract?
[Use proxy address if ABI is for a proxy implementation]
```
:::tip
If using a proxy contract, always specify the **proxy address**, not the implementation address.
:::
**c. Select events to index**
```bash
? Which events would you like to index?
> [x] ClaimRewards(address indexed from, address indexed reward, uint256 amount)
[x] Deposit(address indexed from, uint256 indexed tokenId, uint256 amount)
[x] NotifyReward(address indexed from, address indexed reward, uint256 indexed epoch, uint256 amount)
[x] Withdraw(address indexed from, uint256 indexed tokenId, uint256 amount)
[space to select, → to select all, ← to deselect all]
```
**d. Finish or add more contracts**
You'll be prompted to continue adding more contracts or to complete the setup:
```bash
? Would you like to add another contract?
> I'm finished
Add a new address for same contract on same network
Add a new network for same contract
Add a new contract (with a different ABI)
```
---
### 2. Local ABI Import
Choose this method if the contract ABI is unavailable from a block explorer or you're using an unverified contract.
#### Steps:
**a. Select Local ABI**
```bash
? Would you like to import from a block explorer or a local abi?
Block Explorer
> Local ABI
[↑↓ to move, enter to select]
```
**b. Specify ABI JSON file**
Provide the path to your local ABI file (JSON format):
```bash
? What is the path to your json abi file?
```
**c. Select events to index**
```bash
? Which events would you like to index?
> [x] ClaimRewards(address indexed from, address indexed reward, uint256 amount)
[x] Deposit(address indexed from, uint256 indexed tokenId, uint256 amount)
[space to select, → to select all, ← to deselect all]
```
**d. Choose blockchain**
Specify the blockchain your contract is deployed on:
```bash
? Choose network:
> ethereum-mainnet
goerli
optimism
base
bsc
gnosis
[Custom Network ID]
[↑↓ to move, enter to select]
```
**e. Enter contract details**
- **Contract name**
```bash
? What is the name of this contract?
```
- **Contract address**
```bash
? What is the address of the contract?
[Use proxy address if ABI is for a proxy implementation]
```
**f. Finish or add more contracts**
Complete the import process or continue adding contracts:
```bash
? Would you like to add another contract?
> I'm finished
Add a new address for same contract on same network
Add a new network for same contract
Add a new contract (with a different ABI)
```
---
## Generated Files & Configuration
The Quickstart automatically generates key files:
### 1. `config.yaml`
Automatically configured parameters include:
- **Network ID**
- **Start Block**
- **Contract Name**
- **Contract Address**
- **Event Signatures**
By default, all selected events are included, but you can manually adjust the file if needed. See the detailed guide on [`config.yaml`](configuration-file).
### 2. GraphQL Schema
- Entities are automatically generated for each selected event.
- Fields match the event parameters emitted.
See more details in the [schema file guide](./Guides/schema-file.md).
### 3. Event Handlers
- Handlers are autogenerated for each event.
- Handlers create event-specific entities.
Learn more in the [event handlers guide](./Guides/event-handlers.mdx).
---
**Congratulations!** Your HyperIndex indexer is now ready to run and query data!
Next step: [Running your Indexer locally](./running-locally) or [Deploying to Envio Cloud](./hosted-service).
---
## Other Ways to Start
Contract Import is the recommended path, but you can also bootstrap an indexer from:
- **Templates** — pre-built `ERC20` or [Greeter](./greeter-tutorial) projects, selectable from the `pnpx envio init` interactive prompt.
- **Examples** — copy and adapt an existing indexer from the [Envio Explorer](https://envio.dev/explorer), our [Tutorials](./tutorial-erc20-token-transfers), or the [GitHub repositories](https://github.com/enviodev).
---
# Velodrome & Aerodrome Indexer
> Explore multi-chain Velodrome & Aerodrome DEX data with real-time tracking of pools, swaps, and liquidity.
# Velodrome & Aerodrome DEX Indexer
This blockchain indexer provides a comprehensive implementation for tracking data from the popular Velodrome and Aerodrome decentralized exchanges across multiple chains.
## Overview
The [Velodrome & Aerodrome Indexer](https://github.com/enviodev/velodrome-indexer) demonstrates how to:
- Create a **multichain indexer** that works across Superchain networks
- Index core DEX functionality including pools, swaps, and liquidity positions
- Unify data access through a single GraphQL API
- Build efficient handlers using TypeScript
## Key Features
- Complete tracking of liquidity pools and trading activity
- Real-time synchronization across multiple networks
- Well-documented codebase suitable as a starting point for similar DEX indexers
- Making ad-hoc RPC requests for additional data (use with caution)
For implementation details, usage examples, and setup instructions, see the project's [README](https://github.com/enviodev/velodrome-indexer/blob/main/README.md).
:::note
This indexer was built by Envio partners and community builders and serves as a valuable reference implementation. As with any indexer, perform appropriate testing and data validation before using in production environments.
:::
---
# Cross-Chain Messaging
> Explore cross-chain messaging and track events across multiple chains efficiently.
# Cross-Chain Messaging Indexer
This blockchain indexer demonstrates how to track and index cross-chain messaging protocols at scale, providing a complete view of message passing across multiple blockchains.
:::warning Targets HyperIndex V2
This example links to a third-party community repository that targets HyperIndex **V2** and is now archived (last updated December 2024), so it does not reflect current V3 APIs. We keep it here as a conceptual reference for cross-chain indexing patterns. If you are starting a new project, build on the current [V3 docs](/docs/HyperIndex/overview) and see [Migrate to V3](/docs/HyperIndex/migrate-to-v3) for the latest syntax.
:::
## Overview
The [ORMP Indexer](https://github.com/ringecosystem/ormpexer/tree/main) showcases advanced multichain indexing techniques for tracking cross-chain communication. It offers a practical example of how to:
- Monitor message passing across 14 different EVM-compatible blockchains
- Combine both HyperSync and RPC data sources for optimal performance
- Create a unified data model for cross-chain events
- Build a scalable indexing solution for complex cross-chain applications
## Key Features
- Comprehensive tracking of cross-chain message delivery and execution
- Efficient data collection using a hybrid approach (HyperSync + RPC)
- Well-structured GraphQL API for querying cross-chain messaging data
- Example implementation that can be adapted for other cross-chain protocols
## Use Cases
This blockchain indexer can serve as a foundation for applications requiring visibility into cross-chain operations, including:
- Cross-chain bridges and token transfer monitoring
- Multichain DeFi dashboards
- Cross-chain governance systems
- Message delivery verification tools
:::note
This blockchain indexer was built by Envio partners and community builders. As noted above it targets HyperIndex V2 and is no longer maintained, so always perform appropriate testing and data validation, and follow the current V3 docs, before using these patterns in production.
:::
---
# Sablier Protocol Indexers
> Explore Sablier protocol indexers to track token streams and distributions across multiple chains.
# Sablier Protocol Indexers
The following blockchain indexers serve as exceptional reference implementations for the Sablier protocol, showcasing professional development practices and efficient multichain data processing.
## Overview
[Sablier](https://sablier.com/) is a token streaming protocol that enables real-time finance on the blockchain, allowing tokens to be streamed continuously over time. These official Sablier indexers track streaming activity across many EVM-compatible chains, providing comprehensive data through a unified GraphQL API.
## Professional Indexer Suite
Sablier maintains two public indexers, each targeting a specific part of their protocol:
### 1. [Streams Indexer](https://github.com/sablier-labs/indexers/tree/main/envio/streams)
Tracks Sablier's payment-stream data across both the Lockup and Flow products. Lockup covers streams with fixed durations and amounts (creation, cancellation, and withdrawal events), while Flow covers open-ended streaming with dynamic flow rates. For more detail, see the [Lockup indexer docs](https://docs.sablier.com/api/lockup/indexers) and the [Flow indexer docs](https://docs.sablier.com/api/flow/indexers).
### 2. [Airdrops Indexer](https://github.com/sablier-labs/indexers/tree/main/envio/airdrops)
Tracks Sablier's Merkle airdrop campaigns, which enable efficient batch stream creation using cryptographic proofs. This indexer captures data about campaign creation, claims, and related activity, powering both Airstreams and Instant Airdrops. For more detail, see the [Airdrops indexer docs](https://docs.sablier.com/api/airdrops/indexers).
## Key Features
- **Comprehensive Multichain Support**: Indexes data across many EVM-compatible chains
- **Professionally Maintained**: Used in production by the Sablier team and their partners
- **Extensive Test Coverage**: Includes comprehensive testing to ensure data accuracy
- **Optimized Performance**: Implements efficient data processing techniques
- **Well-Documented**: Clear code structure with extensive comments
- **Backward Compatibility**: Carefully manages schema evolution and contract upgrades
- **Cross-chain Architecture**: Envio promotes efficient cross-chain indexing where all networks share the same indexer endpoint
## Best Practices Showcase
These blockchain indexers demonstrate several development best practices:
- **Modular Code Structure**: Well-organized code with clear separation of concerns
- **Consistent Naming Conventions**: Professional and consistent naming throughout
- **Efficient Event Handling**: Optimized processing of blockchain events
- **Comprehensive Entity Relationships**: Well-designed data model with proper relationships
- **Thorough Input Validation**: Robust error handling and input validation
- **Detailed Changelogs**: Documentation of breaking changes and migrations
- **Preload Optimization**: Envio indexers benefit from always-on [Preload Optimization](/docs/HyperIndex/preload-optimization), which batches entity reads and runs external calls in parallel through the Effect API
## Getting Started
To use these indexers as a reference for your own development:
1. Clone the indexer that matches your needs:
- [Streams Indexer](https://github.com/sablier-labs/indexers/tree/main/envio/streams)
- [Airdrops Indexer](https://github.com/sablier-labs/indexers/tree/main/envio/airdrops)
2. Review the file structure and implementation patterns
3. Examine the event handlers for efficient data processing techniques
4. Study the schema design for effective entity modeling
For complete API documentation and usage examples, see:
- [Sablier API Overview](https://docs.sablier.com/api/overview#envio)
:::note
These are official indexers maintained by the Sablier team and represent production-quality implementations. They serve as an excellent example of professional blockchain indexer development and are regularly updated to support the latest protocol features.
:::
---
# Uniswap V4 Multichain indexer
> Explore real-time Uniswap V4 data across multiple chains with Envio.
# Uniswap V4 Multichain Indexer
The following blockchain indexer example is a reference implementation and can serve as a starting point for applications with similar logic.
This [official Uniswap V4 indexer](https://github.com/enviodev/uniswap-v4-indexer) is a comprehensive implementation for the Uniswap V4 protocol using Envio HyperIndex. This is the same indexer that powers the [v4.xyz](https://v4.xyz) website, providing real-time data for the Uniswap V4 interface.
## Key Features
- **Multichain Support**: Indexes Uniswap V4 deployments across every supported blockchain network in real-time
- **Complete Pool Metrics**: Tracks pool statistics including volume, TVL, fees, and other critical metrics
- **Swap Analysis**: Monitors swap events and liquidity changes with high precision
- **Hook Integration**: In-progress support for Uniswap V4 hooks and their events
- **Production Ready**: Powers the official v4.xyz interface with production-grade reliability
- **Ultra-Fast Syncing**: Processes massive amounts of blockchain data significantly faster than alternative blockchain indexing solutions, reducing sync times from days to minutes
## Technical Overview
This indexer is built using TypeScript and provides a unified GraphQL API for accessing Uniswap V4 data across all supported networks. The architecture is designed to handle high throughput and maintain consistency across different blockchain networks.
### Performance Advantages
The Envio-powered Uniswap V4 indexer offers extraordinary performance benefits:
- **10-100x Faster Sync Times**: Leveraging Envio's HyperSync technology, this indexer can process historical blockchain data orders of magnitude faster than traditional solutions
- **Real-time Updates**: Maintains low latency for new blocks while efficiently managing historical data
## Use Cases
- Power analytics dashboards and trading interfaces
- Monitor DeFi positions and protocol health
- Track historical performance of Uniswap V4 pools
- Build custom notifications and alerts
- Analyze hook interactions and their impact
## Getting Started
To use this indexer, you can:
1. Clone the [repository](https://github.com/enviodev/uniswap-v4-indexer)
2. Follow the installation instructions in the README
3. Run the indexer locally or deploy it to a production environment
4. Access indexed data through the GraphQL API
## Contribution
The Uniswap V4 indexer is actively maintained and welcomes contributions from the community. If you'd like to contribute or report issues, please visit the [GitHub repository](https://github.com/enviodev/uniswap-v4-indexer).
:::note
This is an official reference implementation that powers the v4.xyz website. While extensively tested in production, remember to validate the data for your specific use case. The indexer is continuously updated to support the latest Uniswap V4 features and optimizations.
:::
---
# Fuel
> Explore how to index and query real-time and historical data on Fuel Network with HyperIndex.
# Indexing on Fuel Network
## Introduction
Envio supports the [Fuel Network](https://fuel.network/) on mainnet and testnet. This page shows how to use HyperIndex with Fuel’s architecture and features.
Fuel offers several advantages as a modular execution layer including:
- Parallel transaction execution
- State-minimized design
- UTXO-based architecture
- Advanced FuelVM capabilities
## HyperIndex for Fuel
[HyperIndex](../overview.md) enables developers to easily index and query real-time and historical data on Fuel Network with the same powerful features available for EVM chains.
### Getting Started with Fuel Indexing
You can start indexing Fuel contracts in two ways:
1. **Quick Start (5-minute tutorial)**: Follow our step-by-step [tutorial](../Tutorials/tutorial-indexing-fuel.md) to create your first Fuel indexer quickly.
2. **No-Code Contract Import**: Use our [Contract Import](../contract-import.md) tool to automatically generate configuration and schema files for your Fuel contracts.
### Example Fuel Indexers
Looking for inspiration? Check out these indexers built by projects in the Fuel ecosystem:
| Project | Type | GitHub Repository |
| ------------------------------------- | ---------------- | --------------------------------------------------------------------------------- |
| [Spark](https://sprk.fi/) | Orderbook DEX | [github](https://github.com/compolabs/spark-envio-indexer) |
| [Mira](https://mira.ly/) | AMM DEX | [github](https://github.com/mira-amm/mira-indexer) |
| [Thunder](https://thundernft.market/) | NFT Marketplace | [github](https://github.com/ThunderFuel/thunder-indexer) |
| [Swaylend](https://swaylend.com/) | Lending Protocol | [github](https://github.com/Swaylend/swaylend-monorepo/tree/develop/apps/indexer) |
| Greeter | Tutorial | [github](https://github.com/enviodev/fuel-greeter) |
### Features Supported on Fuel
HyperIndex for Fuel supports all the core features available in the EVM version:
- ✅ [No-code Contract Import](../contract-import.md)
- ✅ [Dynamic Contracts / Factory Tracking](../Advanced/dynamic-contracts.md)
- ✅ [Testing Framework](/docs/HyperIndex/testing)
- ✅ [Envio Cloud](../Hosted_Service/hosted-service.md)
- ✅ [Wildcard Indexing](../Advanced/wildcard-indexing.mdx)
## Fuel-Specific Event Types
### Understanding Fuel's Event Model
Fuel's event model differs significantly from EVM. Instead of predefined events, Fuel uses a more flexible approach with various receipt types that can be indexed.
### LOG_DATA Receipts (Primary Event Type)
The most common event type in Fuel is the `LOG_DATA` receipt, created by the `log` instruction in [Sway](https://docs.fuel.network/docs/sway/) contracts.
Unlike Solidity's `emit` which requires predefined event structures, Sway's `log` function allows passing any data, providing greater flexibility.
#### Configuration Example:
```yaml
ecosystem: fuel
network:
name: "fuel_testnet"
contracts:
- name: SwayContract
abi_file_path: "./abis/SwayContract.json"
start_block: 1
address: "0x123..."
events:
- name: NewGreeting
logId: "8500535089865083573"
```
The `logId` is a unique identifier for the logged struct, which you can find in your contract's ABI file.
#### Auto-detection of logId:
If your event name matches the logged struct name in Sway, you can omit the `logId`:
```yaml
events:
- name: NewGreeting # Will automatically detect logId if it matches the struct name
```
> **Tip**: Instead of manually configuring events, use the [Contract Import](../contract-import.md) tool which automatically detects events and generates the proper configuration.
### Additional Fuel Event Types
Fuel allows indexing several additional receipt types not available in EVM:
| Event Type | Description | Example Configuration |
| ---------- | ------------------------------------------------ | --------------------- |
| `Mint` | Triggered when a contract mints tokens | `- name: Mint` |
| `Burn` | Triggered when a contract burns tokens | `- name: Burn` |
| `Transfer` | Combines `TRANSFER` and `TRANSFER_OUT` receipts | `- name: Transfer` |
| `Call` | Triggered when a contract calls another contract | `- name: Call` |
#### Using Custom Names:
You can rename these events while maintaining their type:
```yaml
events:
- name: MintMyNft # Custom name
type: mint # Actual event type
```
> **Note**: All event types can be used with [Wildcard Indexing](../Advanced/wildcard-indexing.mdx).
### Transfer Event Specifics
The `Transfer` event type combines two Fuel receipt types:
- `TRANSFER`: Emitted when a contract transfers tokens to another contract
- `TRANSFER_OUT`: Emitted when a contract transfers tokens to a wallet
> **Important**: Transfers between wallets are not included in the `Transfer` event type.
## Event Object Structure in Handlers
When handling Fuel events, the event object structure differs from EVM:
```typescript
// Example Fuel event handler
import { indexer } from "envio";
indexer.onEvent(
{ contract: "SwayContract", event: "NewGreeting" },
async ({ event, context }) => {
// Access event parameters
const message = event.params.message;
// Access block information
const blockHeight = event.block.height;
const blockTime = event.block.time;
const blockId = event.block.id;
// Access transaction information
const txId = event.transaction.id;
// Access source contract address
const sourceContract = event.srcAddress;
// Access log position
const logIndex = event.logIndex;
// Store data
context.Greeting.set({
id: event.transaction.id,
message: message,
timestamp: blockTime,
});
},
);
```
## HyperFuel
[HyperFuel](/docs/HyperSync/hyperfuel) is Envio's low-level data API for the Fuel Network (equivalent to [HyperSync](/docs/HyperSync/overview) for EVM chains).
HyperFuel provides:
- High-performance data access
- Flexible query capabilities
- Multiple data formats (Parquet, Arrow, typed data)
- Complete historical data
### Available Clients
Access HyperFuel data using any of these clients:
- **Rust**: [hyperfuel-client-rust](https://github.com/enviodev/hyperfuel-client-rust)
- **Python**: [hyperfuel-client-python](https://github.com/enviodev/hyperfuel-client-python)
- **Node.js**: [hyperfuel-client-node](https://github.com/enviodev/hyperfuel-client-node)
- **JSON API**: [hyperfuel-json-api](https://github.com/enviodev/hyperfuel-json-api)
### HyperFuel Endpoints
- **Mainnet**: https://fuel.hypersync.xyz
- **Testnet**: https://fuel-testnet.hypersync.xyz
For detailed information, see the [HyperFuel documentation](/docs/HyperSync/hyperfuel).
## About Fuel Network
[Fuel](https://fuel.network/) is an operating system purpose-built for Ethereum rollups with unique architecture focused on:
- **P**arallelization: Execute transactions concurrently for higher throughput
- **S**tate-minimized execution: Efficient storage and computation model
- **I**nteroperability: Seamless integration with other blockchain systems
Powered by the FuelVM, Fuel expands Ethereum's capabilities without compromising security or decentralization.
### Resources
- [Website](https://fuel.network/)
- [Twitter](https://twitter.com/fuel_network)
- [Discord](https://discord.com/invite/xfpK4Pe)
- [Documentation](https://docs.fuel.network/)
## Need Help?
If you encounter any issues with Fuel indexing, please:
1. Check our [Troubleshooting](../Troubleshoot/common-issues.md) guides
2. Join our [Discord](https://discord.gg/envio) for community support
3. Create an issue in our [GitHub repository](https://github.com/enviodev/hyperindex)
---
# Block Handlers
> Learn how to run custom logic on every blockchain block or at set intervals using onBlock.
# Block Handlers
Run logic on every block or an interval.
---
`indexer.onBlock` lets you run logic on every block or an interval. This is useful for aggregations and time-series logic.
To get started, import the `indexer` value from `envio` and call `onBlock` in one of your handler files.
```typescript
import { indexer } from "envio";
indexer.onBlock(
{
name: "MyBlockHandler",
},
async ({ block, context }) => {
context.log.info(`Processing block ${block.number}`);
}
);
```
> Block handlers don't require any config changes as well as codegen runs.
In the example above, the handler runs on **every** chain configured in `config.yaml` (the V3 default). To restrict it to a single chain, pass a `where` callback — see below.
## Options
`indexer.onBlock` accepts an options object as the first argument with the following properties:
- `name` (required) — The name of the block handler. It's used for logging, debugging and metrics.
- `where` (optional) — A callback `({ chain }) => false | true | { block: { number: { _gte?, _lte?, _every? } } }` that decides which chains the handler runs on and over which block range/interval. Omit it to run on every chain on every block.
To express the V2-era `chain`, `startBlock`, `endBlock`, and `interval` options, return them from `where`:
```typescript
import { indexer } from "envio";
indexer.onBlock(
{
name: "MyBlockHandler",
where: ({ chain }) => {
if (chain.id !== 1) return false;
return {
block: {
number: {
_gte: 19_000_000, // start block (inclusive)
_lte: 20_000_000, // end block (inclusive)
_every: 100, // run every Nth block
},
},
};
},
},
async ({ block, context }) => {
context.log.info(`Processing block ${block.number}`);
}
);
```
## Handler Function
:::note
Preload Optimization is always enabled in HyperIndex V3 and powers Block Handlers. Don't forget that it makes your handlers run twice.
:::
The second argument is a handler function that receives the block object and the handler context.
- `block` — The block object.
- `number` — The block number.
- More fields will be added in the future. Let us know in [Discord](https://discord.gg/envio) if you need any specific fields. You can also use [Effect API](/docs/HyperIndex/effect-api) to get the data from RPC.
- `context` — Exactly the same as the Event Handlers [Context Object](/docs/HyperIndex/event-handlers#context-object). Use `context.chain.id` to read the current chain ID inside the handler.
## Multichain
By default `indexer.onBlock` runs on every chain in your config. To run different parameters per chain, branch inside the `where` callback:
```typescript
import { indexer } from "envio";
const perChain = {
1: { startBlock: 19783636, interval: (60 * 60) / 12 }, // Every 60 minutes (12s block time)
10: { startBlock: 119534316, interval: (60 * 60) / 2 }, // Every 60 minutes (2s block time)
} as const;
indexer.onBlock(
{
name: "HourlyPrice",
where: ({ chain }) => {
const cfg = perChain[chain.id as keyof typeof perChain];
if (!cfg) return false;
return {
block: {
number: { _gte: cfg.startBlock, _every: cfg.interval },
},
};
},
},
async ({ block, context }) => {
context.log.info(`Processing block ${block.number} on chain ${context.chain.id}`);
}
);
```
## Time Interval
The `_every` option is a number of blocks. But quite often you want to run some logic on a time interval. To convert time interval to blocks, you can use the following formula:
```typescript
// Every 60 minutes
const timeIntervalInSeconds = 60 * 60;
// 12 seconds per block
const secondsPerBlock = 12;
// 300 blocks per 60 minutes
const blockInterval = timeIntervalInSeconds / secondsPerBlock;
```
## Different Historical and Realtime Intervals
Here's the recipe to speed up your historical sync by increasing the interval for historical blocks.
You can achieve this by registering multiple block handlers with the same handler, but different `_gte`, `_lte`, and `_every` values.
```typescript
import { indexer } from "envio";
const realtimeBlocks = {
1: 19783636,
10: 119534316,
} as const;
indexer.onBlock(
{
name: "HistoricalBlockHandler",
where: ({ chain }) => {
const realtime = realtimeBlocks[chain.id as keyof typeof realtimeBlocks];
if (!realtime) return false;
return { block: { number: { _lte: realtime - 1, _every: 1000 } } };
},
},
async ({ block, context }) => {
context.log.info(`Processing block ${block.number}`);
}
);
indexer.onBlock(
{
name: "RealtimeBlockHandler",
where: ({ chain }) => {
const realtime = realtimeBlocks[chain.id as keyof typeof realtimeBlocks];
if (!realtime) return false;
return { block: { number: { _gte: realtime } } };
},
},
async ({ block, context }) => {
context.log.info(`Processing block ${block.number}`);
}
);
```
In this case we'll run the historical handler on every 1000 blocks and from the realtime block we'll start running the second handler on every block.
> We recommend exploring the approach together with [HyperSync](/docs/HyperSync/overview) client to effectively query data for big block ranges.
## Preset Handler
This is not an official feature, but a creative way to use block handlers. You can define a block handler that runs on a single block at the start of the chain and use it to populate the database with the initial data.
```typescript
import { indexer } from "envio";
indexer.onBlock(
{
name: "Preset",
where: ({ chain }) => {
if (chain.id !== 1) return false;
return { block: { number: { _gte: 0, _lte: 0 } } };
},
},
async ({ block, context }) => {
// You don't need preload optimization here,
// so don't forget to disable it to prevent double-run.
if (context.isPreload) return;
const users = await fetch("https://api.example.com/users");
users.forEach((user) => {
context.User.set({
id: user.id,
address: user.address,
name: user.name,
});
});
}
);
```
## Current Limitations
- Only block number is provided in the block object. We'll definitely add more fields in the future.
---
# Envio CLI Reference
> Explore and manage indexer projects with Envio CLI, from setup to development and benchmarking.
# Envio Command Line Interface
This comprehensive reference guide covers all available commands and options in the Envio CLI tool for HyperIndex **V3**. Use this documentation to explore the full capabilities of the `envio` command and its subcommands for managing your blockchain indexing projects.
:::tip Envio Cloud CLI
Looking to manage your hosted indexers from the command line? See the **[Envio Cloud CLI](/docs/HyperIndex/envio-cloud-cli)** for deployment, monitoring, and management commands for Envio Cloud.
:::
## Getting Started
The Envio CLI provides a powerful set of tools for creating, developing, and managing your blockchain indexers. Whether you're starting a new project, running a development server, or deploying to production, the CLI offers commands to simplify and automate your workflow.
The fastest way to get going is `pnpx envio init`, which scaffolds a project interactively. From there, `envio dev` runs your indexer locally while you iterate, and `envio start` runs it in production.
**Command Overview:**
* [`envio`↴](#envio)
* [`envio init`↴](#envio-init)
* [`envio init contract-import`↴](#envio-init-contract-import)
* [`envio init contract-import explorer`↴](#envio-init-contract-import-explorer)
* [`envio init contract-import local`↴](#envio-init-contract-import-local)
* [`envio init template`↴](#envio-init-template)
* [`envio init svm`↴](#envio-init-svm)
* [`envio init svm template`↴](#envio-init-svm-template)
* [`envio init fuel`↴](#envio-init-fuel)
* [`envio init fuel contract-import`↴](#envio-init-fuel-contract-import)
* [`envio init fuel contract-import local`↴](#envio-init-fuel-contract-import-local)
* [`envio init fuel template`↴](#envio-init-fuel-template)
* [`envio dev`↴](#envio-dev)
* [`envio stop`↴](#envio-stop)
* [`envio codegen`↴](#envio-codegen)
* [`envio local`↴](#envio-local)
* [`envio local docker`↴](#envio-local-docker)
* [`envio local docker up`↴](#envio-local-docker-up)
* [`envio local docker down`↴](#envio-local-docker-down)
* [`envio local db-migrate`↴](#envio-local-db-migrate)
* [`envio local db-migrate up`↴](#envio-local-db-migrate-up)
* [`envio local db-migrate down`↴](#envio-local-db-migrate-down)
* [`envio local db-migrate setup`↴](#envio-local-db-migrate-setup)
* [`envio start`↴](#envio-start)
* [`envio metrics`↴](#envio-metrics)
* [`envio metrics runtime`↴](#envio-metrics-runtime)
* [`envio skills`↴](#envio-skills)
* [`envio skills update`↴](#envio-skills-update)
* [`envio tools`↴](#envio-tools)
* [`envio tools search-docs`↴](#envio-tools-search-docs)
* [`envio tools fetch-docs`↴](#envio-tools-fetch-docs)
* [`envio config`↴](#envio-config)
* [`envio config view`↴](#envio-config-view)
## `envio`
**Usage:** `envio [OPTIONS] `
###### **Subcommands:**
* `init` — Initialize an indexer with one of the initialization options
* `dev` — Development commands for starting, stopping, and restarting the indexer. Runs codegen automatically before launching
* `stop` — Stop the local environment - delete the database and stop all processes (including Docker) for the current directory
* `codegen` — Generate indexing code from user-defined configuration & schema files
* `local` — Prepare local environment for envio testing
* `start` — Start the indexer. Runs codegen automatically before launching so the on-disk types stay in sync with `config.yaml` and `schema.graphql`
* `metrics` — Fetch raw Prometheus metrics from the running indexer's /metrics endpoint
* `skills` — Manage Envio-provided Claude Code skills under `.claude/skills/`
* `tools` — Tools for people and AI agents (search-docs, fetch-docs). Run `envio tools help` for details
* `config` — Inspect the indexer config
###### **Options:**
* `-d`, `--directory ` — The directory of the project. Defaults to current dir ("./")
* `--config ` — The file in the project containing the configuration. It can also be set via the `ENVIO_CONFIG` environment variable
Default value: `config.yaml`
## `envio init`
Initialize an indexer with one of the initialization options
**Usage:** `envio init [OPTIONS] [COMMAND]`
###### **Subcommands:**
* `contract-import` — Initialize Evm indexer by importing config from a contract for a given chain
* `template` — Initialize Evm indexer from an example template
* `svm` — Initialization option for creating Svm indexer
* `fuel` — Initialization option for creating Fuel indexer
###### **Options:**
* `-n`, `--name ` — The name of your project
* `-l`, `--language ` — The language used to write handlers
Possible values: `typescript`, `rescript`
* `--package-manager ` — The package manager used for `install` and post-init build steps (default: pnpm)
Possible values: `pnpm`, `npm`, `yarn`, `bun`
* `--api-token ` — The hypersync API key to be initialized in your templates .env file. Falls back to the `ENVIO_API_TOKEN` environment variable
## `envio init contract-import`
Initialize Evm indexer by importing config from a contract for a given chain
**Usage:** `envio init contract-import [OPTIONS] [COMMAND]`
###### **Subcommands:**
* `explorer` — Initialize by pulling the contract ABI from a block explorer
* `local` — Initialize from a local json ABI file
###### **Options:**
* `-c`, `--contract-address ` — Contract address to generate the config from
* `--single-contract` — If selected, prompt will not ask for additional contracts/addresses/chains
* `--all-events` — If selected, prompt will not ask to confirm selection of events on a contract
## `envio init contract-import explorer`
Initialize by pulling the contract ABI from a block explorer
**Usage:** `envio init contract-import explorer [OPTIONS]`
###### **Options:**
* `-b`, `--blockchain ` — Network to import the contract from
Possible values: `abstract`, `amoy`, `arbitrum-nova`, `arbitrum-one`, `arbitrum-sepolia`, `arbitrum-testnet`, `aurora`, `aurora-testnet`, `avalanche`, `b2-testnet`, `base`, `base-sepolia`, `berachain`, `blast`, `blast-sepolia`, `boba`, `bsc`, `bsc-testnet`, `celo`, `celo-alfajores`, `celo-baklava`, `citrea-testnet`, `crab`, `curtis`, `ethereum-mainnet`, `evmos`, `fantom`, `fantom-testnet`, `fhenix-helium`, `flare`, `fraxtal`, `fuji`, `galadriel-devnet`, `gnosis`, `gnosis-chiado`, `goerli`, `harmony`, `holesky`, `hoodi`, `hyperliquid`, `kroma`, `linea`, `linea-sepolia`, `lisk`, `lukso`, `lukso-testnet`, `manta`, `mantle`, `mantle-testnet`, `megaeth-testnet2`, `metis`, `mode`, `mode-sepolia`, `monad`, `monad-testnet`, `moonbase-alpha`, `moonbeam`, `moonriver`, `morph`, `morph-testnet`, `neon-evm`, `opbnb`, `optimism`, `optimism-sepolia`, `plasma`, `poa-core`, `poa-sokol`, `polygon`, `polygon-zkevm`, `polygon-zkevm-testnet`, `rsk`, `saakuru`, `scroll`, `scroll-sepolia`, `sei`, `sei-testnet`, `sepolia`, `shimmer-evm`, `sonic`, `sonic-testnet`, `sophon`, `sophon-testnet`, `swell`, `taiko`, `tangle`, `unichain`, `unichain-sepolia`, `worldchain`, `xdc`, `xdc-testnet`, `zeta`, `zksync-era`, `zora`, `zora-sepolia`
* `--api-token ` — API token for the block explorer
* `--single-contract` — If selected, prompt will not ask for additional contracts/addresses/chains
* `--all-events` — If selected, prompt will not ask to confirm selection of events on a contract
## `envio init contract-import local`
Initialize from a local json ABI file
**Usage:** `envio init contract-import local [OPTIONS]`
###### **Options:**
* `-a`, `--abi-file ` — The path to a json abi file
* `--contract-name ` — The name of the contract
* `-b`, `--blockchain ` — Name or ID of the contract network
* `-r`, `--rpc-url ` — The rpc url to use if the network id used is unsupported by our hypersync
* `-s`, `--start-block ` — The start block to use on this network
* `--single-contract` — If selected, prompt will not ask for additional contracts/addresses/chains
* `--all-events` — If selected, prompt will not ask to confirm selection of events on a contract
## `envio init template`
Initialize Evm indexer from an example template
**Usage:** `envio init template [OPTIONS]`
###### **Options:**
* `-t`, `--template ` — Name of the template to be used in initialization
Possible values: `greeter`, `erc20`, `feature-external-calls`, `feature-factory`
## `envio init svm`
Initialization option for creating Svm indexer
**Usage:** `envio init svm [COMMAND]`
###### **Subcommands:**
* `template` — Initialize Svm indexer from an example template
## `envio init svm template`
Initialize Svm indexer from an example template
**Usage:** `envio init svm template [OPTIONS]`
###### **Options:**
* `-t`, `--template ` — Name of the template to be used in initialization
Possible values: `feature-block-handler`
## `envio init fuel`
Initialization option for creating Fuel indexer
**Usage:** `envio init fuel [COMMAND]`
###### **Subcommands:**
* `contract-import` — Initialize Fuel indexer by importing config from a contract for a given chain
* `template` — Initialize Fuel indexer from an example template
## `envio init fuel contract-import`
Initialize Fuel indexer by importing config from a contract for a given chain
**Usage:** `envio init fuel contract-import [OPTIONS] [COMMAND]`
###### **Subcommands:**
* `local` — Initialize from a local json ABI file
###### **Options:**
* `-c`, `--contract-address ` — Contract address to generate the config from
* `--single-contract` — If selected, prompt will not ask for additional contracts/addresses/chains
* `--all-events` — If selected, prompt will not ask to confirm selection of events on a contract
## `envio init fuel contract-import local`
Initialize from a local json ABI file
**Usage:** `envio init fuel contract-import local [OPTIONS]`
###### **Options:**
* `-a`, `--abi-file ` — The path to a json abi file
* `--contract-name ` — The name of the contract
* `-b`, `--blockchain ` — Which Fuel network to use
Possible values: `mainnet`, `testnet`
* `--single-contract` — If selected, prompt will not ask for additional contracts/addresses/chains
* `--all-events` — If selected, prompt will not ask to confirm selection of events on a contract
## `envio init fuel template`
Initialize Fuel indexer from an example template
**Usage:** `envio init fuel template [OPTIONS]`
###### **Options:**
* `-t`, `--template ` — Name of the template to be used in initialization
Possible values: `greeter`
## `envio dev`
Development commands for starting, stopping, and restarting the indexer. Runs codegen automatically before launching
**Usage:** `envio dev [OPTIONS]`
###### **Options:**
* `-r`, `--restart` — Force restart: clear the database and re-index from scratch. Required when config/schema/ABI changes are incompatible with the existing indexer state
## `envio stop`
Stop the local environment - delete the database and stop all processes (including Docker) for the current directory
**Usage:** `envio stop`
## `envio codegen`
Generate indexing code from user-defined configuration & schema files
**Usage:** `envio codegen`
## `envio local`
Prepare local environment for envio testing
**Usage:** `envio local `
###### **Subcommands:**
* `docker` — Local Envio environment commands
* `db-migrate` — Local Envio database commands
## `envio local docker`
Local Envio environment commands
**Usage:** `envio local docker `
###### **Subcommands:**
* `up` — Start Docker containers (Postgres + Hasura) for local environment
* `down` — Stop and remove Docker containers for local environment
## `envio local docker up`
Start Docker containers (Postgres + Hasura) for local environment
**Usage:** `envio local docker up`
## `envio local docker down`
Stop and remove Docker containers for local environment
**Usage:** `envio local docker down`
## `envio local db-migrate`
Local Envio database commands
**Usage:** `envio local db-migrate `
###### **Subcommands:**
* `up` — Migrate latest schema to database
* `down` — Drop database schema
* `setup` — Setup database by dropping schema and then running migrations
## `envio local db-migrate up`
Migrate latest schema to database
**Usage:** `envio local db-migrate up`
## `envio local db-migrate down`
Drop database schema
**Usage:** `envio local db-migrate down`
## `envio local db-migrate setup`
Setup database by dropping schema and then running migrations
**Usage:** `envio local db-migrate setup`
## `envio start`
Start the indexer. Runs codegen automatically before launching so the on-disk types stay in sync with `config.yaml` and `schema.graphql`
**Usage:** `envio start [OPTIONS]`
###### **Options:**
* `-r`, `--restart` — Clear your database and restart indexing from scratch
## `envio metrics`
Fetch raw Prometheus metrics from the running indexer's /metrics endpoint
**Usage:** `envio metrics [COMMAND]`
###### **Subcommands:**
* `runtime` — Fetch runtime metrics from the running indexer's /metrics/runtime endpoint
## `envio metrics runtime`
Fetch runtime metrics from the running indexer's /metrics/runtime endpoint
**Usage:** `envio metrics runtime`
## `envio skills`
Manage Envio-provided Claude Code skills under `.claude/skills/`
**Usage:** `envio skills `
###### **Subcommands:**
* `update` — Re-extract every skill shipped by this CLI version, overwriting the matching directories under `/.claude/skills/`. Skills not shipped by envio are left untouched
## `envio skills update`
Re-extract every skill shipped by this CLI version, overwriting the matching directories under `/.claude/skills/`. Skills not shipped by envio are left untouched
**Usage:** `envio skills update`
## `envio tools`
Tools for people and AI agents (search-docs, fetch-docs). Run `envio tools help` for details
**Usage:** `envio tools `
###### **Subcommands:**
* `search-docs` — Full-text search over Envio docs; prints matching titles, URLs, and snippets. Pair with `fetch-docs` to read a hit in full
* `fetch-docs` — Print the full markdown of a docs page by URL. Use a URL returned by `search-docs`
## `envio tools search-docs`
Full-text search over Envio docs; prints matching titles, URLs, and snippets. Pair with `fetch-docs` to read a hit in full
**Usage:** `envio tools search-docs `
###### **Arguments:**
* `` — The search query
## `envio tools fetch-docs`
Print the full markdown of a docs page by URL. Use a URL returned by `search-docs`
**Usage:** `envio tools fetch-docs `
###### **Arguments:**
* `` — The full URL of the documentation page to fetch
## `envio config`
Inspect the indexer config
**Usage:** `envio config `
###### **Subcommands:**
* `view` — Print the resolved indexer config as JSON
## `envio config view`
Print the resolved indexer config as JSON
**Usage:** `envio config view`
---
# Configuration File (config.yaml)
> Learn how to configure HyperIndex with contracts, events, networks, and advanced indexing options.
The `config.yaml` file defines your indexer's behavior, including which blockchain events to index, contract addresses, which chains to index, and various advanced indexing options. It is a crucial step in configuring your HyperIndex setup.
:::tip
Whenever you make changes in `config.yaml` that should affect generated types (e.g. adding events, contracts, or chains), run `pnpm codegen` to regenerate types and code for your event handlers.
:::
---
## Example
This is a basic ERC-20 `config.yaml` — it's enough on its own to get an indexer running across Ethereum Mainnet and Gnosis, tracking `Approval` and `Transfer` events on the UNI token.
```yaml
# yaml-language-server: $schema=./node_modules/envio/evm.schema.json
name: erc20-indexer
description: ERC-20 Indexer
contracts:
- name: ERC20
events:
- event: "Approval(address indexed owner, address indexed spender, uint256 value)"
- event: "Transfer(address indexed from, address indexed to, uint256 value)"
chains:
- id: 1 # Ethereum Mainnet
start_block: 0
contracts:
- name: ERC20
address: "0x1f9840a85d5aF5bf1D1762F925BDADdC4201F984" # UNI
- id: 100 # Gnosis Mainnet
start_block: 0
contracts:
- name: ERC20
address: "0x4537e328Bf7e4eFA29D05CAeA260D7fE26af9D74" # UNI
```
**Next steps:**
- Define how each event updates your data → [`src/handlers`](./event-handlers.mdx)
- Shape the entities you'll query → [`schema.graphql`](./schema-file.md)
- Need more than the basics? Keep reading for the full set of configuration options below.
---
## Contracts Definition
The top-level `contracts` block defines each contract once — its events and per-event options. Each chain then references these contracts by `name` and supplies chain-specific values like the on-chain address (see [Contracts (per chain)](#contracts-per-chain)).
```yaml
contracts:
- name: Greeter
events:
- event: "NewGreeting(address user, string greeting)"
- event: "ClearGreeting(address user)"
chains:
- id: 1
start_block: 0
contracts:
- name: Greeter
address: "0x9D02A17dE4E68545d3a58D3a20BbBE0399E05c9c"
```
### Events Selection
The recommended way to declare events is by their **human-readable signature** directly under `events`:
```yaml
contracts:
- name: Greeter
events:
- event: "NewGreeting(address user, string greeting)"
- event: "ClearGreeting(address user)"
```
Only the events listed here are indexed. To stop indexing an event, remove its entry.
#### Custom Event Names
You can assign custom names to events in `config.yaml`. This is handy when
two events share the same name but have different signatures, or when you want
a more descriptive name in your Envio project.
```yaml
events:
- event: Assigned(address indexed recipientId, uint256 amount, address token)
- event: Assigned(address indexed recipientId, uint256 amount, address token, address sender)
name: AssignedWithSender
```
#### Using an ABI File
If you'd rather reference a JSON ABI file (e.g. when you have one already and want to pick a subset of events from it), use `abi_file_path` and refer to events by name:
```yaml
contracts:
- name: Greeter
abi_file_path: ./abis/greeter.json
events:
- event: NewGreeting # signature comes from the ABI file
```
Event signatures are still the recommended default — they keep `config.yaml` self-contained and easier to review.
### Field Selection
To improve indexing performance and reduce credits usage, the `block` and `transaction` fields on events contain only a subset of the fields available on the blockchain.
To access fields that are not provided by default, specify them using the `field_selection` option for your event:
```yaml
events:
- event: "Assigned(address indexed user, uint256 amount)"
field_selection:
transaction_fields:
- transactionIndex
block_fields:
- timestamp
```
See all possible options in the [Config File Reference](/docs/HyperIndex/config-schema-reference#fieldselection) or use IDE autocomplete for your help.
#### Global Field Selection
You can also specify fields globally for all events in the root of the config file:
```yaml
field_selection:
transaction_fields:
- hash
- gasUsed
block_fields:
- parentHash
```
Try to use this option sparingly as it can cause redundant Data Source calls and increased credits usage.
---
## Chains
Everything under the top-level `chains` field configures the networks your indexer connects to — data sources, start/end blocks, reorg behavior, and [multichain indexing](/docs/HyperIndex/multichain-indexing) semantics.
### RPC
The `rpc` option configures an RPC data source per chain. It supports multiple URLs with explicit roles via the `for` field:
- `sync` – use this RPC for historical sync.
- `realtime` – use this RPC for low-latency head tracking once the indexer enters realtime mode. WebSocket endpoints (`wss://...`) are supported here.
- `fallback` – use as a fallback when the primary source is unavailable.
:::tip Optional for HyperSync chains
For chains supported by [HyperSync](../Advanced/hypersync.md), `rpc` is **not required** — HyperSync is used as the primary data source out of the box. You can still add an RPC entry as a fallback for extra reliability. For chains without HyperSync, `rpc` is required and acts as the primary data source.
:::
```yaml
chains:
- id: 1
rpc:
- url: https://eth-mainnet.your-rpc-provider.com
for: sync
- url: wss://eth-mainnet.your-rpc-provider.com
for: realtime
- url: https://fallback.example.com
for: fallback
```
A short form is also supported when you only need a single RPC URL:
```yaml
chains:
- id: 1
rpc: https://eth-mainnet.your-rpc-provider.com
```
After switching to a fallback source, HyperIndex automatically attempts to recover to the primary source 60 seconds later.
See the [Rpc reference](/docs/HyperIndex/config-schema-reference#def-rpc) for advanced tuning options such as `initial_block_interval`, `polling_interval`, `query_timeout_millis`, and backoff parameters.
#### Custom Headers
Since v3.3, `headers` sets custom HTTP headers on an RPC entry — for endpoints gated behind an auth header rather than a key in the URL. Values support `${ENV_VAR}` interpolation:
```yaml
chains:
- id: 1
rpc:
- url: https://eth-mainnet.your-rpc-provider.com
for: sync
headers:
Authorization: "Bearer ${RPC_API_KEY}"
```
See [Custom HTTP Headers](/docs/HyperIndex/rpc-sync#custom-http-headers) for details.
### WebSocket Height Streaming
Pair `wss://` URLs with `for: realtime` to improve head latency by streaming new block heights over a WebSocket connection.
```yaml
chains:
- id: 1
rpc:
url: ${ENVIO_RPC_ENDPOINT}
ws: ${ENVIO_WS_ENDPOINT}
for: realtime
```
:::tip Experimental
WebSocket support for RPC sources is experimental. Please open a GitHub issue if you hit any problems.
:::
### Start Block
Set `start_block` on a chain to control the block at which the indexer begins ingesting data. Setting it to `0` is a safe default for HyperSync — it will automatically skip ahead to the first block that contains data for your configured contracts.
```yaml
chains:
- id: 1
start_block: 0 # HyperSync will fast-forward to the first relevant block
contracts:
- name: Greeter
address: "0x9D02A17dE4E68545d3a58D3a20BbBE0399E05c9c"
```
For finer-grained control, see [Per-Contract Start Block Override](#per-contract-start-block-override) and the per-event start block tip below it.
### End Block
Set `end_block` on a chain to stop indexing once that block is reached. Useful for backfills with a known cutoff or one-off snapshots.
```yaml
chains:
- id: 1
start_block: 18000000
end_block: 19000000
contracts:
- name: Greeter
address: "0x9D02A17dE4E68545d3a58D3a20BbBE0399E05c9c"
```
:::note Don't use this for tests
If you're capping the range to make a deterministic test run, prefer the built-in [testing framework](./testing.mdx) instead — it lets you pin block ranges or feed synthetic events without spinning up a real indexer.
:::
### Contracts (per chain)
Inside each chain you list the contracts you want to index on that chain — referenced by `name` from your top-level [Contracts Definition](#contracts-definition). The chain-level entry is where you pin the on-chain `address` (or list of addresses) and, optionally, a per-contract `start_block`.
:::note
Addresses can be provided in **checksum** format or in **lowercase**.
Envio accepts both and normalizes them internally.
:::
#### Addresses
**Single address:**
```yaml
chains:
- id: 1
start_block: 0
contracts:
- name: MyContract
address: "0xContractAddress"
```
**Multiple addresses for the same contract:**
```yaml
chains:
- id: 1
start_block: 0
contracts:
- name: MyContract
address:
- "0xAddress1"
- "0xAddress2"
```
:::tip
If using a **proxy contract**, always use the **proxy address**, not the implementation address.
:::
#### Per-Contract Start Block Override
By default, contracts use the chain `start_block`. You can also set a per-contract `start_block` to override it. Handy when:
- Contracts were deployed at different blocks
- You only need data from a contract starting at a specific block
- You want to skip unnecessary historical data for some contracts
- Works nicely with [Dynamic Contract Registration](/docs/HyperIndex/dynamic-contracts)
```yaml
chains:
- id: 1 # ethereum-mainnet
start_block: 18000000 # Default start block for all contracts on this chain
contracts:
- name: ERC20
address:
- "0x1111111111111111111111111111111111111111"
- "0x2222222222222222222222222222222222222222"
start_block: 18500000 # Override for this contract
- name: Greeter
address: "0x9D02A17dE4E68545d3a58D3a20BbBE0399E05c9c"
# Uses chain default (18000000)
```
:::tip Per-event start block
For even finer control, you can also specify a start block per event from your handler using the `where.block.number._gte` filter on `indexer.onEvent`. See [Event Handlers](./event-handlers.mdx) for the full API.
:::
#### Indexer Without Contracts
The `contracts` field is **optional** in V3. You can run an indexer that only uses block handlers (`indexer.onBlock`) without declaring any contracts:
```yaml
name: BlockHandlerOnly
chains:
- id: 1
start_block: 18000000
```
---
## Storage
How indexed data is persisted — which backends to use and whether to keep raw event records around.
### Storage Backends
By default, HyperIndex writes entities to Postgres. In V3 you can additionally enable **ClickHouse** as a second storage backend (experimental):
```yaml
storage:
postgres: true
clickhouse: true
```
When both backends are enabled, you must route each entity explicitly via the `@storage` directive in `schema.graphql`:
```graphql
# Stored in both Postgres and ClickHouse
type Transfer @storage(postgres: true, clickhouse: true) {
id: ID!
from: String!
to: String!
value: BigInt!
}
# Stored only in ClickHouse
type Snapshot @storage(clickhouse: true) {
id: ID!
blockNumber: BigInt!
}
```
Alternatively, mark one or both backends as `default` so every entity is written there automatically, and reach for the `@storage` directive only when you need to override routing for a specific entity:
```yaml
storage:
postgres:
default: true
clickhouse:
default: true
```
:::note
The `default` storage option was added in HyperIndex v3.2.
:::
`envio dev` automatically spins up a ClickHouse Docker container for local development. For `envio start`, provide the connection via the environment variables `ENVIO_CLICKHOUSE_HOST`, `ENVIO_CLICKHOUSE_DATABASE`, `ENVIO_CLICKHOUSE_USERNAME`, and `ENVIO_CLICKHOUSE_PASSWORD`.
:::warning
Do not run multiple indexers writing to the same ClickHouse database at the same time.
:::
Envio Cloud currently supports ClickHouse on the Dedicated Plan.
Since v3.4, the `@storage` directive's `clickhouse` argument also accepts an options object for per-entity table tuning — partitioning, sorting key and TTL. See [Per-Entity ClickHouse Tuning](/docs/HyperIndex/schema#per-entity-clickhouse-tuning).
### Column Name Format
Set `column_name_format: snake_case` on a backend to store column names in `snake_case` while keeping the original names in the GraphQL API and your handler types. It works for both Postgres and ClickHouse:
```yaml
storage:
postgres:
column_name_format: snake_case
clickhouse:
column_name_format: snake_case
```
:::note
Added in HyperIndex v3.2.
:::
---
## Address Format
Use `address_format` to control how every address surfaced by the indexer (event fields like `event.srcAddress` / `event.transaction.from`, `chain..addresses`, addresses embedded in entity ids, etc.) is formatted.
```yaml
address_format: lowercase # default: checksum
```
- `checksum` (default) – [EIP-55](https://eips.ethereum.org/EIPS/eip-55) checksummed mixed-case addresses.
- `lowercase` – every address is lowercased globally. Useful when joining against another data source that stores addresses in lowercase, or when you want byte-for-byte deterministic ids without per-handler `.toLowerCase()` calls.
You can still call `.toLowerCase()` ad-hoc inside a handler when you only need a single value lowercased.
---
## Ecosystem
`ecosystem` selects which chain family your indexer targets. EVM is the default and what every example above assumes, but the same `config.yaml` schema is shared with Fuel and Solana (SVM).
```yaml
ecosystem: evm # default — also: fuel, svm
```
Most projects don't set this field explicitly. Use `pnpx envio init fuel` or `pnpx envio init svm` to scaffold non-EVM indexers — the generated `config.yaml` will already have `ecosystem` set correctly. See the [Fuel](../fuel/fuel.md) and [Solana](../solana/solana.md) guides for the ecosystem-specific options.
---
## Environment Variables
Environment variable interpolation is supported anywhere in `config.yaml`, which is useful for keeping RPC URLs, addresses, or chain IDs out of source control.
```yaml
chains:
- id: ${ENVIO_CHAIN_ID:-ethereum-mainnet}
contracts:
- name: Greeter
address: "${ENVIO_GREETER_ADDRESS}"
```
Run your indexer with custom environment variables:
```bash
ENVIO_CHAIN_ID=optimism ENVIO_GREETER_ADDRESS=0xYourContractAddress pnpm dev
```
**Interpolation syntax:**
- `${ENVIO_VAR}` – Use the value of `ENVIO_VAR`
- `${ENVIO_VAR:-default}` – Use `ENVIO_VAR` if set, otherwise use `default`
For more detailed information about environment variables, see our [Environment Variables Guide](./environment-variables).
---
## Advanced
:::note
In ~95% of cases you don't need to touch any of these — the defaults are tuned for the common path. Reach for them only when you have a specific reason.
:::
### Handler File Path
Handlers are auto-discovered from `src/handlers/`. Override the directory with the top-level `handlers` option, or set a per-contract `handler` path when needed:
```yaml
handlers: ./src/my-handlers # optional override of src/handlers
contracts:
- name: Greeter
handler: ./src/GreeterHandler.ts # optional per-contract path
```
### Schema File Path
You can customize the path to the [schema file](./schema-file.md) using the `schema` option:
```yaml
schema: ./path/to/schema.graphql
```
By default, the `schema.graphql` is expected to be in the root directory of your project.
### Block Lag
Set `block_lag` on a chain to keep the indexer a fixed number of blocks behind the chain head. Defaults to `0`.
```yaml
chains:
- id: 1
start_block: 0
block_lag: 5
```
:::warning
Only set `block_lag` if you understand the trade-off — it intentionally trades head latency for additional reorg safety.
:::
### Rollback on Reorg
HyperIndex automatically handles blockchain reorganizations by default. To disable or customize this behavior, set the `rollback_on_reorg` flag in your `config.yaml`:
```yaml
rollback_on_reorg: true # default is true
```
See detailed configuration options [here](./reorgs-support).
### Full Batch Size
Set `full_batch_size` to control how many events are processed in a single batch.
```yaml
full_batch_size: 5000
```
### Raw Events Storage
By default, HyperIndex doesn't store raw event data in the database to optimize performance and reduce storage requirements. However, you can enable this feature for debugging purposes or if you need to access the original event data.
To enable storage of raw events, add the following to your `config.yaml`:
```yaml
raw_events: true
```
When enabled, all indexed events will be stored in the `raw_events` table in the database, which you can view through the Hasura interface. This is particularly useful for:
- Debugging event processing issues
- Verifying that events are being captured correctly
- Creating custom queries against raw blockchain data
Note that enabling this option will increase database storage requirements and may slightly impact indexing performance.
### Skip Chain
Set `skip: true` on a chain to exclude it from indexing and migrations without removing it from your config — handy for temporarily disabling a chain.
```yaml
chains:
- id: 137
skip: true
start_block: 0
```
:::note
Added in HyperIndex v3.1.
:::
---
## Configuration Schema Reference
Explore detailed configuration schema parameters here:
- See the full, deep-linkable reference: [Config Schema Reference](/docs/HyperIndex/config-schema-reference)
:::info For AI/LLM Systems
**Recommended**: Use the [Config Schema Reference](/docs/HyperIndex/config-schema-reference) for programmatic access to schema information. The interactive viewer below is optimized for human users.
:::
import CodeBlock from "@theme/CodeBlock";
import Schema from "@site/static/schemas/config.evm.json";
import JSONSchemaViewer from "@theme/JSONSchemaViewer";
📋 Hierarchical Interactive Schema Explorer(Click to expand - For human reference only)
#### Features
- **Explorer Panel**: The left panel shows all available entities defined in your `schema.graphql` file
- **Query Builder**: The center area is where you write and execute GraphQL queries
- **Results Panel**: The right panel displays query results in JSON format
#### Available Entities
By default, you'll see:
- All entities defined in your `schema.graphql` file
- [`dynamic_contracts`](../Advanced/dynamic-contracts.md) (for dynamically added contracts)
- `raw_events` table (Note: This table is no longer populated by default to improve performance. To enable storage of raw events, add `raw_events: true` to your `config.yaml` file as described in the [Raw Events Storage](/docs/HyperIndex/configuration-file#raw-events-storage) section)
#### Example Query
Try a simple query to test your blockchain indexer:
```graphql
query MyQuery {
User(limit: 5) {
id
latestGreeting
numberOfGreetings
}
}
```
Click the "Play" button to execute the query and see the results.
For more advanced GraphQL query options, see Hasura's [quickstart guide](https://hasura.io/docs/latest/queries/quickstart/).
### Data Tab
The Data tab provides direct access to your database tables and relationships, allowing you to view the actual indexed data.
#### Features
- **Schema Browser**: View all tables in the database (left panel)
- **Table Data**: Examine and browse data within each table
- **Relationship Viewer**: See how different entities are connected
#### Working with Tables
1. Select any table from the "public" schema to view its contents
2. Use the "Browse Rows" tab to see all data in that table
3. Check the "Insert Row" tab to manually add data (useful for testing)
4. View the "Modify" tab to see the table structure
#### Verifying Indexed Data
To confirm your blockchain indexer is working correctly:
1. Check entity tables to ensure they contain the expected data
2. Run the [`_meta` indexing status](/docs/HyperIndex/observability#indexing-status) query to see each chain's latest processed block and confirm the indexer is making progress
## Common Tasks
### Checking Indexing Status
To verify your blockchain indexer is actively processing new blocks:
1. Run the [`_meta` indexing status](/docs/HyperIndex/observability#indexing-status) query to see each chain's latest processed block
2. Monitor those values over time to ensure they're advancing
(Note the TUI is also an easy way to monitor this)
### Troubleshooting Missing Data
If expected data isn't appearing:
1. Check if you've enabled raw events storage (`raw_events: true` in `config.yaml`) and then examine the `raw_events` table to confirm events were captured
2. Verify your event handlers are correctly processing these events
3. Examine your GraphQL queries to ensure they match your schema structure
4. Check console logs for any processing errors
### Resetting Indexed Data
When testing, you may need to reset your database:
1. Stop your indexer
2. Reset your database (refer to the development guide for commands)
3. Restart your indexer to begin processing from the configured start block
## Best Practices
- **Regular Verification**: Periodically check both the API and Data tabs to ensure your blockchain indexer is functioning correctly
- **Query Testing**: Test complex queries in the API tab before implementing them in your application
- **Schema Validation**: Use the Data tab to verify that relationships between entities are correctly established
- **Performance Monitoring**: Watch for tables that grow unusually large, which might indicate inefficient indexing
## Aggregations: local vs hosted (avoid the foot‑gun)
When developing locally with Hasura, you may notice that GraphQL aggregate helpers (for example, count/sum-style aggregations) are available. On Envio Cloud, these aggregate endpoints are intentionally not exposed. Aggregations over large datasets can be very slow and unpredictable in production.
The recommended approach is to compute and store aggregates at indexing time, not at query time. In practice this means maintaining counters, sums, and other rollups in entities as part of your event handlers, and then querying those precomputed values.
### Example: indexing-time aggregation
schema.graphql
```graphql
# singleton; you hardcode the id and load it in and out
type GlobalState {
id: ID! # "global-state"
count: Int!
}
type Token {
id: ID! # incremental number
description: String!
}
```
EventHandler.ts
```typescript
import { indexer } from "envio";
const globalStateId = "global-state";
indexer.onEvent(
{ contract: "NftContract", event: "Mint" },
async ({ event, context }) => {
const globalState = await context.GlobalState.get(globalStateId);
if (!globalState) {
context.log.error("global state doesn't exist");
return;
}
const incrementedTokenId = globalState.count + 1;
context.Token.set({
id: incrementedTokenId,
description: event.params.description,
});
context.GlobalState.set({
...globalState,
count: incrementedTokenId,
});
},
);
```
This pattern scales: you can keep per-entity counters, rolling windows (daily/hourly entities keyed by date), and top-N caches by updating entities as events arrive. Your queries then read these precomputed values directly, avoiding expensive runtime aggregations.
#### Exceptional cases
If runtime aggregate queries are a hard requirement for your use case, please reach out and we can evaluate options for your project on Envio Cloud. Contact us on [Discord](https://discord.gg/envio).
## Disable Hasura for Self-Hosted Blockchain Indexers
Set the `ENVIO_HASURA` environment variable to `false` to disable Hasura integration for self-hosted blockchain indexers.
---
For more information on using GraphQL with your indexed data, refer to the [Hasura GraphQL documentation](https://hasura.io/docs/latest/queries/overview/).
---
# Running The Indexer Locally
> Learn how to start, run, and manage the Envio indexer locally with Docker and Hasura.
## Starting the Indexer
Remember to `cd` into your project directory if you have defined one during `pnpx envio init`.
Before running the Envio CLI command to start the indexer locally, please make sure you have [Docker](https://www.docker.com/products/docker-desktop/) running.
Run the indexer
```bash
pnpm dev
```
This will automatically open up the Hasura dashboard where you can view the data that has been indexed.
Admin-secret / password for local Hasura is `testing`.
## Hot Reload
When running `pnpm dev`, HyperIndex watches for changes to your **event handler** files. When you save a handler file, the indexer will automatically re-process events using your updated logic without needing to restart manually.
**What triggers a hot reload:**
- Changes to event handler files (e.g., `src/handlers`)
**What requires a full restart:**
- Changes to `config.yaml` (adding contracts, events, or chains)
- Changes to `schema.graphql` (adding or modifying entities)
- Changes to ABIs
For config or schema changes, restart the indexer:
```bash
pnpm envio start -r
```
:::tip
If hot reload doesn't seem to be picking up your handler changes, try saving the file again or check the terminal output for compilation errors that may be preventing the reload.
:::
## Avoiding Full Re-indexing During Development
By default, stopping and restarting the indexer will re-index from the start block. This can be slow for large datasets. Here are strategies to iterate faster:
### Use the testing framework
The fastest way to validate handler logic is to skip syncing entirely. Envio's built-in [testing framework](./testing) lets you mock events, simulate database state, and assert handler behavior — all without connecting to a blockchain or waiting for blocks to sync.
```bash
pnpm test
```
This is ideal for iterating on handler logic, catching bugs early, and running tests in CI. See the [Testing guide](./testing) for setup and examples.
### Use a recent start block
During development, temporarily set `start_block` in your `config.yaml` to a recent block number. This lets you test handler logic against fresh data without waiting for a full historical sync:
```yaml
- id: 1 # Ethereum mainnet
start_block: 19000000 # Recent block for faster dev iteration
```
Remember to set it back to the correct start block before deploying to production.
### Force a clean re-index
If you need to re-index from scratch (e.g., after schema changes or to clear corrupted state):
```bash
pnpm envio start -r # Restarts and re-indexes from start_block
```
## Stopping the Indexer
To delete the docker images used for the local development environment, run
```bash
pnpm envio stop
```
## What next?
## Once you have successfully run the indexer locally, you can deploy the indexer onto Envio's [Envio Cloud](./hosted-service).
---
# Entities Schema (schema.graphql)
> Learn how to define GraphQL schemas, manage entities, and handle different types in HyperIndex.
The **`schema.graphql`** file defines the data model for your HyperIndex indexer. Each entity type defined in this schema corresponds directly to a database table, with your event handlers responsible for creating and updating the records. HyperIndex automatically generates a GraphQL API based on these entity types, allowing easy access to the indexed data.
---
## Defining Entity Types
Entities in your schema are defined as GraphQL object types:
**Example:**
```graphql
type User {
id: ID!
greetings: [String!]!
latestGreeting: String!
numberOfGreetings: Int!
}
```
### Requirements:
- Every entity **must** have a unique `id` field, using one of these scalar types:
- `ID!`, `String!`, `Int!`, or `BigInt!`
- The `id` field must be non-nullable, must not be a list, and cannot be a [`@derivedFrom`](#relationships-one-to-many-derivedfrom) field.
### Numeric Entity IDs
`ID` is the usual choice and behaves as a string. Since v3.5, you can also key an entity on `Int` or `BigInt`, which is a better fit when the identifier is genuinely a number — a block number, an auction id, a sequential position:
```graphql
type Auction {
id: BigInt! # the on-chain auction id, not a stringified copy of it
seller: String!
bids: [Bid!]! @derivedFrom(field: "auction")
}
type Bid {
id: ID!
auction: Auction! # inferred as BigInt to match Auction.id
amount: BigInt!
}
```
Relationship fields adopt the referenced entity's id type automatically, so `Bid.auction` above is typed `bigint` in your handlers rather than `string`. You don't declare the foreign key type — keep the two sides in sync by changing the referenced entity's `id`.
---
## Scalar Types
Scalar types represent basic data types and map directly to JavaScript, TypeScript, or ReScript types.
| **GraphQL Scalar** | **Description** | **JavaScript/TypeScript** | **ReScript** |
| ------------------ | -------------------------------------------- | ------------------------- | -------------- |
| `ID` | Unique identifier | `string` | `string` |
| `String` | UTF-8 character sequence | `string` | `string` |
| `Int` | Signed 32-bit integer | `number` | `int` |
| `Float` | Signed floating-point number | `number` | `float` |
| `Boolean` | `true` or `false` | `boolean` | `bool` |
| `Bytes` | UTF-8 character sequence (hex prefixed `0x`) | `string` | `string` |
| `BigInt` | Signed integer (`int256` in Solidity) | `bigint` | `bigint` |
| `BigDecimal` | Arbitrary-size floating-point | `BigDecimal` (imported) | `BigDecimal.t` |
| `Timestamp` | Timestamp with timezone | `Date` | `Js.Date.t` |
| `Json` | JSON object | `Json` | `Js.Json.t` |
Learn more about GraphQL scalars [here](https://graphql.org/learn/).
---
## Working with BigDecimal
The `BigDecimal` scalar type in HyperIndex is based on the [bignumber.js](https://mikemcl.github.io/bignumber.js/) library, which provides arbitrary-precision decimal arithmetic. This is essential for financial calculations and handling numeric values that exceed JavaScript's native number precision.
### Importing BigDecimal
```typescript
// JavaScript/TypeScript
import { BigDecimal } from "envio";
// ReScript
open BigDecimal;
```
### Creating BigDecimal Instances
```typescript
// From string (recommended for precision)
const price = new BigDecimal("123.456789");
// From number (may lose precision for very large values)
const amount = new BigDecimal(123.45);
// From other BigDecimal
const copy = new BigDecimal(price);
```
### Arithmetic Operations
BigDecimal instances are immutable. Operations return new BigDecimal instances:
```typescript
// Basic arithmetic
const a = new BigDecimal("123.45");
const b = new BigDecimal("67.89");
const sum = a.plus(b); // 191.34
const difference = a.minus(b); // 55.56
const product = a.times(b); // 8,381.03
const quotient = a.div(b); // 1.81839...
// Power
const squared = a.pow(2); // 15,239.9025
// Square root
const root = a.sqrt(); // 11.11...
// Absolute value
const abs = new BigDecimal("-123.45").abs(); // 123.45
```
### Comparison Methods
```typescript
const x = new BigDecimal("10.5");
const y = new BigDecimal("10.5");
const z = new BigDecimal("9.9");
x.eq(y); // true (equal)
x.gt(z); // true (greater than)
x.gte(y); // true (greater than or equal)
x.lt(z); // false (less than)
x.lte(y); // true (less than or equal)
// Check for special values
x.isZero(); // false
x.isPositive(); // true
x.isNegative(); // false
x.isFinite(); // true
```
### Rounding and Formatting
```typescript
const value = new BigDecimal("123.456789");
// Get with specific decimal places
value.dp(2); // 123.46 (rounded)
value.dp(2, 1); // 123.45 (rounded down)
// Format as string
value.toString(); // "123.456789"
value.toFixed(2); // "123.46"
value.toExponential(2); // "1.23e+2"
value.toPrecision(5); // "123.46"
```
### Working with Schema-Defined BigDecimal Fields
When you've defined a `BigDecimal` field in your schema:
```graphql
type TokenPair {
id: ID!
name: String!
price: BigDecimal!
volume: BigDecimal!
}
```
You can use it in your handlers:
```typescript
// In your event handler
context.TokenPair.set({
id: event.params.pairId,
name: event.params.name,
price: new BigDecimal(event.params.price),
volume: new BigDecimal("0"), // Start with zero volume
});
// Updating a field
const tokenPair = await context.TokenPair.get(pairId);
if (tokenPair) {
const newVolume = tokenPair.volume.plus(new BigDecimal(tradeAmount));
context.TokenPair.set({
...tokenPair,
volume: newVolume,
});
}
```
### Example: Financial Calculation
```typescript
function calculateFee(amount: BigDecimal, feeRate: BigDecimal): BigDecimal {
// Calculate fee with proper rounding
return amount.times(feeRate).dp(2);
}
const tradeAmount = new BigDecimal("1250.75");
const feeRate = new BigDecimal("0.0025"); // 0.25%
const fee = calculateFee(tradeAmount, feeRate); // 3.13
```
### Best Practices for BigDecimal
1. **Always use strings for initialization** when precision matters:
```typescript
// Preferred
const value = new BigDecimal("123.456789");
// May lose precision
const value = new BigDecimal(123.456789);
```
2. **Set precision explicitly** when doing division:
```typescript
// Set to 8 decimal places for crypto prices
const price = totalValue.div(tokenAmount).dp(8);
```
3. **Handle rounding appropriately** for financial calculations:
```typescript
// Round down (floor) for user-favorable calculations
const userReceives = amount.dp(2, 1); // ROUND_DOWN
// Round up (ceil) for protocol-favorable calculations
const protocolFee = amount.dp(2, 0); // ROUND_UP
```
4. **Compare with equals method** instead of `==` or `===`:
```typescript
// Correct
if (value.eq(new BigDecimal(0))) {
/* ... */
}
// Incorrect - compares object references
if (value === new BigDecimal(0)) {
/* ... */
}
```
5. **Chain operations carefully**, remembering that each operation returns a new instance:
```typescript
// Calculate (a + b) * c with proper precision
const result = a.plus(b).times(c).dp(8);
```
---
## Enum Types
Enums allow fields to accept only a predefined set of values.
**Example:**
```graphql
enum AccountType {
ADMIN
USER
}
type User {
id: ID!
balance: Int!
accountType: AccountType!
}
```
Enums translate to string unions (TypeScript/JavaScript) or polymorphic variants (ReScript):
**TypeScript Example:**
```typescript
import { type Enum } from "envio";
let user = {
id: event.params.id,
balance: event.params.balance,
accountType: "USER" satisfies Enum<"AccountType">, // enum as string
};
```
**ReScript Example:**
```rescript
let user: Types.userEntity = {
id: event.params.id,
balance: event.params.balance,
accountType: #USER, // polymorphic variant
};
```
---
## Relationships: One-to-Many (`@derivedFrom`)
Define relationships between entities using the `@derivedFrom` directive, known as **reverse lookups**.
**Example:**
```graphql
type NftCollection {
id: ID!
contractAddress: Bytes!
name: String!
symbol: String!
maxSupply: BigInt!
currentSupply: Int!
tokens: [Token!]! @derivedFrom(field: "collection")
}
type Token {
id: ID!
tokenId: BigInt!
collection: NftCollection!
owner: User!
}
```
- The `tokens` field in `NftCollection` is a virtual field, populated automatically when querying the API.
- Set relationships in your handlers by assigning `
3. Choose your preferred language (TypeScript, JavaScript, or ReScript):
## Step 2: Import the USDC Token Contract
1. Select **Contract Import** → **Block Explorer** → **Base**
2. Enter the USDC token contract address on Base:
```
0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913
```
[View on BaseScan](https://basescan.org/address/0x833589fcd6edb6e08f4c7c32d4f71b54bda02913)
3. Select the `Transfer` event:
- Navigate using arrow keys (↑↓)
- Press spacebar to select the event
> **Tip:** You can select multiple events to index simultaneously if needed.
4. When finished adding contracts, select **I'm finished**
## Step 3: Start Your Indexer
1. If you have any running indexers, stop them first:
```bash
pnpm envio stop
```
> **Note:** You can skip this step if this is your first time running an indexer.
2. Start your new indexer:
```bash
pnpm dev
```
This command:
- Starts the required Docker containers
- Sets up your database
- Launches the indexing process
- Opens the Hasura GraphQL interface
## Step 4: Understanding the Generated Code
Let's examine the key files that Envio generated:
### 1. `config.yaml`
This configuration file defines:
- Network to index (Base)
- Starting block for indexing
- Contract address and ABI details
- Events to track (Transfer)
### 2. `schema.graphql`
This schema defines the data structures for the Transfer event:
- Entity types based on event data
- Field types for sender, receiver, and amount
- Any relationships between entities
### 3. `src/handlers`
This directory contains the business logic for processing events:
- Functions that execute when Transfer events are detected
- Data transformation and storage logic
- Entity creation and relationship management
## Step 5: Exploring Your Indexed Data
Now you can interact with your indexed USDC transfer data:
### Accessing Hasura
1. Open Hasura at [http://localhost:8080](http://localhost:8080)
2. When prompted, enter the admin password: `testing`
### Monitoring Indexing Progress
1. Click the **Data** tab in the top navigation
2. Find the `_events_sync_state` table to check indexing progress
3. Observe which blocks are currently being processed
> **Note:** Thanks to Envio's [HyperSync](https://docs.envio.dev/docs/hypersync), you can index millions of USDC transfers in just minutes rather than hours or days with traditional methods.
### Querying Indexed Events
1. Click the **API** tab
2. Construct a GraphQL query to explore your data
Here's an example query to fetch the 10 largest USDC transfers:
```graphql
query LargestTransfers {
FiatTokenV2_2_Transfer(limit: 10, order_by: { value: desc }) {
from
to
value
blockTimestamp
}
}
```
3. Click the **Play** button to execute your query
## Conclusion
Congratulations! You've successfully created an indexer for USDC token transfers on Base. In just a few minutes, you've indexed over 3.6 million transfer events and can now query this data in real-time.
### What You've Learned
- How to initialize an indexer using Envio's contract import feature
- How to index ERC20 token transfers on the Base network
- How to query and analyze token transfer data using GraphQL
### Next Steps
- Try customizing the event handlers to add additional logic
- Create aggregated statistics about token transfers
- Add more tokens or events to your indexer
- Deploy your indexer to Envio Cloud
For more tutorials and advanced features, check out our [documentation](https://docs.envio.dev) or watch our [video walkthrough](https://www.youtube.com/watch?v=e1xznmKBLa8) on YouTube.
---
# Indexing Sway Farm on the Fuel Network
> Learn how to index Sway Farm on Fuel and track player events with Envio.
HyperIndex supports any EVM-compatible blockchain and the [Fuel](https://fuel.network/) Network.
Blockchain indexers are vital to the success of any dApp. In this tutorial, we will create an Envio indexer for the Fuel dApp [Sway Farm](https://swayfarm.xyz/) step by step.
Sway Farm is a simple farming game and for the sake of a real-world example, let's create the indexer for a leaderboard of all farmers 🧑🌾
## About Fuel
[Fuel](https://fuel.network/) is an operating system purpose-built for Ethereum rollups. Fuel's unique architecture allows rollups to solve for PSI (parallelization, state minimized execution, interoperability). Powered by the FuelVM, Fuel aims to expand Ethereum's capability set without compromising security or decentralization.
[Website](https://fuel.network/) | [X](https://twitter.com/fuel_network?lang=en) | [Discord](https://discord.com/invite/xfpK4Pe)
## Prerequisites
### Environment tooling
- **[Node.js](https://nodejs.org/en/download/current)** _(v22 or newer recommended)_
- **[pnpm](https://pnpm.io/installation)** _(recommended but not required)_
- **[Docker Desktop](https://www.docker.com/products/docker-desktop/)** _(required to run the Envio indexer locally)_
> **Note:** Docker is specifically required to run your blockchain indexer locally. You can skip Docker installation if you plan only to use Envio Cloud.
## Initialize the project
Now that you have installed the prerequisite packages let's begin the practical steps of setting up the indexer.
Open your terminal in an empty directory and initialize a new indexer by running the command:
```bash
pnpx envio init
```
In the following prompt, choose the directory where you want to set up your project. The default is the current directory, but in the tutorial, I'll use the indexer name:
```bash
? Specify a folder name (ENTER to skip): sway-farm-indexer
```
Then, choose a language of your choice for the event handlers. TypeScript is the most popular one, so we'll stick with it:
```bash
? Which language would you like to use?
JavaScript
> TypeScript
ReScript
[↑↓ to move, enter to select, type to filter]
```
Next, we have the new prompt for a blockchain ecosystem. Previously Envio supported only EVM, but now it's possible to choose between `Evm`, `Fuel` and other VMs in the future:
```bash
? Choose blockchain ecosystem
Evm
> Fuel
[↑↓ to move, enter to select, type to filter]
```
In the following prompt, you can choose an initialization option. There's a Greeter template for Fuel, which is an excellent way to learn more about HyperIndex. But since we have an existing contract, the `Contract Import` option is the best way to create an indexer:
```bash
? Choose an initialization option
Template
> Contract Import
[↑↓ to move, enter to select, type to filter]
```
> A separate [Tutorial](./greeter-tutorial) page provides more details about the `Greeter` template.
Next it'll ask us for an ABI file. You can find it in the `./out/debug` directory after building your [Sway](https://docs.fuel.network/docs/sway/) contract with `forc build`:
```bash
? What is the path to your json abi file? ./sway-farm/contract/out/debug/contract-abi.json
```
After the ABI file is provided, Envio parses all possible events you can use for indexing:
```bash
? Which events would you like to index?
> [x] NewPlayer
[x] PlantSeed
[x] SellItem
[x] InvalidError
[x] Harvest
[x] BuySeeds
[x] LevelUp
[↑↓ to move, space to select one, → to all, ← to none, type to filter]
```
Let's select the events we want to index. I opened the code of the [contract file](https://github.com/FuelLabs/sway-farm/blob/47e3ed5a91593ebcf8d2c67ae6fad41d9954c8a8/contract/src/abi_structs.sw#L365-L406) and realized that for a leaderboard we need only events which update player information. Hence, I left only `NewPlayer`, `LevelUp`, and `SellItem` selected in the list. We'd want to index more events in real life, but this is enough for the tutorial.
```bash
? Which events would you like to index?
> [x] NewPlayer
[ ] PlantSeed
[x] SellItem
[ ] InvalidError
[ ] Harvest
[ ] BuySeeds
[x] LevelUp
[↑↓ to move, space to select one, → to all, ← to none, type to filter]
```
> 📖 For the tutorial we only need to index `LOG_DATA` receipts, but you can also index `Mint`, `Burn`, `Transfer` and `Call` receipts. Read more about [Supported Event Types](/docs/HyperIndex/fuel#features-supported-on-fuel).
Just a few simple questions left. Let's call our contract `SwayFarm`:
```bash
? What is the name of this contract? SwayFarm
```
Set an address for the deployed contract:
```bash
? What is the address of the contract? 0xf5b08689ada97df7fd2fbd67bee7dea6d219f117c1dc9345245da16fe4e99111
[Use the proxy address if your abi is a proxy implementation]
```
Finish the initialization process:
```bash
? Would you like to add another contract?
> I'm finished
Add a new address for same contract on same network
Add a new contract (with a different ABI)
[Current contract: SwayFarm, on network: Fuel]
```
If you see the following line, it means we are already halfway through 🙌
```
Please run `cd sway-farm-indexer` to run the rest of the envio commands
```
Let's open the indexer in an IDE and start adjusting it for our farm 🍅
## Walk through initialized indexer
At this point, we should already have a working indexer. You can start it by running `pnpm dev`, which we cover in more detail later in the tutorial.
Everything is configured by modifying the 3 files below. Let's walk through each of them.
- config.yaml [`Guide`](configuration-file)
- schema.graphql [`Guide`](../Guides/schema-file.md)
- EventHandlers.\* [`Guide`](../Guides/event-handlers.mdx)
> (\* depending on the language chosen for the indexer)
### `config.yaml`
The `config.yaml` outlines the specifications for the indexer, including details such as network and contract specifications and the event information to be used in the indexing process.
```yaml
name: sway-farm-indexer
ecosystem: fuel
chains:
- id: 0
start_block: 0
contracts:
- name: SwayFarm
address:
- "0xf5b08689ada97df7fd2fbd67bee7dea6d219f117c1dc9345245da16fe4e99111"
abi_file_path: abis/swayfarm-abi.json
events:
- name: SellItem
logId: "11192939610819626128"
- name: LevelUp
logId: "9956391856148830557"
- name: NewPlayer
logId: "169340015036328252"
```
In the tutorial, we don't need to adjust it in any way. But later you can modify the file and add more events for indexing.
As a nice to have, you can use a [Sway](https://docs.fuel.network/docs/sway/) struct name without specifying a `logId`, like this:
```yaml
- name: SellItem
- name: LevelUp
- name: NewPlayer
```
### `schema.graphql`
The `schema.graphql` file serves as a representation of your application's data model. It defines entity types that directly correspond to database tables, and the event handlers you create are responsible for creating and updating records within those tables. Additionally, the GraphQL API is automatically generated based on the entity types specified in the `schema.graphql` file, to allow access to the indexed data.
> 🧠 A separate [Guide](../Guides/schema-file.md) page provides more details about the `schema.graphql` file.
For the leaderboard, we need only one entity representing the player. Let's create it:
```graphql
type Player {
id: ID!
farmingSkill: BigInt!
totalValueSold: BigInt!
}
```
We will use the user address as an ID. The fields `farmingSkill` and `totalValueSold` are `u64` in Sway, so to safely map them to JavaScript value, we'll use `BigInt`.
### `EventHandlers.ts`
The event handlers generated by contract import are quite simple and only add an entity to a DB when a related event is indexed.
```typescript
/*
* Please refer to https://docs.envio.dev for a thorough guide on all Envio indexer features
*/
import { indexer, type Entity } from "envio";
indexer.onEvent(
{ contract: "SwayFarm", event: "SellItem" },
async ({ event, context }) => {
const entity: Entity<"SwayFarm_SellItem"> = {
id: `${event.chainId}_${event.block.height}_${event.logIndex}`,
};
context.SwayFarm_SellItem.set(entity);
},
);
```
Let's modify the handlers to update the `Player` entity instead. But before we start, we need to run `pnpm codegen` to generate utility code and types for the `Player` entity we've added.
```bash
pnpm codegen
```
It's time for a little bit of coding. The indexer is very simple; it requires us only to pass event data to an entity.
```typescript
import { indexer } from "envio";
/**
Registers a handler that processes NewPlayer event
on the SwayFarm contract and stores the players in the DB
*/
indexer.onEvent(
{ contract: "SwayFarm", event: "NewPlayer" },
async ({ event, context }) => {
// Set the Player entity in the DB with the initial values
context.Player.set({
// The address in Sway is a union type of user Address and ContractID. Envio supports most of the Sway types, and the address value was decoded as a discriminated union 100% typesafe
id: event.params.address.payload.bits,
// Initial values taken from the contract logic
farmingSkill: 1n,
totalValueSold: 0n,
});
},
);
indexer.onEvent(
{ contract: "SwayFarm", event: "LevelUp" },
async ({ event, context }) => {
const playerInfo = event.params.player_info;
context.Player.set({
id: event.params.address.payload.bits,
farmingSkill: playerInfo.farming_skill,
totalValueSold: playerInfo.total_value_sold,
});
},
);
indexer.onEvent(
{ contract: "SwayFarm", event: "SellItem" },
async ({ event, context }) => {
const playerInfo = event.params.player_info;
context.Player.set({
id: event.params.address.payload.bits,
farmingSkill: playerInfo.farming_skill,
totalValueSold: playerInfo.total_value_sold,
});
},
);
```
Without overengineering, simply set the player data into the database. What's nice is that whenever your ABI or entities in `graphql.schema` change, Envio regenerates types and shows the compilation error.
> 🧠 You can find the indexer repo created during the tutorial on [GitHub](https://github.com/enviodev/sway-farm-indexer).
## Starting the Indexer
> 📢 Make sure you have docker open
The following commands will start the docker and create databases for indexed data. Make sure to re-run `pnpm dev` if you've made some changes.
```bash
pnpm dev
```
Nice, we indexed `1,721,352` blocks containing `58,784` events in 10 seconds, and they continue coming in.
## View the indexed results
Let's check indexed players on the local Hasura server.
```bash
open http://localhost:8080
```
The Hasura admin-secret / password is `testing`, and the tables can be viewed in the data tab or queried from the playground.
Now, we can easily get the top 5 players, the number of inactive and active players, and the average sold value. What's left is a nice UI for the Sway Farm leaderboard, but that's not the tutorial's topic.
> 🧠 A separate [Guide](./navigating-hasura) page provides more details about navigating Hasura.
## Deploy the indexer to Envio Cloud
Once you have verified that the indexer is working for your contracts, then you are ready to deploy the indexer to Envio Cloud.
Deploying an indexer to Envio Cloud allows you to extract information via graphQL queries into your front-end or some back-end application.
Navigate to the [Envio Cloud](https://envio.dev/app/login) to start deploying your indexer and refer to this [documentation](../Hosted_Service/hosted-service.md) for more information on deploying your indexer.
## What next?
Once you have successfully finished the tutorial, you are ready to become a blockchain indexing wizard!
Join our [Discord](https://discord.gg/envio) channel to make sure you catch all new releases.
---
# Indexing Optimism Bridge Deposits
> Learn to quickly index Optimism Bridge deposits and explore OP Bridge event data.
## Introduction
This tutorial will guide you through indexing Optimism Standard Bridge deposits in under 5 minutes using Envio HyperIndex's no-code [contract import](/docs/HyperIndex/quickstart) feature.
The Optimism Standard Bridge enables the movement of ETH and ERC-20 tokens between Ethereum and Optimism. We'll index bridge deposit events by extracting the `DepositFinalized` logs emitted by the bridge contracts on both networks.
## Prerequisites
Before starting, ensure you have the following installed:
- **[Node.js](https://nodejs.org/en/download/current)** _(v22 or newer recommended)_
- **[pnpm](https://pnpm.io/installation)** _(recommended but not required)_
- **[Docker Desktop](https://www.docker.com/products/docker-desktop/)** _(required to run the Envio indexer locally)_
> **Note:** Docker is specifically required to run your blockchain indexer locally. You can skip Docker installation if you plan only to use Envio Cloud.
## Step 1: Initialize Your Indexer
1. Open your terminal in an empty directory and run:
```bash
pnpx envio init
```
2. Name your indexer (we'll use "optimism-bridge-indexer" in this example):
3. Choose your preferred language (TypeScript, JavaScript, or ReScript):
## Step 2: Import the Optimism Bridge Contract
1. Select **Contract Import** → **Block Explorer** → **Optimism**
2. Enter the Optimism bridge contract address:
```
0x4200000000000000000000000000000000000010
```
[View on Optimistic Etherscan](https://optimistic.etherscan.io/address/0x4200000000000000000000000000000000000010)
3. Select the `DepositFinalized` event:
- Navigate using arrow keys (↑↓)
- Press spacebar to select the event
> **Tip:** You can select multiple events to index simultaneously.
## Step 3: Add the Ethereum Mainnet Bridge Contract
1. When prompted, select **Add a new contract**
2. Choose **Block Explorer** → **Ethereum Mainnet**
3. Enter the Ethereum Mainnet gateway contract address:
```
0x99C9fc46f92E8a1c0deC1b1747d010903E884bE1
```
[View on Etherscan](https://etherscan.io/address/0x99C9fc46f92E8a1c0deC1b1747d010903E884bE1)
4. Select the `ETHDepositInitiated` event
5. When finished adding contracts, select **I'm finished**
## Step 4: Start Your Indexer
1. If you have any running indexers, stop them first:
```bash
pnpm envio stop
```
2. Start your new indexer:
```bash
pnpm dev
```
This command:
- Starts the required Docker containers
- Sets up your database
- Launches the indexing process
- Opens the Hasura GraphQL interface
## Step 5: Understanding the Generated Code
Let's examine the key files that Envio generated:
### 1. `config.yaml`
This configuration file defines:
- Networks to index (Optimism and Ethereum Mainnet)
- Starting blocks for each network
- Contract addresses and ABIs
- Events to track
### 2. `schema.graphql`
This schema defines the data structures for our selected events:
- Entity types based on event data
- Field types matching the event parameters
- Relationships between entities (if applicable)
### 3. `src/handlers`
This file contains the business logic for processing events:
- Functions that execute when events are detected
- Data transformation and storage logic
- Entity creation and relationship management
## Step 6: Exploring Your Indexed Data
Now you can interact with your indexed data:
### Accessing Hasura
1. Open Hasura at [http://localhost:8080](http://localhost:8080)
2. When prompted, enter the admin password: `testing`
### Monitoring Indexing Progress
1. Click the **Data** tab in the top navigation
2. Find the `_events_sync_state` table to check indexing progress
3. Observe which blocks are currently being processed
> **Note:** Thanks to Envio's [HyperSync](https://docs.envio.dev/docs/hypersync), indexing happens significantly faster than with standard RPC methods.
### Querying Indexed Events
1. Click the **API** tab
2. Construct a GraphQL query to explore your data
Here's an example query to fetch the 10 largest bridge deposits:
```graphql
query LargestDeposits {
DepositFinalized(limit: 10, order_by: { amount: desc }) {
l1Token
l2Token
from
to
amount
blockTimestamp
}
}
```
3. Click the **Play** button to execute your query
## Conclusion
Congratulations! You've successfully created an indexer for Optimism Bridge deposits across both Ethereum and Optimism networks.
### What You've Learned
- How to initialize a multi-network indexer using Envio
- How to import contracts from different blockchains
- How to query and explore indexed blockchain data
### Next Steps
- Try customizing the event handlers to add additional logic
- Create relationships between events on different networks
- Deploy your indexer to Envio Cloud
For more tutorials and advanced features, check out our [documentation](https://docs.envio.dev) or watch our [video walkthroughs](https://www.youtube.com/watch?v=9U2MTFU9or0) on YouTube.
---
# Scaffold-Eth-2 Envio Extension
> Scaffold-ETH Extension: Get a boilerplate indexer for your deployed smart contracts and start tracking events instantly.
# Scaffold-Eth-2 Envio Extension
## Introduction
The [Scaffold-ETH 2](https://scaffoldeth.io/extensions) Envio extension makes indexing your deployed smart contracts as simple as possible. Generate a boilerplate indexer for your deployed contracts with a single click and start indexing their events immediately.
With this extension, you get:
- 🔍 **Automatic indexer generation** from your deployed contracts
- 📊 **Status dashboard** with links to Envio metrics and database
- 🔄 **One-click regeneration** to update the indexer when you deploy new contracts
- 📈 **GraphQL API** for querying your indexed blockchain data
## Prerequisites
Before starting, ensure you have the following installed:
- **[Node.js v20](https://nodejs.org/en/download/current)** _(v20 or newer required)_
- **[pnpm](https://pnpm.io/installation)** _(for Envio indexer)_
- **[Docker Desktop](https://www.docker.com/products/docker-desktop/)** _(required to run the Envio indexer locally)_
- **[Yarn](https://yarnpkg.com/getting-started/install)** _(for Scaffold-ETH)_
## Step 1: Create a New Scaffold-ETH 2 Project with Envio Extension
To create a new Scaffold-ETH 2 project with the Envio extension already integrated:
```bash
npx create-eth@latest -e enviodev/scaffold-eth-2-extension
```
## Step 2: Start the Local Blockchain
Navigate to your project directory and start the local blockchain:
```bash
cd your-project-name
yarn chain
```
This will start a local blockchain node for development.
## Step 3: Deploy Your Contracts
In a new terminal window, navigate to your project directory and deploy the default smart contracts:
```bash
cd your-project-name
yarn deploy
```
This will deploy the default contracts to the local blockchain. This step is optional and can also be done once you've created your own smart contracts and deployed them using `yarn deploy`.
## Step 4: Start Scaffold-ETH Frontend
From your project directory, start the Scaffold-ETH frontend:
```bash
yarn start
```
This will start the Scaffold-ETH frontend at `http://localhost:3000`.
## Step 5: Generate the Indexer
Navigate to the Envio page in your Scaffold-ETH frontend at `http://localhost:3000/envio` and click the **"Generate"** button. This should only be done once you've created a smart contract and ran `yarn deploy`. This will create the boilerplate indexer from your deployed contracts.
The Envio page also includes a helpful "How to Use" section with step-by-step instructions.
## Step 6: Start the Indexer
Navigate to the Envio package directory and start the indexer:
```bash
cd packages/envio
pnpm dev
```
This will begin indexing your contract events.
## Regenerating the Indexer
When you deploy new contracts or make changes to existing ones, you'll need to regenerate the indexer:
### Via Frontend Dashboard
1. Go to the Envio page at `http://localhost:3000/envio`
2. Click "Generate" to regenerate the boilerplate indexer
### Via Command Line
```bash
cd packages/envio
pnpm update
pnpm codegen
```
> **Note:** Regenerating will overwrite any custom handlers, config, and schema changes, creating a fresh boilerplate indexer based on your deployed contracts. After regenerating, you'll need to stop the running indexer (Ctrl+C) and restart it with `pnpm dev` for the changes to take effect.
---
# What's New in HyperIndex V3
> Discover the new features in HyperIndex V3 — the unified handlers API, ESM and top-level await, automatic handler registration, a new testing framework, ClickHouse storage, Solana support, and more.
# What's New in HyperIndex V3
15 full months have passed since the official HyperIndex v2.0.0. Since then, we have shipped [32 minor releases](https://github.com/enviodev/hyperindex/releases) and multiple patches with **zero breaking changes** to the documented API. We also received PRs from 6 external contributors, grew from 1 GitHub star to over 470, and saw many big projects rely on HyperIndex.
HyperIndex V3 focuses on modernizing the codebase and laying the foundation for many more months of development. This page describes everything that's new. To upgrade an existing project from V2, follow the [Migrate to V3](./migrate-to-v3) guide.
## New Features
### Unified Handlers API
In V3 all handler registrations now happen through a single `indexer` value. Contract-specific exports (`ERC20.Transfer.handler`, `UniV3.PoolFactory.contractRegister`, etc.) have been removed in favor of `indexer.onEvent`, `indexer.contractRegister`, and `indexer.onBlock`.
**Event handlers** with `indexer.onEvent`:
```typescript
import { indexer } from "envio";
indexer.onEvent(
{
contract: "ERC20",
event: "Transfer",
wildcard: true,
where: ({ chain }) => ({
params: [
{ from: chain.Safe.addresses },
{ to: chain.Safe.addresses },
],
}),
},
async ({ event, context }) => {
// Handler logic
},
);
```
**Dynamic contracts** with `indexer.contractRegister`:
```typescript
import { indexer } from "envio";
indexer.contractRegister(
{
contract: "UniV3",
event: "PoolFactory",
},
async ({ event, context }) => {
context.chain.Pool.add(event.params.poolAddress);
},
);
```
**Block handlers** with `indexer.onBlock` consolidate across chains in a single call:
```typescript
import { indexer } from "envio";
indexer.onBlock(
{ name: "EveryBlock" },
async ({ block, context }) => {
// Handler logic
},
);
```
For chain-specific or interval-based block handlers, use the `where` callback:
```typescript
indexer.onBlock(
{
name: "Ranges",
where: ({ chain }) => {
if (chain.id !== 1) return false;
return {
block: {
number: {
_gte: 20_000_000,
_lte: 22_000_000,
_every: 100,
},
},
};
},
},
async ({ block, context }) => {
// Handler logic
},
);
```
### Per-Event Start Block
Handlers can specify custom start blocks per chain via `where.block.number._gte`, overriding contract and chain configuration:
```typescript
indexer.onEvent(
{
contract: "UniV4",
event: "Pool",
where: ({ chain }) => {
let startBlock: number;
switch (chain.id) {
case 1:
startBlock = 18_000_000;
break;
case 8453:
startBlock = 2_000_000;
break;
default: {
const _exhaustive: never = chain.id;
return false;
}
}
return {
block: { number: { _gte: startBlock } },
};
},
},
async ({ event, context }) => {
// Handler logic
},
);
```
### CommonJS → ESM
We migrated HyperIndex from CommonJS-only to ESM-only. This enables:
- Using the latest versions of libraries that have long since abandoned CommonJS support
- **Top-level await** in handler files
### Top-Level Await
Thanks to the migration to ESM, you can now use `await` directly in handler and other files:
```typescript
import { indexer } from "envio";
const ZERO_ADDRESS = "0x0000000000000000000000000000000000000000";
// Load data before registering handlers
const addressesFromServer = await loadWhitelistedAddresses();
indexer.onEvent(
{
contract: "ERC20",
event: "Transfer",
wildcard: true,
where: {
params: [
{ from: ZERO_ADDRESS, to: addressesFromServer },
{ from: addressesFromServer, to: ZERO_ADDRESS },
],
},
},
async ({ event, context }) => {
// ... your handler logic
},
);
```
### 3x Historical Backfill Performance
Achieved by adding chunking logic to request events across multiple ranges at once. This also fixed overfetching for contracts with a much later `start_block` in the config, as well as speeding up dynamic contract registration. If you had data fetching as a bottleneck, 25k events per second is now a standard.
### Automatic Handler Registration (`src/handlers`)
We introduced automatic registration of handler files located in `src/handlers`.
Previously, you needed to specify an explicit path to a handler file for every contract in `config.yaml`. Now you can remove all of the paths from `config.yaml` and simply move the files to `src/handlers`. You can name the files however you want, but we suggest using contract names and having a file per contract.
If you don't like `src/handlers`, use the `handlers` option in `config.yaml` to customize it.
:::note
The explicit `handler` field in `config.yaml` still works, so you don't need to change anything immediately.
:::
### RPC for Realtime Indexing
Built by an external contributor [@cairoeth](https://github.com/cairoeth) to allow specifying `realtime` mode for an RPC data source to embrace low-latency head tracking:
```yaml
rpc:
- url: https://eth-mainnet.your-rpc-provider.com
for: realtime
```
In this case, the RPC won't be used for historical sync but will be used as the primary source once the indexer enters realtime mode.
### Chain State on Context
The Handler Context object provides chain state via the `chain` property:
```typescript
import { indexer } from "envio";
indexer.onEvent(
{ contract: "ERC20", event: "Approval" },
async ({ context }) => {
console.log(context.chain.id); // 1 - The chain id of the event
console.log(context.chain.isRealtime); // true - Whether the indexer entered realtime mode
},
);
```
### Indexer State & Config
As a replacement for the deprecated and removed `getGeneratedByChainId`, we introduce the `indexer` value. It provides nicely typed chains and contract data from your config, as well as the current indexing state, such as `isRealtime` and `addresses`. Use `indexer` either at the top level of the file or directly from handlers. It returns the latest indexer state.
With this change, we also introduce new official types: `Indexer`, `EvmChainId`, `FuelChainId`, and `SvmChainId`.
```typescript
import { indexer } from "envio";
indexer.name; // "uniswap-v4-indexer"
indexer.description; // "Uniswap v4 indexer"
indexer.chainIds; // [1, 42161, 10, 8453, 137, 56]
indexer.chains[1].id; // 1
indexer.chains[1].startBlock; // 0
indexer.chains[1].endBlock; // undefined
indexer.chains[1].isRealtime; // false
indexer.chains[1].PoolManager.name; // "PoolManager"
indexer.chains[1].PoolManager.abi; // unknown[]
indexer.chains[1].PoolManager.addresses; // ["0x000000000004444c5dc75cB358380D2e3dE08A90"]
```
On indexer restart, reading `indexer` at the top level of a handler file returns values restored from the database — including dynamically registered contract addresses — rather than only what's declared in `config.yaml`:
```typescript
import { indexer } from "envio";
// Includes initial + dynamically registered addresses persisted in the DB
console.log(indexer.chains.eth.Pool.addresses);
```
### Conditional Event Handlers
Now it's possible to return a boolean value from the `where` function to disable or enable the handler conditionally.
```typescript
import { indexer } from "envio";
indexer.onEvent(
{
contract: "ERC20",
event: "Transfer",
wildcard: true,
where: ({ chain }) => {
// Skip all ERC20 on Polygon
if (chain.id === 137) {
return false;
}
// Track all ERC20 on Ethereum Mainnet
if (chain.id === 1) {
return true;
}
// Track only whitelisted addresses on other chains
return {
params: [
{ from: ZERO_ADDRESS, to: WHITELISTED_ADDRESSES[chain.id] },
{ from: WHITELISTED_ADDRESSES[chain.id], to: ZERO_ADDRESS },
],
};
},
},
async ({ event, context }) => {
// ... your handler logic
},
);
```
### Automatic Contract Configuration
Started automatically configuring all globally defined contracts. This fixes an issue where `addContract` crashed because the contract was defined globally but not linked for a specific chain. Now it's done automatically:
```yaml
contracts:
- name: UniswapV3Factory
events: # ...
- name: UniswapV3Pool
events: # ...
chains:
- id: 1
start_block: 0
contracts:
- name: UniswapV3Factory
address: "0x1F98431c8aD98523631AE4a59f267346ea31F984"
# UniswapV3Pool no longer needed here - auto-configured from global contracts
- id: 10
start_block: 0
contracts:
- name: UniswapV3Factory
address: "0x1F98431c8aD98523631AE4a59f267346ea31F984"
# UniswapV3Pool no longer needed here - auto-configured from global contracts
```
### ClickHouse Storage (Experimental)
HyperIndex can now run with multiple storage backends at the same time. Postgres remains the primary database, and entities can additionally be written to a ClickHouse database that is restart- and reorg-resistant. Prometheus metrics carry a storage-name label so you can distinguish backends.
Enable backends in `config.yaml` and route each entity explicitly via the `@storage` directive in `schema.graphql`:
```yaml
storage:
postgres: true
clickhouse: true
```
```graphql
# Stored in both Postgres and ClickHouse
type Transfer @storage(postgres: true, clickhouse: true) {
id: ID!
from: String!
to: String!
value: BigInt!
}
# Stored only in ClickHouse
type Snapshot @storage(clickhouse: true) {
id: ID!
blockNumber: BigInt!
}
```
Per-entity routing is more verbose but lets you write some entities to Postgres and others to ClickHouse only.
`envio dev` automatically spins up a ClickHouse Docker container for local development with playground-friendly defaults so you can connect to it without configuring a password. For `envio start`, provide your own connection via the environment variables `ENVIO_CLICKHOUSE_HOST`, `ENVIO_CLICKHOUSE_DATABASE`, `ENVIO_CLICKHOUSE_USERNAME`, and `ENVIO_CLICKHOUSE_PASSWORD`.
Envio Cloud currently supports ClickHouse on the Dedicated Plan.
For high-availability ClickHouse setups, HyperIndex supports two additional environment variables:
- `ENVIO_CLICKHOUSE_REPLICATED` — set to `true` to use replicated table engines.
- `ENVIO_CLICKHOUSE_DATABASE_ENGINE` — override the database engine (for example, `Replicated`).
:::warning
Do not run multiple indexers writing to the same ClickHouse database at the same time.
:::
### HyperSync Source Improvements
Multiple updates on the HyperSync side to achieve smaller latency and less traffic:
- Server-Sent Events instead of polling to get updates about new blocks
- CapnProto instead of JSON for query serialization
- Cache for queries with repetitive filters - huge egress saving when indexing thousands of addresses
- Improved connection establishment behind a proxy
- Configurable log level support via `ENVIO_HYPERSYNC_LOG_LEVEL` environment variable
- Automatic rate-limiting handling on the client side
- Better reconnection logic, logging, and fallbacks for HyperSync SSE and RPC WebSocket height streaming for more stable indexing at the chain head
### Fuel Block Handler Support
Block handlers are now supported for Fuel indexing.
### Solana Support (Experimental)
HyperIndex now supports Solana with RPC as a source. This feature is experimental and may undergo minor breaking changes. Solana exposes its block-stream handler as `indexer.onSlot` (rather than `onBlock`) to match Solana's slot-based model.
To initialize a Solana project:
```bash
pnpx envio init svm
```
See the [Solana documentation](/docs/HyperIndex/solana) for more details.
### `pnpx envio init` Improvements
- Removed language selection to prefer TypeScript by default
- Cleaned up templates to follow the latest good practices
- Added new templates to highlight HyperIndex features: `Feature: Factory Contract`, `Feature: External Calls`
- Pre-configured GitHub Actions workflow for running tests and initialized git repository
- Generated projects include Cursor/Claude skills to support agent-driven development
### Block Handler Only Indexers
Now it's possible to create indexers with only block handlers. Previously, it was required to have at least one event handler for it to work. The `contracts` field became optional in `config.yaml`.
### Flexible Entity Fields
We no longer have restrictions on entity field names, such as `type` and others. Shape your entities any way you want. There are also improvements in generating database columns in the same order as they are defined in the `schema.graphql`.
### Unordered Multichain Mode Only
Unordered multichain mode is now the only mode in V3 — events from different chains are processed in parallel without strict cross-chain ordering, which provides better performance for most use cases. The V2 `unordered_multichain_mode` option and the `multichain: ordered` opt-in have been removed.
### Preload Optimization by Default
Preload optimization is now enabled by default, replacing the previous `loaders` and `preload_handlers` options. This improves historical sync performance automatically.
### TUI Improvements
We gave our TUI some love, making it look more beautiful and compact. It also consumes fewer resources, shares a link to the Hasura playground, and dynamically adjusts to the terminal width.
The TUI now shows an **events-per-second** indicator during backfill so you can see indexing throughput at a glance.
The TUI is also auto-disabled in CI environments and when running under AI agents, so logs stay clean without manual configuration. The legacy `TUI_OFF=true` environment variable was renamed to `ENVIO_TUI=false`.
### New Testing Framework
HyperIndex ships a purpose-built testing framework powered by `createTestIndexer()`. Write tests against the same indexer that runs in production — no database, no Docker, no manual mock wiring.
The framework integrates with [Vitest](https://vitest.dev/), replacing the previous mocha/chai setup with a single package that doesn't require configuration by default and includes snapshot testing out-of-the-box. It also provides typed test assertions and utilities to read/write entities in-between processing runs.
#### Three ways to feed events
**1. Auto-exit** — processes the first block with matching events, then exits. Each subsequent call continues where the last one stopped. Zero config needed.
```typescript
import { describe, it } from "vitest";
import { createTestIndexer } from "envio";
describe("ERC20 indexer", () => {
it("processes the first block with events", async (t) => {
const indexer = createTestIndexer();
const result = await indexer.process({ chains: { 1: {} } });
// Auto-filled by Vitest on first run — just review and commit
t.expect(result).toMatchInlineSnapshot(`
{
"changes": [
{
"Transfer": {
"sets": [
{
"blockNumber": 10861674,
"from": "0x0000000000000000000000000000000000000000",
"id": "1-10861674-23",
"to": "0x41653c7d61609D856f29355E404F310Ec4142Cfb",
"transactionHash": "0x4b37d2f343608457ca...",
"value": 1000000000000000000000000000n,
},
],
},
"block": 10861674,
"chainId": 1,
"eventsProcessed": 1,
},
],
}
`);
});
});
```
**2. Explicit block range** — pin to specific blocks for deterministic CI snapshots.
```typescript
const result = await indexer.process({
chains: {
1: {
startBlock: 10_861_674,
endBlock: 10_861_674,
},
},
});
```
**3. Simulate** — feed typed synthetic events for pure unit tests. No network, no block ranges.
```typescript
await indexer.process({
chains: {
137: {
simulate: [
{
contract: "Greeter",
event: "NewGreeting",
params: { greeting: "Hello", user: "0x123..." },
},
],
},
},
});
```
#### Key capabilities
- **Snapshot-driven assertions** — `result.changes` captures every entity set/delete per block. Pair with `toMatchInlineSnapshot` for auto-generated, reviewable snapshots.
- **Direct entity access** — `indexer.Entity.get()`, `.getOrThrow()`, `.getAll()`, and `.set()` for reading and presetting state.
- **Real pipeline, real confidence** — tests exercise the full indexer pipeline including dynamic contract registration, multi-chain support, and handler context.
- **Parallel test execution** via worker thread isolation.
The test indexer also exposes chain information:
```typescript
const indexer = createTestIndexer();
indexer.chainIds; // [1, 42161]
indexer.chains[1].id; // 1
indexer.chains[1].startBlock; // 0
indexer.chains[1].ERC20.addresses; // ["0x..."]
// Read/write entities between processing runs
await indexer.Account.set({ id: "0x123...", balance: 100n });
const account = await indexer.Account.get("0x123...");
```
See the [Testing documentation](/docs/HyperIndex/testing) for more details.
### Podman Support
Beyond Docker, HyperIndex now supports [Podman](https://podman.io/) for local development environments. This provides an alternative container runtime for developers who prefer Podman or have it available in their environment.
### Nested Tuples for Contract Import
The `envio init` command now supports contracts with nested tuples in event signatures, which was previously a limitation when importing contracts.
### PostgreSQL Update for Local Docker Compose
The local development Docker Compose setup now uses PostgreSQL 18.1 (upgraded from 17.5).
### `contractName` and `eventName` on Event
Events now include `contractName` and `eventName` fields, making it easier to identify which contract and event you're working with in handlers:
```typescript
import { indexer } from "envio";
indexer.onEvent(
{ contract: "ERC20", event: "Transfer" },
async ({ event }) => {
console.log(event.contractName); // "ERC20"
console.log(event.eventName); // "Transfer"
},
);
```
### New Official Exported Types
Generated code now exports official generic types for entities, enums, and events. These replace the previous contract-specific type exports:
```typescript
import type {
MyEntity, // Still exported but Entity<"MyEntity"> is preferred
Entity, // Generic entity type — use as Entity<"MyEntity">
Enum, // Generic enum type — use as Enum<"MyEnum"> (replaces direct MyEnum export)
EvmEvent, // Generic event type — use as EvmEvent<"ERC20", "Transfer">
// Access specific fields: EvmEvent<"ERC20", "Transfer">["block"]
} from "envio";
```
### Support for DESC Indices
A nice way to improve your query performance as well:
```graphql
type PoolDayData
@index(fields: ["poolId", ["date", "DESC"]]) {
id: ID!
poolId: String!
date: Timestamp!
}
```
### RPC Source Improvements
Added `polling_interval` option for RPC source configuration. Also added missing support for receipt-only fields (`gasUsed`, `cumulativeGasUsed`, `effectiveGasPrice`) that are not available via `eth_getTransactionByHash`. HyperIndex will additionally perform the `eth_getTransactionReceipt` request when one of the fields is added in `field_selection`.
### WebSocket Support (Experimental)
Experimental WebSocket support for RPC source to improve head latency. Please create a GitHub issue if you come across any problems.
```yaml
chains:
- id: 1
rpc:
url: ${ENVIO_RPC_ENDPOINT}
ws: ${ENVIO_WS_ENDPOINT}
for: realtime
```
### Prometheus Metrics for Data Providers
Added a Prometheus metric to track requests to data providers, providing better observability into your indexer's data fetching patterns.
### GraphQL-Style `getWhere` API
The `getWhere` query API has been redesigned using GraphQL-style syntax:
```typescript
// Before
const transfers = await context.Transfer.getWhere.from.eq("0x123...");
// After
const transfers = await context.Transfer.getWhere({ from: { _eq: "0x123..." } });
```
Additionally, three new filter operators are available following Hasura-style conventions:
```typescript
context.Entity.getWhere({ amount: { _gte: 100n } })
context.Entity.getWhere({ amount: { _lte: 500n } })
context.Entity.getWhere({ status: { _in: ["active", "pending"] } })
```
### Direct RPC Client
Replaced Ethers.js with a direct RPC client implementation, reducing dependencies and improving performance.
### Block Lag Configuration
A per-chain `block_lag` option to index behind the chain head by a specified number of blocks. Replaces the global `ENVIO_INDEXING_BLOCK_LAG` environment variable. Defaults to 0. This is for advanced use cases — only use it if you know what you're doing.
```yaml
chains:
- id: 1
block_lag: 5
```
### Official `/metrics` Endpoint
Prometheus metrics are now official. We cleaned up metric names, switched time units to seconds instead of milliseconds, and followed Prometheus naming conventions more closely. Metrics also cover data points previously available only via the `--bench` feature. A separate `/metrics/runtime` endpoint with a dedicated Prometheus registry is available for runtime metrics, isolated from the default `/metrics` endpoint.
Starting from the v3.0.0 release, Prometheus metrics will follow semver and be documented.
Breaking changes:
- Cleaned up metric names and switched time units from milliseconds to seconds
- Removed [`--bench`](/docs/HyperIndex/benchmarking) support — use the `/metrics` endpoint instead
Use the new `envio metrics` CLI command to fetch the Prometheus metrics of a locally running indexer without curling the endpoint manually.
### Continue on Config Change
HyperIndex can now keep indexing through some `config.yaml` changes — `rpc` configuration is the first to land — instead of erroring out on every restart. Where a change is incompatible, the CLI prints exactly which fields were touched and offers two clear options (revert, or `envio dev -r` to wipe and re-index). More flexibility will be unlocked over time; open a GitHub issue if you need a specific field supported.
### Double Handler Registration
It's now possible to register multiple handlers for the same event with similar filters:
```typescript
import { indexer } from "envio";
indexer.onEvent(
{ contract: "ERC20", event: "Transfer" },
async ({ event, context }) => {
// Your logic here
},
);
indexer.onEvent(
{ contract: "ERC20", event: "Transfer" },
async ({ event, context }) => {
// And here
},
);
```
### Improved Multiple Data-Sources Support
After switching to a fallback source, HyperIndex now attempts to recover to the primary source 60 seconds later. Previously, it would stay on the fallback until the fallback was down or the indexer was restarted. The source selection logic has also been improved for better indexing resilience and stricter enforcement of the `realtime` mode configuration.
### Updated Dev Docker Flow
`envio dev` no longer uses a generated Docker Compose file and manages containers, network, and volumes directly for greater flexibility. For example, disabling Hasura with `ENVIO_HASURA` now prevents `envio dev` from pulling the Hasura image. Use `envio dev --restart` (or `-r`) to forcefully clear the database even if there are no config changes detected.
### Envio Dev Update
`envio dev` no longer automatically resets the database on incompatible config or schema changes. Use `envio dev -r` to explicitly allow this.
### Envio Start Update
`envio start` now has a clear role: to run HyperIndex in the production environment. Use `envio dev` for local development to enable debugging with Dev Console.
### Optimized `envio codegen`
`envio codegen` is now near-instant. We no longer run `pnpm i` for the `generated` package, and we no longer recompile ReScript every time you change `config.yaml` or `schema.graphql`. The output is also a lot quieter.
### `envio skills update` Command
Pull the latest Claude/Cursor skills into your project so agent-driven development stays in sync with the latest HyperIndex APIs:
```bash
pnpx envio skills update
```
### `envio config view` Command (Experimental)
Inspect your fully resolved indexer configuration as JSON — useful for debugging configuration issues and for tooling that needs to consume the resolved config:
```bash
pnpx envio config view
```
### Improved TypeScript Error Messages
When generated types are missing, the TypeScript error now explicitly suggests running `envio codegen` instead of leaving you to puzzle out the cause.
### Smaller `envio` Package (-88MB)
By eliminating dynamically generated ReScript code, we no longer need to ship or run a ReScript compiler at runtime. The published npm package shrank from 141MB to 53MB.
### No Hard pnpm Requirement
Internal use of pnpm is gone. The `generated` package no longer has its own dependency tree, so HyperIndex works with whichever package manager you prefer.
### Bun Support
Run HyperIndex on Bun:
```bash
bun --bun envio dev
```
### Choose Your Package Manager on `envio init`
`envio init` now accepts `--package-manager=pnpm|npm|bun|yarn` so you can scaffold projects without committing to pnpm.
### Better Tuples Developer Experience
Solidity struct components used to be generated as positional tuples in handler params, which made handler code awkward. They are now generated as objects with named fields:
```solidity
struct CreateEventCommon {
address funder;
address sender;
address recipient;
Lockup.CreateAmounts amounts;
IERC20 token;
bool cancelable;
bool transferable;
Lockup.Timestamps timestamps;
string shape;
address broker;
}
event CreateLockupTranchedStream(
uint256 indexed streamId,
Lockup.CreateEventCommon commonParams,
LockupTranched.Tranche[] tranches
);
```
```typescript
// Before
event.params.commonParams[5];
event.params.commonParams[3][0];
// After
event.params.commonParams.cancelable;
event.params.commonParams.amounts.deposit;
```
### Improved Multichain Backfill
For large multichain indexers, HyperIndex now throttles chains that have already reached the head so they don't compete for resources while the rest finish backfilling. Once every chain has caught up, throttling is lifted and all chains continue indexing equally.
### Toolchain Upgrades
- ReScript upgraded from v11 to v12 (internally and in `envio init` templates)
- TypeScript upgraded from v5 to v6 (internally and in `envio init` templates)
### 2x Cheaper and 2.5x Faster (v3.1)
HyperIndex now requires up to **2x fewer HyperSync queries** during backfill and is **2.5x faster** in many indexing cases. If you had data fetching as a bottleneck, this comes for free on upgrade.
### Descriptions on Entities, Fields, and Relationships (v3.1)
You can now document your entities, fields, and relationships directly in `schema.graphql` using string descriptions. These are exposed through the GraphQL API and appear in introspection:
```graphql
"""
A token transfer between two accounts
"""
type Transfer {
id: ID!
"The address the tokens were sent from"
from: String!
"The address the tokens were sent to"
to: String!
"The amount transferred, in wei"
value: BigInt!
}
```
Only string descriptions (`"""..."""` or `"..."`) are exposed. Hash (`#`) comments are ignored by the GraphQL parser and do **not** appear in introspection.
### Skip Chains From Indexing (v3.1)
A new `skip` field in `config.yaml` lets you exclude a specific chain from indexing and database migrations without removing it from your config:
```yaml
chains:
- id: 1
start_block: 0
contracts: # ...
- id: 137
skip: true
start_block: 0
contracts: # ...
```
### Improved Agentic Indexer Development (v3.1)
New CLI subcommands make it easier to build indexers with AI agents:
```bash
envio tools search-docs