For AI agents: the documentation index is at /llms.txt. Markdown versions of pages are available by appending .md to the URL.
Skip to main content

Benchmarking Your Indexer

Every HyperIndex indexer reports what it's doing over an HTTP metrics endpoint while it runs. There's no benchmarking mode to enable and no profiler to attach — if the indexer is running, the numbers are already there.

This page turns those numbers into an answer to one question: what is my indexer waiting on, and what do I do about it?

Work through it in order:

  1. Measure throughput — how fast is it, in events per second?
  2. Find the bottleneck — which of five things is it waiting on?
  3. Fix it — the section for your bottleneck says what to change.
  4. Prove the fix worked — re-measure and compare.

Step 1: Measure throughput​

Start the indexer in one terminal — locally with envio dev, or in production with envio start:

pnpm envio dev

Read its metrics from a second terminal, in the same project:

pnpm envio metrics          # indexing metrics
pnpm envio metrics runtime  # Node.js process metrics (CPU, memory, GC, event-loop lag)

The output is plain Prometheus text — a # HELP line describing each metric, a # TYPE line, then one sample per line with its labels in braces:

# HELP envio_progress_events The number of events processed and reflected in the database.
# TYPE envio_progress_events gauge
envio_progress_events{chainId="1"} 158205

# HELP envio_process_elapsed_seconds How long the indexer has been running.
# TYPE envio_process_elapsed_seconds gauge
envio_process_elapsed_seconds 45.801

Because every metric documents itself, grepping for the metric you care about — pnpm envio metrics | grep envio_progress — is usually the fastest way to answer a specific question.

Take two snapshots a minute or so apart, and divide the change in events by the change in elapsed time:

pnpm envio metrics > t1.txt
# wait ~60s
pnpm envio metrics > t2.txt

grep -E '^envio_(progress_events|process_elapsed_seconds)' t1.txt t2.txt
# t1: 158205 events at 45.8s
# t2: 262000 events at 105.8s
# (262000 - 158205) / (105.8 - 45.8) = ~1730 events/second

envio_progress_events is reported per chainId, so a multichain indexer prints one sample per chain: compare the same chainId across both snapshots for a per-chain figure, or sum every chain within each snapshot for the indexer as a whole. Both snapshots also have to come from one uninterrupted run — restarting the indexer resets envio_process_elapsed_seconds, so start a fresh baseline after any restart. (Prometheus rate() handles that for you when you scrape instead.)

Interpret the result against what your workload plausibly allows:

Events per secondReading
Over 10,000Excellent — most likely bounded by the data source, not by you
1,000–5,000Good. Worth tuning only if your sync time is still too long
Under 500Something is probably wrong. Continue to step 2
Measure over historical sync, not at the head

Once a chain is caught up (envio_progress_ready is 1), throughput reflects how fast new blocks arrive, not how fast your indexer is. Benchmark while the chain is still syncing history, or against a fixed block range with an end_block in config.yaml, so two runs are comparable.

Step 2: Find the bottleneck​

A slow indexer is waiting on something. Two counters attribute that wait directly — read them first, because they point at the answer instead of hinting at it:

pnpm envio metrics | grep -E 'stalled_on|process_elapsed'
MetricMeaningGo to
envio_processing_stalled_on_fetch_secondsNothing to process — events hadn't been fetched yetFetching
envio_processing_stalled_on_storage_write_secondsProcessing paused — too many changes queued for writingStorage writes

Both are cumulative seconds, so read them as a share of the run:

envio_processing_stalled_on_fetch_seconds / envio_process_elapsed_seconds

Anything above roughly 0.3 (30% of the run) is worth acting on. The same division works for every _seconds counter — that's how you compare parts of the pipeline against each other:

MetricShare of the run spent…
envio_processing_secondsinside your event handlers
envio_preload_secondsloading entities for a batch
envio_storage_write_secondswriting batches to storage
envio_fetching_block_range_secondsfetching block ranges (per chainId)
envio_effect_call_seconds_totalinside Effect API calls (per effect)
Counters, counts, and gauges

Only _seconds counters are durations you can divide by elapsed time. Counts (envio_progress_events, envio_processing_handler_total) and gauges (envio_progress_block, envio_indexing_concurrency) are read as values or as deltas between snapshots. envio_process_metric_time_seconds records when a snapshot was taken, so you can compute rates between two of them by hand.

If you scrape with Prometheus instead of reading snapshots, every counter above works as a rate — rate(envio_processing_stalled_on_fetch_seconds[5m]) — see Scraping with Prometheus.

Step 3: Fix the bottleneck​

Each section below covers one bottleneck: how you recognise it, what it means, and what to change. Work only the one that step 2 pointed you at.

Fetching​

How you know

pnpm envio metrics | grep -E 'stalled_on_fetch|indexing_idle|fetching_block_range_seconds'

envio_processing_stalled_on_fetch_seconds is a large share of the run, and envio_fetching_block_range_seconds dominates the other _seconds counters. envio_indexing_idle_seconds climbing on a specific chainId tells you which chain is holding you up.

What it means

Your handlers are idle waiting for data. Time spent waiting at the chain head for new blocks is deliberately excluded from the stall counter, so a fully synced indexer never looks stalled here.

What to do

  • Use HyperSync if it supports your network — it's the single biggest change available, and RPC is orders of magnitude slower.
  • On RPC, a faster provider or a higher rate limit is the fix; check envio_source_request_seconds_total per source to see what each one costs you.
  • Compare envio_indexing_concurrency with envio_indexing_partitions for the chain — if concurrency sits at its ceiling while events still arrive slowly, the source is saturated, not the indexer.
  • Watch envio_fetching_block_range_parse_seconds: when parsing rivals fetching, you're decoding far more events than your handlers use, so narrow the events in config.yaml.

Entity loading​

How you know

pnpm envio metrics | grep -E 'preload_seconds|storage_load'

envio_preload_seconds or envio_storage_load_seconds_total is a large share of the run. envio_storage_load_size and envio_storage_load_where_size (labeled by operation and storage) show which loads pull the most data.

What it means

Preload optimization batches the entity reads of a whole event batch into a few queries — but only for reads it can see during the preload phase. A read hidden behind a condition that only some events reach, or one that depends on the result of an earlier read, falls back to a single-event round trip.

What to do

  • Read entities unconditionally near the top of the handler so the preload phase can batch them, rather than inside a branch.
  • Avoid chaining reads (get → use the id you just read → get again); load both entities up front where possible.
  • Add database indices for the fields you filter on with getWhere.
  • Reduce how much each load returns: a getWhere that matches thousands of rows costs more than the handler usually needs.

Handlers​

How you know

pnpm envio metrics | grep -E 'processing_handler_(seconds|total)'

envio_processing_seconds is a large share of the run. To find the specific handler, divide cumulative time by call count — both are labeled by contract and event:

envio_processing_handler_seconds / envio_processing_handler_total

What it means

Your own code is the cost. A mean above ~1ms per call is worth a look; compare handlers against each other rather than against an absolute number.

What to do

  • Move external calls (RPC, HTTP, IPFS) out of the handler and into the Effect API, which batches, parallelizes, and caches them.
  • Keep heavy derived computation out of the hot path — store what you need and compute the rest at query time.
  • Look for accidental work per event: parsing the same static data repeatedly, formatting, or building large intermediate objects.

Storage writes​

How you know

pnpm envio metrics | grep -E 'stalled_on_storage_write|storage_write'

envio_processing_stalled_on_storage_write_seconds is a large share of the run, and envio_storage_write_seconds is high next to envio_storage_write_total (labeled by storage).

What it means

The indexer produced entity changes faster than storage accepted them, so processing paused until the write queue drained.

What to do

  • Write fewer entities per event. Updating a running-total entity on every event makes each event a write; aggregating in memory and writing on a coarser boundary avoids most of them.
  • Don't update the same entity several times within one handler — the last write wins anyway.
  • Check your database: indices speed up reads but every extra index slows writes down, and an under-resourced Postgres shows up here first.
  • If reorg support is enabled, entity history is written alongside your entities — envio_rollback_history_prune_seconds per entity shows what that costs.

External calls​

How you know

pnpm envio metrics | grep -E 'envio_effect_'

envio_effect_call_seconds_total is a large share of the run, or envio_effect_queue_wait_seconds shows calls waiting on a rate limit. Both are labeled per effect, so the slow one names itself.

What it means

Handlers are blocked on Effect API calls to something outside the indexer — an RPC node, an HTTP API, IPFS.

What to do

  • Set cache: true on effects whose result is stable for a given input, so a rerun doesn't repeat the call; envio_effect_cache tracks cache size and envio_effect_cache_invalidations how often it's discarded.
  • Raise the effect's rateLimit if the provider allows it — envio_effect_queue sitting above zero means calls are queuing rather than running.
  • If effects are slow and can't be batched, lower full_batch_size so a batch doesn't wait on thousands of pending calls.
  • Compare envio_effect_call_seconds_total with envio_effect_call_total to separate "each call is slow" from "there are too many calls".

Step 4: Prove the fix worked​

Benchmarking is only useful as a loop:

  1. Capture a baseline before changing anything: pnpm envio metrics > before.txt.
  2. Change one thing. Two changes at once make it impossible to attribute the difference.
  3. Re-run over the same block range, ideally from a fresh database so the run does the same work.
  4. Compare the same numbers: events per second from step 1, and the share of the run for the counter you targeted.
pnpm envio metrics > after.txt
diff <(grep -E '^envio_(progress_events|.*_seconds)' before.txt) \
     <(grep -E '^envio_(progress_events|.*_seconds)' after.txt)

Two things to watch while you iterate:

  • Memory. pnpm envio metrics runtime reports heap usage, GC, and event-loop lag. An indexer that speeds up while memory climbs is often deferring work rather than avoiding it.
  • Realistic conditions. Benchmark against a block range with the event mix your indexer actually sees — a quiet range flatters every change you make.

Next Steps​