Nodes Silently Miss Events

TL;DR

• A JSON-RPC response does not carry proof of its own completeness. A node that has lost receipts, or a provider whose fleet is out of sync, can return a well-formed response with logs missing, and this is difficult to tell from the response alone.
• An EVM block header commits to its transactions and receipts through two Merkle-Patricia roots. Logs are part of receipts, so a dropped or altered log changes the receipts root.
• Recomputing those roots requires the complete block, meaning all of its transactions and receipts. A filtered eth_getLogs response cannot be checked this way, and this also limits what proxies and load balancers in front of RPC are able to verify.
• Because HyperSync ingests entire blocks, it is able to recompute both roots before data is served, and to refetch from a different source when they do not match. This tends to catch missing logs at ingest rather than later in an indexer's database.
• The protection is aimed at faulty sources rather than adversarial ones, and some chains need chain-specific handling before the roots can be recomputed.

The problem

Ethereum's JSON-RPC interface returns plain JSON. When a client calls eth_getLogs, the response is a list of log objects, and there is nothing in that list that shows whether it is complete. The same applies to eth_getBlockReceipts. A block returned by eth_getBlockByNumber with full transaction objects does carry enough to recompute transactionsRoot, though a hash-only response does not, and in either form the block omits its receipts, so it says nothing about whether the logs are complete. The client is trusting that the node executed the block correctly, stored the results correctly, and served them correctly.

That trust is usually justified, but it is difficult to verify at the point of use, and when it breaks the failure tends to be quiet. A node with a corrupted receipts database does not usually return an error. It returns fewer logs.

This matters most for indexers, because an indexer's output depends on every log it has received. One missing transfer event produces a balance that would be wrong, and this is not caught downstream.

The Ethereum protocol does provide commitments that make this checkable. The question is where in the stack there is enough data to make use of them.

What a block header commits to

An EVM block header contains, among other fields, three commitments defined in the Ethereum Yellow Paper (Wood, Section 4, "Blocks, State and Transactions"):

Header field	Commits to
transactionsRoot	The root of a Merkle-Patricia trie keyed by transaction index, whose values are the RLP or EIP-2718 encoded transactions of the block.
receiptsRoot	The root of a Merkle-Patricia trie keyed by transaction index, whose values are the encoded receipts. Each receipt contains the status (EIP-658, for blocks since Byzantium; earlier receipts carried an intermediate state root in its place), cumulative gas used, the receipt's bloom filter, and the full list of logs emitted by that transaction.
parentHash	The Keccak-256 hash of the previous block's header, which links headers into a chain.

The header also contains logsBloom, the bitwise OR of every receipt's bloom, and stateRoot, which commits to the world state after execution.

This has a useful consequence for logs. A log is part of the receipt's encoding, so a change to any log in the block changes the receipt's encoding, which changes the trie value, which changes receiptsRoot. Removing a log, adding one, reordering topics, or altering a data field would each produce a different root. The same holds for transactions and transactionsRoot.

So in principle the check is fairly straightforward. Given a block's header and all of its transactions and receipts, recompute both roots and compare them with the header. If they match, the transactions and receipts are the ones the block producer committed to. If they do not, something has been missed or altered.

The condition matters. The check needs all of the block's receipts. A response to eth_getLogs filtered by address and topic is a subset, and there is no commitment to a subset. This is less a limitation of any particular implementation than a consequence of how the header is structured.

Documented cases of incomplete data

It is not hypothetical that logs are missed. This pattern has recurred several times across execution clients over some years, and in 2025 it became the subject of a longer discussion about Ethereum's RPC standards.

"Ethereum needs Standards-Punk"

On 5 October 2025, Sebastian Bürgel, founder of HOPR, published a post on ethresear.ch titled "Ethereum needs Standards-Punk". HOPR's mixnet uses eth_getLogs to map its payment-channel topology, and the team had spent the preceding months chasing channels that appeared closed without ever having been opened. The post opens with the problem as it looks from the application side:

Right now, if a wallet or app queries eth_getLogs from a production Ethereum client, there's a real chance it will silently miss events. The result is simple and devastating: balances don't add up, transaction histories show funds being spent before they are ever received, and applications cannot give users a trustworthy account of what happened. Consensus may still be intact, but the interface developers actually rely on is corrupted.

The post links eight separate reports across Erigon, Nethermind, and HOPR's own tracker, on both Gnosis Chain and Ethereum mainnet, and calls the problem systemic. It also points at the conformance gap. Of roughly 190 RPC compatibility tests in Hive, four cover eth_getLogs, and about half of those had been failing for months without affecting any client's release. The post's proposal is a standards and conformance group for execution-layer RPC, with eth_getLogs as its first focus, and test suites in the hundreds of thousands rather than the hundreds.

The thread that followed is also useful. Mario Vega of the Ethereum Foundation's testing team added the topic to the agenda of All Core Devs Testing call 57 on 13 October 2025. Etan Kissling pointed to EIP-7919, which aims to make RPC answers verifiable so that a provider is trusted only for availability and not for correctness. A later reply from Antony Denyer described what teams do in practice once they stop trusting the RPC boundary: they ingest the entire chain into their own datastore and rebuild derived state from scratch, which is close to the architecture described later in this article.

Bürgel's own follow-up in the thread is relevant to what follows here. Even with better encoding and verification schemes, he argues, the existing JSON-RPC interface will not go away, because every library that dapps and wallets depend on is built on it, and the semantics of filtering logs are "far from trivial, with several edge cases left unspecified for clients to handle as they see fit."

The report behind the post: Nethermind 1.33.1 on Gnosis Chain

Three weeks earlier, on 16 September 2025, Bürgel had opened Nethermind issue #9305 after noticing that his Nethermind node on Gnosis Chain was returning fewer logs than the public Gnosis RPC. His method is worth noting because, from outside the node, comparison is more or less the only detection method available: query two providers, block by block.

for ((b=$START; b<=$END; b+=$RANGE)); do
  e=$((b+RANGE-1))
  n1=$(curl -s -X POST $PROV1 ... eth_getLogs ... | jq '.result|length')
  n2=$(curl -s -X POST $PROV2 ... eth_getLogs ... | jq '.result|length')
  if [ "$n1" -eq "$n2" ]; then echo "$b-$e: OK"; else echo "$b-$e: $n1 vs $n2"; fi
done

The output shows the Nethermind node returning zero logs for a filtered Transfer query on roughly a third of the blocks in a 50-block window, where the other provider returned between 1 and 11:

39556455-39556455: OK
39556456-39556456: 0 vs 2
39556457-39556457: 0 vs 6
39556458-39556458: OK
39556459-39556459: 0 vs 2
...
39556463-39556463: 0 vs 11
39556464-39556464: 0 vs 8

Two details from the thread are relevant here. First, the reference provider was not fully consistent either. Bürgel noted that the public Gnosis RPC gave "a little bit inconsistent" results at a rate of roughly 1 in 50, missing some logs on repeated queries. Second, the root cause was database corruption in the node's receipts store introduced in the previous release. The maintainers closed the issue four days later with "Logs are back after fixing db corruption", and the 1.33.1 release notes were updated to advise operators affected by "the missing receipts from running v1.33.0" to run with Sync.FixReceipts set to true or to delete and re-download the receipts database.

The node did not have a way to know its receipts were incomplete, so it served them. Without a second source to compare against, this would likely have gone unnoticed.

The same failure across clients and years

The reports Bürgel's post links, and older ones like them, share a similar shape. The node is healthy, the block is valid, and the log index or receipt store has drifted from the chain.

• Erigon #16613, August 2025. A Gnosis Chain archive node returned no logs for eth_getLogs over ranges where other nodes returned them. The thread ran to 23 comments over two months before it was closed.
• Erigon #16364, July 2025. A mainnet archive node returned empty results from eth_getLogs while it was creating snapshots, then correct results afterwards.
• Nethermind #9178, August 2025. HOPR's team reported empty eth_getLogs results for historical Gnosis Chain blocks that other providers served correctly. The cause was in an experimental log-index branch, later fixed.
• Besu #1153, June 2020. A query for deposit contract events on Görli returned results with "a number of blocks missing" compared with the same query against Infura.
• go-ethereum #18198, November 2018. A Rinkeby node consistently omitted one log from a range query, and the transaction receipt returned null, while other nodes and Etherscan showed it. Restarting the node restored the log.

All of these were resolved, and the point is not that any particular client is unreliable today. It is that receipt storage and log indexing are handled separately from block validation in clients generally, so they can drift without the node noticing, and the RPC response does not carry the information needed to detect that drift.

Provider fleets

Hosted providers run many nodes behind a load balancer, which adds a second source of inconsistency: two requests can be served by two nodes in different states, or by two different clients. While investigating the Gnosis Chain problems above, HOPR's engineers found that the same eth_getLogs request to a public endpoint sometimes returned a log and sometimes returned an empty array, and that web3_clientVersion called twice in a row came back as Nethermind, then Reth, then Erigon (hoprnet #7437). The endpoint was balancing across three execution clients, one of which was missing the log. The cases below name the providers involved because the details are what make them useful. In each of them the provider was serving what its node software produced. The fault sits with the node implementation, or with the chain's own state during an outage, and a self-hosted node running the same software would most likely have returned the same data. All four providers remain in HyperSync's source pool.

Robinhood Chain, September 2026. This case is what prompted the article, and it is worth describing because the bad data came from Alchemy, one of the best-known providers in the industry. HyperIndex can read from HyperSync or from RPC, and a production configuration usually lists both, with RPC as a fallback that takes over automatically when the primary source stops progressing. On 4 September 2026, Robinhood Chain stopped producing blocks for about 14 minutes. From the indexer's point of view HyperSync had stopped advancing, which looks the same as a source outage, so it failed over to its configured Alchemy RPC endpoint. That endpoint had also stopped advancing, because the chain itself had, and the indexer waited. When the chain resumed and the indexer continued indexing, it was still on the RPC source and continued ingesting from Alchemy for a period before switching back to HyperSync. The invalid data arrived in that window, from a node that had just come through a sequencer halt and was in an abnormal state.

The bad data originated on Alchemy's side, in a node that had just come through a sequencer halt, and nothing in the responses would have let the indexer tell. Data read through HyperSync had passed the root checks described below, while data read through the RPC fallback could not be checked in the same way, because a filtered RPC response cannot be. We are building a solution for this class of downstream bad data: when an indexer is reading the chain head from RPC, issue additional HyperSync requests so that the RPC data can be verified against HyperSync once it is available, and the fallback path inherits the same guarantee as the primary. That is not built yet.

HyperEVM system transactions, August to September 2026. HyperEVM places system transactions, which move funds from HyperCore into the EVM, at the front of blocks. The node implementation used by Dwellir and Alchemy returns these as ordinary transactions in eth_getBlockByNumber, with zero gas used, and returns receipts for them. The chain's official RPC does not return them, and neither did Chainstack. The block header's transactionsRoot and receiptsRoot were identical from every source and matched the view without the system transactions. For at least a week, roughly one in three blocks at the chain head failed HyperSync's root check on first fetch from Dwellir and was patched from Chainstack. Whether the official RPC hides these transactions or the other node build surfaces them is a question about HyperEVM's semantics, not about either provider. What the root check establishes is only that they are not part of what the block producer committed to.

Arbitrum receipt field casing, August 2026. On Arbitrum One, Chainstack's fleet ran a stock Nitro release that serialised one receipt field on eth_getBlockReceipts as L1BlockNumber, while Dwellir, dRPC, and the Arbitrum Foundation's public endpoint, on the next Nitro release, returned it as l1BlockNumber. The values were identical. Only the key's capitalisation differed, and a strict parser reading the two providers would see a field present on one and absent on the other. The difference came from the node software version, and the upstream release resolved it.

None of these cases produced an error from the provider. Each returned well-formed JSON, and the main signal was disagreement between sources or with the header's commitments.

Verifying with complete block data

Because HyperSync stores every block in full, it is able to perform these root checks. To serve an arbitrary filtered query later, it has to ingest every transaction, receipt, and log, and on some chains every trace. That happens to be the input the root check needs, so the check adds relatively little once the data is there.

Every batch of blocks is validated before it is written to storage or served. At a high level the checks fall into three groups.

• Cryptographic commitments. For each block, a complete receipt is rebuilt for every transaction from its stored receipt fields, such as status, cumulative gas used, and bloom, together with the logs that belong to it, and the receipts trie root is recomputed and compared with the header's receiptsRoot. The transactions are re-encoded as signed envelopes across the standard transaction types, and the transactions trie root is compared with transactionsRoot. Each block's parentHash is checked against the hash of the block before it, including across batch boundaries.
• Structural consistency. Block numbers are sequential with no gaps, transaction and log indices are contiguous, cumulative gas used adds up transaction by transaction, and gas used does not exceed the gas limit.
• Cross-references. Every transaction, log, and trace carries the hash of the block it belongs to, and every log and trace points at the transaction that produced it.

The root checks are the ones most relevant to completeness, and they are tested against real mainnet blocks. Removing a single log, or a single transaction, from a block produces a root mismatch.

When a check fails, HyperSync refetches rather than accepting the batch. It keeps track of which source served each block, transaction, receipt, and trace, refetches the affected block or transactions from a different source, excluding the sources that served the failing data, validates the refetched data again, and patches it into the batch. After a few patch rounds without success, the range is marked unverified and scheduled for backfill. On the ingest leader that data can still be served until the backfill lands, so the guarantee is that unverified ranges are known and tracked, not that they never reach a query.

Recomputing a root only works if the reconstruction reproduces the chain's exact encoding. On Ethereum mainnet that is mostly mechanical. On several other chains, system or deposit transactions with non-standard types and receipt formats, and hard forks that change which transactions the tries include, need to be handled differently, and a fair amount of the engineering effort goes there. On chains where a transaction type cannot yet be reconstructed, the root checks are held back until it can be, and the structural and cross-reference checks still run.

The validation is aimed at faulty sources, such as nodes with corrupted stores or providers serving mixed views, rather than at a source constructing a deliberately consistent but false block. In practice, disagreement between providers has been the signal in the cases we have seen.

Where a proxy sits in the stack

It is instructive to compare this with what can be done at the RPC proxy layer, because that is where many teams look to solve the reliability problem. eRPC is a thorough open-source example, and its authors have thought carefully about this question.

eRPC has three mechanisms for data correctness.

Block-tip and availability enforcement, always on. It prevents eth_blockNumber from going backwards across upstreams, pre-screens eth_getLogs ranges against each upstream's known height, and treats null responses for tagged blocks as retryable (docs). This addresses lagging nodes, which is a common failure in practice.

Consensus, opt-in. Requests are fanned out to several upstreams and responses are grouped by a hash of their canonicalised JSON. A result wins when enough upstreams agree (docs). This is a majority vote, not a cryptographic check, and eRPC's own documentation is explicit about the assumption. Consensus "catches a minority bad upstream", and the case it cannot see is "when every serving upstream returns the same wrong value".

Data-integrity checks, opt-in, shipped in eRPC 0.2.0 on 31 August 2026. This module does perform cryptographic recomputation. For eth_getBlockByNumber and eth_getBlockByHash it recomputes the block hash from the header and, when the response carries full transaction objects of types it can model, the transactions root from the transaction bodies. Hash-only, empty, and unsupported transaction lists are skipped rather than checked. For eth_getBlockReceipts it recomputes the receipts root and compares it with a header fetched by block hash (checks_recompute.go). It also recovers the sender from the signature for eth_getTransactionByHash. The implementation uses go-ethereum's DeriveSha and StackTrie, and it is conservative: any header or receipt with a field the reference encoder does not know is skipped rather than rejected.

That covers block-shaped responses. For eth_getLogs, which most indexers depend on, the situation is different, and eRPC's specification says so directly (specs/data-integrity/getlogs-receipt-crosscheck.md). It describes eth_getLogs as "the one method that cannot be validated intrinsically", because "every other check recomputes a commitment from the response itself (block hash, transactions root, receipts root, logs bloom)", whereas a getLogs response is "a filtered subset of logs across a block range" for which "there is no root to recompute and no self-contained invariant". The consequence, in the specification's words: "an upstream that silently drops logs (the single worst failure mode for an indexer) passes every existing check."

eRPC's answer is two checks in checks_getlogs.go. The first confirms that every returned log matches the requested filter and range, which catches fabricated or out-of-range logs but says nothing about missing ones. The second compares the response against block receipts that happen to be in an in-memory cache from earlier, unrelated traffic. It does not fetch receipts in order to perform the comparison. When the receipts are not cached, the block is skipped, and the specification lists the consequence under what the check misses: "ranges not in cache (large cold backfills); consistent drops from a single bad source that fed both sides."

None of this is intended as a criticism of eRPC. It is a well-engineered system that publishes its own false-positive rates and is candid about its assumptions. The limitation seems structural. A proxy that forwards a filtered log query sees a filtered response, and verifying that response would mean fetching the block's full receipts, which is much of the work the proxy exists to avoid. A proxy can reasonably verify completeness only when something else has already fetched everything.

An ingestion system that stores whole blocks has already done that work. That is most of the argument. The verification is less a feature added to HyperSync than a property that follows from holding the complete data, and the engineering effort goes into making the reconstruction correct on each chain rather than into obtaining the inputs.

Practical implications

For teams reading data through RPC, the practical advice follows from Bürgel's script. Completeness of eth_getLogs is best established by comparison with a second, independent source, and ideally against the block's receipts rather than another filtered query. If an indexer's correctness matters, it is worth making that comparison routine rather than doing it after a discrepancy is noticed.

For teams reading through HyperSync, the root checks run on every block before it is served, on every chain where the transaction encodings can be reconstructed. A query's results are drawn from data that matched the block header's commitments at ingest time, or from a range that is recorded as unverified and scheduled for backfill.

References

• Wood, G. Ethereum: A Secure Decentralised Generalised Transaction Ledger. Section 4, "Blocks, State and Transactions". ethereum.github.io/yellowpaper
• EIP-658: Embedding transaction status code in receipts. eips.ethereum.org/EIPS/eip-658
• EIP-2718: Typed Transaction Envelope. eips.ethereum.org/EIPS/eip-2718
• EIP-7919: Pureth Meta. eips.ethereum.org/EIPS/eip-7919
• Ethereum Execution APIs, eth_getLogs. ethereum.github.io/execution-apis
• Bürgel, S. "Ethereum needs Standards-Punk". ethresear.ch, 5 October 2025. ethresear.ch/t/ethereum-needs-standards-punk/23151
• Bürgel, S. "1.33.1 not returning event logs in some range". NethermindEth/nethermind #9305, 16 September 2025. github.com/NethermindEth/nethermind/issues/9305
• Nethermind 1.33.1 release notes. github.com/NethermindEth/nethermind/releases/tag/1.33.1
• All Core Devs Testing call 57 agenda, 13 October 2025. github.com/ethereum/pm/issues/1756
• "Broken eth_getLogs responses on RPC endpoints". hoprnet/hoprnet #7437, 2 September 2025. github.com/hoprnet/hoprnet/issues/7437
• "No log response for eth_getLogs Gnosis Mainnet Archival". erigontech/erigon #16613, 13 August 2025. github.com/erigontech/erigon/issues/16613
• "RPC methods return empty responses during snapshot creation". erigontech/erigon #16364, 30 July 2025. github.com/erigontech/erigon/issues/16364
• "eth_getLogs returns empty results for historical blocks on Gnosis Chain". NethermindEth/nethermind #9178, 20 August 2025. github.com/NethermindEth/nethermind/issues/9178
• "Missing results from eth_getLogs request". besu-eth/besu #1153, 25 June 2020. github.com/besu-eth/besu/issues/1153
• "missing logs in eth_getLogs". ethereum/go-ethereum #18198, 28 November 2018. github.com/ethereum/go-ethereum/issues/18198
• Wang, W. and Van Cutsem, T. "Depermissioning Web3: a Permissionless Accountable RPC Protocol for Blockchain Networks". arXiv:2506.03940, June 2025. arxiv.org/abs/2506.03940
• eRPC integrity checks documentation. docs.erpc.cloud/config/failsafe/integrity
• eRPC, architecture/evm/integrity/checks_recompute.go and checks_getlogs.go, and specs/data-integrity/getlogs-receipt-crosscheck.md, at tag 0.2.0 (31 August 2026). github.com/erpc/erpc/tree/0.2.0