I worked on a production off-chain service: a system that watches Bitcoin activity and coordinates downstream operations on Ethereum for a cross-chain interoperability system, with more than one independent operator watching and acting on the same chain activity. It’s a monitoring and coordination role, not custody or key management: the service observes Bitcoin, decides what that activity means, and triggers the corresponding action elsewhere. Getting that decision right, without producing duplicate downstream effects, in the face of a chain that can change its mind, turns out to be most of the engineering problem.
Why Bitcoin monitoring is hard#
If you’re used to Ethereum, Bitcoin’s quirks are easy to underestimate. Three things make chain-watching harder than it looks:
- Reorgs are a normal outcome, not an attack. A block your service already acted on can be replaced by a different block at the same height. This isn’t a rare edge case reserved for 51%-attack scenarios; short reorgs (one or two blocks) happen as an ordinary consequence of how proof-of-work chains resolve competing miners, and a service that isn’t built to expect them will eventually act on a block that stops existing.
- There’s no finality below N confirmations. Bitcoin doesn’t have an on-chain finality gadget the way some proof-of-stake chains do. “Final enough to act on” is a threshold you choose, not a guarantee the protocol gives you, and choosing it is a real trade-off between how long you wait and how much reorg risk you’re willing to eat.
- The same event can be observed more than once. During a reorg, a transaction can appear in a block, disappear when that block gets orphaned, and reappear in a later block once the chain resolves, sometimes with a different set of surrounding transactions. This isn’t unique to Bitcoin’s UTXO model, any chain re-observes activity across a reorg, but it’s the case this kind of service has to be built around. From the watcher’s point of view, that’s the same real-world deposit surfacing as multiple, slightly different observations over time.
None of these are solved by watching harder. They’re solved by designing for the chain telling you something, then later telling you something else.
Start with the invariants#
Before getting into specific mechanisms, it’s worth naming what actually has to hold, independent of implementation detail. A handful of invariants define what “correct” means for this kind of system:
- One canonical Bitcoin observation must not produce the same downstream effect twice.
- State derived from an orphaned block must be reversible.
- Restarting a process must not change the outcome.
- Adding another service instance must not change the logical outcome.
- One logical consumer processing an event must not prevent another logical consumer from independently processing it.
Everything below is one of these invariants, made concrete.
Four failure modes, four patterns#
Chain reorganizations#
The direct fix is a confirmation-depth threshold: don’t treat an observation as final until it’s buried under N additional blocks. But depth alone isn’t enough, because you still need to detect when a reorg happens within that window. In this design, the canonical-chain check runs before advancing the persisted tip: before doing anything else on a given tick, compare the block hash your service has stored at a given height against the chain’s actual hash at that height right now. A mismatch means the block you built state on is no longer canonical, and everything derived from it has to be rolled back before you continue.
graph TD
A[Start tick] --> B["Compare stored hash at
height H vs. chain's
current hash at H"]
B -->|match| C[Proceed: fetch new
blocks since last tip]
B -->|mismatch: reorg| D["Roll back state derived
from the orphaned block(s)"]
D --> C
Running that check before advancing the tip matters: check for a reorg only after fetching new blocks, and you can end up building new state on top of a branch that’s already been abandoned. A single-height comparison catches the common case (the reorg replaced the block right at your last checkpoint), but it’s a simplification: a deeper reorg needs the same idea applied over a window of recent heights, walking back until you find a height where the stored and current hashes actually agree, and rolling back everything after that common ancestor.
Retries#
Any call to an upstream data source, a node, an indexing API, can fail transiently: timeouts, rate limits, a dropped connection. Retrying a read is cheap, since nothing has happened yet that a duplicate read could compound. Bounded retries at the client level, a small retry budget with backoff, handle the common transient-failure case without changing application semantics. The failure mode that actually needs design is different: a retry issues the same operation again without knowing whether the first attempt’s side effect, recording an observation, triggering a downstream action, already landed on the far side before the client timed out or crashed. That’s not a retry problem anymore, it’s a duplicate-processing problem, and it needs its own answer.
Duplicate processing#
The same on-chain event can also reach your service more than once for reasons that have nothing to do with retries: once before a reorg, again after, or because two polling windows briefly overlap. Either way, the fix is the same: idempotency, not making duplicates impossible, but making them harmless. Give every observation a deterministic identity, protocol-specific rather than generic: for a Bitcoin output, that’s (txid, vout); for an EVM log, it’s (transaction hash, log index). Enforce uniqueness on that identity at the database level: it prevents duplicate observation records and gives the downstream path a stable deduplication boundary to build on, provided recording the observation and deciding to trigger the downstream effect stay correctly coupled.
Race conditions and consistency#
This is the failure mode that’s easiest to get wrong, because the instinct is to reach for a distributed lock or leader election: pick one instance to be authoritative, and have everyone else defer to it. That works, but it’s more machinery than the problem needs, and it also runs two different situations together that deserve separate answers.
Different logical consumers watching the same activity. One component might react to a Bitcoin deposit by updating a balance; another might react to the same deposit by feeding an entirely separate downstream process. Both need to see the same event independently. The pattern here is called Idempotent Consumer. Every domain event carries an idempotency key, enforced unique at the database level exactly as above. Each consumer then keeps its own dedup record for that event, keyed on the pair (consumer, event) rather than on the event alone, so whether one consumer has processed something is invisible to every other consumer. A minimal version of this is just an existence check; a fuller implementation tracks pending, done, or failed on top of it to support retries and observability. Either way, consistency between different consumers comes from that isolation, each one owning its own delivery record, not from making them take turns.
Multiple replicas of the same logical consumer. Running several instances of one consumer for throughput or availability is a different problem. They share the same work stream, and per-consumer isolation doesn’t separate them from each other, since they’re the same consumer. Two replicas can still race to dequeue the same pending row. Solving that takes a narrow, row-level claim at dequeue time, a SELECT ... FOR UPDATE SKIP LOCKED-style lock on one row, not the system-wide lock the earlier instinct reaches for, layered on top of the pattern above rather than something it gives you for free.
graph TD
E["Domain event
(idempotency key)"] --> U{"Unique constraint on
idempotency key"}
U -->|duplicate| R[Rejected: no-op]
U -->|new| S[Event stored]
S --> I1["Consumer A's dedup record
(own delivery state)"]
S --> I2["Consumer B's dedup record
(own delivery state)"]
I1 --> H1["Consumer A's handler
(idempotent)"]
I2 --> H2["Consumer B's handler
(idempotent)"]
One more thing both cases force you to confront: handlers have to be idempotent even beyond the uniqueness check, because delivery is at-least-once. If a handler finishes its work but the process crashes before its delivery record is marked done, that event gets redelivered. What makes redelivery safe isn’t any single technique: an upsert is the common shape when the effect is itself a piece of state (a balance, a status), but a conditional state transition (“only advance from pending to confirmed, never from confirmed itself”) or a separate idempotency record written atomically alongside the real state change both work too. The requirement is that reapplying the same operation lands in the same externally relevant business effect as applying it once, not that every handler looks like an upsert or that every last timestamp, audit record, or metric is byte-identical across replays.
Ethereum-side ingestion with Ponder#
The same principle applies on the Ethereum side: canonical chain data should become durable, queryable state rather than being reconstructed ad hoc during every decision. In this system I used Ponder for EVM indexing, feeding the same persistent store the Bitcoin-side decision logic reads from, so “did this already happen” is always a database read, not a re-derivation from raw events. I’ve written a deeper, code-driven walkthrough of how that indexing layer works, including backfills, reorg handling, and crash recovery on the Ethereum side specifically.
Takeaway#
Correctness in a chain-watching service is dominated by handling the chain changing its mind, not by the happy-path event-processing logic. Reorgs, retries, and duplicate observations aren’t corner cases to patch in later. A confirmation-depth threshold with reorg rollback, protocol-specific idempotency keys enforced at the database level, and per-consumer delivery state layered with per-replica claims where they’re actually needed are what keep the system’s behavior predictable under reorgs, retries, duplicate delivery, process crashes, and concurrent consumers, rather than merely correct on the happy path.