Share
Share
Share
Share
On a Tuesday morning in July 2023, a clearing engineer at the Federal Reserve watched a payment settle across three data centers in under fifteen seconds, and a system that had taken nearly a decade to build quietly began moving money for the United States twenty-four hours a day. That is the practical face of distributed systems in finance: not a whiteboard diagram, but a settlement that has to be correct, ordered, and survivable across regions. This piece walks through what makes those systems hard.
Why finance breaks the easy distributed system patterns
Most consumer software can tolerate eventual consistency. A like count that lags by a second is fine. A bank balance that lags by a second is a regulatory incident. Finance imposes three constraints at once: strong consistency for the ledger, ordering guarantees for payments, and low latency for matching and pricing. Almost every architectural choice in a US finance system is a negotiation among those three.
The ledger constraint is the hardest. Double-entry accounting requires that for every credit there is a matching debit, and that the sum across all accounts is invariant. Translating that to a distributed setting means either a single writer with replicas, or a consensus protocol that orders writes globally. Both have costs, and neither makes the constraint disappear.
The ordering constraint shows up in payments. If two transfers race against a single account with $100, only one can succeed. The system that says yes first has to be the system that says yes everywhere. Without total order, a balance can briefly go negative on one node and positive on another, and reconciliation becomes a manual job for back office staff. The combined demand for engineers who can reason about these failure modes is one reason US software developer employment is projected to grow 17 percent through 2033 according to the BLS Occupational Outlook Handbook, with finance employers concentrated in New York and Chicago.
Consensus protocols US finance actually runs
Raft is the dominant consensus protocol in newer US finance systems. It elects a leader, replicates a log to followers, and commits an entry once a quorum has acknowledged it. CockroachDB, etcd, and TiKV all run Raft, and each appears in production at multiple US fintechs and banks. The protocol is well documented, easier to reason about than Paxos, and supported by widely used libraries. US fintechs routinely operate Raft clusters of three or five nodes spread across availability zones, with quorum reads served from the leader and stale reads from followers when a use case can tolerate them.
Paxos and its variants still anchor older systems. Google Spanner, which underpins the back end of several US payment companies, uses Multi-Paxos with TrueTime to provide externally consistent transactions across continents. The advantage is global ordering with bounded clock uncertainty. The cost is that Spanner is a managed service, not something a bank can rebuild in house. Several US banks now run Spanner or its open compatible peer YugabyteDB behind their newer ledger services, with Raft-based systems used for control-plane work.
Two-phase commit, the textbook protocol for distributed transactions, is mostly avoided for new payment systems. A coordinator failure between the prepare and commit phases leaves participants holding locks, and the recovery path is operationally painful. Modern US finance systems prefer saga patterns, idempotent operations, and compensating transactions, which trade absolute atomicity for survivability.
What FedNow and the exchanges actually built
FedNow, the Federal Reserve’s instant payment service that went live in July 2023, is the most visible US distributed finance system of the decade. It settles payments in seconds, twenty-four hours a day, seven days a week, across more than 1,300 participating institutions as of early 2026. The system is built on a clustered architecture with synchronous replication across Federal Reserve data centers, designed so that no single site failure can stop settlement. The Federal Reserve’s FedNow service page describes the participation model, settlement finality, and the role of liquidity management transfers between banks.
The exchanges face a different distributed problem: deterministic ordering at microsecond scale. The Intercontinental Exchange’s matching engines, the Nasdaq INET system, and NYSE’s Pillar platform all run colocated, single-threaded matching cores fronted by distributed gateways. The matching itself is intentionally not distributed because total order at nanosecond resolution is easier to guarantee in one process. The distributed parts are the gateways, the market data fanout, and the post-trade pipelines that feed clearing. Each US equity exchange publishes order book updates to thousands of subscribers, and the fanout layer has to deliver every update in the same order to every consumer, or arbitrage strategies break.
For real-time risk and clearing, the model flips. The Options Clearing Corporation and DTCC run distributed risk calculations across clusters that aggregate position data from hundreds of clearing members. Those systems prioritize throughput and fault tolerance over single-process determinism, because the answer can be recomputed if a node fails.
Data structures that make distributed finance survive
Conflict-free replicated data types, known as CRDTs, are starting to appear in US finance for limited use cases. They let independent nodes merge updates without coordination, which suits scenarios like fraud feature aggregation across regions. They are not used for ledgers themselves, because monetary arithmetic does not commute in the way CRDT operations require. The same goes for fraud rule outputs, where conflicting signals from different regions are reconciled at a central decision layer rather than merged in place.
Event sourcing is the more common pattern for ledgers. Every state change is recorded as an immutable event, the current balance is a fold over those events, and replication is a matter of streaming the event log to every replica. Kafka, Redpanda, and Pulsar are the workhorse event logs in US finance, and Confluent’s Tiered Storage feature has become a near-default for keeping the full transaction history available without unbounded broker disks. The log itself becomes the source of truth, and any new service can be built by replaying it from the beginning.
Idempotency keys are the small but indispensable detail. Every payment API exposed by a US fintech accepts an idempotency key from the caller, so a retried request never produces a duplicate transfer. The pattern is encoded in Stripe’s and Plaid’s public APIs, and it is increasingly required by US bank partners during integration reviews. Engineering teams looking for more on the design tradeoffs can scan TechBullion’s payments coverage and the recurring cloud finance modernization notes.
What is changing in the next eighteen months
The shift to T+1 securities settlement, which took effect on May 28, 2024 under the SEC’s final rule, has already pushed brokerages, custodians, and clearing firms to rebuild parts of their distributed back ends. The SEC’s T+1 final rule release set the standard, and the operational changes show up as tighter affirmation windows, more aggressive overnight batch redesign, and stricter requirements on cross-system reconciliation. Several US custodians have publicly said T+1 forced a long-deferred replacement of legacy message queues. The reconciliation systems that used to run overnight now run continuously, and the engineering teams that own them sit on call alongside the trading desks.
The next move is T+0, sometimes called atomic settlement, where trade execution and settlement happen at the same instant. The technology to do it exists, the regulatory and liquidity questions do not yet. US clearing firms have signaled that any move to T+0 would require coordinated changes to margin calculations, central counterparty risk models, and intraday funding lines. TechBullion’s fintech news hub tracks the regulatory threads. The deeper question for US finance is whether the same distributed systems that now move retail payments in seconds can be trusted to move institutional securities in seconds, with the same audit trail. The next eighteen months of FedNow volume, T+1 stress, and exchange outage data will start to answer it. Engineering hiring patterns at US clearing firms suggest the industry is already preparing for the answer to be yes.
