A Price That Goes Backwards Is a Liability: Building a Sub-80ms Market-Data Fan-Out in Nineteen Days

A trader looks at a number on a screen and commits capital against it. That sentence is the entire reason this project was harder than it looked.

When the client first described the work, they framed it as a throughput problem. They're a boutique commodities and crypto derivatives desk; the fund was scaling headcount, the existing terminal buckled under the new seat count, and there were roughly four weeks before those seats went live. Reasonable enough as a brief. What took a couple more conversations to surface was the requirement underneath the performance one, and that one was much less forgiving. The prices had to be fast, and on top of that it had to be structurally impossible for a price to move backwards on screen. Fast and monotonic. Those two goals pull against each other, and almost all of the interesting engineering over the next nineteen working days happened in the gap between them.

The shape of the problem

The numbers first, because they set the shape of everything after. Six venue connections, roughly 2,400 tracked instruments, normalized ingest peaking around 120,000 ticks per second and spiking past 190,000 around scheduled macro prints. Fan-out target was 3,500 concurrent terminals. We provisioned for 6,000 and have since observed a 4,200 peak.

The first real design decision was conflation. Rather than pushing every tick, the fan-out tier collapses per-symbol state into a 40ms frame: 25 frames per second per connection, which at peak works out to about 105,000 WebSocket frames per second across the tier. Twenty milliseconds of mean added latency bought an order-of-magnitude reduction in frame count — 25Hz is well above what a human can act on, trading something imperceptible for something measurable. The exception is the instruments currently loaded in a trader's order ticket. Those ride a priority channel that bypasses conflation entirely.

tick-to-glass budget (p99 target: 80 ms)
────────────────────────────────────────────────────
venue timestamp → feed-handler decode          4 ms
normalization + per-symbol sequencing          2 ms
conflation frame boundary (0-40 ms)           20 ms   (mean)
encode + fan-out enqueue                       3 ms
kernel → WAN (RTT/2, median trader)           14 ms
client decode + store commit + paint           9 ms
────────────────────────────────────────────────────
budgeted p99                                  52 ms
measured p99 @ 4,200 connections              71 ms

Terminals connect through an L4 load balancer, not L7. An L7 proxy re-frames and re-buffers WebSocket traffic, which added measurable latency and, more annoyingly, destroyed our ability to reason about when bytes actually left the box.

Monotonic by construction

The first production-shaped failure was not a crash. It was flicker. A price would tick to 1,842.75, then briefly show 1,842.25 again, then correct. Cosmetically minor, commercially not, because a trader who stops trusting the tape stops using the terminal.

I spent most of a day convinced the deduplication logic was broken. It wasn't. The cause was our own redundancy, working exactly as designed. Each venue is consumed by two feed-handler processes running active/active, so failover stays invisible. Exact duplicates get deduplicated by venue update id, but under load the two handlers drift — we measured that drift at 6 to 9 milliseconds. A last-write-wins broadcast will happily apply the older of two versions if it arrives last. Arbitrating on wall clock doesn't help either: NTP-disciplined hosts still sat 1-3ms apart, and timestamps from different venues aren't comparable in the first place.

So ordering had to be per key and causal rather than temporal. Symbols are hashed across sequencer lanes, and a single writer per lane gives a total order per symbol with no global coordination. Every tick carries an (epoch, seq) version, anchored to the venue's own update id where one exists and synthesized where it doesn't. The epoch persists and increments on feed-handler restart, so a freshly started process can never emit a counter that looks like stale data.

// One writer per symbol lane ⇒ total order per symbol, no global lock.
// Comparison is lexicographic on (epoch, seq); a restarted feed handler
// bumps epoch, so its reset counter can never masquerade as fresh.
function admit(book: SymbolState, tick: Tick): boolean {
  const cur = book.version;                          // { epoch, seq }
  if (tick.epoch < cur.epoch) return false;
  if (tick.epoch === cur.epoch && tick.seq <= cur.seq) {
    metrics.sequenceRegression.inc({ venue: tick.venue });
    return false;                                    // never applied
  }
  book.version = { epoch: tick.epoch, seq: tick.seq };
  book.apply(tick);
  return true;
}

The identical guard runs in the terminal SDK too, because a client can be moved between fan-out nodes on reconnect and receive a snapshot concurrently with in-flight deltas from the old node. Snapshots carry a version; deltas at or below it are discarded on arrival. The sequenceRegression counter turned out to be the most valuable metric we shipped, running at roughly 340 per minute at peak — 340 flickers we had previously been rendering straight to human beings.

A timestamp tells you when somebody saw a thing. A sequence number tells you what actually came after what. Only one of those is safe to put in front of a trader, and it isn't the clock.

The open is not Poisson

Staging was green, and staying green was the problem. We had driven synthetic load at twice the mean production arrival rate and watched everything hold. Then the session open would arrive, the tier would wobble, connect times ballooned, and a fraction of terminals sat on a spinner for ten to fifteen seconds.

The trouble is that trading-desk traffic is correlated. At the ring open, and again at scheduled data releases, thousands of terminals reconnect and resubscribe inside the same second or two. A Poisson arrival process with a matching mean never produces that, which is exactly why our load test kept assuring us we were fine.

So we stopped generating load and started replaying it. Six weeks of anonymized connection-timestamp logs, reduced to a per-second connect/disconnect arrival trace plus the real distribution of subscription-set cardinality, then driven against staging by a harness spread over twelve hosts. The real trace, at 1x its own mean, broke staging in about 900 milliseconds on the second run. I'd be lying if I said that wasn't a relief. A bug you can reproduce in under a second beats one traders complain about once a day.

Three distinct causes fell out.

First, token verification. jwt.verify(token, pemString) re-parses the PEM into a key object on every single call, roughly 0.9ms of CPU each, synchronously, on the event loop. Four thousand of those in a burst is about 3.6 seconds of blocked loop, during which nothing else gets accepted. Hoisting crypto.createPublicKey() to module initialization took it to ~0.05ms. Moving the verify itself onto the libuv threadpool and memoizing verified claims by token fingerprint took most of what was left to zero, since a reconnect storm presents the same 15-minute tokens repeatedly — about a 97% hit rate.

Second, snapshot hydration. Each resubscribe built its snapshot per connection per symbol from Redis and encoded it — roughly 150,000 encodes inside the burst window. We replaced that with one pre-encoded binary snapshot per symbol, invalidated by version, so hydration became a refcounted buffer reference. Hydration p99 went from 340ms to 4ms.

Third, and this is the one that actually worried me, metastability. The accept backlog overflowed, clients hit their 10-second timeout, and every timed-out client retried, which made the second wave larger than the first. Once a system does that, it won't recover on its own — somebody has to turn things off. We added a token bucket on the upgrade path (1,200 upgrades/sec sustained, 2,500 burst) that returns 503 with Retry-After rather than queueing, full-jitter exponential backoff in the client, and a server-issued reconnect_after_ms in the close frame so a rolling deploy can deliberately stagger its own herd. At 2% of connections per 250ms you drain 4,200 terminals in under a minute without a stampede.

If your load test's arrival process is smoother than production's, you're measuring throughput and calling it load. Nobody's users show up one at a time.

The p99.9 you cannot see

Traders kept telling us the tape "freezes for a beat." Our APM insisted nothing was happening: p99 flat, error rate zero. Averaging will do that to you. One-minute rollups, means of means, and 5% span sampling will hide a 55ms stall that occurs every eight seconds, because that stall moves a 60-second mean by well under a millisecond.

So we went and built the instrument we should have had on day one. HDR histograms, three significant figures, 1µs to 60s, roughly 48 KB of fixed memory each, zero allocation after initialization, recorded per tick rather than per request, at five pipeline stages, rotated and serialized every ten seconds. The property that actually mattered is that HDR histograms merge additively and without loss, which meant that a day later we could reconstruct the exact p99.9 for any forty-second window we felt like interrogating.

We were also honest about clocks, which is easier to skip than to do: chrony with hardware timestamping held the ingest LAN under ~100µs of offset, and for the terminal side we reported RTT/2 on a sampled 0.5% cohort as an explicit upper bound rather than pretend we had synchronized clocks on people's laptops.

Then we overlaid perf_hooks GC entries on the latency timeline, and the answer was just sitting there waiting for us. Every excursion above 120ms sat inside a mark-compact window: 40-70ms pauses recurring roughly every eight seconds at peak. We had spent days quietly suspecting the network, and it had been our own heap the whole time.

The allocator was the bottleneck

The v1 fan-out loop did the obvious thing: for each changed symbol, for each subscriber, build a message object and serialize it. At peak that is around 1.5 million JSON encodes per second and about 1.6 GB/s of short-lived garbage.

Here's the part that surprised me. High allocation on its own is survivable, because young-generation scavenges are cheap when everything dies young. Our problem was that the garbage kept refusing to die young. Buffers handed to backpressured sockets stayed reachable across a scavenge, got promoted into old space, and old space then filled at hundreds of megabytes per second. The slowest connections on the tier were manufacturing GC pauses for every other connection — a fairly grim way to run a shared service.

The rewrite had four parts.

We hash each connection's subscription set once at subscribe time, so within a frame all connections sharing a signature share one encoded buffer. On the desk's actual watchlist distribution, 4,200 connections collapse to about 260 distinct payloads per 40ms frame, roughly 6,500 encodes per second instead of 1.5 million.

We also disabled permessage-deflate on the market-data channel, because a per-connection sliding-window compression context makes a shared buffer impossible by construction, and replaced it with a static-dictionary binary codec (symbol ids interned to uint16, prices as scaled int64 deltas). 1.1 KB of JSON became about 310 bytes, and egress dropped from ~115 MB/s to ~33 MB/s.

Third, we got the writes off the sequencing loop. Node cannot hand a live TCP socket to a worker_thread, since handles aren't transferable across that boundary, so the fan-out tier ended up as processes rather than threads: four writer processes per host with SO_REUSEPORT so the kernel distributes accepts and no master process sits in the path. One sequencer process per host encodes each frame exactly once and publishes it into an mmap'd ring buffer in /dev/shm (a small N-API shim, since SharedArrayBuffer does not cross process boundaries). Worker threads still earn their keep, but only for CPU work that never touches a handle.

Fourth, bounded queues. Each connection gets a 64-frame outbound ring with drop-oldest. That's only safe because the payload is state-replacing rather than an event log, and only if the client can tell it missed something. So a drop sets a gap flag, and the next frame for that connection is a full snapshot at the current version rather than a delta. A slow terminal degrades to a lower refresh rate. It never renders a delta against a stale base.

An unbounded queue isn't backpressure, it's a slower way to run out of memory. I'll take bounded and lossy over unbounded and honest-until-it-dies any day, as long as the client is told what it missed.

Allocation fell from ~1.6 GB/s to under 80 MB/s, mark-compact from every eight seconds to roughly every four minutes, and p99 from 210ms to 71ms.

Ninety-six hours of hostile weather

Trader connections churn all day. Laptops sleep, VPNs drop, sessions get force-quit on the way out the door. A four-hour QA cycle proves nothing about that class of behaviour, so before go-live we ran a 96-hour soak replaying real churn patterns at 1.3x production rate.

At hour three, an invariant tripped, and my first thought was that we had broken something worse than whatever we'd just fixed. We had instrumented a cheap 30-second consistency check: the sum of subscriber-set cardinalities across the symbol index must equal the sum of per-connection subscription counts. It went non-zero and climbed monotonically.

There were two teardown paths, which in hindsight was one too many. A graceful close ran through the close handler into unsubscribeAll(conn), which walks the per-symbol subscriber sets. An abrupt close, meaning a peer RST, roughly 6% of disconnects in this population, hit the error handler first; that destroyed the socket and deleted the connection-table entry. close then fired as expected, but its first line was const conn = connections.get(id); if (!conn) return;. The entry was already gone, so cleanup was silently skipped for every abrupt disconnect — two code paths, one deleting the index the other depended on.

By hour sixty, about 19% of the entries in the hot subscriber sets pointed at connections that no longer existed, and staging p99 had drifted from 68ms to 141ms. Total leaked memory was around 90 MB, invisible against a 4 GB heap — a latency regression long before it would ever have become a memory alert, and in production it would have surfaced during the desk's first live week, precisely when you don't want to be reading heap dumps. The fix was single ownership: one idempotent teardown(id, reason) reachable from every path, ordered so index cleanup happens before the connection entry is dropped. The invariant check still runs in production, and it costs about 2ms every thirty seconds.

Gauges tell you what a number currently is. Invariants tell you the moment your mental model of the system stopped being true. We only caught this one because a subtraction that should always come out to zero didn't.

What shipped

We went live on day nineteen of a four-week window. At 4,200 concurrent terminals, tick-to-glass runs p50 11ms, p99 71ms, p99.9 128ms, with headroom validated to 6,000 connections. Across roughly 150 trading sessions since go-live there has not been a single stale or rolled-back price incident, while the sequenceRegression counter continues to discard around 340 out-of-order ticks per minute at peak. The ordering discipline is doing real, continuous work, and the desk never sees it.

Rust might have taken another 10-15ms off the tail. With nineteen days and a TypeScript SDK sharing the wire codec, that wasn't the constraint that mattered — picking which constraint matters is most of the job. The invoice said latency. What the desk actually bought is the ability to say that when a number changes on their screen, it changed because the market changed. Everything above is what that costs under load.