Designing for the Link You Don't Have: Offline-First Coordination for an Autonomous Survey Fleet

At 1,900 metres, in a steep-sided valley on a high-voltage transmission corridor, one of the operator's VTOL aircraft logged a pack cell-imbalance fault severe enough to arm a precautionary landing. The ground station found out thirty-eight minutes later, when a sister ship crested the ridge and re-acquired LTE.

That was the good outcome.

The bad one is the reason we were brought in. On an earlier sortie, an event of the same class never arrived at all. It had been written into a fixed-size replicated buffer, and that buffer did what fixed-size buffers do: it evicted in FIFO order, which meant a battery-critical alert got discarded behind several hundred megabytes of entirely uninteresting position samples. Nobody had written a bug. The thing did exactly what it was built to do, which is the part that stays with you.

The client flies corridor survey, transmission lines and pipeline right-of-way, across mountainous terrain. Just over forty airframes, typically eighteen to twenty-four of them airborne at once on independent missions. Sorties run 60–90 minutes along corridor legs of 20–140 km. And for roughly a third of a typical sortie there is simply no usable link home: terrain shadowing down in the valleys, plain distance at the far end of a leg, plus a handful of coordinated radio-silence windows near controlled airspace where the operator is contractually obliged to go quiet.

We put one sentence on the whiteboard on day one and never moved off it: intermittent connectivity is not a failure mode on this platform. It is the operating condition. Any design that treats "link up" as normal and "link down" as an exception is wrong for a third of every flight.

Chapter one: ordering events when the clock is lying to you

Reconciliation is meaningless without a defensible notion of before, and wall-clock time is not that notion. Each aircraft disciplines a TCXO against the GNSS PPS, which works beautifully right up until GNSS itself degrades. Canyon multipath, poor geometry (we treat HDOP > 4 as untrusted), a deliberate denial exercise, and from there the aircraft is coasting on the oscillator. Across the thermal profile of a real sortie, +31 °C on the pad down to −8 °C at altitude, we measured up to ~90 ms of accumulated error over a three-hour coasting stretch.

Ninety milliseconds sounds harmless. Now picture two aircraft executing a deconfliction manoeuvre, logging related events 200 ms apart, and a merged timeline at the ground station that has to explain to a safety investigator which one happened first. I have sat in that kind of review. "We're fairly confident" is not a sentence you want to be the one saying.

So the event log moved onto hybrid logical clocks: 48 bits of physical milliseconds plus a 16-bit logical counter, which keeps Lamport causality intact while staying close enough to wall time that the incident tooling stays readable by a human. The part doing the real work, though, is the bound. HLC assumes a maximum skew, we set ε_max at 2 s, and any inbound timestamp beyond it is treated as a clock anomaly, quarantined and surfaced rather than silently absorbed. One aircraft with a sick oscillator must not be able to drag the whole fleet's logical time forward with it. Each replica also carries a compact version vector, so a sync partner can work out what it is actually missing instead of replaying everything it has.

Chapter two: merges that are only allowed to add

The event log is a delta-state CRDT: a causally-tagged grow-only set, where each event carries a dot (node_id, seq) and merge is set union against a causal context. Safety-classified events live only here. Battery and thermal faults, collision-avoidance activation, geofence breach, GNSS integrity loss, autonomy fallback transitions, mission abort. There is no code path anywhere in the system where two replicas disagreeing about a safety event resolve to one of them.

Mutable operational state is a different animal. Current waypoint assignment and mission phase really are last-writer-wins-shaped, and we still didn't use LWW: concurrent writes survive as a conflict set in a multi-value register and get pushed to the operator console. LWW is permitted for cosmetic, derived fields and nowhere else.

def merge(local, remote):
    for e in remote.events:
        if e.dot in local.causal_ctx:            # seen, or already GC'd
            continue
        if not hlc_within_skew(e.hlc, local.hlc, EPS_MAX):
            local.quarantine(e)                  # bad clock: keep, don't order
            continue
        local.insert(e)                          # G-Set: insert only, never replace
        local.causal_ctx.add(e.dot)

    for key, r in remote.registers.items():
        assert r.klass is not EventClass.SAFETY  # enforced at ingest and here
        local.registers[key] = mv_register_merge(local.registers[key], r)

    local.hlc = hlc_max(local.hlc, remote.hlc)
    return local

Anti-entropy has to finish inside a contact window measured in seconds, so shipping the log was never on the table. We reconcile with a Merkle prefix tree keyed by (node_id, seq), exchanging range fingerprints and descending only into the subtrees that disagree. Closing a 90-minute divergence usually takes three to four round trips and around 12 KB.

If a field can be overwritten by a merge, one day it will be, and it'll pick the worst possible moment to do it. So safety data doesn't get a register. It gets a set.

Chapter three: deciding what to forget

512 MB. That is the cap on the replicated log, and it has nothing to do with storage; the aircraft has plenty of that. The cap is about relayability. The log has to stay small enough that a full reconciliation is feasible over a 2.4 GHz mesh link during a passing encounter. A log you cannot ship is not a replicated log, it's a diary.

Which means eviction is unavoidable. The original FIFO ring wasn't too small — it was the wrong shape. So we tiered it:

Tier 0 is safety-classified, and it is never evicted. Full stop. That guarantee is only safe because tier 0 is bounded at ingest by event coalescing: a repeating fault collapses into one event carrying an occurrence count and first/last HLC, so an oscillating sensor cannot flood the one class it is forbidden to drop. Promise "never evict" without a source-side bound and you have written yourself a denial-of-service.

Tier 1 is state transitions, task-allocation outcomes, link events. Evicted oldest-first, and only once tier 2 is exhausted.

Tier 2 is routine pose and attitude, and it isn't dropped, it's decimated: Ramer–Douglas–Peucker trajectory simplification with an error bound that rises as buffer pressure rises, from 2 m cross-track at 50% pressure out to 15 m at 90%. The flight's shape survives; its fidelity degrades smoothly. Anchored windows are exempt, meaning full-rate telemetry for ±10 s around every tier-0 event is preserved regardless of pressure, because the five seconds before a collision-avoidance trigger is the most valuable data on the aircraft and nothing gets to decimate it.

on_pressure(p):                       # EWMA over 5 s, p in [0,1]
    if p < 0.5: return
    eps = lerp(2.0, 15.0, (p - 0.5) / 0.4)          # metres, cross-track
    for seg in tier2.segments_outside(anchors(tier0, window=10s)):
        seg.simplify(rdp, eps)                       # lossy, shape-preserving
    if pressure() > 0.9:
        tier1.evict_oldest_until(0.85)
    invariant(tier0.retained == tier0.ingested)      # runtime + CI assertion

Chapter four: someone else's airframe as your transport

The mesh radio had been sitting on these aircraft the whole time. Nobody was using it as a data plane. We built a store-carry-forward layer with Bundle Protocol v7 semantics over a custom convergence layer on that existing 2.4 GHz radio, then went and measured what a contact actually looks like: median 26 s of usable link at roughly 5 Mbps effective when two aircraft pass within ~900 m line-of-sight.

Routing is where we got humbled. Epidemic flooding was the obvious first cut, and written down now it's the kind of thing you'd flag in someone else's design review in about four seconds: buffer amplification across the fleet crowded out exactly the tier-2 history the operator had asked us to protect. We learned it in the field instead, which is a more expensive classroom. What we settled on is binary spray-and-wait with L = 4 copies for tier-0 bundles and single-copy direct delivery for everything else, biased by a contact-graph heuristic. We already know each aircraft's planned corridor legs, so a bundle prefers a carrier whose remaining plan intersects a known coverage cell soonest.

BPv7 deliberately dropped custody transfer, so we implemented acknowledged hand-off at the application layer. When a carrier accepts custody, the originator downgrades its own copy from tier 0 to tier 1, still retained but now evictable. That is the moment where chapters three and four pay each other back. Ground-station delivery receipts propagate back through the mesh as tiny anti-entropy hints, so stale copies get reaped.

Chapter five: confidence, quantized

Everything so far has been about moving data around. This part is about the aircraft deciding things on its own, which is much harder to be relaxed about.

Corridor survey has a genuinely nasty perception problem. Conductors, static wires and guy wires are thin, low-texture, and nearly invisible to stereo at range. The onboard stack runs stereo depth, a monocular depth prior and a thin-structure segmentation head on a 16 GB Orin NX, inside a 15 W platform budget shared with the survey payload, at 25 ms end-to-end and 20 Hz.

Getting there cost 1.2 mAP: INT8 post-training quantization with per-channel scales, plus QAT on the depth head and distillation from a heavier teacher. That was fine, we had budgeted for it. What we had not budgeted for, and would never have caught without field data, is that calibration degraded four times worse than accuracy. Expected calibration error went from 0.021 to 0.088, and it degraded asymmetrically — worst in backlit haze and low light, which is precisely where you need the confidence score to be honest. The first time we put the field numbers next to the lab numbers I assumed somebody had mixed up the runs. Nobody had. A threshold tuned on lab data was simply meaningless.

So we re-fit temperature scaling post-quantization on field data stratified by illumination and terrain class, and gated on min(calibrated_confidence, ood_score), where the OOD term is a feature-space Mahalanobis distance against per-class covariances. About 0.4 ms, cheap enough to run every frame.

Below threshold the aircraft does not keep guessing. It walks a small, exhaustively tested state machine: NOMINAL → CAUTION (speed capped at 4 m/s, obstacle inflation widened) → HOLD (position hold, re-evaluate every 3 s) → CLIMB (to a surveyed-clear altitude above known corridor structure) → RTL. Entering a transition requires k-of-n agreement, 7 of 10 frames, 350 ms; leaving one requires a 2 s dwell. Chattering between autonomy modes is a hazard in its own right.

Tuning that threshold turned out not to be a hyperparameter sweep at all. It was a negotiation. Our first setting triggered an unnecessary RTL roughly every nine sorties, which the operator rejected, correctly: a re-flown corridor leg costs a crew-hour and a battery cycle, and these are people who fly for a living. So we mined 6,400 labelled degradation clips from real flights, built the ROC, and put the cost asymmetry on the table in writing. A missed thin-wire detection priced at roughly 400:1 against a spurious RTL, agreed and signed off by the client's safety team. We landed at one unnecessary RTL per ~60 sorties with no missed-detection incidents in the evaluation period.

A confidence threshold is just a cost ratio wearing a number. If your safety lead hasn't seen the ratio, you made their call for them and didn't mention it.

Chapter six: when the thing that's down is central command

For a while, the most frustrating failure in the fleet had nothing to do with radios at all. The scheduler was central. An aircraft that lost its link mid-leg would finish its assigned waypoints and then wait, burning endurance on a task queue it could no longer refill. The broken link wasn't the drone's link to a target. It was the fleet's link to the scheduler, and it stalled aircraft that were otherwise perfectly healthy.

We replaced it with a contract-net auction using CBBA-style consensus over whichever peers a given aircraft can currently reach. Bids are priced in watt-hours to complete rather than distance, drawn from the energy model using the EKF's estimated wind states and temperature-derated pack energy, because a 20 km leg upwind and the same leg downwind are not the same task. Bundle construction enforces diminishing marginal gain, which is what buys the convergence guarantees. Re-auction among reachable peers settles in under 2 s.

Assignments carry a lease, epoch plus expiry. A partitioned aircraft that cannot renew relinquishes its claim, and its reachable peers re-auction the work. On re-merge you can end up with duplicate coverage, and we chose that deliberately. Surveying 3% of a corridor twice is waste. A partitioned fleet that stops flying because it cannot reach quorum is an outage. In this domain, idempotent duplication beats blocking consensus every single time.

Stalled sorties requiring manual re-tasking fell from 6.1% to 1.0%, an 83% reduction, and what's left is now dominated by genuine hardware issues rather than coordination.

Chapter seven: making the failure repeatable

None of this is testable by flying more. You cannot put a GNSS denial event on the calendar, and you cannot sit around waiting for the right multipath geometry to turn up.

So we built a hardware-in-the-loop rig: real flight controller and real Orin running production firmware, an RF constellation simulator injecting multipath, degraded HDOP and full denial, an SDR injecting interference into the 2.4 GHz mesh band, and a link emulator computing path loss from a 10 m DEM with a Longley-Rice model. That last piece matters more than it sounds like it should. Random packet loss does not reproduce the correlated, geometry-driven outages that actually break these systems. Real shadowing takes the whole link down for four minutes as an aircraft descends behind a ridge, and it takes down the ground link and half the mesh neighbours at the same time, because they are all on the wrong side of the same rock.

Everything is deterministic: fixed-step sim clock, seeded RNG, recorded sensor traces. 214 scenarios run nightly across four rigs in about 46 minutes, and any PR touching sync or autonomy runs a 38-scenario safety subset in nine. The assertions are invariants rather than outputs: tier-0 ingested equals tier-0 delivered-or-in-custody, no causal-order violation at any replica, no undefined edge in the fallback FSM.

I'll be straight about why that rig mattered to me. Before it existed, the honest answer to "are you certain the safety class never drops?" was "we believe so, and we've reasoned about it carefully." Those are not the same sentence, and on an aircraft that difference isn't academic. The rig earned its cost in the first month. A three-hour coasting scenario surfaced a 16-bit logical-counter overflow that only appeared under coalescing storms, and a separate run caught a spray-and-wait copy leak where a carrier landed holding custody and never re-acquired a link, which we fixed with custody expiry and ground-station reissue. Neither would have been found by flying. Both would have been found by an incident.

The number that mattered

Over nine months of operation, 99.98% of 2.31 million safety-classified events reached the ground station. Median time-to-ground for those events fell from 41 minutes to 6.5, p99 from 2h07 to 22 minutes. The 0.02% we did not deliver were held on two airframes that were physically lost. No event was ever dropped by the buffer, the merge, or the relay — which was the entire point.

What the operator bought was not uptime. Uptime was never on offer; the terrain does not negotiate. What they bought was a bounded, stated answer to the only question that matters when an aircraft goes dark over a ridge: if something happened out there, do we find out, and how fast?

Everything else on the aircraft is allowed to degrade. That answer isn't.