Nobody Is Coming: 41,000 Custom Energy Endpoints, From Bare PCB to Real-Time Analytics

Behind a steel door in a medium-voltage switchroom, a board we designed is counting energy pulses into a hardware timer while its microcontroller sleeps at 2.6 µA. There are 41,318 of them across the operator's territory, bolted inside or beside energized switchgear.

From the first design review onwards we worked to a single assumption, and nothing that happened afterwards let us soften it: no technician would ever touch any of them again. Everything below is downstream of that one sentence: the layout discipline, the coulomb ledger, the bootloader's connectivity watchdog, the way we ended up splitting the ingestion pipeline in two. When nobody is coming, a bug is not a ticket. It is a truck roll, or a dead endpoint, forty-one thousand times over.

The build-versus-buy call, and what it actually costs to own

The client is a distribution utility with a heavy industrial customer base, and the brief fit in one sentence: settlement-aligned 15-minute interval energy at feeder and large-circuit granularity, everywhere, permanently. The obvious path was a commercial cellular metering modem, and we took that seriously enough to evaluate four of them before killing the approach on two independent grounds. Two grounds is a comfortable place to argue from, because you only need one of them to hold up under scrutiny.

Economics first. At €118 per endpoint for module, enclosure and antenna plus €1.85/month per SIM, 41,000 endpoints modelled to €15.9M across a ten-year program once an 18% replacement rate was included. The custom path modelled to €4.2M: €1.4M of NRE across four board spins, EMC lab time and tooling, a €37.40 landed board cost at volume, 287 private LoRaWAN gateways at €2,450 installed, and ten years of network and backend operations on top of all of it. You do not engineer away an €11.7M delta later. That number is the difference between the program existing and not existing.

The physics killed it independently, though it took fourteen months before we could say so with any authority. That is how long we left temperature and RF loggers sitting in 40 representative cabinets. Interior temperature ranged from −22 °C to +71 °C, peaking at +78 °C inside a south-facing pad-mount enclosure in August, and the modems we were evaluating were rated 0–55 °C. Worse, a grounded steel switchgear cabinet is a decent Faraday cage. Closing the door cost 19–31 dB, and at 22 of the 40 sites the LTE RSRP with the door shut sat between −103 and −117 dBm. That is the region where a modem holds a session right up until it doesn't, and every re-attach after that is paid for out of a battery. At 868 MHz with an external bulkhead antenna, the same enclosure loss is something you absorb with margin left over.

Choosing custom means choosing to own failure modes a pure-software team never meets: moisture ingress, solder fatigue, silicon errata, an EMC lab that says no, a reel of shunt resistors with a different tempco than the last reel. We knew that going in. It still surprised us.

Designing for a cabinet that fights back

The endpoint is a four-layer board carrying an STM32L451CE (Cortex-M4F, 512 KB flash). It runs an Arduino-framework build, which we picked for iteration speed and would pick again, with the low-power, DMA and metrology paths dropped down to the LL layer where cycles and microamps actually matter. Metrology is an ADE7880 poly-phase front end. The radio is an SX1262 on EU868, LoRaWAN 1.0.4, Class A.

That is the easy half of the description. The hard half is what happens around the board. Vacuum breaker operation in a switchgear cubicle produces prestrike and restrike bursts with dV/dt in the kV/µs range and radiated energy smeared from 100 kHz to well past 100 MHz. We designed toward the IEEE 1613 / IEC 61850-3 envelope and tested to IEC 61000-4-2, -4-4, -4-5, -4-6 and, the one that really matters here, -4-18 damped oscillatory wave.

Layout carried most of the load. We deliberately did not split the ground plane between analog and digital sections, because the classic split is an excellent way to turn a reference plane into a slot antenna. One uninterrupted plane on L2, return-current geometry controlled by component placement above it, the boost converter's hot loop kept small and far from the sense chain, and the RF section in its own shield can, fed by a coplanar waveguide with pi-filtered feedthroughs on every conductor crossing the can wall or the enclosure gland. CT secondaries arrive on twisted pair through a common-mode choke into a bidirectional TVS and an RC anti-alias corner well below the modulator's band.

None of which stopped us failing IEC 61000-4-4 at ±2 kV on the first lab visit. There were two causes and I had signed off on both: the AFE's SPI chip-select ran 40 mm parallel to the harvester's switching node, and the gland let common-mode current onto the sense pair. I had reviewed that routing twice and read straight past it both times. It took a burst generator and a failed test report to make me see what was in front of me. Re-route with a ground guard, add the choke, move the hot loop, and we passed at ±4 kV. But you never filter out 100% of it in a real cubicle, so the firmware assumes corruption:

• Every ADE7880 register transaction is wrapped in a CRC over the SPI frame, which catches bus corruption specifically.
• The AFE's own CHECKSUM register, a CRC across its configuration map, is polled every 64 accumulation windows. This catches the genuinely dangerous failure: an EMI event silently flipping a gain register bit, after which the unit reports plausible, wrong data forever.
• Reported RMS and power-factor values are the median of seven 200 ms line-cycle windows, with windows rejected on overrange, PLL unlock, or excess second-difference energy. That last test is an impulsive-noise detector rather than a checksum, because analog corruption is arithmetically valid and no checksum will ever object to it.
• Energy itself comes from a hardware pulse count, bounded by a physical plausibility clamp. We know the maximum possible CF rate at Imax, so any count above it is noise by definition, and pulse-derived energy is cross-checked against the register accumulators each interval with a 0.5% divergence tolerance.

Radio backoff is tuned to the same disturbance. Uplink loss correlated tightly with breaker operation, and a naive immediate retry drops you straight back inside the same burst envelope. So the first retry is deferred by a randomized 3–11 s, then exponential with full jitter, and the transmitter is simply inhibited for 400 ms after the metrology block raises a transient flag. Do not transmit into the storm.

Eight years on a single A-size cell

Eight years against 3.06 Ah of usable capacity works out to an average current budget of 43 µA. Everything in this section follows from that one number, and it is worth sitting with how little it buys you before reading on.

The cell is a Saft LS 17500, A-size lithium thionyl chloride, 3.6 Ah, paired with a hybrid layer capacitor to source the transmit pulse, because Li-SOCl₂ cells have high internal impedance and will sag under a 45 mA burst.

The key architectural move is that the battery mostly does not pay for measurement at all. Energy is an integral; you cannot compute kWh by sampling 200 ms every fifteen minutes and hoping the rest averages out. So the ADE7880 runs continuously from a dedicated harvesting CT, which delivers over 90 mW above roughly 35 A of primary current and therefore covers about 91% of feeder-hours, and its CF1 pulse output clocks an LPTIM counter while the MCU sits in STOP2. Waking every fifteen minutes is a counter read, not a measurement session. Below the harvest threshold the load is small enough that duty-cycled battery sampling at reduced resolution is defensible, and Class 1's error envelope widens below 0.05 Ib anyway.

Measured sleep current was 1.4 µA at 25 °C and 6.8 µA at 70 °C. Datasheet typicals are quoted at 25 °C; the cabinets are not 25 °C. Temperature-weighted against the logger data, sleep annualizes to 2.6 µA, and the full model came to 31 µA against the 43 µA ceiling. Comfortable. We believed that number right up until the network went away.

Then a fiber cut plus a misconfigured LTE failover APN took 63 gateways off the air for 31 hours, and 11,400 devices lost the network. LoRaWAN's ADR back-off did exactly what the spec says it should: after ADR_ACK_LIMIT uplinks with no downlink the device sets ADRACKReq, after ADR_ACK_DELAY more it steps to maximum power, and then it walks the data rate down toward SF12. Per-uplink energy went from 2.3 µAh to roughly 19 µAh, multiplied by confirmed-uplink retransmissions, for a day and a half.

Nobody saw it happen. You cannot observe the telemetry of a device that cannot transmit, and thionyl-chloride cells have a discharge curve so flat that terminal voltage tells you nothing until the cell is nearly gone. What told us, afterwards, was the firmware's coulomb ledger: an NVM accumulator that integrates modelled energy per operation using per-unit TX current measured at manufacture and a temperature-compensated leakage curve, reported in every uplink. When the fleet came back, the ledger showed a fleet-wide step of 4.1% of total budget consumed in 31 hours.

I ran that query three times hoping I had written it wrong. 3.9 months of design life, gone in a day and a half, and there had been nothing to watch while it happened. Six events like that in a year would have taken the program from eight years to under three.

A device that can't reach the network will spend itself to death trying, and it'll do it quietly. That's the one I'd put on the wall: on a battery-powered fleet a connectivity outage is an energy outage. You just don't get the invoice until later.

The fix was an energy governor that outranks the application layer.

// Runs before every transmit decision.
typedef enum { LINK_NOMINAL, LINK_DEGRADED, LINK_ISLANDED } link_state_t;

static bool tx_permitted(sched_ctx_t *ctx) {
    // 1. Hard 24 h quota, enforced against the coulomb ledger, not a timer.
    if (ledger_spent_since(ctx->day_epoch) +
        tx_cost_estimate(ctx->dr, ctx->len) > ctx->day_quota_nah)
        return false;                    // the app does not get a vote

    // 2. Never transmit into a switching transient.
    if (metro_transient_age_ms() < 400) return false;

    switch (ctx->link) {
    case LINK_NOMINAL:
        return true;
    case LINK_DEGRADED:                  // >32 uplinks, zero downlinks
        ctx->confirmed = false;          // stop paying for ACKs nobody sends
        ctx->dr = ctx->dr_last_known_good;   // refuse the MAC's slide to SF12
        return true;
    case LINK_ISLANDED:                  // >6 h of silence: probe only.
        // The AFE never stopped integrating; full-resolution data keeps
        // landing in the ring buffer. Probe cadence 60 s -> 30 min, jittered.
        return ctx->now >= ctx->next_probe_at;
    }
}

Deviating from reference MAC behaviour is a decision you write down and defend in a review, not one you slip in quietly. It is defensible here because the utility runs its own ChirpStack network server and we control both ends of the link. Server-side, gateway-cluster silence now alerts within 90 seconds. Across 34 months and four subsequent outages, the ledger projects a fleet median of 9.2 years and a 5th percentile of 7.6 years against the 8-year target.

Forty-one thousand individually calibrated instruments

This is not revenue metering and we did not pursue type approval. But the utility reconciles our data against revenue meters at the substation boundary, which means our numbers get compared against somebody else's numbers continuously, so we held the IEC 62053-21 Class 1 error envelope as a hard internal requirement.

Most of that budget goes to a single term, and the arithmetic is unforgiving. At power factor 0.5, an uncompensated CT phase shift of 0.3° produces a 0.91% energy error. Three tenths of a degree. It looks like nothing at all written on a page, and it eats most of what you were given, which is why per-unit phase compensation is mandatory here rather than a nice-to-have.

The production trade-off was thermal. A full three-point chamber characterization per board is 41 minutes of fixture time, and at 41,000 units that is not a test step, it is a second factory. So we do a five-point gain, offset and phase calibration per unit at 25 °C against a Class 0.05 reference source, 96 seconds per board, plus a full −25/+25/+70 °C characterization on a 5% sample per component lot, because AFEs and shunts from one reel drift together. Each unit's coefficient set is its own gains composed with its lot's second-order thermal polynomial. Field result: 95th-percentile error of 0.61% at −25 °C.

Coefficients, DevEUI and root key are written to a protected flash page and signed by the factory HSM, with the same record mirrored into a provisioning database keyed by board serial, AFE lot and CT serial. Every device reports a hash of its calibration record in its boot uplink. That check earned its keep before we had shipped a single unit: an ICT fixture bug wrote a 64-board tray with an off-by-one serial mapping, and 64 units left the line carrying their neighbour's phase compensation. The hash mismatch caught all 64.

Field drift is audited without touching anything, by exploiting conservation of energy. At each substation, the sum of downstream endpoints should equal the busbar meter minus modelled transformer losses. We run a recursive estimator over the residual that attributes scale error to individual endpoints, and the identifiability comes from switching events: feeder reconfigurations and breaker operations change the mixture matrix, which is what makes individual units separable inside a correlated load population. Over 26 months this flagged 214 units. 189 were confirmed on the bench as out of tolerance or trending, and 25 were false positives. A 400-unit random audit found zero out-of-tolerance units the estimator had missed.

The traffic pattern that isn't on the spec sheet

Steady state is 41,000 devices at one uplink per fifteen minutes: 46 messages per second. Trivial. Steady state is also the least informative number in this entire system.

When those 63 gateways came back, 11,400 devices each held up to 124 buffered intervals and began draining as fast as duty cycle permitted. Peak sustained write rate hit 118,000 rows/s. That part was merely a scaling problem. The real problem was that backlog records carry timestamps up to 31 hours old, interleaved with live data, and almost every component downstream quietly assumes append-mostly, roughly-ordered writes. Nobody writes that assumption down anywhere. You find it by violating it.

ClickHouse told us first, in the most recognizable way possible: Too many parts. Merges are processing significantly slower than inserts. Small, frequent inserts spread across a wide event-time range produce part explosion. We fixed the write path with an accumulating consumer that batches to 30 seconds or 200k rows, whichever comes first, against a (device_id, ts) sort key and monthly partitioning. Parts per table per day fell from about 1,900 to 96.

The subtler damage was in the stream layer. Our Flink jobs are keyed per device and hold windowed state, so setting allowed lateness to 31 hours to accommodate replay means windows never close and state grows without bound across 41,000 keys. There is no tuning parameter that saves you from that, because it isn't a tuning problem, it's a modelling error. So we split the pipeline at the source of truth. Every record carries an origin tag of live or replay, which the device already knows because it knows whether the record came out of the ring buffer. Live records go down the low-latency event-time path with 90 seconds of allowed lateness and never see a late watermark. Replay records go to a separate path that writes storage directly and enqueues targeted recomputation of exactly the affected windows. The live watermark is never held hostage by history.

Backpressure ultimately has to reach the endpoint, and the awkward part is that downlink, not uplink, is the scarce resource on a LoRaWAN network, because gateways are half-duplex and duty-cycle limited. Individually throttling 11,400 devices would cost more airtime than the backlog itself. Instead, the drain rate is piggybacked on the first downlink each device receives on reconnect, which it has to receive anyway for MAC state. Zero incremental airtime, fleet-wide flow control. Live-path p99 held at 1.4 s throughout the worst drain.

Cardinality is the problem, not volume

We sized the storage tier for volume, and volume turned out to be the wrong thing to worry about. Forty-one thousand series at 96 points per day is under four million points daily, which is a rounding error. Cardinality was the entire problem: 41,000 independent, low-rate, individually-behaved time series, each one needing its own baseline.

We landed on three tiers. Tier 1 runs for every endpoint on every point at O(1) cost: an additive seasonal state of 672 quarter-hour-of-week buckets holding an EWMA mean and a robust scale estimate, updated in about twenty flops. That is 5.4 KB per device and 224 MB fleet-wide, which is small enough to keep hot and cheap enough that we never have to decide which endpoints deserve monitoring. The anomaly score is a robust z-score of the residual against the matching bucket.

Tier 2 adds detectors that need no model at all: CUSUM on the residual for slow drift, change-point detection, phase current unbalance, power factor trend, and a temperature-normalized loss coefficient. Tier 3 is expensive — gradient-boosted forecasting with weather covariates, per-feeder multivariate models — and it only ever runs on the few hundred endpoints that Tier 1 or 2 has already promoted. Spend compute where the surprise is.

Alerts are evaluated in-stream rather than by polling storage, with current state materialized into a narrow keyed table so the operator's map is a single 41,000-row scan. p95 from device transmit to operator alert is 4.1 s, p99 6.8 s, of which roughly a second is airtime and network server. Day-ahead feeder forecasts, aggregated up from endpoint level, run at 3.1% MAPE.

Two catches show what the tiering buys. A distribution transformer whose temperature-normalized loss intercept crept up 4.6% over eleven weeks was flagged long before any threshold alarm would have fired; thermography found a degrading LV bushing connection running a 68 °C hotspot, replaced in a planned outage the utility estimates would otherwise have become an unplanned one affecting about 1,900 customers. The second one took longer to believe. A feeder showed a 7.8% divergence between head-end and downstream sum, but only between 01:00 and 05:00 on weekdays, with a sub-interval on/off edge. That shape is diagnostic: metering error is proportional and time-invariant, whereas a bypass being switched in and out is time-localized and edge-sharp. It was a bypass. Across 26 months the system has produced 31 confirmed non-technical-loss cases worth roughly €2.1M in recovered billable energy and 17 pieces of plant replaced before failure.

Shipping firmware you cannot take back

Pressing go on a multicast firmware campaign, aimed at a fleet you will never visit, does not feel like any deploy I have done in a data centre. There is no ninety-second rollback and no way to stop it halfway. Everything in this section exists to make that moment survivable.

The flash map is 32 KB of write-protected bootloader, two 224 KB application slots, and 32 KB of provisioning and persistent state, with RDP Level 2 set in production. The bootloader verifies an Ed25519 signature over the SHA-256 of the reassembled image against a root key inside the protected region. Nothing unsigned ever executes.

Delivery is the LoRaWAN FUOTA stack: TS003 clock sync, TS005 multicast setup, TS004 fragmented data block transport. A full image is 118 KB, and a binary delta against the previous release is 11.8 KB, which comes to 252 fragments of 48 bytes. We ship 38 additional redundancy fragments, about 15% overhead, so a device that misses any 38 fragments still reconstructs the image with zero retransmission requests. That matters enormously, because per-device NACK-and-retransmit does not scale on a downlink-starved network. During a campaign devices enter Class C for a bounded window, which is only affordable because the harvesting CT is supplying them, so campaigns are scheduled into loaded-feeder windows and units below harvest threshold defer to the next campaign. A full-fleet rollout that would take six weeks unicast completes in four days of multicast sessions, and we still spread it deliberately across three weeks of canary gates.

Then v2.4.0 moved the LoRaWAN persistent context into a new wear-levelled key-value store, and the commit path only flushed on clean shutdown. Clean shutdowns never happen in the field. A cold-booted device would rejoin with a DevNonce it had already used, the network server would reject the replay exactly as the spec requires, and the device would join forever, never receive a JoinAccept, drain its budget and never come back. Catastrophic, and completely silent.

The canary was 0.5%, 206 devices, soaked for 72 hours, and it saw nothing whatsoever, because cold boots essentially never happen in normal operation. Then a scheduled substation outage cold-booted 41 canaries and 38 went dark. The trial-boot flag did not save them, because the firmware ran beautifully. What saved them was the bootloader's connectivity watchdog.

slot_t choose_slot(void) {
    slot_t s = nvm->active;

    if (!ed25519_verify(nvm->sig[s], sha256(slot_image(s)), ROOT_PUBKEY))
        return other(s);                  // never execute an unsigned image

    if (nvm->trial[s] && nvm->boot_count[s] > 3)
        return revert(s);                 // classic A/B: it won't even run

    // The application stamps last_downlink_utc whenever the network server
    // proves it can still hear us. 26 h > one diurnal cycle plus the longest
    // legitimate gateway outage we have measured. This is the check that
    // caught v2.4.0: the image runs perfectly. It just can't be reached.
    if (rtc_now() - nvm->last_downlink_utc > HOURS(26))
        return revert(s);

    return s;
}

All 38 reverted to v2.3.6, whose nonce store commits before transmit in the correct order, and rejoined within four hours. I sat and watched them come back on the join log one at a time, and I counted every one of them off out loud. Measured fleet cold-boot rate is about 0.7% per month, so at full rollout this bug would have permanently darkened roughly 3,400 devices in its first year, around €620K in truck rolls before counting the lost data. The canary gate now includes a forced-reset probe: a downlink deliberately watchdog-resets 20% of canaries at hour six, so the reboot path gets exercised in the field rather than only on a bench.

A/B partitions protect you from firmware that crashes. They do nothing about firmware that runs flawlessly and can't be reached, and that's the one that takes the whole fleet with it. Stop asking whether the image boots and start asking how the device gets back.

What thirty-four months in the field actually produced

41,318 endpoints are live, from a 400-unit pilot through a staged 14-month rollout. Battery telemetry projects a fleet median of 9.2 years and a 5th percentile of 7.6 against an 8-year design target, with the governor now absorbing outages that used to cost months of life apiece. Accuracy holds inside the Class 1 envelope, with drift caught statistically and repaired during maintenance windows that were already scheduled. Operators see an anomaly 4.1 seconds after the device transmits it. The program cost €11.7M less over ten years than the modem-based alternative that could not have survived the cabinets anyway.

None of that came from a clever algorithm. It came from accepting early that the hard constraints were physical and irreversible: a steel box at 71 °C, a 43 µA budget, a downlink channel measured in hundreds of milliseconds per second, and a device you will never touch again. After that, the work was letting every layer above the metal, right up to the stream processor, be shaped by them.