Proposals, Not Actions: The Control Plane Behind Autonomous Infrastructure Agents

There is a specific moment in the life of an agentic infrastructure platform when the abstraction stops being charming. A model returns a plan. The plan is well-formed, internally consistent, articulate about its own reasoning, and it will destroy a stateful resource that fourteen other things depend on. A human engineer proposing the same change would have leaked signal somewhere along the way: hedging in the PR description, a message asking whether anyone still uses this thing, a half-second pause before typing the resource name. Agents don't hedge. They have no proprioception about consequence, and they are exactly as fluent when they are wrong as when they are right. That asymmetry is the actual engineering problem in letting autonomous software operate production infrastructure, and it has surprisingly little to do with model quality, prompt engineering, or tool design.

The platform is multi-tenant agentic infrastructure. A customer describes intent in natural language, and a fleet of specialized agents (provisioning, cost optimization, security posture, incident response) plans, provisions, deploys, and then continuously operates the resulting cloud estate. Calling it a one-shot generator undersells the hard part: it is a control loop that runs unattended, indefinitely, across roughly 3,400 concurrent customer environments and a little over a million managed resources at the point this story reaches. At that scale, "the agent occasionally does something catastrophic" stops being an anecdote and becomes a rate you can compute. Once it is a rate, you either budget for it or you design it out. We were brought in to design it out.

Agents propose. The control plane decides.

The first decision is the one everything else hangs off. An agent has no network path to a cloud API. Not a rate-limited one, not a scoped-down one — none at all. The only artifact an agent can produce is a document.

That document is a plan IR, a provider-neutral intermediate representation of intent. We normalize provider plan output into a typed set of mutations: canonical resource URN, operation (create / update / replace / destroy), the before/after attribute diff at field-path granularity, and the dependency edges the mutation implies. Everything downstream reads the IR and nothing else, whether that is policy evaluation, risk scoring, locking, credential minting, or audit.

None of that came free, and to be fair to the people who pushed back, building an IR when the provider already emits a machine-readable plan does look like ceremony on a whiteboard. It stops looking like ceremony the first time you have to express one policy that means the same thing across three clouds, or hash a plan so you can prove that the thing approved is byte-for-byte the thing executed, or diff a proposal against an approval envelope from six weeks ago. The IR turned out to be the load-bearing abstraction of the entire system, and nearly every mechanism below is a pure function over it.

The gate that does not reason

Between the proposal and the execution sits a deterministic evaluator: Rego-style policy-as-code, compiled per organization, with no model anywhere in the enforcement path. We were dogmatic about that and I would take the same position again. A model may help author a policy and it may explain a policy, but it may never be the component that decides whether a policy passed. Put something probabilistic in the enforcement path and the whole thing quietly degrades into a very expensive suggestion.

Policy answers whether a change is permitted. It has nothing useful to say about how much the change matters, which is the question that actually decides whether a human needs to look. For that we built a blast-radius estimator that reduces a plan to a risk vector.

# blast_radius.py — deterministic, no model in this path.
# Input is the normalized plan IR. Output is a 0-100 scalar AND the
# vector that produced it. Reviewers read the vector; the router reads the scalar.

def assess(plan: PlanIR, graph: StateDAG, envelope: ApprovalEnvelope) -> RiskVector:
    writes = plan.mutations

    # 1. Direct footprint, log-scaled: 3 resources and 300 resources
    #    differ in kind, not merely in degree.
    footprint = 12.0 * log10(1 + len(writes))

    # 2. Transitive fan-out: what breaks if this proposal is wrong?
    #    Walked over the state DAG, never over the plan — a plan knows
    #    what it touches, not what depends on it.
    downstream = graph.downstream_closure(writes.urns(), max_depth=4)
    fanout = 18.0 * log10(1 + len(downstream))

    # 3. Novelty: has a plan of this SHAPE been approved for this tenant
    #    before? Shape = (resource types, ops, field paths) — never values.
    novelty = 0.0 if envelope.contains(plan.shape_hash) else 25.0

    # 4. Tenant criticality tier, declared by the customer, not inferred by us.
    tier = TIER_WEIGHT[graph.tenant.tier]        # prod-critical 1.4 … sandbox 0.6

    # 5. Reversibility is not a term in the sum. It is an override.
    irreversible = any(
        (m.op is DESTROY and graph.node(m.urn).is_stateful)
        or m.forces_replacement_of_stateful()
        or m.crosses(DATA_DELETION_BOUNDARY)
        or m.revokes(LAST_ADMIN_PATH)
        for m in writes
    )

    raw = (footprint + fanout + novelty) * tier
    return RiskVector(
        score=100.0 if irreversible else min(raw, 99.0),
        fanout=len(downstream), novelty=novelty,
        irreversible=irreversible, rules_fired=plan.policy_trace.rule_ids,
    )

Three things in there cost us iterations. Irreversibility is an override instead of a weighted term, because no amount of "but it's only two resources" should ever be able to sum its way under a threshold when the operation deletes data. Fan-out is computed against the dependency graph rather than the plan, since self-reported scope is the one number an agent is structurally unable to be honest about. And the artifact that matters is the vector, not the scalar: a reviewer sees fanout=214, novelty=25, irreversible=true, rules_fired=[data.infra.pci.tenancy] rather than the number 78.

I had no real confidence any of this would hold up under production traffic. My worry was that reviewers would learn the score distribution, find the threshold, and start reading anything underneath it as pre-approved noise — which is exactly why the scalar is never shown on its own.

Below the threshold, plans auto-apply: 91.4% of them. Above it there is a hard human gate with no override path, no break-glass "approve anyway" button for a plan the estimator flagged as irreversible. Removing that button was an argument, and not a short or a polite one. The case for keeping it was legitimate: at 3am during a real incident, somebody is going to want it. What settled it was asking what the button would be for, given that the only plans it unlocks are the ones you cannot undo.

A risk score is doing its job when nobody finds it interesting. Once people start haggling with the number instead of reading the vector behind it, you have built a slot machine and hung a sign on it that says gate.

Two agents, one resource, on an ordinary Tuesday

Everything above quietly assumes one agent at a time, and that assumption survives exactly until the second agent ships. After that, the cost agent and the security agent are reasoning about the same node group at the same moment, and the question of what happens when both of them are right becomes load-bearing.

The obvious answer is one lock per tenant. It works, and it is unusable. It serializes an entire customer's infrastructure behind whichever agent got there first, and an incident-response agent sitting in a queue behind a 90-second cost sweep has stopped being a throughput problem and become an availability one.

So we lock only the write-set subgraph a plan actually touches. Locks are acquired in canonical URN order, which makes deadlock structurally impossible rather than merely unlikely, and they are held as leases with fencing tokens so a stalled agent process cannot apply after its lease has expired. On top of that sits optimistic concurrency control, lifted more or less directly from distributed transaction design: every plan records the generation counters of its read set, and at apply time those generations are revalidated. If anything moved, the plan is invalidated and the agent replans against fresh state.

About 3.1% of plans get invalidated this way. That is cheap, because replanning is a playbook-tier decision, and it is very cheap next to the alternative, which is two correct plans composing into one incorrect outcome. p95 plan queue wait fell from 4m 12s under the global-lock prototype to 840ms, with the fleet sustaining roughly 600 applies per minute.

The storm that never paged

The control loop reconciles agent-authored desired state against observed cloud state continuously, Kubernetes-operator style, healing drift without asking anyone first. The first genuinely instructive production incident that design produced did not look like an incident at all.

A tenant's cost agent kept migrating a node group to a cheaper instance family. That same tenant's security agent had a posture baseline requiring an instance family with a specific attestation capability for a compliance-scoped workload. Both agents were locally correct. Both were authoritative over the same field. Every reconcile cycle, one of them undid the other.

Nothing was down. No SLO burned, and no page fired. It ran for nine days and 1,447 transitions, and we only found it because a cloud provider API throttling alarm tripped in one region and someone went looking for the caller. The $9,400 anomaly on that tenant's bill turned up as corroboration, and that is the part I still think about: not our alerting, not our dashboards, not any of the observability we were fairly proud of. An invoice told us.

The fix has two halves. The first is ownership: exactly one agent role owns each (resource, field path) pair, resolved through a priority lattice where security posture outranks incident response, which outranks provisioning, which outranks cost. A non-owner does not mutate an owned field. It files a proposal to the owner, and the owner either accepts it or explains why not. Kubernetes reached the same insight with field managers in server-side apply; we got there the expensive way. The second half is a damper, because ownership only resolves the conflicts somebody declared, not the ones that emerge.

// Reconciliation is not "converge harder." A loop that fights itself
// must stop, and must be able to say who it was fighting.

func (r *Reconciler) Admit(f FieldPath, want Value, by AgentRole) Decision {
    if owner := r.ownership.Owner(f); owner != by {
        return Propose{To: owner, Reason: ConflictingIntent} // non-owners file, never write
    }
    if br := r.breakers.Get(f); br.State() == Open {
        return Hold{Until: br.ResetAt(), Reason: FlapSuppressed}
    }
    // A "revisit" is a transition back to a value this field already held
    // inside the window. That is churn, not convergence.
    win := r.history.Window(f, 30*time.Minute)
    if win.Revisits(want) && r.breakers.Get(f).Trip(threshold: 5) {
        // Escalate with BOTH claimants' provenance traces side by side,
        // so a human sees two arguments, not one mystery.
        r.escalate(f, win.Traces())
        return Hold{Reason: FlapSuppressed}
    }
    return Apply{}
}

Five revisits in thirty minutes opens the breaker. Reconciliation on that field path pauses, the resource is pinned at its current value, and the conflict is escalated with both agents' reasoning attached. There is a half-open probe when a human resolves it, or after 24 hours if nobody gets to it.

The failure modes that survive longest in agentic systems are the ones that don't look like failures. For nine days nothing was down; a value was just quietly flipping back and forth. If the only thing you alert on is outage, you will never see it. Alert on churn too.

Paying for thought

The fleet makes about 2.3 million reconcile decisions a day. Running a full agent reasoning loop for every one of them would have produced an inference bill larger than the cloud bill the cost agent was hired to reduce, along with a decision latency that made the control loop slower than the drift it was chasing. Neither problem is subtle; we could see both coming without much arithmetic.

The property worth exploiting is that reconciliation decisions are overwhelmingly Zipfian. The same handful of situations recur across thousands of environments. So when a decision shape recurs with a stable policy outcome, we compile it into a parameterized playbook: a deterministic matcher over the plan IR shape, with slots filled and validated rather than reasoned from scratch. Routing then falls out fairly naturally. 61.2% of decisions resolve on a playbook hit with zero inference, 33.0% go to a fast, cheap model, and 5.8% escalate to a frontier model on shape miss, low confidence, policy near-boundary, or a blast-radius floor. Prompt prefixes carry the tenant's topology digest and compiled policy summary, both of which are stable across cycles, so they cache at an 89% hit rate and invalidate on a state-generation bump.

Inference cost per environment-month dropped 71%, and p95 decision latency went from 6.4s to 410ms. The second-order effect mattered more than the first. Once decisions were cheap, a tighter reconcile cadence became affordable, which shrank drift windows across the fleet. The escalation rate also turned into one of our better leading indicators: a tenant whose frontier-tier share is climbing is a tenant whose environment has wandered outside every shape we have seen before, and that is usually a week or two ahead of them having an incident.

"Why did it do that?"

When autonomous software changes a customer's production infrastructure, "why did it do that" cannot resolve to a shrug. Every decision emits a provenance span: model and version, the content-addressed rendered prompt, the IDs and content hashes of every retrieved context document, tool calls and their results, the resulting plan IR and its hash, the policy decision log down to the specific rule IDs that fired, the risk vector, and the identity of the approving human if there was one. Spans nest across agents, so a change made by the provisioning agent in response to a finding from the security agent reads as one causal tree.

Distributed tracing for cognition, essentially, and the reverse index is where it earns its keep. When a retrieved document turns out to have been stale or wrong, you can enumerate every action anywhere in the fleet that descended from it. That is the blast radius of a bad fact, and it is a question no cloud audit log has ever been able to answer.

It also rescued the human gate, which I did not see coming. Before provenance, reviewing a gated plan meant reconstructing an agent's reasoning out of a diff, so reviews were slow, and slow reviews drift toward rubber-stamping. That is the specific failure mode that makes a safety control worse than useless, because from the outside it still looks like it is working. With the trace attached, median time-to-approve fell from 31 minutes to 4, and gated plans abandoned or expired in the queue dropped from 11% to under 1%.

The last mile: no standing credentials

Everything above constrains what an agent may propose. None of it constrains what an agent's process can do if it holds standing infrastructure-wide credentials, and that is a strictly worse problem than the hallucination one, because it does not require the model to be wrong at all. A compromised or simply buggy agent with broad standing access is dangerous while behaving perfectly. Our threat model included prompt injection arriving through resource tags and IAM policy descriptions ingested as context, which sounds exotic right up until you remember that reading the estate it operates is the agent's whole job.

So agents hold no standing production credentials. At apply time, a credential broker takes the approved plan ID, verifies that the plan IR hash still matches the hash a human or the auto-apply router approved (this is what closes the gap where an approved plan A becomes an executed plan B), and mechanically derives a least-privilege session policy from the plan's write set: exactly those resource identifiers, exactly the actions each operation type requires, nothing broader. The credential is single-use, bound to the workload identity and the plan ID, issued with a TTL of min(10 minutes, 3 × estimated apply duration), and revoked at apply completion rather than left to expire. Planning runs on a separate, read-only, equally short-lived path.

No human writes those policies. They are a function of an approved plan, which is the third time the IR investment paid for itself.

What it actually bought

Across the two quarters following the guardrail and credential redesign, agent-attributable Sev-2-or-higher incidents fell from 23 in a quarter to 2, and there have been zero incidents involving an agent acting outside an approved plan across roughly 2.6 million applies. The fleet reconciles ~3,400 concurrent environments with no global-lock serialization, auto-applies 91.4% of plans without a human in the loop, and gets the remaining 8.6% through review in a median of 4 minutes.

I slept badly the first night we left the router applying unattended, with the apply feed open on my phone and checked more often than I would like to write down. What eventually made that feel survivable was not growing confidence in the model, but being able to name, in order, every component that would have to fail before one of the model's mistakes could reach a cloud API.

The number we actually care about is the last one, because it is the one that shows the system is not merely safe but usable. Safety mechanisms that make autonomy slow get switched off. A control plane rarely fails by blocking something it should have allowed; it fails when people quietly start routing around it. None of this worked because we found a better model. It worked because we accepted that the model's authority has to be bounded by components that cannot reason: a deterministic policy evaluator, a risk function with an override for irreversibility, a lock manager, a breaker, and a credential broker that derives permissions from an approved artifact rather than from trust. Autonomy is not a property of the agent. It is a property of the system the agent is allowed to run inside.