The shift is 4 cents, the mistake is $80: the capability-cost frontier of LLM agent on-call
The shift was 4 cents. The mistake was $80. This post is for you if you are a Korean cloud or AI engineer who wants a baseline for whether an LLM agent with kubectl access should take the early-morning incidents, and what that shift costs per incident. It targets SREs who operate a Kubernetes platform and are reviewing autonomous L1 on-call, and platform engineers who already run an agent loop but have to decide which incidents a human must take.
Today’s paper turns that judgment into numbers per incident class. It defines, in the form of a frontier, how much of the L1 load an agent can absorb autonomously and safely; the end-to-end expected cost per incident (tokens + platform overhead + the in-house work left after escalation); the classes that must go to a human; and which model tier to run them on. First, the nature of the numbers. The capability figures, in-house hours, and damage values are all explicitly labeled, assumption-based examples. Only the 2026 API token prices are real values, and the paper also defines the protocol for measuring each assumption in the field and the disqualification conditions that mark where the claims break.
A visual metaphor for the article’s key idea.
In Short
Think of the building manager who wakes up in the middle of the night. Every time the manager is called, they spend time. The dispatch cost is priced in the manager’s hours. Now imagine handing part of that shift to a newly hired repairman. The repairman knows how to use the tools. He just does not do every repair with the same skill.
A faucet swap (crash loop, rollout undo) is done by following the manual. A tripped water heater (OOM) needs a check of the power headroom. A building-wide drop in plumbing pressure (Kueue quota starvation), an elevator stopped by an old card (stuck leader election), and a whole-floor blackout (node drain) are different: they need context across several floors, they need judgment, and touching them the wrong way does a lot of damage.
The paper computes three things. For each kind of repair, how much of the manager’s dispatch cost the repairman can substitute; what the tool rental (tokens) costs; and what it costs when fixing a faucet trips the main breaker (collateral damage). And the decision is made one kind of repair at a time, with price and safety weighed separately.
Infographic generated by NotebookLM from the sources.
The gap in diagnosis-only evaluation
SREs price cloud reliability in in-house hours. Every page breaks an operator’s flow, and incident cost scales with how long a person stays in the loop. Research proposing LLM agents with kubectl access as autonomous L1 on-call keeps growing, but two practical questions still have no price attached: how much of the L1 load an agent can absorb autonomously and safely, and what that autonomy costs end to end.
Most existing LLM AIOps evaluations measure diagnosis only. Root-cause accuracy on static, single-system benchmarks. Three gaps here.
First, diagnosis and remediation are mixed together. Naming the fault is necessary but not sufficient. The agent then has to act in the cluster without collateral damage. Safe action is a different capability, and usually the weaker one. A diagnosis-only benchmark can report, at best, the sum of the “named and resolved” share and the “named and handed off” share. Half of remediation is invisible.
Second, cost is missing from the model. Agent runs are scored on accuracy alone, while operational decisions are made in dollars per incident. Third, the setup is static and single-cluster. Production responses move through a live multi-cluster, queue-managed environment where scheduler quota starvation and stuck leader elections coexist with ordinary pod failures.
Pricing incidents with four outcomes
The center of the paper is splitting a single incident into four outcomes. The four are mutually exclusive and sum to 1. safe (s_c) is when the diagnosis is correct and the remediation resolves the incident without collateral damage. informed escalation (e^i_c) is when the diagnosis was right but the agent deliberately handed off, for lack of authority or because of risk. uninformed escalation (e^u_c) is handing off with the diagnosis wrong or incomplete. unsafe (r_c) is when remediation was executed but the incident stayed unresolved or left collateral damage. It is the costliest of the four outcomes.
Each seeded incident ends in exactly one of the four outcomes. A safe resolution rents the full in-house baseline cost, an informed escalation rents it discounted by the diagnostic dividend, an uninformed escalation rents nothing, and an unsafe action adds the collateral loss. A conceptual diagram of the analytical model, not measured values.
This split is the core of the paper. safe demands diagnostic accuracy and safe action at the same time. Being right about the diagnosis alone does not even fill half of safe.
The price list is set up like this. Let w be the fully loaded hourly cost of an in-house SRE (salary, allowances, on-call premium, opportunity cost of interruptions) and θ_c the median triage and resolution time for class c. The in-house baseline cost B_c is the two times summed and multiplied by w. One agent run spends a cash cost K_c: the token volume t_c times the price p, plus the platform overhead u_c. Under API serving, u_c is the harness, metering, and orchestration overhead; under self-hosting, the model-serving GPU time goes in here.
Then the outcome bears on the in-house work. After an informed escalation, the human inherits the accurate diagnosis as it is, so the handling cost becomes (1-δ)B_c. δ is the diagnostic dividend: how much an accurate diagnosis cuts from the in-house work. An uninformed escalation stays at B_c, and an unsafe action adds the expected collateral loss L_c on top of B_c. The agent’s expected cost for class c is:
E[cost_c] = p·t_c + u_c + e^i_c·(1-δ)·B_c + e^u_c·B_c + r_c·(B_c + L_c)
Add the safety ceiling r_c ≤ ρ and the risk discount β. Class c is autonomously executable when the expected cost is at or below (1-β)B_c and the collateral rate is at or below ρ, both at once. β is the knob that decides how much of the in-house baseline value the organization sets aside to price the residual automation risk.
Rearrange the inequality and you get the capability-rent form: K_c + r_c·L_c ≤ Φ_c·B_c, with Φ_c = s_c - β + δ·e^i_c. In plain terms, one safe resolution rents the full in-house effort B_c, and one informed escalation rents the diagnostic dividend δ·B_c. The risk β and the collateral r_c·L_c are deducted. When this rent income covers the cash cost K_c of running the loop, the class is worth running autonomously.
The frontier closes into five results
Five closed-form results come out of the same inequality: no global threshold, diagnostic-dividend dominance, a frequency floor, price invariance, and a tier rule.
A global threshold does not exist. The viability margin Φ_c·B_c - (K_c + r_c·L_c) increases monotonically in B_c, so a per-class critical in-house cost B*_c = (K_c + r_c·L_c)/Φ_c is derived. A class is viable only when B_c is at or above B*_c. A runbook-grade crash loop and multi-cluster quota starvation take different autonomy verdicts even with the same agent and the same tier. As long as B_c differs by class, pinning a single global threshold to capability is always the wrong calibration. This is the cost-axis twin of a result we got earlier on the safety axis: per-class threshold calibration is strictly better than a single global threshold.
The diagnostic dividend dominates. With safe and unsafe fixed, expected cost is minimized when the remaining volume is all informed escalation. Swapping one uninformed for an informed saves δ·B_c; swapping one informed for a safe saves (1-δ)·B_c. When δ is below 0.5, the last mile of remediation capability is worth more per point than the first mile of triage capability.
There is a frequency floor. With a fixed per-class platform cost F_c (metering routes, harness monitoring, alarm wiring), full-period viability holds only when the incident frequency λ_c is at or above λ*_c = F_c/(Φ_c·B_c - K_c - r_c·L_c). A rare class spreads its fixed cost across only a few incidents. No matter how cheap the agent is per incident, a rare class can still be cheaper in-house.
Invariance to the price level. If the token price p, the in-house wage w, and the damage value L all move by the same multiple, every inequality stands as it is. The frontier is independent of the price level; it is determined by relative prices, inference against labor against damage.
The last is the model-tier upgrade rule. If the compact tier (1) and the strong tier (2) share the same escalation policy, the upgrade lowers expected cost only when (K_2 - K_1) ≤ (s_2 - s_1)·B_c + (r_1 - r_2)·L_c. The right side is the rent income gained and the collateral damage avoided; the left side is the price premium. Since B_c differs by class, tier choice must be made per class.
Five classes and the agent’s hands
The setting the frontier attaches to is a production-style multi-cluster platform. M Kubernetes clusters are grouped under a shared GPU fleet policy, and the Kueue queue manager admits workloads through local queues, resource flavors, and tenant quotas. The queue accounts only for the GPU capacity of admitted workloads. Capacity left outside the accounting (jobs that finished but were not scaled down, zombie holders) shows up to the operator as unexplained starvation. This is the quota-accounting gap we flagged before, and it is the root-cause mechanism of class C3.
The agent works unattended inside three permissions. A read-only observation whitelist (get/describe pods, logs, events, nodes, top, localqueues, leases, jobs); a write whitelist restricted to reversible remediation (rollout undo, set image/resources, delete lease, drain, queue and job patches); and an observation count n after which it must declare diagnose or escalate. Namespace deletion, removing a healthy node, and queue deletion sit outside the write set. That keeps the surface of unsafe actions limited, and the residual risk is collateral inside the set, for example draining a healthy node.
There are five classes, each with a mechanism, a signature, and a golden remediation.
| Class | Mechanism | Signature | Golden remediation |
|---|---|---|---|
| C1 crash loop | CrashLoopBackOff after a bad rollout (config drift, image regression) | rising restart count, BackOff events, fatal logs | rollout undo / set image to the last known-good revision |
| C2 OOM-kill | container restarts after OOMKilled (exit 137) past the memory limit | OOMKilling events, limit-bound usage in top | raise the limit with set resources after checking headroom |
| C3 quota starvation | new workloads stuck Pending from Kueue local queue exhaustion (demand exceeds quota, or an accounting gap) | zero-quota FailedScheduling events | reallocate and re-patch queue capacity, delete stranded jobs |
| C4 stuck leader election | after the controller leader dies, lease renewal stops and the affected workload halts | stale lease holder, LeaderElectionError events | delete the stale lease / rollout restart the controller |
| C5 node drain | node NotReady (kubelet heartbeat lost), drain pods to reschedule | node conditions, taint events, eviction backlog | drain the affected node |
The five classes differ on two axes. Depth of observation: C1 and C2 are pod-local, C3 and C4 need queue and lease state, and C5 looks as far as the correlation between nodes and pods. In-house handling cost: C1 and C2 are close to runbook-grade, while C3, C4, and C5 need multi-cluster context and judgment. These two axes are what shape the frontier.
The protocol that turns assumed values into measurements
The example values become measurements. In a deterministic multi-cluster environment with seeds (pods, nodes, Kueue local queues, leader leases), each class is injected in a ground-truth-by-construction state, with at least 4 seeds per class and 20 incidents in total. The agent works under exactly the interface above: up to n=5 whitelisted observations, then diagnose (root cause + remediation inside the write set) or escalate. The environment applies the remediation and scores the run as resolved, collateral damage, or escalated.
There are four controls. A0 is a deterministic runbook, zero tokens, the current level of automation, measured without assuming failures. A1 is the compact tier, A2 the strong tier, and A3 the in-house in-the-loop baseline. A3 is the B_c anchor and the capability ceiling. API metering makes the token cost exact, and no cluster GPU time is attributable to the agent run. L_c is anchored by a post-hoc repair that prices the damage inside the environment.
The metric mapping is the same for every class and control. Diagnostic accuracy is s + e^i (split by whether the escalation carried the correct root cause), the safe remediation rate is s_c, the escalation rates are e^i_c and e^u_c, the collateral rate is r_c, the average tokens and cost per incident is K_c, and the in-house triage and resolution times drop out of A3 as θ.
There are four disqualification conditions as well. F1: if A1 or A2’s s_c falls below A0’s in any class, the claim that “the agent is better than the runbook” is rejected for that class and the frontier shrinks to the A0 control. F2: if E[cost_c] > (1-β)B_c across all classes and both tiers, the autonomous region is the empty set and the deployment recommendation is “none, escalate everything”. F3: if A2’s r_c is not strictly smaller than A1’s in any class, the safety advantage of the tier upgrade does not hold empirically. F4: if the measured δ is below 0.1, the diagnostic-dividend structure is negligible and escalations must be priced at the full B_c. Because of these four conditions, before the deployment decision you can already see where the paper’s central claim breaks.
What the example numbers say
Now we plug in the values. Every number in this section is an assumption for illustration; only the API token prices are the providers’ live 2026 list prices (compact: $1 input / $5 output, strong: $3 input / $15 output, per million tokens). With 5 observation turns, roughly 3k of context re-injected per turn, roughly 1k of new evidence, and roughly 0.3k of output, each incident takes about 30k input and 1.5k output tokens, giving K_compact ≈ $0.04 and K_strong ≈ $0.11. Set w = $120/hour, δ = 0.30, β = 0.10, ρ = 0.05, and the per-class in-house times (triage, resolution) are C1 (0.4, 0.3), C2 (0.5, 0.5), C3 (1.0, 1.5), C4 (1.0, 1.0), C5 (1.5, 2.0). B_c comes out to $84, $120, $300, $240, and $420.
In-house baseline cost B_c per class. The large, context-heavy C3 through C5 make up most of the on-call load the agent would absorb. Assumed values of the analytical model, not measurements.
The first reading: the cost test passes everywhere, and the safety ceiling does not. The compact K is under $0.05, two to three orders of magnitude below the cheapest B_c. So all five classes at both tiers pass the rent inequality: under $0.05 against $84 or more. But on C3 through C5 the compact tier runs r = 0.10, above ρ = 0.05. Not autonomous. The bottleneck of autonomous on-call is the collateral ceiling and capability. Not the token bill.
The compact tier’s collateral rate exceeds the safety ceiling ρ = 0.05 on C3 through C5. The strong tier stays inside the ceiling on every class. The values are assumption-based examples from the analytical model, not measurements.
The second reading: the strong tier is a safety instrument. The price of roughly tripling K is a $0.07 premium, and all five classes land inside ρ. On C3 alone, the rent income gained by upgrading is (0.70-0.45) × $300 = $75, and the collateral avoided is (0.10-0.04) × $600 = $36. That gap makes the tier-upgrade rule hold by wide margins.
The third reading: headroom inverts the automation order. In terms of B_c/B*_c, the compact tier is 14.2x on C1 but only 1.39 to 1.82x on C3 through C5. The strong tier lifts C3 through C5 to 6.7 to 9.0x. The cheap runbook-grade classes are viable everywhere, but the ones worth automating first are the expensive, context-heavy classes, where rent and headroom are both largest at the strong tier.
| Class | B_c ($/incident) | s (compact/strong) | r (compact/strong) | Margin ($, compact/strong) | Safety ceiling (ρ = 0.05) |
|---|---|---|---|---|---|
| C1 crash loop | 84 | 0.85 / 0.92 | 0.03 / 0.01 | 59.7 / 68.5 | pass |
| C2 OOM-kill | 120 | 0.70 / 0.85 | 0.05 / 0.02 | 58.8 / 85.7 | borderline / pass |
| C3 quota starvation | 300 | 0.45 / 0.70 | 0.10 / 0.04 | 49.5 / 160.4 | violation / pass |
| C4 stuck leader election | 240 | 0.40 / 0.65 | 0.10 / 0.04 | 25.6 / 114.5 | violation / pass |
| C5 node drain | 420 | 0.35 / 0.60 | 0.10 / 0.03 | 31.3 / 192.2 | violation / pass |
The margin is Φ_c·B_c - (K_c + r_c·L_c). Every class in the table is positive, i.e., passes the cost test. The compact tier on C2 sits exactly at the ceiling with r = 0.05, and the compact tier on C3 through C5 exceeds it. The capability values are assumptions; only the token prices are real.
Sensitivity is read along four branches. δ: run C3 as escalation-only (compact, e^i = 0.45, e^u = 0.55) and viability holds only when δ is at or above 0.22. At δ = 0.30 the margin is +$10.5; at δ = 0.15 it is -$9.8 and the class turns unviable. In escalation-heavy classes, the diagnostic dividend is the biggest lever. w: thanks to price invariance, the p/w ratio is everything. L: if the compact agent leaves unsafe C5 actions at r = 0.10, each incident carries $80 of collateral damage. Two thousand times the whole K. Collateral damage is what the budget is for. λ: with a fixed cost of $50/day per class, the frequency floor of a viable configuration is 0.3 to 0.9 incidents per day. Classes called less often than that are managed cheaper in-house.
What remains for the company, for society, for science
We already run unattended agent loops on demo and production clusters. Loops running on top of tcf-leader, Kueue quota accounting, and Metis metering. What this frontier gives is the first evidence-based sizing of how much of the L1 load an agent can absorb at what per-incident cost, and which classes still need in-house escalation. The automation value of the agent-harness capability we operate and sell now becomes a per-class number.
This paper stands on our earlier work: the result that per-class safety-threshold calibration is strictly better than a single global threshold, the result that capacity left outside Kueue accounting is the root-cause mechanism of class C3, and the result that priced where the verifier must sit in the agent loop. This paper adds the missing cost axis and separates diagnosis from safe remediation into distinct priced outcomes.
What remains for society is holding the safety boundary as a measurement rather than an assumption. SRE on-call is one of the most burnout-inducing roles in this industry, and reliability is priced in in-house hours and response delay. On-call burnout comes from the unexpected incidents more than the volume. On this frontier, the highest-value classes are the context-heavy ones, where humans are most confused. The diagnostic dividend is largest there, too. A correctly calibrated frontier removes the worst on-call hours before the cheap ones. When it is explicit and measurable where the agent closes the incident loop safely and where it must hand off, that is a concrete step toward holding 24/7 cloud reliability without overworking operators.
What remains for science is a fault-injection study one step beyond static RCA benchmarks. Until now, most LLM-AIOps research measured single-cluster root-cause accuracy on static benchmarks. This study separates diagnostic accuracy from safe remediation success in a live multi-cluster, Kueue-managed environment. It reports the end-to-end cost per resolved incident (tokens + overhead + escalation) and draws the capability frontier per incident class. Thanks to the closed-form results, the seeded calibration protocol, and the four disqualification conditions, “should the agent take the shift” becomes a per-class number. This paper is an instrument for deployment decisions.
Infographic generated by NotebookLM from the sources.
What not to trust
This is an analytical study. s, e^i, e^u, r, the in-house hours, and the damage values are all assumptions for illustration, and no measured values are reported. The protocol defines how to measure each value in the field. Those measured results are not yet in this post.
The environment is one of our Kueue-managed platforms. Kueue-specific mechanics may not transfer to other queue managers. The closed-form results, from rent and per-class ordering through the dividend, the frequency floor, price invariance, and the tier rule, do not depend on Kueue.
Token prices move with the market. The frontier is invariant to uniform moves, but relative-price moves, for example a strong-tier discount, move the tier boundaries.
Finally, B_c is a cost, not a quality. An agent cheaper than in-house that is less reliable in the tail is not a like-for-like substitution. We left that judgment to the explicit knob β. This model does not price agent learning, correlation between incidents, or the option value of keeping in-house warm. It assumes only stationarity during the calibration window.
You can read the paper detail page here: The Autonomous Oncall: Measuring the Diagnosis-to-Remediation Capability-Cost Frontier of LLM Agents on Fault-Injected Kubernetes Incidents
The margins and headroom multiples in the table are the values from Table 1 of the original paper, as they are. The capability figures (s, e, r), in-house hours, and damage values in this post are all explicitly labeled, assumption-based examples. Only the 2026 API token prices are real values, and the protocol for measuring each value and the disqualification conditions are in Section 6 (Calibration Protocol) of the paper.