Wrong Costs the Whole Task; Doubt Costs One Call: The Veto Frontier of a 2,000-Skill Router
If you operate an agent that auto-routes through 2,000 skills on every turn, or you sit on the platform team that measures that routing cost, this post is for you. The price of picking one skill wrong and sending an execution is that the whole task disappears; the price of doubting and calling an AI judge once more is that single call. The moment that asymmetry goes into the math, the router’s “accuracy” is no longer the answer. The paper answers two questions. The first is how well the score gap between rank one and rank two reflects the actual probability of being wrong. The second is at which point auto-execution should stop and hand off to a judge call so that total cost is smallest. There is one conclusion: calibration is what makes the difference, and the break-even point of calibration condenses into a single constant, p* = p_j + C_e/C_r.
A visual metaphor for the article’s key idea.
In Plain Terms
Imagine a restaurant with 2,000 items on the menu. When an order comes in, the waiter picks one item and sends it straight to the kitchen. If the waiter misreads it, the dish is ruined, and the whole meal is ruined. If instead the waiter says “I’m not sure” and checks with the manager once, the price is that single check, and the meal goes out properly.
This paper computes, in the world of a waiter who must pick one out of 2,000 menu items, the point where “it is worth handing off to the manager”. The waiter is the router that picks a skill; the misreading is the wrong routing; the manager is the AI judge, the LLM judge, called only when there is no confidence. The waiter’s “confidence” is the size of the margin, the score gap between the first-place and second-place menu items. There are two key points. If you know how well that gap describes the probability of a misreading, the point from which you should go to the manager is fixed.
Infographic generated by NotebookLM from the sources.
Wrong Costs the Whole Task; Doubt Costs One Call
In the deployment environment that motivated this work, the skill catalog sits around 2,000 entries. A prior compression study measured 1,910; a sequence routing study measured 2,275. On every turn, the router must pick one of them, the top-1, and hand it to execution. The router fuses two orderings, lexical BM25 and dense embedding, with reciprocal rank fusion (RRF). The gap in RRF score between the top-1 and second place, the score margin, is the only per-query confidence this environment has.
The cost is asymmetric. If auto-routing is right, the price is only the downstream execution cost of the skill. If it is wrong, the execution tokens, the failing or misleading artifact, and the recovery path that follows all disappear at once. Escalation, by contrast, is a single metered call over the top-5 shortlist. The judge is wrong with probability p_j, but in a world where the error penalty C_r is far larger than the judge call cost C_e, doubt is always cheaper than error. So the question is not the accuracy of the router. It is from which margin onward the expected cost of auto-execution is lower than the expected cost of doubt.
For that question to hold, the margin must be calibrated. Reading a margin value should immediately give you the probability of being wrong. Only then does where to place the threshold become a cost calculation; otherwise, no matter where you move the threshold, the calculation is meaningless. Existing skill routing research is all accuracy-centric. It optimizes hit@k and MRR and keeps the router always on. Confidence appears rarely, and only as a tie-breaker. Whether the router can stop its own decision, and how much it spends and how much it saves when it does, is a question no one has asked.
Every query is scored by the hybrid RRF router. When the score gap clears the certified threshold, the veto layer executes the top-1 skill immediately; when it does not, it spends a single metered judge call. The asymmetry that C_r is far larger than C_e is what makes “doubt” the cheap side. Conceptual illustration.
p* = p_j + C_e/C_r: Where Doubt Becomes Insurance
The break-even posterior is simple. If the probability of being wrong on query q, p(q), is larger than p* = p_j + C_e/C_r, judge escalation is always cheaper than auto-execution on that query. If it is smaller, the reverse: auto-execution is cheaper, and the judge call is wasted money. This constant is what separates “confidence” from “doubt”, so it deserves a name. In plain terms, p* is the exchange rate that converts “one judge call” into “probability of being wrong”. In restaurant terms, it is the price of one check with the manager converted into the probability of a misreading.
Four consequences follow. First, abstention is not a fallback. It is insurance. If the judge’s error rate on the escalated set approaches that of the auto-router, p* is pinned to 1, and no matter how well the margin is calibrated, doubt is almost meaningless. Second, if the judge becomes free, C_e drops to 0, p* drops to p_j, and a completely free judge leaves only queries with zero posterior error in auto-execution. Third, if p_j + C_e/C_r is 1 or more, escalation is always a net cost. The right answer is to change the judge tier. Widening the rejection range only increases cost. Fourth, C_r differs by skill class, so p* differs by class, and a single global threshold targets the wrong thing.
The second result is a regret bound. Build an estimate of the probability of being wrong from the margin, and use a policy that goes to auto-execution whenever the estimate is at or below p*. The excess cost this policy pays over the optimal policy (the Bayes policy) is bounded above by C_r times the mean absolute difference between the estimate and the true probability. If calibration is off by ε, no excess cost beyond C_r × ε per query is produced. Even if the margin perfectly orders the risk of error, the mismatch when the value is converted into probability is the entire cost. What binds on cost is calibration, not accuracy.
The third result compresses the research question into a single sign test. Use a policy that auto-executes up to a margin threshold c, and to beat “always auto-execute”, the conditional error rate r(c) of the escalated set, the queries with margin at or below c, must exceed p. If the margin cannot exceed p in any interval, the veto layer is a net cost no matter where the threshold is placed. So when the router is confidently wrong, when error concentrates in the high margins, the direction of investment is not tuning the threshold. It must go toward the quality of skill descriptions, the judge tier, and retrieval coverage. Conversely, if the margin is a trustworthy risk signal, r(c) crosses p* somewhere, and that crossing point is the operating point.
Always auto-executing produces a constant expected cost. Threshold rejection is most expensive when every query is escalated; it crosses the always-auto-execute line where the error rate of the escalated set passes through p, reaches its minimum at the operating point, and returns to the original cost when everything is auto-executed. Schematic of the analytical model, not measured values.*
The RRF Margin Is an Ordinal, Not a Probability: A Finite Lattice
The RRF margin is not the confidence of a neural network. It is a property of rank fusion. Its geometry makes the resolution of any threshold policy coarse. First, the margin is determined only by four rank integers: the lexical ranks and the dense ranks of the top-1 and the top-2. As long as these four ranks are preserved, the veto decision does not change no matter how much the underlying scores wobble. It is the same even when the wobble is as large as the entire score gap between adjacent ranks. Two queries with the same rank profile but different topical ambiguity, for example a query where two skills are paired and their roles are swapped, receive the exact same veto decision.
Second, the range is finite. With two rankers and N candidates, the theoretical maximum of the margin is 2/(k+1) - 2/(k+N). At k=60, N=2,000, it cannot exceed 3.2 percent. The margin of the most contested top-2, where first and second are adjacent in both rankers, is smaller than one two-thousandth. Since the margin is a function of the rank tuple, the support of its values is a finite lattice, at most N^4 points, and the coverage-risk curve is piecewise constant over it. In other words, the margin lives on a coarse scale, and that scale is coarsest right near where the veto layer cuts.
So the place where a threshold policy’s resolution is coarsest is exactly where the veto layer works. Near the top of the margin distribution, different score configurations collapse into the same margin value. The number of valid positions where a threshold can actually be placed is far smaller than a continuous scale suggests. There is one practical consequence. The deployment environment must store both the RRF margin (the decision statistic the fusion has already produced) and the score-space gap (a second signal that carries separation information within the ranks). The two play different roles in the certified threshold of the next section.
The RRF margin is a value on a finite rank lattice, so its reachable range is finite. At k=60, N=2,000, the maximum margin is 0.0318, and the margin of the top two, adjacent in both rankers, is 0.00053, that is 2/((k+1)(k+2)). Threshold resolution is coarsest in the adjacent top region where the veto layer actually works. Conceptual illustration; the bar lengths are not computed values. For explanation only.*
The Composition Where the Veto Layer Buys Quantization Headroom
The immediately preceding study asked whether the confidence signal of this router could be made cheaper. It bounded how much the fused score wobbles when the embedding side is quantized, and derived a rank-safety band that certifies the top-1 is invariant at a given bit width b. The uncertified queries, the at-risk fraction ρ(b), were all priced at the cost of “one wrong routing”. But the router did not even go through that band. It executed everything with confidence.
This paper composes that safety band with the calibration break-even point. Bundle the score gap t* where the calibrated posterior touches p* with the quantization wobble bound, and a single certified threshold comes out: auto-execute when the gap is at least max(t*, Δ_RRF(b)). Under this policy, three things hold at once. Every auto-executed query receives a quantization certification, so compression creates no rank-flip error. Excess escalation is at most the at-risk band fraction, at most ρ(b) × C_e per query. The error rate never rises.
The core is repricing. The prior study’s adoption condition was “savings ≥ at-risk fraction × wrong routing cost”. Now that band is repriced at the judge call cost C_e. Since C_e is a tiny fraction of C_r, a router equipped with a veto layer tolerates far more compression than the same router would if it auto-executed everything. The veto layer is buying quantization headroom. What the prior study captured conservatively as a worst-case band becomes a cheap escalation band. In restaurant terms: since the manager absorbs the misreadings, the menu can move to a thinner, cheaper printing.
The actual deployment fusion is RRF and has no blend weights, so the prior bound must be re-derived in units of rank movement. If quantization on the embedding side moves the dense rank of a top-2 candidate by up to δ_r(b) positions, the worst-case margin erosion is at most 2δ_r/(k+1)². At k=60, a one-position rank movement cuts the RRF margin by about one two-thousandth, the same order as an adjacent top-2 margin. Certification in RRF space is inherently coarse. Only queries whose top-2 are not adjacent on the dense side get certified, so the lattice that bounded the resolution of calibration also bounds the precision of quantization certification at the same level. The two diagnostics share a common resolution limit under RRF. The bound is first-order worst-case. If δ_r is loose, it over-reports, and that is safe. If δ_r fails to cover reality, the certification silently becomes void.
What Remains for the Company, for Society, and for Science
For the company, two things remain. First, the veto layer reduces the executions that disappear whole from wrong routing in ThakiCloud’s own 2,000-skill registry. Judge calls are used only where the escalation pays for its cost. It is a lever that touches directly on Metis’s cost per task and on the reliability of unattended automation. Second, the certified threshold greatly relaxes the adoption condition for quantization on the embedding side. If the router can hold the top-1 at a lower bit width, the inference cost of routing comes down with it.
For society, an agent that “knows what it does not know” remains. An unattended agent’s budget is spent on two things: execution and doubt. Accuracy-centric programs optimize the former and treat the latter as if it were free. This paper shows that the two are exchanged at a single constant, p*. When a 2,000-skill registry runs indefinitely, the difference between a router that is often right and a router that may be wrong comes out of the calibration of that constant. High-cost, or hallucinated, skill executions decrease. The token and energy footprint of unattended work automation comes down with them.
For science, it is the first calibration and selective prediction analysis of LLM agent skill routing under asymmetric loss. Existing learning-to-defer and cascade routing research escalates answers between model tiers. Even if the cheaper tier fails, the answer is merely degraded. If a skill router fails, the execution itself disappears whole. The asymmetry of loss does not come from magnitude. It comes from kind. That is why the break-even and regret are derived for the action pair commit/doubt, the pair of confidence and doubt. The actions themselves differ from the route/escalate pair that cascade research uses. The calibration error of the RRF margin (ECE, expected calibration error), the coverage-risk curve, and the break-even abstention threshold are axes that accuracy-centric routing research has never measured, and this paper formalizes them.
Infographic generated by NotebookLM from the sources.
What Is Not Yet Trustworthy
This paper is an analytical study. Beyond the structural facts that follow from the definition of RRF, it makes no empirical claims. Every open quantity is handed to the five-stage validation protocol and, over the 63-entry golden routing set (sra_bench, route_bench), fixed respectively by live margins and Metis metered judge calls. With n=63, the per-bin estimates are wide. Stage 4 is a sign test, not an estimate of magnitude. If the margin is on the confidently-wrong side, the veto layer is a net cost, and that sign is read at stage 4.
There are a few idealizations. The judge error rate p_j is treated as a constant over the whole escalated set, but if the escalated set is heterogeneous, that assumption breaks. The paper partially corrects for this by measuring p_j not as a full-range average but on the escalated set where the judge is actually used. The gold labels themselves carry noise. That noise lands exactly on the ambiguous queries the veto layer is trying to protect, pairs of skills with equivalent capability. C_e is an endpoint-specific value that changes with the model tier, the shortlist size, and the prompt length, so if the judge tier changes, p* must be re-derived.
Also note where rejection lives and where it does not. The veto layer only moves single-step routing risk onto the judge tier. If the gold skill is not in the shortlist, the judge shares the same retrieval blind spot. A prior sequence routing study measured that, in composite tasks, the top-20 retrieval pool misses about 24 percent of the gold skill. It is insurance against the router being wrong inside the pool, not insurance against the pool itself. Class-specific thresholds are needed for the same reason: the C_r of read-only skills and destructive skills differs, and a single global threshold targets the wrong thing. The margin goes stale with catalog drift, that is, the new skills arriving every day, description rewrites, and deduplication. Recalibration over a rolling window (conformal recalibration) is the natural companion.
The paper’s detail page is here: The Veto Frontier: Measuring Skill-Router Calibration and the Asymmetric Cost of Abstaining vs. Auto-Routing in a 2,000-Skill Agent Harness
All three figures in this post are conceptual illustrations and analytical model schematics. No curve is a measured value. The numbers will be filled in by the live margins and metered judge calls of the protocol.