The Selection Break-Even: The Registry Size at Which Indexing and Reranking First Recoup Their Own Cost
If you run unattended agent harnesses, or you are the engineer who answers for their inference bill, this post is today’s work. Over a registry of N skills, every query must be routed to a single skill before it starts, and that call stands on the hot path of every query, adding its tokens and latency straight onto the serving bill. Two architectures compete for that routing call. The first is in-context selection: the whole manifest, the concatenated text of N skill descriptions, goes into the system prompt and the LLM picks directly. The second is retrieval-based routing: build an index, retrieve the top k shortlist per query, and pick from within it. There is also a third arm that optionally slots in an LLM reranker. The paper puts both on the same cost-quality axis and asks the question nobody has answered yet: at what registry size N do indexing and reranking first recoup their own cost? All prior studies measured only retrieval quality at a fixed, single scale, and that boundary is open. Every number here is an interpretation-model prediction derived under stated assumptions. The paper pre-writes the measurement protocol and falsification criteria that would overturn its prediction.
A visual metaphor for the article’s key idea.
The Blank Space Left by Fixed-Scale Evaluation
The tool-selection literature has studied the two arms separately, and at fixed scale. ToolLLM ran retrieval-based tool selection over 16,000 or more real-world APIs, and Gorilla wired a large API corpus through RAG. PORTS trained a preference-optimized retriever from the premise that LLMs struggle with large tool sets, and Toollery showed that when a library grows from hundreds to tens of thousands of functions, whole-prompting collapses on cost, latency, and reliability all at once, and built a way to compress candidates without training. A production skill router reaching 34,396 skills is also documented. The common point is one. Every reported quality, recall and precision and selection accuracy, was measured at a single, frozen registry size.
So the break-even question for the architectures is open. If N is small enough, the overhead of retrieval infrastructure can cost more than a single manifest call; if it is large enough, the manifest call collapses on both cost and quality. Fixed-scale evaluation was not designed to find that boundary.
The paper raises that evaluation a step. It sweeps registry size N and promotes direct in-context selection over the whole manifest to a first-class competitor on the cost-quality frontier, judged on the same axis as the retrieval arm by per-query cost and off-route-penalty-weighted quality.
ThakiCloud’s previous three studies sit in the same context. They established the limits of compositional skill routing on a production harness of roughly 1,600 skills, the relation between dense-side quantization error and fusion weights on a hybrid BM25+embedding router, and the per-stage cost-quality frontier of router inference, respectively. All three assumed retrieval routing was already built and asked only how to run it well. This paper asks the question one step ahead of them. Is retrieval routing worth building at all, and does the answer change with N?
Infographic generated by NotebookLM from the sources.
The Cost-Quality Model for Three Arms, and Two Crossovers
First, the arms. A1 in-context ends in a single model call with a system prompt carrying the manifest M_N. A2 retrieval has a retriever that fuses BM25 lexical scores with dense embedding similarity and returns the top k shortlist S_k(q) per query, and the model picks from within it. A3 has the retriever return the top k’ (k’≥k), an LLM reranker reorders or compresses them down to the top k, and the model picks.
The cost model writes the three as functions of N. A1’s per-query cost is C1(N)=b1+p1·N. b1 is the base cost of a single model call, and p1 is the effective manifest token price per skill. The manifest is static between queries, so prefix-caching serving discounts cached input by a discount rate ρ, making p1=(1-ρ)·ℓ·p_in. Caching does not change linearity; it only lowers the slope. A2’s cost is C2(N,k)=b2+p2·k+r(N)+a(N). p2·k is the shortlist token price that varies per query and cannot be cached, and r(N), the per-query retrieval cost of query embedding, inverted-index BM25 scan, and hybrid fusion, is nearly logarithmic in N for a tuned index. a(N) is the O(N log N) index build cost amortized by the query count between rebuilds, plus the upkeep share for skill additions and deletions and description edits. A3 only adds the reranker call p3·k’+b3 on top. Retrieving k’ leaves r(N) and a(N) unchanged to first order.
The A2-vs-A1 cost crossover comes out in closed form. In the operating regime where A2’s N dependence is nearly logarithmic and the amortized index cost reads as a constant, writing the sum of shortlist price, retrieval cost, and amortized index cost as r0 gives N_cost=(b2+r0-b1)/p1, beyond which A2 is cheaper than A1. The crossover is a line crossing: linear A1 and nearly flat A2 meet at N_cost. The lower p1 and the larger b2 and r0, the farther right N_cost is pushed; a high prefix-caching discount ρ lowers p1, so the manifest call stays competitive to larger N. k’ enters only A3 and does not participate in the A2-vs-A1 crossover.
Per-query cost by arm as registry size N grows. The in-context arm’s cost C1(N) is linear in N, and the retrieval arm, nearly flat, crosses the linear manifest-call curve at N_cost. Interpretation-model curves, not measurements.
The quality model writes each arm’s error as a function of N. A1’s selection error is the sum of three channels: readout confusion η_readout, distractor and lexical-competition mass η_distractor, and positional bias η_positional. All three channels grow with N. The evidence for readout confusion points at where selection failure lives. In real tool-selection failures, the per-candidate attention argmax already lands on the correct tool 80 percent of the time, and chance is 21 percent. The model sees the answer but cannot pick it. Prompt-side repairs, reordering or duplicating the correct tool, recover only 23 percent of those failures, while readout-side interventions recover 59 to 91 percent. Blaming the failure on visibility would be wrong. The bottleneck is decision readout. Distractor mass grows with the registry’s near-duplicate density. Exposing all skill descriptions in the prompt does not help. Partial exposure creates lexical competition among similar descriptions, an effect that pins correct selection down. There is also measurement showing that as a library grows from hundreds to tens of thousands of functions, each added candidate brings a new distractor and reliability collapses. Positional bias is the sum of two length effects. Long-context usage is U-shaped in position, and reasoning performance drops as input length grows even for the same task. A description buried deep in a long manifest is disadvantaged by position and by reasoning at once.
A2’s quality is a product of two factors: Q2(N,k)=R@k(N)·S(k). S(k) is the probability of picking the correct skill when it is inside the k-list, the shortlist selection confidence. For small k it is nearly saturated. That matches the evidence that attention already localizes the answer in small pools. R@k(N) is full-registry recall, and its bound is not the algorithm but the registry’s own discrimination limit B(N). One limit is the cross-lingual gap. In ThakiCloud’s previous production harness, roughly 1,600 skills under Korean-English mixed queries, even the full manual-decomposition ORACLE stayed at 63.6 percent step-coverage. The bound is that the retriever’s cross-lingual lexical bridge binds recall with it. That number is a measurement of that harness, and this paper writes a working assumption that transfers it to N=2,029. The SRA protocol tests exactly that premise. The second limit is functional duplication. Near-duplicate skills create recall that no single correct-answer expression can reach.
A3’s gain is paid only on near ties. The precision the reranker buys is R@k’(N)·S(k) plus Δ_rerank, bounded by Δ_rerank ≤ Δ_tie·τ(k’). Δ_tie is the maximum per-query quality lift when both the correct skill and the current top-1 sit in the tie band, and τ(k’) is the expected probability that both the correct skill and the top-1 are in the tie band inside the top k’ list. When retriever scores split cleanly at a cliff or a semantic boundary, the order is already right, and dynamic top-K at that point captures most of the precision with fewer tokens. The residual value is only the mass the base retriever leaves in the tie band. The price is measurable. An LLM reranker adds roughly $0.2 to $0.3 per 1,000 queries and about 1 second of tail latency. A learned per-query controller skips 51 percent of reranking calls, gives up 1.2 points of held-out coverage as the price, and explicitly accounts for the queries where skipping would hurt.
Selection quality by arm as registry size N grows. In-context quality Q1(N) declines toward a positive error floor, and retrieval quality Q2(N), nearly flat, crosses it once at N_qual, after which the retrieval arm dominates. Interpretation-model curves, not measurements.
The quality crossover is unique as well. If Q1(N) is non-increasing in N with a positive error floor, the error gap between the two arms e2(N)-e1(N) is non-decreasing in N, A1 leads at small N0, and the error floor crosses retrieval error at the end of the interval, then there exists a unique N_qual where Q2 overtakes Q1. Below N_qual, A1 dominates because the small manifest is selected almost trivially; above it, the retrieval arm dominates.
The Break-Even N* and the Deployment Rules
Combining the two crossovers gives the break-even N. δ(N)=μ·(Q2-Q1)(N)-(C2-C1)(N), and N is the smallest N where δ(N) crosses 0. Here μ is the off-route penalty, priced as the product of the expected downstream quality drop and the execution cost of a single off-route event. An off-route event is spent immediately at the value the execution was supposed to produce. If N is above both N_cost and N_qual, both terms of δ are plus at the same time; below both, both are minus. If the relevant functions are monotone in between, N* sits in the window between min(N_cost, N_qual) and max(N_cost, N_qual), and finding N* reduces to a one-dimensional sweep over registry size. If μ is so small that μ(Q2-Q1) cannot touch C2-C1 over the range of interest, N* does not exist in that range, and the manifest call is the net-profit choice all the way.
Once N is past N_qual, δ increases in μ. A larger μ pushes N* down. High-stakes routing is obligated to buy the index earlier. Break-even is not just a property of the registry; it is a property of the stakes the registry carries.
Reranking comes with a residual-k rule. A3 beats A2 when its net gain δ_r is positive, and since the rerank gain is bounded by Δ_tie·τ(k’), the call paying for itself requires τ(k’)·Δ_tie·μ > p3·k’+b3+λ. λ is the price of reranker tail latency. The tail latency of a production LLM reranker is about 1 second. The inequality reversing is a sufficient condition to skip the call. The practical proxies are two. One is to monitor retriever score cliffs and semantic boundaries and skip the call where tie mass is low. The other is per-query control that removes the call from queries with negative expected gain. In one production setting of the latter, 51 percent of reranking calls are skipped at the price of 1.2 points of held-out coverage.
Compressed into deployment rules, this is four lines. If N ≤ N, do not build the index; route with a single LLM call over the whole manifest. If N > N, build the index and use two-stage routing that retrieves the top k and then selects. Turn on LLM reranking only when the residual duplication in the top list exceeds the residual-k threshold, gated by per-query score-cliff monitoring. And N* must be recomputed every time the registry grows tenfold, every time the model changes. The p1, p2, and η terms depend on the model and serving.
Plugging in the 2,029-skill registry gives the prediction. A hybrid BM25+dense router, H200 self-hosted quantized serving, and all arms share the same model. At 40 to 120 tokens per description, the 2,029-item manifest is roughly 80k to 250k input tokens per call. Input-length performance degradation, U-shaped position usage, and the evidence for readout confusion: that is the regime that predicts direct selection as unreliable. The retrieval arm’s recall is bound by the ceiling transferred from the previous harness’s 63.6 percent ORACLE step-coverage. The model predicts that at this size the two-stage arm A2 dominates A1, the margin grows with μ, and reranking value concentrates in the registry interval where near duplicates are dense, where tie mass is large. All of this is model prediction, not measurement in this paper. It is what the SRA protocol confirms or overturns.
The 63-Case SRA Audit and Pre-Registered Falsification
The design that hands the prediction to measurement is the SRA, the Selection-Retrieval Audit. A 63-case stratified audit. This paper presents only the protocol specification; it reports no measurements.
The 63 cases split into three near-duplicate-density strata, low/medium/high, 21 cases each, and test the η_distractor and rerank tie-mass terms intensively. Within a stratum, 12 cases have single-skill gold, and 9 are compositional gold multi-skill tasks. The latter are scored by F1, testing the set-selection behavior of diversity-aware reranking. 14 cases are cross-lingual: 7 Korean queries against the English registry and 7 English queries against the Korean annotation segment. The remaining 7 cases are single-word queries. Every case is labeled with density, gold type, and language alignment.
All three arms run the full 63 cases on the same model under H200 self-hosted quantized serving. The manifest prefix is cacheable, so cached and uncached tokens are recorded per arm, and per-query cost is computed by using the C1, C2, and C3 formulas as-is with the measured cache discount rate ρ for p1. Per-case (C_a, Q_a) points are collected case by case to estimate each arm’s frontier, and confidence intervals are obtained by stratum bootstrapping.
The three analytic claims are tied one-to-one to pre-registered falsification predictions per model. First is the central prediction P1. The prediction that A2 dominates A1 at N=2,029 is overturned if the measured break-even is N* > 2,029, equivalently δ(2,029) ≤ 0. Second is the quality ordering P2. The prediction Q1(2,029) < Q2(2,029) is judged a wrong η floor when measured Q̂1 ≥ Q̂2 and measured R@k exceeds the discrimination bound the model emits, so the ordering cannot be explained by recall. Third is the cost crossover P3. The linear cost model is overturned if the measured A1/A2 cost crossover is nonlinear in N beyond the stated logarithmic perturbation, including the case where the manifest call cost is nonlinear in N even under the measured cache discount rate.
Case allocation across near-duplicate-density strata for the 63-case Selection-Retrieval audit. Three strata (low/medium/high) of 21 cases each, a design that tests the distractor-mass and rerank tie-mass terms. A design diagram of the interpretation model, not a measurement.
What Remains for the Company, for Society, and for Science
For ThakiCloud, this rule is directly the deployment criterion. The 2,029-skill registry is expected to sit in the two-stage routing regime, the margin grows with the off-route penalty, and reranking turns on only under the residual-k condition. The decision rule for the token factory router configuration compresses to one line: at or below N, a single LLM call; above N, two-stage routing; reranking only when tie mass crosses the threshold.
Socially, it lowers the barrier to the agent skill ecosystem for small organizations. This rule is negative information. Until N* is crossed, there is no need to build the index, the retrieval stack, or the reranker. Below it, the single LLM call is the net-profit architecture, and skill routing is a prompt-design problem, not a systems-engineering problem. This analysis supplies the threshold itself, up to the moment the registry crosses it.
Scientifically, it raises the fixed-scale tool/skill retrieval evaluation paradigm a step. Retrieval quality measured at a single frozen size does not say whether it was worth building the index. Break-even is the quantity that says exactly that. The paper sweeps N and puts the controlled head-to-head of in-context selection vs retrieval vs retrieval+reranking, and the N* curve, on the same cost-quality frontier.
Infographic generated by NotebookLM from the sources.
What Not to Trust
This paper is analytic and reports no measurements at all. Every number is a value derived from stated parameters.
First, N* is parameter-sensitive. If there is uncertainty in p1, ρ, or the η slopes, N* cannot be read as a point. It must be read as an interval, and the deployment rule must be read at the conservative end of that interval. Second, the plug-in is a single harness. One registry, one serving stack; moving to another harness requires re-estimating the model- and serving-dependent terms, including the transferred cross-lingual bound. Third, the registry moves. Staleness and churn in skill descriptions move r(N) and a(N) and the discrimination limit, so N* drifts. The recomputation trigger is part of the design, not something that can be pushed off into a footnote.
The paper detail page is available here: The Selection Break-Even: Measuring the Registry Size at Which Retrieval-Based Skill Routing Beats Direct In-Context LLM Selection on the Cost-Quality Frontier
The three figures in this post are all interpretation-model curves and a stratified design diagram, and contain no measurements. The 63.6 percent ORACLE step-coverage cited in the body is a measured value from a previous production harness (roughly 1,600 skills); it is a transfer based on a working assumption, not a measurement taken at 2,029 skills.
