The Index Rots: Staleness Frontier and Re-Embedding Cadence of a 2,000-Skill Router
This post is for Korean cloud and AI engineers who run self-evolving agent skill fleets, or who own the retrieval index cost of such a fleet. The skill catalog changes every night, but the retrieval index does not keep up. The routing accuracy that this index once delivered is quietly rotting. This paper answers how fast a frozen index’s accuracy rots and what re-embedding cadence minimizes total cost. The answer comes in two formulas. A law that gives how accuracy decays, and a re-embedding cadence set by the ratio of two costs.
A visual metaphor for the article’s key idea.
In Plain Terms
Imagine a library that takes in new books every day and reshelves its stacks. The librarian updates the card catalog whenever a book is added or moved. But rewriting the whole catalog is expensive. For each volume, the content has to be re-read and the location re-written on the card. So the library rewrites the catalog on a fixed cadence, and in between, the catalog and the stacks drift a little out of sync. A reader pulls a card and walks to the location written on it. If the book has moved, the reader ends up holding the wrong book. The question is how often to rewrite the catalog. Rewrite it daily and the librarian’s work piles up. Never rewrite it and readers keep picking the wrong book. This paper prices that balance and derives the optimal cadence.
The library is the 2,000-skill registry, the cards are the retrieval index entries, and the librarian is the local embedding server that rewrites the cards. A reader picking the wrong book is the same as the router sending a request to the wrong skill and the whole task being wasted.
Infographic generated by NotebookLM from the sources.
The Problem: The Registry Moves, the Index Is Frozen
The skill registry in this deployment is about 2,000 entries. The overnight self-evolution loop edits skills and rewrites descriptions, and over three weeks the registry grew from 1,600 to 2,029. The router, which sends each turn’s request to a single skill, merges the lexical BM25 lane and the embedding lane with reciprocal rank fusion (RRF).
The prior study Curse of Skills made registry size the variable and showed that the recorded monitoring point is the signature of a size-axis survival model. That monitoring point has Recall@5 of 86.7% and top-1 of 48.9%. But in that study the index was implicitly fresh. The decay in that model is in size N, not in time.
The embedding index is a frozen artifact. Between re-embeddings, the registry moves underneath the index. New skills are not in the index, and changed descriptions do not remain in it. Dynamic vector index research keeps index structure consistent at sublinear cost when entries are added or removed, but it assumes the vectors are immutable. In a self-evolving registry, the source text of the vector itself is rewritten by the evolution loop. Structure maintenance tricks cannot rescue stale vectors. The intervention is re-embedding, and the question is when.
So the research question has two parts. First, as a function of index age t and churn rate κ, how fast do top-1 and Recall@5 fall. Second, given the full re-embedding cost and the downstream cost wasted by wrong routing, what cadence minimizes total cost. In library terms, it asks which is cheaper: the librarian’s work of rewriting the catalog, or the reader’s waste of picking the wrong book.
The Decay Law: A Floor, Not a Cliff
The model rests on four assumptions. Entries change at a constant failure rate, stale entries give the embedding lane uninformative scores, the lexical lane always holds the latest text, and collisions among competing entries are nearly independent. Under these assumptions, the staleness fraction s(t) = 1 - e^(-κt) drives everything. When the correct entry goes stale, the probability of finding the answer drops by d_r, and the expected number of collisions per query rises by Δλ. Top-1 accuracy is P(t) = r(t)·e^(-λ(t)), and Recall@5 is the same form multiplied by the Poisson probability that fewer than five competing entries beat the correct one.
The consistency check passes. At t = 0, the law reproduces the recorded points exactly. If the churn rate is 0, it becomes a constant in time at the recorded level, which is precisely the domain of the size-axis survival model. The size axis and the time axis meet at (N recorded, t = 0), and when the two studies are combined, the two-axis accuracy surface A(N, t; κ) is complete.
At the observed churn rate (about 0.02 per skill per day), the model says this. Top-1 loses 2.8 points by day 7 and 9.1 points by day 30. The total loss is 18.3 points, and the floor is about 30.5%.
Analytical model prediction at the observed churn rate (κ = 0.0193/skill-day) and reference parameters (d_r = 0.071, Δλ = 0.397). Only the t = 0 anchor (top-1 0.489 / Recall@5 0.867) is measured; every other curve value is model output, not measurement. Top-1 falls concavely by 2.8 points at day 7, 9.1 points at day 30, and 18.3 points in total (floor 0.305), and Recall@5 falls 6.4 points in total (floor 0.803). The dashed line is the lexical-only floor predicted by the model. The paper discloses that the reference parameters disagree with the recorded terminal point (BM25-only arm, hit@1 0.5278 / hit@3 0.7778, 36 golden queries, 1,910-entry catalog). (Analytical model, not measured)
In other words, in human terms: leave it a month and 40 percent of the registry is stale and first-recommendation success drops by more than 9 points. Recall@5 loses 6.4 points in total and ends at 80.3%. Half of the loss happens within about 37 days.
The shape is concave. Fast at first, flat later. Because the staleness fraction saturates, the last 20 percent of the registry accounts for only the last 4 points of first-recommendation accuracy. Where the floor sits matters more. A fully stale index is precisely the configuration of switching off the embedding lane and using the lexical lane only. The companion quantization study measures that configuration on its own golden set and records hit@1 52.8% and hit@3 77.8%. The library is stale, but the reader still finds books. The floor is a working configuration, not a dead state.
An important premise disclosed at the front of the paper. The terminal parameter pair (d_r, Δλ) is not anchored to any recorded measurement in this suite. The paper carries reference values (d_r about 7%, Δλ about 0.4) and discloses that they disagree with the recorded terminal point. At the reference values, terminal top-1 is about 30.5%. That is about 22 points below the recorded terminal point of about 0.5. Until the frozen arm re-anchors this pair, every decay magnitude is model output of a reference pair that recorded evidence does not support.
The Re-Embedding Cadence: Square-Root Law and Break-Even
Two cost scalars price that balance. C_re is the cost of re-embedding the full 2,029-entry registry and rewriting the index on the local embedding server, and c_q is the cost a wrong routing wastes downstream per day per point of accuracy. The total cost rate for a Δt-day cadence is C(Δt) = C_re/Δt + c_q·L̄(Δt). The first term is the re-embedding cost amortized per cycle, and the second converts the average accuracy loss within the cycle into downstream cost.
In the cheap regime, where the cadence is much shorter than the time constant (about 52 days), the loss starts like a linear ramp and the optimum takes a square-root form. Δt* = √(2·C_re / (c_q·P’(0))). Halve the re-embedding cost and the optimal cadence shortens by 30 percent. Double the wrong-routing cost and it shortens by the same 30 percent. It is the same form as the classic inventory replenishment cycle. The library’s catalog rewrite has become an inventory problem.
In dimensionless form, the operating point is set by a single number, ρ. ρ is the cost of one re-embedding divided by the total accuracy loss over one time constant (about 52 days). If ρ is about 1/1000, the optimum shortens to about 2 days; at 5 percent it is about 17 days; at 50 percent it stretches to about 3 months. Never re-embedding at all is optimal only when ρ is about 1 or more, that is, only when one re-embedding costs more than the total loss over one time constant.
The cost landscape is flat around the optimum. At ρ = 1 percent, the 7-day cadence works out as the cheapest. Five days costs about 5 percent more, and 30 days about twice as much. Never re-embedding sits far above the optimum. So an operator can pick a cadence that fits the calendar and lose only a few percent.
Analytical model prediction at ρ = 0.01 and the reference parameters. The daily total cost C(Δt) = C_re/Δt + c_q·L̄(Δt) is U-shaped in the re-embedding cadence. The minimum sits at the 7-day cadence (0.0281, in c_q units); 5 days is 1.05x and 30 days is 1.94x. Never re-embedding at all sits far above the minimum at c_q·L∞ = 0.1834. (Analytical model, not measured)
The break-even rule turns the judgment into a single inequality. For a fixed cadence to beat doing nothing, the cost of one re-embedding must be smaller than the downstream loss that cadence prevents. C_re < Δt·c_q·(L∞ - L̄(Δt)) is that condition. On a weekly scale, the one-week average loss is 0.01 points, and doing nothing sits at 0.18 points.
The paper also evaluates a threshold policy that re-embeds once cumulative loss since the last re-embedding reaches C_re/c_q. When ρ is very small it tracks the optimum within 5 percent, but at ρ = 1 percent it lags by more than 4.6x because it does not know the curve where it saturates. The paper recommends the fixed cadence.
The operating recipe is short. Measure C_re once, measure c_q over a week of real traffic, compute ρ, read off the cadence, and schedule the re-embeddings. ρ must be recomputed every month. κ flows with the evolution loop, and the more descriptions get rewritten, the shorter the optimum becomes. In library terms, if the librarian reshelves the stacks often, the catalog has to be rewritten often.
What Remains for the Company, for Society, and for Science
For the company, a concrete re-embedding budget remains. Instead of re-embedding the 2,000-skill registry every night unconditionally, or never, the fleet re-embeds only when the measured accuracy decay cost exceeds the measured index rewrite cost. That protects the recorded routing metrics from the quiet decay created by the evolution loop’s own churn.
For society, a budget that self-evolving agents were missing remains. Self-improving agent systems are spreading, but without a freshness budget, the autonomous skill evolution those systems depend on quietly rots the router that allocates the work. A published accuracy decay law and break-even cadence rule lower the compute and energy cost of trustworthy self-evolution automation for any team running an agent harness.
For science, the time axis that static benchmarks cannot see remains. This paper makes index age the variable for a hybrid retrieval index under controlled registry churn and derives the accuracy decay law and the cost-optimal re-embedding cadence. It extends the static registry size scaling law onto the time axis. When the two studies are combined, the accuracy surface A(N, t; κ) spanning size and time is complete. The paper pre-registers a measurement protocol to re-anchor the terminal parameters in a 30-day frozen arm and to falsify the decay law under controlled churn injection and numeric criteria.
Infographic generated by NotebookLM from the sources.
What Is Not Yet Trustworthy
First, the reference values themselves. The recorded terminal point sits about 22 points above the terminal top-1 the reference values predict. That means the recorded evidence does not support the reference values. Until the frozen arm re-anchors them, the loss depth and the floor are all model output. The model’s loss depth is 18.3 points for top-1 and 6.4 points for Recall@5, and the floors are 30.5% and 80.3%.
The best evidence for where the real floor sits is the recorded terminal configuration on its own set, and it sits far above the model’s floor. Second, the constant failure rate assumption. Churn comes in overnight waves, and κ is a time-averaged rate. The frozen arm measures the rate that is actually effective. Third, the assumption that stale entries are uninformative. If stale descriptions actively mislead, the terminal state sits below the lexical-only configuration, and the model’s floor becomes an optimistic upper bound. In that case the measured floor is below the model value, and the protocol’s falsification criteria are designed to catch exactly that.
Fourth, the unmeasured cost scalars. C_re and c_q are inputs that the protocol measures, and this paper measures neither. Until then, the operating claim is the form of the square root in the cheap regime, the break-even inequality, and the rule that doing nothing is optimal only at ρ about 1 or more. Not calendar dates. Fifth, a single harness. It is one Korean-English bilingual registry, one RRF fusion, and one local encoder. English-only registries and learned fusion are unverified. The portable claim is the structure: decay that falls exponentially in the staleness fraction, a measurable floor, and the square-root cadence.
The paper detail page is available here: The Staleness Frontier: Accuracy Decay of Frozen Skill-Router Embedding Indexes Under Registry Churn and the Cost-Optimal Re-Embedding Cadence
Both figures in this post are analytical model output at the reference parameter values. The measured points are only the t = 0 anchor (top-1 48.9% / Recall@5 86.7%) and the recorded terminal configuration (hit@1 52.8% / hit@3 77.8%, 36 golden queries, 1,910-entry catalog).