Does Adding Votes to a Skill Router Improve Accuracy? We Costed the Ensemble Before Building It

If you run an agent harness that has to pick one skill out of many, and you have been looking for a way to make that router smarter, this post is for you, especially if you have already had the thought that adding one more model to vote might raise accuracy. This research is an analysis that builds the cost model and the pre-registered evaluation procedure before actually implementing that intuition. To state the result up front: this paper did not run an experiment. Instead, it built a calculator for deciding whether running one is worth it, and that calculator already produces an interesting answer.

The Question: Why Does a Router Always Look at Only One Signal

A production LLM agent harness has to decide, for every incoming request, which of thousands of installed skills to load into context. Once the skill count passes a few hundred, this single decision governs the quality of the entire harness. A skill that is never retrieved is as good as absent, and a skill that is retrieved incorrectly eats into the context budget while steering downstream agents in the wrong direction.

The system this research studies is an actual production harness with over 1,000 skills, where Korean and English frequently mix within a single request. Today this router combines a BM25 lexical search score and a 512-dimensional dense embedding similarity via reciprocal rank fusion (RRF) to compute one score per candidate skill for every query, then returns only the candidates that clear a threshold. This structure is not unusual. The routing and cascading literature broadly converges on this shape: pick one model, decide whether to escalate, or produce one ranked list. The number of independent judgments aggregated per query is always one.

A separate strand of literature, meanwhile, has built up evidence that combining multiple independent judgments raises accuracy. Self-consistency samples multiple reasoning paths and takes the mode answer, mixture-of-agents combines the outputs of multiple models, and confidence-weighted protocols give different weight to different participants. The problem is that these two strands have rarely met. Routing papers do not ensemble, and ensemble papers do not route. This paper asks its question in exactly that gap: if you combine votes from cheap heterogeneous model tiers (haiku- and sonnet-class judgments, for example) with the existing BM25/embedding hybrid retrieval score using confidence weighting, does gated Recall@5 and Top-1 accuracy rise above what a single-signal router achieves, and what does that cost in tokens?

This research is also a direct follow-up to two earlier pieces of work on the same system. One had already exhausted the single-signal lever (fixing the retrieval signal itself) and left open the question of what lever comes next. The other attempted online-bandit auto-recalibration on this exact same router and returned a null result. This paper formalizes the lever proposed as the next step: treating the router’s output not as a single score but as one vote among several.

What We Analyzed and How: We Built a Calculator, Not an Experiment

It is worth being precise here. This paper is not an empirical study that actually ran an ensemble router and measured its accuracy. It builds a cost model directly on top of three numbers already measured on this system. Looking at the current production router’s hybrid retrieval performance, gated Recall@5 rose from 66.7% to 77.8% and Top-1 rose from 33.3% to 53.3%, with the hallucination rate at 0.0% in both cases. It also carries over the cause of the null result from online-bandit recalibration on the same system: a positive-only reward function that never saw the hallucination-rate dimension at all. On top of this, it incorporates a cost-quality Pareto frontier measured on the same harness for adversarial verification, a mechanism structurally similar to voting, across 27 configurations and 180 real API calls. The cheapest tier was the sole Pareto-optimal point, and adding more voters was confirmed to actually reduce safety through a Condorcet-style vote-dilution effect.

Two necessary conditions are derived from this cost model. One is a break-even inequality between marginal accuracy and marginal cost, and the other is a precondition that each individual voter’s accuracy must satisfy. Showing that these two conditions are non-trivial is the paper’s substantive contribution. And the paper pre-registers a concrete evaluation protocol for testing these conditions and hands it off to future empirical work, specifying exactly which arms to run, on which benchmark, against which success criteria.

The Proposed Architecture: Keep the Retrieval Signal as Voter 1, Do Not Replace It

The proposed ensemble router keeps the existing retrieval score as voter 1 and adds cheap model-tier classifiers as voters 2, 3, and so on. Each voter returns a probability distribution over the candidate skill set, including an abstention option meaning “this request needs no skill,” along with a confidence signal. These distributions are combined with confidence-modulated softmax weighting.

A query is narrowed to a candidate set Cm (m=8). Voter 1 is the free hybrid retrieval signal, and voters 2 through K are model-tier classifiers. These are combined with confidence-weighted softmax and an abstention threshold. This is a conceptual diagram, not a measurement, and does not reflect actual benchmark results.

Two decisions in this design are deliberate. First, the retrieval signal is used additively, never as a replacement. It is already computed for free on every query, it is Pareto-optimal among the single signals measured so far, and it fails in structurally different ways from LLM voters. The paper’s judgment is that there is no reason to discard a signal that is free, optimal, and structurally distinct, only to replace it with a paid one. Second, it uses confidence weighting rather than simple majority vote. There is a counterexample showing that pooling models that share training data and architecture under plain majority vote does not cancel errors but actually amplifies them, so the design lets stronger voters carry more weight. The retrieval signal’s weight is structurally floored so it never falls below zero, meaning that in the extreme case, reverting to current production behavior is a single parameter change away.

Costing It Out: Adding One Voter and Adding Three Are Completely Different Stories

The most substantive result in this paper comes from the cost side. The relative per-call token cost of the haiku, sonnet, and opus tiers is 1x, 4x, and 19x respectively. Since the marginal cost of the current retrieval-only approach is zero, adding a single haiku voter costs 1 unit, keeping haiku plus sonnet always on costs 5 units, and keeping haiku always on while escalating to opus only when they disagree (assuming a 10% disagreement rate) costs 2.9 units.

You can calculate how much accuracy gain is needed to justify this cost. If we conservatively value turning one wrong answer into a right one at 20 or 50 haiku calls, adding a single haiku voter pays for itself with just a 5.0 percentage-point (conservative estimate) or 2.0 percentage-point (generous estimate) rise in Top-1 accuracy. Keeping haiku plus sonnet always on, by contrast, requires 25.0 percentage points, which is larger than the largest improvement this system has ever achieved from any single change (20.0 percentage points, from moving BM25 alone to the hybrid signal). In other words, “add one cheap voter” is a realistic bar to clear, but “keep a mid tier always on” demands an improvement no single change has ever delivered on this system so far.

The Top-1 accuracy improvement each arm needs to clear break-even, under two assumptions about the value of recovering one wrong answer (gamma=20, gamma=50). This is an analytical model, not a measurement, derived by substituting the per-tier token cost ratios (haiku 1x, sonnet 4x, opus 19x) into equation (3). The escalation arm (D) assumes a 10% disagreement rate.

The additional token cost each voting scheme incurs over the free production baseline, in units of haiku calls. The cost of the escalation arm (D) is a stochastic value that depends on the disagreement rate, so it is shown at the 10% disagreement rate (=2.9 units) and compared against the always-on arms B and C. Latency cost is not included.

This cost model also explicitly pulls in two risks already measured on this system. The first is the failure pattern seen in bandit recalibration: tuning to optimize Top-1 alone reduces abstention probability, which quietly raises the hallucination rate. So the hallucination rate is treated not as a tradeoff variable but as a constraint pinned at its current value (0.0%). The second is Condorcet-style vote dilution: if an individual voter’s accuracy falls below a threshold, adding more voters makes results worse rather than better, so the paper pins down a precondition that each candidate voter’s standalone accuracy must be measured before assembling the ensemble.

The Strongest Objection and the Reply: The Kind of Heterogeneity Differs

The strongest objection to this proposal comes from an ensemble study covering 67 frontier models. That study showed that simple majority voting was worse than the single best model in most cases, and traced the cause to a co-failure ceiling: models that share training data and architecture family fail on the same items, so errors do not cancel out.

This paper takes that objection head-on but answers it narrowly. The mechanism that study identified concerns correlation between voters, not that ensembling is meaningless in general. In this proposal, voter 1 is not a language model but a similarity function over a fixed representation. It fails on lexical mismatch, descriptions written in different tones, and lexically similar duplicate skills. LLM voters fail on contextually plausible distractors, ambiguous instructions, and position bias within long candidate lists. These two failure sets are different. The paper’s answer is that mixing voters with structurally different input modalities and judgment procedures is a stronger form of decorrelation than diversity within a pool of language models alone.

The paper is honest, however, that this answer is still an untested hypothesis. How much the error sets of the retrieval voter and the model voters actually overlap has never been measured, and it flags measuring this as the single most valuable number future empirical work could return. If the error sets turn out to overlap heavily, the co-failure ceiling applies as-is, and the paper itself states that the right response would be to discard the proposal, not tune it. There is a real chance of that, given that LLM voters only ever see the candidates retrieval has already selected.

What This Contributes to the Company, Society, and Science: Why This Calculation Needed to Happen Now

From the company’s perspective, this paper aims directly at an actual production system that gates skill routing on every turn. It builds a basis, grounded in measured numbers rather than intuition, for revisiting the earlier decision to hold heterogeneous model-candidate aggregation for a new build. Whether adding one cheap voter turns out to be worth it or whether the current setup is already right, either conclusion substantially lowers the cost of deciding the next execution step.

Socially, this feeds into the broader question of whether an ensemble of cheap heterogeneous models can substitute for one larger single model. If it holds, it opens a path to lowering the compute and energy cost of correctly executing one agent action without scaling up model size. Scientifically, the key contribution is placing two literatures that have rarely met under one aggregation rule so they can be compared directly: the retrieval-routing literature, which has converged on a single signal, and the multi-agent consensus literature, which has barely considered non-model voters. Turning the error correlation between two structurally different decision-makers, a retrieval score and an LLM judgment, into a measurable question has implications that reach beyond this one router.

Limitations: This Paper Sharpens a Question Rather Than Answering It

As noted above, the most important limitation is that this paper does not report empirical results. The proposed protocol is written to be executed directly as future work, and no statement in this paper should be read as an empirical verdict on the effectiveness of the proposed method. Every quantitative figure used here comes from unpublished, non-peer-reviewed internal technical-report numbers maintained by the same organization across two internal evaluation harnesses (a 63-item regression suite and a 12-item compound-routing suite), not public benchmarks. What is fully specified is the protocol itself: the arm configurations, metrics, precondition gates, and success criteria are all defined, so it can be applied to any similarly labeled skill-routing benchmark.

The per-tier cost ratios (1x, 4x, 19x) are approximations that shift with provider pricing and model versions and would need to be recomputed before real-world application. The break-even inequality is only a first-pass screening estimate, not a proof of optimality, and it prices accuracy linearly while ignoring latency. The study behind the co-failure objection comes from general LLM aggregation tasks rather than routing, so whether that ceiling transfers to retrieval-augmented routing is itself an open question. The vote-dilution threshold is a conservative placeholder carried over from a measurement on binary-decision tasks into an eight-way-choice task, and it needs to be re-derived. And because LLM voters only ever see the candidate set retrieval has already filtered, the paper also states plainly that this ensemble has a structural limitation: it cannot recover a correct answer that retrieval missed entirely, and can only improve re-ranking and abstention within the candidates retrieval already found.

The full paper is available here: https://huggingface.co/datasets/thaki-AI/daily-paper-2026-08-06-ensemble-tier-voting-skill-router