You Reason Once. You Get Billed Every Turn: The Formula for the Reasoning Verbosity Tax in Agent Loops
This post is for Korean cloud and AI engineers who serve multi-turn agentic LLMs directly or set cost budgets for tool-calling workloads. Whether to switch on thinking, how much thinking budget to allow, whether to leave reasoning in context: this post introduces a paper that puts a formula on the price those three decisions pay in the agent loop. A reasoning token is not finished once it is generated. Left in context, it is billed again on every turn. This paper names and prices the phenomenon of that bill growing quadratically with the horizon as the reasoning verbosity tax.
A visual metaphor for the article’s key idea.
Why Long Reasoning Gets Billed Again
An agent is not a system that answers one question; it is a system that runs a loop. In a multi-turn tool-calling agent, the model is invoked again on every turn, and the context it sees then has accumulated the reasoning traces, actions, and tool responses from previous turns. Running with thinking switched on leaves reasoning tokens in context on every turn. Those leftover tokens are re-billed at input token price starting from the next turn.
In practice, teams often fix thinking on/off once at the deployment level and just run it. The price of that fixation never shows up in a ledger that only watches dollars per task. In a ledger that reads only totals, extra reasoning tokens compound quietly on every turn. This paper starts by making that fixation, and its bill, visible.
The study that opened this axis is BDH-CQ. A 150M-parameter in-context model ran recurrent latent reasoning, reasoning that never speaks in language, and recorded pass@2 of 29.5% at a computational reasoning cost of USD 0.0007 per task on ARC-AGI-1. A small model broke the cost-accuracy Pareto frontier. The format and density of reasoning, explicit tokens versus compact computation, is itself a cost-quality axis independent of model capacity. This paper moves that axis into real agent serving. It fixes the model and varies only the reasoning state, isolating the format/density axis. What the agent loop adds on top of BDH-CQ’s single-task setting is re-billing. Explicit reasoning tokens left in context pay again on every subsequent turn. That re-billing, accumulated across turns, is the reasoning verbosity tax.
Infographic generated by NotebookLM from the sources.
Casting the Verbosity Tax into a Formula
The setup fixes the model family and the task set, and varies only the state. On H200, the Qwen3-8B student is served in bf16 and the Qwen3.8-27B teacher in NVFP4. The task set is BFCL simple, multiple, and parallel, plus the SRA routing bench with 63 cases. The reasoning state is switched with the thinking toggle of the Qwen3 chat template. The on/off switch and the thinking budget b are handles already attached to the serving stack.
On a turn with context length m, what is the per-turn tax of the explicit state E (reasoning v_E tokens) relative to the compact state C?
τ₁ = (p_in·m + p_out·(v_E + o)) / (p_in·m + p_out·o) > 1 (v_E > 0)
p_in and p_out are the amortized input and output per-token prices for H200 self-hosted serving, and o is the number of output/action tokens per turn. The tax increases in v_E and decreases in m. A long fixed context dilutes the tax, and a large thinking budget grows it. The first-order approximation is τ₁ ≈ 1 + p_out·v_E/(p_in·m + p_out·o). Looking at a single turn, it is nothing but a small ratio. What changes materially is the compounding.
The result that handles compounding is a lemma. Under the persistent policy, where reasoning remains in context, if v_E, o, and w (tool response tokens) stay constant over T turns, the absolute cost gap between explicit and compact is
v_E·(p_out·T + p_in·T(T-1)/2) → Θ(T²)
and under the discard policy, which wipes the chain-of-thought on every turn, it is p_out·v_E·T, that is, Θ(T). The two terms read differently. p_out·v_E·T is the decode tax: the cost of reasoning tokens generated once per turn at output price. p_in·v_E·T(T-1)/2 is the prefill tax: the same tokens re-sent on every later turn and re-billed at input price. Discard structurally removes that second term. That is why the quadratic-versus-linear curve is a measurable signature separating the context policies. The shape of the curve in the ledger tells you whether the reasoning was kept or wiped.
With persistent context, the absolute cost gap of explicit over compact grows quadratically in the horizon T, while under the discard policy it grows linearly. The graph is a conceptual illustration, not a measured value.
There is an upper bound as well. Under the standard context condition p_in·m > p_out·w, where the fixed system/tool context m is large, the relative tax τ_T of the persistent policy starts at τ₁ and increases monotonically toward the finite structural bound U = 1 + v_E/(o+w). The tax widens with the horizon, but it does not exceed the bound set by the reasoning share inside the growing context. Under the discard policy, τ_T decreases from τ₁ toward 1.
Under the standard context condition, the relative tax of the persistent policy opens monotonically from the single-turn tax τ₁ toward the structural bound U, and the tax of the discard policy converges to 1. This is a schematic of the analytic model, not measured data.
The Quality-Per-Dollar Break-Even Point
Cost alone cannot pick the state. A closed-form solution gives the condition under which compact beats explicit on quality per dollar. The condition is
τ_T > (q_E/q_C)^γ
where q_E and q_C are per-turn success probabilities and γ is the number of turns over which quality compounds. Where you place γ yields two quality models.
Model A is the deep dependency chain. Task success is Q = q^T, so γ = T. The quality premium (q_E/q_C)^T is exponential in T, while τ_T is bounded by max{τ₁, U}. If Δq > 0, then under either policy explicit wins once you cross the crossover horizon. On long dependency chains, the tax is worth paying.
Model B is the case where k critical turns carry the quality. A single routing decision is the representative example. Quality does not compound over T; γ = k, and the premium R = (q_E/q_C)^k is fixed. Under the persistent policy, if R < τ₁, compact wins at every horizon. If τ₁ ≤ R < U, compact wins for T > T_crit, and T_crit is given in closed form. If R ≥ U, explicit wins at every horizon. Under the discard policy, τ_T decreases from τ₁ to 1, so if R < τ₁, compact wins at short horizons and explicit wins at long horizons.
Both models point to a single practical conclusion. For workloads where the quality premium R sits below the bound U, with routing as the representative case, the winner at long horizons is decided by the context policy. In a persistent re-billing environment, compact is favored; in a discard environment, the decode tax drops to 1 and explicit is favored. This reversal, and the quadratic-versus-linear curve gap, are what the protocol measures.
An over-provisioned turn is defined as well. If the relative cost increment of turn t, p_in(t-1)v_E + p_out·v_E, exceeds the relative quality increment (p_in·m + p_out·o)·Δq/q_C, then that turn paid the tax and came back without quality. Two testable conditions follow. For Model B workloads with many routing decisions and short dependencies (k ≪ T), the set of over-provisioned turns is predicted to be non-empty; for the deep chain of Model A, it is predicted to be empty or rare.
The Optimal Thinking Budget Moves with the Horizon
Fix a task class and sweep the thinking budget b from 0 to b_max, and two structural facts hold. Accuracy Q(b) is non-monotone in b. It rises from Q(0), peaks, and can then fall in the range where the budget inflates the reasoning without the answer improving by as much. The evidence for this non-monotonicity comes from an earlier study that swept six budget values from 0 to 512 across 200 function-calling agent tasks. Cost C(b) increases in b. Under the persistent policy, a large b not only spends more on decode every turn but also generates more re-send prefill on every later turn, so the effective cost of a large b grows with the horizon.
The quality-per-dollar function F(b) = Q(b)/C(b) can have an interior optimum b. If there is a b where the quality gain of the peak outruns the cost increase, b lies between 0 and b_max; otherwise b* = 0. If b* = 0, the compact point itself is the quality-per-dollar optimum for that task class. The paper defines the maximum quality-per-dollar loss of the compact regime as the price of optimality, PoO = F(b)/F(0) ≥ 1. If the operator’s quality tolerance is ε, compact is acceptable when PoO - 1 ≤ ε. Under the persistent policy, b depends on the horizon as b*(T), and the frontier becomes a family of curves indexed by horizon rather than a single curve. Operating with one budget fixed across multiple horizons is an implicit trade of the frontier points the ledger should be tracking.
Because accuracy is non-monotone in the thinking budget b while cost increases monotonically, the quality-per-dollar curve can have an interior optimum b. If the price of optimality stays inside the operator’s tolerance, fixed compact (b = 0) is acceptable. Conceptual illustration, not measured values.*
The interaction with low-bit serving is captured by formula as well. Quantization inflates the reasoning length by a factor α > 1. Under NVFP4, the same regime emits α·v_E reasoning tokens. The tax excess τ₁ - 1 grows in proportion to α, and T_crit decreases monotonically in α, converging toward T = 1 as α grows. The bound U rises to 1 + α·v_E/(o+w) too. A regime that was break-even neutral in bf16 becomes clearly compact-favorable in NVFP4. A cheaper per-token price and more tokens per turn are a multiplication, not a competition. Serving precision is an input to the break-even calculation, not an afterthought.
Workload structure also classifies the state. Schema-based single tool calls and routing decisions sit near b = 0. Small reasoning models are strong instruction followers on function calling, and the SRA routing bench is a routing workload whose per-turn cost is dominated by the regime. The compositional multiple and parallel tool calls are the class with an interior b, where the budget buys call coordination. Under the BFCL mapping, the simple split is b ≈ 0, and the multiple and parallel splits are classes where an interior b is plausible. The SRA routing bench is the natural home of Model B, because routing decisions form a small set of critical turns, k ≪ T.
The 4arm Protocol Built to Falsify the Paper
Every result is structural; the protocol fills in the values. The protocol is a fully pre-registered, inspectable, rule-based design.
The platform is H200 continuous batching serving. There are two model points. The Qwen3-8B student in bf16 is the reference with no token inflation; the Qwen3.8-27B teacher in NVFP4 is the point that pairs token inflation with a cheaper per-token price to measure the α interaction. Metis meters the tokens and reports per-task amortized dollars against p_in and p_out.
The workloads are BFCL simple, multiple, and parallel, the SRA routing bench with 63 cases, and the multi-turn variants of each. The horizon T takes 1, 4, 8, and 16 to probe the compounding prediction of the lemma. Each multi-turn run declares its context policy (persistent or discard), and persistent runs are paired with discard pairs so that the quadratic-versus-linear signature is measurable.
There are four arms. Arm A is the explicit reference with thinking always on at the default budget; arm B is the compact baseline with thinking always off, v_t = 0. Arm C is the allocation arm that uses a difficulty trigger for a per-turn adaptive regime. Arm D fills the frontier with a budget sweep over b = 0, 256, 512, 1024, 2048, and 4096. Decoding pins temperature and seed, declared per run. Differences across arms are attributed to the regime rather than to sampling, and the task set within an arm is fixed to control budget-versus-difficulty confounding. All run-level controls are declared in the run config before metrics are computed, and the per-turn decomposition is preserved. The tax is verified cell by cell, not reconstructed from aggregates.
The metrics are per-split tool-call accuracy on BFCL, SRA routing recall@1, per-turn reasoning tokens, total input/output tokens, wall-clock latency, and Metis dollars per task. Everything decomposes per turn.
The way to read the arms is fixed. A versus B measures τ_T directly per workload class and horizon. D locates b* and PoO from measured Q(b) and C(b), confirming the sign of the proposition. C checks the over-provisioned corollary: if the difficulty trigger switches a turn to compact and the measured quality of that turn matches arm B, that turn was over-provisioned. The 8B versus 27B comparison uses matched per-turn accuracy as the control to contrast size and precision, bounding the reach of the α interaction.
The protocol is designed so that it can falsify the paper. There are four refutation criteria. R1: under the declared context policy, if the measured per-task cost delta fits Θ(T) better than Θ(T²) at T = 1, 4, 8, and 16, the persistent compounding mechanism is falsified. R2: if the measured quality-per-dollar ranking in arm B versus A, and in D, contradicts the sign predicted by the proposition in two or more workload classes, the break-even condition is falsified. R3: if the set of over-provisioned turns is empty in every workload class, the regime allocation prediction is refuted. R4: at matched per-turn accuracy, if the measured tax of NVFP4-27B is at or below that of bf16-8B, the α inflation interaction is falsified.
Composition of Operating Rules and Levers
The decision rule replaces state fixation with the ledger. Schema-based single tool turns and routing default to compact. Compositional tool calls keep an interior budget, and the cost meter sets the regime per workload class by the proposition’s conditions. In the routing class, the ledger compares the measured τ_T against (q_E/q_C)^k. Below the threshold, explicit stays on; above it, the class flips to compact. T_crit is the horizon at which that flip becomes permanent.
The four levers, effort, format, size, and serving, are mutually orthogonal axes. Effort routes the amount of reasoning per subtask within a fixed state, format measures the per-turn price of the state itself, size swaps the model, and serving changes precision and speculation. Because the axes are independent, the per-axis break-evens compose as multiplication in cost. Moving each axis to its own break-even multiplies the savings.
The price of compact is monitorability. Short chains are hard to monitor, and a misleading hint affects the answer even when it is mentioned less often. The operating response is a sampled audit path. Leave explicit CoT on a small fraction of traffic, record the regime and budget used in the audit sample, then contrast the audited quality against the metered tax.
Translated into carbon, reducing the compounding tax is reducing the re-send prefill compute per task. It is the p_in term of the lemma. A dial that cuts energy without touching the accuracy of turns that did not need it, and that runs without any training.
What Remains for the Company, for Society, and for Science
For the company, what remains is the tax the ledger sees. Our token factory prices text tokens, but unattended agent loops re-emit reasoning every turn. The BFCL baseline of the 27B teacher was measured with the thinking state implicitly set. This paper turns the enable_thinking/no_think handles already attached to the repo into measured per-turn verbosity cost and quality-per-dollar break-even. Metis can now set thinking on/off and budget per workload class on H200, and instead of continuing to pay an invisible compounding tax, it watches the tax and trims the budget.
For society, what remains is the dial. Reasoning tokens are a large and fast-growing share of inference spend and energy. Making the explicit-versus-compact frontier reproducible on open weights (the Qwen3 family) with our harness keeps agent AI in the cheap price band. What is left for the operator is a measured dial that cuts cost and carbon without giving up tool-calling and routing reliability.
For science, what remains is the first compounding formalization. BDH-CQ’s latent reasoning efficiency claim is converted into a per-turn cost-quality frontier of agentic serving. The format/density axis is orthogonal to the effort axis (our 07-29 Effort-Routing study) and to the model size axis (distillation). This paper is the first to show how the per-turn verbosity tax compounds in a multi-turn agent loop on H200.
Infographic generated by NotebookLM from the sources.
What Not to Trust
There are no measurements. This is an analytic, positioning paper. Every result is structural, the protocol is designed to fill in the values, and the refutation criteria are the worth of the claims.
The state toggle is template-level. The Qwen3 hybrid thinking toggle only switches between the two ends of explicit and no-CoT; it does not include a trained latent recurrent reasoner, the BDH-CQ lineage. The frontier between the two ends, and what fills the space in between, is an open question that this formalization leaves behind.
The cost model assumes steady state. Amortized per-token prices presume a uniform load, and burst workloads break the p_in/p_out split. A prefix caching policy that amortizes re-sent context changes the effective input price of re-sent tokens. τ₁ and U change with that value. p_out > p_in reflects the decode/prefill price structure of self-hosted serving. If that price relationship inverts, the first-order term of τ₁ changes, but the compounding structure of the lemma stands.
Per-turn quality independence is also an assumption. Error propagation on long chains can only widen the tax. Propagated errors force corrective re-invocations, and re-invocations re-price the context.
The benchmarks cover tool calling and routing. BFCL and the SRA routing bench are those two domains. Other agentic domains such as code and retrieval may have different critical turn counts k and γ, and the break-even must be re-derived per domain. To interpret T_crit, you first have to estimate k.
The paper’s detail page is here: The Reasoning Verbosity Tax: Per-Turn Cost-Quality Frontiers of Explicit vs. Compact Chain-of-Thought in Self-Hosted Agentic LLMs on H200
Every curve in this paper is analytic, not measured. The three figures are conceptual illustrations and schematics of the analytic model. The measured values will be filled in by the 4arm protocol, and refutation criteria R1 through R4 declare the failure conditions.