The Concurrency Tax: The Tax You Pay When Function-Calling Accuracy Drops as Concurrency Rises, and the Closed Form of the Safe Concurrency Ceiling
If you are a cloud/AI engineer in Korea who serves agentic tool-calling workloads on a self-hosted GPU (H200 class), or who owns the cost bill for that serving, the question this post answers is an operational one. Raising GPU utilization lowers the per-token cost. But there are workloads whose accuracy falls with it, and function calling, which scores structure against a contract, is the prime candidate. ThakiCloud’s paper “The Concurrency Tax: Measuring the Accuracy-Drift-versus-Utilization Cost-Quality Frontier of Agentic Tool-Calling in Self-Hosted H200 Serving” formalizes the size and direction of the concurrency tax, the accuracy score a function-calling workload loses as serving concurrency is raised from light load to saturation under a fixed model, fixed precision, temperature-zero harness. And it gives a closed form for the safe concurrency ceiling, the point where that tax exceeds the per-token savings. This is an analytical paper. Every figure is a closed-form prediction for the stated assumptions, and the falsifiable protocol that fills those predictions with measurement is registered alongside.
Problem: Every Frontier in the Series Was Measured at Light Load
A self-hosted inference factory is sized by utilization. An operator running agentic tool-call workloads on a fixed accelerator must decide how many requests to keep in flight simultaneously, and each in-flight request buys throughput and cuts the dollars and kWh per token. Continue until the serving stack saturates. Absent evidence to the contrary, the natural operating point is maximum concurrency, maximum utilization, minimum cost per token.
Our cost-quality frontier series has priced that decision across four levers, and every lever was measured at light load. The weight-precision lever priced the structural tool-call accuracy tax tau_q and the cliff ratio kappa_q by anchoring to the series’ 89.6% NVFP4 reference row. The (model, precision) pair upgrades the FrugalGPT cascade into a two-dimensional frontier over effective passing throughput E = tQ/ℓ and prices it under the judge-free gate g = 0.85. The sampling lever priced one accuracy point as a sample cost and gave a silent floor that no retry budget can beat, along with the gate-versus-majority crossover. The context lever priced the bill that compounds at theta(T^2). Across all of these levers, serving concurrency was held at light load and was an implicit constant.
The serving literature reads load as a throughput or latency variable. Continuous batching and paged KV attention make throughput a function of in-flight concurrency, and that side has produced goodput, SLO-aware latency budgets, and cost-quality-latency Pareto atlases. But all of that side conditions throughput on per-request timing targets. No paper conditions on end-task correctness, and no agentic evaluation reads load as an accuracy variable. The gap this paper fills is exactly that point.
This paper is an explicit upgrade of the Tool-Call Cliff. In the same sense in which the Precision Ladder upgraded the FrugalGPT cascade, it adds a utilization axis. Under a fixed model, fixed weight precision, temperature-zero harness, it asks how much function-calling accuracy drifts as serving concurrency moves from light load to saturation, and where the safe concurrency ceiling lies, the point where the tax exceeds the per-token savings. This upgrade completes the cliff series. The precision cliff and the utilization tax compose into a double cliff, where load-induced drift hits low-bit checkpoints earlier and farther.
Where does the belief that the load axis moves accuracy come from? The serving nondeterminism literature answers. Even with the prompt, the weights, and sampling all fixed, the serving context changes the output. Batch shape moves up to 56.7 percentage points of probability mass in bf16. Server-level prompt-cache settings shift run-to-run divergence, which rises from 36.2% to 75.0% of agent episodes as precision drops from 16-bit to 4-bit. Cache-disabled runs are bit-identical. This paper prices exactly these effects. Nondeterminism is not a reproducibility hazard; it is the substrate of the accuracy tax.
Design: The Concurrency Tax, the Noise Floor, and the Load-Dependent Cost Model
We fix the platform and the harness so that concurrency is the only variable swept. Serving runs on a vLLM OpenAI-compatible endpoint behind a single NVIDIA H200, with –max-num-seqs 256, –max-model-len 8192, –gpu-memory-utilization 0.90, and prefix caching off in the base configuration. Cache-on is a separate attribution arm. Sampling is greedy, temperature 0, top-p 1. The model m and the weight precision q are fixed throughout. The concurrency level C counts in-flight requests; the saturation capacity is C_max = 256, utilization is u = C/C_max, and the light-load reference is u_min = 1/C_max. The workload is a fixed 60-item BFCL-v3 subset with 30 single-call and 30 parallel-call items, seeded once and held fixed. Single-turn function-calling completions are scored by judge-free exact match, so quality is a function of the serving state alone.
The central definition is the concurrency tax tau_u = Q_ref - Q(u), the accuracy score the workload loses as it moves from light load to utilization u. The analogy to the series is direct. The precision tax tau_q in the cliff paper prices accuracy lost along the weight-precision axis at fixed load, and tau_u prices accuracy lost along the load axis at fixed precision. Composing the two taxes on the orthogonal axes of the same frontier is the double cliff.
A drift claim must clear the noise floor. Let d(u) be the item-flip distance, the number of items whose pass/fail status flipped relative to the first light-load run. A same-session C=1 repeat of the reference run yields the null flip distance F. The drift claim at level u is significant only when d(u) > max(1, F). Because the null is a physical repetition of the reference level within the same session, F captures every load-independent cause of drift in a single number. The item-majority granularity of a 60-item set is 1/60, about 1.67 percentage points.
The cost model prices that axis. Let r be the declared self-hosted rate, and t(u) the aggregate output throughput at utilization u (tokens per second, non-decreasing and saturating). If ℓ is the roughly constant number of output tokens per tool call on the item set, then c(u) = r/t(u), and the cost per passed call is beta(u) = c(u)ℓ/Q(u). The effective passing throughput E(u) = t(u)Q(u)/ℓ is the number of calls passed per wall time, and it makes the Precision Ladder’s metric load-dependent. Since ℓ and r are constants, all load dependence hangs on t(u) and Q(u), and beta(u) = r/E(u) is the ratio of two falling curves. The shape of this ratio is what the next section controls.
The forms of the two regimes are simple. Aggregate throughput follows the saturating (hill) law t(u) = t_sat*u/(u + K_u): continuous batching lifts in-flight work almost linearly, and KV and scheduling pressure bend the curvature until the growth rate turns. Accuracy drift is absent up to the KV saturation knee u_kv and grows linearly past it: tau(u) = s(u - u_kv). The knee is a capacity-planning quantity read from engine metrics; it is not the throughput knee. The protocol’s first prediction registers this distinction. s is the drift price per unit utilization, a workload-class parameter.
Up to the KV saturation knee, tool-call accuracy drift is flat within the noise floor; past it, it grows linearly faster than the free-form control, and the concurrency cliff ratio exceeds 1. (Analytic model, not measurement.)
Finally, the concurrency cliff ratio kappa_u = tau_u^(tc) / max(tau_u^(ff), eps_0) asks whether the load axis taxes structural tool calls more heavily than the free-form prose that carries the same content. It is the load analogue of the cliff paper’s kappa_q. The expected regime is kappa_u > 1, because the mechanism is structural amplification: a parallel-call item has more required structural decoding steps. Each call boundary requires one more function name, one more argument boundary, and one more list close, so the same load drift flips more steps.
Results: The Utilization Illusion and the Closed Form That Governs Maximum Utilization
The main theorem gives the shape of E(u). On (0, u_kv], E is strictly increasing, because throughput is rising while accuracy is untouched. Past the knee, E’(u) has at most one zero, and that zero is a strict global maximum when it exists. Let s_crit = K_uQ_ref / (1 + K_u(2 - u_kv)). Then E attains its maximum at an interior point u* < 1 in (0,1] only when s > s_crit, and in that case E(u) > E(1). If s <= s_crit, the maximizer is u = 1. Maximum utilization is not always optimal. When the drift slope crosses the closed-form threshold, running at saturation is strictly dominated.
With a steep drift slope, effective passing throughput reaches an interior maximum below full load; with a gentle slope, the maximum stays at saturation. (Analytic model, not measurement.)
The corollary is the economics that gives this paper its name. Operating at u = 1 instead of u* costs H = E(u)/E(1) > 1 times as much per passed call, because beta(u) = r/E(u). H is a closed-form function of (K_u, u_kv, s, Q_ref), and the throughput scale t_sat, the call length ℓ, and the declared rate r all cancel. The safe concurrency ceiling is C = floor(u* C_max), refined downward by the gate floor. The paper’s slogan, “headroom is priced, not assumed”, is this corollary: there is a price on the margin that protects accuracy. The same ratio also bounds energy cost, because request-level energy attribution depends on the batch context.
The sizing rule reads the frontier; it does not assume it. For each workload class, sweep the declared levels and read the knee from engine metrics. Estimate Q_ref, s, and K_u from the frontier curve. If s > s_crit, operate at C <= min(C*, C_safe) and carry the headroom at its price H; if s <= s_crit, operate at the top of the sweep. Since s is a property of the mix, re-audit it whenever the workload class changes. This rule turns “run at maximum GPU utilization” into a cost-per-accuracy decision. Both terms are billed in dollars per passed call.
The sampling lever deforms consistently under load as well. The validator’s silent floor drops by the undetectable share of the load drift. The portion of load-induced wrong calls that survives schema/AST validation, phi_s(u)tau_u, is the part no retry budget can recover. The post-gate quality ceiling falls by at least phi_s(u)tau_u/(1 - p_d(0)) below the light-load ceiling. The dividend shrinks the busier the factory. The marginal wall-clock price of recovering one quality point, Pi(u) = c_s(u) / (delta_eff(u) * q_bar), is non-decreasing past the knee, because the resampling cost c_s(u) rises with queueing while the effective detectable fraction delta_eff(u) = delta(1 - phi_s(u)) falls. The gate-versus-majority crossover moves upward: in the regime where q(u) > 3/4, the crossover detectable fraction p_d*(q(u)) is non-decreasing in load, and the higher the utilization, the higher the detectable fraction at which the gate beats k-sample consensus. The two levers pay the same queueing cost per draw. Only the gate’s dividend is eroded by the silent share, and the consensus’s quality gain is not, so consensus voting gets relatively cheaper the busier the factory.
Double Cliff: Why Low-Bit Checkpoints Drop Earlier and Farther
The precision axis and the load axis meet in a structural mechanism. We carry over the low-margin structural-step assumption from the cliff paper. The required structural decoding steps (function name, argument boundary, list close) have a decision margin m_struct, and below it they flip under logit perturbation. The flip probability per step is p_flip = Phi(-m_struct/sigma_eff), with sigma_eff = sqrt(sigma_q^2 + eps_u^2). sigma_q is the weight-quantization noise, the precision axis, and eps_u is the load-dependent logit perturbation. Batch shape and batch-composition numerics, prefix-cache divergence amplified by quantization, and silent shape-dependent GEMM effects all hang on it. An item with r_s required structural steps fails under load with probability tau_u(q) = 1 - (1 - p_flip)^r_s. The result is monotone: tau_u(q) is non-decreasing in sigma_q at every u, the drift knee is non-increasing in sigma_q, and the endpoint tax is non-decreasing. That is, a low-bit checkpoint drifts earlier and farther than a 16-bit one.
On the same item set, the drift knee of a low-bit checkpoint is pulled earlier and its endpoint tax is larger relative to the 16-bit baseline: the double cliff where low-bit checkpoints drop earlier and farther. (Analytic model, not measurement.)
The routing implication upgrades the series gate into a load-aware three-dimensional condition. The per-category routing condition, which routes to a task category only when the (model, precision) pair passes the judge-free gate Q(m,q) >= g, g = 0.85, at light-load quality, must instead be Q(m,q,u) >= g at the factory’s operating utilization. A pair that passes the gate at light load can fail at the operating point. The routing table is indexed by the three axes of (model, precision), sampling, and load. The load-free row of the cliff paper, the 89.6% NVFP4 reference on the 800-case BFCL-style suite, is only the u = u_min slice of the surface, not a point estimate of factory behavior.
Where does the drift come from? Four channels carry load into accuracy, and each gets an attribution arm in the protocol. Numerics and batch composition: continuous batching changes the shape the kernel sees, and floating-point non-associativity makes results shape-dependent. The batch-shape effect is quantified up to 56.7 percentage points in bf16. KV memory and preemption: past the knee, the KV cache working set no longer fits, and the engine preempts, evicts, and recomputes. A recompute pass sees a different batch composition than the original. The prefix-cache channel is precision-conditional: cache-on/off divergence rises from 16-bit to 4-bit, and cache-disabled runs are bit-identical. Wall time and harness: even when every token is correct, TTFT/TPOT inflation eats into the per-episode wall budget, and timeouts and retries turn slow requests into wrong trajectories. Co-tenants and interference: the host is not isolated, and co-located CPU workloads amplify the long tail. One measurement has unprotected co-location cutting throughput by 78.8% and inflating TTFT by 429.5%, and gateway-level cross-tenant effects complete the channel. The decomposition tau_u = sum_c tau_c(u) plus interaction terms is the account of the mechanisms; it does not claim the measured share of each channel. So the attribution arms turn channels on and off. They do not assume that the sum separates.
Meaning: What Remains for the Company, Society, and Science
For ThakiCloud’s H200 token factory, the remaining asset is a measured accuracy-tax curve per workload class (agentic tool-calling versus free-form). “Run at max GPU utilization” becomes a cost-per-accuracy sizing decision, and the safe concurrency ceiling keeps the BFCL-validated 27B NVFP4 router baseline protected from silent load-induced drift. The series’ three levers now index a single surface: the precision ladder picks the per-category (model, precision) under the gate, the validator’s dividend picks the sampling policy per (category, load), and this paper picks the operating utilization per workload class. The join condition is the three-dimensional gate Q(m,q,u) >= g, and the routing table is the (model, precision, load) triple.
For society, the economics of self-hosted inference become honest. Cost-per-token comparisons measured at light load structurally understate the quality erosion that utilization buys. A published load-aware methodology cuts both the over-provisioned headroom and the energy spent keeping GPUs idle to protect accuracy. Now there is a price on the margin that protects accuracy.
For science, this is the first result of a controlled characterization that treats serving load, continuous-batching and paged-KV numerics, preemption under KV pressure, and wall-time erosion as hidden variables in agentic LLM evaluation. Adding the utilization axis on top of the precision (tool-call cliff) and model-tier/quantization (precision ladder) frontiers completes the cliff series, with the utilization illusion (closed-form interior maximum) and the double cliff (precision interaction) as the main results.
Limitations: A Paper That Registered the Measurement, Not the Measurement
This is an analytical and a positional paper. The formalization, the closed-form conditions, the channel-decomposition account of where load-induced drift comes from, and the pre-registered falsifiable protocol are all it has. There is no new measurement, and every figure is a closed-form prediction for the stated assumptions.
The scope is narrow by design: a single accelerator (H200), a single model family (Qwen3 series checkpoints), a single-turn 60-item BFCL-v3 subset. Multi-turn trajectories and the token-wise replay discipline are outside the base protocol. MoE routing, speculative decoding, and distributed scheduling are future axes that enter the theory only as t(u) and eps_u. The declared self-hosted rate, input $1.50 and output $0.60 per MTok, is a series constant, not a market quote. H is a rate-invariant ratio, and only the level of beta(u) rescales linearly.
The pre-registered protocol takes on the role of filling the frontier with measurement: four predictions on the declared H200 platform, each with a 1:1 refutation criterion. P1 claims that Q(u) is flat within the noise floor up to the KV knee read from engine metrics and decreases past it; it is falsified if the flat region ends below u_kv/2 or if Q decreases monotonically from C=4 without a flat region. P2 claims that past the knee, drift in the parallel category exceeds drift in the simple category (kappa_u > 1); it is falsified if, at any level above the noise floor, simple drift is at least as large as parallel drift. P3 claims that, on the same item set, the drift knee of a low-bit checkpoint is pulled earlier and its endpoint tax is larger than 16-bit’s; the primary comparison is the same-size precision pair, 27B bf16 versus 27B NVFP4, so that precision is not confounded with model size, and the base sweep checkpoint is Qwen3-8B bf16. P4 claims that validation-gated resampling recovers at least delta_eff(u) = delta(1 - phi_s(u)) of the drift induced at load u; the silent share phi_s is a measured value, not an assumption. The concurrency sweep runs at levels C = {1, 2, 4, 8, 16, 32, 64, 128, 256}, that is, from 1 to 256, on the fixed 60-item function-calling subset, with each level repeated until in-flight requests stay at C or above.
The frame itself has a registered falsifier. If drift is flat within the noise floor across the whole sweep, the utilization axis is a null result on this platform and workload class: tau_u is 0, and the interior maximum is an artifact of the assumed tax shape. If the throughput law departs from the hill form, the form of s_crit changes, but beta(u) = r/E(u) and the headroom-price structure survive on the shape of E alone. If phi_s is negligible, the gate proposition degenerates to the load-free dividend and the crossover does not move. Every drift claim on the filled frontier must clear the run-to-run null d(u) > max(1, F), and the flat-region test (P1) is the same discipline applied to the load axis.
The safe concurrency ceiling that the protocol reads from the frontier is the factory’s operating number. We run slightly below a point just past the knee, and pay for headroom in the same currency as everything else.
The three figures in this post are all closed-form predictions of the analytic model, not measurements.
The paper and its accompanying materials are available on Hugging Face.
https://huggingface.co/datasets/thaki-AI/daily-paper-2026-10-13-concurrency-tax-agentic-tool-calling