The Parallelism Dividend: The Parallel-Call Dividend of Unattended Agent Loops, Priced in Two Units, Wall Time and Tokens, Where the Floor Belongs to the Harness and the Ceiling to the Model
This is for Korean cloud and AI engineers who run, or plan to run, multi-step unattended agent loops on a self-hosted GPU (H200 class). The reason to read is that the cost of the choice, whether to leave parallelism inside the loop to the model or have the harness do it instead, is the subject of this post. If you operate a loop where every tool call is billed in wall time and token charges, the answer depends on window length and that model’s parallel calling gap. ThakiCloud’s paper “The Parallelism Dividend: Measuring Wall-Time and Token Cost Savings of Harness-Flattened Parallel Tool-Calling at Fixed Task Accuracy in Unattended Agent Loops on Self-Hosted H200” prices that dependency in closed form, in the two units of wall time and billed tokens at fixed task accuracy. This is an analytical paper. Every number is either a closed-form prediction for a declared nominal assumption or a measured result attributed to a cited reference. It ships with a falsifiable 3-arm protocol with pre-registered falsification criteria.
The Problem: Unattended Loops Serialize Independent Calls, and Model Parallel Calling Is a Separate Skill
If the default agent harness is ReAct-style, it emits one tool call per model turn, blocks on the tool result, and feeds the accumulated results back into context. So every turn of a fan-out stage re-prefills everything that came before it. In a 6-call gather phase, results are billed 6, 5, 4, 3, 2, and 1 times, and the GPU idles once per tool latency inside the window.
Meanwhile, the model’s ability to emit multiple calls in one message is a separate skill from the ability to emit one. The Berkeley Function Calling Leaderboard (BFCL) reports it as a distinct category, and parallel scores sit below simple. The same gap shows up on our 27B NVFP4 checkpoint in the serving environment: simple 90.8% vs parallel 88.0%, a gap g = 2.8pp (declared nominal).
Two parallelism levers have been confused, and that confusion hides the cheaper one. Generation-side parallelism is the model emitting a batch in one message. The LLM-Compiler line, which compiles plans into parallel execution, AsyncLM, which fine-tunes interrupt semantics, and recent agentic serving systems such as AgentSpec, SPORK, and SAGA all belong here. They all change what the model generates or how the cluster schedules, and they treat the model’s parallel calling capability as given. Execution-side parallelism is different. The model generates one call per turn under the same simple prompting as always, and a deterministic harness post-processor finds the certified independent calls, withholds results from context until the window closes, and then runs the window concurrently. The post-processor adds no inference calls, uses no fine-tuning and no predictor, and pays no capability gap between parallel and simple.
This paper is an explicit upgrade of Manifest Order, which priced the fixed prefix. That paper priced the fixed prefix P that is retransmitted every turn. This one prices rolling accumulation, the intermediate results that the serial harness re-prefills in every subsequent turn. The zero-inference post-processor removes it, and the dividend composes with the prefix-cache dividend, peaking in the same regime where cache contention is largest.
Design: Arms Over Certified Fan-Out Windows, Zero Additional Inference
The platform is a single H200 Tensor GPU (141 GB HBM3e). It serves the 27B NVFP4 agentic checkpoint with vLLM-style paged KV and automatic prefix caching (exact-match prefix semantics, LRU eviction), and K concurrent loops share the engine. The nominal regime is K = 8.
A fan-out window is the maximal block of n consecutive turns that structures such as read/gather phases and disjoint writes produce. Inside the window, each call’s arguments must depend only on the prefix, the task input, and the earlier calls, and never on earlier results. The window closes at an absorb turn, where the whole window, all calls and all results, enters context at once. This is identical across all arms.
The deterministic harness post-processor certifies a window under three conditions. D1 requires every tool in the window to be read-only or idempotent, or to have pairwise disjoint write targets. D2 checks, at the schema level, that no argument references a window result, i.e. that the argument template has no result placeholder. D3 requires a phase signal marking the window as fan-out, either a skill-manifest phase annotation or a slot-closure signal. A false fill is a genuinely violated certification, a dependent edge inside the window. The schema-sound detector has a structurally zero false-fill probability, and the phase-heuristic detector has a declared nominal of 0.03.
There are four arms for comparison. S (serial baseline) emits one call per turn and blocks on each result. F (harness-flattened) uses the same serial prompting and withholds results from context only inside certified windows, until the window closes. The n calls run concurrently, and the absorb turn receives the full window as-is. F has no additional inference calls. It simply performs the exact same n generation calls as S and the same absorb turn. P (native-parallel) is the BFCL parallel regime where the model emits the window’s n calls in one message, and it pays the capability gap g. P+V is P with a free deterministic validator gating the window and resampling buying back g.
Structure of each arm over a fan-out window of n certified independent calls. S blocks on each result and re-prefills the accumulated results. F withholds results and runs the certified window concurrently at zero additional inference. The P family relies on the model emitting the batch itself and pays the capability gap g. When the window closes, the absorb turn takes in the full window identically across all arms. (Conceptual diagram, not a measurement.)
The nominal regime parameters are as follows. The fixed prefix P is 4096 tok, the average call tokens per turn c is 64, the average result tokens r is 192, the serial stride s = c + r is 256, and the absorb turn output a is 128. Marginal prefill latency is 1.5 ms/tok, decode latency is 5.0 ms/tok, per-call serving overhead is 0.1 s, and average tool latency L is 0.5 s. The steady-state prefix hit fraction h is 0.80, and the rolling-stride hit fraction h_R, with its shorter retention distance, is declared at 0.50. The harness accuracy budget ε is 1.0pp, and the expected number of certified windows per task w is 2.
Results: The Floor Is Capability-Invariant, the Ceiling Is Gap-Priced
The wall-time dividend of harness flattening is closed form. For each certified window of n ≥ 2 calls, the serial arm cost is T_S(n) = n T_0 + t_pf (1-h_R) s n(n-1)/2 + nL + T_abs(n), the harness-flattened arm cost is T_F(n) = T_S(n) - Δ_T^F, and the dividend is
Δ_T^F = (n-1) L + t_pf (1-h_R) r n(n-1)/2 > 0
, strictly positive in every certified window. The two components are the (n-1) tool latencies that overlap when the window runs concurrently, and the re-prefill of the withheld results. Because the number of generated tokens does not change, no accuracy or billing terms enter, and the dividend is capability-invariant. It depends only on serving and tool parameters, and never on the model’s parallel calling gap g.
At the nominal values, the F dividend is 4.66 s at n = 6, 24.7% of T_S = 18.89 s, and 10.98 s at n = 10, 32.3%. The native arm P dominates in raw wall time. It saves 12.02 s (63.6%) at n = 6 and 25.11 s (73.8%) at n = 10, because it removes n-1 full turns and has no re-prefill at all inside the window. But that dominance comes at the price of paying the capability gap g. Pricing it is the next section.
The token dividend is real but small at the nominal cache temperature. The billed window token difference between the serial arm and the F arm is D_tok^F = r n(n-1)/2, exactly the re-prefill of the withheld intermediate results. It is the rolling-stride counterpart of Manifest Order’s fixed prefix retransmission. At the nominal n = 6 window it is 8.3% of the window bill. The regime structure is clean. The dividend peaks when the rolling stride is cache-cold (h_R → 0), which coincides exactly with the high-contention regime (large K, short retention) where Manifest Order’s fixed-prefix dividend also peaks. The two dividends are priced by the same retention parameters. Flattening is also cache-friendly. By removing (n-1) stride re-prefills from KV inflow per window, it increases retention, which creates a positive coupling between the two levers.
At the declared nominal regime, the fraction of the serial arm’s billed window tokens saved by the F arm (withholding intermediate results) and the P arm (withholding the fixed prefix and strides). The F dividend grows quadratically with window length but stays modest, while the P dividend is much larger at the cost of paying the capability gap g. (Analytical model, not a measurement.)
The two arms fail in different ways, and that failure sets the price. P fails in generation. The parallel prompting regime changes the model’s structured-output behavior, incurring a prompt-side cost from the Format Tax and losing g points that only resampling or a stronger model can recover. F fails in certification. Generation outside the window is bit-identical to S, and only a false fill changes behavior. The schema-sound detector makes that event null. That is why the F dividend is capability-invariant and the P dividend is not. F keeps the fixed accuracy budget ε = 1.0pp. The schema-sound detector sets equality at φ = 0, and the phase heuristic sits inside the budget at nominal at 0.03 × 2 × 0.9 = 0.54pp. A false fill is the flattening of a dependent edge, a cascade seed as ParaRecover documented, and it is detected by the schema-level validator. The residual cost is repair, not silent error.
The Frontier: At the Declared Gap, the Wall-Time Strategy Flips at Window n ≈ 8
Pricing the native arm’s gap g in both units with validation-gated resampling yields two crossover gaps. The redo price is one window redraw per point, matching the Validator’s Dividend pricing. The wall-time redo is A_T(n) = P̂ + t_dec n c, the token redo is A_B(n) = P(1-h) + n c. P+V is net-better than F if and only if the gap is below the crossover, and the crossover gaps are g_T^(n) in wall time and g_B^(n) in tokens.
In the nominal regime, g_T^* rises from 0.74pp (n = 2) through 2.34pp (n = 6) to 2.81pp (n = 8), and g_B^* rises from 4.39pp (n = 2) to 17.8pp (n = 6). At the declared gap g = 2.8pp, the only policies where P+V is net-better in wall time are windows n ≥ 8, and harness flattening dominates for n ≤ 7. The crossover window is n* ≈ 8. Solving T_P+V(n) = T_F(n) gives n* = 7.96. In the token bill, P+V is net-better for all n ≥ 2. g_B^* is minimized at n = 2, at 4.39pp, above the declared gap.
Wall time per certified window at the declared nominal regime, normalized to the common n = 1 value. The serial arm S grows fastest and the native arm P slowest, and the crossover where the P+V arm, which buys the gap with validation, drops below the F arm sits around n = 8. (Analytical model, not a measurement.)
Here the structure is the result. The parallelism dividend has a capability-invariant floor (F) and a gap-priced ceiling (P+V), and in wall time what separates the two is g_T^*(n). The floor belongs to the harness. The ceiling, because it requires paying the model’s parallel gap in accuracy points, is reachable only when the gap is below the crossover. Operators with strong parallel callers, large models or fine-tuned checkpoints with small g, live above the ceiling and use native parallel plus validation at every window length. Operators with the declared 27B class gap live in the floor regime at windows up to n ≈ 8, and that stretch is exactly the small gather phases that dominate loop count. There, flattening is right. At n ≈ 8 and above, P’s prefix-turn savings beat the validation cost. The flip is a choice of window length, not a choice of model. A single harness uses both arms. It flattens short certified windows and prompts natively with the validator in the path for long windows. The operational rule is “flatten what the model cannot afford to emit in parallel”.
Implications: What Remains for the Company, Society, and Science
What remains for ThakiCloud’s agent platform is the zero-extra-inference post-processor. It reduces loop wall time and redundant prefix tokens on the H200 and provides cost/latency-accuracy frontiers for choosing between model-side native parallelism and harness-side call flattening. At fixed accuracy, busy single-H200 wall time converts to energy at the declared 750 W. The F dividend at n = 6 is 4.66 × 750 ≈ 3.5 kJ/window, or ≈ 1.9 Wh per task at the declared w = 2. Small per task, material per fleet-hour, super-linear in the window.
For society, unattended work automation becomes faster and cheaper per task, because it removes the serial tool-call overhead with deterministic post-processing and no additional LLM inference. The capability-invariant floor is exactly that: for any self-hosted agent model, regardless of its BFCL parallel score, it collects the (n-1)L tool-latency overlap and the withheld-result re-prefill per window at fixed task accuracy. Lower energy costs for small-team automation on self-hosted GPUs come from this floor.
For science, it is the first separation that prices model-side parallel function-calling capability and harness-side parallelism enforcement at fixed accuracy. The two crossover gaps g_T^(n) and g_B^(n) tell the operator when native-parallel-plus-validation beats harness flattening: nominal n ≥ 8 in wall time, and all n at the declared gap in tokens. The protocol extends BFCL’s parallel category from capability measurement to serving economics, turning the leaderboard’s single gap number into a frontier and keeping the question of native versus flatten out of folklore.
The falsifiable 3-arm protocol changes no formula and converts every closed form into a measurement. Model weights, harness, and prefix are held identical across arms S, F, P, and P+V. F runs the schema-sound detector, and the phase heuristic is attached as a reporting control. The workload is 48 deterministic seeded loop tasks over a typed read/write toolset (16 fan-out, 16 chain, 16 mixed), with gold success labels built by deterministic construction. To anchor g, 32 seeded BFCL-v3 simple and 32 parallel instances at temperature 0 are added. Chain tasks certify no windows at all and form the null arm, where no arm beats S above noise. The pre-registered falsification criteria are four. R1: the measured Δ_T^F on n ≥ 6 windows must be at least half of the closed form at the nominal point. A violation means the homogeneous-L assumption or the overlap efficiency is wrong, and the E[max_i L_i] correction is reported. R2: the measured D_tok^F must be non-negative. A negative value means position sensitivity in the withheld-result context broke the accounting or induced re-decoding. R3: the measured a_F - a_S must not exceed -1.0pp on gold tasks. Exceeding it counts as a violation of detector soundness on the real toolset, and φ̂ and ê_v are reported to recompute the accuracy theorem. R4: the measured BFCL simple versus parallel gap at temperature 0 must fall within g ± 1.5, i.e. [1.3, 4.3]pp. If it goes outside, the nominal gap is unstable. The formula is not changed, only the parameters are recomputed.
Limits: Where the Dividend Is Zero, and What This Paper Does Not Adjudicate
This paper is explicit that it is analytical and positional. It provides no new measurements. Every figure is a closed-form prediction for a declared nominal assumption or a measured result attributed to a cited reference. The protocol above exists to fill the frontier with measurements without changing the formulas.
In write-heavy loops where every window has a dependent edge, the dividend is 0. The detector rejects, and F degrades to S, leaving only log overhead. A small window like n = 2 earns only (n-1)L = 0.5 s at nominal. Real, but below the validation cost of the P+V gap. With non-homogeneous latency, L is replaced by E[max_i L_i] and the concurrent window cost is L + σ_L κ_n (κ_n → √(2 ln n)). The dividend is more sensitive to the slowest tool than to the mean, and with σ_L ≤ 0.3L declared it stays positive for every window with n ≥ 2 at nominal.
Withholding results changes the input the model sees inside the window. Context is load-bearing, and the plan signal dropping 4.1× in a single step is why withholding is safe only inside certified windows and why the absorb turn reattaches the full window verbatim. Information is not lost, only delayed. Two defenses bound the risk. Since the certified window is the only place where the input differs, a wrong certification is priced as a false fill in R3, not as silent drift.
The declared gap g = 2.8pp is a nominal input for our serving environment, and R4 pre-registers the range in which the nominal gap is stable. Energy conversion uses the declared 750 W average busy draw, and net-cost conversion is concurrency-aware. Dividend sizes depend on the declared cache hit fractions h = 0.80 and h_R = 0.50, and the high-contention, short-retention regime where both dividends peak is also the regime where the declared hit fractions are least certain.
The waiting in unattended agent loops has two owners, with different bills. The model owns the capability gap g, and native parallelism pays it in every window regardless of the harness. The harness owns the serialization, producing (n-1) tool latencies and r n(n-1)/2 re-prefilled results per certified window. The deterministic zero-inference flattener removes it without depending on g. The parallelism dividend is a floor every self-hosted checkpoint can collect and a ceiling only strong parallel callers reach. Flatten short windows, prompt natively with the validator in the path for long windows, and the retention parameter decides how much of each dividend the cache gives back.
The paper and its accompanying materials are available on Hugging Face.
https://huggingface.co/datasets/thaki-AI/daily-paper-2026-10-12-parallelism-dividend-tool-calling-harness