We Left the Feature On and the Speedup Went From 1.4x to 0.8x

As riders fill the bus, the empty seats disappear. Those seats are exactly what speculative decoding was living on.

The number to check before you turn on speculative decoding is not the speedup in a paper. It is how many concurrent users your server actually carries. We kept the same drafter switched on and raised concurrency from one user to eight, and the speedup fell from 1.4x to 0.8x. This is for anyone running inference in production who keeps asking why their own numbers never match the published ones.

In plain terms

Picture a shuttle bus. For a large language model to write a single character, it has to read the entire model out of memory. That is a bus driving from the depot to the stop to pick up one passenger. The round trip costs roughly the same whether you carry one rider or ten, so anyone you add to a trip you were making anyway rides close to free. Speculative decoding is the technique that uses those empty seats. An attentive guide points at the next few likely passengers and loads them, and the driver checks everyone at once on arrival. Whoever was guessed right saves a whole round trip. Whoever was guessed wrong just gets off.

The catch is that the empty seats are not always there. When riders crowd in, the bus is already full and there is nowhere to put anyone extra. Swap in a smaller bus and each trip gets cheaper, but the spare seats shrink with it. This post is a record of when those seats disappear, measured on our own hardware.

The speedup is attached to conditions, not to the feature

First, what we actually turned on. The target is our own 27B checkpoint, and the drafter is DFlash2 proposing seven tokens at a time. The engine is vLLM 0.28.0, and sampling matches what the live service uses. We paired the drafter-on and drafter-off arms inside the same run, and we only quote a speedup within that pair.

Measured that way, the speedup is not one number. It is a curve, and the curve crosses below 1.0. Where it crosses is the whole point of this post.

Fill the bus and there are no seats left to take

Concurrency is the axis that collapses first. Running free-form generation on an INT4 checkpoint under an edge memory budget, one concurrent user gives 1.4x. Two gives 1.2x, four gives 1.1x, and eight gives 0.8x. Leave the feature on and eight simultaneous users make the server slower than it would be with the feature off. Push our production checkpoint to a larger batch and the same collapse gets more dramatic. At 64 concurrent users the baseline reaches 2,306 tokens per second, and turning the drafter on drops it to 595. That is 0.3x.

Drag the slider to see where each condition crosses 1.0. The three series run different serving configurations, so we do not compare their absolute values against each other.

Here is the same drafter attached to three conditions, all on free-form generation. Columns are concurrent users, rows are the target checkpoint.

Condition	1	2	4	8	16	64
NVFP4-GPTQ, 131k, batch 256	1.34x	not measured	1.22x	not measured	1.05x	0.26x
KV fp8, 256k, edge budget	1.45x	1.34x	1.28x	1.27x	not measured	not measured
INT4, 256k, edge budget	1.43x	1.23x	1.06x	0.82x	not measured	not measured

Every row falls from left to right. They fall at different rates and they cross 1.0 at different places. Knowing that the feature is switched on tells you nothing about which cell you are sitting in.

In plain speech: when nobody is waiting, the guide helps. When everybody is waiting, the guide is taking up a seat.

The mechanism is continuous batching. Engines like vLLM do not process requests one after another. On every step they gather whatever requests are currently alive and compute them as one batch. With several users in flight, that batch alone already saturates the compute units. The bus is full. What speculative decoding does at that point is not create seats. It shoves unconfirmed candidate tokens onto a full bus, and everything rejected is compute thrown away. What was free when the bus was empty is pure waste when it is packed.

So what operations needs is not an on/off switch but a threshold concurrency. In our case that threshold moved with the checkpoint. With the same drafter, the arm running an fp8 KV cache held 1.3x even at eight users, while the INT4 arm was already underwater at the same point.

Grading every branch in one pass

To see why the acceptance rate matters so much, you need to know how verification works. The drafter does not pick a single next token. It proposes several branches at once. Those branches get flattened into one long sequence, and an attention mask hides everything from each candidate except its own ancestors. Without that mask, candidates on different branches would read each other and contaminate the grading. With it, the target model grades every branch in a single forward pass, simultaneously but independently. When grading finishes, the longest surviving path is kept and the rest are discarded.

Back on the bus, the guide is not pointing at one passenger. The guide is building branches: this person, then that one, or otherwise that other one. The driver inspects all the branches in a single check. So adding branches barely changes the cost of checking, and what you actually save is the length of the path that survives. That length is called the acceptance length.

Branches are flattened into one line, masked so each candidate sees only its ancestors, then graded in a single pass. Only the longest surviving path is kept.

Acceptance length governs the speedup. In our measurements, copy-style work that reproduces the prompt had an acceptance length of 3.1. Free-form writing came in at 2.2. Under identical conditions, copy-style held 1.1x even at eight concurrent users while free-form sank to 0.8x. The only difference is how often the drafter guesses right.

Put your own acceptance rate in

Acceptance length follows from the acceptance rate by arithmetic. Call the chance that a single token is accepted alpha and the number of tokens proposed per round gamma, and the expected acceptance length is a geometric sum. Put your own acceptance rate into the calculator below and you can watch why a small drop in the rate collapses the speedup so quickly. We have marked where our own free-form value of 2.2 falls on the curve.

The curve is computed from the formula. Only the two dashed lines are measured. You can see why free-form work sits on the flat part of the curve at 2.2.

Quantization shrinks the gain without removing it

The second axis is the checkpoint, and this is where a common misreading starts. We suspected at first that quantization was breaking speculative decoding. Changing only the checkpoint under otherwise identical conditions showed otherwise.

On the original bf16 model the baseline was 88.7 tokens per second, and turning on the drafter gave 266.4, a 3.0x gain. On our 4-bit checkpoint the baseline rises to 129.0, which is what you would expect when there are fewer weights to read. Adding the same drafter there gives 286.9, a 2.2x gain. The multiplier fell from 3.0x to 2.2x, and yet the highest absolute throughput belongs to the arm running quantization and the drafter together.

On the bus: quantization swapped in a smaller, faster vehicle. The round trip got cheaper, so the cost per rider fell, and the spare seats shrank along with it. Those spare seats were what speculative decoding had been eating. The two optimizations attack the same bottleneck, so they partly overlap. Running both is still the fastest option.

Quantization raises the baseline by leaving fewer weights to read, which shrinks the idle time speculative decoding was living on. Both optimizations attack the same bottleneck.

This distinction matters because dropping the drafter on the grounds that its multiplier shrank also drops your absolute speed. The number to judge on is final throughput, not the multiplier.

How you pick the guide also flips the sign

How the drafter is built is another axis. One approach searches the prompt for a plausible continuation and proposes that. Another predicts from the model’s own hidden state after being trained to do so. The first is a guide reading names off a board at the stop. The second is a guide who knows the regulars by face.

Reading the board only works when the answer is already somewhere in the prompt. We measured that approach on four of our quantized checkpoints and it was slower than baseline on free-form generation in every one. On the original bf16 model the same approach produced a gain. Same workload, and the sign of the conclusion flipped with the checkpoint. The trained drafter absorbed the shrinking headroom. The board-reading one did not.

Sampling adds one more layer. Published speedup tables are usually measured at temperature zero. Our service runs at temperature 1.0 because the model config says so. Holding the model and the configuration fixed and changing only the temperature, the board-reading approach fell from a range of 1.3x to 4.2x on math problems down to a range of 0.3x to 1.0x. The reason is simple. The guide points at the most plausible passenger, and at a higher temperature the bus takes a different one. The guess is rejected and only the checking cost remains.

The price is seats, not latency

The last axis is the memory budget. A drafter is a model too, and it keeps its own KV cache. Under a simulated edge budget, the KV cache pool shrank by roughly 36 percent. That is directly fewer conversations you can hold. Converted into sessions at a 256k context, what fit as 6.7 concurrent sessions becomes 4.3.

In plain speech, the guide also occupies a seat. A service handling many short chats will never notice. A service working through long documents may feel this loss before it ever feels the speed gain.

What to change

Start by measuring your own workload mix. Of the tokens we generated on real traffic, 93.5 percent were free-form, and the share that copies straight out of the prompt sat around 0.1. Public benchmarks lean heavily on copy-like tasks, so they hand you a more generous multiplier than you will see. There is a good chance the benchmark column is not your column.

Next, draw the speedup curve on your own hardware. What you need is not one point but the place where the curve crosses 1.0. Measuring it is not hard. Pair a drafter-on arm and a drafter-off arm inside the same run, raise concurrency through 1, 2, 4, 8, and 16, and record tokens per second at each point. Use the prompts your service actually receives. Public benchmark prompts drift toward copy-style work and will draw the curve higher than reality. And do not let engine versions or quantization methods differ between the arms. A comparison mixed like that measures the difference between two servers, not the effect of the drafter.

Look at your serving configuration before you turn anything on. We have already found that the difference between compilation off and compilation on dwarfs anything speculative decoding contributes. Bolting a drafter onto an endpoint launched with defaults is doing things in the wrong order.

There is no per-request switch. Whether speculation runs is decided when the engine starts and it applies to the whole endpoint. So if latency-sensitive traffic and bulk batch traffic share one endpoint today, separating them is the first step.

Once it is on, keep watching the acceptance rate. vLLM exports the number of proposed and accepted tokens as metrics, and the ratio between them is the alpha you were moving in the calculator. If the character of your traffic shifts after deployment, that value quietly drops and the speedup follows it down. When someone reports that things got slower even though the model never changed, look at this metric first. The same applies when you swap checkpoints. We saw the useful range move when we switched to a third-party INT4 checkpoint. Every new checkpoint means measuring the threshold concurrency again.

Finally, judge on final throughput and session capacity rather than on the multiplier. The multiplier grows as the baseline gets worse, so the better tuned your server is, the smaller it will look.

What not to trust here

Stated plainly. Each point on the concurrency curve comes from a single run, and the low-concurrency free-form region has noticeable spread. The edge-budget measurements used a fixed 8k prompt, so how the drafter behaves on a real 256k prompt is still unmeasured. We measured on B200, and the eventual target hardware is a different generation with different kernels. The sign of the result will probably carry over, but the absolute values will not. The collapse at 64 users is measured, though we did not go looking for the batch settings that would suit that point. The temperature comparison was measured on the board-reading drafter, so it should not be transferred to a trained one.

How we read this at ThakiCloud

This changed how we handle speculative decoding in Metis inference serving. Rather than enabling it globally, we measure the threshold concurrency per checkpoint and attach the drafter only to endpoints that live below it. On-premise appliances, where concurrent users are few and response latency matters, are where this technique pays best. Bulk batch processing is where it should stay off. Knowing when to switch a feature off protects serving cost just as much as knowing when to switch it on.

Further reading

This post is about speed, not quality. There is a good piece of teaching material that digs into the same gap from the quality side. Speculative Decoding: How It Evolved, When It Stays Lossless, and What’s Next by Lily Zhang and Madison Kanna, submitted to the NeurIPS 2026 Education Track, walks through the lineage of the technique and where the lossless guarantee breaks down in deployment, using figures and interactive demos. Their point that only a fraction of public traffic comes from measured domains is the same observation we reached from our own traffic mix.

Earlier posts in this series may help as well. The fundamentals are in a plain guide to speculative decoding, the combination with quantization is in 97 tokens per second became 458, and the choice of drafter is covered in it was lookup that did not fit.

The measurement ledgers are docs/measurements/2026-08-29-metis-edge-dflash2-drafter.json and 2026-08-31-metis-edge-dflash2-edge-budget.json.