The Friend Writes First, the Teacher Grades It: Speculative Decoding Made Easy

If you have ever watched a chatbot type out letters one by one and wondered why it is so slow, this single post gives you both the answer and the fix. Modern inference engines use a technique called speculative decoding that speeds things up two to six times without changing a single letter of the answer, and the idea behind it is simple enough for a grade schooler to follow. A quick-witted friend writes first, and the teacher only grades the work.

Writing One Letter Means a Trip to the Library

The way large language models write sentences is more frustrating than it looks. Once the model settles on a letter, it appends that letter back to the input, runs the whole calculation from scratch, and appends whatever letter comes out next. A hundred-letter answer means repeating this process a hundred times.

That part is well known. The real problem comes next. What eats up time in a single pass is not the calculation itself. Pulling the model’s weights from memory into the compute unit takes far longer.

Picture a library. To write one letter, the teacher has to pull hundreds of very heavy books off the shelves and spread every one of them open on the desk. Actually reading the books, though, takes no time at all. Most of the time goes into pulling the books out and opening them up. Then the teacher writes one letter and puts the books back. To write the next letter, the teacher has to pull them all out again from scratch.

It sounds wasteful, but there is no other way. You need to know the next letter before you can compute the one after that. In technical terms, this bottleneck is called a memory bandwidth limitation. The GPU has plenty of compute power to spare; it is just sitting idle because the path carrying the data is too narrow.

Seat a Quick-Witted Friend Next to You

There is a way out once you notice one thing. Not every letter is equally hard to write.

Think of a well-known opening line, one that everyone can finish without a second of thought. Once the first few words are set, the rest follows almost automatically. If you open a bracket, you eventually have to close it, and once you have written “so I,” what comes next is usually predictable. Names of people, unfamiliar technical terms, or a conjunction that turns a sentence in a new direction, on the other hand, really do call for careful thought.

There is no reason for the teacher to spread out every heavy book just to write an easy letter. So we seat a small, fast model in the next chair. We call this friend the draft model. Because it is small, its books are thin too, so it writes much faster. In exchange, it is sometimes wrong.

This friend writes out about five letters in a row first. Then the teacher grades those five letters. Correct ones pass through, and starting from the first mistake, the teacher erases the rest and writes them instead.

Grading Five Letters Costs Almost the Same as Grading One

Why this method actually pays off is the most important part of this post. Since the teacher still has to grade in the end, it is natural to ask what gets faster.

Here is the answer. Once the books are open, checking five letters at once is nearly free.

Writing one letter still requires spreading out all the heavy books, and grading five letters only requires opening them the same single time. What ate up the time was pulling the books out in the first place. Once everything is spread across the desk, scanning five spots is handled by the spare capacity that was already sitting idle. Earlier we said the GPU’s compute power was going unused; speculative decoding is exactly the technique that puts that idle capacity to work.

So the math comes out like this. Before, getting five letters meant opening the heavy books five times. Now it means opening the thin books five times plus the heavy books once. Five thin-book passes are far cheaper than one heavy-book pass.

The Guarantee That the Answer Doesn’t Change

This is where the worry comes in. Doesn’t using the friend’s letters as-is make the answer worse?

It doesn’t. And this is the real appeal of the technique. With the right grading rule in place, the final output ends up statistically identical to what the teacher would have written alone, start to finish.

Here is the rule. For any given letter, call the friend’s confidence q and the teacher’s confidence p. Grading comes down to comparing these two numbers.

If the teacher would have wanted that letter at least as much as the friend did (p is greater than or equal to q), it passes automatically. The friend was being modest, so there is no issue. But if the friend was overconfident (q is greater than p), the letter gets erased with a probability proportional to that gap. To be precise, it passes with a probability equal to p divided by q. If the friend wrote with 0.9 confidence but the teacher only had 0.45, the letter only has a fifty-fifty chance of passing.

What happens after erasing matters too. The teacher cannot just resample as usual, because the distribution has already been skewed by that first filter. Instead, the teacher resamples from what is left over after subtracting the friend’s confidence from the teacher’s own, clipping any negative values to zero, renormalizing the whole thing back to one, and then drawing from that.

Follow these two rules and the math works out to the exact same distribution as the original. This is not trading quality for speed; the speedup is simply free.

How Many Letters Should the Friend Draft Ahead?

Does raising the number of letters the friend drafts ahead always help? Not really.

Say the friend’s odds of getting a single letter right are 70 percent. The first letter has a 70 percent chance of passing. Surviving to the second letter requires both the first and second to be correct, which drops to 49 percent. The third falls to 34 percent, the fourth to 24 percent, and it keeps sliding. The moment an earlier letter gets erased, the context changes and every letter after it is thrown out too.

So even if you have the friend draft ten letters ahead, only three or four survive on average. The rest is wasted effort on the friend’s part, and that wasted effort still takes time. The average number of letters that pass per round is called tau, and past a certain point, cranking up how many letters get drafted barely moves tau at all.

Around a 40 percent hit rate, drafting 3 letters ahead or 7 gives almost the same result. To gain anything from a higher count, the friend has to get smarter first.

This chart sums up where research has been heading for the past few years. Because raising the draft count hits a wall quickly, people shifted toward making the friend better at guessing right.

The Ways of Building That Friend Kept Getting Better

The earliest approach was simple: bring in a smaller model from the same family and seat it next to the teacher. It requires no extra training, which is convenient, and it delivers roughly 2x to 3x. The catch is that you now have two models running, so it costs that much more GPU memory.

Medusa came next, and instead of keeping a separate friend, it bolted several extra hands onto the teacher’s head. Each hand predicts the second, third, and fourth letters at the same time. It delivered 2.2x to 3.6x, but it had a weakness. The hands do not consult each other, so the farther-out hands tend to produce off-target letters.

MTP (multi-token prediction) takes the opposite approach: instead of bolting the extra hands onto a finished model, it raises them together with the model from the very start of training. As a bonus, this even sharpens the base model’s own ability, so the 3x inference speedup ends up being almost a side effect.

The EAGLE family is the most widely used approach today, and it came from a shift in framing. Instead of asking the friend “what is the next letter,” it asks “what will the teacher’s internal state look like next.” Inheriting the internal state turned out to line up much better than inheriting a letter, and this alone delivered 2.7x to 3.5x. EAGLE-2 followed by preparing several branches at once whenever the friend was unsure, pushing the range up to 3.05x through 4.26x. EAGLE-3 went back to predicting letters directly, but trained the friend under the exact conditions it would face in production, and reported up to 6.5x.

The most recent arrival, DFlash, takes a completely different approach. Instead of writing letters one at a time in order, it produces a whole block at once. It applies the same idea diffusion models use to sharpen a blurry image step by step, but to a block of letters. It averages 4x to 6x, and it scales well as the friend model grows larger.

Add DSpark to the mix and the recent trend becomes clear. This approach does not chase a better draft. Instead, it watches how much load the server is under right now together with how confident the friend is, and adjusts when grading happens. It boosted the speed a single user feels by 60 to 85 percent, at the cost of more moving parts and a more complex implementation.

Method	How the draft is made	Avg. accepted per round	Speedup
Separate small model	One letter at a time, in order	About 3.6	2x to 3x
Medusa	Several hands at once	3.0 to 3.5	2.2x to 3.6x
EAGLE-3	One letter at a time, in order	5 to 7.5	Up to 6.5x
DFlash	Whole block at once	4 to 8	6x or more
DSpark	Semi-autoregressive plus verification scheduling	3.1 to 6.2	1.6x to 1.85x felt by users

The Payoff Shrinks as the Crowd Grows

This is where a lot of people trip up. They turn the feature on expecting the 6x from the paper, and in production it barely speeds anything up. There is usually one reason.

When you are alone in the library, the time spent pulling books out really does feel wasted. But once a hundred people are lined up, the picture changes. Pulling the books out once now serves all hundred people at the same time, so that cost is already split a hundred ways. There is nothing left to save.

Speculative decoding wins the most when concurrent users are few. Once enough users pile in that the GPU’s compute is already saturated, the whole premise of putting spare capacity to work disappears. In bad cases, all you add is the cost of drafting, and you actually come out behind.

So here is a simple rule of thumb. If how quickly a single response comes back matters to your service, it is worth turning on. Conversational chatbots, coding assistants, and agents that reason through many steps on their own all fit here. On the other hand, if you are running an overnight batch job processing documents in bulk, you do not really need it.

Turning It On Yourself

Every major inference engine supports this today. vLLM, SGLang, llama.cpp, and MLX all need just a line or two of configuration. Here is an example of launching with the DFlash method on SGLang.

python -m sglang.launch_server \
  --model-path <path-to-target-model> \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path <path-to-draft-model> \
  --speculative-num-draft-tokens 5

Once it is on, check two numbers in the logs. One is the acceptance rate, and the other is the average accepted length per round. If the accepted length cannot get past 2, the friend and the teacher just are not a good match, and swapping out the draft model is the right move, not raising the draft count.

What to Check Before You Turn It On

Running GPU serving ourselves taught us one lesson the expensive way. Speculative decoding is something you turn on last, not first.

In August 2026, we measured throughput on our B200 hardware while changing nothing but the serving configuration. Same model, same GPU, same engine, and yet two settings alone produced an 18.8x difference in single-stream throughput. Compilation was off, and the number of requests that could be handled concurrently was stuck at the default of 32. What had been 7.4 tokens per second became 138.8.

What happens if you turn on speculative decoding in that state? You would be using a technique that gets you 6x to try to claw back an 18.8x loss. The order is backwards. Fix the base configuration first, and only then layer speculative decoding on top, so the technique can actually deliver what it is capable of.

Our inference product, Metis, exists precisely so tenants do not have to fine-tune these serving settings themselves. And the place this technique fits especially well is Paxis. An agent handling a single request calls tools, reads the results, and reasons again, over and over, and during that loop concurrent users tend to be few while the latency of each step becomes the felt speed directly. That is exactly the “wins the most when the crowd is small” condition described earlier.

If you tried speculative decoding once already and gave up on it, our own record of running into the same thing and reopening the question might help. In It Wasn’t Speculative Decoding That Was Slow, It Was the Lookup Method, we walk through how a single method choice produced an 11.9x swing. DFlash Block-Diffusion Drafting and Running EAGLE on vLLM also cover the actual configuration values we used.

Wrapping Up

Speculative decoding boils down to one sentence. A fast friend writes first, and a slow but precise teacher grades it all at once. With the grading rule set up correctly, not a single letter of the answer changes.

The trajectory of the field is worth noting too. At first, it was all about “how do we make the friend smarter.” It started with a separate model, moved on to bolting hands onto the teacher’s head, and arrived at inheriting internal state instead of letters. But recent research is asking a different question. As more users pile in, when and how should grading be batched together, and scheduling verification is turning out to be the harder problem, more so than drafting itself. This looks like where the fight will be over the next few years.

The original explanation behind this post is Leonie Monigatti’s Speculative Decoding. If you need the formulas and paper links, we recommend reading the original alongside this one.