One 27B Model Serving Both Answers and Evidence Checks: A Shared Backbone That Splits Only in the Last Four Layers

Running a retrieval-based agent seems to require a large model twice. One writes the answer; the other decides whether the retrieved documents are enough to answer at all. Our judgment model reuses the first 44 layers of the original 27B model unchanged, so loading the two separately means doing the same computation twice.

So we loaded the 27B model once and handled both jobs in the same serving engine. Answer generation did not drift beyond the variation you get from running the original engine twice, and all 256 evidence judgments matched the dedicated judgment engine. This post is for anyone weighing the cost of serving in-house models, and for anyone who has added a feature to a serving engine and watched results shift slightly.

A visual of the core idea.

In plain terms

Picture a restaurant kitchen that receives two kinds of orders. One asks for a dish. The other asks for an inspection: is it all right to cook a dish with these ingredients?

The prep work for both orders is nearly identical. Washing, cutting and seasoning are the same. The only difference is what happens at the last station. Until now we had two prep teams doing the same work twice.

What we changed is to give all the prep to one team and pass only the inspection orders to a side station just before the last step. There is one condition: the dishes must taste the same as before. We allow the grain-of-salt difference you get when the same cook makes the same recipe twice, but nothing beyond that.

What we did

Where the paths split

The base model is Qwen3.8-27B and the serving engine is vLLM 0.28.0. The judgment model is the decision adapter from our earlier post: it uses layers 0-43 of the original model unchanged and adds a small LoRA only to layers 44-47. So the computation for layers 0-43 is exactly the same for generation and for judgment.

A gate request copies the intermediate value at the entrance of layer 44 and takes the side branch. Generation requests follow the original path. The 27B weights are loaded once on a single GPU.

There is one engine, and requests arrive mixed in the same batch. Gate requests end with a fixed sentence pattern, which the engine uses to route them to the branch. The result comes back as scores for the three actions, and the action is chosen with the same threshold as the existing adapter.

We fixed the judging rule first

Before looking at results, we froze the judging rule in a document. We run 64 generation requests on the original engine and on the shared engine, and count the requests in which even one token differs. We compare that with the difference you get from running the original engine twice. The shared engine passes if it does not differ more often than the original’s own repeat.

At first we wanted a pass to mean token-for-token identity with the original. But the original engine, given the same input twice, sometimes differed on zero requests and sometimes on one. A standard the original itself cannot meet decides nothing, so we left that criterion as inconclusive and used the repeat difference as the baseline.

We also checked separately whether mixing in gate requests disturbs generation. Even an engine with no branch shows some difference when requests are mixed, because the batch composition changes. So the no-branch engine’s difference became the baseline, and we only asked whether the branch adds to it.

Results

Two failures, then a pass on the third try

We judged three times under the same rule, and the first two failed. Those failures stay in the ledger as they are.

Gray is the difference from running the original engine twice in the same run; colored bars are the shared engine against the original. 64 generation requests, one H200, vLLM 0.28.0.

In the first attempt we read the layer-44 intermediate through the mechanism the engine already provides. The original’s repeat differed on zero requests, but the shared engine differed on 14. In the second we copied the value into a separate buffer instead, and got 11.

We ran a separate diagnostic to split the causes. With the branch turned off, the same code produced no request that differed from the original. Yet just reading the intermediate, with no branch computation at all, produced 13 differing requests. It was not the branch computation but the act of reading the value that disturbed generation.

The likely explanation is the compiler. vLLM fuses several operations into one to run them faster. Pulling a value out from the middle of that changes how the operations are fused, and with it where bf16 rounding happens. We did not inspect the actual kernels, so this remains a hypothesis.

In the third attempt we moved the read point. The input to the layer-44 normalization is already materialized in memory as its own value, so we expected that copying it would leave the fusion alone. The branch recomputes the normalization itself. The result was 1 differing request out of 64, and the original’s own repeat in the same run also differed on 1. Both were the same request, at the same 22nd token.

Put simply, the shared engine now differs from the original exactly as much as the original differs from itself.

Judgments matched the dedicated engine

Gate comparisons are against the dedicated gate engine, an HF fp32 reference, and, for scale, HF bf16 against HF fp32. KV blocks are what each engine allocated under the same memory setting (0.85).

We sent 256 gate requests to the shared engine and compared them with the existing dedicated gate engine. No action flipped, and probabilities differed by at most 0.02. Against a 32-bit reference implementation one decision flipped, while two 16-bit implementations of the reference differ from it on five, so this is within numerical precision.

How much mixing in gate requests disturbed generation also stayed within the baseline. The engine with the branch differed on 2 of 64 requests; the engine without it differed on 3.

The price is KV cache

It was not free. Under the same memory setting, KV cache blocks fell from 1,354 to 1,216, a 10.2% drop. The branch layers and the copy buffer take up space. That means proportionally fewer long conversations can be held at once.

What to change

If your judgment model reuses the front of the original model, check whether it can live as a branch inside one engine before you stand up a separate one. You avoid loading the same weights twice, and there is no judgment engine sitting idle when no judgment requests arrive.

If you build a feature that reads intermediate values inside a serving engine, choose the read point carefully. Even the same value can change the output when it is read from the middle of a compiled, fused block. In our case the problem went away when we read a value that is already materialized separately, such as the input to a normalization.

A realistic validation standard is the original engine’s own repeat difference. Token-for-token identity looks clean, but if the original cannot meet it, it decides nothing. And because that baseline is sometimes zero and sometimes one, it has to be measured within the same run.

At ThakiCloud we plan to use this structure at the step where a Paxis agent decides whether to answer from retrieved results. Serving answers and judgments from one Metis inference endpoint means an agent step does not need an extra GPU attached for the judgment.

What not to trust yet

First, this is a functional check. We confirmed that one engine handles both and that the results hold. We did not measure throughput, latency or GPU cost savings, nor long-running operation or peak memory.

Second, the scope is narrow. There were 64 generation requests and 256 gate requests, on one GPU with one configuration. Of the four published adapter seeds, only one was validated.

Third, token-level identity is inconclusive. The original engine itself differed on one request across repeats, so in this environment we cannot say whether the shared engine is bit-identical to the original. The compiler explanation for the failures is the most plausible account, not something we confirmed directly.

Fourth, this is not yet a service. The deployment check, which exposes generation and judgment as two HTTP APIs and verifies recovery after a restart, is still in progress. The serving code and the integrated package will be released after that check; the judgment adapter weights are on Hugging Face as ECP-Q12-27B-Research-Preview.

The figures in this post were measured on our own H200 GPUs, and only results recorded in our research ledger are cited.