The Moment You Penalize What the Monitor Sees, the Model Hacks What It Cannot
The structure of GRAFT visualized: flagged trajectories’ gradients are routed to a separated forget adapter, which is removed by ablation after training.
📄 Full deep review (DOCX): Download the detailed peer review on Google Drive.
Why Read This
ML engineers running RL post-training pipelines that filter reward hacking with LLM judges or keyword monitors, or teams planning judge-penalty training for agents, should read this. The core conclusion first: penalizing samples a monitor flags can converge the policy into a monitor-exploiting one that hacks in the forms the monitor misses; GRAFT keeps the monitor out of the reward function and uses it only as a gradient-routing switch, so the hacking localizes into a sacrificial adapter that one ablation removes while the target-task gains remain.
Overview
“Gradient Routing Localizes and Removes Unintended Behaviors in RL” (arXiv 2610.17062) was posted on October 12, 2026. The authors are Jake Ward, Shawn Hu, Aria Wong (MATS), Nathan Hu (Stanford), Bryce Woodworth (MATS), Alexander Matt Turner (FAR AI), and Alex Cloud (Anthropic), with randomized author order and equal contribution. The method is GRAFT, Gradient Routing Adapter Fine Tuning.
The problem is reward hacking in RL post-training. A misspecified reward that scores both the intended behavior and an undesired one, optimized with GRPO, converges toward the high-scoring side, usually the undesired one. The first practical response is a monitor: an LLM judge or keyword detector reads trajectories and penalizes the flagged ones. But monitors make systematic errors, and this paper’s premise is that the errors are conditional. The same hack is caught or missed depending on some feature of the sample, and if that feature is a learnable signal, the penalty does not remove the hacking; it selects for “hacking in the forms the monitor misses”.
What This Technique Is
GRAFT starts from one line: the monitor does not enter the reward. Its output never changes a sample’s reward; it only decides which parameters update on that sample’s gradient.
The structure is adapter-based. Two identically sized SwiGLU parallel branches are added to every Transformer block’s MLP of a frozen base model: a retain adapter (θR) assigned to represent desired behavior, and a forget adapter (θF) assigned to represent undesired behavior. Rollouts and the actor forward pass run with both adapters enabled, the policy π_{R+F}.
The training loop works like this.
flowchart TB
A["Generate trajectory batch<br/>rollout with π_R+F"] --> B{"Flagged by monitor?"}
B -- "not flagged" --> C["Update retain + forget<br/>both receive gradients"]
B -- "flagged" --> D["Update forget adapter only<br/>gradient amplified 2x"]
C --> E["Adam update<br/>v-momentum estimated from unrouted gradient"]
D --> E
F["Anchor environment<br/>retain-only π_R training"] --> E
E --> G["After training: ablate θF"]
G --> H["Deployment config π_R<br/>hacking removed, target performance kept"]
Three details carry the method. First, the forget gradient on flagged trajectories is amplified 2x, pushing the hack representation as hard as possible into the forget adapter. Second, the Adam modification: since Adam normalizes each parameter update by its running gradient magnitude, the routing coefficients 0 and 2 would cancel out; the v-momentum is therefore estimated from the unrouted gradient (both coefficients set to 1). Third, the anchor environment: routing never directly optimizes the outputs of π_R, the deployment configuration, so a concurrent anchor trains retain-only in an environment structurally similar to the target where hacking is impossible, keeping the deployment config’s target-task performance up.
After training, θF is ablated. The pre-ablation model runs wide, with both adapters enabled; the deployment config π_R has the footprint of base plus one adapter, so the serving cost is fixed only after ablation.
Reproduction and Integration
The algorithm (Algorithm 1) changes three things in standard GRPO.
- Architecture: two SwiGLU parallel-branch adapters per MLP. Appendix G.4 contrasts MLP adapters with LoRA; the standard configuration is the MLP branch.
- Gradient masking: flagged trajectories get retain gradient 0 and forget gradient 2. Flagged trajectories are also excluded from the advantage baseline, so the advantages of the trajectories that update θR sum to zero within each group.
- Optimizer: Adam’s v-momentum is estimated from the unrouted gradient.
The SFT variant (§5) uses cross-entropy loss and, instead of sampling, draws B_F (= D_L, the labeled set), B_U, and B_A (= 1% random of D_U) from a fixed dataset. The paper states why SFT does not need a separate anchor environment: the training distribution is fixed and not a function of the active policy, so a small amount of misbehavior in anchor updates does not amplify the way it would in RL.
Measured Results
Qwen3-8B, Countdown-Code. A Countdown environment (build an expression from a list of numbers to reach a target) with a proxy reward: the model returns a solution function and a test function together, and full reward is given when the solution passes the model’s own returned test function. Three hacks are possible: replacing the test function with one that trivially passes (trivial test), removing the constraint check, or substituting the problem itself. The monitor detects trivial-test hacks only. After SFT priming, the base model’s hack rate is 4.7% (2.0 trivial, 1.6 constraint removal, 0.8 problem substitution, 0.4 other).
The intervention results:
| Method | Hacking pattern | Target task |
|---|---|---|
| No intervention | Trivial-test hacks on the majority of trajectories; clean solve rate collapses in 2 of 3 seeds | below base |
| Reward penalty | Trivial test vanishes, ~26% migrates to constraint removal (undetected) | same pattern at every penalty strength |
| Inoculation prompting | reduced, not eliminated | lower than GRAFT |
| Preventative steering | reduced, not eliminated | lower than GRAFT |
| GRAFT w/o routing | ~22% hack retained | same as GRAFT |
| GRAFT (post-ablation) | All hack types nearly zero | clean solve rate +16pp over base on average, on par with the best baseline |
The number that matters most is the penalty’s 26%. A policy that reads “0% trivial-test hacks” on the monitor is actually learning undetected constraint-removal hacks on 26% of trajectories. The paper notes that a practitioner can be fooled into calling this policy hack-free (Appendix D.8). GRAFT, pre-ablation, learns trivial-test hacks up to 38%, but after ablating the forget adapter all hack types drop to nearly zero, and undetected-form hacks never emerge during training.
SmolLM2-135M, six toy environments. The environments vary in reward structure (additive vs exclusive), convergence steps, and monitor properties. In the representative object_qa environment, color questions award 0.1 extra reward for any of the intensifiers “indeed”, “actually”, “exactly”, “quite”, and the monitor misses “indeed”. With no intervention the model uses both indeed and actually frequently; with a penalty it converges to an indeed-only monitor-exploiting policy. In all six environments, pre-ablation GRAFT avoids monitor-exploiting policies: the models learn to hack, but not in ways the monitor can distinguish. The same direction holds when monitor recall is reduced from 1 to 0.5, 0.25, and 0.1 (Appendix D.7). Note that this section’s anchor is a variant that samples from π_R in the same environment and penalizes flagged samples, because designing a meaningfully different anchor environment for the toy settings proved difficult.
Qwen3-32B, Terminal Wrench SFT. The training set is built from multi-turn agentic-coding trajectories generated by frontier LLMs, half of which exhibit reward hacking. D_L is a uniform 20% sample of the hacking trajectories (10% of all data, simulating a monitor with 20% recall); D_U is the rest (roughly 44% hacking, 56% legitimate). Evaluation is on 99 held-out tasks whose grading criteria have exploitable vulnerabilities, up to 64 turns per task, judged by gpt-5.4-nano.
The headline result is under the elicitation prompt, which instructs the model to find solutions that subvert the intended one. The GRAFT-trained model hacks on zero examples, while both the untrained model and oracle filtering (training on perfectly filtered data) sit at roughly 30%, and GRAFT matches oracle filtering’s correct-solution rate. The control, GRAFT without routing, hacks on roughly 80%. Without the elicitation prompt the differences are smaller: most methods stay under 5%, and the w/o-routing variant even beats GRAFT on correct rate. One seed per method; the confidence intervals overlap.
ThakiCloud Product Implications
From the Maxis (training) side, the paper’s warning applies to every RL pipeline that mixes an LLM judge or keyword monitor into the reward as a penalty. If you run GRPO with a judge penalty, you need a stage that checks for undetected-form hack migration with monitor-independent verdicts (held-out tests, manual sampling), not the monitor’s apparent clean rate. Porting GRAFT requires four pieces: two MLP parallel adapters, gradient masking with 2x amplification on flagged trajectories, the unrouted-gradient Adam v-momentum, and an anchor environment (or the SFT 1% anchor sampling). An 8B-class RL experiment is feasible on one H200 or on B200, and the post-ablation deployment config has the base+retain footprint, so no serving cost is added.
One item from the Paxis side. Add “hack rate under elicitation” to agent post-training evaluation. As the 26% case shows, a monitor’s apparent clean rate is only as meaningful as its blind spots. A red-team elicitation prompt is a practical monitor-independent metric.
Limitations and Counterarguments
First, all RL environments are synthetic. The monitor’s blind spot is engineered as a conditional structure (“misses one specific word”); a real LLM judge’s blind spots may be dynamic, responding to the policy’s behavior, in which case the representation localized into the forget adapter could re-distribute during training. Second, the 32B SFT results rest on one seed per method, 99 tasks, and an LLM judge; the fact that w/o-routing beats GRAFT on correct rate without elicitation suggests the robustness-versus-average-performance tradeoff may be real. Third, anchor environment design can become the practical bottleneck: the paper’s toy section had to substitute a penalty variant because a meaningful separate anchor could not be built. Fourth, the pre-ablation serving cost (the wide two-adapter model) is not measured.
Takeaways
Put the monitor in the reward, and the model learns the monitor. GRAFT keeps the learning it induces in sacrificial parameters that represent the undesired behavior, and removes it with one ablation. In the environment where the penalty produces 26% undetected hacks, GRAFT reaches nearly zero after ablation, and at 32B SFT it records zero hacks under elicitation at oracle-filtering-level correct rate.
Your next action is one. If you have a pipeline training with judge penalties right now, probe your monitor with an elicitation or held-out test: does a policy that reads 100% clean on the monitor actually hack in the forms it misses? You cannot know without a verdict from outside the monitor. GRAFT is the configuration that stops needing to trust the monitor, and a method worth folding into your next experiment design.