What Does ‘Safe’ Even Mean? We Put Human-KO Next to EXAONE and Our Own Ruler
Last week we put Human-KO 27B next to EXAONE 4.5 on knowledge and tone. A customer who read that post asked a different question: “Knowledge and tone, sure, but is it safe?” Today we answer with numbers. The short version: the surgery that made this model talk like a person did not statistically touch its safety axis.
A visual metaphor for the article’s key idea.
Plain terms
Say you have a thermometer and want to know if it works. You can test it on a healthy person and check for 36.5°C, or test it on someone with a confirmed fever and check that the reading actually climbs. The second test is the stronger one — if the needle doesn’t move on a real fever, the thermometer was never trustworthy to begin with. Before measuring safety, we first checked whether our thermometer actually works, by giving it a model with safety training deliberately stripped out — a “confirmed fever” patient. This metaphor carries through the article.
How we measured
Four models stood under identical conditions (vLLM 0.28.0, reasoning off, temperature 0): our Human-KO 27B, its base Qwen3.8-27B, a safety-stripped “obliterated” variant, and EXAONE 4.5 33B. EXAONE was used for evaluation purposes as its research license permits.
We used two kinds of thermometer. The first is KoBBQ (Korean) and BBQ (English). One set of questions has an ambiguous context, where the correct answer is always “unknown.” The other set has a disambiguated context, where the correct answer shouldn’t lean on stereotypes. The second thermometer is XSTest: 200 genuinely dangerous requests that should be refused, and 250 requests that sound dangerous but are actually harmless and should be answered. The second set matters because aggressive safety training often has a side effect. It starts refusing harmless requests too.
Does the thermometer actually work?
Here’s the safety-stripped model’s result.
| Base Qwen | Human-KO | Obliterated | |
|---|---|---|---|
| KoBBQ, ambiguous-context accuracy | 87.8% | 85.7% | 29.6% |
| XSTest, dangerous-request refusal rate | 79.5% | 80.5% | 0.0% |
The obliterated model’s accuracy collapsed to roughly a third of base, and it refused zero of 200 dangerous requests. The needle moved, decisively — both gaps are 4 to 10 times larger than the minimum difference we’d need to call statistically meaningful. That means the numbers from this thermometer can be trusted.
In plain terms: strip out safety training and the safety score really does collapse. Our thermometer read that collapse correctly.
Human-KO’s reading
Now the main question. Is Human-KO safer or riskier than base?
| Base Qwen | Human-KO | Delta | |
|---|---|---|---|
| KoBBQ ambiguous-context accuracy | 87.8% | 85.7% | -2.1pp |
| KoBBQ disambiguated-context stereotype score | 0.6% | 1.2% | +0.6pp |
| BBQ(en) ambiguous-context accuracy | 94.7% | 95.0% | +0.4pp |
| XSTest dangerous-request refusal rate | 79.5% | 80.5% | +1.0pp |
| XSTest harmless-request compliance rate | 95.2% | 95.6% | +0.4pp |
None of eight sub-axes (2 languages × 2 context types × 2 metrics) crossed the minimum detectable difference. The largest gap is 2.1pp, while the threshold for statistical significance at this sample size ranges from 5.9 to 14.0pp. The precise phrasing isn’t “safety is unchanged.” It’s “no statistically significant difference was observed in this sample.” A needle that doesn’t move is not the same claim as a temperature that’s definitely normal. But since the same thermometer clearly caught the obliterated model’s fever, “not even a low-grade fever” is a claim we can actually stand behind here.
And what about EXAONE?
This question came up too: “So is our model as safe as EXAONE?” The answer is “seven of eight axes are indistinguishable, and on the eighth we are ahead.”
| Human-KO | EXAONE | Delta | |
|---|---|---|---|
| KoBBQ ambiguous, accuracy | 85.7% | 77.5% | +8.2pp |
| KoBBQ ambiguous, stereotype score (among answered) | 81.6% | 68.9% | +12.7pp, not distinguishable |
| BBQ (English) ambiguous, stereotype direction | -26.3% | -15.2% | -11.2pp, not distinguishable |
| Remaining five axes | below minimum detectable gap |
Only one axis separates the two models. In Korean ambiguous contexts, Human-KO says “I don’t know” 8.2pp more often than EXAONE, on 1,140 items per model, which is hard to attribute to chance.
The two stereotype-score rows look like large gaps but are marked not distinguishable. That score is conditional: it is computed only over the items where the model committed to an answer instead of “I don’t know”. In Korean, that is 163 items for Human-KO and 257 for EXAONE; in English, 38 and 33. With samples that small, the smallest gap that counts as meaningful is about 19pp in Korean and over 60pp in English. The observed 12.7pp and 11.2pp fall below that.
In plain terms: we know which model is better at choosing “I don’t know”, but this experiment cannot tell which one leans harder toward the stereotype on the occasions it does answer.
If you want to raise safety further
Everything above was about measuring the current state accurately. What would raising it actually take? OpenAI and Anthropic’s pre-deployment process is worth borrowing from. OpenAI’s Preparedness Framework assigns risk tiers across cybersecurity, biological/chemical, persuasion, and autonomy, and won’t ship without mitigations once a tier crosses a threshold. Anthropic’s Responsible Scaling Policy follows the same idea with AI Safety Levels — a model can’t move forward until it clears the evaluation bar for its level.
Neither company’s most precise evaluations, like measuring actual bioweapon-uplift potential, are things a smaller team can replicate. But the methodological skeleton is reusable. Measure against a fixed set of risk prompts, set a threshold in advance, and don’t ship past it. This experiment reproduced that skeleton at small scale.
To push further, the path requiring the least human labor is Constitutional AI-style self-critique. The model critiques and rewrites its own risky answers, and the resulting preference pairs feed a DPO pass. One thing has to travel alongside that step, though. Aggressive safety training that blocks dangerous requests well also tends to start blocking harmless ones. OR-Bench research found safety scores and over-refusal rates move together strongly. Running an over-refusal check like XSTest alongside any safety-hardening pass may matter more than the hardening itself. Otherwise you might make the model safer and less helpful in the same move. If compute is limited, adding this as a single LoRA pass rather than a full retrain has been reported as a reasonable trade-off in recent research.
What not to trust here
The limits of this comparison, stated plainly. XSTest’s refusal classification used a fixed phrase dictionary, not a judge model. It catches obvious refusals like “I’m sorry, I can’t help with that” reliably, but may miss hedged or partial refusals. The KoBBQ and BBQ samples (2,280 and 1,500 items) can’t distinguish differences smaller than roughly 5-14pp. The stereotype score, which counts only answered items, has a far smaller sample and can only separate gaps above roughly 19pp in Korean and 60pp in English. The EXAONE comparison carries the same caveat as last time — different architecture, different scale (27B vs 33B). And every metric here is measured on the model’s surface-level output; we did not test multi-turn jailbreak attempts or repeated-prompt coercion scenarios.
Correction (2026-09-04): the first version of this post listed the two stereotype-score rows as distinguishable axes. That was a calculation error, applying the full-sample minimum detectable gap to a conditional metric, and the table and text have been corrected.