Stop Saying ‘95% Accurate’: Guarantee an Error Ceiling on What You Answer Instead

This post is for engineers and PMs at Korean enterprises in finance and the public sector who want to attach an LLM to a rule-based decision. The short version: “95% accurate” is not a sentence you can defend to an auditor. What you can defend is this: “with 90% confidence, the error rate on the questions this model actually answers stays under a target α, and to keep that promise it abstains on X% of questions and routes them to a person or a larger model.” We put this on top of ThakiCloud’s K-Decision and report what actually happened.

How this model was trained and how it compares to Jev is covered in two earlier posts, Teaching a 4B Model to Judge Numbers in Korean Statutes and Why We Built a Korean Jev. This post adds a layer that turns that model’s probability output into an auditable guarantee.

Why accuracy alone does not survive an audit

K-Decision is a typed decision model: given a document state and a question, it returns a probability over the options for one of three types, choice, noul (yes or no), or score. Confidence is the maximum probability in that distribution. The problem with “95% accurate” as a single number is that it tells you nothing about whether the question in front of you falls inside that 95% or inside the 5%. An audit log that reads “a 95%-accurate model answered yes” forces a human reviewer to go back and re-read the source to figure out how much to trust that particular verdict.

A family of methods from conformal prediction, specifically Learn-then-Test (Angelopoulos et al.), attacks this from a different angle. Instead of raising accuracy, it sets a threshold on the model’s own confidence and abstains entirely on questions below it. For the questions that clear the threshold, the ones it actually answers, it gives a statistical guarantee that the error rate stays under a target α you choose in advance. That guarantee rests on the assumption that new questions look like the ones used for calibration, and later in this post we show exactly where that assumption breaks.

The threshold is set separately for each of K-Decision’s three question types. Candidate thresholds are tested one by one against calibration data with exact binomial p-values, and because there are three types, the overall error budget δ=0.10 is split across them with a Bonferroni correction. Calibration and test data are split by statute, so articles from the same law never end up on both sides.

What tightening the target α does to statute questions

The public KD-Lawset-KO set is generated by code from Korean statute text (the National Law Information Center, legalize-kr). The questions ask whether a fine cap was exceeded, whether a deadline was met (including date arithmetic, the exclusion of the first day, and holidays), what the fine cap of a given paragraph is, and what the prison-term cap is. Under lawset-v2 there are 532 calibration items, 542 test items, and a held-out shift set of 393 items drawn from ministries never used in calibration or testing.

Tightening the target α raises the abstain rate and lowers the error rate on answered questions. At α=0.01 and 0.02, both domains abstain on everything and cannot be certified at all.

At α=0.05, the model abstains on 59.6% of statute questions and on the remaining 40.4% it guarantees an error rate of at most 5%; the observed error on this test set was zero. Relaxing α to 0.10 drops the abstain rate to 18.1%, but the error rate on answered questions rises to 2.5%. Push α down further to 0.01 or 0.02 and this set cannot be certified at all: no threshold clears the bar, so everything is abstained on. The point of this chart is not that a lower abstain rate is better. It is that abstain rate and error rate move on the same dial. Tighten it and fewer questions get answered; loosen it and the answered ones get less reliable.

On this set, the 4B model already gets every choice and score item right. The dial is effectively moved by noul (yes/no) questions, and within those, mostly deadline arithmetic. When forced to answer everything, the error rate on deadline items runs around 31%, a clear outlier against the other types. Confidence appears to wobble on rule-dense calculations like excluding the first day or accounting for holidays, so tightening α concentrates abstentions almost entirely there.

The dial runs the other way on internal documents

The internal-only AI Hub document QA set, with human-made labels, has 1,402 items and is not released. Its base accuracy starts from a higher point than the statute set: 97.4% on choice, 95.1% on noul.

The same α=0.05 threshold produces an abstain rate of 26.6%, less than half of the statute set, with 1.3% error on answered questions. Relaxing to α=0.10 drops the abstain rate to just 0.7%, but the error rate climbs to 3.7%, actually higher than the statute set at the same α. You can see the two lines cross between α=0.05 and 0.10 in the chart, and that crossing is exactly this. Statutes abstain heavily and clamp error down hard; AI Hub documents start from higher base accuracy and can afford to abstain less at the same α while landing in a similar error range.

This set carries one clear limitation. The split was done by question, not by document, so multiple questions from the same source passage may end up on both the calibration and test sides. Correlation within a passage could then make the guarantee look more optimistic than it actually is. Re-splitting by document id is the next step to close this gap.

Where the guarantee bends: re-calibrate when the domain shifts

To see how solid this guarantee is, we re-ran it across 500 law-grouped re-splits. In-distribution, the exact 95% lower bound on answered-question error exceeded the target α in at most 1.2% of those 500 re-splits. In other words, as long as the distribution does not change, the guarantee holds almost every time.

The problem shows up when the domain actually shifts. Applying the same thresholds to the held-out shift set, drawn from ministries never used in evaluation, pushed the noul error rate close to the target α. At α=0.15, for example, the measured error climbed to 14.2%, narrowing the margin to within about 1 percentage point of the target. We are reporting this as it is. Carrying a calibration from one domain straight into another erodes nearly all of the safety margin, which means re-calibrating thresholds on the new domain’s own documents is a precondition for using this method, not an optional extra.

Where an abstained question goes

Abstained questions are routed to an in-house Qwen3.8-27B model, with no external API calls. We verified this path on a separate confirmation set, 591 items drawn from Enforcement Decree text and never used for calibration. The 4B abstained on 406 of those, and we isolated that subset to compare the two paths.

Accuracy on the 406 deadline-arithmetic questions the 4B abstained on, forced to answer with the 4B versus escalated to the 27B. The 27B does not judge the answer directly; it only extracts the two dates and the day count, and code performs the calculation.

Forcing the 4B to answer those 406 questions anyway yields 58.4% accuracy. The 27B, in contrast, does not judge the answer directly. It extracts two reference dates and a deadline day count from the document, and a separate piece of code performs the date arithmetic, including the exclusion of the first day and holiday handling. That path reaches 99.3% accuracy, a 40.9 percentage point gap with a law-clustered bootstrap 95% confidence interval of [36.0, 45.8]. This was a preregistered hypothesis, and the result confirmed it. Median latency was about one second. Abstention here is not giving up on the question; it is routing the exact spot where the 4B is weak into a form code can compute.

How trustworthy are the labels themselves

Every guarantee in this post ultimately rests on the labels being correct. KD-Lawset-KO’s labels are computed by two independent programs. To re-check label quality, all 2,812 items were re-solved blind by gpt-5.6-luna, without seeing the original label, and the 25 items where the two disagreed were adjudicated by gpt-5.6-sol; to rule out the judge being anchored by the label, sol also re-solved the same 25 without seeing it, and all 25 matched the code label. The result was zero wrong labels and one ambiguous item, with a 95% upper bound on label error of 0.13%. This was a model-based re-verification, not a human audit, and we want that distinction on the record. It also missed something. A rule scan of the exact statute line each question targets, looking for caps that depend on an amount the question does not give (“three times the compensation”, “whichever is higher”), found 4 such items; the model audit had marked only 1 of them ambiguous. All four are flagged as ambiguous in the dataset, and removing them leaves the α=0.10 result almost unchanged at 19.0% abstained and 1.8% error. The generator no longer produces this form.

ThakiCloud product angle

This work connects directly to Paxis’s human-approval workflows. An agent that makes decisions grounded in regulation and automatically abstains on low-confidence cases, routing them to a human approval step, is exactly what conformal abstention statistically justifies. The audit log then holds only two states: “answered, error under α guaranteed at 90% confidence” or “abstained, routed to human review,” not a bare “yes.”

From the Aegis lens, the path after abstention is already self-contained on-prem. Abstained questions go to a self-hosted 27B model rather than an external API, and even that model does not judge the answer directly, only extracting numbers for code to compute, which maps directly onto finance, insurance, public-sector, and defense customers who cannot send documents outside their network. From the Maxis lens, the finding that thresholds need re-calibration when the domain shifts becomes a literal product requirement: re-calibrating on a new document population inside the customer’s own network has to be part of an ongoing fine-tuning and calibration pipeline, not a one-time step.

Limits and what is next

A few things should be clear before taking this post’s conclusions at face value. First, the AI Hub set was split by question rather than by document, a known limitation that could make the guarantee look more optimistic than it is. Second, the domain-shift result is a single case, and we cannot assume every new domain erodes the safety margin by the same amount. Third, label verification was model-based re-solving, not human review. Fourth, the 40.9 percentage point gain from escalating to the 27B comes from a narrow question type, deadline arithmetic, and there is no guarantee the same size of gain shows up on other question types.

What comes next is re-splitting the AI Hub set by document id to re-verify the guarantee, and measuring how often, and with how little data, thresholds can be re-calibrated each time the domain shifts. Until both of those are filled in, the numbers in this post should be read as an interim report, not a final verdict.

References

Public sources cited in this post.

• Angelopoulos et al. · Learn Then Test: Calibrating Predictive Algorithms to Achieve Risk Control (arXiv)
• ThakiCloud · ThakiCloud/kd-4b-ko-v0 (Hugging Face)
• ThakiCloud · ThakiCloud/kd-lawset-ko (Hugging Face Datasets)
• AI Hub (National Information Society Agency, Korea) · Legal and regulatory text analysis data (AI Hub)