The Alignment Paradox: How Post-Training Amplifies Confident Hallucinations in Language Models

Qingjia Huang; Yakai Li; Jianguo Wu; Qihang Zhou; Aimin Yu; Xiaoqi Jia; Luping Ma; Weijuan Zhang

cs.CLSep 26, 2026

The Alignment Paradox: How Post-Training Amplifies Confident Hallucinations in Language Models

Authors: Qingjia Huang, Yakai Li, Jianguo Wu, Qihang Zhou, Aimin Yu, Xiaoqi Jia, Luping Ma, Weijuan Zhang

Organizations: Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China

Abstract

Large language models (LLMs) can produce factually incorrect answers with high confidence, undermining their reliability and limiting the effectiveness of uncertainty-based error detection. While prior research attributes confident hallucinations to factors such as missing knowledge in training data, reasoning errors, or stochastic decoding, we uncover that post-training alignment itself is a primary driver of these errors, a phenomenon we call the Alignment Paradox. Across five model families evaluated on factual benchmarks, unaligned base models produce few high-confidence errors on long-tail factual queries, whereas instruction-tuned models multiply high-confidence errors ( $p \ge 0.95$ ) by more than an order of magnitude (10 $\times$ to 35 $\times$ ). Layer-wise probing with the Logit Lens reveals that this overconfidence emerges in late layers, where wrong-answer margins expand past 4.0 points after remaining near zero across early and intermediate layers. These findings motivate limiting margin growth during post-training. We implement this principle through an entropy-dependent margin bound in direct preference optimization (DPO). In multi-epoch experiments with Mistral-7B, the bounded objective reduces high-confidence errors by up to 35.3% relative to standard DPO while maintaining performance on evaluated general reasoning benchmarks. These results show that bounded margins mitigate confident hallucinations during post-training.

Figures & tables

Figure 1: The Alignment Paradox. Post-training alignment shifts output probabilities into high-confidence regimes and multiplies near-certain errors on factual queries. (a) Sequence confidence distributions on factual queries ( Mistral-7B ). (b) Progression of high-confidence error rates (

p\geq 0.95

) and first-token decision entropy across alignment stages in the LLaMA-3.1-8B lineage (Base

\to

SFT

\to

DPO

\to

RLVR). (c) Layer-wise logit margin across normalized depth, showing logit margin expansion in late layers (

l/L\geq 0.85

, Mistral-7B ).

Model Family	Checkpoint	Acc (%)	Gate A	Gate B	Ratio	Gate C	Gate D	ECE
Mistral-7B	Base	52.3	2,378 (10.7%)	97 (0.4%)	14.1 $\times$	10,525 (47.3%)	9,256 (41.6%)	0.097
Mistral-7B	Instruct	41.6	4,133 (18.6%)	1,366 (6.1%)	14.1 $\times$	11,605 (52.1%)	5,125 (23.0%)	0.378
Qwen3-8B	Base	45.9	1,885 (8.5%)	50 (0.2%)	35.4 $\times$	11,986 (53.9%)	8,318 (37.4%)	0.042
Qwen3-8B	Instruct	37.9	5,140 (23.1%)	1,768 (7.9%)	35.4 $\times$	11,993 (53.9%)	3,308 (14.9%)	0.400
LLaMA-3.1-8B	Base	55.3	1,406 (6.3%)	38 (0.2%)	15.7 $\times$	9,909 (44.5%)	10,910 (49.0%)	0.018
LLaMA-3.1-8B	Instruct	45.6	4,996 (22.4%)	596 (2.7%)	15.7 $\times$	11,494 (51.6%)	5,157 (23.2%)	0.262

Table 1: Base versus aligned models across 22,267 queries. Post-training alignment multiplies high-confidence errors (Gate B) and degrades confidence reliability.

Panel of Figure 2 — Figure 2: Confidence reliability and distribution dynamics across 22,267 queries. (a) Reliability diagrams show post-training overconfidence gaps. (b) Sequence confidence distributions ( $\bar{p}$ ) reveal that unaligned base errors remain in the low-confidence regime, whereas post-training alignment drives thousands of errors into the near-certain regime ( $p\geq 0.95$ ). (c) Confident errors concentrate disproportionately in long-tail tiers ( $s_{\text{pop}}\in[10,10^{4})$ ).

Model Family	Evaluation Condition	Gate B Count	Gate B (%)	Inflation vs. Base
Mistral-7B	C1: Base Few-Shot	63	0.28%	1.0 $\times$
	C2: Instruct Chat Template	1,630	7.32%	25.9 $\times$
	C3: Instruct Bare Completion	864	3.88%	13.7 $\times$
LLaMA-3-8B	C1: Base Few-Shot	18	0.08%	1.0 $\times$
	C2: Instruct Chat Template	637	2.86%	35.4 $\times$
	C3: Instruct Bare Completion	108	0.48%	6.0 $\times$

Table 2: Causal disentanglement of chat templates versus model weights. Aligned models retain high error inflation even under bare completion prompts.

Panel of Figure 3 — Figure 3: Internal representation dynamics on high-confidence completions ( Mistral-7B , all cohorts conditioned on terminal $p\geq 0.95$ ). (a) Commitment depth ( $l^{*}$ ). Token predictions stabilize strictly in late layers ( $l^{*}/L\geq 0.85$ ), showing early commitment does not hold for factual errors. (b) Late-layer margin expansion. Logit margins hover near zero through intermediate layers before expanding sharply in late layers ( $l/L\geq 0.85$ ). (c) Target token suppression. Correct entity tokens drop deep into the vocabulary on confident factual errors ( $<4\%$ top-1 recall vs. $>94\%$ for correct completions).

Inference-Time Defense	Gate B Count	Gate B Reduction	Correct Retention	Effective Trade-off
Standard Instruct Baseline	324	0.0%	100.0%	Baseline
Defensive System Prompting	306	5.6%	65.2%	Severe over-hedging
Iterative Self-Correction	308	4.9%	81.4% (1.9% on LLaMA-3.1)	Corrupts valid answers
Self-Consistency (10 samples)	302	6.8%	97.4%	68.4% rigid errors
DoLa Layer-Contrastive	297	8.3%	97.1%	Late margin limits gain
Entropy Thresholding	308	4.8%	97.0%	Bypassed by overconfidence

Table 3: Inference-time mitigation on LLaMA-3-8B-Instruct. Interventions eliminate at most 8.3% of confident errors under

\geq 97\%

retention.

Starting Checkpoint	Optimization Objective	Acc (%)	Gate A	Gate B	$\Delta$ Gate B	Gate B / Errors	ECE	Reduction
Mistral-7B-SFT	Standard DPO	45.2	8,120	1,180	–	12.69%	0.224	Baseline
Mistral-7B-SFT	BDM-DPO (Ours)	45.4	8,204	1,118	-62	11.99%	0.218	-5.3%
Mistral-7B-Instruct	Standard DPO	47.6	8,973	1,630	–	15.37%	0.282	Baseline
Mistral-7B-Instruct	BDM-DPO (Ours)	47.6	9,012	1,591	-39	15.01%	0.279	-2.4%
LLaMA-3-8B-SFT	Standard DPO	48.2	9,450	102	–	1.07%	0.142	Baseline
LLaMA-3-8B-SFT	BDM-DPO (Ours)	48.3	9,510	91	-11	0.95%	0.138	-10.8%

Table 4: Factorial matrix across model families under single-epoch tuning. BDM-DPO safely suppresses early-stage overconfidence with zero degradation to baseline accuracy. (Single-epoch tuning precedes severe margin inflation; see Table 5 for multi-epoch stress tests).

Epoch	Acc (%)	Gate B	$M_{1}$	ECE	Acc (%)	Gate B	$M_{1}$	ECE	$\Delta$ Gate B	Reduction	$\Delta$ Acc
	Standard DPO				BDM-DPO (Ours)				Mitigation Effect
Epoch 1	47.6	1,630	3.61	0.282	47.6	1,591	3.60	0.279	-39	-2.4%	+0.0pp
Epoch 2	47.2	1,745	3.75	0.301	47.6	1,605	3.68	0.285	-140	-8.0%	+0.4pp
Epoch 3	46.8	1,890	3.92	0.324	47.5	1,612	3.71	0.291	-278	-14.7%	+0.7pp
Epoch 5	45.8	2,045	4.12	0.354	47.5	1,610	3.74	0.298	-435	-21.3%	+1.7pp
Epoch 8	43.8	2,409	4.38	0.391	47.6	1,559	3.78	0.332	-850	-35.3%	+3.8pp

Table 5: Multi-epoch dynamics on Mistral-7B across 10 DPO epochs. BDM-DPO constrains unconstrained margin growth, eliminating up to 850 confident errors (-35.3%) and preserving +4.7pp factual accuracy.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Model Family	HuggingFace Repository Identifier	Parameters	Layers	Vocabulary
Mistral-7B Base	mistralai/Mistral-7B-v0.3	7.25B	32	32,768
Mistral-7B Instruct	mistralai/Mistral-7B-Instruct-v0.3	7.25B	32	32,768
Qwen3-8B Base	Qwen/Qwen3-8B-Base	8.21B	36	151,936
Qwen3-8B Instruct	Qwen/Qwen3-8B	8.21B	36	151,936
LLaMA-3.1-8B Base	meta-llama/Meta-Llama-3.1-8B	8.03B	32	128,256
LLaMA-3.1-8B Instruct	meta-llama/Meta-Llama-3.1-8B-Instruct	8.03B	32	128,256

Appendix

Table 6: Evaluated Model Checkpoints and Architectural Specifications. All models are publicly accessible open-weight releases evaluated on their native tokenizers and context windows, matching the primary benchmark in Table 1 .

Audit Classification Category	Count ( $n$ )	Percentage (%)
Category 1: Genuine Factual Hallucination	493	98.6%
Category 2: Geographic / Granularity Ambiguity	3	0.6%
Category 3: Unlisted Colloquial Alias / Nickname	2	0.4%
Category 4: Syntactic Parsing Truncation	2	0.4%
Total Audited Sample	500	100.0%

Appendix

Table 7: Comprehensive Breakdown of the 500-Sample Double-Blind Human Verification Audit. Over 98.6% of candidate Gate B errors are confirmed as severe, unambiguous factual hallucinations. Less than 1.4% involve minor boundary or alias edge cases.

Epoch	Full Population Gate B (%)			Gate B Suppression (%)		Base Bias
Epoch	Base	Standard DPO	BDM-DPO	Pilot Subset	Full Population	$\Delta$ Gate B
1	7.25	7.32	7.15	16.0	2.4	-5.95pp
2	7.25	7.56	7.16	23.0	5.3	-5.95pp
3	7.25	9.19	7.18	34.9	21.9	-5.95pp
5	7.25	10.02	7.50	45.9	25.1	-5.95pp
8	7.25	10.82	7.00	50.0	35.3	-5.95pp

Appendix

Table 8: Empirical Quantification of Sampling Bias: Pilot Cohort (

n=2{,}000

) vs. Full Population (

N=22{,}267

). Side-by-side comparison of Gate B error rates and relative suppression rates across multi-epoch preference fine-tuning.

Model	Benchmark	Acc (%)	Gate A	Gate B	Gate C	Gate D	ECE	Brier
Mistral-7B Base	PopQA	28.5	3,010	42	8,895	562	0.045	0.182
Mistral-7B Base	TriviaQA	45.1	2,862	21	5,347	1,528	0.036	0.145
Mistral-7B Instruct	PopQA	38.2	4,520	1,180	6,559	250	0.312	0.245
Mistral-7B Instruct	TriviaQA	59.6	4,453	450	3,473	1,382	0.243	0.188
LLaMA-3-8B Base	PopQA	31.4	3,450	12	8,575	472	0.028	0.174
LLaMA-3-8B Base	TriviaQA	47.8	3,090	6	5,085	1,577	0.019	0.138

Appendix

Table 9: Extended Dataset Breakdown on PopQA and TriviaQA across Evaluated Model Lineages. Across every model family, instruction tuning triggers a multi-fold explosion of Gate B errors (up to 28-fold in Mistral-7B and 40-fold in LLaMA-3-8B on PopQA) alongside severe confidence reliability degradation.

Architecture	Stage	Checkpoint	Len. (Corr. / Wrong)	Gate A (%)	Gate B (%)	High-Conf. Err. ( $n$ )	First-Tok. Ent.	First-Tok. Marg.
Mistral-7B Lineage	Base	Mistral-7B-v0.1	3.0 / 3.2	12.96	0.84	186	3.148	1.586
	SFT	mistral-7b-sft-beta	9.6 / 12.5	0.36	0.06	13	1.981	1.466
	DPO	zephyr-7b-beta	29.3 / 40.9	14.97	2.99	655	0.698	3.847
LLaMA-3.1-8B Lineage	Base	Meta-Llama-3.1-8B	2.5 / 2.6	6.49	0.28	63	3.625	1.256
	SFT	Llama-3.1-Tulu-3-8B-SFT	3.1 / 3.3	13.25	1.16	258	2.818	1.465
	DPO	Llama-3.1-Tulu-3-8B-DPO	3.2 / 3.5	21.29	2.79	621	2.150	1.776

Appendix

Table 10: The Post-Training Causal Ladder: Base

\to

SFT

\to

DPO

\to

RLVR Across Open-Science Lineages. Evaluated on the full 22,267 queries (12,509 PopQA + 9,758 TriviaQA) under identical decoding protocols. Across both Mistral and LLaMA architectures, preference tuning (DPO) and rule-based reinforcement learning (RLVR) act as massive accelerators of confident hallucinations, expanding Gate B error volume by up to

13.7\times

. Length-invariant first-token entropy and margin confirm that probability sharpening occurs monotonically across post-training stages. ( mistral-7b-sft-beta ’s low Gate A/B stems from joint probability dilution over full sentences at

\bar{p}\geq 0.95

; at

\bar{p}\geq 0.80

its confident errors exceed Base, and first-token entropy already collapses by 37.1%.)

Model	Sampling Parameter	Value	Gate B Persist. (%)	Top-1 Agree (%)	Gold Elicitation (%)	Output Ent. ( $H_{K}$ )	AUROC (SC)
Mistral-7B-Inst	Temperature ( $T$ )	0.0 (Greedy)	100.0%	100.0%	0.0%	0.000	0.500
	Temperature ( $T$ )	0.2	94.2%	95.8%	1.2%	0.182	0.512
	Temperature ( $T$ )	0.5	82.6%	86.4%	3.8%	0.445	0.528
	Temperature ( $T$ )	0.7	68.4%	74.2%	7.1%	0.782	0.535
	Temperature ( $T$ )	1.0	44.1%	52.8%	12.4%	1.340	0.548
	Nucleus ( $p$ )	0.90 ( $T=0.7$ )	67.8%	73.9%	7.4%	0.791	0.534

Appendix

Table 11: Ablation of Stochastic Sampling Parameters on Confident Hallucination Stability. Evaluated on 1,000 randomly selected Gate B errors from PopQA and TriviaQA across Mistral-7B-Instruct and Qwen3-8B-Instruct. We sample

K=10

independent stochastic paths per prompt across sampling temperatures (

T

) and nucleus cutoffs (

p

). Gate B errors are remarkably stable across temperatures up to

T=0.7

, confirming that confident errors represent stable representational attractors rather than greedy decoding quirks.

Panel of Figure 4 — Figure 4: Response Rigidity and Architectural Scaling Dynamics. (a) Persistence of confident hallucinations under increasing temperature $T\in[0.0,1.2]$ . Even at $T=0.7$ , over 68%–72% of error trajectories persistently regenerate the identical hallucinated entity. (b) Gate B error rates across model parameter scales (7B to 14B). Scaling parameters alone does not suppress the Alignment Paradox; larger models exhibit equal or heightened susceptibility to confident hallucinations under preference optimization.

Model	Outcome	Sample Size ( $n$ )	Total Layers ( $L$ )	Mean $l^{*}$	Mean $l^{*}/L$	Final Top-1 Prob.
Mistral-7B (Instruct)	Wrong	500	32	28.36	0.886	0.948
Mistral-7B (Instruct)	Correct	500	32	27.26	0.852	0.971
Mistral-7B (Base)	Wrong	97	32	27.94	0.873	0.892
Mistral-7B (Base)	Correct	500	32	27.19	0.850	0.935
Qwen-8B (Instruct)	Wrong	500	36	35.34	0.982	0.961
Qwen-8B (Instruct)	Correct	500	36	35.35	0.982	0.984

Appendix

Table 12: Layer-wise Logit Lens Commitment Statistics. Probing 500 confident errors (passing Gate A+B) and 500 confident correct answers per model. Delineating the empirical scope of early commitment, both base and instruct models commit to their final token in late layers (

l^{*}/L\geq 0.85

). Within each arm, errors commit slightly later than correct answers. The primary distinction lies in mid-layer entropy (

d=2.72

at layer 16), not the layer index of commitment.

Figure 5: Layer-wise Dynamics of Commitment and Logit Margins. Tracking top-1 accuracy, logit margin (

z_{\text{top1}}-z_{\text{top2}}

), and layer entropy across all layers. Both base and instruct models resolve their top-1 prediction late in the network (

l^{*}/L\geq 0.85

). However, instruct models undergo a severe mid-layer entropy collapse, which transitions into a steep logit margin surge in the final 2–4 layers ( Late-Layer Margin Expansion ).

Model	Dataset	Sample Size ( $n$ )	R@1	R@2	R@5	R@10	R@100	Median Rank
Mistral-7B	PopQA (Long-tail)	320	0.019	0.100	0.209	0.309	0.653	42
Mistral-7B	TriviaQA (Head)	180	0.072	0.244	0.461	0.622	0.917	6
Qwen-8B	PopQA (Long-tail)	310	0.016	0.126	0.281	0.432	0.810	16
Qwen-8B	TriviaQA (Head)	190	0.116	0.316	0.558	0.716	0.937	5

Appendix

Table 13: Ground-Truth Token Rank Distribution across Knowledge Popularity. Comparing PopQA (long-tail facts) and TriviaQA (common/head facts) on 500 confident errors and correct controls. True entity ranks are evaluated by taking the minimal rank across all morphological and whitespace tokenizations.

Figure 6: Ground-Truth Latency Profiling across Knowledge Popularity. Recall@

k

curves for ground-truth entity tokens under confident hallucinations (

p\geq 0.95

) versus confident correct predictions. In head knowledge (TriviaQA), ground truth is recovered within Top-5 for

>50\%

of queries (median rank 5–6). In long-tail knowledge (PopQA), ground truth is suppressed deep in the vocabulary (median rank 16–42), with over 34% of entities completely absent from the top-100 vocabulary distribution.

Model Arm	Layer 8 AUROC	Layer 16 AUROC	Layer 24 AUROC	Layer 31 AUROC
Mistral-7B Base	$0.739\pm 0.017$	$0.753\pm 0.014$	$0.781\pm 0.015$	$0.809\pm 0.010$
Standard DPO (Epoch 10)	$0.735\pm 0.012$	$0.774\pm 0.007$	$0.786\pm 0.012$	$0.804\pm 0.012$
BDM-DPO (Epoch 10, Ours)	$0.728\pm 0.009$	0.779 $\pm$ 0.011	0.807 $\pm$ 0.014	0.823 $\pm$ 0.009

Appendix

Table 14: Hidden Representation Geometry: Linear Probing across Intermediate Layers. Training

L_{2}

-regularized logistic regression probes across layers

L\in\{8,16,24,31\}

on 4,000 balanced instances (2,000 correct vs. 2,000 confident errors). Evaluated via 5-fold cross-validation. Middle layers (

L=16,24

) sustain high AUROC (

\sim 0.77

–

0.81

) across all models, proving that factual truthfulness remains linearly separable in hidden state representations; distortion is localized to terminal logit divergence.

Model	Layer	Mode	$\alpha$	Recovery (%)	Retention (%)	Net Gain	Gold Rank Shift
Mistral-7B	31 ( $L-1$ )	Raw	0.2	0.0	99.0	$-2$	16.0 $\to$ 16.0
	31 ( $L-1$ )	Normalized	0.5	1.5	97.0	$-3$	16.0 $\to$ 21.5
	30 ( $L-2$ )	Normalized	0.5	3.5	88.5 †	$-16$	16.0 $\to$ 23.0
	20 (Mid)	Normalized	0.5	11.0	92.0 †	$+6$	16.0 $\to$ 20.0
Qwen-8B	35 ( $L-1$ )	Normalized	0.2	2.0	100.0	$+4$	7.0 $\to$ 7.5
	34 ( $L-2$ )	Normalized	0.5	4.0	55.5 †	$-81$	7.0 $\to$ 11.0

Appendix

Table 15: Empirical Diagnostic of Layer Contrastive Decoding (CD-Layer). Benchmarking 24 configurations on 200 confident errors and 200 confident correct answers under the pre-registered constraint: Retention

\geq 97\%

. Net Gain = Recovered Errors

-

Lost Correct. Across all layers and scaling modes, CD-Layer fails to produce meaningful net gains, exhibiting severe retention degradation as

\alpha

scales.

Model	Detector / Probe	AUROC	AUPR	FPR@95TPR
Mistral-7B (Instruct)	Softmax Max Prob. ( $\bar{p}$ )	0.661	0.641	0.852
	Final Layer Entropy	0.686	0.676	0.850
	Self-Consistency ( $K=10$ )	0.535	0.981	0.082
	Normalized Commit Layer ( $l^{*}/L$ )	0.608	0.665	0.902
	Final Logit Margin	0.609	0.591	0.874
	Margin Rise ( $M_{24:32}-M_{8:16}$ )	0.622	0.603	0.888

Appendix

Table 16: Detection Benchmark in the High-Confidence Regime (

\bar{p}\geq 0.95

). Evaluated on 500 confident errors (passing Gate A+B) and 500 confident correct answers per model. AUROC indicates ability to detect errors (higher is better). Self-consistency collapses to random guessing (

\approx 0.51

) due to rigid fabricated outputs. Final-layer micro-entropy serves as the strongest epistemic probe, achieving AUROC 0.686–0.732.

Model	Abstention Criterion	AURC $\downarrow$	Acc@100%	Acc@80%	Acc@70%
Mistral-7B (Deployment $n=5{,}537$ )	Random Rejection	0.508	74.6	74.6	74.6
	Sequence Confidence ( $1-\bar{p}$ )	0.364	74.6	77.7	79.1
	Token Confidence ( $1-p_{t_{1}}$ )	0.368	74.6	77.5	78.8
	Final-Layer Micro-Entropy ( $H_{L}$ )	0.355	74.6	77.7	79.1
Qwen-8B (Deployment $n=6{,}965$ )	Random Rejection	0.495	73.8	73.8	73.8
	Sequence Confidence ( $1-\bar{p}$ )	0.309	73.8	79.3	82.0

Appendix

Table 17: Selective Abstention under Realistic High-Confidence Deployment (

\bar{p}\geq 0.95

). Evaluated across all 5,537 high-confidence predictions in Mistral-7B and 6,965 in Qwen-8B. While selective rejection via micro-entropy (

H_{L}

) or sequence confidence outperforms random rejection, all post-hoc signals hit an epistemic ceiling, improving accuracy by only +3.1pp to +5.5pp at 80% coverage.

Figure 7: Risk-Coverage Trade-Off Curves for Selective Abstention under Alignment. Empirical selective risk (error rate) as a function of coverage on deployment queries (

n=5{,}537

for Mistral-7B,

n=6{,}965

for Qwen-8B). Confidence-based filtering (

1-\bar{p}

) and micro-entropy (

H_{L}

) achieve only marginal risk reductions (+3.1pp to +5.5pp accuracy gain at 80% coverage), hitting a strict epistemic ceiling. Because alignment forces error tokens into near-zero entropy states, post-hoc thresholding cannot distinguish between true and fabricated memories.

Model	Method	Paradigm	HCE Elim. (%)	Abstain / Hedge (%)	Retention (%)	Corrupted ( $n$ )	Net Gain
Mistral-7B-Inst	Unconstrained Greedy (Control)	Baseline	3.7%	0.8%	99.8%	1	+21
	Hedging Prompt ( Kadavath et al., 2022 )	Prompt	7.8%	3.2%	97.2%	17	+30
	Self-Correction ( Huang et al., 2024 )	2-Turn Critic	6.8%	3.0%	99.2%	5	+36
	Self-Consistency ( $K=5$ ) ( Wang et al., 2023 )	Sampling	6.3%	5.0%	99.0%	6	+32
	DoLa ( $L-2$ ) ( Chuang et al., 2024 )	Decoding	6.2%	1.2%	99.5%	3	+34
	ET-Abstention (Raw $\tau=[1.5,1.2]$ )	Epistemic Gate	42.0%	40.3%	83.5% *	99	+153

Appendix

Table 18: Comprehensive Inference-Time Mitigation Benchmark across Four Aligned Model Families. Evaluated on 4,422 balanced instances (1:1 matched Gate B confident errors vs. Gate A confident correct controls). Pre-registered safety constraint:

\text{Retention}\geq 97.0\%

(marked with * if violated). Net Gain = Eliminated Confident Errors

-

Corrupted Correct Answers. Literature baselines and uncalibrated internal gating either hit a strict 5%–8% elimination ceiling or trigger severe destruction of correct knowledge (e.g., Tulu-3 Self-Correction retention collapsing to 1.9%, Qwen-8B Hedging retention falling to 65.2%).

Tier	Configuration	$M_{1}$	$H_{1}$	Gate B (%)	#Gate B	Acc. (%)	ECE
Reference	Unaligned Base (Mistral-7B)	3.252	0.863	7.00%	140	46.8%	0.3322
Unconstrained	Standard DPO (1 Epoch Control)	3.584	0.731	7.65%	153	46.1%	0.3603
Static Clamp	$M\equiv M_{0}$	3.332	0.829	7.10%	142	46.5%	0.3430
Linear Decay	$M_{0}\max(0,1-\gamma H)$	3.320	0.837	6.95%	139	46.8%	0.3384
Dynamic Grid	$M_{0}=2.0,\gamma=1.0$	3.315	0.839	6.95%	139	46.9%	0.3367
	$M_{0}=2.0,\gamma=2.0$	3.306	0.838	7.00%	140	46.8%	0.3380

Appendix

Table 19: Comprehensive Ablation of BDM-DPO Formulations and Hyperparameter Sensitivity (Pilot

n=2{,}000

). Evaluating 13 controlled configurations on a balanced cohort. Comparing unconstrained Standard DPO against static clamping, linear decay, and a

3\times 3

dynamic BDM grid. Bounding the preference margin yields consistent Gate B error reductions across all configurations.

Paradigm	Objective / Method	Ref. Policy	$M_{1}$	$H_{1}$	Gate B (%)	#Gate B	Acc. (%)	ECE
Reference Checkpoint	Mistral-7B-Instruct-v0.3	None	3.609	0.907	7.25%	1,615	47.8%	0.3569
Reference-Free	Vanilla SimPO ( $\gamma=1.0$ )	None	3.799	0.986	7.58%	1,687	45.4%	0.3896
Reference-Based	Standard DPO (Unbounded)	$\pi_{\text{ref}}$	3.916	0.796	7.37%	1,641	47.1%	0.3777
Epistemic Bounded	BDM-DPO (Ours)	$\pi_{\text{ref}}$	3.698	0.876	7.10%	1,580	47.7%	0.3608

Appendix

Table 20: Cross-Paradigm Preference Alignment Benchmark on the Full 22,267-Query Corpus. Comparing the pre-DPO reference checkpoint ( Mistral-7B-Instruct-v0.3 ) against reference-free preference optimization ( Vanilla SimPO ; Meng et al. 2024 ), standard unconstrained reference-based optimization ( Standard DPO ; Rafailov et al. 2023 ), and our epistemic-bounded method ( BDM-DPO ). All models are trained under identical 1-epoch conditions on 3,000 preference pairs and evaluated across all 22,267 questions under clean zero-shot protocol. Reference-free SimPO suffers the most severe overconfidence and accuracy degradation, while BDM-DPO achieves the superior Pareto frontier.

$\beta$	Method	$M_{1}$	$H_{1}$	Gate B (%)	#Gate B	Acc. (%)	ECE
0.05	Standard DPO	4.071	0.743	7.64%	1,701	46.8%	0.3889
0.05	BDM-DPO (Ours)	3.687	0.877	7.18%	1,599	47.6%	0.3621
0.10	Standard DPO	3.902	0.798	7.32%	1,630	47.2%	0.3771
0.10	BDM-DPO (Ours)	3.691	0.878	7.15%	1,591	47.6%	0.3617
0.20	Standard DPO	3.785	0.840	7.16%	1,595	47.4%	0.3699
0.20	BDM-DPO (Ours)	3.686	0.878	7.15%	1,593	47.6%	0.3620

Appendix

Table 21: Robustness to the DPO Regularization Weight

\beta

Across the Full 22,267-Query Corpus. Evaluated under identical reference model ( Mistral-7B-Instruct-v0.3 ), preference dataset (UltraFeedback 3K), and zero-shot protocol across 1 epoch. Standard DPO exhibits monotonic deterioration in margin

M_{1}

, Gate B confident hallucinations, and ECE as

\beta

decreases. In contrast, BDM-DPO remains invariant across

\beta

values, establishing a robust epistemic safety floor.

Preference Dataset	Optimization Method	$M_{1}$	$H_{1}$	Gate B (%)	#Gate B	Acc. (%)	ECE
UltraFeedback ( Cui et al., 2024 )	Standard DPO	3.902	0.798	7.32%	1,630	47.2%	0.3771
UltraFeedback ( Cui et al., 2024 )	BDM-DPO (Ours)	3.691	0.878	7.15%	1,591	47.6%	0.3617
Anthropic HH-RLHF ( Bai et al., 2022 )	Standard DPO	3.665	0.867	7.72%	1,718	47.5%	0.3623
Anthropic HH-RLHF ( Bai et al., 2022 )	BDM-DPO (Ours)	3.640	0.889	7.31%	1,627	47.8%	0.3584

Appendix

Table 22: Robustness to the Post-Training Preference Dataset (UltraFeedback vs. Anthropic HH-RLHF). Evaluated under identical architecture ( Mistral-7B-Instruct-v0.3 ), hyperparameters (

\beta=0.10

M_{0}=3.0

\gamma=2.0

), optimizer, and unified zero-shot evaluation across all 22,267 queries. The preference dataset is the sole independent variable. BDM-DPO consistently suppresses Gate B confident hallucinations across distinct preference corpora.

Architecture	Epoch	Method	$M_{1}$	$H_{1}$	Gate B (%)	#Gate B	Acc. (%)
Mistral-7B-Inst	Epoch 1	Standard DPO	3.916	0.796	7.37%	1,641	47.1%
	Epoch 1	BDM-DPO	3.698	0.876	7.10%	1,580	47.7%
	Epoch 10	Standard DPO	4.514	0.521	9.60%	2,138	42.9%
	Epoch 10	BDM-DPO	3.784	0.812	6.65%	1,481	47.6%
Llama-3.1-Tulu-3	Epoch 1	Standard DPO	1.766	1.482	0.46%	102	54.2%
	Epoch 1	BDM-DPO	1.748	1.512	0.41%	91	54.4%

Appendix

Table 23: Cross-Architecture Multi-Epoch Progression (Unified Zero-Shot Protocol,

N=22{,}267

). Comparing 1-epoch vs. 2-epoch preference optimization across distinct architectures. On LLaMA SFT, Standard DPO inflates

M_{1}

(

1.766\to 1.855

) and escalates Gate B errors (+29.4%,

102\to 132

), while BDM-DPO pins margin (1.750) and Gate B (94), expanding the BDM advantage from ratio

0.892\to 0.712

(-28.8% error reduction).

Figure 8: Longitudinal Multi-Epoch Dynamics on Full Population (

N=22{,}267

): Standard DPO Margin Expansion vs. BDM-DPO Stability. Left: First-token decision logit margin

M_{1}

continuously explodes under Standard DPO (

3.609\to 4.514

), whereas BDM-DPO acts as an epistemic circuit breaker, clamping the margin at

M_{1}\leq 3.80

. Right: Unconstrained margin push under Standard DPO inflates Gate B errors (+49.2%), while BDM-DPO eliminates up to 850 confident hallucinations.

Detection / Abstention Paradigm	Evaluation Metric	Reference SFT	Standard DPO	BDM-DPO (Ours)
Sampling Self-Consistency ( $K=5$ ) ( Wang et al., 2023 )	AUROC $\uparrow$	0.535	0.508 (Blinded)	0.658 (Revitalized)
Sequence Confidence ( $1-\bar{p}$ ) ( Geifman & El-Yaniv, 2017 )	AUROC $\uparrow$	0.612	0.564 (Degraded)	0.714 (Sharpened)
Final-Layer Micro-Entropy ( $H_{L}$ ) (Ours)	AUROC $\uparrow$	0.686	0.615 (Compressed)	0.781 (Restored)
Safe Selective Abstention ( $\text{Retention}\geq 97.0\%$ )	Net Gain $\uparrow$	+21	+8 (Collateral Loss)	+54 (Clean Rejection)

Appendix

Table 24: Revitalizing Post-Hoc Detectability: Standard DPO Blinding vs. BDM-DPO Restoration. Evaluating detector separability in the high-confidence regime (

\bar{p}\geq 0.95

, PopQA and TriviaQA). BDM-DPO restores the discriminative capability of literature detectors with zero knowledge corruption.

Model Variant	MMLU (5-shot)	GSM8K (8-shot CoT)	ARC-C (25-shot)	McNemar $p$ -value
Mistral-7B Base	62.69%	49.51%	63.65%	(Baseline)
Standard DPO (Epoch 10)	61.76%	49.51%	63.65%	Degradation ( $p<10^{-4}$ )
BDM-DPO (Epoch 10)	62.70%	51.55%	63.65%	Preserved ( $p=0.63$ )

Appendix

Table 25: Downstream benchmark performance across general reasoning tasks. BDM-DPO eliminates confident errors while preserving general reasoning on MMLU (5-shot), GSM8K (8-shot CoT), and ARC-Challenge (25-shot).

Question	Ground Truth ( $\mathcal{Y}^{*}$ )	Model Output ( $\hat{y}$ )	$\bar{p}(\hat{y})$	$\mathcal{H}(P_{1})$	$M_{1}$	Gold Rank (L16 $\to$ L-1)	Failure Mechanism
Domain: Entertainment & Cinema (Mistral-7B-Instruct, PopQA)
Who directed the movie Star Wars?	George Lucas	Guillermo del Toro	0.992	0.038	4.85	Rank 3 $\to$ Rank 42	Mid-layer competition; final-layer suppression.
Domain: World History (Qwen3-8B-Instruct, TriviaQA)
In which country was the Battle of Waterloo fought?	Belgium	France	0.989	0.045	4.62	Rank 2 $\to$ Rank 18	Strong semantic prior; gold token suppressed at L27.
Domain: Literature (Qwen3-8B-Instruct, PopQA)
Who wrote the novel “The Spy Who Came in from the Cold”?	John le Carré	Ian Fleming	0.995	0.021	5.21	Rank 5 $\to$ Rank 64	Genre author confusion; rapid late-layer margin surge.

Appendix

Table 26: Qualitative Case Studies of Confident Hallucinations across Diverse Knowledge Domains. Tracing model confidence

\bar{p}

, first-token decision metrics (

\mathcal{H}(P_{1})

M_{1}

), and layer-wise ground-truth token latency. Across all domains, the model commits to a false attractor with near-certainty (

p\geq 0.97

), while the true answer is suppressed in late layers.

Hallucination in Language Models

Hallucination in language models studies how systems produce unsupported or false claims and how to detect, explain, and reduce them.

See all 247 papers

Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection
Feb 18, 2026Xie Xiaohu, Liu Xiaohu, Yao BenjaminLanguage Model Error Detection Confidence Estimation in Language Models
Alexa AI, Amazon, Seattle, WA, USA
Anchored Confabulation: Partial Evidence Non-Monotonically Amplifies Confident Hallucination in LLMs
Apr 2, 2026Ashish Balkishan LathkarHallucination in Language Models Confidence Estimation in Language Models
M.S. Data Science, Florida State University
Emergent Unfaithfulness: How Alignment Training Causes Language Models to Silently Override Task Faithfulness
Sep 30, 2026Pardis Sadat Zahraei, Janvijay Singh, Gokhan Tur +1LLM Alignment Instruction Following
University of Illinois Urbana-Champaign

Feb 18, 2026J/K move · Enter open · S save

Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection

Xie Xiaohu, Liu Xiaohu, Yao Benjamin

Alexa AI, Amazon, Seattle, WA, USA

Language Model Error Detection Confidence Estimation in Language Models LLM Reliability Language Model Calibration

Open paper AskPDF arXiv

As large language models (LLMs) are increasingly deployed in critical decision-making systems, the lack of reliable methods to measure their uncertainty presents a fundamental trustworthiness risk. We introduce a normalized confidence score based on output anchor token probabilities: classification labels for structured tasks and self-evaluation responses (Yes/No) for open-ended generation. This enables direct detection of errors and hallucinations with minimal overhead and without external validation. We make three key contributions. First, we propose a normalized confidence score and self-evaluation framework that exposes reliable confidence estimates for error detection across seven diverse benchmark tasks and five LLMs of varying architectures and sizes. Second, our theoretical analysis reveals that supervised fine-tuning (SFT) yields well-calibrated confidence through maximum-likelihood estimation, whereas reinforcement learning methods (PPO, GRPO) and DPO induce overconfidence via reward exploitation. Third, we propose post-RL SFT with self-distillation to restore confidence reliability in RL-trained models. Empirical results demonstrated that SFT improved average confidence-correctness AUROC from 0.806 to 0.879 and reduced calibration error from 0.163 to 0.034 on Qwen3-4B, while GRPO and DPO degraded confidence reliability. We demonstrated practical value through adaptive retrieval-augmented generation (RAG) that selectively retrieves context when the model lacks confidence, using only 58% of retrieval operations to recover 95% of the maximum achievable accuracy gain on TriviaQA

Abstract

Hallucination in Language Models

Explore similar work

Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection

Anchored Confabulation: Partial Evidence Non-Monotonically Amplifies Confident Hallucination in LLMs

Emergent Unfaithfulness: How Alignment Training Causes Language Models to Silently Override Task Faithfulness