Organizations: Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
Large language models (LLMs) can produce factually incorrect answers with high confidence, undermining their reliability and limiting the effectiveness of uncertainty-based error detection. While prior research attributes confident hallucinations to factors such as missing knowledge in training data, reasoning errors, or stochastic decoding, we uncover that post-training alignment itself is a primary driver of these errors, a phenomenon we call the Alignment Paradox. Across five model families evaluated on factual benchmarks, unaligned base models produce few high-confidence errors on long-tail factual queries, whereas instruction-tuned models multiply high-confidence errors (p≥0.95) by more than an order of magnitude (10× to 35×). Layer-wise probing with the Logit Lens reveals that this overconfidence emerges in late layers, where wrong-answer margins expand past 4.0 points after remaining near zero across early and intermediate layers. These findings motivate limiting margin growth during post-training. We implement this principle through an entropy-dependent margin bound in direct preference optimization (DPO). In multi-epoch experiments with Mistral-7B, the bounded objective reduces high-confidence errors by up to 35.3% relative to standard DPO while maintaining performance on evaluated general reasoning benchmarks. These results show that bounded margins mitigate confident hallucinations during post-training.
Figures & tables
Figure 1: The Alignment Paradox. Post-training alignment shifts output probabilities into high-confidence regimes and multiplies near-certain errors on factual queries. (a) Sequence confidence distributions on factual queries ( Mistral-7B ). (b) Progression of high-confidence error rates ( p≥0.95 ) and first-token decision entropy across alignment stages in the LLaMA-3.1-8B lineage (Base → SFT → DPO → RLVR). (c) Layer-wise logit margin across normalized depth, showing logit margin expansion in late layers ( l/L≥0.85 , Mistral-7B ).
Model Family
Checkpoint
Acc (%)
Gate A
Gate B
Ratio
Gate C
Gate D
ECE
Mistral-7B
Base
52.3
2,378 (10.7%)
97 (0.4%)
14.1 ×
10,525 (47.3%)
9,256 (41.6%)
0.097
Instruct
41.6
4,133 (18.6%)
1,366 (6.1%)
11,605 (52.1%)
5,125 (23.0%)
0.378
Qwen3-8B
Base
45.9
1,885 (8.5%)
50 (0.2%)
35.4 ×
11,986 (53.9%)
8,318 (37.4%)
0.042
Instruct
37.9
5,140 (23.1%)
1,768 (7.9%)
11,993 (53.9%)
3,308 (14.9%)
0.400
LLaMA-3.1-8B
Base
55.3
1,406 (6.3%)
38 (0.2%)
15.7 ×
9,909 (44.5%)
10,910 (49.0%)
0.018
Instruct
45.6
4,996 (22.4%)
596 (2.7%)
11,494 (51.6%)
5,157 (23.2%)
0.262
Table 1: Base versus aligned models across 22,267 queries. Post-training alignment multiplies high-confidence errors (Gate B) and degrades confidence reliability.
Figure 2: Confidence reliability and distribution dynamics across 22,267 queries. (a) Reliability diagrams show post-training overconfidence gaps. (b) Sequence confidence distributions ( pˉ ) reveal that unaligned base errors remain in the low-confidence regime, whereas post-training alignment drives thousands of errors into the near-certain regime ( p≥0.95 ). (c) Confident errors concentrate disproportionately in long-tail tiers ( spop∈[10,104) ).
Model Family
Evaluation Condition
Gate B Count
Gate B (%)
Inflation vs. Base
Mistral-7B
C1: Base Few-Shot
63
0.28%
1.0 ×
C2: Instruct Chat Template
1,630
7.32%
25.9 ×
C3: Instruct Bare Completion
864
3.88%
13.7 ×
LLaMA-3-8B
C1: Base Few-Shot
18
0.08%
1.0 ×
C2: Instruct Chat Template
637
2.86%
35.4 ×
C3: Instruct Bare Completion
108
0.48%
6.0 ×
Table 2: Causal disentanglement of chat templates versus model weights. Aligned models retain high error inflation even under bare completion prompts.
Figure 3: Internal representation dynamics on high-confidence completions ( Mistral-7B , all cohorts conditioned on terminal p≥0.95 ). (a) Commitment depth ( l∗ ). Token predictions stabilize strictly in late layers ( l∗/L≥0.85 ), showing early commitment does not hold for factual errors. (b) Late-layer margin expansion. Logit margins hover near zero through intermediate layers before expanding sharply in late layers ( l/L≥0.85 ). (c) Target token suppression. Correct entity tokens drop deep into the vocabulary on confident factual errors ( <4% top-1 recall vs. >94% for correct completions).
Inference-Time Defense
Gate B Count
Gate B Reduction
Correct Retention
Effective Trade-off
Standard Instruct Baseline
324
0.0%
100.0%
Baseline
Defensive System Prompting
306
5.6%
65.2%
Severe over-hedging
Iterative Self-Correction
308
4.9%
81.4% (1.9% on LLaMA-3.1)
Corrupts valid answers
Self-Consistency (10 samples)
302
6.8%
97.4%
68.4% rigid errors
DoLa Layer-Contrastive
297
8.3%
97.1%
Late margin limits gain
Entropy Thresholding
308
4.8%
97.0%
Bypassed by overconfidence
Table 3: Inference-time mitigation on LLaMA-3-8B-Instruct. Interventions eliminate at most 8.3% of confident errors under ≥97% retention.
Starting Checkpoint
Optimization Objective
Acc (%)
Gate A
Gate B
Δ Gate B
Gate B / Errors
ECE
Reduction
Mistral-7B-SFT
Standard DPO
45.2
8,120
1,180
–
12.69%
0.224
Baseline
BDM-DPO (Ours)
45.4
8,204
1,118
-62
11.99%
0.218
-5.3%
Mistral-7B-Instruct
Standard DPO
47.6
8,973
1,630
–
15.37%
0.282
Baseline
BDM-DPO (Ours)
47.6
9,012
1,591
-39
15.01%
0.279
-2.4%
LLaMA-3-8B-SFT
Standard DPO
48.2
9,450
102
–
1.07%
0.142
Baseline
BDM-DPO (Ours)
48.3
9,510
91
-11
0.95%
0.138
-10.8%
Table 4: Factorial matrix across model families under single-epoch tuning. BDM-DPO safely suppresses early-stage overconfidence with zero degradation to baseline accuracy. (Single-epoch tuning precedes severe margin inflation; see Table 5 for multi-epoch stress tests).
Standard DPO
BDM-DPO (Ours)
Mitigation Effect
Epoch
Acc (%)
Gate B
M1
ECE
Acc (%)
Gate B
M1
ECE
Δ Gate B
Reduction
Δ Acc
Epoch 1
47.6
1,630
3.61
0.282
47.6
1,591
3.60
0.279
-39
-2.4%
+0.0pp
Epoch 2
47.2
1,745
3.75
0.301
47.6
1,605
3.68
0.285
-140
-8.0%
+0.4pp
Epoch 3
46.8
1,890
3.92
0.324
47.5
1,612
3.71
0.291
-278
-14.7%
+0.7pp
Epoch 5
45.8
2,045
4.12
0.354
47.5
1,610
3.74
0.298
-435
-21.3%
+1.7pp
Epoch 8
43.8
2,409
4.38
0.391
47.6
1,559
3.78
0.332
-850
-35.3%
+3.8pp
Table 5: Multi-epoch dynamics on Mistral-7B across 10 DPO epochs. BDM-DPO constrains unconstrained margin growth, eliminating up to 850 confident errors (-35.3%) and preserving +4.7pp factual accuracy.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Model Family
HuggingFace Repository Identifier
Parameters
Layers
Vocabulary
Mistral-7B Base
mistralai/Mistral-7B-v0.3
7.25B
32
32,768
Mistral-7B Instruct
mistralai/Mistral-7B-Instruct-v0.3
7.25B
32
32,768
Qwen3-8B Base
Qwen/Qwen3-8B-Base
8.21B
36
151,936
Qwen3-8B Instruct
Qwen/Qwen3-8B
8.21B
36
151,936
LLaMA-3.1-8B Base
meta-llama/Meta-Llama-3.1-8B
8.03B
32
128,256
LLaMA-3.1-8B Instruct
meta-llama/Meta-Llama-3.1-8B-Instruct
8.03B
32
128,256
Appendix
Table 6: Evaluated Model Checkpoints and Architectural Specifications. All models are publicly accessible open-weight releases evaluated on their native tokenizers and context windows, matching the primary benchmark in Table 1 .
Audit Classification Category
Count ( n )
Percentage (%)
Category 1: Genuine Factual Hallucination
493
98.6%
Category 2: Geographic / Granularity Ambiguity
3
0.6%
Category 3: Unlisted Colloquial Alias / Nickname
2
0.4%
Category 4: Syntactic Parsing Truncation
2
0.4%
Total Audited Sample
500
100.0%
Appendix
Table 7: Comprehensive Breakdown of the 500-Sample Double-Blind Human Verification Audit. Over 98.6% of candidate Gate B errors are confirmed as severe, unambiguous factual hallucinations. Less than 1.4% involve minor boundary or alias edge cases.
Epoch
Full Population Gate B (%)
Gate B Suppression (%)
Base Bias
Base
Standard DPO
BDM-DPO
Pilot Subset
Full Population
Δ Gate B
1
7.25
7.32
7.15
16.0
2.4
-5.95pp
2
7.25
7.56
7.16
23.0
5.3
-5.95pp
3
7.25
9.19
7.18
34.9
21.9
-5.95pp
5
7.25
10.02
7.50
45.9
25.1
-5.95pp
8
7.25
10.82
7.00
50.0
35.3
-5.95pp
Appendix
Table 8: Empirical Quantification of Sampling Bias: Pilot Cohort ( n=2,000 ) vs. Full Population ( N=22,267 ). Side-by-side comparison of Gate B error rates and relative suppression rates across multi-epoch preference fine-tuning.
Model
Benchmark
Acc (%)
Gate A
Gate B
Gate C
Gate D
ECE
Brier
Mistral-7B Base
PopQA
28.5
3,010
42
8,895
562
0.045
0.182
TriviaQA
45.1
2,862
21
5,347
1,528
0.036
0.145
Mistral-7B Instruct
PopQA
38.2
4,520
1,180
6,559
250
0.312
0.245
TriviaQA
59.6
4,453
450
3,473
1,382
0.243
0.188
LLaMA-3-8B Base
PopQA
31.4
3,450
12
8,575
472
0.028
0.174
TriviaQA
47.8
3,090
6
5,085
1,577
0.019
0.138
Appendix
Table 9: Extended Dataset Breakdown on PopQA and TriviaQA across Evaluated Model Lineages. Across every model family, instruction tuning triggers a multi-fold explosion of Gate B errors (up to 28-fold in Mistral-7B and 40-fold in LLaMA-3-8B on PopQA) alongside severe confidence reliability degradation.
Architecture
Stage
Checkpoint
Len. (Corr. / Wrong)
Gate A (%)
Gate B (%)
High-Conf. Err. ( n )
First-Tok. Ent.
First-Tok. Marg.
Mistral-7B Lineage
Base
Mistral-7B-v0.1
3.0 / 3.2
12.96
0.84
186
3.148
1.586
SFT
mistral-7b-sft-beta
9.6 / 12.5
0.36
0.06
13
1.981
1.466
DPO
zephyr-7b-beta
29.3 / 40.9
14.97
2.99
655
0.698
3.847
LLaMA-3.1-8B Lineage
Base
Meta-Llama-3.1-8B
2.5 / 2.6
6.49
0.28
63
3.625
1.256
SFT
Llama-3.1-Tulu-3-8B-SFT
3.1 / 3.3
13.25
1.16
258
2.818
1.465
DPO
Llama-3.1-Tulu-3-8B-DPO
3.2 / 3.5
21.29
2.79
621
2.150
1.776
Appendix
Table 10: The Post-Training Causal Ladder: Base → SFT → DPO → RLVR Across Open-Science Lineages. Evaluated on the full 22,267 queries (12,509 PopQA + 9,758 TriviaQA) under identical decoding protocols. Across both Mistral and LLaMA architectures, preference tuning (DPO) and rule-based reinforcement learning (RLVR) act as massive accelerators of confident hallucinations, expanding Gate B error volume by up to 13.7× . Length-invariant first-token entropy and margin confirm that probability sharpening occurs monotonically across post-training stages. ( mistral-7b-sft-beta ’s low Gate A/B stems from joint probability dilution over full sentences at pˉ≥0.95 ; at pˉ≥0.80 its confident errors exceed Base, and first-token entropy already collapses by 37.1%.)
Model
Sampling Parameter
Value
Gate B Persist. (%)
Top-1 Agree (%)
Gold Elicitation (%)
Output Ent. ( HK )
AUROC (SC)
Mistral-7B-Inst
Temperature ( T )
0.0 (Greedy)
100.0%
100.0%
0.0%
0.000
0.500
Temperature ( T )
0.2
94.2%
95.8%
1.2%
0.182
0.512
Temperature ( T )
0.5
82.6%
86.4%
3.8%
0.445
0.528
Temperature ( T )
0.7
68.4%
74.2%
7.1%
0.782
0.535
Temperature ( T )
1.0
44.1%
52.8%
12.4%
1.340
0.548
Nucleus ( p )
0.90 ( T=0.7 )
67.8%
73.9%
7.4%
0.791
0.534
Appendix
Table 11: Ablation of Stochastic Sampling Parameters on Confident Hallucination Stability. Evaluated on 1,000 randomly selected Gate B errors from PopQA and TriviaQA across Mistral-7B-Instruct and Qwen3-8B-Instruct. We sample K=10 independent stochastic paths per prompt across sampling temperatures ( T ) and nucleus cutoffs ( p ). Gate B errors are remarkably stable across temperatures up to T=0.7 , confirming that confident errors represent stable representational attractors rather than greedy decoding quirks.
Figure 4: Response Rigidity and Architectural Scaling Dynamics. (a) Persistence of confident hallucinations under increasing temperature T∈[0.0,1.2] . Even at T=0.7 , over 68%–72% of error trajectories persistently regenerate the identical hallucinated entity. (b) Gate B error rates across model parameter scales (7B to 14B). Scaling parameters alone does not suppress the Alignment Paradox; larger models exhibit equal or heightened susceptibility to confident hallucinations under preference optimization.
Model
Outcome
Sample Size ( n )
Total Layers ( L )
Mean l∗
Mean l∗/L
Final Top-1 Prob.
Mistral-7B (Instruct)
Wrong
500
32
28.36
0.886
0.948
Correct
500
32
27.26
0.852
0.971
Mistral-7B (Base)
Wrong
97
32
27.94
0.873
0.892
Correct
500
32
27.19
0.850
0.935
Qwen-8B (Instruct)
Wrong
500
36
35.34
0.982
0.961
Correct
500
36
35.35
0.982
0.984
Appendix
Table 12: Layer-wise Logit Lens Commitment Statistics. Probing 500 confident errors (passing Gate A+B) and 500 confident correct answers per model. Delineating the empirical scope of early commitment, both base and instruct models commit to their final token in late layers ( l∗/L≥0.85 ). Within each arm, errors commit slightly later than correct answers. The primary distinction lies in mid-layer entropy ( d=2.72 at layer 16), not the layer index of commitment.
Figure 5: Layer-wise Dynamics of Commitment and Logit Margins. Tracking top-1 accuracy, logit margin ( ztop1−ztop2 ), and layer entropy across all layers. Both base and instruct models resolve their top-1 prediction late in the network ( l∗/L≥0.85 ). However, instruct models undergo a severe mid-layer entropy collapse, which transitions into a steep logit margin surge in the final 2–4 layers ( Late-Layer Margin Expansion ).
Model
Dataset
Sample Size ( n )
R@1
R@2
R@5
R@10
R@100
Median Rank
Mistral-7B
PopQA (Long-tail)
320
0.019
0.100
0.209
0.309
0.653
42
TriviaQA (Head)
180
0.072
0.244
0.461
0.622
0.917
6
Qwen-8B
PopQA (Long-tail)
310
0.016
0.126
0.281
0.432
0.810
16
TriviaQA (Head)
190
0.116
0.316
0.558
0.716
0.937
5
Appendix
Table 13: Ground-Truth Token Rank Distribution across Knowledge Popularity. Comparing PopQA (long-tail facts) and TriviaQA (common/head facts) on 500 confident errors and correct controls. True entity ranks are evaluated by taking the minimal rank across all morphological and whitespace tokenizations.
Figure 6: Ground-Truth Latency Profiling across Knowledge Popularity. Recall@ k curves for ground-truth entity tokens under confident hallucinations ( p≥0.95 ) versus confident correct predictions. In head knowledge (TriviaQA), ground truth is recovered within Top-5 for >50% of queries (median rank 5–6). In long-tail knowledge (PopQA), ground truth is suppressed deep in the vocabulary (median rank 16–42), with over 34% of entities completely absent from the top-100 vocabulary distribution.
Model Arm
Layer 8 AUROC
Layer 16 AUROC
Layer 24 AUROC
Layer 31 AUROC
Mistral-7B Base
0.739±0.017
0.753±0.014
0.781±0.015
0.809±0.010
Standard DPO (Epoch 10)
0.735±0.012
0.774±0.007
0.786±0.012
0.804±0.012
BDM-DPO (Epoch 10, Ours)
0.728±0.009
0.779 ± 0.011
0.807 ± 0.014
0.823 ± 0.009
Appendix
Table 14: Hidden Representation Geometry: Linear Probing across Intermediate Layers. Training L2 -regularized logistic regression probes across layers L∈{8,16,24,31} on 4,000 balanced instances (2,000 correct vs. 2,000 confident errors). Evaluated via 5-fold cross-validation. Middle layers ( L=16,24 ) sustain high AUROC ( ∼0.77 – 0.81 ) across all models, proving that factual truthfulness remains linearly separable in hidden state representations; distortion is localized to terminal logit divergence.
Model
Layer
Mode
α
Recovery (%)
Retention (%)
Net Gain
Gold Rank Shift
Mistral-7B
31 ( L−1 )
Raw
0.2
0.0
99.0
−2
16.0 → 16.0
31 ( L−1 )
Normalized
0.5
1.5
97.0
−3
16.0 → 21.5
30 ( L−2 )
Normalized
0.5
3.5
88.5 †
−16
16.0 → 23.0
20 (Mid)
Normalized
0.5
11.0
92.0 †
+6
16.0 → 20.0
Qwen-8B
35 ( L−1 )
Normalized
0.2
2.0
100.0
+4
7.0 → 7.5
34 ( L−2 )
Normalized
0.5
4.0
55.5 †
−81
7.0 → 11.0
Appendix
Table 15: Empirical Diagnostic of Layer Contrastive Decoding (CD-Layer). Benchmarking 24 configurations on 200 confident errors and 200 confident correct answers under the pre-registered constraint: Retention ≥97% . Net Gain = Recovered Errors − Lost Correct. Across all layers and scaling modes, CD-Layer fails to produce meaningful net gains, exhibiting severe retention degradation as α scales.
Model
Detector / Probe
AUROC
AUPR
FPR@95TPR
Mistral-7B (Instruct)
Softmax Max Prob. ( pˉ )
0.661
0.641
0.852
Final Layer Entropy
0.686
0.676
0.850
Self-Consistency ( K=10 )
0.535
0.981
0.082
Normalized Commit Layer ( l∗/L )
0.608
0.665
0.902
Final Logit Margin
0.609
0.591
0.874
Margin Rise ( M24:32−M8:16 )
0.622
0.603
0.888
Appendix
Table 16: Detection Benchmark in the High-Confidence Regime ( pˉ≥0.95 ). Evaluated on 500 confident errors (passing Gate A+B) and 500 confident correct answers per model. AUROC indicates ability to detect errors (higher is better). Self-consistency collapses to random guessing ( ≈0.51 ) due to rigid fabricated outputs. Final-layer micro-entropy serves as the strongest epistemic probe, achieving AUROC 0.686–0.732.
Model
Abstention Criterion
AURC ↓
Acc@100%
Acc@80%
Acc@70%
Mistral-7B (Deployment n=5,537 )
Random Rejection
0.508
74.6
74.6
74.6
Sequence Confidence ( 1−pˉ )
0.364
74.6
77.7
79.1
Token Confidence ( 1−pt1 )
0.368
74.6
77.5
78.8
Final-Layer Micro-Entropy ( HL )
0.355
74.6
77.7
79.1
Qwen-8B (Deployment n=6,965 )
Random Rejection
0.495
73.8
73.8
73.8
Sequence Confidence ( 1−pˉ )
0.309
73.8
79.3
82.0
Appendix
Table 17: Selective Abstention under Realistic High-Confidence Deployment ( pˉ≥0.95 ). Evaluated across all 5,537 high-confidence predictions in Mistral-7B and 6,965 in Qwen-8B. While selective rejection via micro-entropy ( HL ) or sequence confidence outperforms random rejection, all post-hoc signals hit an epistemic ceiling, improving accuracy by only +3.1pp to +5.5pp at 80% coverage.
Figure 7: Risk-Coverage Trade-Off Curves for Selective Abstention under Alignment. Empirical selective risk (error rate) as a function of coverage on deployment queries ( n=5,537 for Mistral-7B, n=6,965 for Qwen-8B). Confidence-based filtering ( 1−pˉ ) and micro-entropy ( HL ) achieve only marginal risk reductions (+3.1pp to +5.5pp accuracy gain at 80% coverage), hitting a strict epistemic ceiling. Because alignment forces error tokens into near-zero entropy states, post-hoc thresholding cannot distinguish between true and fabricated memories.
Model
Method
Paradigm
HCE Elim. (%)
Abstain / Hedge (%)
Retention (%)
Corrupted ( n )
Net Gain
Mistral-7B-Inst
Unconstrained Greedy (Control)
Baseline
3.7%
0.8%
99.8%
1
+21
Hedging Prompt ( Kadavath et al., 2022 )
Prompt
7.8%
3.2%
97.2%
17
+30
Self-Correction ( Huang et al., 2024 )
2-Turn Critic
6.8%
3.0%
99.2%
5
+36
Self-Consistency ( K=5 ) ( Wang et al., 2023 )
Sampling
6.3%
5.0%
99.0%
6
+32
DoLa ( L−2 ) ( Chuang et al., 2024 )
Decoding
6.2%
1.2%
99.5%
3
+34
ET-Abstention (Raw τ=[1.5,1.2] )
Epistemic Gate
42.0%
40.3%
83.5% *
99
+153
Appendix
Table 18: Comprehensive Inference-Time Mitigation Benchmark across Four Aligned Model Families. Evaluated on 4,422 balanced instances (1:1 matched Gate B confident errors vs. Gate A confident correct controls). Pre-registered safety constraint: Retention≥97.0% (marked with * if violated). Net Gain = Eliminated Confident Errors − Corrupted Correct Answers. Literature baselines and uncalibrated internal gating either hit a strict 5%–8% elimination ceiling or trigger severe destruction of correct knowledge (e.g., Tulu-3 Self-Correction retention collapsing to 1.9%, Qwen-8B Hedging retention falling to 65.2%).
Tier
Configuration
M1
H1
Gate B (%)
#Gate B
Acc. (%)
ECE
Reference
Unaligned Base (Mistral-7B)
3.252
0.863
7.00%
140
46.8%
0.3322
Unconstrained
Standard DPO (1 Epoch Control)
3.584
0.731
7.65%
153
46.1%
0.3603
Static Clamp
M≡M0
3.332
0.829
7.10%
142
46.5%
0.3430
Linear Decay
M0max(0,1−γH)
3.320
0.837
6.95%
139
46.8%
0.3384
Dynamic Grid
M0=2.0,γ=1.0
3.315
0.839
6.95%
139
46.9%
0.3367
M0=2.0,γ=2.0
3.306
0.838
7.00%
140
46.8%
0.3380
Appendix
Table 19: Comprehensive Ablation of BDM-DPO Formulations and Hyperparameter Sensitivity (Pilot n=2,000 ). Evaluating 13 controlled configurations on a balanced cohort. Comparing unconstrained Standard DPO against static clamping, linear decay, and a 3×3 dynamic BDM grid. Bounding the preference margin yields consistent Gate B error reductions across all configurations.
Paradigm
Objective / Method
Ref. Policy
M1
H1
Gate B (%)
#Gate B
Acc. (%)
ECE
Reference Checkpoint
Mistral-7B-Instruct-v0.3
None
3.609
0.907
7.25%
1,615
47.8%
0.3569
Reference-Free
Vanilla SimPO ( γ=1.0 )
None
3.799
0.986
7.58%
1,687
45.4%
0.3896
Reference-Based
Standard DPO (Unbounded)
πref
3.916
0.796
7.37%
1,641
47.1%
0.3777
Epistemic Bounded
BDM-DPO (Ours)
πref
3.698
0.876
7.10%
1,580
47.7%
0.3608
Appendix
Table 20: Cross-Paradigm Preference Alignment Benchmark on the Full 22,267-Query Corpus. Comparing the pre-DPO reference checkpoint ( Mistral-7B-Instruct-v0.3 ) against reference-free preference optimization ( Vanilla SimPO ; Meng et al. 2024 ), standard unconstrained reference-based optimization ( Standard DPO ; Rafailov et al. 2023 ), and our epistemic-bounded method ( BDM-DPO ). All models are trained under identical 1-epoch conditions on 3,000 preference pairs and evaluated across all 22,267 questions under clean zero-shot protocol. Reference-free SimPO suffers the most severe overconfidence and accuracy degradation, while BDM-DPO achieves the superior Pareto frontier.
β
Method
M1
H1
Gate B (%)
#Gate B
Acc. (%)
ECE
0.05
Standard DPO
4.071
0.743
7.64%
1,701
46.8%
0.3889
BDM-DPO (Ours)
3.687
0.877
7.18%
1,599
47.6%
0.3621
0.10
Standard DPO
3.902
0.798
7.32%
1,630
47.2%
0.3771
BDM-DPO (Ours)
3.691
0.878
7.15%
1,591
47.6%
0.3617
0.20
Standard DPO
3.785
0.840
7.16%
1,595
47.4%
0.3699
BDM-DPO (Ours)
3.686
0.878
7.15%
1,593
47.6%
0.3620
Appendix
Table 21: Robustness to the DPO Regularization Weight β Across the Full 22,267-Query Corpus. Evaluated under identical reference model ( Mistral-7B-Instruct-v0.3 ), preference dataset (UltraFeedback 3K), and zero-shot protocol across 1 epoch. Standard DPO exhibits monotonic deterioration in margin M1 , Gate B confident hallucinations, and ECE as β decreases. In contrast, BDM-DPO remains invariant across β values, establishing a robust epistemic safety floor.
Preference Dataset
Optimization Method
M1
H1
Gate B (%)
#Gate B
Acc. (%)
ECE
UltraFeedback ( Cui et al., 2024 )
Standard DPO
3.902
0.798
7.32%
1,630
47.2%
0.3771
BDM-DPO (Ours)
3.691
0.878
7.15%
1,591
47.6%
0.3617
Anthropic HH-RLHF ( Bai et al., 2022 )
Standard DPO
3.665
0.867
7.72%
1,718
47.5%
0.3623
BDM-DPO (Ours)
3.640
0.889
7.31%
1,627
47.8%
0.3584
Appendix
Table 22: Robustness to the Post-Training Preference Dataset (UltraFeedback vs. Anthropic HH-RLHF). Evaluated under identical architecture ( Mistral-7B-Instruct-v0.3 ), hyperparameters ( β=0.10 , M0=3.0 , γ=2.0 ), optimizer, and unified zero-shot evaluation across all 22,267 queries. The preference dataset is the sole independent variable. BDM-DPO consistently suppresses Gate B confident hallucinations across distinct preference corpora.
Architecture
Epoch
Method
M1
H1
Gate B (%)
#Gate B
Acc. (%)
Mistral-7B-Inst
Epoch 1
Standard DPO
3.916
0.796
7.37%
1,641
47.1%
BDM-DPO
3.698
0.876
7.10%
1,580
47.7%
Epoch 10
Standard DPO
4.514
0.521
9.60%
2,138
42.9%
BDM-DPO
3.784
0.812
6.65%
1,481
47.6%
Llama-3.1-Tulu-3
Epoch 1
Standard DPO
1.766
1.482
0.46%
102
54.2%
BDM-DPO
1.748
1.512
0.41%
91
54.4%
Appendix
Table 23: Cross-Architecture Multi-Epoch Progression (Unified Zero-Shot Protocol, N=22,267 ). Comparing 1-epoch vs. 2-epoch preference optimization across distinct architectures. On LLaMA SFT, Standard DPO inflates M1 ( 1.766→1.855 ) and escalates Gate B errors (+29.4%, 102→132 ), while BDM-DPO pins margin (1.750) and Gate B (94), expanding the BDM advantage from ratio 0.892→0.712 (-28.8% error reduction).
Figure 8: Longitudinal Multi-Epoch Dynamics on Full Population ( N=22,267 ): Standard DPO Margin Expansion vs. BDM-DPO Stability. Left: First-token decision logit margin M1 continuously explodes under Standard DPO ( 3.609→4.514 ), whereas BDM-DPO acts as an epistemic circuit breaker, clamping the margin at M1≤3.80 . Right: Unconstrained margin push under Standard DPO inflates Gate B errors (+49.2%), while BDM-DPO eliminates up to 850 confident hallucinations.
Detection / Abstention Paradigm
Evaluation Metric
Reference SFT
Standard DPO
BDM-DPO (Ours)
Sampling Self-Consistency ( K=5 ) ( Wang et al., 2023 )
Table 24: Revitalizing Post-Hoc Detectability: Standard DPO Blinding vs. BDM-DPO Restoration. Evaluating detector separability in the high-confidence regime ( pˉ≥0.95 , PopQA and TriviaQA). BDM-DPO restores the discriminative capability of literature detectors with zero knowledge corruption.
Model Variant
MMLU (5-shot)
GSM8K (8-shot CoT)
ARC-C (25-shot)
McNemar p -value
Mistral-7B Base
62.69%
49.51%
63.65%
(Baseline)
Standard DPO (Epoch 10)
61.76%
49.51%
63.65%
Degradation ( p<10−4 )
BDM-DPO (Epoch 10)
62.70%
51.55%
63.65%
Preserved ( p=0.63 )
Appendix
Table 25: Downstream benchmark performance across general reasoning tasks. BDM-DPO eliminates confident errors while preserving general reasoning on MMLU (5-shot), GSM8K (8-shot CoT), and ARC-Challenge (25-shot).
Question
Ground Truth ( Y∗ )
Model Output ( y^ )
pˉ(y^)
H(P1)
M1
Gold Rank (L16 → L-1)
Failure Mechanism
Domain: Entertainment & Cinema (Mistral-7B-Instruct, PopQA)
Who directed the movie Star Wars?
George Lucas
Guillermo del Toro
0.992
0.038
4.85
Rank 3 → Rank 42
Mid-layer competition; final-layer suppression.
Domain: World History (Qwen3-8B-Instruct, TriviaQA)
In which country was the Battle of Waterloo fought?
Belgium
France
0.989
0.045
4.62
Rank 2 → Rank 18
Strong semantic prior; gold token suppressed at L27.
Domain: Literature (Qwen3-8B-Instruct, PopQA)
Who wrote the novel “The Spy Who Came in from the Cold”?
Table 26: Qualitative Case Studies of Confident Hallucinations across Diverse Knowledge Domains. Tracing model confidence pˉ , first-token decision metrics ( H(P1) , M1 ), and layer-wise ground-truth token latency. Across all domains, the model commits to a false attractor with near-certainty ( p≥0.97 ), while the true answer is suppressed in late layers.
As large language models (LLMs) are increasingly deployed in critical decision-making systems, the lack of reliable methods to measure their uncertainty presents a fundamental trustworthiness risk. We introduce a normalized confidence score based on output anchor token probabilities: classification labels for structured tasks and self-evaluation responses (Yes/No) for open-ended generation. This enables direct detection of errors and hallucinations with minimal overhead and without external validation. We make three key contributions. First, we propose a normalized confidence score and self-evaluation framework that exposes reliable confidence estimates for error detection across seven diverse benchmark tasks and five LLMs of varying architectures and sizes. Second, our theoretical analysis reveals that supervised fine-tuning (SFT) yields well-calibrated confidence through maximum-likelihood estimation, whereas reinforcement learning methods (PPO, GRPO) and DPO induce overconfidence via reward exploitation. Third, we propose post-RL SFT with self-distillation to restore confidence reliability in RL-trained models. Empirical results demonstrated that SFT improved average confidence-correctness AUROC from 0.806 to 0.879 and reduced calibration error from 0.163 to 0.034 on Qwen3-4B, while GRPO and DPO degraded confidence reliability. We demonstrated practical value through adaptive retrieval-augmented generation (RAG) that selectively retrieves context when the model lacks confidence, using only 58% of retrieval operations to recover 95% of the maximum achievable accuracy gain on TriviaQA
We identify a previously unknown calibration property of large language models: providing one confirmed intermediate fact toward a multi-step reasoning chain increases the model's confident-wrong-answer rate before full evidence eliminates it. We call this anchored confabulation: a partial anchor commits the model to confident parametric completion of remaining reasoning steps. We formalize it as Parametric Hallucination Confidence (PHC) and establish it across six lines of evidence including a causal injection experiment (PHC 0.613 to 0.656 to 0.595 to 0.536, N=160) and capability scaling across five model families (Spearman rho=0.900, p=0.037). The Anchoring Threshold Law k*(n)=floor(n/3) predicts PHC amplification by hop depth with four confirmed predictions. Applied to RAG routing, a LearnedRouter exploiting PHC closes 81.1% of the oracle performance gap (macro F1=0.426, p<1e-6) on 1,800 queries across four benchmarks with no model fine-tuning and 50x fewer labels than prior RL-based work. An epistemic humility prompt reduces the PHC spike by -0.118; explicit self-rating (PHC=0.684, p<0.001) outperforms lexical confidence as a routing signal.
Large language models are characterized by three key properties: capability, alignment, and faithfulness. Prior work studies the tradeoffs between capability and alignment, and between capability and faithfulness, but a third tension remains underexplored: the alignment-faithfulness conflict. We show that aligned models systematically deviate from their inputs on unsafe or sensitive content without disclosing the modification, a failure mode we call alignment-induced unfaithfulness (AIU). Unlike capability-driven unfaithfulness, which comes from errors in knowledge or reasoning, this is induced by post-training mechanisms that override adherence to the input. We introduce FaithConflict, a controlled dataset isolating both conflicts, and two complementary taxonomies: behavioral (B1-B8) and chain-of-thought reasoning (C0-C6). Across models, AIU increases with scale and more sharply than capability-driven unfaithfulness, a reverse scaling law; intermediate checkpoints show it is amplified during post-training, with DPO the stage at which the gap both grows most and becomes least visible. Prompting-based mitigation does not resolve it, revealing a capability-alignment-faithfulness trilemma in the design and evaluation of LLMs.
Pardis Sadat Zahraei, Janvijay Singh, Gokhan Tur +1