cs.LGMar 25, 2026

The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimation

Authors: Mingyi Liu

Organizations: Independent Researcher

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

RLHF-aligned language models exhibit response homogenization: on TruthfulQA (n=790), 40-79% of questions produce a single semantic cluster across 10 i.i.d. samples. On affected questions, sampling-based uncertainty methods have zero discriminative power (AUROC=0.500), while free token entropy retains signal (0.603). This alignment tax is task-dependent: on GSM8K (n=500), token entropy achieves 0.724 (Cohen's d=0.81). A base-vs-instruct ablation confirms the causal role of alignment: the base model shows 1.0% single-cluster rate vs. 28.5% for the instruct model (p < 10^{-6}). A training stage ablation (Base 0.0% -> SFT 1.5% -> DPO 4.0% SCR) localizes the cause to DPO, not SFT. Cross-family replication on four model families reveals alignment tax severity varies by family and scale. We validate across 22 experiments, 5 benchmarks, 4 model families, and 3 model scales (3B-14B), with Jaccard, embedding, and NLI-based baselines at three DeBERTa scales (all ~0.51 AUROC). Cross-embedder validation with two independent embedding families rules out coupling bias. Cross-dataset validation on WebQuestions (58.0% SCR) confirms generalization beyond TruthfulQA. The central finding -- response homogenization -- is implementation-independent and label-free. Motivated by this diagnosis, we explore a cheapest-first cascade (UCBD) over orthogonal uncertainty signals. Selective prediction raises GSM8K accuracy from 84.4% to 93.2% at 50% coverage; weakly dependent boundaries (|r| <= 0.12) enable 57% cost savings.

Figures & tables

MethodBoundaryCostCascade Role
Token entropy (ours)B1 FluencyFreeStage 1 (always-on)
LogTokU / PRO ( Ma et al., 2025b ; Chen et al., 2025c )B1 FluencyFreeStage 1 (drop-in)
Semantic Energy ( Ma et al., 2025a )B1 FluencyNN samplesSingle-cluster remedy
SINdex ( Abdaljalil et al., 2025 )B1 FluencyNN samplesStage 1 (escalation)
Semantic Entropy ( Kuhn et al., 2023 )B1 Fluency5–10 ×\timesStage 1 (escalation)
CoCoA ( Huang et al., 2026 )B1 Fluency1–5 ×\timesStage 1 (escalation)
Table 1: Positioning of UCBD relative to existing approaches. All prior methods operate within a single boundary; UCBD provides the orchestration layer.
Figure 1: UCBD framework: four-column architecture mapping brain mechanisms, boundary detectors, detection signals, and response strategies. Solid borders indicate experimentally validated components (B1–B2, cascade B1 →\to B2); dashed borders indicate theoretical components awaiting empirical validation (B3–B5). The Pointer Model (center, PFC analogue) connects to all five boundary detectors—solid arrows for validated boundaries (B1–B2), dashed arrows for theoretical ones (B3–B5)—dispatching queries into the cheapest-first cascade. Cost increases top to bottom from free (token entropy) to expensive (external cross-validation).
#HypothesisDataKey ResultStatus
1B1 domain specificityTruthfulQA 790qCV AUC=0.658 (eff.) / 0.395 (blind)✓
2B1–B2 independenceTruthfulQA 401qrr =0.119, dcor=0.143✓
3Cascade ≥\geq parallelTruthfulQA 401qpp =0.498, 57.4% cost saving✓
4Cross-model stability3 models ×\times 790q3B AUC=0.676 >> 14B=0.537✓
5B3 freshness decayFreshQA 1500q11–13 ×\times acc. drop; B1–B3 rr = −- 0.067✓
6Label robustnessLLM-judgeB1: 0.571 →\to 0.599 (cross-family)✓
Table 2: Summary of experiments across five benchmarks, three model scales, four model families, and five baseline methods. Strongest results: B1=0.724 on GSM8K (Exp 11), alignment tax with NLI validation (Exp 12), base-vs-instruct + cross-family + SFT/DPO ablation + cross-chain replication (Exp 13–18), quantization sensitivity (Exp 19), B5 rescue (Exp 7).
ModelEffective%Blind%Eff. AUCOverall
LLaMA-3.2-3B79%21%0.6760.622
Qwen3-4B50%50%0.6250.537
Qwen3-14B36%64%0.5370.490
Table 3: Scale effect: B1 effectiveness decreases with model size.
JudgeCorrect%B1 AUC95% CI
Word-overlap25.9%0.571[0.526, 0.617]
LLaMA-3.2-3B (cross family)54.6%0.599[0.563, 0.634]
Table 4: Cross-family judge validation confirms B1 robustness.
FeatureAUROC95% CICost
B1 mean entropy0.706[.635,.772]Free
B1 std entropy0.715[.643,.782]Free
B1 max entropy0.724[.650,.793]Free
Combined entropy (4 feat, CV)0.724 ±\pm .033—Free
P(True)0.608[.52,.70]1 call
Response length (tokens) †0.849[.791,.903]Free
Table 5: GSM8K error detection ( nn =500): B1 entropy and behavioral features. Incorrect answers show 49% higher mean entropy (Cohen’s dd =0.81, p<10−8p<10^{-8} ).
MethodAUROC95% CICost
B1 mean entropy0.599[0.559, 0.637]Free
SelfCheck-Emb ( kk =5)0.588[0.547, 0.626]6 ×\times
SE-Jaccard ( NN =10)0.548[0.510, 0.589]11 ×\times
SE-Embedding ( NN =10)0.542[0.513, 0.572]11 ×\times
SE-NLI † (DeBERTa-large)0.511[0.419, 0.594]11 ×\times +NLI
SE-NLI † (DeBERTa-base)0.512[0.421, 0.593]11 ×\times +NLI
Table 6: B1 entropy vs. sampling-based baselines on TruthfulQA (LLM-judge labels). B1 (free) matches or outperforms all baselines including canonical NLI-based SE at three model scales (70M–435M) and the full SINdex metric with intra-cluster coherence weighting. DeLong tests with Holm-Bonferroni correction: vs. SE-Emb padjp_{\text{adj}} =0.033*, vs. SE-Jaccard padjp_{\text{adj}} =0.040*, vs. SelfCheck pp =0.65 ns. † 200-question subset; DeBERTa-large and SINdex recomputed with LLM-judge labels for fair comparison.
Figure 2: The alignment tax mechanism (Jaccard clustering, primary metric). On single-cluster questions (40.0%), SE drops to exact chance (0.500, dashed red) because all 10 samples produce the same answer. B1 retains discriminative power (0.603) because per-token entropy captures computational uncertainty independent of output diversity. Under embedding clustering (sensitivity), 79% of questions are single-cluster.
MetricBaseInstructDifference
Single-cluster rate1.0%28.5%+27.5pp
Mean clusters9.263.58−- 5.68
Mean SE2.1580.832−- 1.326
Wilcoxon signed-rank (base >> instruct): WW =18,331, p<10−6p<10^{-6}
Table 7: Base-vs-instruct response diversity on TruthfulQA ( nn =200). Alignment reduces mean clusters by 2.6 ×\times and increases single-cluster rate from 1% to 28.5%.
ModelSCRMean NCWilcoxon vs. Qwen
Qwen3-14B-Instruct28.5%3.58—
LLaMA-3.2-3B-Instruct5.5%7.27p<10−6p<10^{-6}
Mistral-7B-Instruct1.0%7.87p<10−6p<10^{-6}
Qwen3-14B-Base1.0%9.26p<10−6p<10^{-6}
Table 8: Cross-family alignment tax ( nn =200). Homogenization varies widely across model families and scales.
StrategyParametersSCRMean NCMean SE
BaselineTT =1.028.5%3.580.832
NucleusTT =1.0, pp =0.933.5%3.400.786
NucleusTT =1.0, pp =0.9530.0%3.550.827
Low tempTT =0.730.0%2.960.668
Table 9: Decoding strategy ablation ( nn =200). The alignment tax persists across strategies.
StageModelSCRMean NCMean SE
BaseMistral-7B-v0.10.0%9.282.170
SFTmistral-7b-sft-beta1.5%8.632.024
SFT+DPOzephyr-7b-beta4.0%8.011.897
Wilcoxon: Base →\to SFT pp =0.002, SFT →\to DPO pp =0.0001, Base →\to DPO p<10−6p<10^{-6}
Table 10: Training stage ablation. DPO is the primary driver of homogenization.
max_tokensSCRMean NCMean SEAvg Words
4032.0%3.020.67626.9
10010.0%7.461.76968.8
2008.0%8.301.929115.2
Base model SCR ≈\approx 0% at all lengths. 5/16 single-cluster questions persist across all settings.
Table 11: Generation length sensitivity (50q). SCR decreases with length but persists at all settings: 8% at 200 tokens vs. 0% for base model ( p<0.05p<0.05 ), confirming alignment-driven homogenization.
StageModelSCRMean NCMean SE
BaseLlama-3.1-8B0.0%9.462.209
SFTTulu-3-8B-SFT0.0%9.022.123
SFT+DPO+RLVRtulu3-8b0.5%9.312.174
Wilcoxon: Base →\to SFT pp =0.00004, SFT →\to DPO pp =0.008, Base →\to DPO pp =0.43
Table 12: Tulu-3 chain ablation. DPO effect is recipe-dependent : Tulu-3’s DPO produces minimal homogenization compared to Zephyr (Exp 16).
 Mean J[email protected][email protected]Mean [email protected]
Q4_K_M (4-bit)0.60863.3%6.7%7.67
Q8_0 (8-bit)0.57626.7%6.7%7.30
Δ\Delta (Q8 −- Q4)−- 0.032−- 36.6pp0.0pp−- 0.37
Table 13: Quantization sensitivity: Q4 vs Q8 on Mistral-7B-Instruct. At semantic-level thresholds ( tt =0.7), both quantizations produce identical SCR.
Embedder[email protected][email protected][email protected]
Qwen3-Embedding (1.5B)94.0%78.0%14.0%
Nomic-embed-text (137M)98.0%92.0%52.0%
Per-question cluster-count Pearson rr =0.033 at τ\tau =0.85; both detect single-cluster collapse.
Table 14: Cross-embedder validation. An independent embedder detects more homogenization, not less—ruling out coupling bias.
Setting[email protected][email protected][email protected]
40 tokens, 200q (Exp. 12)79.0%79.0%—
200 tokens, 200q (this exp.)61.5%33.5%14.0%
Table 15: Extended length sensitivity at scale (200q vs. original 50q). Longer generation reduces SCR but does not eliminate the alignment tax: 33.5% of questions remain single-cluster at τ\tau =0.85.
Dataset[email protected][email protected][email protected]
TruthfulQA 200q (100tok)79.0%79.0%—
WebQuestions 200q (100tok)77.5%58.0%34.0%
Table 16: Cross-dataset SCR validation. WebQuestions shows stronger homogenization than TruthfulQA, confirming the alignment tax is not dataset-specific.
MethodAUROCΔ\Delta vs. JaccardCost
NLI-SE (cross-encoder)0.639+0.01411 ×\times +NLI
Jaccard-SE0.625—11 ×\times
95% bootstrap CIs overlap ( Δ\Delta =+0.014 crosses zero).
Table 17: Cross-encoder NLI-SE vs. Jaccard proxy on matched samples (200q, 200 tokens). Bootstrap 95% CIs overlap; canonical NLI architecture provides negligible improvement.
StageModelMean HH (nats)Std HHRetention
BaseMistral-7B-v0.11.1750.312100%
SFTmistral-7b-sft-beta1.0550.30289.8%
SFT+DPOzephyr-7b-beta0.7760.17966.0%
Kruskal-Wallis HH =183.89, pp =1.17 ×10−40\times 10^{-40} ; all pairwise Mann-Whitney p<0.001p<0.001 .
Table 18: Stage-wise token entropy across the alignment pipeline. DPO compresses token entropy by 34% but retains 66%—while response diversity collapses (SCR 0% →\to 4%, Exp 16). This asymmetric compression confirms the decoupling between token-level and response-level uncertainty.
BenchmarkMethodAUROC95% CICost
TruthfulQAB1 entropy0.599[0.559, 0.637]Free
(factual QA)SelfCheck ( kk =5)0.588[0.547, 0.626]6 ×\times
 Semantic entropy ( NN =10)0.548[0.510, 0.589]11 ×\times
 P(True)0.427[0.408, 0.446]1 call
FreshQAB1 entropy0.657[0.610, 0.703]Free
(temporal)P(True)0.399[0.366, 0.432]1 call
Table 19: B1 entropy vs. baselines across three benchmarks. TruthfulQA: LLM-judge labels ( nn =790); FreshQA/GSM8K: exact-match labels. B1 matches or outperforms all baselines at zero additional cost.
Figure 3: AUROC for error detection across tasks (dashed red = chance). The alignment tax is visible on TruthfulQA: B1 (free, 0.599) matches SelfCheckGPT (6 ×\times , 0.588, pp =0.65) and significantly outperforms Jaccard-approximated SE (11 ×\times , 0.548, padjp_{\text{adj}} =0.04). On GSM8K, where alignment does not suppress entropy, B1 reaches 0.724 ( dd =0.81).
PairPearson rrdcorHSIC ppMI (bits)nnSource
B1–B20.119*0.143*0.020*0.008401Exp. 2
B1–B3−- 0.067——0.015500Exp. 5
B1–B40.0540.0720.2520.006790Exp. 10
B1–B50.070——0.012790Exp. 7
B2–B4−- 0.0860.156*0.000*0.003401Exp. 10
* p<0.05p<0.05 . Effect sizes remain small ( ∣r∣≤0.12|r|\leq 0.12 , dcor ≤0.16\leq 0.16 ); superadditive coverage holds approximately.
Table 20: Pairwise boundary dependence. MI: Freedman-Diaconis binning, 1000 permutation null (mean permuted MI ≈\approx 0.003 bits; max observed = 0.015 ≈\approx 5 ×\times null). dcor/HSIC: 500 permutations. Weak dependence confirmed.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

ComparisonExpEffectpp -valueTest
B1 vs SE-Jaccard AUROC12Δ\Delta =+0.0510.040*DeLong+Holm
B1 vs SE-Embed AUROC12Δ\Delta =+0.0570.033*DeLong+Holm
B1 vs SelfCheck AUROC12Δ\Delta =+0.0110.65 nsDeLong
B1 vs NLI-SE (large)12Δ\Delta =+0.088[0.419,0.594]Bootstrap CI
Base vs Instruct SCR131.0% vs 28.5%<10−6<10^{-6}Wilcoxon
Base→SFT (Zephyr)16Δ\Delta NC= −- 0.640.002Wilcoxon
Appendix
Table 21: Statistical summary of all key comparisons. Tests: DeLong (AUROC), Wilcoxon (paired distributions), bootstrap (CIs). All pp -values two-sided.
More in this topic

Homogenization in Language Models

See all 8 papers

Explore similar work

CardsList
Sep 26, 2026

The Alignment Paradox: How Post-Training Amplifies Confident Hallucinations in Language Models

Large language models (LLMs) can produce factually incorrect answers with high confidence, undermining their reliability and limiting the effectiveness of uncertainty-based error detection. While prior research attributes confident hallucinations to factors such as missing knowledge in training data, reasoning errors, or stochastic decoding, we uncover that post-training alignment itself is a primary driver of these errors, a phenomenon we call the Alignment Paradox. Across five model families evaluated on factual benchmarks, unaligned base models produce few high-confidence errors on long-tail factual queries, whereas instruction-tuned models multiply high-confidence errors (p≥0.95p \ge 0.95) by more than an order of magnitude (10×\times to 35×\times). Layer-wise probing with the Logit Lens reveals that this overconfidence emerges in late layers, where wrong-answer margins expand past 4.0 points after remaining near zero across early and intermediate layers. These findings motivate limiting margin growth during post-training. We implement this principle through an entropy-dependent margin bound in direct preference optimization (DPO). In multi-epoch experiments with Mistral-7B, the bounded objective reduces high-confidence errors by up to 35.3% relative to standard DPO while maintaining performance on evaluated general reasoning benchmarks. These results show that bounded margins mitigate confident hallucinations during post-training.

Qingjia Huang, Yakai Li, Jianguo Wu +5

Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China

Hallucination in Language ModelsLLM Post-TrainingLLM Alignment
AskPDF
Sep 30, 2026

Emergent Unfaithfulness: How Alignment Training Causes Language Models to Silently Override Task Faithfulness

Large language models are characterized by three key properties: capability, alignment, and faithfulness. Prior work studies the tradeoffs between capability and alignment, and between capability and faithfulness, but a third tension remains underexplored: the alignment-faithfulness conflict. We show that aligned models systematically deviate from their inputs on unsafe or sensitive content without disclosing the modification, a failure mode we call alignment-induced unfaithfulness (AIU). Unlike capability-driven unfaithfulness, which comes from errors in knowledge or reasoning, this is induced by post-training mechanisms that override adherence to the input. We introduce FaithConflict, a controlled dataset isolating both conflicts, and two complementary taxonomies: behavioral (B1-B8) and chain-of-thought reasoning (C0-C6). Across models, AIU increases with scale and more sharply than capability-driven unfaithfulness, a reverse scaling law; intermediate checkpoints show it is amplified during post-training, with DPO the stage at which the gap both grows most and becomes least visible. Prompting-based mitigation does not resolve it, revealing a capability-alignment-faithfulness trilemma in the design and evaluation of LLMs.

Pardis Sadat Zahraei, Janvijay Singh, Gokhan Tur +1

University of Illinois Urbana-Champaign

LLM AlignmentInstruction Following
AskPDF
May 8, 2026

Topology-Enhanced Alignment for Large Language Models: Trajectory Topology Loss and Topological Preference Optimization

Alignment of large language models (LLMs) via SFT and RLHF/DPO typically ignores the global geometry of the representation space, relying instead on local token likelihoods or scalar scores. We view generation as tracing a semantic trajectory in hidden space and propose a topology-enhanced alignment framework that regularizes these trajectories using 0-dimensional persistent homology. First, for SFT, we introduce Trajectory Topology Loss (TTL). Treating prompt and gold-answer embeddings as a mixed point cloud, we use a 0D persistent homology algorithm to extract "prompt-answer bridges." TTL aligns the model's actual update direction with these topological bridges rather than arbitrary directions. Second, for DPO, we propose Topological Preference Optimization (TPO). TPO constructs topic-specific semantic preference vectors and aligns the improvement direction between rejected and chosen responses with these vectors in an intermediate hidden layer. We also introduce a dynamic weighting scheme to balance DPO and TPO losses. Evaluating on Qwen2.5-7B-Instruct using UltraChat and Anthropic HH-RLHF, our topology-enhanced objectives consistently outperform strong non-topological baselines (e.g., per-example, nearest-neighbor, random regularizers) on automatic preference metrics and LLM-judge evaluations, while maintaining or improving toxicity. Results show persistent homology and trajectory geometry offer a promising direction for controllable alignment.

Yurui Pan, Ke Xu, Bo Peng

School of Computing and Intelligent Innovation, Fudan University · School of Economics and Management, Tongji University · College of Information Technology, Shanghai Ocean University

LLM AlignmentDirect Preference Optimization
AskPDF