Truth in Contrast: Calibrated Factual Evidence from Language Model Likelihoods

Ponhvoan Srey; Jason Lucas; Pittawat Taveekitworachai; Junyoung Lee; Anh Tuan Luu

cs.CLOct 16, 2026

Truth in Contrast: Calibrated Factual Evidence from Language Model Likelihoods

Authors: Ponhvoan Srey, Jason Lucas, Pittawat Taveekitworachai, Junyoung Lee, Anh Tuan Luu

Organizations: Nanyang Technological University · University of Colorado at Boulder · National University of Singapore · VinUniversity

Abstract

Large Language Models (LLMs) encode substantial factual knowledge. Yet, estimating confidence in factual claims reliably remains challenging. We study likelihood-based factual confidence from the complementary perspectives of discrimination and calibration across multilingual factual statements. Specifically, we instantiate contrastive generative likelihoods as Contrastive Factual Evidence (CFE), which compares the likelihood of the same claim under competing factual and non-factual conditions, and derive its probabilistic counterpart, the Contrastive Factual Posterior (CFP). We show analytically that CFP calibration depends on the differential error between the two surrogate conditionals, allowing errors shared across them to cancel and yield tighter calibration bounds. Empirically, CFP substantially improves calibration while maintaining factual discrimination comparable to its strongest competitor. Nonetheless, discrimination is uniformly weak across methods in lower-resource languages, which simultaneously undermines the utility of the measured calibration. To this end, we interpret a likelihood geometry, within which an evidence-bearing direction is induced, along with an orthogonal evidence-invariant direction. In our investigation, we selectively align target-language claims with stronger English evidence only along the factual-evidence direction, improving discrimination and open-ended factual generation without target-language labels.

Figures & tables

Figure 1: Overview. (1) We instantiate CFP as factual confidence and evaluate through complementary calibration and discrimination lenses (§ 4 ). (2) Discrimination degrades in lower-resource languages despite low ECE , motivating selective evidence transfer from stronger languages (§ 5 ).

Table 1: Main results by language, with rightmost column representing mean across all languages. Each cell reports mean with five-fold cross-validation standard error (SE). † Gen-Z outputs hard predictions, so ECE and BS reduce to classification error and are reported only for completeness.

Component	ECE ( $\downarrow$ )	BS ( $\downarrow$ )	AUROC ( $\uparrow$ )
$\log q_{1}$	$.2903{\scriptstyle\,\pm\,.0035}$	$.3235{\scriptstyle\,\pm\,.0027}$	$.6309{\scriptstyle\,\pm\,.0142}$
$\log q_{0}$	$.2927{\scriptstyle\,\pm\,.0035}$	$.3286{\scriptstyle\,\pm\,.0028}$	$.6032{\scriptstyle\,\pm\,.0148}$
CFE ${}_{\text{raw}}$	$.1420{\scriptstyle\,\pm\,.0084}$	$.2369{\scriptstyle\,\pm\,.0025}$	$.6894{\scriptstyle\,\pm\,.0144}$
CFE ${}_{\text{single}}$	$.1373{\scriptstyle\,\pm\,.0079}$	$.2411{\scriptstyle\,\pm\,.0023}$	$.6729{\scriptstyle\,\pm\,.0146}$
CFE	$.1446{\scriptstyle\,\pm\,.0080}$	$.2387{\scriptstyle\,\pm\,.0024}$	$.6888{\scriptstyle\,\pm\,.0144}$
CFP	$.1150{\scriptstyle\,\pm\,.0103}$	$.2405{\scriptstyle\,\pm\,.0007}$	$.6888{\scriptstyle\,\pm\,.0144}$

Table 2: Ablation results with different formulation choices.

Figure 2: ECE for varying non-factual proportions with fixed

\hat{\alpha}=0

and corrected

\hat{\alpha}=\alpha^{\star}

Figure 3: Geometry of CFE alignment.

Table 3: Alignment results on True-False discriminability (AUROC) and general-task retention on Belebele (Accuracy). “Full” and “CFE” are finetuned on True-False training sets with full-consistency and CFE objective, respectively. English Base row reports the frozen English reference used as alignment anchor.

Language	Objective	Factual Precision (%)	# Claims	# Correct
English	Base	41.29	12.74	5.26
	Full	38.93 (-2.36)	12.87 (+0.13)	5.01 (-0.25)
	CFE	42.32 (+1.03)	11.67 (-1.07)	4.94 (-0.32)
Hindi	Base	27.47	11.72	3.22
	Full	32.42 (+4.95)	8.29 (-3.43)	2.69 (-0.53)
	CFE	32.88 (+5.41)	10.92 (-0.80)	3.59 (+0.37)

Table 4: Long-form biography generation results using Qwen-3.5-4B, base and finetuned on Hindi statements.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Factual and non-factual generative conditions for CFE. Pair #1 is the default for CFE

{}_{\text{single}}

Figure 5: P(True) / P(True)-norm prompt.

Method	No Held-out $\rightarrow$ Held-out
Energy	.1726 $\rightarrow$ .1735
Entropy	.3065 $\rightarrow$ .2842
LN-Entropy	.1257 $\rightarrow$ .1357
NLL	.2937 $\rightarrow$ .2666
Perplexity	.4781 $\rightarrow$ .4689
Average	.2753 $\rightarrow$ .2658

Appendix

Table 5: ECE with

\operatorname{min-max}

scaling using the entire dataset without a held-out calibration set or with one.

Method	IT	Chat	ECE ( $\downarrow$ )	BS ( $\downarrow$ )	AUROC ( $\uparrow$ )
P(True)	✗	✗	$.2026{\scriptstyle\,\pm\,.0050}$	$.2816{\scriptstyle\,\pm\,.0014}$	$.6548{\scriptstyle\,\pm\,.0144}$
	✗	✓	$.4530{\scriptstyle\,\pm\,.0014}$	$.4543{\scriptstyle\,\pm\,.0011}$	$.6081{\scriptstyle\,\pm\,.0158}$
	✓	✗	$.3167{\scriptstyle\,\pm\,.0041}$	$.3589{\scriptstyle\,\pm\,.0023}$	$.5760{\scriptstyle\,\pm\,.0154}$
	✓	✓	$.4987{\scriptstyle\,\pm\,.0008}$	$.4965{\scriptstyle\,\pm\,.0008}$	$.5420{\scriptstyle\,\pm\,.0163}$
P(True)-norm	✗	✗	$.1824{\scriptstyle\,\pm\,.0076}$	$.2399{\scriptstyle\,\pm\,.0004}$	$.7823{\scriptstyle\,\pm\,.0113}$
	✗	✓	$.2294{\scriptstyle\,\pm\,.0043}$	$.2889{\scriptstyle\,\pm\,.0019}$	$.6501{\scriptstyle\,\pm\,.0148}$

Appendix

Table 6: Results with different model types and prompt templates: “IT” ✗ and ✓ refer to base model and instruction-tuned model respectively, and “Chat” indicates usage of chat template.

Method	IT	Native	ECE ( $\downarrow$ )	BS ( $\downarrow$ )	AUROC ( $\uparrow$ )
P(True)	✗	✗	$.1991{\scriptstyle\,\pm\,.0042}$	$.2866{\scriptstyle\,\pm\,.0013}$	$.6399{\scriptstyle\,\pm\,.0148}$
	✗	✓	$.1974{\scriptstyle\,\pm\,.0056}$	$.2734{\scriptstyle\,\pm\,.0015}$	$.6600{\scriptstyle\,\pm\,.0142}$
	✓	✗	$.3326{\scriptstyle\,\pm\,.0038}$	$.3664{\scriptstyle\,\pm\,.0025}$	$.5789{\scriptstyle\,\pm\,.0156}$
	✓	✓	$.3006{\scriptstyle\,\pm\,.0041}$	$.3522{\scriptstyle\,\pm\,.0119}$	$.5614{\scriptstyle\,\pm\,.0154}$
P(True)-norm	✗	✗	$.1745{\scriptstyle\,\pm\,.0077}$	$.2411{\scriptstyle\,\pm\,.0003}$	$.7806{\scriptstyle\,\pm\,.0113}$
	✗	✓	$.1769{\scriptstyle\,\pm\,.0076}$	$.2296{\scriptstyle\,\pm\,.0004}$	$.7631{\scriptstyle\,\pm\,.0019}$

Appendix

Table 7: Results with different model types and English versus native prompting in the target language on non-English datasets.

Figure 6: Calibration and discrimination across model families and scales. Columns report ECE (

\downarrow

) and AUROC (

\uparrow)

. Each point is averaged over eight languages, base and IT model variants, and model/prompt-format settings (including English and native prompting but excluding base models with chat template).

Language	# Claims	Human—Human	LLM-Human
English	109	88.1% (0.734)	75.7% (0.527)
German	71	88.7% (0.761)	85.9% (0.719)
Hindi	71	100.0% (1.000)	83.1% (0.634)

Appendix

Table 8: Alignment between human and LLM judgment: agreement % (Cohen’s

\kappa

Training Topic	Objective	German			Hindi
Training Topic	Objective	True–False	BB Target	BB English	True–False	BB Target	BB English
Animals	Base	63.8	81.9	86.9	50.0	42.8	86.9
	Full	56.9 $(-6.9)$	81.9 $(0.0)$	86.6 $(-0.3)$	43.9 $(-6.0)$	40.7 $(-2.1)$	87.0 $(+0.1)$
	CFE	70.7 $(+6.8)$	81.9 $(0.0)$	87.1 $(+0.2)$	59.9 $(+9.9)$	40.7 $(-2.1)$	86.7 $(-0.2)$
Chemical Elements	Base	81.1	81.9	86.9	54.5	42.8	86.9
	Full	78.4 $(-2.7)$	81.3 $(-0.6)$	86.7 $(-0.2)$	54.5 $(-0.1)$	40.2 $(-2.6)$	86.6 $(-0.3)$

Appendix

Table 9: Per-topic alignment results for Qwen-3.5-2B. Each topic denotes the True–False training split used to fit a separate adapter. True–False reports held-out AUROC, and Belebele (BB) reports accuracy.

Training Topic	Objective	German			Hindi
Training Topic	Objective	True–False	BB Target	BB English	True–False	BB Target	BB English
Animals	Base	71.7	92.4	92.9	45.4	65.3	92.9
	Full	46.6 $(-25.2)$	91.9 $(-0.6)$	92.9 $(0.0)$	50.6 $(+5.3)$	66.0 $(+0.7)$	93.2 $(+0.3)$
	CFE	70.0 $(-1.8)$	92.2 $(-0.2)$	93.2 $(+0.3)$	61.2 $(+15.9)$	65.8 $(+0.4)$	93.0 $(+0.1)$
Chemical Elements	Base	93.5	92.4	92.9	60.5	65.3	92.9
	Full	91.5 $(-2.0)$	92.0 $(-0.4)$	93.1 $(+0.2)$	63.6 $(+3.1)$	64.3 $(-1.0)$	93.0 $(+0.1)$

Appendix

Table 10: Per-topic alignment results for Qwen-3.5-4B. Each topic denotes the True–False training split used to fit a separate adapter. True–False reports held-out AUROC, and Belebele (BB) reports accuracy.

Training Topic	Objective	English Generation			Hindi Generation
Training Topic	Objective	Precision (%)	# Claims	# Correct	Precision (%)	# Claims	# Correct
Animals	Base	41.29	12.74	5.26	27.47	11.72	3.22
	Full	40.24 $(-1.05)$	11.68 $(-1.06)$	4.70 $(-0.56)$	33.57 $(+6.10)$	8.34 $(-3.38)$	2.80 $(-0.42)$
	CFE	42.17 $(+0.89)$	12.14 $(-0.60)$	5.12 $(-0.14)$	37.04 $(+9.57)$	13.12 $(+1.40)$	4.86 $(+1.64)$
Chemical Elements	Full	39.46 $(-1.83)$	12.52 $(-0.22)$	4.94 $(-0.32)$	29.96 $(+2.48)$	9.08 $(-2.64)$	2.72 $(-0.50)$
Chemical Elements	CFE	44.98 $(+3.70)$	11.56 $(-1.18)$	5.20 $(-0.06)$	34.12 $(+6.64)$	11.08 $(-0.64)$	3.78 $(+0.56)$

Appendix

Table 11: Per-topic biography-generation results for Qwen-3.5-4B finetuned on Hindi True–False statements. Each topic denotes the training split used to fit a separate adapter. Factual precision is pooled across biographies containing at least one claim; # Claims and # Correct are per-entity means.

Confidence Estimation in Language Models

Confidence estimation in language models measures whether a model’s uncertainty signals predict answer correctness and support reliable decisions.

See all 248 papers

Claim-Level Confidence Calibration for Reliable Decision Making with Large Language Models
Aug 23, 2026Toghrul Abbasli, Kentaroh Toyoda, Yuan Wang +1Confidence Estimation in Language Models Language Model Calibration
Tsinghua University, Haidian District, Beijing, 100084, P. R. China · Vulcan Research, AIFT, Singapore · Keio Global Research Institute (KGRI), Japan +2
Retrieval-Augmented Linguistic Calibration
May 19, 2026Yi-Fan Yeh, Linwei Tao, Minjing Dong +4Confidence Estimation in Language Models Faithfulness of Language Model Reasoning
School of Computer Science University of Sydney Sydney, Australia · City University of Hong Kong Hong Kong · Shanghai Jiao Tong University Shanghai, China +1
Think Through Uncertainty: Improving Long-Form Generation Factuality via Reasoning Calibration
Apr 13, 2026Xin Liu, Lu WangLLM Hallucination Mitigation Language Model Calibration
Computer Science and Engineering, University of Michigan, Ann Arbor, MI, USA

Aug 23, 2026J/K move · Enter open · S save

Claim-Level Confidence Calibration for Reliable Decision Making with Large Language Models

Toghrul Abbasli, Kentaroh Toyoda, Yuan Wang, Li Chen

Tsinghua University, Haidian District, Beijing, 100084, P. R. China · Vulcan Research, AIFT, Singapore · Keio Global Research Institute (KGRI), Japan+2

Confidence Estimation in Language Models Language Model Calibration LLM Reliability Confidence Estimation

Open paper AskPDF arXiv

Large Language Models (LLMs) increasingly support decision-making in high-stakes domains, but they often hallucinate and express confidence that is misaligned with factual correctness. Response-level confidence is a coarse signal: a single generation can mix correct and incorrect statements, so a single number is not actionable for users that must accept, reject, or verify individual pieces of information. We study claim-level confidence calibration as a decision-relevant uncertainty signal: each response is decomposed into atomic, verifiable claims, and each claim is assigned a calibrated confidence using inference-time signals from consistency across samples and self-verification. Our framework operates in closed-box settings (no logits, no fine-tuning) and applies post-hoc calibration directly at the claim level, enabling selective intervention such as evidence retrieval or human review for low-confidence claims. Across TriviaQA and TruthfulQA we evaluate seven baselines on six recent models (Llama-3.1, Mistral, Qwen2.5, DeepSeek-R1, GPT-4, GPT-4o), and show that claim-level decomposition combined with post-hoc calibration reduces expected calibration error on factual questions while exposing failure modes on adversarial false-premise questions where decision-makers most need reliable uncertainty estimates.

Abstract

Confidence Estimation in Language Models

Explore similar work

Claim-Level Confidence Calibration for Reliable Decision Making with Large Language Models

Retrieval-Augmented Linguistic Calibration

Think Through Uncertainty: Improving Long-Form Generation Factuality via Reasoning Calibration