Confidence Estimation in Language Models

Follow this topic Ask about this topic

Momentum

24 papers in the last four weeks, up 500% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 134

All topics

CALIBER: Calibrating Confidence Before and After Reasoning in Language Models
Jun 23, 2026Conor Finlay, Joshua Kurien, Saurabh Dash +2Confidence Estimation in Language Models Language Model Calibration
1Cohere Labs
The Score Granularity Gap in Black-Box LLM Classification: A Comparative Study of Confidence Constructions
Jun 20, 2026Ao Sun, Tian Sun, Jiaxing GengLLM Evaluation Selective Prediction
Latent Confidence Alignment for LLM Self-Assessment
Jun 20, 2026Ting-Yu Chen, Tingting Yu, Pei-Cing Huang +3LLM Evaluation Confidence Estimation in Language Models
Department of Information Management National Sun Yat-Sen University Kaohsiung, Taiwan · Kaohsiung Medical University Hospital Kaohsiung Medical University Kaohsiung, Taiwan
Speaking in Self-Assessing Tongues: On the Verbalized Confidence of LLMs in Machine Translation
Jun 15, 2026Ali Marashian, Alexis Palmer, Katharina von der WenseConfidence Estimation in Language Models Language Model Calibration
University of Colorado Boulder, USA · Johannes Gutenberg University Mainz, Germany
The Value Axis: Language Models Encode Whether They're on the Right Track
Jun 15, 2026Nick Jiang, Isaac Kauvar, Jack LindseyConfidence Estimation in Language Models Language Model Calibration
1Stanford University · 2Anthropic
SAGE: Answer-Conditioned Uncertainty Targets for Verbal Uncertainty Alignment
Jun 9, 2026Kaiwen Shi, Zheyuan Zhang, Yanfang YeConfidence Estimation in Language Models Preference Optimization
University of Notre Dame
Code Is More Than Text: Uncertainty Estimation for Code Generation
Jun 8, 2026Yuling Shi, Caiqi Zhang, Yuexian Li +4Confidence Estimation in Language Models LLM Reliability
Shanghai Jiao Tong University · University of Cambridge
The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and Trust
Jun 5, 2026Nishant Subramani, Palash Goyal, Yiwen Song +4Confidence Estimation in Language Models Language Model Calibration
Carnegie Mellon University · Google · Scale AI
CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction
Jun 4, 2026Mohammad Anas Jawad, Cornelia CarageaConfidence Estimation in Language Models Language Model Calibration
Computer Science, University of Illinois Chicago, USA.
Calibrating Overconfidence Without Sacrificing Confidence: Probe-Conditioned Head Intervention for LLMs
Jun 2, 2026Ke Li, Chongzhe Zhang, Zifan Zeng +3Confidence Estimation in Language Models Language Model Calibration
Huawei Heisenberg Research Center · EPFL · TU Berlin +1
Integrating Local and Global Entropy for Uncertainty Quantification in LLMs
Jun 2, 2026Johanne Medina, Tianyi Zhou, Keivin Isufaj +2Uncertainty Quantification Confidence Estimation in Language Models
Qatar Computing Research Institute, HBKU Doha, Qatar · KTH Royal Institute of Technology Stockholm, Sweden
Quantifying Faithful Confidence Expression in Large Reasoning Models
Jun 2, 2026Areeb Gani, Asal Meskin, Gabrielle Kaili-May Liu +1LLM Evaluation Confidence Estimation in Language Models
1Yale University
Clustered Self-Assessment: A Simple yet Effective Method for Uncertainty Quantification in Large Language Models
Jun 2, 2026Qi Cao, Takeshi Kojima, Andrew Gambardella +3LLM Evaluation Uncertainty Quantification
The University of Tokyo, Japan
Large Language Models Are Overconfident in Their Own Responses
Jun 2, 2026Mario Sanz-Guerrero, Manuel Mager, Katharina von der WenseConfidence Estimation in Language Models LLM Prompting
∇Johannes Gutenberg University Mainz, Germany · ♠University of Colorado Boulder, USA
Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction
Jun 1, 2026Yujia Tong, Yuxi Wang, Yunyang Wan +3LLM Quantization LLM Evaluation
Wuhan University of Technology · Nanyang Technological University
Decoding in Order-Agnostic Language Models: Chain-Rule Deviation and Uniform Spreading
May 31, 2026Lin YaoLanguage Model Decoding Confidence Estimation in Language Models
School of Computer Science, Shanghai Jiao Tong University, Shanghai, 200240, China · Zhongguancun Academy, Beijing, 100097, China
Shared Doubt: Zero-Shot Cross-Lingual Confidence Estimation for Language Models
May 29, 2026Athina Kyriakou, Dennis Ulmer, Ivan TitovMultilingual Language Models Zero-Shot Learning
° ILCC, University of Edinburgh · M ILLC, University of Amsterdam
Human-Alignment, Calibration, and Activation Patterns in Large Language Model Uncertainty
May 29, 2026Kyle Moore, Jesse Roberts, Daryl Watson +2Confidence Estimation in Language Models LLM Interpretability
1Vanderbilt University · 2Tennessee Technological University
Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence?
May 27, 2026Gabrielle Kaili-May Liu, Arman CohanConfidence Estimation in Language Models Language Model Calibration
Yale University
When Confidence Misleads: Suffix Anchoring and Anchor-Proximity Confidence Modulation for Diffusion Language Models
May 27, 2026Jungwon Park, Jimyeong Kim, Jungmin Ko +2Non-Autoregressive Text Generation Language Model Decoding
1RICS · 2AIIS · 3IPAI +2
Do Models Know Why They Changed Their Mind? Interpretability and Faithfulness of Chain-of-Thought Under Knowledge Conflict
May 26, 2026Pruthvinath Jeripity VenkataCoT Faithfulness Faithfulness of Language Model Explanations
Independent Researcher
Asking Is Not Enough: Protocol Sensitivity in LLM Confidence Calibration
May 26, 2026Hankyeol Kim, Pilsung KangLLM Evaluation Confidence Estimation in Language Models
Seoul National University
Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals
May 25, 2026Federico Torrielli, Peter Schneider-Kamp, Lukas Galke PoechConfidence Estimation in Language Models LLM Interpretability
University of Turin · University of Southern Denmark
Second Guess: Detecting Uncertainty Through Abstention and Answer Stability in Small Language Models
May 25, 2026Ashwath Vaithinathan Aravindan, Mayank KejriwalConfidence Estimation in Language Models Multiple-Choice Question Answering
University of Southern California Los Angeles, 90007, California, United States of America · Information Sciences Institute 4676 Admiralty Way #1001, Los Angeles, 90292, California, United States of America
Decision-Making with Lightweight Confidence-Aware Language Model for Autonomous Driving
May 25, 2026Ruoyu Yao, Ruiguo Zhong, Pei Liu +3Confidence Estimation in Language Models Language Model Distillation
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China
Inference Time Optimization with Confidence Dynamics
May 24, 2026Yu Wang, Minghao Liu, Jiayun Wang +3Inference-Time Optimization Confidence Estimation in Language Models
Center for Advanced AI, Accenture.
Trust but Verify: Prover-Verifier Deliberation for Selective LLM Prediction
May 24, 2026João Sedoc, Baotong Zhang, Dean FosterLLM Answer Verification Selective Prediction
New York University
Understanding and Mitigating Premature Confidence for Better LLM Reasoning
May 23, 2026Jingchu Gai, Guanning Zeng, Christina Baek +4Confidence Estimation in Language Models RL for Language Model Reasoning
1Carnegie Mellon University · 2Tsinghua University
LLMs Show No Signs Of Individuated Metacognition
May 22, 2026M. Moran, Mark WhitingLLM Evaluation Confidence Estimation in Language Models
Pareto AI
Calibration vs Decision Making: Revisiting the Reliability Paradox in Unlearned Language Models
May 20, 2026Divyaksh Shukla, Ashutosh ModiConfidence Estimation in Language Models Language Model Calibration
Indian Institute of Technology Kanpur (IIT Kanpur)

Jun 23, 2026, cs.CLJ/K move · Enter open · S save

CALIBER: Calibrating Confidence Before and After Reasoning in Language Models

Conor Finlay, Joshua Kurien, Saurabh Dash, Marzieh Fadaee+1

1Cohere Labs

Confidence Estimation in Language Models Language Model Calibration LLM Uncertainty Estimation Large Reasoning Models

Open paper AskPDF arXiv

Reasoning language models are increasingly asked not only to answer difficult questions, but also to estimate their likelihood of success. Existing methods typically elicit confidence only once: either before thinking or after answering. We argue that confidence in reasoning models is state-dependent: before thinking, confidence should estimate the chance of the model correctly solving the prompt, while after thinking it should predict whether the realized answer is likely to be correct. This distinction determines the appropriate supervision target: prompt-level success should supervise confidence estimates made after seeing the prompt, while individual answer-level correctness should supervise confidence estimates made after answering. We introduce CALIBER (Calibration Before and After Reasoning), which elicits both estimates and supervises each with the target matched to its information state. Under this unified protocol, CALIBER reduces Expected Calibration Error (ECE) by 52.5% over the strongest single-confidence baseline on BigMathDigits for the 7B model, while achieving the best Brier score and AUROC, and remains within 2.1 points of the best accuracy. Further, on a larger 30B model, CALIBER achieves the best ECE on BigMathDigits while remaining competitive in Brier score and AUROC. Out of distribution, it achieves the best ECE and Brier score on GPQA and TriviaQA, and remains competitive on SimpleQA. Ablations further show that this position-target alignment is most beneficial under distribution shift where it consistently reduces calibration error across all out-of-distribution benchmarks.

Momentum

Related topics

Latest papers 134