Verbalized Confidence Estimation

Latest papers

25 papers1 paper in the last four weeks, down 80% on the four weeks before. 0.0% of all new papers.

Measurement Validity and Representation Dynamics of Confidence Signals in Large Language Models
Oct 19, 2026Debojeet Ghosh, Shreyansh Kumar, Ayushman GargConfidence Estimation in Language Models Language Model Calibration
When Words Fall Short: Iterative Synergy Between Verbalized Reasoning and Hidden Features for LLM Confidence Estimation
Sep 28, 2026Yekun Xu, Ante Wang, Jingyi Ren +3Confidence Estimation in Language Models LLM Reliability
College of AI, Tsinghua University, Beijing, China · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China · Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China
An Analysis of Training-Free Self-Reported Confidence in Language Models
Sep 17, 2026Lukas Meyer, Sofia Rossi, Wei Chen +2Confidence Estimation in Language Models Language Model Self-Assessment
DreamAI
The Mirage of Calibrated Confidence: Trajectory-Independence of Verbalized Confidence in Vision-Language Models
Sep 16, 2026Jisoo Yang, Jaeho Han, Trung X. Pham +1VLM Evaluation Confidence Estimation in Language Models
Department of Artificial Intelligence, Chung-Ang University, Republic of Korea · Van Lang University, Ho Chi Minh City, Vietnam
Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models
Sep 10, 2026Yu-Chung HsiaoLanguage Model Calibration LLM-as-a-Judge
Cisco Systems
"very likely" Means "uncertain"? How LLMs Diverge from Humans in Linguistic Uncertainty Quantification
Sep 6, 2026Jinhao Duan, Zicheng Liu, Zijie Liu +2LLM Uncertainty Estimation Verbalized Confidence Estimation
Department of Computer Science, UNC-Chapel Hill · Department of Mathematics, The University of Hong Kong · Department of Data Science, City University of Hong Kong
When Linguistic and Internal Confidence Diverge in Large Language Models
Aug 28, 2026Hefan Zhang, Bingquan Zhang, Ming Cheng +3Confidence Estimation in Language Models Verbalized Confidence Estimation
Dartmouth College · Oakland University
DirEAG: Dirichlet Evidence Aggregation for Calibrating Verbalized Confidence in Mathematical Reasoning
Aug 21, 2026Haorui Xu, Yuzhou Zhu, Liyuan GaoConfidence Estimation in Language Models Confidence Estimation
School of Mathematics, Jilin University, Changchun, China · Leicester International Institute, Dalian University of Technology, Dalian, China
Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classification
Aug 5, 2026Elena Merdjanovska, Omar Zaidan, Andreas RückléLLM Evaluation Selective Prediction
Humboldt-Universität zu Berlin Science of Intelligence · Amazon
Reported Confidence in LLMs Tracks Commitment More Than Correctness
Jun 28, 2026Dharshan KumaranLLM Uncertainty Estimation LLM Interpretability
Google DeepMind
CALIBER: Calibrating Confidence Before and After Reasoning in Language Models
Jun 23, 2026Conor Finlay, Joshua Kurien, Saurabh Dash +2Confidence Estimation in Language Models Language Model Calibration
1Cohere Labs
Speaking in Self-Assessing Tongues: On the Verbalized Confidence of LLMs in Machine Translation
Jun 15, 2026Ali Marashian, Alexis Palmer, Katharina von der WenseConfidence Estimation in Language Models Language Model Calibration
University of Colorado Boulder, USA · Johannes Gutenberg University Mainz, Germany
Calibrating Overconfidence Without Sacrificing Confidence: Probe-Conditioned Head Intervention for LLMs
Jun 2, 2026Ke Li, Chongzhe Zhang, Zifan Zeng +3LLM Reliability Causal Interventions in Language Models
Huawei Heisenberg Research Center · EPFL · TU Berlin +1
Quantifying Faithful Confidence Expression in Large Reasoning Models
Jun 2, 2026Areeb Gani, Asal Meskin, Gabrielle Kaili-May Liu +1Large Reasoning Models Confidence Estimation in Language Models
1Yale University
Asking Is Not Enough: Protocol Sensitivity in LLM Confidence Calibration
May 26, 2026Hankyeol Kim, Pilsung KangLLM Evaluation Confidence Estimation in Language Models
Seoul National University
Retrieval-Augmented Linguistic Calibration
May 19, 2026Yi-Fan Yeh, Linwei Tao, Minjing Dong +4Confidence Estimation in Language Models Faithfulness of Language Model Reasoning
School of Computer Science University of Sydney Sydney, Australia · City University of Hong Kong Hong Kong · Shanghai Jiao Tong University Shanghai, China +1
ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Models
May 12, 2026Chen Li, Xiaoling Hu, Songzhu Zheng +2Confidence Estimation in Language Models Language Model Calibration
Stony Brook University, NY, USA · Massachusetts General Hospital and Harvard Medical School, MA, USA · Morgan Stanley, NY, USA
Measuring Black-Box Confidence via Reasoning Trajectories: Geometry, Coverage, and Verbalization
May 7, 2026Marc Boubnovski Martell, Josefa Lia Stoisser, Kaspar Märtens +4LLM Uncertainty Estimation CoT Reasoning
1Novo Nordisk · University of Oxford
Distilling Self-Consistency into Verbal Confidence: A Pre-Registered Negative Result and Post-Hoc Rescue on Gemma 3 4B
Apr 27, 2026Jon-Paul CacioliSupervised Fine-Tuning Confidence Estimation in Language Models
Independent Researcher, Melbourne, Australia
Verbal Confidence Saturation in 3-9B Open-Weight Instruction-Tuned LLMs: A Pre-Registered Psychometric Validity Screen
Apr 24, 2026Jon-Paul CacioliLLM Evaluation Confidence Estimation in Language Models
Independent Researcher, Melbourne, Australia
Wired for Overconfidence: A Mechanistic Perspective on Inflated Verbalized Confidence in LLMs
Apr 1, 2026Tianyi Zhao, Yinhan He, Wendy Zheng +2Confidence Estimation in Language Models Language Model Calibration
University of Virginia
Closing the Confidence-Faithfulness Gap in Large Language Models
Mar 26, 2026Miranda Muqing Miao, Lyle UngarConfidence Estimation in Language Models Verbalized Confidence Estimation
University of Pennsylvania · Department of Computer and Information Science
How Uncertainty Estimation Scales with Sampling in Reasoning Models
Mar 19, 2026Maksym Del, Markus Kängsepp, Marharyta Domnich +4LLM Uncertainty Estimation Uncertainty Quantification
Institute of Computer Science, University of Tartu
How do LLMs Compute Verbal Confidence
Mar 18, 2026Dharshan Kumaran, Arthur Conmy, Federico Barbero +3Verbalized Confidence Estimation Confidence Estimation in Language Models
Google DeepMind
Rescaling Confidence: What Scale Design Reveals About LLM Metacognition
Mar 10, 2026Yuyang Dai, Yuxia WangConfidence Estimation in Language Models Language Model Calibration
INSAIT

Oct 19, 2026J/K move · Enter open · S save

Measurement Validity and Representation Dynamics of Confidence Signals in Large Language Models

Debojeet Ghosh, Shreyansh Kumar, Ayushman Garg

Confidence Estimation in Language Models Language Model Calibration Verbalized Confidence Estimation Language Model Probing Semantic Entropy

Open paper AskPDF arXiv

Whether a language model "knows what it knows" matters for trustworthy deployment and for deciding when it should abstain. Prior work usually measures this through one channel: verbalized confidence, a linear probe on hidden states, or agreement among sampled answers (semantic entropy), and often treats these as interchangeable. We compare all three in a pre-registered audit of five Qwen2.5 models (0.5B-7B, including a base/instruct pair at 7B) on factual retrieval (PopQA, SimpleQA) and multi-step reasoning (GSM8K, MATH). First, probes trained on prefill states to predict whether sampled answers will disagree become informative earlier for retrieval (about 7% of layer depth on average) than for reasoning (about 28%). The pre-registered five-layer test supported this only marginally; a finer 21-layer follow-up, designed after those results, supports it more clearly. In most retrieval cells, however, a prompt-only TF-IDF classifier matches the probe, so the early signal may reflect surface features of the question. Second, verbalized confidence is unstable: slightly different prompt templates yield scores that barely correlate. Third, Expected Calibration Error is misleading: verbal confidence looks well calibrated because it clusters near the base rate while discriminating poorly, whereas sampling agreement has worse ECE but about twice the resolution. Of five pre-registered hypotheses, two were falsified, one was partly supported, one was not supported, and one could not be tested because the base model's verbal confidence was degenerate.