LLM Uncertainty Estimation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

38 papers in the last four weeks, up 111% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 298

All topics

From token probabilities to calibrated confidence: An empirical study of mathematical question answering
Aug 8, 2026Avery Ma, Lorne Schell, Vin Bhaskara +1LLM Answer Verification Confidence Estimation in Language Models
RBC Borealis
Provable Limits and Certified Deferral for Verbalized Uncertainty in Small Language Models
Aug 5, 2026Jianru ShenSelective Prediction Confidence Estimation in Language Models
University of Montana, Missoula, MT 59812, USA
Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classification
Aug 5, 2026Elena Merdjanovska, Omar Zaidan, Andreas RückléLLM Evaluation Selective Prediction
Humboldt-Universität zu Berlin Science of Intelligence · Amazon
PPDL: LLM-Based Flows as Probabilistic Programs
Aug 5, 2026Louis Mandel, Guillaume Baudart, Mandana Vaziri +1LLM Inference LLM Reliability
IBM, New York, USA · Université Paris Cité, Inria, CNRS, IRIF, France
DUD: Decoupled Update Dynamics for Reliable Uncertainty Quantification in Large Language Models
Aug 4, 2026Yixin Bu, Runze Xia, Guanyun Zou +3LLM Uncertainty Estimation Uncertainty Calibration
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, China · 2The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China · 3Researcher
When Policies Change Probabilities: Modular Decision-Making for LLM Code Review
Aug 2, 2026Rasvik Kudum, Max Corbett, Hitansh Paliwal +3LLM Evaluation LLM Decision-Making
Conformalized Large Language Models under Configuration Shift
Aug 2, 2026Yuqicheng Zhu, Jialin Yu, Lin Li +7Conformal Prediction Language Model Calibration
University of Stuttgart · Robert Bosch GmbH · University of Oxford +4
TrAC: Trace-Conditioned Answer Consistency for Efficient Uncertainty Quantification in LLMs
Aug 1, 2026Dahai Yu, Lin Jiang, Rongchao Xu +1LLM Evaluation Reasoning Consistency in Language Models
Florida State University Tallahassee, Florida, USA
SymboUQ: Symbolic Uncertainty Quantification for Spatial Reasoning in LLMs
Aug 1, 2026Dahai Yu, Lin Jiang, Rongchao Xu +1Uncertainty Quantification LLM Reliability
Florida State University, Tallahassee, Florida
Think Short, Defer Smart, Act, and Repeat: Calibrated Reasoning and Uncertainty-Aware Deferral for Edge LLM Agents
Jul 29, 2026Amirmohammad Farzaneh, Osvaldo SimeoneLLM Inference Efficiency AI Agent Reliability
Institute for Intelligent Networked Systems (INSI) Northeastern University London London, UK
Thinking Under Uncertainty: Evidence Use and Information-Seeking in Language Models
Jul 29, 2026Hua-Dong Xiong, Xinyuan Yan, Ji-An Li +3LLM Decision-Making LLM Uncertainty Estimation
School of Psychological and Brain Sciences, Georgia Tech · Department of Neurosurgery, Baylor College of Medicine · Department of Psychology, New York University +1
Model-Driven Requirements Configuration with Three-Valued Uncertainty Scoring
Jul 28, 2026Ahmed IbrahimLogical Reasoning Requirements Engineering
aWestern University, London, Canada
Spend Experts Where You Are Unsure: Confidence-Adaptive Routing for Mixture-of-Experts LoRA
Jul 28, 2026Tom Saliencro, Rohan Desai, Priya Nair +2Parameter-Free Mixture-of-Experts Routing LLM Inference Acceleration
University of California, Irvine · University of Washington
Beyond Self-Knowledge: Propagating Uncertainty Across Reasoning and Retrieval in LLMs
Jul 28, 2026Chandan Kumar Sah, Li Zhang, Xiaoli LianRetrieval-Augmented Generation Confidence Estimation in Language Models
Beihang University, Beijing, China
Laplace-PSN-IRT: Uncertainty Quantification for Neural Item Response Theory Models of LLM Benchmarks
Jul 28, 2026Juan Francisco, Mandujano ReyesLLM Evaluation Language Model Calibration
Independent Research
Beyond "What to Retrieve": Uncertainty in Retrieval-Augmented Code Generation
Jul 27, 2026Chandan Kumar Sah, Li Zhang, Xiaoli LianRetrieval-Augmented Generation LLM Uncertainty Estimation
Beihang University, Beijing, China
Statistically Supported LLM Ingredient and Recipe Data Collection in Computational Nutrition
Jul 25, 2026James Izzard, Hassan Eshkiki, Fabio CaraffiniLLM Grounding Training Data Curation
Swansea University, Department of Computer Science, Bay Campus, Swansea, SA1 8EN, Wales, UK
The Knowing-Saying Gap: When Probes See Errors that Confidence Misses
Jul 21, 2026Jyotin Goel, Ipshita Bandyopadhyay, Justin ShenkLanguage Model Error Detection Confidence Estimation in Language Models
Calibrating Semantic Uncertainty from Observable Language-Model Probabilities
Jul 20, 2026Matthew F. DixonUncertainty Quantification Confidence Estimation in Language Models
Artificial Intelligence Finance Institute (AIFI), 69 Charlton St, New York, NY 10014, USA
Beyond Semantic Equivalence: Logical Graphs for LLM Uncertainty Quantification
Jul 18, 2026Yanni Dong, Minghua Liu, Meilin Zhu +2Reasoning Consistency in Language Models LLM Reliability
Institute of Intelligent Software, Guangzhou, China · Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Liverpool, Liverpool, UK
The Severance Problem: LLMs are Unaware of the Person Beyond the Prompt
Jul 15, 2026Dor Litvak, Liu LeqiLLM Sycophancy LLM Hallucination Mitigation
The University of Texas at Austin
From Critic to Confidence: PPO for Language-Based Quantitative Prediction with Confidence Estimation
Jul 14, 2026Mehak Dhaliwal, Rasta Tadayon, Andong Hua +2RL for Language Models Confidence Estimation in Language Models
University of California, Santa Barbara
The Computational Basis of Confidence in Large Language Models
Jul 14, 2026Dharshan Kumaran, Viorica Patraucean, Maks Ovsanikov +2Confidence Estimation in Language Models LLM Uncertainty Estimation
Google DeepMind · Google DeepMind and École Polytechnique · École Polytechnique +1
Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agents
Jul 14, 2026Yaopei Zeng, Congchao Wang, JianHang Chen +3AI Agent Reliability LLM Agent Evaluation
Pennsylvania State University · Virginia Tech · Purdue University +1
Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs
Jul 14, 2026Xiaoning Ren, Yinxing Xue, Lei Ma +1Uncertainty Quantification LLM Uncertainty Estimation
Xi’an Jiaotong University, China · Institute of AI for Industries, Chinese Academy of Sciences, China · The University of Tokyo, Japan +1
Confidently Wrong: Detecting Hallucinations in Financial Question Answering from LLM Internal States
Jul 13, 2026Richard Zhe WangFinancial QA LLM Hallucination Detection
St. John Fisher University Rochester, NY, USA
Reliability Scaling Laws for Quantized Large Language Models
Jul 12, 2026Sirine Ayadi, Sándor Daróczi, Stephan Günnemann +1LLM Quantization Language Model Calibration
School of Computation, Information and Technology, Technical University of Munich Munich Data Science Institute · Pruna AI
Hallucination Detection in Large Language Models Using Diversion Decoding
Jul 11, 2026Basel Abdeen, S M Tahmid Siddiqui, Meah Tahmeed Ahmed +4Hallucination in Language Models LLM Hallucination Detection
The University of Texas at Dallas, Richardson, TX 75080, USA · National Institute of Standards and Technology, Gaithersburg, USA · Carnegie Mellon University, PA, USA
Two Axes of LLM Abstention: Answer Correctness and Question Answerability
Jul 9, 2026Benedikt J. WagnerSelective Prediction Confidence Estimation in Language Models
City St George’s, University of London
Eigenvalue Calibration for Semantic Embeddings of Large Language Models
Jul 9, 2026Sebastian G. Gruber, Nassim Walha, Francis Bach +1Confidence Estimation in Language Models Language Model Calibration
1ESAT-PSI, KU Leuven, Belgium · 2German Cancer Research Center (DKFZ), Heidelberg, Germany · 3German Cancer Consortium (DKTK), Germany +3

Aug 8, 2026, cs.LGJ/K move · Enter open · S save

From token probabilities to calibrated confidence: An empirical study of mathematical question answering

Avery Ma, Lorne Schell, Vin Bhaskara, Leila Pishdad

RBC Borealis

LLM Answer Verification Confidence Estimation in Language Models Language Model Calibration LLM Uncertainty Estimation LLM Mathematical Reasoning

Open paper AskPDF arXiv

Confidence estimation for large language models (LLMs) aims to estimate the probability that a generated answer is correct, while calibration aligns these estimates with empirical accuracy. Prior work has shown that token probabilities are often overconfident, we investigate whether these readily available signals can nevertheless provide well-calibrated confidence estimation for mathematical question answering. We compare single-pass estimators, which reuse token probabilities from the original generation, with multi-pass estimators, which obtain additional confidence signals through verification or stochastic forward passes. While individual token probabilities can be highly saturated, we find that aggregating token probabilities over the full sequence captures small but consistent differences between correct and incorrect generations, yielding more informative confidence estimates. Multi-pass methods can yield calibrated confidence estimates. We study two such approaches: self-verification through re-prompting, including a lower-cost in-situ variant, and Monte Carlo Dropout, which derives confidence from variation across stochastic forward passes. We further evaluate two post-hoc calibration methods, Platt scaling and isotonic regression, both of which substantially reduce in-domain calibration error. However, their data efficiency varies with dataset difficulty, and the calibration mappings often transfer asymmetrically across datasets and models.

Momentum

Related topics

Latest papers 298