LLM Uncertainty Estimation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

38 papers in the last four weeks, up 111% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 298

All topics

Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs
May 8, 2026Sree Bhattacharyya, Samarth Khanna, Leona Chen +3LLM Evaluation Confidence Estimation in Language Models
Department of Informatics and Intelligent Systems, College of Information Sciences, The Pennsylvania State University. · Department of Computer Science and Engineering, College of Engineering, The Pennsylvania State University.
Tracing Uncertainty in Language Model "Reasoning"
May 8, 2026Nils Grünefeld, Bertram Højer, Philipp Mondorf +5LLM Evaluation CoT Reasoning
Data Science Section, IT University of Copenhagen · Pioneer Centre for Artificial Intelligence · MaiNLP, Center for Information and Language Processing, LMU Munich +2
POETS: Uncertainty-Aware LLM Optimization via Compute-Efficient Policy Ensembles
May 8, 2026Nicolas Menet, Andreas Krause, Abbas RahimiThompson Sampling Black-Box Optimization
IBM Research – Zurich · Department of Computer Science, ETH Zürich
Confidence-Aware Alignment Makes Reasoning LLMs More Reliable
May 8, 2026Kejia Chen, Jiawen Zhang, Yihong Wu +5LLM Alignment Direct Preference Optimization
Zhejiang University · Université de Montréal · Sun Yat-sen University +1
Bias and Uncertainty in LLM-as-a-Judge Estimation
May 7, 2026James FiedlerLLM Evaluation LLM-as-a-Judge
Indeed Inc.
Can LLMs Take Retrieved Information with a Grain of Salt?
May 7, 2026Behzad Shayegh, Mohamed Osama Ahmed, Fred Tung +1Retrieval-Augmented Generation LLM Evaluation
RBC Borealis
LLMs are not (consistently) Bayesian: Quantifying internal (in)consistencies of LLMs' probabilistic beliefs
May 7, 2026Chacha Chen, Matthew Jörke, Adam Goliński +4Belief Revision LLM Auditing
1Apple · 2Stanford University · †Work done while an intern at Apple. +1
Rethinking Vacuity for OOD Detection in Evidential Deep Learning
May 7, 2026Claire McNamaraLLM Evaluation Evidential Deep Learning
∗Accenture Labs · †Trinity College Dublin.
Measuring Black-Box Confidence via Reasoning Trajectories: Geometry, Coverage, and Verbalization
May 7, 2026Marc Boubnovski Martell, Josefa Lia Stoisser, Kaspar Märtens +4Confidence Estimation in Language Models CoT Reasoning
1Novo Nordisk · University of Oxford
Towards Generation-Efficient Uncertainty Estimation in Large Language Models
May 7, 2026Mingcheng Zhu, Yu Liu, Tingting ZhuLLM Inference Efficiency Uncertainty Quantification
University of Oxford
Estimating the Black-box LLM Uncertainty with Distribution-Aligned Adversarial Distillation
May 7, 2026Huizi Cui, Huan Ma, Qilin Wang +2Language Model Distillation LLM Uncertainty Estimation
School of Computer Science and Technology, Tianjin University, China · School of Future Technology, Tianjin University, China · Georgia Tech Shenzhen Institute, Tianjin University, China +1
The First Token Knows: Single-Decode Confidence for Hallucination Detection
May 6, 2026Mina GabrielUncertainty Quantification Self-Consistency Decoding
Department of Computer and Information Sciences Temple University Philadelphia, PA 19122, USA
Detecting Hallucinations in Large Language Models via Internal Attention Divergence Signals
May 6, 2026Gijs van DijkLanguage Model Error Detection Confidence Estimation in Language Models
Utrecht University
NoisyCoconut: Counterfactual Consensus via Latent Space Reasoning
May 6, 2026Michael Jerge, David EvansSelective Prediction LLM Inference
University of Virginia
Elicitation Matters: How Prompts and Query Protocols Shape LLM Surrogates under Sparse Observations
May 6, 2026Ge Lei, Samuel J. CooperSurrogate Modeling Bayesian Optimization
Dyson School of Design Engineering Imperial College London Exhibition Road, London SW7 2AZ
Gradients with Respect to Semantics Preserving Embeddings Tell the Uncertainty of Large Language Models
May 6, 2026Mingda Li, Rundong Lv, Xinyu Li +2Uncertainty Quantification LLM Uncertainty Estimation
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, Harbin, China · Suzhou Research Institute, Harbin Institute of Technology, Harbin, China
LLMs Uncertainty Quantification via Adaptive Conformal Semantic Entropy
May 5, 2026Hamed Karimi, Vaishali Meyappan, Reza SamaviSelective Prediction Semantic Entropy
Toronto Metropolitan University, Toronto, Ontario, Canada · Vector Institute, Toronto, Ontario, Canada
Geometric Deviation as an Unsupervised Pre-Generation Reliability Signal: Probing LLM Representations for Answerability
May 4, 2026Yucheng DuRepresentation Geometry LLM Reliability
University of Southern California
Zero-Shot Confidence Estimation for Small LLMs: When Supervised Baselines Aren't Worth Training
May 4, 2026Luong N. NguyenConfidence Estimation in Language Models Language Model Self-Assessment
Feature Rivalry in Sparse Autoencoder Representations: A Mechanistic Study of Uncertainty-Driven Feature Competition in LLMs
May 3, 2026HarshavardhanLLM Interpretability Sparse Autoencoders
Independent Researcher
Hallucinations Undermine Trust; Metacognition is a Way Forward
May 2, 2026Gal Yona, Mor Geva, Yossi MatiasLLM Hallucination Mitigation Hallucination in Language Models
Google Research · Tel Aviv University
CLEAR: Revealing How Noise and Ambiguity Degrade Reliability in LLMs for Medicine
May 1, 2026Kevin H. Guo, Chao Yan, Avinash Baidya +5Healthcare LLM Reliability
Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA · Intuit AI Research, Mountain View, CA, USA · Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, USA +2
"I Don't Know" -- Towards Appropriate Trust with Certainty-Aware Retrieval Augmented Generation
May 1, 2026Daan Di Scala, Maaike de Boer, Pınar YolumRetrieval-Augmented Generation LLM Reliability
TNO Netherlands Organisation for Applied Scientific Research, Department Data Science · Utrecht University, Department of Information and Computing Sciences
Confidence Estimation in Automatic Short Answer Grading with LLMs
Apr 30, 2026Longwei Cong, Sonja Hahn, Sebastian Gombert +3Uncertainty Quantification Selective Prediction
DIPF | Leibniz Institute for Research and Information in Education, 60323 Frankfurt am Main, Germany · Faculty of Computer Science, Goethe University Frankfurt, 60629 Frankfurt am Main, Germany · Chemnitz University of Technology, 09111 Chemnitz, Germany
Belief-Guided Inference Control for Large Language Model Services via Verifiable Observations
Apr 30, 2026Wenhao Yuan, Chenchen Lin, Jian Chen +3LLM Inference Cost-Aware Inference
The University of Hong Kong · Sun Yat-sen University
Entropy Centroids as Intrinsic Rewards for Test-Time Scaling
Apr 28, 2026Wenshuo Zhao, Qi Zhu, Xingshan Zeng +4LLM Inference Test-Time Scaling
May · HKUST · ZJU +1
Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models
Apr 28, 2026Chun-Yi Kuan, Wei-Ping Huang, Hung-yi LeeSemantic Entropy Audio Understanding
Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · Artificial Intelligence Center of Research Excellence (AI-CoRE), National Taiwan University, Taiwan
Quantifying Aleatoric Uncertainty of In-Context Learning for Robust Measure of LLM Prediction Confidence
Apr 28, 2026Jinseok Chung, Minkyoung Song, Hyunji Jung +1Uncertainty Quantification In-Context Learning
POSTECH
Representational Curvature Modulates Behavioral Uncertainty in Large Language Models
Apr 27, 2026Jack King, Evelina Fedorenko, Eghbal A. HosseiniNeural Representation Geometry Autoregressive Language Modeling
Brain and Cognitive Sciences Massachusetts Institute of Technology Cambridge, MA 02139 · Google DeepMind
What Did They Mean? How LLMs Resolve Ambiguous Social Situations across Perspectives and Roles
Apr 27, 2026Qiming Yuan, Linyi Han, Nam Ling +1LLM Uncertainty Estimation Social Reasoning
Santa Clara University, USA

May 8, 2026, cs.CLJ/K move · Enter open · S save

Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs

Sree Bhattacharyya, Samarth Khanna, Leona Chen, Lucas Craig+2

Department of Informatics and Intelligent Systems, College of Information Sciences, The Pennsylvania State University. · Department of Computer Science and Engineering, College of Engineering, The Pennsylvania State University.

LLM Evaluation Confidence Estimation in Language Models Language Model Self-Assessment Language Model Calibration LLM Reliability LLM Uncertainty Estimation

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly used in settings where reliable self-assessment is critical. Assessing model reliability has evolved from using probabilistic correctness estimates to, more recently, eliciting verbalized confidence. Confidence, however, has been shown to be an inconsistent and overoptimistic predictor of model correctness. Drawing on cognitive appraisal theory, a framework from human psychology that decomposes self-evaluation into multiple components, we propose a multidimensional perspective on model self-assessment. We elicit six appraisal-based dimensions of self-assessment, alongside confidence, and evaluate their utility for predicting model failure across 12 LLMs and 38 tasks spanning eight domains. We find that competence-related appraisal dimensions, particularly effort and ability, consistently match or outperform confidence across most settings. Effort additionally yields less overoptimistic estimates that remain stable across model sizes. In contrast, affective dimensions provide marginally predictive signals. Furthermore, the most informative dimension varies systematically with task characteristics: effort is most predictive for reasoning-intensive tasks, while ability and confidence dominate on retrieval-oriented tasks. Broadly, our findings indicate that structured multidimensional self-assessment is a promising approach to improving the reliability and safety of language model deployment across diverse real-world settings.

Momentum

Related topics

Latest papers 298