LLM Uncertainty Estimation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

38 papers in the last four weeks, up 111% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 298

All topics

Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents
Mar 27, 2026Nicholas Edwards, Sebastian SchusterMulti-Agent LLM Systems AI Coding Agents
Faculty of Computer Science, University of Vienna, Vienna, Austria · UniVie Doctoral School Computer Science, University of Vienna, Vienna, Austria
Do LLMs Know What They Know? Measuring Metacognitive Efficiency with Signal Detection Theory
Mar 26, 2026Jon-Paul CacioliConfidence Estimation in Language Models Language Model Calibration
Independent Researcher Melbourne, Australia
The Anatomy of Uncertainty in LLMs
Mar 26, 2026Aditya Taparia, Ransalu Senanayake, Kowshik Thopalli +1LLM Auditing LLM Reliability
School of Computing and Augmented Intelligence, Arizona State University · Lawrence Livermore National Laboratory
INTRYGUE: Induction-Aware Entropy Gating for Reliable RAG Uncertainty Estimation
Mar 23, 2026Alexandra Kuleshova, Andrei Volodichev, Daria Kotova +1Hallucination Detection LLM Interpretability
Applied AI Institute
Learning When to Sample: Confidence-Aware Selective Sampling for Efficient Chain-of-Thought Reasoning
Mar 9, 2026Juming Xiong, Kevin Guo, Congning Ni +7CoT Reasoning Efficient Language Model Reasoning
1Vanderbilt University · 2Vanderbilt University Medical Center · 3Intuit AI Research
Fine-Grained Uncertainty Quantification for Long-Form Language Model Outputs: A Comparative Study
Feb 19, 2026Dylan Bouchard, Mohit Singh Chauhan, Viren Bajaj +1LLM Hallucination Mitigation Hallucination in Language Models
CVS Health, Wellesley, MA
On Calibration of Large Language Models: From Response To Capability
Feb 14, 2026Sin-Han Yang, Cheng-Kuang Wu, Chieh-Yen Lin +3LLM Evaluation Confidence Estimation in Language Models
Appier AI Research · National Taiwan University
When Agents Disagree With Themselves: Behavioral Consistency as an Uncertainty Signal for LLM Agents
Feb 12, 2026Aman MehtaAI Agent Reliability Selective Prediction
Snowflake AI Research
Decomposed Prompting Does Not Fix Knowledge Gaps, But Helps Models Say "I Don't Know"
Feb 4, 2026Dhruv Madhwal, Lyuxin David Zhang, Dan Roth +2LLM Prompting Question Decomposition
Arizona State University · University of Pennsylvania · Oracle AI
Semantic Self-Distillation for Language Model Uncertainty
Feb 4, 2026Edward Phillips, Sean Wu, Fredrik K. Gustafsson +2Uncertainty Quantification Confidence Estimation in Language Models
Department of Engineering Science, University of Oxford · Oxford Suzhou Centre for Advanced Research, University of Oxford, Suzhou
Uncertainty Localization in LLM Reasoning via Embedding Perturbations
Feb 2, 2026Qihao Wen, Jiahao Wang, Yang Nan +3LLM Reliability LLM Uncertainty Estimation
University of Arizona · Michigan State University
Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations
Jan 12, 2026Yuxi Xia, Dennis Ulmer, Terra Blevins +3LLM Evaluation Confidence Estimation in Language Models
Faculty of Computer Science, UniVie Doctoral School Computer Science · Faculty of Philological and Cultural Studies, University of Vienna, Austria · ILLC, University of Amsterdam, Netherlands +2
Aligning LLMs with Biomedical Knowledge using Balanced Fine-Tuning
Nov 26, 2025Zhenchao Tang, Fang Wang, Haohuai He +13LLM Alignment LLM Fine-Tuning
Tencent AI for Life Sciences Lab, Shenzhen, China. · Sun Yat-sen University, Shenzhen, China. · The Hong Kong Polytechnic University, Hong Kong SAR, China. +2
MedBayes-Lite: A Clinical Uncertainty Governance Layer for Risk-Aware Medical Decision Support
Nov 20, 2025Elias Hossain, Md Mehedi Hasan Nipu, Maleeha Sheikh +5Selective Prediction Healthcare
College of Engineering and Computer Science, University of Central Florida, Orlando, FL 32816, USA · Department of Computer Science and Engineering, North South University, Dhaka 1229, Bangladesh · Department of Electrical and Computer Engineering, Purdue University Fort Wayne, Fort Wayne, IN 46805, USA +4
Sequential Bayesian Evaluation of Large Language Model Behavior
Nov 4, 2025Saatvik Kher, Shang Wu, Rachel Longjohn +2LLM Evaluation Language Model Generation Evaluation
Department of Computer Science, University of California, Irvine · Department of Statistics, University of California, Irvine
When Guessing is Rewarded: Rethinking Language Model Evaluation with Distributional Uncertainty Scoring
Oct 5, 2025Thomas F BurnsLLM Evaluation Language Model Generation Evaluation
Aleph Alpha Research
Geometric Uncertainty for Detecting and Correcting Hallucinations in LLMs
Sep 17, 2025Edward Phillips, Sean Wu, Soheila Molaei +3Uncertainty Quantification LLM Hallucination Mitigation
Department of Engineering Science, University of Oxford · GlaxoSmithKline · Oxford Suzhou Centre for Advanced Research
PiCSAR: Probabilistic Confidence Selection And Ranking for Reasoning Chains
Aug 29, 2025Joshua Ong Jun Leang, Zheng Zhao, Aryo Pradipta Gema +7Confidence Estimation in Language Models Best-of-N Sampling
Imperial College London · University of Edinburgh · UCL
Toward Efficient Uncertainty in LLMs through Evidential Knowledge Distillation
Jul 24, 2025Lakshmana Sri Harsha Nemani, P. K. Srijith, Tomasz KuśmierczykLLM Uncertainty Estimation Uncertainty-Aware Knowledge Distillation
Department of Computer Science and Engineering, Indian Institute of Technology Hyderabad, India · Jagiellonian University, Faculty of Mathematics and Computer Science, Kraków, Poland
On the Effect of Uncertainty on Layer-wise Inference Dynamics
Jul 9, 2025Sunwoo Kim, Haneul Yoo, Alice OhLLM Inference LLM Interpretability
School of Computing, KAIST, Daejeon, South Korea.
Popular but Wrong: Understanding and Mitigating LLM Overconfidence through Knowledge Popularity
May 23, 2025Shiyu Ni, Keping Bi, Jiafeng Guo +1LLM Hallucination Mitigation Confidence Estimation in Language Models
CAS Key Lab of Network Data Science and Technology, ICT, CAS · State Key Laboratory of AI Safety · University of Chinese Academy of Sciences
Label-Confidence-Aware Uncertainty Estimation in Natural Language Generation
Dec 10, 2024Qinhong Lin, Yinglun Feng, Yuhao Zhang +2Uncertainty Quantification LLM Uncertainty Estimation
School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing, China · QuanCheng Laboratory, Jinan, China
Prompting the Unknown: Understanding Response Uncertainty in Large Language Models
Jul 20, 2024Ze Yu Zhang, Arun Verma, Finale Doshi-Velez +1LLM Prompting LLM Reliability
Department of Computer Science, National University of Singapore, Republic of Singapore · Singapore-MIT Alliance for Research and Technology Centre · Harvard University
Evidence for Limited Metacognition in LLMs
Date pendingChristopher AckermanLLM Evaluation LLM Uncertainty Estimation
Towards Reliable Medical LLMs: Benchmarking and Enhancing Confidence Estimation of Large Language Models in Medical Consultation
Date pendingZhiyao Ren, Yibing Zhan, Siyuan Liang +3Healthcare Clinical Reasoning
Nanyang Technological University, Singapore, Singapore. · Wuhan University, Wuhan, China.
Rescaling Confidence: What Scale Design Reveals About LLM Metacognition
Date pendingYuyang Dai, Yuxia WangConfidence Estimation in Language Models Language Model Calibration
INSAIT
Decomposing LLM-Judge Uncertainty to Target Expert Labels
Date pendingRyan LailAnnotator Disagreement LLM-as-a-Judge
Composo AI
ActMap: Single-Pass Uncertainty Quantification from Generation-Time Activation Maps
Date pendingJacopo Dardini (University of Bologna), Roberta Calegari (University of Bologna)Uncertainty Quantification Confidence Estimation in Language Models
University of Bologna, Italy

Mar 27, 2026, cs.CLJ/K move · Enter open · S save

Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents

Nicholas Edwards, Sebastian Schuster

Faculty of Computer Science, University of Vienna, Vienna, Austria · UniVie Doctoral School Computer Science, University of Vienna, Vienna, Austria

Multi-Agent LLM Systems AI Coding Agents LLM Agents LLM Uncertainty Estimation

Open paper AskPDF arXiv

As Large Language Model (LLM) agents are increasingly deployed in open-ended domains like software engineering, they frequently encounter underspecified instructions that lack crucial context. While human developers naturally resolve underspecification by asking clarifying questions, current agents are largely optimized for autonomous execution. In this work, we systematically evaluate the clarification-seeking abilities of LLM agents on an underspecified variant of SWE-bench Verified. We propose an uncertainty-aware multi-agent scaffold that decouples underspecification detection from code execution. Across both proprietary and open-weight frontier LLMs, our scaffold achieves a 69.40% task resolve rate, significantly outperforming a standard single-agent setup and closing the performance gap with agents operating on fully specified instructions. Furthermore, we find that the multi-agent system exhibits well-calibrated information-seeking behavior, conserving queries on simple tasks while proactively seeking information on more complex issues. These findings indicate that current models can be turned into proactive collaborators, where agents independently recognize when to ask questions to elicit missing information in real-world, underspecified tasks.

Momentum

Related topics

Latest papers 298