Confidence Estimation in Language Models

Follow this topic Ask about this topic

Momentum

24 papers in the last four weeks, up 500% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 134

All topics

ProbPlug: A Plugin Uncertainty Network for Reliable Confidence in LLM Binary Classification
Sep 9, 2026Jianzong Wang, Chuhang Liu, Botao Zhao +6Confidence Estimation in Language Models Language Model Calibration
Ping An Technology (Shenzhen) Co., Ltd., Shenzhen, China · Tsinghua Shenzhen International Graduate School, Shenzhen, China · Hubei University of Technology, Hubei, China
Credal Large Language Models for Semantic Commitment under Uncertainty
Aug 24, 2026Shireen Kudukkil Manchingal, Sofiia Nikolenko, Fabio CuzzolinSelective Prediction Confidence Estimation in Language Models
Oxford Dynamics Oxford · Ludwig-Maximilians-Universität München Munich · Institute for Artificial Intelligence, Data Analysis and Systems (AIDAS) School of Engineering Computing & Mathematics Oxford Brookes University, Oxford, UK
Are You Sure You're Sure? On the Impact of Instruction Tuning on Confidence and Lexical Diversity
Aug 13, 2026Irina Proskurina, Mayank Kumar, Oyindolapo O. KomolafeConfidence Estimation in Language Models Question Answering
Cohere Labs Community · Laboratoire Hubert Curien, UMR CNRS 5516, Saint-Étienne, France · School of Computer Science Engineering and Technology (SCSET), Bennett University, Greater Noida, India +1
Beyond Single-Turn Confidence: Trajectory-Adapted Uncertainty Quantification for LLM Agents
Aug 12, 2026Dylan Bouchard, Mohit Singh ChauhanConfidence Estimation in Language Models LLM Agent Evaluation
Attention-Path Fragility as an Uncertainty Signal in Large Language Models
Aug 11, 2026Minsoo Kim, Sungyoung Ji, Kisung Moon +1Confidence Estimation in Language Models LLM Uncertainty Estimation
POSCO Holdings Future Technology Research Institute 440, Teheran-ro, Gangnam-gu, Seoul, Republic of Korea
Unsure but Certain: Uncovering the Representation-Confidence Gap in Diffusion Language Models
Aug 9, 2026Saurabh Yadav, Badri Narayana Patro, Vijay Srinivas AgneeswaranConfidence Estimation in Language Models Language Model Calibration
Microsoft India
From token probabilities to calibrated confidence: An empirical study of mathematical question answering
Aug 8, 2026Avery Ma, Lorne Schell, Vin Bhaskara +1LLM Answer Verification Confidence Estimation in Language Models
RBC Borealis
Provable Limits and Certified Deferral for Verbalized Uncertainty in Small Language Models
Aug 5, 2026Jianru ShenSelective Prediction Confidence Estimation in Language Models
University of Montana, Missoula, MT 59812, USA
Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classification
Aug 5, 2026Elena Merdjanovska, Omar Zaidan, Andreas RückléLLM Evaluation Selective Prediction
Humboldt-Universität zu Berlin Science of Intelligence · Amazon
Can You Trust the Confidence? ConfBench for Vision-Language Models on Document Extraction
Aug 3, 2026Priyashree Roy, Sujitha Martin, Mohammad Rostami +6VLM Evaluation Document Understanding
Amazon Web Services
Beyond Self-Knowledge: Propagating Uncertainty Across Reasoning and Retrieval in LLMs
Jul 28, 2026Chandan Kumar Sah, Li Zhang, Xiaoli LianRetrieval-Augmented Generation Confidence Estimation in Language Models
Beihang University, Beijing, China
Bigger or Cheaper? Scale and Quantization Effects on Uncertainty Signals in Vision-Language Models Under Image Degradation
Jul 27, 2026M M Asif FerdousVLM Robustness Selective Prediction
Independent Researcher
Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation
Jul 24, 2026M M Asif FerdousVLM Robustness Vision-Language Models
Independent Researcher
Sound Probabilistic Safety Bounds for Large Language Models
Jul 22, 2026Mahdi Nazeri, Anne-Kathrin Schmuck, Sadegh Soudjani +1Language Model Safety Evaluation Confidence Estimation in Language Models
Department of Computer Science, University of Oxford, UK · Max Planck Institute for Software Systems, Germany · School of Computer Science, University of Birmingham, UK
The Knowing-Saying Gap: When Probes See Errors that Confidence Misses
Jul 21, 2026Jyotin Goel, Ipshita Bandyopadhyay, Justin ShenkLanguage Model Error Detection Confidence Estimation in Language Models
Calibrating Semantic Uncertainty from Observable Language-Model Probabilities
Jul 20, 2026Matthew F. DixonUncertainty Quantification Confidence Estimation in Language Models
Artificial Intelligence Finance Institute (AIFI), 69 Charlton St, New York, NY 10014, USA
Efficient Sequential Evaluation of Large Language Models
Jul 19, 2026Chia-Yu Hsu, Shubhanshu ShekharLLM Evaluation Confidence Estimation in Language Models
EECS Department, University of Michigan
Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape Pre-, Intra-, and Post-CoT Calibration
Jul 15, 2026Shuhao Li, Guodong Du, Anhao Zhao +3Confidence Estimation in Language Models Language Model Calibration
Eastern Institute of Technology, Ningbo · The Hong Kong Polytechnic University
From Critic to Confidence: PPO for Language-Based Quantitative Prediction with Confidence Estimation
Jul 14, 2026Mehak Dhaliwal, Rasta Tadayon, Andong Hua +2RL for Language Models Confidence Estimation in Language Models
University of California, Santa Barbara
The Computational Basis of Confidence in Large Language Models
Jul 14, 2026Dharshan Kumaran, Viorica Patraucean, Maks Ovsanikov +2Confidence Estimation in Language Models LLM Uncertainty Estimation
Google DeepMind · Google DeepMind and École Polytechnique · École Polytechnique +1
Two Axes of LLM Abstention: Answer Correctness and Question Answerability
Jul 9, 2026Benedikt J. WagnerSelective Prediction Confidence Estimation in Language Models
City St George’s, University of London
Eigenvalue Calibration for Semantic Embeddings of Large Language Models
Jul 9, 2026Sebastian G. Gruber, Nassim Walha, Francis Bach +1Confidence Estimation in Language Models Language Model Calibration
1ESAT-PSI, KU Leuven, Belgium · 2German Cancer Research Center (DKFZ), Heidelberg, Germany · 3German Cancer Consortium (DKTK), Germany +3
Future Confidence Distillation in Large Language Models
Jul 8, 2026Sahil KaleConfidence Estimation in Language Models Language Model Calibration
University of California, Los Angeles Los Angeles, USA
Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs
Jul 7, 2026Andrea Bacciu, Andrea Alfarano, Saab Mansour +2Multilingual Language Model Evaluation Selective Prediction
Amazon · INSAIT, Sofia
SpanUQ: Span-Level Uncertainty Quantification for Large Language Model Generation
Jul 7, 2026Yimeng Zhang, Yingying Zhuang, Ziyi Wang +12LLM Evaluation Confidence Estimation in Language Models
Amazon · Northeastern University
Uncertainty-Aware Abstention in Large Language Models with Provable Alignment Guarantees
Jul 5, 2026Sijin Dong, Hiroyuki ShinnouSelective Prediction Confidence Estimation in Language Models
1Ibaraki University, Japan.
On the effectiveness of reward functions in reinforcement learning for confidence calibration of large language models
Jul 5, 2026Chee Heng Tan, Zhuoyi Lin, Mehul Motani +1Reward Hacking Confidence Estimation in Language Models
School of Computing National University of Singapore · Institute of Advanced Intelligence and Computing Agency for Science, Technology and Research · Department of Electrical & Computer Engineering National University of Singapore
When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs
Jun 29, 2026Zhichao Yang, Caiqi Zhang, Ruihan Yang +3LLM Evaluation Confidence Estimation in Language Models
Fudan University · University of Cambridge
Reported Confidence in LLMs Tracks Commitment More Than Correctness
Jun 28, 2026Dharshan KumaranConfidence Estimation in Language Models LLM Interpretability
Google DeepMind
AB-RAG: Adaptive Budgeted Retrieval-Augmented Generation for Reliable Question Answering
Jun 27, 2026Ansh KamthanRetrieval-Augmented Generation LLM Answer Verification
Department of Artificial Intelligence and Machine Learning Manipal University Jaipur Jaipur, India

Sep 9, 2026, cs.CLJ/K move · Enter open · S save

ProbPlug: A Plugin Uncertainty Network for Reliable Confidence in LLM Binary Classification

Jianzong Wang, Chuhang Liu, Botao Zhao, Zuheng Kang+5

Ping An Technology (Shenzhen) Co., Ltd., Shenzhen, China · Tsinghua Shenzhen International Graduate School, Shenzhen, China · Hubei University of Technology, Hubei, China

Confidence Estimation in Language Models Language Model Calibration LLM Reliability Classification LLM Uncertainty Estimation

Open paper AskPDF arXiv

Large language models (LLMs) have achieved strong performance across a broad range of classification settings, yet the reliability of their predictions remains a major obstacle to deployment in high-stakes scenarios. Although confidence estimation for LLMs has been widely studied, confidence calibration for LLM-based classification remains underexplored. We introduce ProbPlug, a lightweight confidence estimation framework for LLM-based binary classification, which predicts whether an output is correct using internal token features extracted from a frozen LLM. ProbPlug employs a self-attention module to aggregate hidden representations and can be integrated into the original inference pipeline without modifying the base model. Experiments across multiple tasks involving both text-based and multimodal large models show that ProbPlug provides more reliable confidence estimates, improves classification performance with negligible additional overhead, and exhibits strong generalization across tasks. These results indicate that ProbPlug serves as a practical solution for confidence estimation in LLM-based classification. Our code is publicly available at Github.

Momentum

Related topics

Latest papers 134