Confidence Estimation in Language Models

Follow this topic Ask about this topic

Momentum

24 papers in the last four weeks, up 500% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 134

All topics

Reading Calibrated Uncertainty from Language Model Trajectories
May 19, 2026Aliai Eusebi, Alexander Herzog, Xiaoyu Liang +3Confidence Estimation in Language Models LLM Reliability
University College London, London, United Kingdom · Core46, London, United Kingdom
Retrieval-Augmented Linguistic Calibration
May 19, 2026Yi-Fan Yeh, Linwei Tao, Minjing Dong +4Confidence Estimation in Language Models Language Model Calibration
School of Computer Science University of Sydney Sydney, Australia · City University of Hong Kong Hong Kong · Shanghai Jiao Tong University Shanghai, China +1
Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution
May 19, 2026Xiaoou Liu, Tiejin Chen, Dengjia Zhang +3Uncertainty Quantification Confidence Estimation in Language Models
1Arizona State University · 2Johns Hopkins University · 3Purdue +1
Position: Uncertainty Quantification in LLMs is Just Unsupervised Clustering
May 19, 2026Tiejin Chen, Longchao Da, Xiaoou Liu +1LLM Evaluation Uncertainty Quantification
School of Computing and Augmented Intelligence, Arizona State University
Confidence Geometry Reveals Trace-Level Correctness in Large Language Model Reasoning
May 16, 2026Shuo Liu, Ding Liu, Shi-Ju RanConfidence Estimation in Language Models LLM Reliability
School of Computer Science and Technology, Tiangong University, Tianjin, China. · Center for Quantum Physics and Intelligent Sciences, Department of Physics, Capital Normal University, Beijing, China.
Margin-Adaptive Confidence Ranking for Reliable LLM Judgement
May 14, 2026Gaojie Jin, Yong Tao, Lijia Yu +1LLM Evaluation LLM-as-a-Judge
Department of Computer Science, University of Exeter · Institute of AI for Industries, Chinese Academy of Sciences · Department of Mathematics and Computer Science, Eindhoven University of Technology
Inducing Artificial Uncertainty in Language Models
May 13, 2026Sophia Hager, Simon Zeng, Nicholas AndrewsConfidence Estimation in Language Models Language Model Calibration
Johns Hopkins University · Microsoft
LLMs as Implicit Imputers: Uncertainty Should Scale with Missing Information
May 13, 2026Stef van BuurenLLM Evaluation Confidence Estimation in Language Models
1. TNO - Netherlands Organization for Applied Scientific Research, Leiden · 2. Dept. of Methodology and Statistics, University of Utrecht
ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Models
May 12, 2026Chen Li, Xiaoling Hu, Songzhu Zheng +2Confidence Estimation in Language Models Language Model Calibration
Stony Brook University, NY, USA · Massachusetts General Hospital and Harvard Medical School, MA, USA · Morgan Stanley, NY, USA
Confidently Deceptive: On the Relationship Between Confidence and Deception in LLMs
May 12, 2026Ali Asad, Stephen Obadinma, Anshul Pattoo +2Language Model Safety Evaluation Confidence Estimation in Language Models
Department of Electrical and Computer Engineering & Ingenuity Labs Research Institute Queen’s University, Kingston, Canada
VERDI: Single-Call Confidence Estimation for Verification-Based LLM Judges via Decomposed Inference
May 11, 2026Jasmine Qi, Danylo Dantsev, Muyang SunPost-Hoc Calibration LLM-as-a-Judge
Indeed Inc.
Grounded or Guessing? LVLM Confidence Estimation via Blind-Image Contrastive Ranking
May 11, 2026Reza Khanmohammadi, Erfan Miahi, Simerjot Kaur +4VLM Evaluation Vision-Language Models
Michigan State University · Independent AI Researcher · JPMorgan AI Research +1
A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering
May 8, 2026Zhanliang Wang, Jiancong Xiao, Ruochen Jin +3LLM Evaluation Confidence Estimation in Language Models
University of Pennsylvania, Philadelphia, PA · Dartmouth College, Hanover, NH
Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs
May 8, 2026Sree Bhattacharyya, Samarth Khanna, Leona Chen +3LLM Evaluation Confidence Estimation in Language Models
Department of Informatics and Intelligent Systems, College of Information Sciences, The Pennsylvania State University. · Department of Computer Science and Engineering, College of Engineering, The Pennsylvania State University.
Measuring Black-Box Confidence via Reasoning Trajectories: Geometry, Coverage, and Verbalization
May 7, 2026Marc Boubnovski Martell, Josefa Lia Stoisser, Kaspar Märtens +4Confidence Estimation in Language Models CoT Reasoning
1Novo Nordisk · University of Oxford
Towards Generation-Efficient Uncertainty Estimation in Large Language Models
May 7, 2026Mingcheng Zhu, Yu Liu, Tingting ZhuLLM Inference Efficiency Uncertainty Quantification
University of Oxford
HyperLens: Quantifying Cognitive Effort in LLMs with Fine-grained Confidence Trajectory
May 7, 2026Chengda Lu, Xiaoyu Fan, Wei XuConfidence Estimation in Language Models LLM Interpretability
1IIIS, Tsinghua University · 2Shanghai Qi Zhi Institute.
Detecting Hallucinations in Large Language Models via Internal Attention Divergence Signals
May 6, 2026Gijs van DijkLanguage Model Error Detection Confidence Estimation in Language Models
Utrecht University
Zero-Shot Confidence Estimation for Small LLMs: When Supervised Baselines Aren't Worth Training
May 4, 2026Luong N. NguyenConfidence Estimation in Language Models Language Model Self-Assessment
Geometry-Calibrated Conformal Abstention for Language Models
Apr 30, 2026Rui Xu, Yi Chen, Sihong Xie +1Uncertainty Quantification Selective Prediction
Information Hub, AI Thrust Hong Kong University of Science and Technology (Guangzhou) Guangzhou, Guangdong, China
Distilling Self-Consistency into Verbal Confidence: A Pre-Registered Negative Result and Post-Hoc Rescue on Gemma 3 4B
Apr 27, 2026Jon-Paul CacioliSupervised Fine-Tuning Confidence Estimation in Language Models
Independent Researcher, Melbourne, Australia
Process Supervision of Confidence Margin for Calibrated LLM Reasoning
Apr 25, 2026Liaoyaqi Wang, Chunsheng Zuo, William Jurayj +2Confidence Estimation in Language Models RL for Language Model Reasoning
Johns Hopkins University
How LLMs Detect and Correct Their Own Errors: The Role of Internal Confidence Signals
Apr 24, 2026Dharshan Kumaran, Viorica Patraucean, Simon Osindero +2LLM Self-Correction Language Model Error Detection
Google DeepMind · Princeton University
Calibration Drift Under Reasoning: How Chain-of-Thought Budgets Induce Overconfidence in Large Language Models
Apr 24, 2026Prakul Sunil Hiremath, Harshit R. HiremathConfidence Estimation in Language Models CoT Reasoning
Department of Computer Science and Engineering Visvesvaraya Technological University, Belagavi · Department of Computer Science and Business System SG Balekundri Institute of Technology, Belagavi
Verbal Confidence Saturation in 3-9B Open-Weight Instruction-Tuned LLMs: A Pre-Registered Psychometric Validity Screen
Apr 24, 2026Jon-Paul CacioliLLM Evaluation Confidence Estimation in Language Models
Independent Researcher, Melbourne, Australia
Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generation
Apr 21, 2026Thomas Zollo, Jimmy Wang, Richard ZemelConfidence Estimation in Language Models Language Model Calibration
Columbia University
Beyond "I Don't Know": Evaluating LLM Self-Awareness in Discriminating Data and Model Uncertainty
Apr 19, 2026Jingyi Ren, Ante Wang, Yunghwei Lai +5LLM Evaluation Selective Prediction
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China
Instinct vs. Reflection: Unifying Token and Verbalized Confidence in Multimodal Large Models
Apr 19, 2026Yunkai Dang, Yifan Jiang, Yizhu Jiang +3Selective Prediction Confidence Estimation in Language Models
School of Artificial Intelligence, Nanjing University.
Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification
Apr 18, 2026Kimia Hamidieh, Veronika Thost, Walter Gerych +2Selective Prediction Confidence Estimation in Language Models
1MIT · MIT-IBM Watson AI Lab · 3WPI +1
Beyond Surface Statistics: Robust Conformal Prediction for LLMs via Internal Representations
Apr 17, 2026Yanli Wang, Peng Kuang, Xiaoyu Han +2Confidence Estimation in Language Models Conformal Prediction
Imperial College London · Zhejiang University · University of Illinois Urbana-Champaign +1

May 19, 2026, cs.LGJ/K move · Enter open · S save

Reading Calibrated Uncertainty from Language Model Trajectories

Aliai Eusebi, Alexander Herzog, Xiaoyu Liang, Marie Vasek+2

University College London, London, United Kingdom · Core46, London, United Kingdom

Confidence Estimation in Language Models LLM Reliability LLM Uncertainty Estimation Representation Geometry in Language Models Uncertainty Calibration Language Model Probing

Open paper AskPDF arXiv

The maximum softmax probability (MSP) represents a default approach when evaluating uncertainty quantification for language model generation with structured output. Although cheap, it is often miscalibrated. Methods that probe the model's internal activations feed raw hidden states into opaque classifiers, reading activations as static snapshots and leaving implicit the layer-wise trajectory by which a representation is formed. Yet, similar endpoints can arise from very different paths, and how evidence accumulates, reinforces, or reverses across depth might reveal uncertainty that final probabilities obscure. We extract eleven scale-invariant geometric features, tracing the cumulative path of per-layer MLP updates, and feed them to a sparse linear probe. The probe outperforms MSP under selective abstention, with gains scaling with baseline miscalibration up to 21 AURC points. Because every feature has a closed-form geometric meaning, the probe's coefficients trace how and where along depth errors take shape -- which layers commit prematurely, which contradict the running state, where trajectories drift away from their endpoint.

Momentum

Related topics

Latest papers 134