Confidence Estimation in Language Models

Overview

The field studies whether language models can tell when their answers are likely to be wrong, which matters for deciding when to answer, abstain, retrieve evidence, or spend more computation. Methods include verbalized confidence, token probabilities and sampling-based agreement, as well as hidden-state probes, activation signals, post-hoc calibration, and conformal prediction. These approaches are tested across settings such as factual question answering, mathematical reasoning, code generation, multilingual tasks, and tool-using agents. A central challenge is that confidence signals can disagree or shift with prompts, training, and distribution changes; good calibration or error ranking alone does not guarantee useful decisions or reliable risk guarantees.

Latest papers

CardsList

271 papers39 papers in the last four weeks, down 13% on the four weeks before. 0.3% of all new papers.

  1. Stability-Compensated Threshold Calibration for Selection-Conditioned Risk Control in Large Language Models

    Nov 16, 2026Zhiyuan Wang, Yuanchang Ye, Jiahao Yuan +5Selective PredictionLanguage Model Calibration

  2. Learning a calibrated decision head from logged bandit feedback

    Nov 11, 2026Ye YuanLanguage Model CalibrationConfidence Estimation in Language Models

  3. BARS: Budget-Aware Sequential Sampling with Refusal for Majority Identification under Unknown Distributions

    Nov 10, 2026Shenxi Liu, Yiwei Li, Mingyang Zhao +2Adaptive SamplingSelective Prediction

  4. Improving Confidence Estimation in Language Models with Linear Representations of Ambiguity

    Nov 9, 2026Themistoklis Nikas, Nishant Subramani, Aaron MuellerConfidence Estimation in Language ModelsLLM Uncertainty Estimation

  5. A Layer-wise Study of How Style Rewriting Breaks the Calibration of Zero-Shot LLM Fake-News Detectors

    Nov 8, 2026Prabhjot Singh, Praney Goyal, Raksha SharmaLanguage Model RobustnessLanguage Model Calibration

  6. Ensemble Methodology for Reference-Free Confidence Estimation in LLM-based Code Summarization

    Nov 6, 2026Marco Colombo, Balaji A Jayaram, Sowmyakumari Selvaraj +5Confidence Estimation in Language ModelsSummarization Evaluation

  7. Evaluating Token Probabilities as Confidence Estimators in Human Behavior Simulations: A Case Study in Luxembourgish Next Like Prediction

    Nov 5, 2026Nils Schwager, Simon Münker, Alistair Plum +1Confidence Estimation in Language ModelsLanguage Model Calibration

  8. When Does Better Confidence Improve a Knowledge-Graph Audit?

    Nov 4, 2026Zehao Lin, Kai Chen, Xixuan Hao +4Knowledge GraphsLLM Auditing

  9. Confidence-Aware Aggregation for Multimodal Audio Reasoning

    Nov 3, 2026Michel Olvera, Paraskevas Stamatiadis, Ga{ë}l RichardAudio-Language ModelsAudio Reasoning

  10. Calibration-Constrained Diagnostics for Evaluating LLM Self-Evaluation

    Nov 1, 2026Jae Oh Woo, Mengdie Flora Wang, Rahul Ghosh +2Language Model Self-AssessmentConfidence Estimation in Language Models

  11. XDecideBench: Evaluating Typed Decision Models and Confidence Policies Across Languages

    Nov 1, 2026Marek Šuppa, Jaroslav KopčanMultilingual Language Model EvaluationTyped Decision Models

  12. Do Language Models Know What Compelling Scientific Evidence Looks Like?

    Oct 29, 2026Conor Igoe, Gabriel Grand, Zane Koch +6Scientific Reasoning in Language ModelsEvidence-Grounded Reasoning

  13. SCARL: Sequence Confidence-Aligned Reinforcement Learning

    Oct 29, 2026Zhenghong Ke, Jingbo Zhou, Hua WuEntropy Collapse in RLReinforcement Learning

  14. Blind Spots: Where Small Language Models Fall Behind Frontier Models on Terminal Tasks

    Oct 28, 2026Ashima Suvarna, Ruta Desai, Ahmed ElgoharySmall Language ModelsLLM Evaluation

  15. Confidence Is Not a Promise: A Reference Audit of Typed Decision Models

    Oct 26, 2026Pedro Carvalho BromTyped Decision ModelsLLM Reliability

  16. Not One Belief: Answer Payloads, Persistence, and Dissociated Confidence Signals in Conversational Question Answering

    Oct 26, 2026Duc-Vinh TranConversational QABelief Updating in LLMs

  17. Test-Time Confidence Adaptation under RL-Induced Trajectory Shift

    Oct 26, 2026Zhenzhen Ren, Xiaoqing Zhang, GuoBiao Li +4Confidence Estimation in Language ModelsInference-Time Adaptation of Language Models

  18. CalibraRL: Confidence-Calibrated Reinforcement Learning for Mitigating Hallucinations in RAG

    Oct 26, 2026Qianchi Zhang, Hainan Zhang, Liang Pang +2Hallucination in RAGRL for RAG

  19. STEP: Soft Trajectory Estimation of Prediction Correctness via Per-Step Probability Scoring

    Oct 25, 2026Chenyang Yu, Zifan Zeng, Qunli Zhang +1Confidence Estimation in Language ModelsLanguage Model Performance Prediction