Language Model Calibration

Follow this topic Ask about this topic

Momentum

43 papers in the last four weeks, up 330% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 247

All topics

Unsure but Certain: Uncovering the Representation-Confidence Gap in Diffusion Language Models
Aug 9, 2026Saurabh Yadav, Badri Narayana Patro, Vijay Srinivas AgneeswaranConfidence Estimation in Language Models Language Model Calibration
Microsoft India
From token probabilities to calibrated confidence: An empirical study of mathematical question answering
Aug 8, 2026Avery Ma, Lorne Schell, Vin Bhaskara +1LLM Answer Verification Confidence Estimation in Language Models
RBC Borealis
Do LLM Recommenders Know When They're Hallucinating? Auditing Confidence Calibration in Catalog Faithfulness
Aug 7, 2026Srijith RavikumarLLM Auditing Language Model Calibration
Amazon.com LLC Seattle, Washington, USA
Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibration
Aug 7, 2026Ruochen Jin, Zhanliang Wang, Zongyu Dai +2LLM Fine-Tuning Language Model Calibration
Dartmouth College, Hanover, NH · University of Pennsylvania, Philadelphia, PA · National University of Singapore, Singapore
Mitigating Scoring Bias in LLM-as-a-Judge via Random Number Generation
Aug 6, 2026Yuma Asato, Kiyoaki Shirai, Natthawut KertkeidkachornLLM Evaluation LLM-as-a-Judge
Japan Advanced Institute of Science and Technology
Provable Limits and Certified Deferral for Verbalized Uncertainty in Small Language Models
Aug 5, 2026Jianru ShenSelective Prediction Confidence Estimation in Language Models
University of Montana, Missoula, MT 59812, USA
Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classification
Aug 5, 2026Elena Merdjanovska, Omar Zaidan, Andreas RückléLLM Evaluation Selective Prediction
Humboldt-Universität zu Berlin Science of Intelligence · Amazon
SocietyBench: Forecasting Counterfactual Social-World Evolution
Aug 4, 2026Zhenran Wang, Zhonghan Bian, Jinsong Li +1Counterfactual Evaluation Forecasting Benchmarks
Quantization Effects on Biomedical LLM Reliability
Aug 4, 2026Anton Rasmussen, Hong QinLLM Quantization Text Classification
Old Dominion University Norfolk, VA 23529, USA
Conformalized Large Language Models under Configuration Shift
Aug 2, 2026Yuqicheng Zhu, Jialin Yu, Lin Li +7Conformal Prediction Language Model Calibration
University of Stuttgart · Robert Bosch GmbH · University of Oxford +4
Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learning
Jul 31, 2026Xujun Che, Yuchen Yuan, Weida Zhao +1Reinforcement Learning KL-Regularized RL
Department of Cybersecurity, University of North Carolina at Charlotte, Charlotte, NC, USA · Department of Information Sciences and Technology, George Mason University, Fairfax, VA, USA · Department of Computer Science and Software Engineering, Auburn University, Auburn, AL, USA +1
Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoning
Jul 31, 2026Xinyan Guan, Jiali Zeng, Chunlei Xin +5Overthinking in Language Models RL for Language Model Reasoning
1Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · 3Weixin AI, Tencent Inc, China
Cybersecurity Detection Classification with Reasoning-enabled Language Models
Jul 30, 2026Amol Khanna, Manu Nandan, Cristian Viorel Popa +10CoT Reasoning Language Model Calibration
CrowdStrike
Selecting Open-Weight Language Models for Zero-Shot Intent Classification: A Systematic Evaluation of 41 Models
Jul 29, 2026Parishruthi Ganesh, Gerry Dozier, Cheryl SealsLLM Evaluation Zero-Shot Text Classification
Department of Computer Science and Software Engineering Auburn University Auburn, Alabama, USA
Beyond the Bidirectional Promise: Re-evaluating the Robustness of Diffusion Language Models
Jul 29, 2026Saurabh Yadav, Badri Narayana Patro, Vijay Srinivas AgneeswaranLanguage Model Calibration Diffusion Language Models
Microsoft
Laplace-PSN-IRT: Uncertainty Quantification for Neural Item Response Theory Models of LLM Benchmarks
Jul 28, 2026Juan Francisco, Mandujano ReyesLLM Evaluation Language Model Calibration
Independent Research
Hierarchical Group-Conditional Conformal Risk Control for Selective Prediction in Language Models
Jul 27, 2026Murilo Salem, Luísa Böhm, Daniel Pontes +1Selective Prediction Group-Conditional Conformal Prediction
CDTec (Centro de Desenvolvimento Tecnológico) Universidade Federal de Pelotas Pelotas, RS, Brasil
A Self-Calibrating Agentic AI Framework for Autonomous Edge Resource Allocation
Jul 24, 2026Fin Gentzen, Marla Grunewald, Iulisloi Zacarias +2Edge Computing Language Model Calibration
Institut für Datentechnik und Kommunikationsnetze Technische Universität Braunschweig, Germany
Artificial Epanorthosis: Why large language models overuse a classical rhetorical figure, and how to mitigate it
Jul 23, 2026Federico BoggiaLanguage Model Calibration
Calibrating Semantic Uncertainty from Observable Language-Model Probabilities
Jul 20, 2026Matthew F. DixonUncertainty Quantification Confidence Estimation in Language Models
Artificial Intelligence Finance Institute (AIFI), 69 Charlton St, New York, NY 10014, USA
A Better Start for Language Models: Domain-Conditional Position Offsets
Jul 17, 2026Ye QiaoDomain Adaptation Language Model Calibration
University of California, Irvine
Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape Pre-, Intra-, and Post-CoT Calibration
Jul 15, 2026Shuhao Li, Guodong Du, Anhao Zhao +3Confidence Estimation in Language Models Language Model Calibration
Eastern Institute of Technology, Ningbo · The Hong Kong Polytechnic University
Temperature Scaling Is Not Enough: Calibration Gaps Under Human Label Distributions
Jul 15, 2026Wisdom DogahPost-Hoc Calibration Language Model Calibration
Faculty of Computing and Mathematical Sciences, University of Mines and Technology (UMaT), Tarkwa, Ghana. · BlackMatrix AI Research, Accra, Ghana.
LLM Judges Can Be Too Generous When There Is No Reference Answer
Jul 14, 2026Chalamalasetti Kranti, Sowmya VajjalaLLM Evaluation LLM-as-a-Judge
Computational Linguistics, Department of Linguistics, University of Potsdam, Germany · National Research Council, Ottawa, Canada
From Critic to Confidence: PPO for Language-Based Quantitative Prediction with Confidence Estimation
Jul 14, 2026Mehak Dhaliwal, Rasta Tadayon, Andong Hua +2RL for Language Models Confidence Estimation in Language Models
University of California, Santa Barbara
CHARM: Charge Calibration and Acoustic Rescue for LLM-based Multimodal Sarcasm Detection
Jul 13, 2026Qiyang Sun, Yi Chang, Yupei Li +3Language Model Calibration Multimodal Sentiment Analysis
GLAM – the Group on Language, Audio, & Music, Imperial College London, UK · College of Telecommunications and Information Engineering, Nanjing University of Posts and Telecommunications, China · College of Computer Science and Electronic Engineering, Hunan University, China +2
Reliability Scaling Laws for Quantized Large Language Models
Jul 12, 2026Sirine Ayadi, Sándor Daróczi, Stephan Günnemann +1LLM Quantization Language Model Calibration
School of Computation, Information and Technology, Technical University of Munich Munich Data Science Institute · Pruna AI
Validity of LLMs as data annotators: AMALIA on authority
Jul 9, 2026Manuel PitaMultilingual Language Model Evaluation Moral Reasoning in Language Models
Artificial Intelligence, Social Interaction and Complexity Laboratory · CICANT, Universidade Lusófona
Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution
Jul 9, 2026Ethan Leung, Elias Lumer, Corey Feld +3LLM-as-a-Judge Citation Verification
Commercial Technology and Innovation Office, PricewaterhouseCoopers, U.S.
Eigenvalue Calibration for Semantic Embeddings of Large Language Models
Jul 9, 2026Sebastian G. Gruber, Nassim Walha, Francis Bach +1Confidence Estimation in Language Models Language Model Calibration
1ESAT-PSI, KU Leuven, Belgium · 2German Cancer Research Center (DKFZ), Heidelberg, Germany · 3German Cancer Consortium (DKTK), Germany +3

Aug 9, 2026, cs.CLJ/K move · Enter open · S save

Unsure but Certain: Uncovering the Representation-Confidence Gap in Diffusion Language Models

Saurabh Yadav, Badri Narayana Patro, Vijay Srinivas Agneeswaran

Microsoft India

Confidence Estimation in Language Models Language Model Calibration LLM Uncertainty Estimation Diffusion Language Models Language Model Robustness

Open paper AskPDF arXiv

Diffusion language models use broad context to create text, suggesting they might handle input noise better than standard models. Testing reveals this is only partially true. Internally, diffusion models detect text errors highly accurately. Externally, their reported certainty ignores this signal. As accuracy drops due to noise, confidence stays near its maximum and the ability to correctly rank answers degrades toward random chance. We call this mismatch the representation confidence gap. The visible concentration of high certainty scores is a misleading surface symptom. Standard math adjustments remove this concentration but fail to fix the underlying loss of ranking order. This ranking deficit favors standard models under noisy conditions and resists common remedies. Matching training recovers accuracy but not ranking, while score recalibration and input level error signals cannot reorder the final answers. However, the information needed to properly evaluate an answer survives in the hidden states. A lightweight extraction tool uses this signal to improve ranking. This approach is highly efficient because it leaves the base model completely frozen and requires zero additional text generation steps. We present this tool to prove the signal exists, while clearly noting its limits. Ultimately, certainty reliability is a more pressing limit than overall accuracy under noisy conditions.

Momentum

Related topics

Latest papers 247