Language Model Calibration

Overview

Language model calibration asks whether a model’s stated or computed confidence corresponds to how often its answers are correct, so systems can decide when to answer, retrieve evidence, defer, or abstain. Researchers estimate confidence from verbal reports, token probabilities, self-consistency, or hidden representations, then improve it with post-hoc methods such as temperature scaling and isotonic regression, conformal prediction, or training methods including supervised fine-tuning and reinforcement learning. Evaluations commonly use question answering and mathematical reasoning benchmarks, and increasingly test retrieval routing, model judges, medical tasks, and performance under domain or prompt shifts. A central tension is that calibration can change with the elicitation protocol and deployment context, and well-calibrated probabilities do not necessarily rank errors well or guarantee safe decisions.

Latest papers

CardsList

416 papers48 papers in the last four weeks, down 35% on the four weeks before. 0.4% of all new papers.

  1. Stability-Compensated Threshold Calibration for Selection-Conditioned Risk Control in Large Language Models

    Nov 16, 2026Zhiyuan Wang, Yuanchang Ye, Jiahao Yuan +5Selective PredictionLanguage Model Calibration

  2. Benchmarking Large Language Models for Linguistic Annotation

    Nov 16, 2026Ondřej Tichý, Anna Marklová, Barbora Bulantová +2LLM EvaluationLanguage Model Calibration

  3. One-Shot Calibration via Model Editing

    Nov 16, 2026Jiaxing Qiu, Luke Hakso, Roxana Daneshjou +2Model EditingLanguage Model Editing

  4. Can decision models extract clinical data from cancer pathology reports? An evaluation of accuracy, calibration, and failure modes

    Nov 15, 2026Indranil Mallick, Paromita RoyClinical Information ExtractionClinical Language Model Evaluation

  5. Advancing the Calibration-Compute Frontier of LLM Classifiers via Asymmetric Duos

    Nov 13, 2026Berke Akturk, Ali Can Sahin, Rasit Cem Solatay +3Language Model CalibrationLLM Uncertainty Estimation

  6. GraphScent: Calibrating Directional Relevance Signals for Bounded-Context Code Localization

    Nov 12, 2026Ramesh Arvind NaagarajanSoftware EngineeringLanguage Model-Guided Search Heuristics

  7. Learning a calibrated decision head from logged bandit feedback

    Nov 11, 2026Ye YuanLanguage Model CalibrationConfidence Estimation in Language Models

  8. Can We Make an LLM Dumber, or Is It Just Playing Dumb? An Empirical Study of Believable Performance Throttling

    Nov 11, 2026Chengguang Gan, Yunhao Liang, Hanjun Wei +2Language Model Reasoning EvaluationLanguage Model Calibration

  9. Improving Confidence Estimation in Language Models with Linear Representations of Ambiguity

    Nov 9, 2026Themistoklis Nikas, Nishant Subramani, Aaron MuellerConfidence Estimation in Language ModelsLLM Uncertainty Estimation

  10. A Layer-wise Study of How Style Rewriting Breaks the Calibration of Zero-Shot LLM Fake-News Detectors

    Nov 8, 2026Prabhjot Singh, Praney Goyal, Raksha SharmaLanguage Model RobustnessLanguage Model Calibration

  11. When Calibration Data Matter in Sub-4-Bit Language Model Quantization

    Nov 8, 2026Yehong Jiang, Yen-Kuang Chen, Xinmin TianLLM QuantizationPost-Training Quantization

  12. Evaluating Token Probabilities as Confidence Estimators in Human Behavior Simulations: A Case Study in Luxembourgish Next Like Prediction

    Nov 5, 2026Nils Schwager, Simon Münker, Alistair Plum +1Confidence Estimation in Language ModelsLanguage Model Calibration

  13. Zuse: A Decision Model for European Languages

    Nov 4, 2026Umair Mehmood ImamMultilingual Language Model EvaluationLanguage Model Calibration

  14. Calibration-Constrained Diagnostics for Evaluating LLM Self-Evaluation

    Nov 1, 2026Jae Oh Woo, Mengdie Flora Wang, Rahul Ghosh +2Language Model Self-AssessmentConfidence Estimation in Language Models

  15. XDecideBench: Evaluating Typed Decision Models and Confidence Policies Across Languages

    Nov 1, 2026Marek Šuppa, Jaroslav KopčanMultilingual Language Model EvaluationTyped Decision Models

  16. VoirDire: Counterfactual Calibration for Reliable LLM-as-a-Judge Aggregation

    Oct 31, 2026Changho Shin, Frederic SalaLLM-as-a-JudgeBias in LLM-as-a-Judge

  17. Decision and System One models in rheumatology: accuracy and calibration on specialty-training examination questions and diagnostic case vignettes

    Oct 29, 2026Alfredo Madrid-García, Diego Benavent, Fabian Lechner +3Clinical Language Model EvaluationMedical LLMs