Language Model Calibration

Ask about this topic

Overview

Language model calibration asks whether a model’s stated or computed confidence corresponds to how often its answers are correct, so systems can decide when to answer, retrieve evidence, defer, or abstain. Researchers estimate confidence from verbal reports, token probabilities, self-consistency, or hidden representations, then improve it with post-hoc methods such as temperature scaling and isotonic regression, conformal prediction, or training methods including supervised fine-tuning and reinforcement learning. Evaluations commonly use question answering and mathematical reasoning benchmarks, and increasingly test retrieval routing, model judges, medical tasks, and performance under domain or prompt shifts. A central tension is that calibration can change with the elicitation protocol and deployment context, and well-calibrated probabilities do not necessarily rank errors well or guarantee safe decisions.

Latest papers

CardsList

386 papers67 papers in the last four weeks, up 40% on the four weeks before. 0.4% of all new papers.

  1. Decision and System One models in rheumatology: accuracy and calibration on specialty-training examination questions and diagnostic case vignettes

    Oct 29, 2026Alfredo Madrid-García, Diego Benavent, Fabian Lechner +3Clinical Language Model EvaluationMedical LLMs

    Independent researcher, Madrid, Spain · Rheumatology Department, Hospital Universitari de Bellvitge, Barcelona, Spain · Institute for Digital Medicine, Marburg University, Marburg, Germany +1

  2. Lev-3B: A Data Recipe for Compact Probabilistic Log Decision Models

    Oct 28, 2026Leo Betthauser, Yingqi Liu, Uday Shah +5Language Model CalibrationCurriculum Learning

    Cisco

  3. Decision Models as Judges for Hallucination Detection

    Oct 27, 2026Haozheng Liu, Zhaoqian Xue, Zhongyin Wang +4LLM Hallucination DetectionLLM-as-a-Judge

    New York University · University of Arizona · University of Illinois Urbana-Champaign +4

  4. Prior Predictive Matching for Bayesian LoRA

    Oct 26, 2026Tomasz Kuśmierczyk, Eliezer de Souza da Silva, Marcin SenderaBayesian LoRABayesian Neural Networks

  5. CalibraRL: Confidence-Calibrated Reinforcement Learning for Mitigating Hallucinations in RAG

    Oct 26, 2026Qianchi Zhang, Hainan Zhang, Liang Pang +2Hallucination in RAGRL for RAG

    School of Artificial Intelligence, Beihang University, Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · Beijing Academy of Blockchain and Edge Computing, Beijing, China

  6. Reinforcement Learning Done Right for Explicit Confidence Targets

    Oct 26, 2026Zefeng He, Yichen He, Yuan Lin +2LLM AbstentionSelective Prediction

    ByteDance Seed · Shanghai Jiao Tong University · NTU

  7. Why Are LLMs Confidently Wrong? Correcting Overconfident Errors via Causal Head Intervention

    Oct 26, 2026Jing Ren, Bowen Li, Ziqi Xu +2Language Model CalibrationConfidence Estimation in Language Models

    RMIT University

  8. SHIFT-JEV: Certified Abstention and Gain-Gated Continual Adaptation for LLM-Based Decision Models under Distribution Shift

    Oct 25, 2026Muhammad Atif Saeed, Azka Atiq, Adarsh Kumar +3Online Language Model AdaptationSelective Prediction

    Department of AI & Data Science, National University of Computer & Emerging Sciences, Islamabad, Pakistan · Data Engineer, University of Arkansas at Little Rock, Arkansas, USA · Department of Computer Science, National University of Computer & Emerging Sciences, Islamabad, Pakistan +2

  9. The Hitchhiker's Guide to System 1 Models: Fast Decision-Making Beyond LLMS

    Oct 24, 2026Haggai RoitmanLanguage Model CalibrationLLM Decision-Making

  10. Small Edits, Shifting Decisions: Benchmarking LLM Accuracy and Reliability on French Input Variants

    Oct 24, 2026Jiaming Yu, Xiaotian Xie, Jian Ma +1Language Model RobustnessLLM Evaluation

    Hangzhou International Innovation Institute, Beihang University, Hangzhou, China · School of Automation, Central South University, Changsha, China · School of Reliability and Systems Engineering, Beihang University, China

  11. Two Uncertainties Are Not One: Disagreement versus Decision-Risk in Cheap Structured Content Moderation

    Oct 23, 2026Karan YadavContent ModerationLLM Evaluation

    Google

  12. Each Insufficient in Its Own Way: Calibrating Language Models Across Varying Evidence States

    Oct 23, 2026Jiayi Li, Dennis L. Shung, Bradly C. StadieConfidence Estimation in Language ModelsLanguage Model Calibration

    Northwestern University · Mayo Clinic

  13. System One models Are Not Better Rerankers

    Oct 23, 2026Michael Dinzinger, Kanishka Ghosh Dastidar, Michael GranitzerLLM RerankingInformation Retrieval

    University of Passau, Passau, Germany · Interdisciplinary Transformation University Austria (IT:U), Linz, Austria

  14. Disentangling Question, Response, and Verifier Languages in Multilingual P(True) Self-Evaluation

    Oct 23, 2026Jewon Yeom, Jaewon Sok, Seonghyeon Park +3Language Model Self-AssessmentMultilingual Language Model Evaluation

    Graduate School of Data Science, Seoul National University

  15. Cost-Aware Abstention for LLM-Based IT Ticket Classification

    Oct 22, 2026Sachin GuptaSelective PredictionText Classification

    Independent Researcher United States

  16. S-EDL: Eliciting Self-Evidence from Sequence Likelihoods for Semantic Calibration of LLMs

    Oct 21, 2026Yawei Li, Jiazheng Li, David Rügamer +3Language Model CalibrationEvidential Deep Learning

    LMU Munich · Munich Center for Machine Learning (MCML) · King’s College London

  17. Auditing Conformal Prediction for Language Models Under Inference-Time Model Shift

    Oct 20, 2026Achref Doula, Max MühlhäuserConformal Prediction for Language ModelsLanguage Model Calibration

    Tongji University, Shanghai, China · Technical University of Darmstadt, Darmstadt, Germany

  18. DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systems

    Oct 20, 2026Xialie Zhuang, Mingcong Lei, Zunhai Su +7LLM EvaluationLLM Decision-Making

    StartLux

  19. BCT: Bayes-Consistent Tempering for Flow Language Models

    Oct 20, 2026Daisuke Oba, Hiroki Furuta, Naoaki OkazakiLanguage ModelingFlow-Based Generative Modeling

    Institute of Science Tokyo · AIST · NII LLMC

  20. Reasoning-Aware Multicalibrated Reliability Predictors for Language Models

    Oct 19, 2026Ekaterina Fedorova, Alessandro Morosini, Chara PodimataLanguage Model CalibrationLLM Reliability

    Massachusetts Institute of Technology.

  21. Training Negatives Still Matter After Calibrating Answerability Readouts

    Oct 19, 2026Jinchen Zhang, Zheyuan Liu, Yihan Zhu +1Selective QALanguage Model Calibration

    University of Notre Dame

  22. Typed Decision Models under Prevalence Shift: Decision Training, Calibration, and Implicit Priors

    Oct 19, 2026Abdul Rahman Iqbal, Muhammad Fawad Nasim, Amir SaleemLanguage Model CalibrationModel Calibration

    Faculty of Computer Science and Information Technology, The Superior University, Lahore, Pakistan · Independent Researcher

  23. Measurement Validity and Representation Dynamics of Confidence Signals in Large Language Models

    Oct 19, 2026Debojeet Ghosh, Shreyansh Kumar, Ayushman GargConfidence Estimation in Language ModelsLanguage Model Calibration

  24. Narrow Decision Models: A System-1 Architecture for Calibrated, Typed Decisions

    Oct 18, 2026Carlos André Antunes, Aldry Albuquerque, Gabriel de Melo Schmith +7LLM Decision-MakingLanguage Model Calibration

    Softplan

  25. System X: An LLM Decision Model That Replies from Its Own KV Cache

    Oct 18, 2026Syed Usama Khalid Bukhari, Syed Safwan Khalid, Muhammad AamirLLM Decision-MakingKV-Cache Reuse

    Turing Logic LTD · University of Oxford

  26. Truth in Contrast: Calibrated Factual Evidence from Language Model Likelihoods

    Oct 16, 2026Ponhvoan Srey, Jason Lucas, Pittawat Taveekitworachai +2Confidence Estimation in Language ModelsLanguage Model Calibration

    Nanyang Technological University · University of Colorado at Boulder · National University of Singapore +1

  27. Literal Lock-In: How Test Language Shifts Literal Attraction in Typed Decision Models on Bengali Idioms

    Oct 15, 2026Adib Sakhawat, Syed Rifat RaiyanCognitive Bias in Language ModelsMultilingual Language Model Evaluation

    Systems and Software Lab (SSL) Department of Computer Science and Engineering Islamic University of Technology, Dhaka, Bangladesh

  28. Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence

    Oct 15, 2026Arijit Dey, Bhartendu Kumar, Ramya Hebbalaguppe +1Confidence Estimation in Language ModelsLanguage Model Calibration

    MBZUAI · TCS Research Labs, India · IIT Delhi

  29. From Unreliable to Usable: Decomposing Failure Modes and Confidence Signals in LLM Entity Extraction - Evidence from Cyber Threat Intelligence

    Oct 14, 2026Jiahong QianLLM Information ExtractionNamed Entity Recognition

    Shanghai Jian Qiao University, Shanghai, China

  30. Single-Pass Decision Scores Enable Rapid Classification with Adjustable Thresholds in Medical Multimodal Large Language Models

    Oct 14, 2026Patrick Wienholt, Frederik Hauke, Christiane Kuhl +3Medical LLMsMultimodal Classification

    Lab for Artificial Intelligence in Medicine, Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany. · Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany. · Department of Medical Oncology, National Center for Tumor Diseases (NCT), Heidelberg University Hospital, Heidelberg, Germany. +2