Language Model Calibration

Follow this topic Ask about this topic

Momentum

43 papers in the last four weeks, up 330% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 247

All topics

LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Data
Jun 17, 2026Akshat Dasula, Prasanna Desikan, Jaideep SrivastavaLanguage Model Calibration LLM Uncertainty Estimation
Centific AI Research · Department of Computer Science & Engineering, University of Minnesota-Twin Cities, Minneapolis, USA.
MortarBench: Evaluating Mortgage Loan Origination Agents
Jun 17, 2026Matthew Toles, Yunan Lu, Manav Munjal +6Financial Services Language Model Calibration
Columbia University · Tidalwave
Rescaling MLM-Head for Neural Sparse Retrieval
Jun 17, 2026Youngjoon Jang, Seongtae Hong, Jonah Turner +1Masked Language Modeling Learned Sparse Retrieval
Korea University Seoul, South Korea · Independent Seoul, South Korea
Speaking in Self-Assessing Tongues: On the Verbalized Confidence of LLMs in Machine Translation
Jun 15, 2026Ali Marashian, Alexis Palmer, Katharina von der WenseConfidence Estimation in Language Models Language Model Calibration
University of Colorado Boulder, USA · Johannes Gutenberg University Mainz, Germany
The Value Axis: Language Models Encode Whether They're on the Right Track
Jun 15, 2026Nick Jiang, Isaac Kauvar, Jack LindseyConfidence Estimation in Language Models Language Model Calibration
1Stanford University · 2Anthropic
PromptShift-CRC: Drift-Aware Conformal Risk Control for Foundation Models Under Prompt and Domain Shift
Jun 14, 2026Jeffery Opoku, David BanaheneConformal Prediction Language Model Calibration
The University of Texas Rio Grande Valley, Edinburg, TX, USA · Florida International University, Miami, FL, USA
Calibrated Triage, Not Autonomy: Confidence Estimation for Medical Vision-Language Models
Jun 14, 2026Reza Khanmohammadi, Kundan Thind, Mohammad M. GhassemiVLM Evaluation Selective Prediction
Michigan State University East Lansing, MI, USA · Henry Ford Health Detroit, MI, USA
Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity
Jun 12, 2026Daniel Lee, Harsh Sharma, Eunkyu Park +5LLM-as-a-Judge Language Model Calibration
Salesforce AI Research · University of Colorado Boulder · Carnegie Mellon University +2
Closing the Reflection Gap: A Free Calibration Bonus for Agentic RL
Jun 12, 2026Yinglun ZhuLLM Agent Evaluation Language Model Calibration
University of California, Riverside
From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation
Jun 11, 2026Bora Kargi, David SalinasLLM Evaluation Pairwise Comparison
ELLIS Institute Tübingen · OpenEuroLLM
CalBrief: A Pilot Diagnostic Benchmark for Evidence-Calibrated Scientific Briefing with Large Language Models
Jun 11, 2026Yu Fu, Yongqi Kang, Yong ZhaoLLM Evaluation AI-Assisted Scientific Research
Sichuan University
Measuring Human Value Expression in Social Media Texts: Calibrated LLM Annotation and Encoder Transfer
Jun 9, 2026Maria Milkova, Maksim RudnevSoft-Label Learning LLM-Assisted Annotation
Independent researcher, Lisbon, Portugal · University of Waterloo, ON, Canada
TRIAGE: Dialectical LLM Reasoning for Explainable Risk Prediction on Irregularly Sampled Medical Time Series
Jun 8, 2026Hyeongwon Jang, Gyouk Chu, Changhun Kim +4Irregular Time-Series Modeling Logical Reasoning
KAIST · AITRICS · University of Wisconsin-Madison
From Hazard Functions to Language Space: Cox-Supervised Distillation of Survival Risk into a Large Language Model
Jun 8, 2026Nicholas I-Hsien Kuo, Blanca Gallego, Louisa JormSurvival Analysis Language Model Calibration
Centre for Big Data Research in Health, the University of New South Wales, Sydney, Australia
Calibration of Structured Ignorance Certificates for Diagnosing Unknown Unknowns in Reasoning Models
Jun 7, 2026Subramanyam SahooHallucination in Language Models Language Model Calibration
Independent Researcher.
Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method
Jun 6, 2026Kyoungmin Kim, Martin Catheland, Anastasia AilamakiSemantic Search LLM Inference Acceleration
EPFL Lausanne, Switzerland
From 'May' to 'Is': Certainty Distortion in Language Model Rewriting
Jun 6, 2026Catarina G Belem, Shang Wu, Hongyu Yao +3Scientific Communication Language Model Calibration
Department of Computer Science, University of California Irvine · Massachusetts Institute of Technology · Department of Cognitive Science, University of California Irvine
The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and Trust
Jun 5, 2026Nishant Subramani, Palash Goyal, Yiwen Song +4Confidence Estimation in Language Models Language Model Calibration
Carnegie Mellon University · Google · Scale AI
Sycophantic Praise: Evaluating Excessive Praise in Language Models
Jun 5, 2026Daniel Vennemeyer, Phan Anh Duong, Meryl Ye +2LLM Evaluation LLM Sycophancy
University of Cincinnati · Carnegie Mellon University · Texas A&M University
FAIR-Calib: Frontier-Aware Instability-Reweighted Calibration for Post-Training Quantization of Diffusion Large Language Models
Jun 4, 2026Haoyu Huang, Linlin Yang, Sheng Xu +5LLM Quantization Diffusion Model Quantization
National College for Excellent Engineers, Beihang University, Beijing, China · State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing, China · Independent researcher +4
CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction
Jun 4, 2026Mohammad Anas Jawad, Cornelia CarageaConfidence Estimation in Language Models Language Model Calibration
Computer Science, University of Illinois Chicago, USA.
Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data
Jun 3, 2026XiuYu Zhang, Yi Shan, Junfeng Fang +1LLM Evaluation LLM-as-a-Judge
National University of Singapore · Beijing University of Technology
Calibrating Overconfidence Without Sacrificing Confidence: Probe-Conditioned Head Intervention for LLMs
Jun 2, 2026Ke Li, Chongzhe Zhang, Zifan Zeng +3Confidence Estimation in Language Models Language Model Calibration
Huawei Heisenberg Research Center · EPFL · TU Berlin +1
Quantifying Faithful Confidence Expression in Large Reasoning Models
Jun 2, 2026Areeb Gani, Asal Meskin, Gabrielle Kaili-May Liu +1LLM Evaluation Confidence Estimation in Language Models
1Yale University
Conformal Language Modeling via Posterior Sampling
Jun 2, 2026Nicolas Emmenegger, Theo X. Olausson, Armando Solar-Lezama +1Hallucination in Language Models Conformal Prediction
Massachusetts Institute of Technology
Large Language Models Are Overconfident in Their Own Responses
Jun 2, 2026Mario Sanz-Guerrero, Manuel Mager, Katharina von der WenseConfidence Estimation in Language Models LLM Prompting
∇Johannes Gutenberg University Mainz, Germany · ♠University of Colorado Boulder, USA
Evidence-Gated LLM Priors for Multi-Objective Bayesian Optimization
Jun 1, 2026Jiangyu Chen, Ban YiBayesian Optimization Large Language Model-Guided Optimization
State Key Laboratory for Novel Software Technology, Nanjing University
Identifying High-Confidence Social Biases in LLMs for Trustworthy Conversational Tutoring Agents
Jun 1, 2026Aitor Arronte Alvarez, Naiyi Xie FinchamIntelligent Tutoring Systems Social Bias in Language Models
University of Hawaii at Manoa, Honolulu HI 96822, USA
Low-Resource Safety Failures Are Action Failures, Not Representation Failures
May 31, 2026Rashad Aziz, Ikhlasul Akmal Hanif, Fajri KotoLanguage Model Calibration LLM Safety
Mohamed bin Zayed University of Artificial Intelligence
A Finite-Calibration Regime Map for LLM Judge Panels
May 31, 2026Bin Zhu, Yanghui RaoLLM-as-a-Judge Language Model Calibration
School of Computer Science and Engineering Sun Yat-sen University

Jun 17, 2026, cs.AIJ/K move · Enter open · S save

LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Data

Akshat Dasula, Prasanna Desikan, Jaideep Srivastava

Centific AI Research · Department of Computer Science & Engineering, University of Minnesota-Twin Cities, Minneapolis, USA.

Language Model Calibration LLM Uncertainty Estimation Clinical Language Model Evaluation Epistemic Uncertainty

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly applied to structured clinical data, yet whether they can recognize the limits of their own knowledge on such tasks remains unexplored. We study this question through the lens of cross-model attribution divergence with the goal of reducing epistemic uncertainty for structured tasks, comparing Qwen 2.5 7B and XGBoost on a prediction task via attribution divergence analysis. We report four findings. First, LLM verbalized confidence is epistemically vacuous, it outputs a near-constant (0.856-0.937) regardless of whether accuracy is 49% or 75.3%, tracking prompt format rather than prediction quality. Second, the LLM exhibits an inverse difficulty effect: accuracy drops to 64.8% when XGBoost is 99% correct, but matches XGBoost (73.8% vs. 73.1%) when it is moderately uncertain. Third, few-shot examples and SHAP-derived feature evidence are orthogonal, super-additive interventions: they reduce the Attribution Disagreement Score (ADS) from 1.54 to 0.38 and improve accuracy from 49% to 75.3% without training. Fourth, a cross-model calibrator that determined LLM reliability using attribution divergence signals reduces expected calibration error from 0.254 to 0.080, replacing uninformative verbalized confidence with patient-specific reliability estimates, without accessing model internals or requiring repeated inference. We frame these findings as a cold start problem for LLMs on structured data and outline a path toward genuine epistemic self-awareness.

Momentum

Related topics

Latest papers 247