Language Model Calibration

Follow this topic Ask about this topic

Momentum

43 papers in the last four weeks, up 330% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 247

All topics

CalVerT: Augmenting Agents with Calibrated Verifier Telemetry Improves Action and Learning in Knowledge-Intensive Tasks
Jun 19, 2026Ashwin Vinod, Ying Ding, Elias Stengel-EskinLLM Answer Verification LLM Agent Verification
The University of Texas at Austin
Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software
Jun 18, 2026Arastoo Zibaeirad, Marco VieiraLLM Evaluation Software Vulnerability Detection
University of North Carolina at Charlotte, Charlotte, NC, USA
Confidence Calibration for Multimodal LLMs: An Empirical Study through Medical VQA
Jun 18, 2026Yuetian Du, Yucheng Wang, Ming Kong +4Multimodal QA Language Model Calibration
College of Computer Science and Technology, Zhejiang University · School of Computer Science and Technology, Xidian University · Zhihui Medical Technology (Shanghai) Co., Ltd., Shanghai, China +1
LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Data
Jun 17, 2026Akshat Dasula, Prasanna Desikan, Jaideep SrivastavaLanguage Model Calibration LLM Uncertainty Estimation
Centific AI Research · Department of Computer Science & Engineering, University of Minnesota-Twin Cities, Minneapolis, USA.
MortarBench: Evaluating Mortgage Loan Origination Agents
Jun 17, 2026Matthew Toles, Yunan Lu, Manav Munjal +6Financial Services Language Model Calibration
Columbia University · Tidalwave
Rescaling MLM-Head for Neural Sparse Retrieval
Jun 17, 2026Youngjoon Jang, Seongtae Hong, Jonah Turner +1Masked Language Modeling Learned Sparse Retrieval
Korea University Seoul, South Korea · Independent Seoul, South Korea
Speaking in Self-Assessing Tongues: On the Verbalized Confidence of LLMs in Machine Translation
Jun 15, 2026Ali Marashian, Alexis Palmer, Katharina von der WenseConfidence Estimation in Language Models Language Model Calibration
University of Colorado Boulder, USA · Johannes Gutenberg University Mainz, Germany
The Value Axis: Language Models Encode Whether They're on the Right Track
Jun 15, 2026Nick Jiang, Isaac Kauvar, Jack LindseyConfidence Estimation in Language Models Language Model Calibration
1Stanford University · 2Anthropic
PromptShift-CRC: Drift-Aware Conformal Risk Control for Foundation Models Under Prompt and Domain Shift
Jun 14, 2026Jeffery Opoku, David BanaheneConformal Prediction Language Model Calibration
The University of Texas Rio Grande Valley, Edinburg, TX, USA · Florida International University, Miami, FL, USA
Calibrated Triage, Not Autonomy: Confidence Estimation for Medical Vision-Language Models
Jun 14, 2026Reza Khanmohammadi, Kundan Thind, Mohammad M. GhassemiVLM Evaluation Selective Prediction
Michigan State University East Lansing, MI, USA · Henry Ford Health Detroit, MI, USA
Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity
Jun 12, 2026Daniel Lee, Harsh Sharma, Eunkyu Park +5LLM-as-a-Judge Language Model Calibration
Salesforce AI Research · University of Colorado Boulder · Carnegie Mellon University +2
Closing the Reflection Gap: A Free Calibration Bonus for Agentic RL
Jun 12, 2026Yinglun ZhuLLM Agent Evaluation Language Model Calibration
University of California, Riverside
From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation
Jun 11, 2026Bora Kargi, David SalinasLLM Evaluation Pairwise Comparison
ELLIS Institute Tübingen · OpenEuroLLM
CalBrief: A Pilot Diagnostic Benchmark for Evidence-Calibrated Scientific Briefing with Large Language Models
Jun 11, 2026Yu Fu, Yongqi Kang, Yong ZhaoLLM Evaluation AI-Assisted Scientific Research
Sichuan University
Measuring Human Value Expression in Social Media Texts: Calibrated LLM Annotation and Encoder Transfer
Jun 9, 2026Maria Milkova, Maksim RudnevSoft-Label Learning LLM-Assisted Annotation
Independent researcher, Lisbon, Portugal · University of Waterloo, ON, Canada
TRIAGE: Dialectical LLM Reasoning for Explainable Risk Prediction on Irregularly Sampled Medical Time Series
Jun 8, 2026Hyeongwon Jang, Gyouk Chu, Changhun Kim +4Irregular Time-Series Modeling Logical Reasoning
KAIST · AITRICS · University of Wisconsin-Madison
From Hazard Functions to Language Space: Cox-Supervised Distillation of Survival Risk into a Large Language Model
Jun 8, 2026Nicholas I-Hsien Kuo, Blanca Gallego, Louisa JormSurvival Analysis Language Model Calibration
Centre for Big Data Research in Health, the University of New South Wales, Sydney, Australia
Calibration of Structured Ignorance Certificates for Diagnosing Unknown Unknowns in Reasoning Models
Jun 7, 2026Subramanyam SahooHallucination in Language Models Language Model Calibration
Independent Researcher.
Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method
Jun 6, 2026Kyoungmin Kim, Martin Catheland, Anastasia AilamakiSemantic Search LLM Inference Acceleration
EPFL Lausanne, Switzerland
From 'May' to 'Is': Certainty Distortion in Language Model Rewriting
Jun 6, 2026Catarina G Belem, Shang Wu, Hongyu Yao +3Scientific Communication Language Model Calibration
Department of Computer Science, University of California Irvine · Massachusetts Institute of Technology · Department of Cognitive Science, University of California Irvine
The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and Trust
Jun 5, 2026Nishant Subramani, Palash Goyal, Yiwen Song +4Confidence Estimation in Language Models Language Model Calibration
Carnegie Mellon University · Google · Scale AI
Sycophantic Praise: Evaluating Excessive Praise in Language Models
Jun 5, 2026Daniel Vennemeyer, Phan Anh Duong, Meryl Ye +2LLM Evaluation LLM Sycophancy
University of Cincinnati · Carnegie Mellon University · Texas A&M University
FAIR-Calib: Frontier-Aware Instability-Reweighted Calibration for Post-Training Quantization of Diffusion Large Language Models
Jun 4, 2026Haoyu Huang, Linlin Yang, Sheng Xu +5LLM Quantization Diffusion Model Quantization
National College for Excellent Engineers, Beihang University, Beijing, China · State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing, China · Independent researcher +4
CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction
Jun 4, 2026Mohammad Anas Jawad, Cornelia CarageaConfidence Estimation in Language Models Language Model Calibration
Computer Science, University of Illinois Chicago, USA.
Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data
Jun 3, 2026XiuYu Zhang, Yi Shan, Junfeng Fang +1LLM Evaluation LLM-as-a-Judge
National University of Singapore · Beijing University of Technology
Calibrating Overconfidence Without Sacrificing Confidence: Probe-Conditioned Head Intervention for LLMs
Jun 2, 2026Ke Li, Chongzhe Zhang, Zifan Zeng +3Confidence Estimation in Language Models Language Model Calibration
Huawei Heisenberg Research Center · EPFL · TU Berlin +1
Quantifying Faithful Confidence Expression in Large Reasoning Models
Jun 2, 2026Areeb Gani, Asal Meskin, Gabrielle Kaili-May Liu +1LLM Evaluation Confidence Estimation in Language Models
1Yale University
Conformal Language Modeling via Posterior Sampling
Jun 2, 2026Nicolas Emmenegger, Theo X. Olausson, Armando Solar-Lezama +1Hallucination in Language Models Conformal Prediction
Massachusetts Institute of Technology
Large Language Models Are Overconfident in Their Own Responses
Jun 2, 2026Mario Sanz-Guerrero, Manuel Mager, Katharina von der WenseConfidence Estimation in Language Models LLM Prompting
∇Johannes Gutenberg University Mainz, Germany · ♠University of Colorado Boulder, USA
Evidence-Gated LLM Priors for Multi-Objective Bayesian Optimization
Jun 1, 2026Jiangyu Chen, Ban YiBayesian Optimization Large Language Model-Guided Optimization
State Key Laboratory for Novel Software Technology, Nanjing University

Jun 19, 2026, cs.CLJ/K move · Enter open · S save

CalVerT: Augmenting Agents with Calibrated Verifier Telemetry Improves Action and Learning in Knowledge-Intensive Tasks

Ashwin Vinod, Ying Ding, Elias Stengel-Eskin

The University of Texas at Austin

LLM Answer Verification LLM Agent Verification Language Model Calibration Agentic RAG Knowledge-Intensive QA LLM Agent Reliability

Open paper AskPDF arXiv

LLM agents in knowledge intensive question answering take retrieval and reasoning actions with incomplete knowledge about whether their current answer is uncertain, unsupported, or already complete. This produces two failure modes: committing to confident but unsupported answers, which hurts accuracy, and over-retrieving when the evidence in hand already suffices, resulting in wasted compute. To give agents a more complete picture of the state space they are operating in, we introduce calibrated verifier telemetry (CalVerT), which augments the agent's state with additional telemetry: a calibrated self-confidence score and a grounding verifier score. We show that CalVerT can improve agents in both training-free and training-based settings. On four QA benchmarks, we find that CalVerT raises F1 by triggering retrieval in cases where agents over-rely on parametric knowledge, while cutting redundant retrieval in cases where agents have sufficient context to answer. We show that CalVerT can augment existing QA frameworks without training. Moreover, CalVerT also improves trained systems: by simply augmenting an agent's state with telemetry, we observe improvements after reinforcement learning, as compared to an agent with identical training but no CalVerT telemetry.

Momentum

Related topics

Latest papers 247