Language Model Calibration

Follow this topic Ask about this topic

Momentum

43 papers in the last four weeks, up 330% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 247

All topics

NOVA: NOise-aware Verbal Confidence CAlibration for Robust Large Language Models in RAG Systems
Jan 16, 2026Jiayu Liu, Rui Wang, Qing Zong +9Retrieval-Augmented Generation Confidence Estimation in Language Models
HKUST · UIUC · Northwestern University
Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations
Jan 12, 2026Yuxi Xia, Dennis Ulmer, Terra Blevins +3LLM Evaluation Confidence Estimation in Language Models
Faculty of Computer Science, UniVie Doctoral School Computer Science · Faculty of Philological and Cultural Studies, University of Vienna, Austria · ILLC, University of Amsterdam, Netherlands +2
Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Models
Oct 3, 2025Ej Zhou, Caiqi Zhang, Tiancheng Hu +4Multilingual Language Models Multilingual Language Model Evaluation
Language Technology Lab, University of Cambridge
Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
Jul 29, 2025Carel van Niekerk, Renato Vukovic, Benjamin Ruppik +3RL for Language Models RL for Language Model Reasoning
Heinrich Heine University Düsseldorf, Germany
Rescaling Confidence: What Scale Design Reveals About LLM Metacognition
Date pendingYuyang Dai, Yuxia WangConfidence Estimation in Language Models Language Model Calibration
INSAIT
How Proper Scoring Rules Shape LLM Forecasting
Date pendingBenjamin Turtel, Paul Wilczewski, Kris Skotheim +2Probabilistic Forecasting Proper Scoring Rules
Lightning Rod Labs · Technology and Operations Management, INSEAD, Fontainebleau, France · Wharton School and School of Arts & Sciences, University of Pennsylvania
ActMap: Single-Pass Uncertainty Quantification from Generation-Time Activation Maps
Date pendingJacopo Dardini (University of Bologna), Roberta Calegari (University of Bologna)Uncertainty Quantification Confidence Estimation in Language Models
University of Bologna, Italy

Jan 16, 2026, cs.CLJ/K move · Enter open · S save

NOVA: NOise-aware Verbal Confidence CAlibration for Robust Large Language Models in RAG Systems

Jiayu Liu, Rui Wang, Qing Zong, Yumeng Wang+8

HKUST · UIUC · Northwestern University

Retrieval-Augmented Generation Confidence Estimation in Language Models Language Model Calibration LLM Reliability Language Model Robustness

Open paper AskPDF arXiv

Accurately assessing model confidence is essential for deploying large language models (LLMs) in mission-critical factual domains. While retrieval-augmented generation (RAG) is widely adopted to improve grounding, confidence calibration in RAG settings remains poorly understood. We conduct a systematic study across four benchmarks, revealing that LLMs exhibit poor calibration performance especially when noisy contexts are retrieved. Specifically, contradictory or irrelevant evidence tends to exacerbate the model's overconfidence issue. To address this, we propose NOVA Rules (NOise-Aware Verbal Confidence CAlibration Rules) to provide a principled foundation for resolving overconfidence under noise. We further design NOVA, a noise-aware calibration framework that synthesizes supervision from ~2K HotpotQA examples guided by these rules. By performing supervised fine-tuning (SFT) with this data, NOVA equips models with intrinsic noise awareness without relying on stronger teacher models. Empirical results show that NOVA yields substantial gains, improving ECE scores by 10.9% in-domain and 8.0% out-of-domain. By bridging the gap between retrieval noise and verbal calibration, NOVA paves the way for both accurate and epistemically reliable LLMs.

Momentum

Related topics

Latest papers 247