Language Model Calibration

Follow this topic Ask about this topic

Momentum

43 papers in the last four weeks, up 330% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 247

All topics

Sequential statistical inference for Large Language Models: Representation, validity, and monitoring
May 30, 2026Yao XieLanguage Model Calibration LLM Reliability
H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology
EST-PRM: Stress-Testing Process Reward Models Before They Become Load-Bearing
May 30, 2026Ibne Farabi Shihab, Fariya Afrin, Sanjeda Akter +1Process Reward Models Language Model Calibration
Department of Computer Science, Iowa State University · Department of Computer Science, Kalinga Institute of Industrial Technology · Department of Civil, Construction & Environmental Engineering, Iowa State University
BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on Bengali
May 29, 2026Shefayat E Shams Adib, Ahmed Alfey Sani, Ekramul Alam Esham +3Multilingual Language Model Evaluation Language Model Error Detection
Department of Computer Science and Engineering, Islamic University of Technology · Department of Computer Science and Engineering, University of California
Human-Alignment, Calibration, and Activation Patterns in Large Language Model Uncertainty
May 29, 2026Kyle Moore, Jesse Roberts, Daryl Watson +2Confidence Estimation in Language Models LLM Interpretability
1Vanderbilt University · 2Tennessee Technological University
Counterfactual Graph for Multi-Agent LLM Calibration
May 28, 2026Jiatan Huang, Mingchen Li, Ziming Li +3AI Agent Reliability Language Model Calibration
University of Connecticut · University of Massachusetts, Amherst
Conformal Certification of Reasoning Trace Prefixes
May 28, 2026Matt Y. Cheung, Ashok Veeraraghavan, Hanjie Chen +1Conformal Prediction Language Model Calibration
Department of Electrical & Computer Engineering, Rice University · Department of Computer Science, Rice University
FinVerBench: Benchmark Validity and Calibration in Large Language Model Financial Statement Verification
May 28, 2026Silu PandaLLM Evaluation Language Model Calibration
Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence?
May 27, 2026Gabrielle Kaili-May Liu, Arman CohanConfidence Estimation in Language Models Language Model Calibration
Yale University
Functional Entropy: Predicting Functional Correctness in LLM-Generated Code with Uncertainty Quantification
May 27, 2026Dylan Bouchard, Mohit Singh Chauhan, Zeya Ahmad +1Language Model Calibration LLM Uncertainty Estimation
CVS Health®, Wellesley, MA, USA
Asking Is Not Enough: Protocol Sensitivity in LLM Confidence Calibration
May 26, 2026Hankyeol Kim, Pilsung KangLLM Evaluation Confidence Estimation in Language Models
Seoul National University
Prefix-Safe Bayesian Belief Tracking for LLM Reasoning Reliability:Separating Calibration from Ranking
May 26, 2026Zhenghan Song, Yunyi Li, Yulong LiuBayesian Filtering Language Model Calibration
Cornell University · Columbia University
Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecasting
May 26, 2026Hui Dai, Ryan Teehan, Parsa Torabian +1Post-Hoc Calibration Probabilistic Forecasting
Agentic Learning AI Lab, New York University · Chronologies AI
Using Probabilistic Programs to Train Inductive Reasoning in Large Language Models
May 26, 2026Liyi Zhang, Akshay K. Jagadish, Brenden M. Lake +1Language Model Calibration LLM Uncertainty Estimation
Department of Computer Science Princeton University · Princeton AI Lab Princeton University · Department of Psychology and Computer Science Princeton University
Innovation: An Almost Characterization of Hallucination
May 26, 2026Nishant P. Das, Piyush SrivastavaHallucination in Language Models Language Model Calibration
School of Technology and Computer Science, Tata Institute of Fundamental Research, Mumbai, Maharashtra - 400 005, India.
Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals
May 25, 2026Federico Torrielli, Peter Schneider-Kamp, Lukas Galke PoechConfidence Estimation in Language Models LLM Interpretability
University of Turin · University of Southern Denmark
Toward a Benchmark for Controllable Simulation of Imperfect Students with Large Language Models
May 25, 2026Alexander Apartsin, Omri Sason, Yehudit ApersteinLLM Evaluation Language Model-Based Control
Holon Institute of Technology, Holon, Israel. · Afeka Tel Aviv Academic College of Engineering, Tel Aviv, Israel
Uncertainty Decomposition via Cyclical SG-MCMC and Soft-label Learning for Subjective NLP
May 23, 2026Keito Inoshita, Takato UenoSoft-Label Learning Bayesian Neural Networks
Faculty of Business and Commerce, Kansai University, Suita, Japan · Data Science and AI Innovation Research Promotion Center, Shiga University, Hikone, Japan · Graduate School of Data Science, Shiga University, Hikone, Japan
LLMs Show No Signs Of Individuated Metacognition
May 22, 2026M. Moran, Mark WhitingLLM Evaluation Confidence Estimation in Language Models
Pareto AI
Self-Improving In-Context Learning
May 22, 2026Baturay Saglam, Dionysis KalogeriasLanguage Model Calibration In-Context Learning
Department of Electrical and Computer Engineering Yale University
MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination
May 21, 2026Joss ArmstrongAI Agent Reliability Multi-Agent Coordination
Ericsson, Athlone, Ireland
Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most
May 21, 2026Nick Merrill, Jaeho Lee, Ezra KargerLanguage Model Scaling Laws Forecasting Benchmarks
Forecasting Research Institute · UC Berkeley
Why Semantic Entropy Fails: Geometry-Aware and Calibrated Uncertainty for Policy Optimization
May 20, 2026Zheyuan Zhang, Kaiwen Shi, Han Bao +3RL for Language Model Reasoning Policy Optimization
University of Notre Dame · †Corresponding Author
Calibration vs Decision Making: Revisiting the Reliability Paradox in Unlearned Language Models
May 20, 2026Divyaksh Shukla, Ashutosh ModiConfidence Estimation in Language Models Language Model Calibration
Indian Institute of Technology Kanpur (IIT Kanpur)
Reinforcement Learning over Predictive Distributions for LLM Regression
May 20, 2026Jungsoo Park, Hyungjoo Chae, Ethan Mendes +4Distributional RL RL for Language Models
Georgia Institute of Technology · Allen Institute for AI
BalanceRAG: Joint Risk Calibration for Cascaded Retrieval-Augmented Generation
May 19, 2026Zijun Jia, Yuanchang Ye, Sen Jia +6Retrieval-Augmented Generation Selective Prediction
Beihang University · Zhejiang University of Finance & Economics · Shenzhen Institute of Advanced Technology +1
Retrieval-Augmented Linguistic Calibration
May 19, 2026Yi-Fan Yeh, Linwei Tao, Minjing Dong +4Confidence Estimation in Language Models Language Model Calibration
School of Computer Science University of Sydney Sydney, Australia · City University of Hong Kong Hong Kong · Shanghai Jiao Tong University Shanghai, China +1
Diagnosing Korean-Language LLM Political Bias via Census-Grounded Agent Simulation
May 18, 2026Sungwoo KangPolitical Bias in Language Models Language Model Calibration
Department of Electrical and Computer Engineering Korea University, Seoul, Republic of Korea
ClaHF: A Human Feedback-inspired Reinforcement Learning Framework for Improving Classification Tasks
May 17, 2026Tianxiang Xu, Xiaoyan Zhu, Xin Lai +1Reward Modeling Reinforcement Learning
School of Computer Science and Technology, Xi’an Jiaotong University
Improving Cross-Cultural Survey Simulation with Calibrated Value Personas
May 15, 2026Axel Abels, Elias Fernandez Domingos, Apurva Shah +1Language Model Calibration Cultural Bias in Language Models
1Machine Learning Group, Universit´e Libre de Bruxelles · 2AI Lab, Vrije Universiteit Brussel · 3FARI Institute, Universit´e Libre de Bruxelles - Vrije Universiteit Brussel +2
Calibrating LLMs with Semantic-level Reward
May 15, 2026Fengfei Yu, Ruijia Niu, Dongxia Wu +2Language Model Calibration LLM Reliability
Department of Computer Science and Engineering, University of California San Diego, La Jolla, California, USA · Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, California, USA · Department of Statistics, Stanford University, Stanford, California, USA

May 30, 2026, cs.LGJ/K move · Enter open · S save

Sequential statistical inference for Large Language Models: Representation, validity, and monitoring

Yao Xie

H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology

Language Model Calibration LLM Reliability LLM Uncertainty Estimation

Open paper AskPDF arXiv

This discussion argues that sequential statistical inference can naturally contribute to LLM trustworthiness. In deployment, LLM systems are queried repeatedly, conditioned on evolving contexts, and incorporate user or tool feedback, and may exhibit behavioral shifts after model updates or distribution changes. The discussion is organized around three tasks: representation, modeling LLM interactions as dependent stochastic processes rather than isolated prompt--response pairs; validity, developing uncertainty guarantees that remain meaningful under dependence, repeated use, and adaptation; and monitoring, using sequential alarms and change-point detection to identify shifts in calibration, hallucination rates, refusal behavior, fairness, or other task-relevant properties. This perspective complements recent surveys by viewing trustworthy LLM deployment as a problem of statistical process control.

Momentum

Related topics

Latest papers 247