Probability Calibration

Follow this topic Ask about this topic

Probability calibration studies whether predictions made with a given confidence are correct at the corresponding rate, a property that matters when decisions depend on estimated risk rather than just the most likely class. Common approaches include post-hoc methods such as temperature scaling and isotonic regression, recalibration of full predictive distributions, and online or group-conditional methods that seek reliability across changing sequences or subpopulations. Researchers assess calibration with reliability diagnostics and proper scoring rules, and apply these ideas to classifiers, forecasts, medical systems, and language-model agents. A central challenge is that good average calibration may hide subgroup or distribution-shift failures, while improving calibration can interact with accuracy, ranking, and the costs of downstream decisions.

Latest papers

135 papers32 papers in the last four weeks, up 60% on the four weeks before. 0.2% of all new papers.

Whether or Where? Tail Calibration Signals Finite-Budget Reward Hacking in Best-of-N Selection
Oct 25, 2026Cheng Gui, Yihua Shao, Lei Li +2Best-of-N Sampling Reward Over-Optimization
City University of Hong Kong · University of Science and Technology Beijing · University of Washington +1
VX-One: Typed, Calibrated Decision Heads for Real-Time Perimeter Triage
Oct 21, 2026Juan S. SantillanaNetwork Intrusion Detection Probability Calibration
Independent research
When Multiple Choices Are Correct: Asymmetric Probability Allocation in Jev and Other Decision Models
Oct 21, 2026Rushab MunotLLM Decision-Making LLM Reliability
Independent Researcher
Teacher Supervision, Fixed Representations and Calibration: Exact Risk Calculations for Candidate-Scoring Decision Models
Oct 19, 2026Agus Sudjianto, Denis BurakovModel Calibration Probability Calibration
Native Probability Outputs for Clinical Triage: An Evaluation of Calibration and Decision Accuracy
Oct 17, 2026Liang You, Zhiyin Zhou, Dawei Tang +2Clinical Language Model Evaluation Clinical Triage
University of Pittsburgh · Pratt Institute · Icahn School of Medicine at Mount Sinai +2
Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence
Oct 15, 2026Arijit Dey, Bhartendu Kumar, Ramya Hebbalaguppe +1Confidence Estimation in Language Models Language Model Calibration
MBZUAI · TCS Research Labs, India · IIT Delhi
Sharp and Stable Label-Free Calibration Certificates from Logical Coherence
Oct 15, 2026Takahiro KatoProbability Calibration
NP-hard, Inc.
Noisy-Label Calibration with Multiple Unreliable Labels
Oct 14, 2026Keisuke Hanada, Futoshi FutamiModel Calibration Probability Calibration
Department of Biostatistics, Wakayama Medical University, Wakayama, Japan · RIKEN AIP, Tokyo, Japan · Graduate School of Engineering Science, The University of Osaka, Osaka, Japan +1
Recalibrating Many Questions with Few Labels: Minimax Rates and Identification Limits
Oct 13, 2026Xiaoyu Li, Chiwun YangProbability Calibration Minimax Estimation
University of New South Wales. · City University of Hong Kong.
Does Personalization Break Trust? Open-Set Rejection Calibration Under Few-Shot On-Device Intent Adaptation
Oct 12, 2026Vaibhav Prajapati, Pushp GargOpen-Set Recognition OOD Detection
University of Technology Nuremberg, Nuremberg, Germany · Indian Institute of Information Sri City, Chittoor, India
ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty
Oct 12, 2026Yuyang Dai, Yuting Tang, Futoshi Futami +1LLM Uncertainty Estimation Probability Calibration
Florida State University · The University of Tokyo · RIKEN AIP +1
On the Calibration of Crowdsourcing Aggregators
Oct 12, 2026Gabriel Singer, Reda Arab, Maïssa Frikha +4Probability Calibration Post-Hoc Calibration
ENS Paris-Saclay, Université Paris-Saclay, France · SNCF, Paris · CentraleSupélec, Université Paris-Saclay, France +1
QBFF: Deep Quantum Bottleneck for Automatic Speaker Verification Anti-Spoofing
Oct 12, 2026Anton Firc, Nouhaila Innan, Vojtěch Staněk +6Audio Deepfake Detection Speaker Verification
eBRAIN Lab, Division of Engineering, New York University Abu Dhabi (NYUAD), Abu Dhabi, UAE · Center for Quantum and Topological Systems (CQTS), NYUAD Research Institute, NYUAD, Abu Dhabi, UAE · SDU Microelectronics, Institute of Mechanical and Electrical Engineering, University of Southern Denmark, Odense, Denmark
Agreement versus Softmax under Spatial Disambiguation
Oct 9, 2026Michał Zakrzewski, Mariia Shpir, Taras Rumezhak +1Vision-Language Grounding 3D Visual Grounding
AGH University of Krakow, Krakow, Poland · SoftServe · ETH Zurich
Beyond Discrimination: Evaluating Probability Calibration in Radiomics-Image Fusion for Renal Cell Carcinoma Classification
Oct 9, 2026Yuan Liang, Morgan May, Abraham Campbell +3Medical Image Classification Probability Calibration
Research Ireland Centre for Research Training in Machine Learning, ML-Labs, Dublin, Ireland · School of Computer Science, University College Dublin, Dublin, Ireland · School of Computer Science, Technological University Dublin, Dublin, Ireland +2
TabReliBench: Evaluating Tabular Foundation Models Beyond Predictive Accuracy
Oct 9, 2026Tianqi Zhao, Jia Guo, Yan Shuo Tan +1Tabular Foundation Models Probability Calibration
Renmin University of China · Cerebras Systems, Inc. · National University of Singapore +1
Missing Modality-Aware Calibration for Trustworthy Brain Tumor Segmentation
Oct 8, 2026Sol Lee, Hyunji Kim, Sungrae Hong +2Probability Calibration Brain Tumor Segmentation
Korea Advanced Institute of Science and Technology, Daejeon, South Korea
Confidence Reasoning Graphs: Structured Confidence Estimation for LLM Agents
Oct 6, 2026Brendan King, Farima Fatahi Bayat, Jean-Flavien Bussotti +2Probability Calibration Confidence Estimation in Language Models
University of California, Santa Cruz · Megagon Labs
High-dimensional online calibration from harmonic weights
Oct 6, 2026Maxwell Fishelson, Mehryar MohriOnline Calibration Probability Calibration
Institute for Advanced Study. · Google Research and Courant Institute of Mathematical Sciences, New York.
Explicit Asymptotic Bounds for Sequential Calibration Beyond $T^{2/3}$
Oct 6, 2026Eric Dai, Maxwell FishelsonProbability Calibration Probabilistic Forecasting
Edison Academy Magnet School · Institute for Advanced Study
Certification of Real Images through Calibrated Content Authentication
Oct 5, 2026Sarim Hashmi, Abdelrahman Elsayed, Mohammed Talha Alam +2Probability Calibration Digital Image Forensics
Mohamed bin Zayed University of Artificial Intelligence Mabuhay, Abu Dhabi, UAE
Sharp Integrality Gaps in Calibration Distance
Oct 5, 2026Zinan Wang, Xinhao YangProbability Calibration Model Calibration
University of Manchester · University of Southern California
SearchJev: A Calibrated System-1 Model for Search Agents
Oct 4, 2026Congfeng Cao, Lipeng Zuo, Konstantinos Papakostas +7Soft-Label Learning Probability Calibration
Huawei Technologies Co., Ltd. · Leiden University
CalCErt: Bin-wise Certification of Confidence Calibration in Medical Image Classification
Oct 2, 2026Leo Fillioux, Stergios Christodoulidis, Stergios Christodoulidis +2Model Calibration Probability Calibration
Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, CDDU, IHU PRISM, Gif-sur-Yvette · ILLS · LIVIA, ILLS, ETS Montréal
Learning When to Commit from Partial Speech for End-to-End Simultaneous Speech Translation
Oct 2, 2026Hieu Hoang, Amittai AxelrodSpeech Translation Simultaneous Speech Translation
Microsoft
From Retrieval to Typed Decisions: Calibrated System One Models from Biomedical Sentence Encoders
Oct 1, 2026Pritam DekaText Classification Probability Calibration
Queen’s University Belfast Belfast, United Kingdom
Beyond Answer Confidence: A Controlled Audit of Self-Knowledge in a Black-Box Decision Model
Oct 1, 2026Sharath M Shankaranarayana, Davor Runje, Jan JanninkProbability Calibration Black-Box Model Auditing
Reliability-Aware Checkpoint Selection for Domain Generalization
Sep 30, 2026Jinshi Liu, Jiahao Li, Pan Liu +5Probability Calibration Domain Generalization
Shenzhen University · Xiamen University · The Hong Kong University of Science and Technology (Guangzhou) +3
OmniMed-Jev: Calibrating LVLM Confidence for Trustworthy Medical Multimodal Decisions via System One
Sep 30, 2026Luyao Tang, Cheng ChenProbability Calibration Multimodal Classification
Department of Electrical and Computer Engineering The University of Hong Kong, Hong Kong SAR, China
How Much Can Reliability Drift Under a Fixed Confidence Distribution?
Sep 30, 2026Wenhao Liang, Lin Yue, Wei Emma Zhang +3Distribution Shift Robustness Probability Calibration
Adelaide University

Oct 25, 2026J/K move · Enter open · S save

Whether or Where? Tail Calibration Signals Finite-Budget Reward Hacking in Best-of-N Selection

Cheng Gui, Yihua Shao, Lei Li, Jingcai Guo+1

City University of Hong Kong · University of Science and Technology Beijing · University of Washington+1

Best-of-N Sampling Reward Over-Optimization Model Calibration Probability Calibration

Open paper AskPDF arXiv

Best-of-N (BoN) inference returns the candidate with the highest proxy reward, so a larger

N

tests the proxy's extreme tail more than its average ranking quality. We show that the exact BoN value is a Beta-weighted integral of conditional true reward over within-prompt score quantiles. It follows that BoN depends on scores only through their ranks. It also follows that BoN can decline only if the calibration curve in the tail turns downward, and the budget where the decline appears depends on where that turn lies. Under a bounded additive model, the error tail sets the limit: Gaussian errors keep improving, Laplace errors reach a plateau, and heavy-tailed errors return to the prior mean. When error variance is largest at low true reward, even Gaussian noise can push BoN below random selection. Synthetic phase maps agree with this taxonomy. On real candidate pools spanning several verifiable domains, base policies, and learned reward models, tail failures are rare and all occur in code generation. A held-out tail calibration statistic reliably predicts where a cross-fitted budget cap helps. It does about as well as rank accuracy, and the cap is competitive with a Soft-BoN baseline. Tail calibration therefore does not replace ranking metrics. It is an audit that is unchanged by any monotone rescaling of the reward scores and shows which budgets are at risk.