Probability Calibration

Follow this topic Ask about this topic

Probability calibration studies whether predictions made with a given confidence are correct at the corresponding rate, a property that matters when decisions depend on estimated risk rather than just the most likely class. Common approaches include post-hoc methods such as temperature scaling and isotonic regression, recalibration of full predictive distributions, and online or group-conditional methods that seek reliability across changing sequences or subpopulations. Researchers assess calibration with reliability diagnostics and proper scoring rules, and apply these ideas to classifiers, forecasts, medical systems, and language-model agents. A central challenge is that good average calibration may hide subgroup or distribution-shift failures, while improving calibration can interact with accuracy, ranking, and the costs of downstream decisions.

Latest papers

136 papers32 papers in the last four weeks, up 60% on the four weeks before. 0.2% of all new papers.

Towards Reliable Simulation-based Inference
Mar 9, 2026Arnaud DelaunoySimulation-Based Inference Probability Calibration
Department of Electrical Engineering and Computer Science, Faculty of Applied Sciences
Micro-Diffusion Compression - Binary Tree Tweedie Denoising for Online Probability Estimation
Mar 9, 2026Roberto TacconelliLossless Data Compression Probability Calibration
Independent Researcher
MedQ-Deg: A Multidimensional Benchmark for Evaluating MLLMs Across Medical Image Quality Degradations
Mar 8, 2026Jiyao Liu, Junzhi Ning, Chenglong Ma +14MLLM Evaluation Medical VLMs
Fudan University, Shanghai, China · Shanghai AI Lab, Shanghai, China · Shanghai Jiaotong University, Shanghai, China +1
Discrete Tokenization Unlocks Transformers for Calibrated Tabular Forecasting
Mar 8, 2026Yael S. ElmatadTransformer Probability Calibration
An intuitive rearranging of the Yates covariance decomposition for probabilistic verification of forecasts with the Brier score
Mar 4, 2026Bruno Hebling VieiraProbability Calibration Probabilistic Forecasting
Methods of Plasticity Research, Department of Psychology, University of Zurich, Zurich, Switzerland
V-ECE: Estimating General Expected Calibration Errors
Feb 27, 2026Eugène Berta, Sacha Braun, Francis Bach +2Probability Calibration Uncertainty Calibration
INRIA · Ecole Normale Sup´erieure, PSL Research University · UC Berkeley
Tipping the Balance: Impact of Class Imbalance Correction on the Performance of Clinical Risk Prediction Models
Feb 27, 2026Amalie Koch Andersen, Hadi Mehdizavareh, Arijit Khan +8Class-Imbalanced Learning Probability Calibration
Medical Informatics and Image Analysis, Department of Health Science and Technology, Aalborg University, Denmark. · Medical & Translational Science, Novo Nordisk A/S, Søborg, Denmark · Department of Computer Science, Aalborg University, Aalborg, Denmark +10
Epistemic Filtering and Collective Hallucination: A Jury Theorem for Confidence-Calibrated Agents
Feb 25, 2026Jonas KargeSocial Choice Theory AI Agent Reliability
Technische Universität Dresden
Multiclass Calibration Assessment and Recalibration of Probability Predictions via the Linear Log Odds Calibration Function
Feb 20, 2026Amy Vennos, Xin Xing, Christopher T. FranckProbability Calibration Model Calibration
Department of Statistics, Virginia Polytechnic Institute and State University
Diagnosing Corruption-Induced Reliability Failures in Vision-Language Models
Nov 24, 2025Xiangjie Sui, Songyang Li, Hanwei Zhu +3VLM Evaluation VLM Robustness
City University of Macau, Macau SAR, China · Jiangxi University of Finance and Economics, Nanchang, China · Nanyang Technological University, Singapore
Risk-Calibrated Bayesian Streaming Intrusion Detection with SRE-Aligned Decisions
Sep 17, 2025Michel A. YoussefCost-Sensitive Learning Probability Calibration
Independent Researcher · Independent Researcher, Lebanon
On the Reliability and Stability of Selective Methods in Malware Classification Tasks
May 28, 2025Alexander Herzog, Aliai Eusebi, Lorenzo CavallaroMalware Classification Selective Prediction
University College London and core64 London, United Kingdom · University College London London, United Kingdom
Optimal and Provable Calibration in High-Dimensional Binary Classification: Angular Calibration and Platt Scaling
Feb 21, 2025Yufan Li, Pragya SurProbability Calibration Binary Classification
Department of Statistics Harvard University Cambrdige, MA 02138
Breaking the $T^{2/3}$ Barrier for Sequential Calibration
Jun 19, 2024Yuval Dagan, Constantinos Daskalakis, Maxwell Fishelson +3Probability Calibration
Tel Aviv University · MIT & Archimedes AI · MIT +1
A probability calibrator for machine-learning classifiers
Feb 21, 2023K. Dyrland, A. S. Lundervold, P. G. L. Porta ManaProbability Calibration
Dept of Computer science, Electrical Engineering and Mathematical Sciences, Western Norway University of Applied Sciences, Bergen, Norway · †& Mohn Medical Imaging and Visualization Centre, Dept of Radiology, Haukeland University Hospital, Bergen, Norway
MetaMax: Improved Open-Set Deep Neural Networks via Weibull Calibration
Nov 20, 2022Zongyao Lyu, Nolan B. Gutierrez, William J. BeksiProbability Calibration Open-Set Recognition
The University of Texas at Arlington Arlington, TX, USA

Mar 9, 2026J/K move · Enter open · S save

Towards Reliable Simulation-based Inference

Arnaud Delaunoy

Department of Electrical Engineering and Computer Science, Faculty of Applied Sciences

Simulation-Based Inference Probability Calibration Bayesian Neural Networks Density Ratio Estimation

Open paper AskPDF arXiv

Scientific knowledge expands by observing the world, hypothesizing some theories about it, and testing them against collected data. When those theories take the form of statistical models, statistical analyses are involved in the process of testing and refining scientific hypotheses. In this thesis, we focus on statistical models that take the form of scientific simulators and provide background about how machine learning can be used for statistical analyses in this context. The first part of this thesis is about showing empirically that performing statistical analyses with machine learning involves a degree of approximation. Specifically, all statistical analyses involve a level of uncertainty in the conclusions drawn, and we show that approximations can lead to overconfident conclusions. We draw caution regarding such overconfident conclusions and introduce a criterion to diagnose overconfident approximations. In the second part, we introduce balancing, a way to regularize machine learning models to reduce overconfidence and favor calibrated or underconfident approximations. Balancing is first introduced for neural ratio estimation algorithms and then extended to other algorithms. Intuition about why balancing leads to less overconfident solutions is provided, and it is shown empirically that balanced algorithms are often either close to calibrated or underconfident. The third part shows that Bayesian neural networks can also be used to mitigate the overconfidence of approximations. Unlike balancing, no regularization is required, and this solution can then work with few training samples and, hence, computationally expensive simulators. To that end, a new Bayesian neural network prior tailored for simulation-based inference is developed, and empirical results show a reduction in overconfidence compared to similar solutions without Bayesian neural networks.