Model Calibration

Follow this topic Ask about this topic

For classifiers and language models, calibration asks whether confidence matches empirical correctness and develops post-hoc methods such as temperature scaling, isotonic regression, and local corrections, alongside training-time objectives. Online and multicalibration methods extend reliability to changing data, subgroups, and selective prediction, while conformal methods provide coverage guarantees for prediction sets or intervals. A related line calibrates mechanistic models and simulators to measurements using Bayesian inference, evolutionary optimization, differentiable surrogates, and data assimilation in areas such as climate, traffic, medicine, and engineering. Across settings from CIFAR and ImageNet to language-model question answering and clinical data, a central challenge is that aggregate calibration may fail for particular groups or under distribution shift, while flexible corrections require representative validation data and may affect accuracy or decisions.

Latest papers

218 papers39 papers in the last four weeks, up 39% on the four weeks before. 0.2% of all new papers.

Monitoring the calibration of probability forecasts with an application to concept drift detection involving image classification
Oct 29, 2025Christopher T. Franck, Anne R. Driscoll, Zoe Szajnfarber +1Concept Drift Model Calibration
Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging
Oct 20, 2025Tiancheng Hu, Benjamin Minixhofer, Nigel CollierLLM Alignment Model Calibration
University of Cambridge
Calibrating Biophysical Models for Grape Phenology Prediction via Multi-Task Learning
Aug 5, 2025William Solow, Sandhya SaisubramanianMulti-Task Learning Model Calibration
Oregon State University
Improving Mixup Calibration with Wasserstein Distributionally Robust Optimization
Jun 22, 2025Jiaming Hu, Yeping Jin, Debarghya Mukherjee +1Image Corruption Robustness Data Augmentation
Department of Mathematics & Statistics Boston University · Department of Electrical and Computer Engineering, Division of Systems Engineering Department of Biomedical Engineering Faculty of Computing & Data Sciences Boston University
Towards Understanding The Calibration Benefits of Sharpness-Aware Minimization
May 29, 2025Chengli Tan, Yubo Zhou, Haishan Ye +7Sharpness-Aware Minimization Model Calibration
Northwestern Polytechnical University · SGIT AI Lab, State Grid Corporation of China · Xi’an Jiaotong University +2
Optimizable Model Averaging for Ensembling Heterogeneous Model Outputs
May 28, 2025Mijung ParkEnsemble Learning Model Calibration
Department of Computer Science University of British Columbia, Vancouver
MetaMax: Improved Open-Set Deep Neural Networks via Weibull Calibration
Nov 20, 2022Zongyao Lyu, Nolan B. Gutierrez, William J. BeksiProbability Calibration Open-Set Recognition
The University of Texas at Arlington Arlington, TX, USA
How many labelers do you have? A closer look at gold-standard labels
Jun 24, 2022Chen Cheng, Hilal Asi, John DuchiModel Calibration Noisy-Label Learning
Department of Statistics, Stanford University · Department of Electrical Engineering, Stanford University · Departments of Statistics and Electrical Engineering, Stanford University

Oct 29, 2025J/K move · Enter open · S save

Monitoring the calibration of probability forecasts with an application to concept drift detection involving image classification

Christopher T. Franck, Anne R. Driscoll, Zoe Szajnfarber, William H. Woodall

Concept Drift Model Calibration Change-Point Detection

Open paper AskPDF arXiv

Machine learning approaches for image classification have led to impressive advances in that field. For example, convolutional neural networks are able to achieve remarkable image classification accuracy across a wide range of applications in industry, defense, and other areas. While these machine learning models boast impressive accuracy, a related concern is how to assess and maintain calibration in the predictions these models make. A classification model is said to be well calibrated if its predicted probabilities correspond with the rates events actually occur. While there are many available methods to assess machine learning calibration and recalibrate faulty predictions, less effort has been spent on developing approaches that continually monitor predictive models for potential loss of calibration as time passes. We propose a cumulative sum-based approach with dynamic limits that enable detection of miscalibration in both traditional process monitoring and concept drift applications. This enables early detection of operational context changes that impact image classification performance in the field. The proposed chart can be used broadly in any situation where the user needs to monitor probability predictions over time for potential lapses in calibration. Importantly, our method operates on probability predictions and event outcomes and does not require under-the-hood access to the machine learning model.