Probability Calibration

Follow this topic Ask about this topic

Probability calibration studies whether predictions made with a given confidence are correct at the corresponding rate, a property that matters when decisions depend on estimated risk rather than just the most likely class. Common approaches include post-hoc methods such as temperature scaling and isotonic regression, recalibration of full predictive distributions, and online or group-conditional methods that seek reliability across changing sequences or subpopulations. Researchers assess calibration with reliability diagnostics and proper scoring rules, and apply these ideas to classifiers, forecasts, medical systems, and language-model agents. A central challenge is that good average calibration may hide subgroup or distribution-shift failures, while improving calibration can interact with accuracy, ranking, and the costs of downstream decisions.

Latest papers

136 papers32 papers in the last four weeks, up 60% on the four weeks before. 0.2% of all new papers.

Mode-as-Sequence: Translating Multimodal Motion Prediction into Unified Sequential Mode Modeling
May 21, 2026Zikang Zhou, Haibo Hu, Xinhong Chen +5Probability Calibration Autonomous Driving
City University of Hong Kong · City University of Hong Kong (Dongguan) · Hon Hai Research Institute +1
Expectation Consistency Loss: Rethink Confidence Calibration under Covariate Shift
May 20, 2026Jinzong Dong, Zhaohui Jiang, Bo YangProbability Calibration Unsupervised Domain Adaptation
School of Automation, Central South University, Changsha, China.
Testable and Actionable Calibration for Full Swap Regret
May 18, 2026Konstantina Bairaktari, Lunjia Hu, Huy L. Nguyen +1Probability Calibration Swap Regret
Department of Computer Science, Aarhus University. · Khoury College of Computer Sciences, Northeastern University.
When Individually Calibrated Models Become Collectively Miscalibrated
May 14, 2026Zhaohui WangProbability Calibration Game Theory
USC Viterbi School of Engineering, University of Southern California
Fair and Calibrated Toxicity Detection with Robust Training and Abstention
May 13, 2026Mokshit SuranaProbability Calibration Group Distributionally Robust Optimization
University of Illinois Chicago
When to Trust Confidence Thresholding: Calibration Diagnostics for Pseudo-Labelled Regression
May 12, 2026Marcell T. KurbuczProbability Calibration Pseudo-Labeling
Institute for Global Prosperity, The Bartlett, University College London, 9–11 Endsleigh Gardens, London, WC1H 0EH, United Kingdom
ASD-Bench: A Four-Axis Comprehensive Benchmark of AI Models for Autism Spectrum Disorder
May 11, 2026Shubhankit Singh, Hassan Shaikh, Kuldeep Raghuwanshi +1Probability Calibration Autism Spectrum Disorder
Research Commons AI · IIT Bombay · IIT Delhi
Instance-Adaptive Online Multicalibration
May 10, 2026Zhiming Huang, Jamie Morgenstern, Aaron Roth +1Probability Calibration Model Calibration
Paul G. Allen School of Computer Science and Engineering, University of Washington · Department of Computer and Information Sciences, University of Pennsylvania
The Manokhin Probability Matrix: A Diagnostic Framework for Classifier Probability Quality
May 5, 2026Valery ManokhinPost-Hoc Calibration Probability Calibration
Independent Researcher
Conformalized Percentile Interval: Finite Sample Validity and Improved Conditional Performance
May 4, 2026Ran Zou, Wanrong Zhu, Bin NanProbability Calibration Group-Conditional Conformal Prediction
Department of Statistics, University of California, Irvine
Multi-Rater Calibrated Segmentation Models
May 4, 2026Meritxell Riera-Marín, Javier García López, Júlia Rodríguez-Comas +2Inter-Rater Reliability Image Segmentation
aSycai Technologies SL, Scientific and Technical Department, Barcelona, Spain · bBCN Medtech, Universitat Pompeu Fabra, Barcelona, Spain · cInstitució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona, Spain +1
Beyond ECE: Calibrated Size Ratio, Risk Assessment, and Confidence-Weighted Metrics
May 3, 2026Fernando Martin-Maroto, Nabil Abderrahaman, Gonzalo G. de PolaviejaProbability Calibration Uncertainty Calibration
Mathematics of Behavior and Intelligence Lab. Champalimaud Research. Avenida de Brasília, Lisbon · Algebraic AI Calle Castelló, 117, Madrid
RCProb: Probabilistic rule extraction from classification tree ensembles
Apr 28, 2026Josue ObregonProbability Calibration Interpretable ML
Department of Industrial Engineering, Seoul National University of Science and Technology (SeoulTech), Seoul, 01811, Republic of Korea
Exploring Hierarchical Consistency and Unbiased Objectness for Open-Vocabulary Object Detection
Apr 25, 2026Sanghoon Lee, Geon Lee, Hyekang Park +1Open-Vocabulary Object Detection Probability Calibration
Yonsei University · Korea Institute of Science and Technology (KIST)
The Sample Complexity of Multicalibration
Apr 23, 2026Natalie Collina, Jiuyao Lu, Georgy Noarov +1Probability Calibration Sample Complexity
University of Pennsylvania
Multicalibration for Unbiased Model-Based Prevalence Estimation
Apr 23, 2026Fridolin Linder, Thomas Leeper, Daniel Haimovich +3Probability Calibration Distribution Shift
Meta Platforms Inc. · The London School of Economics and Political Science
Multi-modal Test-time Adaptation via Adaptive Probabilistic Gaussian Calibration
Apr 21, 2026Jinglin Xu, Yi Li, Chuxiong Sun +3Test-Time Adaptation Probability Calibration
Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences · National Defense University
Unified Multimodal Uncertain Inference
Apr 9, 2026Dengjia Zhang, Alexander Martin, William Jurayj +3Probability Calibration Multimodal Reasoning
Johns Hopkins University · Human Language Technology Center of Excellence
Calibeating Made Simple
Mar 23, 2026Yurong Chen, Zhiyi Huang, Michael I. Jordan +1Online Calibration Probability Calibration
Inria, École Normale Supérieure, PSL Research University · The University of Hong Kong · University of California, Berkeley +1
Exponential Family Discriminant Analysis: Generalizing LDA-Style Generative Classification to Non-Gaussian Models
Mar 21, 2026Anish LakkapragadaExponential Family Classification
Yale University, New Haven, CT 06511
ProCal: Probability Calibration for Neighborhood-Guided Source-Free Domain Adaptation
Mar 19, 2026Ying Zheng, Yiyi Zhang, Yi Wang +1Source-Free Domain Adaptation Domain Adaptation
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, China. · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China.
Computational and Statistical Hardness of Calibration Distance
Mar 19, 2026Mingda QiaoProbability Calibration
University of Massachusetts Amherst
Simplex-to-Euclidean Bijection for Conjugate and Calibrated Multiclass Gaussian Process Classification
Mar 17, 2026Bernardo Williams, Harsha Vardhan Tetali, Arto Klami +1Gaussian Processes Multiclass Classification
Department of Computer Science, University of Helsinki, Finland
From the Inside Out: Progressive Distribution Refinement for Confidence Calibration
Mar 17, 2026Xizhong Yang, Yinan Xia, Huiming Wang +1Test-Time RL Reinforcement Learning
Southeast University · Kuaishou Technology · SUTD
The Importance of Being Smoothly Calibrated
Mar 16, 2026Parikshit Gopalan, Konstantinos Stavropoulos, Kunal Talwar +1Model Calibration Probability Calibration
Apple · UT Austin · Harvard
Decomposing Probabilistic Scores: Reliability, Information Loss and Uncertainty
Mar 16, 2026Arthur Charpentier, Agathe Fernandes MachadoProbability Calibration Uncertainty Decomposition
UQAM, Montréal, Canada · Kyoto University, Japan
Soft Mean Expected Calibration Error (SMECE): A Calibration Metric for Probabilistic Labels
Mar 14, 2026Michael LeznikProbability Calibration Model Calibration
Frequentist Consistency of Prior-Data Fitted Networks for Causal Inference
Mar 12, 2026Valentyn Melnychuk, Vahid Balazadeh, Stefan Feuerriegel +1Causal Effect Estimation Prior-Data Fitted Networks
LMU Munich & Munich Center for Machine Learning (MCML), Munich, Germany · University of Toronto & Vector Institute, Toronto, Canada
Trustworthy Predictive Distributions for Tail Events with Semiparametric Diagnostic Transport Maps
Mar 11, 2026Elizabeth Cucuzzella, Rafael Izbicki, Ann B. LeeProbability Calibration Probabilistic Forecasting
Department of Statistics and Data Science, Carnegie Mellon University · Department of Statistics, Federal University of Sao Carlos
CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents
Mar 11, 2026Marta Sumyk, Oleksandr KosovanComputer-Use Agent Evaluation AI Agent Auditing
Ukrainian Catholic University Lviv, Ukraine

May 21, 2026J/K move · Enter open · S save

Mode-as-Sequence: Translating Multimodal Motion Prediction into Unified Sequential Mode Modeling

Zikang Zhou, Haibo Hu, Xinhong Chen, Yifan Zhang+4

City University of Hong Kong · City University of Hong Kong (Dongguan) · Hon Hai Research Institute+1

Probability Calibration Autonomous Driving Spatiotemporal Forecasting Multimodal Trajectory Prediction Sequence Modeling Mode Collapse

Open paper AskPDF arXiv

Multimodal motion forecasting is inherently under-supervised: each training scene provides only one realized future, yet multiple plausible futures exist. This sparse supervision often leads to mode collapse (redundant hypotheses and insufficient mode coverage) and unreliable confidence ranking when predicting a small set of trajectories. We propose Mode-as-Sequence, a unified decoding framework that translates an unordered mode set into an ordered mode sequence and explicitly models mode-to-mode dependency. Under this framework, we develop two complementary instantiations. ModeSeq performs recurrent mode decoding, where each mode is generated conditioned on the previously generated modes, encouraging diverse, non-redundant hypotheses with calibrated confidence ordering. To remove the mode-by-mode autoregressive bottleneck, we further propose Parallel ModeSeq, which preserves the same causal dependency using masked mode-to-mode self-attention while decoding all modes in a single forward pass, enabling efficient large-

K

inference and scalable joint-scene prediction. To learn representative modes and calibrated confidence under sparse labels, we introduce Early-Match-Take-All (EMTA) and its joint-scene extension MA-EMTA, together with a lightweight ranking regularizer that reduces confidence inversions. Extensive experiments on large-scale benchmarks demonstrate consistent improvements in both ranking-oriented metrics and best-of-K accuracy across datasets, horizons, and object types. In the Waymo Open Dataset challenges, ModeSeq achieves 1st place in the 2024 LiDAR-free motion prediction track, and Parallel ModeSeq achieves 1st place in the 2025 Interaction Prediction Challenge, validating the effectiveness of Mode-as-Sequence for both accuracy and efficiency.