Probability Calibration

Ask about this topic

Overview

Probability calibration studies whether predictions made with a given confidence are correct at the corresponding rate, a property that matters when decisions depend on estimated risk rather than just the most likely class. Common approaches include post-hoc methods such as temperature scaling and isotonic regression, recalibration of full predictive distributions, and online or group-conditional methods that seek reliability across changing sequences or subpopulations. Researchers assess calibration with reliability diagnostics and proper scoring rules, and apply these ideas to classifiers, forecasts, medical systems, and language-model agents. A central challenge is that good average calibration may hide subgroup or distribution-shift failures, while improving calibration can interact with accuracy, ranking, and the costs of downstream decisions.

Latest papers

CardsList

136 papers32 papers in the last four weeks, up 60% on the four weeks before. 0.2% of all new papers.

May 21, 2026

Mode-as-Sequence: Translating Multimodal Motion Prediction into Unified Sequential Mode Modeling

Multimodal motion forecasting is inherently under-supervised: each training scene provides only one realized future, yet multiple plausible futures exist. This sparse supervision often leads to mode collapse (redundant hypotheses and insufficient mode coverage) and unreliable confidence ranking when predicting a small set of trajectories. We propose Mode-as-Sequence, a unified decoding framework that translates an unordered mode set into an ordered mode sequence and explicitly models mode-to-mode dependency. Under this framework, we develop two complementary instantiations. ModeSeq performs recurrent mode decoding, where each mode is generated conditioned on the previously generated modes, encouraging diverse, non-redundant hypotheses with calibrated confidence ordering. To remove the mode-by-mode autoregressive bottleneck, we further propose Parallel ModeSeq, which preserves the same causal dependency using masked mode-to-mode self-attention while decoding all modes in a single forward pass, enabling efficient large-KK inference and scalable joint-scene prediction. To learn representative modes and calibrated confidence under sparse labels, we introduce Early-Match-Take-All (EMTA) and its joint-scene extension MA-EMTA, together with a lightweight ranking regularizer that reduces confidence inversions. Extensive experiments on large-scale benchmarks demonstrate consistent improvements in both ranking-oriented metrics and best-of-K accuracy across datasets, horizons, and object types. In the Waymo Open Dataset challenges, ModeSeq achieves 1st place in the 2024 LiDAR-free motion prediction track, and Parallel ModeSeq achieves 1st place in the 2025 Interaction Prediction Challenge, validating the effectiveness of Mode-as-Sequence for both accuracy and efficiency.

Zikang Zhou, Haibo Hu, Xinhong Chen +5

City University of Hong Kong · City University of Hong Kong (Dongguan) · Hon Hai Research Institute +1

Probability CalibrationAutonomous DrivingSpatiotemporal ForecastingMultimodal Trajectory PredictionSequence ModelingMode Collapse
AskPDF
May 20, 2026

Expectation Consistency Loss: Rethink Confidence Calibration under Covariate Shift

Confidence calibration for classification models is vital in safety-critical decision-making scenarios and has received extensive attention. General confidence calibration methods assume training and test data are independent and identically distributed, limiting their effectiveness under covariate shifts. Previous calibration methods under covariate shift struggle with class-wise or canonical calibrations and often rely on unstable importance weighting when density ratios are large or unbounded. Given the above limitations, this paper rethinks confidence calibration under covariate shifts. First, we derive a necessary and sufficient condition for confidence calibration under covariate shifts, named Expectation consistency condition, which reveals covariate shifts do not necessarily lead to uncalibrated confidence and provides a weaker condition for confidence calibration than global covariate distribution alignment. Then, utilizing Expectation consistency condition, this paper proposes an unsupervised domain adaptation loss to calibrate confidence of the target domain, named Expectation consistency loss (ECL), which is compatible with canonical calibration, class-wise calibration, and top-label calibration. Third, we prove that computing ECL loss has the same sample complexity as Expected Calibration Error (ECE) and provide a theoretically grounded mini-batch trainable scheme for ECL loss. Finally, we validate the effectiveness of our method on both simulated and real-world covariate shift datasets.

Jinzong Dong, Zhaohui Jiang, Bo Yang

School of Automation, Central South University, Changsha, China.

Probability CalibrationUnsupervised Domain AdaptationCovariate ShiftModel CalibrationUncertainty Calibration
AskPDF
May 18, 2026

Testable and Actionable Calibration for Full Swap Regret

AI generated predictions increasingly inform decision making in critical tasks, and therefore must be trustworthy. One widely used measure of trustworthiness is calibration, which requires that the predictions match the true frequencies and can be treated like real probabilities of a given outcome. However, defining calibration is subtle, and designing good measures of calibration error has been an active topic of recent research. The first goal is to find calibration measures that are actionable, meaning they can inform decision makers about their utility loss when predictions are treated as true probabilities, which is known as swap regret. The second goal is to find calibration measures that are testable, meaning that calibration error can be measured from a small sample of predictions and outcomes. Although these are very basic requirements, there is no existing calibration measure that fully satisfies both properties, and all existing measures relax actionability by bounding a weaker notion of swap regret, or relax testability by having suboptimal estimation error. We introduce a new calibration measure, Soft-Binned Calibration Decision Loss (SCDL), which we prove is fully actionable without weakening either requirement, and testable with nearly optimal error rate. In addition, SCDL satisfies other desired properties such as continuity and consistency. We also provide a set of experiments confirming that the theoretical advantages of SCDL compared to other measures lead to better performance in practice.

Konstantina Bairaktari, Lunjia Hu, Huy L. Nguyen +1

Department of Computer Science, Aarhus University. · Khoury College of Computer Sciences, Northeastern University.

Probability CalibrationSwap Regret
AskPDF
May 14, 2026

When Individually Calibrated Models Become Collectively Miscalibrated

Probabilistic prediction systems often aggregate probability estimates from multiple models into a single decision. A common assumption is that if each model is individually calibrated, the aggregate prediction will also be well calibrated. We show that this assumption fails in multi-agent settings: individually calibrated predictors can become collectively miscalibrated when their predictions interact strategically, in the game-theoretic sense of Brier-optimal local response, even without deliberate coordination. This phenomenon arises naturally when agents are independently trained on overlapping data. We prove that under Brier-score-based aggregation with positively correlated beliefs, each agent's individually optimal report systematically underestimates the positive-class probability, yielding a Price of Anarchy greater than one whenever Cov(b_i, b_j) > 0. In a canonical setting (n = 5 agents, pairwise correlation = 0.5, base rate = 0.3), the empirically measured PoA in false-negative rate reaches 7.25x. In contrast, VCG-based aggregation aligns incentives by rewarding marginal contribution, achieving dominant-strategy incentive compatibility and near-optimal performance. Experiments on three real-world datasets (NSL-KDD, UNSW-NB15, Credit Card Fraud) show that VCG provides strong robustness while maintaining comparable accuracy. It performs particularly well in data-sparse and adversarial settings, and adaptive weighting further improves performance under distribution shift.

Zhaohui Wang

USC Viterbi School of Engineering, University of Southern California

Probability CalibrationGame TheoryProbabilistic ForecastingMulti-Agent SystemsPrice of Anarchy
AskPDF
May 13, 2026

Fair and Calibrated Toxicity Detection with Robust Training and Abstention

Fairness in toxicity classification involves three integrated axes: ranking, calibration, and abstention. Training-time interventions and post-hoc safety mechanisms cannot be evaluated independently because the former determines the efficacy of the latter. We compare Empirical Risk Minimization (ERM), instance-level reweighting, and Group DRO across these axes, combined with temperature scaling, confidence-based abstention, and per-identity threshold optimization. Evaluation uses subgroup AUC, BPSN/BNSP AUC, error gaps, and per-subgroup Expected Calibration Error (ECE) with bootstrap CIs (n=1000n = 1000). We report four findings. (1) Calibration disparity is a hidden fairness violation. ERM has near-perfect aggregate calibration (0.0130.013) but is significantly miscalibrated across all identity subgroups (+0.029+0.029 to +0.134+0.134). (2) Training interventions reshape rather than eliminate disparity. Reweighted ERM improves ranking (BPSN AUC +0.06+0.06 to +0.12+0.12) but worsens the calibration-fairness gap by up to +0.232+0.232. Group DRO eliminates calibration disparity but only by becoming uniformly miscalibrated globally (ECE 0.1180.118). (3) Post-hoc methods inherit training failure modes. Temperature scaling fails because miscalibration is non-uniform. Confidence-based abstention works under ERM but breaks under DRO, where the risk-coverage curve rises with deferral. (4) Abstention itself is unfair. Confidence-based deferral helps background content far more than identity-mentioning content. We argue that SRAI fairness requires a multi-axis framework: methods that differ only in aggregate ranking can differ sharply in failure modes that determine real-world harm.

Mokshit Surana

University of Illinois Chicago

Probability CalibrationGroup Distributionally Robust OptimizationSelective PredictionAlgorithmic FairnessToxicity DetectionContent Moderation
AskPDF
May 12, 2026

When to Trust Confidence Thresholding: Calibration Diagnostics for Pseudo-Labelled Regression

Calibrated probability outputs of trained classifiers are increasingly used as inputs to downstream regression estimands such as effects, prevalences, or disparities for a latent group observed only on a small labelled subset. A standard practice is to threshold the calibrated score at a confidence cutoff and treat the hard label as the truth. Building on a recent identification result for the underlying moment equation, we develop a calibration-aware diagnostic apparatus for pseudo-labelling pipelines. We derive a closed-form expression for the attenuation bias that confidence thresholding induces in the downstream regression coefficient, and show that the bias can be predicted, before any inference is run, from the residual score variance V∗=E[Var⁡(p∣X)]V^{*}=\mathbb{E}[\operatorname{Var}(p\mid X)] on the unlabelled set after partialling out the downstream controls XX. We further obtain a sharp sensitivity bound under bounded calibration drift, and identify the boundary V∗=0V^{*}=0, which holds iff pp is a deterministic function of XX; this motivates a structural separation between classifier features WW and downstream controls X⊊WX\subsetneq W. Five controlled simulations and a UCI Adult illustration trace the predictions. The contribution is operational: a (V∗,κ)(V^{*}, κ) decision rule that practitioners can compute from any classifier output to decide whether confidence thresholding is safe.

Marcell T. Kurbucz

Institute for Global Prosperity, The Bartlett, University College London, 9–11 Endsleigh Gardens, London, WC1H 0EH, United Kingdom

Probability CalibrationPseudo-Labeling
AskPDF
May 11, 2026

ASD-Bench: A Four-Axis Comprehensive Benchmark of AI Models for Autism Spectrum Disorder

Automated ASD screening tools remain limited by single-architecture evaluations, axis-restricted assessment, and near-exclusive focus on adult cohorts, obscuring age-specific diagnostic patterns critical for early intervention. We introduce ASD-Bench, a systematic tabular benchmark evaluating ML, deep learning, and foundation model configurations across three age cohorts (children 1-11 yr, adolescents 12-16 yr, adults 17-64 yr) on four axes: predictive performance, calibration, interpretability, and adversarial robustness. Applied to a curated v3 dataset of 4,068 AQ-10 records, our benchmark spans classical models (XGBoost, AdaBoost, Random Forest, Logistic Regression), neural networks (MLP), deep tabular transformers (TabNet, TabTransformer, FT-Transformer), and TabPFN v2. We introduce the Heuristic Aggregate Penalty (HAP): a cost-sensitive metric penalising false negatives more heavily and incorporating cross-validation variance for deployment stability. Adult classification yields high performance (10/17 models achieve perfect F1 and AUC), while adolescents present a harder task (F1 ceiling 0.837 vs. 0.915 for children). Feature hierarchies shift across cohorts: A9 (social motivation) dominates for children, A5 (pattern recognition) leads for adolescents, and adults exhibit a flatter importance profile consistent with developmental social masking. Accuracy and calibration are dissociated: AdaBoost achieves F1=1.000 on adults with ECE=0.302, confirming single-metric evaluation is insufficient for clinical AI. Cohort-specific deployment recommendations are provided. All findings should be interpreted as proof-of-concept evidence on questionnaire-derived labels rather than clinically validated diagnostic performance.

Shubhankit Singh, Hassan Shaikh, Kuldeep Raghuwanshi +1

Research Commons AI · IIT Bombay · IIT Delhi

Probability CalibrationAutism Spectrum DisorderInterpretable MLAdversarial RobustnessTabular ML
AskPDF
May 10, 2026

Instance-Adaptive Online Multicalibration

We study online multicalibration beyond the worst-case. We give a single, efficient algorithm which dynamically interpolates between benign and worst-case sequences by adaptively refining a dyadic grid of prediction values. Its error is controlled by the number of leaves in the refinement tree. Our analysis recovers the known O~(T2/3)\widetilde O(T^{2/3}) worst-case-optimal rate for online multicalibration, while simultaneously automatically adapting to easier instances: in the marginal stochastic setting it obtains a rate of O~(T)\widetilde O(\sqrt T), and for piecewise-stationary means with JJ segments its rate is O~(JT)\widetilde O(\sqrt{JT}). More generally, the rate depends on a threshold-complexity measure of the predictable mean process relative to the group family. We show that this dependence is tight up to logarithmic factors.

Zhiming Huang, Jamie Morgenstern, Aaron Roth +1

Paul G. Allen School of Computer Science and Engineering, University of Washington · Department of Computer and Information Sciences, University of Pennsylvania

Probability CalibrationModel CalibrationStatistical Learning TheoryAdaptive LearningMulticalibrationOnline Calibration
AskPDF
May 5, 2026

The Manokhin Probability Matrix: A Diagnostic Framework for Classifier Probability Quality

The Brier score conflates two distinct properties of probabilistic predictions: reliability (calibration error) and resolution (discriminatory power). We introduce the Manokhin Probability Matrix, a BCG-style two-dimensional diagnostic framework that separates them. Classifiers are placed on a 2x2 grid by Spiegelhalter Z-statistic and AUC-ROC expected rank, then assigned to one of four archetypes: Eagle (good on both axes), Bull (strong discrimination, poor calibration), Sloth (well-calibrated, weak discriminator), and Mole (poor on both). Each archetype carries a distinct prescription. We populate the matrix from a large-scale empirical study spanning 21 classifiers, 5 post-hoc calibrators, and 30 real-world binary classification tasks from the TabArena-v0.1 suite. The assignment is unambiguous. CatBoost, TabICL, EBM, TabPFN, GBC, and Random Forest are Eagles. XGBoost, LightGBM, and HGB are Bulls; Venn-Abers calibration cuts log-loss by 6.5 to 12.6% on Bulls but degrades Eagles by 2.1%. SVM, LR, LDA, and the empirical base-rate predictor are Sloths. MLP, KNN, Naive Bayes, and ExtraTrees are Moles. A theoretical asymmetry follows: no order-preserving post-hoc calibrator can add discriminatory power (Proposition 1), so calibration is the fixable part and discrimination is the hard part. The practical rule is direct: do not optimise aggregate Brier score without first decomposing it; optimise discrimination first, then fix calibration post-hoc. Code and raw experimental data are available at https://github.com/valeman/classifier_calibration.

Valery Manokhin

Independent Researcher

Post-Hoc CalibrationProbability CalibrationModel CalibrationBinary ClassificationClassification
AskPDF
May 4, 2026

Conformalized Percentile Interval: Finite Sample Validity and Improved Conditional Performance

Conformal prediction provides distribution-free predictive intervals with finite-sample marginal coverage. However, achieving conditional validity and interval efficiency (in terms of short interval length) remains challenging, particularly in complex settings with heteroskedasticity, skewed responses, or estimation errors. We propose a conformal-style calibration method for responses obtained by the probability integral transform (PIT) of the conditional cumulative distribution function (CDF) estimated via neural networks to construct a finite-sample-adjusted percentile interval with the shortest length determined by the estimated conditional CDF. Calibrating in PIT space is effective because PIT values are asymptotically feature-independent when the CDF estimator is accurate, which mitigates feature-dependent miscoverage and improves conditional calibration. On the other hand, our percentile calibration adapts to the empirical PIT distribution, which is robust against a possibly imperfect estimation of the conditional CDF. We prove the finite-sample marginal coverage property of the proposed method and show its asymptotic conditional coverage under mild consistency conditions. Experiments on diverse synthetic and real-world benchmarks demonstrate better conditional calibration and substantially shorter intervals than existing methods.

Ran Zou, Wanrong Zhu, Bin Nan

Department of Statistics, University of California, Irvine

Probability CalibrationGroup-Conditional Conformal PredictionConformal PredictionRegression Prediction Intervals
AskPDF
May 4, 2026

Multi-Rater Calibrated Segmentation Models

Objective: Accurate probability estimates are essential for the safe deployment of medical image segmentation models in clinical decision-making. However, modern deep segmentation networks are often poorly calibrated, a problem exacerbated when multiple expert annotations exhibit substantial disagreement. While inter-rater variability is typically treated as noise, it provides valuable information about intrinsic annotation ambiguity that must be reflected in model confidence. Methods: We improve the probabilistic calibration of medical image segmentation models by reformulating multi-rater supervision as an ordinal learning problem. Voxel-wise annotator agreement is treated as an ordered target, linking predictive confidence to the empirical variability in training data. This formulation allows the use of ordinal-aware scoring rules, such as the Ranked Probability Score ordinal loss, combined with a standard binary objective to preserve discriminative performance. Results: We evaluated the proposed approach across four public segmentation benchmarks spanning ophthalmology, histopathology, and thoracic imaging. Calibration was assessed using a multi-rater extension of expected calibration error. Results consistently show that ordinal-aware training yields substantially improved calibration with respect to inter-rater agreement without degrading segmentation accuracy. Conclusions: Treating multi-rater annotations as ordered information provides a principled and architecture-agnostic route to more reliable probabilistic segmentation models.

Meritxell Riera-Marín, Javier García López, Júlia Rodríguez-Comas +2

aSycai Technologies SL, Scientific and Technical Department, Barcelona, Spain · bBCN Medtech, Universitat Pompeu Fabra, Barcelona, Spain · cInstitució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona, Spain +1

Inter-Rater ReliabilityImage SegmentationProbability CalibrationMedical Image SegmentationSegmentation Uncertainty Estimation
AskPDF
May 3, 2026

Beyond ECE: Calibrated Size Ratio, Risk Assessment, and Confidence-Weighted Metrics

Confidence calibration has been dominated by the Expected Calibration Error (ECE), a linear metric that counts calibration offset equally regardless of the confidence level at which it occurs. We show that ECE can remain small even under arbitrarily large overconfidence risk, so we propose Calibrated Size Ratio (CSR) instead, an interpretable metric that equals 1 under perfect calibration, from which we derive the risk probability PriskP_{\mathrm{risk}} that quantifies the statistical evidence for overconfidence. We further argue that overconfidence risk assessment must be complemented by a measure of discriminative value: whether the assigned confidences actively distinguish correct from incorrect predictions. We show that confidence-weighted accuracy cwA\mathrm{cwA} is the natural such complement, and that confidence-weighting extends to all standard classification metrics. In particular, we prove that the confidence-weighted AUC (cwAUC) captures the information about calibration while the classical AUC cannot. We validate the proposed indicators on several synthetic confidence distributions under multiple controlled calibration profiles and find that CSR separates risky from non-risky assignments. We also test the metrics on fifteen real datasets, with and without post-hoc calibration, and find that standard methods can yield risky confidence profiles.

Fernando Martin-Maroto, Nabil Abderrahaman, Gonzalo G. de Polavieja

Mathematics of Behavior and Intelligence Lab. Champalimaud Research. Avenida de Brasília, Lisbon · Algebraic AI Calle Castelló, 117, Madrid

Probability CalibrationUncertainty CalibrationConfidence Estimation
AskPDF
Apr 28, 2026

RCProb: Probabilistic rule extraction from classification tree ensembles

Tree ensembles provide strong classification performance but usually behave as black-box models. Post-hoc interpretability techniques such as RuleCOSI+ extract a small ruleset that approximates the ensemble, but this simplification can leave the probabilities attached to the extracted rules unreliable. In particular, RuleCOSI+ assigns empirical class probabilities to the extracted rules and repeatedly uses those rule statistics during its greedy combination and simplification procedure. We present RCProb, a probabilistic extension that uses smoothed atomic class-conditional evidence for the expensive search stages and a support-adaptive mixture with an ensemble-informed m-estimate for the final rule probabilities. The method is evaluated on 18 binary and 5 multiclass datasets using random forest (RF) and gradient boosting machine (GBM) ensembles. Relative to RuleCOSI+, the median paired log-loss reduction is 71.9% for RF and 62.5% for GBM, with both differences remaining significant after Holm correction. The number of rules decreases by 38.7% for RF and 38.5% for GBM, while the primary tests do not detect a macro-F1 difference. Confidence-ECE also decreases for both ensembles, with statistical support for RF after correction. A separate controlled experiment with dedicated calibration data shows that native RCProb probabilities are competitive with RuleCOSI+ followed by temperature scaling, although additional post-hoc calibration can still improve RCProb. The results show that probability estimation is an important part of rule extraction and not only a post-processing step.

Josue Obregon

Department of Industrial Engineering, Seoul National University of Science and Technology (SeoulTech), Seoul, 01811, Republic of Korea

Probability CalibrationInterpretable ML
AskPDF
Apr 25, 2026

Exploring Hierarchical Consistency and Unbiased Objectness for Open-Vocabulary Object Detection

Conventional object detectors typically operate under a closed-set assumption, limiting recognition to a predefined set of base classes seen during training. Open-vocabulary object detection (OVD) addresses this limitation by leveraging vision-language models (VLMs) to generate pseudo labels for novel object classes. However, existing OVD methods suffer from two critical drawbacks: (1) inaccurate class label assignments, as VLMs are optimized for image-level predictions rather than the region-level predictions required for pseudo labeling, and (2) unreliable objectness scores from region proposal networks (RPNs) trained exclusively on base object classes. To address these issues, we propose a novel pseudo labeling framework for OVD. Our approach introduces a hierarchical confidence calibration (HCC) technique, which ensures reliable class label estimation by assessing consistency across hierarchical semantic levels (class, super- and sub-category). We also present LoCLIP, a parameter-efficient adaptation of CLIP that incorporates an objectness token to mitigate base class bias problem of RPNs and provide reliable objectness estimations for novel object classes. Extensive experiments on standard OVD benchmarks, including COCO and LVIS, demonstrate that our approach clearly sets a new state of the art, validating the effectiveness of our approach. Project site: https://cvlab.yonsei.ac.kr/projects/HCC

Sanghoon Lee, Geon Lee, Hyekang Park +1

Yonsei University · Korea Institute of Science and Technology (KIST)

Open-Vocabulary Object DetectionProbability CalibrationVLM AdaptationPseudo-LabelingObject Detection
AskPDF
Apr 23, 2026

The Sample Complexity of Multicalibration

We study the minimax sample complexity of multicalibration in the batch setting. A learner observes nn i.i.d. samples from an unknown distribution and must output a (possibly randomized) predictor whose population multicalibration error, measured by Expected Calibration Error (ECE), is at most ε\varepsilon with respect to a given family of groups. For every fixed κ>0κ> 0, in the regime ∣G∣≤ε−κ|G|\le \varepsilon^{-κ}, we prove that Θ~(ε−3)\widetildeΘ(\varepsilon^{-3}) samples are necessary and sufficient, up to polylogarithmic factors. The lower bound holds even for randomized predictors, and the upper bound is realized by a randomized predictor obtained via an online-to-batch reduction. This separates the sample complexity of multicalibration from that of marginal calibration, which scales as Θ~(ε−2)\widetildeΘ(\varepsilon^{-2}), and shows that mean-ECE multicalibration is as difficult in the batch setting as it is in the online setting, in contrast to marginal calibration which is strictly more difficult in the online setting. In contrast we observe that for κ=0κ= 0, the sample complexity of multicalibration remains Θ~(ε−2)\widetildeΘ(\varepsilon^{-2}) exhibiting a sharp threshold phenomenon. More generally, we establish matching upper and lower bounds, up to polylogarithmic factors, for a weighted LpL_p multicalibration metric for all 1≤p≤21 \le p \le 2, with optimal exponent 3/p3/p. We also extend the lower-bound template to a regular class of elicitable properties, and combine it with the online upper bounds of Hu et al. (2025) to obtain matching bounds for calibrating properties including expectiles and bounded-density quantiles.

Natalie Collina, Jiuyao Lu, Georgy Noarov +1

University of Pennsylvania

Probability CalibrationSample ComplexityModel CalibrationStatistical Learning TheoryMulticalibration
AskPDF
Apr 23, 2026

Multicalibration for Unbiased Model-Based Prevalence Estimation

Estimating the prevalence of a category in a population using imperfect measurement devices (diagnostic tests, classifiers, or large language models) is fundamental to science, public health, and online trust and safety. Standard approaches correct for known device error rates but assume these rates remain stable across populations. We show this assumption fails under covariate shift and that multicalibration, which enforces calibration conditional on the input features rather than just on average, is sufficient for unbiased prevalence estimation under such shift. Standard calibration and quantification methods fail to provide this guarantee. Our work connects recent theoretical work on fairness to a longstanding measurement problem spanning nearly all academic disciplines. A simulation confirms that standard methods exhibit bias growing with shift magnitude, while a multicalibrated estimator maintains near-zero bias. While we focus the discussion mostly on LLMs, our theoretical results apply to any classification model. Two empirical applications -- estimating employment prevalence across U.S. states using the American Community Survey, and classifying political texts across four countries using an LLM -- demonstrate that multicalibration substantially reduces bias in practice, while highlighting that calibration data should cover the key feature dimensions along which target populations may differ.

Fridolin Linder, Thomas Leeper, Daniel Haimovich +3

Meta Platforms Inc. · The London School of Economics and Political Science

Probability CalibrationDistribution ShiftCovariate ShiftModel CalibrationMulticalibration
AskPDF
Apr 21, 2026

Multi-modal Test-time Adaptation via Adaptive Probabilistic Gaussian Calibration

Multi-modal test-time adaptation (TTA) enhances the resilience of benchmark multi-modal models against distribution shifts by leveraging the unlabeled target data during inference. Despite the documented success, the advancement of multi-modal TTA methodologies has been impeded by a persistent limitation, i.e., the lack of explicit modeling of category-conditional distributions, which is crucial for yielding accurate predictions and reliable decision boundaries. Canonical Gaussian discriminant analysis (GDA) provides a vanilla modeling of category-conditional distributions and achieves moderate advancement in uni-modal contexts. However, in multi-modal TTA scenario, the inherent modality distribution asymmetry undermines the effectiveness of modeling the category-conditional distribution via the canonical GDA. To this end, we introduce a tailored probabilistic Gaussian model for multi-modal TTA to explicitly model the category-conditional distributions, and further propose an adaptive contrastive asymmetry rectification technique to counteract the adverse effects arising from modality asymmetry, thereby deriving calibrated predictions and reliable decision boundaries. Extensive experiments across diverse benchmarks demonstrate that our method achieves state-of-the-art performance under a wide range of distribution shifts. The code is available at https://github.com/XuJinglinn/AdaPGC.

Jinglin Xu, Yi Li, Chuxiong Sun +3

Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences · National Defense University

Test-Time AdaptationProbability CalibrationMultimodal Robustness
AskPDF
Apr 9, 2026

Unified Multimodal Uncertain Inference

We introduce Unified Multimodal Uncertain Inference (UMUI), a multimodal inference task spanning text, audio, and video, where models must produce calibrated probability estimates of hypotheses conditioned on a premise in any modality or combination. While uncertain inference has been explored in text, extension to other modalities has been limited to single-modality binary entailment judgments, leaving no framework for fine-grained probabilistic reasoning in or across other modalities. To address this, we curate a human-annotated evaluation set with scalar probability judgments across audio, visual, and audiovisual settings, and additionally evaluate on existing text and audio benchmarks. We introduce CLUE (Calibrated Latent Uncertainty Estimation), which combines self-consistent teacher calibration and distribution-based confidence probing to produce calibrated predictions. We demonstrate that our 3B-parameter model achieves equivalent or stronger performance than zero-shot baselines up to 32B parameters across all modalities.

Dengjia Zhang, Alexander Martin, William Jurayj +3

Johns Hopkins University · Human Language Technology Center of Excellence

Probability CalibrationMultimodal ReasoningMultimodal Model EvaluationMultimodal Understanding
AskPDF
Mar 23, 2026

Calibeating Made Simple

We study calibeating, the problem of post-processing external forecasts online to minimize cumulative losses and match an informativeness-based benchmark. Unlike prior work, which analyzed calibeating for specific losses with specific arguments, we reduce calibeating to existing online learning techniques and obtain results for general proper losses. More concretely, we first show that calibeating is minimax-equivalent to regret minimization. This recovers the O(log⁡T)O(\log T) calibeating rate of Foster and Hart [FH23] for the Brier and log losses and its optimality, and yields new optimal calibeating rates for mixable losses and general bounded losses. Second, we prove that multi-calibeating is minimax-equivalent to the combination of calibeating and the classical expert problem. This yields new optimal multi-calibeating rates for mixable losses, including Brier and log losses, and general bounded losses. Finally, we obtain new bounds for achieving calibeating and calibration simultaneously for the Brier loss. For binary predictions, our result gives the first calibrated algorithm that at the same time also achieves the optimal O(log⁡T)O(\log T) calibeating rate.

Yurong Chen, Zhiyi Huang, Michael I. Jordan +1

Inria, École Normale Supérieure, PSL Research University · The University of Hong Kong · University of California, Berkeley +1

Online CalibrationProbability Calibration
AskPDF
Mar 21, 2026

Exponential Family Discriminant Analysis: Generalizing LDA-Style Generative Classification to Non-Gaussian Models

We introduce Exponential Family Discriminant Analysis (EFDA), a unified generative framework that extends classical Linear Discriminant Analysis (LDA) beyond the Gaussian setting to any member of the exponential family. Under the assumption that each class-conditional density belongs to a common exponential family, EFDA derives closed-form maximum-likelihood estimators for all natural parameters and yields a decision rule that is linear in the sufficient statistic, recovering LDA as a special case and capturing nonlinear decision boundaries in the original feature space. We prove that EFDA is asymptotically calibrated and statistically efficient under correct specification, and we generalise it to K≥2K \geq 2 classes and multivariate data. Through extensive simulation across five exponential-family distributions (Weibull, Gamma, Exponential, Poisson, Negative Binomial), EFDA matches the classification accuracy of LDA, QDA, and logistic regression while reducing Expected Calibration Error (ECE) by 22-6×6\times, a gap that is structural: it persists for all nn and across all class-imbalance levels, because misspecified models remain asymptotically miscalibrated. We further prove and empirically confirm that EFDA's log-odds estimator approaches the Cramér-Rao bound under correct specification, and is the only estimator in our comparison whose mean squared error converges to zero. Complete derivations are provided for nine distributions. Finally, we formally verify all four theoretical propositions in Lean 4, using Aristotle (Harmonic) and OpenGauss (Math, Inc.) as proof generators, with all outputs independently machine-checked by AXLE (Axiom).

Anish Lakkapragada

Yale University, New Haven, CT 06511

Exponential FamilyClassificationProbability Calibration
AskPDF
Mar 19, 2026

ProCal: Probability Calibration for Neighborhood-Guided Source-Free Domain Adaptation

Source-Free Domain Adaptation (SFDA) adapts pre-trained models to unlabeled target domains without requiring access to source data. Although state-of-the-art methods leveraging local neighborhood structures show promise for SFDA, they tend to over-rely on prediction similarity among neighbors. This over-reliance accelerates the forgetting of source knowledge and increases susceptibility to local noise overfitting. To address these issues, we introduce ProCal, a probability calibration method that dynamically calibrates neighborhood-based predictions through a dual-model collaborative prediction mechanism. ProCal integrates the source model's initial predictions with the current model's online outputs to effectively calibrate neighbor probabilities. This strategy not only mitigates the interference of local noise but also preserves the discriminative information from the source model, thereby achieving a balance between knowledge retention and domain adaptation. Furthermore, we design a joint optimization objective that combines a soft supervision loss with a diversity loss to guide the target model. Our theoretical analysis shows that ProCal converges to an equilibrium where source knowledge and target information are effectively fused, reducing both knowledge forgetting and overfitting. We validate the effectiveness of our approach through extensive experiments on 31 cross-domain tasks across four public datasets. Our code is available at: https://github.com/zhengyinghit/ProCal.

Ying Zheng, Yiyi Zhang, Yi Wang +1

Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, China. · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China.

Source-Free Domain AdaptationDomain AdaptationUnsupervised Domain AdaptationProbability Calibration
AskPDF
Mar 19, 2026

Computational and Statistical Hardness of Calibration Distance

The distance from calibration, introduced by Błasiok, Gopalan, Hu, and Nakkiran (STOC 2023), has recently emerged as a central measure of miscalibration for probabilistic predictors. We study the fundamental problems of computing and estimating this quantity, given either an exact description of the data distribution or only sample access to it. We give an efficient algorithm that exactly computes the calibration distance when the distribution has a uniform marginal and noiseless labels, which improves the O(1/∣X∣)O(1/\sqrt{|\mathcal{X}|}) additive approximation of Qiao and Zheng (COLT 2024) for this special case. Perhaps surprisingly, the problem becomes NP\mathsf{NP}-hard when either of the two assumptions is removed. We extend our algorithm to a polynomial-time approximation scheme for the general case. For the estimation problem, we show that Θ(1/ε3)Θ(1/ε^3) samples are sufficient and necessary for the empirical calibration distance to be upper bounded by the true distance plus εε. In contrast, a polynomial dependence on the domain size -- incurred by the learning-based baseline -- is unavoidable for two-sided estimation. Our positive results are based on simple sparsifications of both the distribution and the target predictor, which significantly reduce the search space for computation and lead to stronger concentration for the estimation problem. To prove the hardness results, we introduce new techniques for certifying lower bounds on the calibration distance -- a problem that is hard in general due to its co-NP\textsf{co-NP}-completeness.

Mingda Qiao

University of Massachusetts Amherst

Probability Calibration
AskPDF
Mar 17, 2026

Simplex-to-Euclidean Bijection for Conjugate and Calibrated Multiclass Gaussian Process Classification

We propose a conjugate and calibrated Gaussian process (GP) model for multi-class classification by exploiting the geometry of the probability simplex. Our approach uses Aitchison geometry to map simplex-valued class probabilities to an unconstrained Euclidean representation, turning classification into a GP regression problem with fewer latent dimensions than standard multi-class GP classifiers. This yields conjugate inference and reliable predictive probabilities without relying on distributional approximations in the model construction. The method is compatible with standard sparse GP regression techniques, enabling scalable inference on larger datasets. Empirical results show well-calibrated and competitive performance across synthetic and real-world datasets.

Bernardo Williams, Harsha Vardhan Tetali, Arto Klami +1

Department of Computer Science, University of Helsinki, Finland

Gaussian ProcessesMulticlass ClassificationProbability Calibration
AskPDF
Mar 17, 2026

From the Inside Out: Progressive Distribution Refinement for Confidence Calibration

Leveraging the model's internal information as the self-reward signal in Reinforcement Learning (RL) has received extensive attention due to its label-free nature. While prior works have made significant progress in applying the Test-Time Scaling (TTS) strategies to RL, the discrepancy in internal information between test and training remains inadequately addressed. Moreover, Test-Time Training based on voting-based TTS strategies often suffers from reward hacking problems. To address these issues, we propose DistriTTRL, which leverages the distribution prior of the model's confidence during RL to progressively optimize the reward signal, rather than relying solely on single-query rollouts. Additionally, we mitigate the phenomenon of consistent reward hacking caused by the voting-based TTS strategies through diversity-targeted penalties. Benefiting from this training mechanism where model capability and self-reward signals complement each other, and the mitigation of reward hacking, DistriTTRL has achieved significant performance improvements across multiple models and benchmarks.

Xizhong Yang, Yinan Xia, Huiming Wang +1

Southeast University · Kuaishou Technology · SUTD

Test-Time RLReinforcement LearningReward Design for RLProbability CalibrationReward Hacking
AskPDF
Mar 16, 2026

The Importance of Being Smoothly Calibrated

Recent work has highlighted the centrality of smooth calibration [Kakade and Foster, 2008] as a robust measure of calibration error. We generalize, unify, and extend previous results on smooth calibration, both as a robust calibration measure, and as a step towards omniprediction, which enables predictions with low regret for downstream decision makers seeking to optimize some proper loss unknown to the predictor. We present a new omniprediction guarantee for smoothly calibrated predictors, for the class of all bounded proper losses. We smooth the predictor by adding some noise to it, and compete against smoothed versions of any benchmark predictor on the space, where we add some noise to the predictor and then post-process it arbitrarily. The omniprediction error is bounded by the smooth calibration error of the predictor and the earth mover's distance from the benchmark. We exhibit instances showing that this dependence cannot, in general, be improved. We show how this unifies and extends prior results [Foster and Vohra, 1998; Hartline, Wu, and Yang, 2025] on omniprediction from smooth calibration. We present a crisp new characterization of smooth calibration in terms of the earth mover's distance to the closest perfectly calibrated joint distribution of predictions and labels. This also yields a simpler proof of the relation to the lower distance to calibration from [Blasiok, Gopalan, Hu, and Nakkiran, 2023]. We use this to show that the upper distance to calibration cannot be estimated within a quadratic factor with sample complexity independent of the support size of the predictions. This is in contrast to the distance to calibration, where the corresponding problem was known to be information-theoretically impossible: no finite number of samples suffice [Blasiok, Gopalan, Hu, and Nakkiran, 2023].

Parikshit Gopalan, Konstantinos Stavropoulos, Kunal Talwar +1

Apple · UT Austin · Harvard

Model CalibrationProbability Calibration
AskPDF
Mar 16, 2026

Decomposing Probabilistic Scores: Reliability, Information Loss and Uncertainty

Calibration is a conditional property that depends on the information retained by a predictor. We develop decomposition identities for arbitrary proper losses that make this dependence explicit. At any information level A\mathcal A, the expected loss of an A\mathcal A-measurable predictor splits into a proper-regret (reliability) term and a conditional entropy (residual uncertainty) term. For nested levels A⊆B\mathcal A\subseteq\mathcal B, a chain decomposition quantifies the information gain from A\mathcal A to B\mathcal B. Applied to classification with features X\boldsymbol{X} and score S=s(X)S=s(\boldsymbol{X}), this yields a three-term identity: miscalibration, a {\em grouping} term measuring information loss from X\boldsymbol{X} to SS, and irreducible uncertainty at the feature level. We leverage the framework to analyze post-hoc recalibration, aggregation of calibrated models, and stagewise/boosting constructions, with explicit forms for Brier and log-loss.

Arthur Charpentier, Agathe Fernandes Machado

UQAM, Montréal, Canada · Kyoto University, Japan

Probability CalibrationUncertainty DecompositionProper Scoring Rules
AskPDF
Mar 14, 2026

Soft Mean Expected Calibration Error (SMECE): A Calibration Metric for Probabilistic Labels

The Expected Calibration Error (ece), the dominant calibration metric in machine learning, compares predicted probabilities against empirical frequencies of binary outcomes. This is appropriate when labels are binary events. However, many modern settings produce labels that are themselves probabilities rather than binary outcomes: a radiologist's stated confidence, a teacher model's soft output in knowledge distillation, a class posterior derived from a generative model, or an annotator agreement fraction. In these settings, ece commits a category error - it discards the probabilistic information in the label by forcing it into a binary comparison. The result is not a noisy approximation that more data will correct. It is a structural misalignment that persists and converges to the wrong answer with increasing precision as sample size grows. We introduce the Soft Mean Expected Calibration Error (smece), a calibration metric for settings where labels are of probabilistic nature. The modification to the ece formula is one line: replace the empirical hard-label fraction in each prediction bin with the mean probability label of the samples in that bin. smece reduces exactly to ece when labels are binary, making it a strict generalisation.

Michael Leznik

Probability CalibrationModel Calibration
AskPDF
Mar 12, 2026

Frequentist Consistency of Prior-Data Fitted Networks for Causal Inference

Foundation models based on prior-data fitted networks (PFNs) have shown strong empirical performance in causal inference by framing the task as an in-context learning problem. However, it is unclear whether PFN-based causal estimators provide uncertainty quantification that is consistent with classical frequentist estimators. In this work, we address this gap by analyzing the frequentist consistency of PFN-based estimators for the average treatment effect (ATE). (1) We show that existing PFNs, when interpreted as Bayesian ATE estimators, can exhibit prior-induced confounding bias: the prior is not asymptotically overwritten by data, which, in turn, prevents frequentist consistency. (2) As a remedy, we suggest employing a calibration procedure based on a one-step posterior correction (OSPC). We show that the OSPC helps to restore frequentist consistency and can yield a semi-parametric Bernstein-von Mises theorem for calibrated PFNs (i.e., both the calibrated PFN-based estimators and the classical semi-parametric efficient estimators converge in distribution with growing data size). (3) Finally, we implement OSPC through tailoring martingale posteriors on top of the PFNs. In this way, we are able to recover functional nuisance posteriors from PFNs, required by the OSPC. In multiple (semi-)synthetic experiments, PFNs calibrated with our martingale posterior OSPC produce ATE uncertainty that (i) asymptotically matches frequentist uncertainty and (ii) is well calibrated in finite samples in comparison to other Bayesian ATE estimators.

Valentyn Melnychuk, Vahid Balazadeh, Stefan Feuerriegel +1

LMU Munich & Munich Center for Machine Learning (MCML), Munich, Germany · University of Toronto & Vector Institute, Toronto, Canada

Causal Effect EstimationPrior-Data Fitted NetworksBayesian InferenceSemiparametric InferenceProbability Calibration
AskPDF
Mar 11, 2026

Trustworthy Predictive Distributions for Tail Events with Semiparametric Diagnostic Transport Maps

Machine learning forecast systems are moving beyond point predictions to full predictive distributions for future outcomes y conditional on complex inputs x. However, these distributions are often locally miscalibrated, especially for high-stakes tail events where accurate uncertainty quantification is most needed to establish trust in models. Local miscalibration occurs because training data often lack examples of low-frequency events. The goal of this paper is to describe a simple, yet flexible framework that, at deployment, produces interpretable diagnostics and a robust correction mechanism of predictive distributions when train examples are limited. With this goal in mind, we introduce a semiparametric version of the Local Amortized Diagnostic and Reshaping (LADaR) framework that posits a covariate-dependent parametric model for a diagnostic transport map regressed nonparametrically on inputs to describe how to correct tail probabilities across the feature space to match calibration data. These maps provide the user with local, real-time diagnostics and a reshaped predictive distribution that can be related back to physical processes in the input space. We apply these semiparametric diagnostic transport maps to short-term tropical cyclone intensity forecasting to detect evolutionary modes linked to local miscalibration in the National Hurricane Center's forecasts and improve predictions for severe weather hazards.

Elizabeth Cucuzzella, Rafael Izbicki, Ann B. Lee

Department of Statistics and Data Science, Carnegie Mellon University · Department of Statistics, Federal University of Sao Carlos

Probability CalibrationProbabilistic ForecastingTropical Cyclone Intensity ForecastingUncertainty CalibrationWeather Forecasting
AskPDF
Mar 11, 2026

CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents

Computer-Use Agents (CUAs) are emerging as a new paradigm in human-computer interaction, enabling autonomous execution of tasks in desktop environment by perceiving high-level natural-language instructions. As such agents become increasingly capable and are deployed across diverse desktop environments, evaluating their behavior in a scalable and reliable manner becomes a critical challenge. Existing evaluation pipelines rely on static benchmarks, rule-based success checks, or manual inspection, which are brittle, costly, and poorly aligned with real-world usage. In this work, we study Vision-Language Models (VLMs) as autonomous auditors for assessing CUA task completion directly from observable interactions and conduct a large-scale meta-evaluation of five VLMs that judge task success given a natural-language instruction and the final environment state. Our evaluation spans three widely used CUA benchmarks across macOS, Windows, and Linux environments and analyzes auditor behavior along three complementary dimensions: accuracy, calibration of confidence estimates, and inter-model agreement. We find that while state-of-the-art VLMs achieve strong accuracy and calibration, all auditors exhibit notable performance degradation in more complex or heterogeneous environments, and even high-performing models show significant disagreement in their judgments. These results expose fundamental limitations of current model-based auditing approaches and highlight the need to explicitly account for evaluator reliability, uncertainty, and variance when deploying autonomous CUAs in real-world settings.

Marta Sumyk, Oleksandr Kosovan

Ukrainian Catholic University Lviv, Ukraine

Computer-Use Agent EvaluationAI Agent AuditingVision-Language Model-as-a-JudgeComputer-Use AgentsProbability Calibration
AskPDF