Multicalibration

Follow this topic Ask about this topic

Multicalibration studies predictors whose outcomes match their predicted probabilities within many groups defined by features or auditing rules, rather than only on average. Common approaches include iterative boosting and post-hoc recalibration, as well as online learning methods that use regret guarantees and oracle-efficient algorithms. These tools are used to assess fairness and reliability, handle shifts between populations, and support tasks such as prevalence estimation and evaluating language-model judges. A central tension is that stronger group guarantees can require substantially more data and computation, while calibration alone does not ensure accuracy or valid coverage in every subgroup.

Latest papers

17 papers6 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Reasoning-Aware Multicalibrated Reliability Predictors for Language Models
Oct 19, 2026Ekaterina Fedorova, Alessandro Morosini, Chara PodimataLanguage Model Calibration LLM Reliability
Massachusetts Institute of Technology.
A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge
Oct 13, 2026Cesare Barbera, Luca Stradiotti, Lorenzo Cascioli +6Bias in LLM-as-a-Judge LLM-as-a-Judge
University of Pisa · University of Trento · KU Leuven +1
Instance-Adaptive Online Multiclass Multicalibration
Oct 11, 2026Zhiming Huang, Jamie Morgenstern, Aaron Roth +1Multicalibration Model Calibration
Paul G. Allen School of Computer Science and Engineering, University of Washington · Amazon · Department of Computer and Information Sciences, University of Pennsylvania
The Sample Complexity of Multicalibration with an ERM Oracle
Oct 9, 2026Princewill OkoroaforMulticalibration Sample Complexity
Harvard University
TabReliBench: Evaluating Tabular Foundation Models Beyond Predictive Accuracy
Oct 9, 2026Tianqi Zhao, Jia Guo, Yan Shuo Tan +1Tabular Foundation Models Probability Calibration
Renmin University of China · Cerebras Systems, Inc. · National University of Singapore +1
Multigroup Fairness and Omniprediction: Separations and Equivalences
Oct 5, 2026Sílvia Casacuberta, Parikshit Gopalan, Varun Kanade +3Group Fairness Multicalibration
Stanford University · Apple · University of Oxford +2
Augmented Feature Boosting for Multicalibration
Sep 27, 2026Ira Globus-Harris, Inbal Livni NavonMulticalibration
Cornell University · Ben-Gurion University of the Negev
Sample Complexity of Multicalibration for Multilevel Properties
Aug 4, 2026Jiuyao Lu, Krishnakumar Balasubramanian, Aleksandr Podkopaev +1Sample Complexity Model Calibration
Department of Statistics and Data Science, The Wharton School, University of Pennsylvania · 2Amazon · Department of Statistics, University of California, Davis
Fast Rates for Swap-Agnostic Learning of Proper Losses
Jul 30, 2026Princewill OkoroaforBlackwell Approachability Statistical Learning Theory
Harvard University
Optimal Deterministic Multicalibration and Omniprediction
Jun 18, 2026Georgy Noarov, Aaron RothMinimax Estimation Model Calibration
University of Pennsylvania
COPF: An Online Framework for Deployment-Stable Counterfactual Fairness in Evolving Graphs
May 30, 2026Sheng'en Li, Dongmian ZouAlgorithmic Fairness Counterfactual Fairness
Zu Chongzhi Center and DIRC, Duke Kunshan University, Kunshan, China.
Multicalibration Boosting: Theory, Convergence, and Transferability
May 23, 2026Hanxuan Ye, Hongzhe LiPost-Hoc Calibration Model Calibration
Department of Biostatistics and Epidemiology University of Pennsylvania Philadelphia, PA 19104-6021, USA · Department of Biostatistics and Epidemiology University of Pennsylvania2026 Philadelphia, PA 19104-6021, USA
Unified Approach for Weakly Supervised Multicalibration
May 11, 2026Futoshi Futami, Takashi IshidaPost-Hoc Calibration Model Calibration
The University of Osaka / RIKEN AIP / The University of Tokyo · The University of Tokyo / RIKEN AIP
Instance-Adaptive Online Multicalibration
May 10, 2026Zhiming Huang, Jamie Morgenstern, Aaron Roth +1Probability Calibration Model Calibration
Paul G. Allen School of Computer Science and Engineering, University of Washington · Department of Computer and Information Sciences, University of Pennsylvania
The Sample Complexity of Multicalibration
Apr 23, 2026Natalie Collina, Jiuyao Lu, Georgy Noarov +1Probability Calibration Sample Complexity
University of Pennsylvania
Multicalibration for Unbiased Model-Based Prevalence Estimation
Apr 23, 2026Fridolin Linder, Thomas Leeper, Daniel Haimovich +3Probability Calibration Distribution Shift
Meta Platforms Inc. · The London School of Economics and Political Science
An Efficient Black-Box Reduction from Online Learning to Multicalibration, and a New Route to $Φ$ -Regret Minimization
Apr 21, 2026Gabriele Farina, Juan Carlos PerdomoModel Calibration Regret Minimization
MIT EECS · New York University

Oct 19, 2026J/K move · Enter open · S save

Reasoning-Aware Multicalibrated Reliability Predictors for Language Models

Ekaterina Fedorova, Alessandro Morosini, Chara Podimata

Massachusetts Institute of Technology.

Language Model Calibration LLM Reliability Multicalibration LLM Safety Hallucination in Language Models

Open paper AskPDF arXiv

As people increasingly rely on large language models (LLMs) for information, model providers need to communicate meaningful estimates of whether individual responses are correct or, more generally, reliable. In the context of hallucinations, existing work uses multicalibration to make such reliability predictions meaningful across application-relevant groups, such as prompt topics. Topic groups alone, however, can conceal systematic errors: within the same topic and confidence level, responses with different chain-of-thought (CoT) properties may exhibit opposing errors that cancel on average. In this work, we introduce reasoning-aware multicalibration, a framework that supplements application-relevant topic groups with groups derived from observable properties of CoT traces. We highlight that, theoretically, groupings based on CoT are "internally important", i.e., groupings for which the multicalibration guarantee is not societally crucial as it is for high-risk topics, but whose inclusion in the multicalibration procedure could easily and significantly improve predictive accuracy of the final reliability predictor. Empirically, we apply our framework to hallucination prediction and unsafe-response detection across a variety of benchmarks, and find that adding CoT groups to topic-based calibration reduces Brier score in both tasks. Interestingly, we find that multicalibrating with respect to CoT groups alone (without any topic information) already improves calibration across topics, suggesting that the predictor's errors depend on how the model reasoned, rather than what it was asked about in the prompt.