cs.LGNov 13, 2026

Advancing the Calibration-Compute Frontier of LLM Classifiers via Asymmetric Duos

Authors: Berke Akturk, Ali Can Sahin, Rasit Cem Solatay, Emanuel Sommer, Yawei Li, David Rügamer

Organizations: Department of Statistics, LMU Munich, Munich, Germany · Munich Center for Machine Learning (MCML)

Abstract

The importance of reliable uncertainty estimation in large language models (LLMs) has become increasingly pronounced, yet modern models remain poorly calibrated and frequently overconfident. In this study, we address this by proposing an efficient method for LLM classifier calibration using the Asymmetric Duos framework. Our approach combines a strong base model with a smaller sidekick model via logit-space fusion, utilizing global scaling parameters optimized on a validation set. Across diverse benchmarks, this Asymmetric LLM Duo consistently achieves best or second-best performance on expected calibration error and negative log-likelihood, without degrading predictive accuracy. More importantly, these gains are realized at a fraction of the computational cost of more complex baselines, placing our method on the calibration-compute Pareto frontier. We further demonstrate that our method acts orthogonal to various established calibration-aware techniques, meaning it can be easily combined with them to consistently boost their performance. Finally, our analysis of the logit-space behavior illustrates how asymmetric fusion successfully mitigates base model overconfidence. Overall, our findings demonstrate that the benefits of an asymmetric fusion successfully transfer to the language domain, providing a simple, highly cost-efficient approach to improving uncertainty estimation in LLM classifiers.

Figures & tables

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Mar 6, 2026

From Entropy to Calibrated Uncertainty: Training Language Models to Reason About Uncertainty

Large Language Models (LLMs) that can express interpretable and calibrated uncertainty are crucial in high-stakes domains. While methods to compute uncertainty post-hoc exist, they are often sampling-based and therefore computationally expensive or lack calibration. We propose a three-stage pipeline to post-train LLMs to efficiently infer calibrated uncertainty estimates for their responses. First, we compute fine-grained entropy-based uncertainty scores on the training data, capturing the distributional variability of model outputs in embedding space. Second, these scores are calibrated via Platt scaling, producing reliable and human-interpretable uncertainty signals. Finally, the target LLM is post-trained via reinforcement learning to align its policy with these calibrated signals through a verifiable reward function. Unlike post-hoc uncertainty estimation methods, our approach provides interpretable and computationally efficient uncertainty estimates at test time. Experiments show that models trained with our pipeline achieve better calibration than baselines and generalize to unseen tasks without further processing, suggesting that they learn a robust uncertainty reasoning behavior.
Oct 26, 2026

Why Are LLMs Confidently Wrong? Correcting Overconfident Errors via Causal Head Intervention

Modern large language models (LLMs) often exhibit overconfident errors, assigning high confidence to incorrect predictions and thereby undermining reliability in real-world use. Existing calibration methods typically rely on post-hoc adjustment or retraining, operating at the output level without addressing the underlying causes of miscalibration. We propose CoCHI, (Class-oriented Causal Head Intervention), a framework that improves calibration by identifying and intervening on internal components responsible for distinct confidence behaviours. By analysing predictions through correctness-confidence patterns, CoCHI enables targeted intervention on model internals, reducing overconfident errors while preserving correct predictions. Our approach provides a mechanistic perspective on calibration, linking reliability to identifiable structures within the model rather than treating it as an output-level issue. Experiments across multiple LLMs show consistent improvements in calibration metrics, including Expected Calibration Error, Negative Log-Likelihood, and Brier Score, while maintaining accuracy on five multiple-choice question answering benchmarks. These results suggest that effective calibration requires not only adjusting outputs, but also addressing the underlying causal mechanisms within the model.
Sep 4, 2026

Evaluating and Recalibrating Probabilistic Classifiers from One-Bit Threshold Queries

Standard calibration evaluation and recalibration methods assume access to predicted probabilities. We study one-bit threshold access, a setting in which probabilities are hidden and the evaluator observes only whether the predicted probability exceeds a single chosen threshold per sample. Under this restriction, we show that classifiers can nevertheless be evaluated and recalibrated without reconstructing their hidden scores. We derive a consistent estimator of True Calibration Error, unbiased estimators of strictly proper scoring rules, and a one-bit objective for Temperature Scaling whose expectation matches the standard log-loss objective. Across these settings, we quantify the statistical cost incurred by replacing standard probability-access procedures by our one-bit estimators. Finally, we instantiate the framework for black-box LLMs, where the logit_bias API parameter allows us to realize the required threshold queries in zero-shot binary question answering.