Advancing the Calibration-Compute Frontier of LLM Classifiers via Asymmetric Duos
Organizations: Department of Statistics, LMU Munich, Munich, Germany · Munich Center for Machine Learning (MCML)
Abstract
The importance of reliable uncertainty estimation in large language models (LLMs) has become increasingly pronounced, yet modern models remain poorly calibrated and frequently overconfident. In this study, we address this by proposing an efficient method for LLM classifier calibration using the Asymmetric Duos framework. Our approach combines a strong base model with a smaller sidekick model via logit-space fusion, utilizing global scaling parameters optimized on a validation set. Across diverse benchmarks, this Asymmetric LLM Duo consistently achieves best or second-best performance on expected calibration error and negative log-likelihood, without degrading predictive accuracy. More importantly, these gains are realized at a fraction of the computational cost of more complex baselines, placing our method on the calibration-compute Pareto frontier. We further demonstrate that our method acts orthogonal to various established calibration-aware techniques, meaning it can be easily combined with them to consistently boost their performance. Finally, our analysis of the logit-space behavior illustrates how asymmetric fusion successfully mitigates base model overconfidence. Overall, our findings demonstrate that the benefits of an asymmetric fusion successfully transfer to the language domain, providing a simple, highly cost-efficient approach to improving uncertainty estimation in LLM classifiers.
Figures & tables
| Metric | Method | ARC-C | ARC-E | CSQA | OBQA | RACE | SciQ |
|---|---|---|---|---|---|---|---|
| NLL | S | ||||||
| B | |||||||
| B TS | |||||||
| B DE | |||||||
| B LA | |||||||
| B IB-EDL |
| Metric | Method | ARC-C | ARC-E | CSQA | OBQA | RACE | SciQ |
|---|---|---|---|---|---|---|---|
| NLL | Duo ( ) | ||||||
| Duo | |||||||
| Acc | Duo ( ) | ||||||
| Duo | |||||||
| ECE | Duo ( ) | ||||||
| Duo |
| Metric | Method | Base Correct | Base Wrong |
|---|---|---|---|
| NLL | Base | 0.003 | 17.705 |
| Duo | 2.552 | ||
| ECE | Base | 0.28 | |
| Duo | 65.59 |
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
| Comparison | Base method | ARC-C | ARC-E | CSQA | OBQA | RACE | SciQ |
|---|---|---|---|---|---|---|---|
| Base TS Base | Plain LoRA | ||||||
| Laplace-LoRA | |||||||
| IB-EDL | |||||||
| Duo Base TS | Plain LoRA | ||||||
| Laplace-LoRA | |||||||
| IB-EDL |
| Metric | Method | ARC-C | ARC-E | CSQA | OBQA | RACE | SciQ |
|---|---|---|---|---|---|---|---|
| NLL | B | ||||||
| B TS | |||||||
| Duo | |||||||
| B DE | |||||||
| Acc | B | ||||||
| B TS |
| Metric | Method | ARC-C | ARC-E | CSQA | OBQA | RACE | SciQ |
|---|---|---|---|---|---|---|---|
| NLL | B | ||||||
| B TS | |||||||
| Duo | |||||||
| S | |||||||
| B DE | |||||||
| Acc | B |
| Metric | Method | ARC-C | ARC-E | CSQA | OBQA | RACE | SciQ |
|---|---|---|---|---|---|---|---|
| NLL | B | ||||||
| B TS | |||||||
| Duo | |||||||
| S | |||||||
| B DE | |||||||
| Acc | B |
| Coefficient | Val. subset | ARC-C | ARC-E | CSQA | OBQA | RACE | SciQ |
|---|---|---|---|---|---|---|---|
| 20% | |||||||
| 40% | |||||||
| 100% | |||||||
| 20% | |||||||
| 40% | |||||||
| 100% |
| Val. subset | ARC-C | ARC-E | CSQA | OBQA | RACE | SciQ |
|---|---|---|---|---|---|---|
| 20% | ||||||
| 40% | ||||||
| 100% |
| Measure | Subset | ARC-C | ARC-E | CSQA | OBQA | RACE | SciQ |
|---|---|---|---|---|---|---|---|
| Test-set fraction | Prediction unchanged | ||||||
| Prediction changed | |||||||
| Changed: wrong to right | |||||||
| Changed: right to wrong | |||||||
| Share of NLL gain | Prediction unchanged | ||||||
| Prediction changed |
| Variant | Parameters | Fusion rule |
|---|---|---|
| Base | 0 | |
| Base TS | 1 | |
| Symmetric learned | 1 | |
| Duo | 2 | |
| Duo oracle | 2 | , fitted on test |
| Variant | ARC-C | ARC-E | CSQA | OBQA | RACE | SciQ |
|---|---|---|---|---|---|---|
| Base | ||||||
| Base TS | ||||||
| Symmetric learned | ||||||
| Duo | ||||||
| Duo oracle (lower bound) |
| Metric | Method | ARC-C | ARC-E | CSQA | OBQA | RACE | SciQ |
|---|---|---|---|---|---|---|---|
| NLL | B LA | ||||||
| Duo LA | |||||||
| B BLoB | |||||||
| Duo BLoB | |||||||
| B IB-EDL | |||||||
| Duo IB-EDL |
| Method | Weight | ARC-C | ARC-E | CSQA | OBQA | RACE | SciQ |
|---|---|---|---|---|---|---|---|
| Duo | |||||||
| Duo LA | |||||||
| Duo BLoB | |||||||
| Metric | Dataset | Method | Base Correct | Base Wrong |
|---|---|---|---|---|
| NLL | ARC-C | Base | 0.001 | |
| Duo | 2.996 | |||
| ARC-E | Base | 0.001 | ||
| Duo | 2.935 | |||
| OBQA | Base | 0.001 | ||
| Duo | 2.224 |
| Dataset | Base ECE | Sidekick ECE | |
|---|---|---|---|
| ARC-C | 0.0439 | 0.1118 | |
| 0.0467 | 0.0986 | ||
| 0.1207 | 0.0538 | ||
| 0.1633 | 0.0898 | ||
| ARC-E | 0.0424 | 0.0548 | |
| 0.0684 | 0.0508 |
Language Model Calibration
Language model calibration studies whether models’ confidence estimates match their actual accuracy across tasks, prompts, and deployment settings.
See all 420 papers