Specification-Aware Predictive Uncertainty Decomposition
Organizations: MBZUAI · AXXX · S-NLP · Huawei Technologies Ltd. ·
Abstract
Classical decompositions of predictive uncertainty, such as the information-theoretic one, are defined for a fixed representation of the input object. They do not consider that may be incomplete, nor do they measure how much the missing information can affect the prediction. In Large Language Models, users can clarify an underspecified input without any retraining. The clarified input can describe the user's unobserved intent more accurately and change the predictive uncertainty about the outcome. In this paper, we formalize this setting probabilistically and derive a three-term decomposition of prediction risk into specification, residual aleatoric, and epistemic uncertainties. We show that at the original information level, when only is available, the aleatoric uncertainty, represented by a Bayes risk, has two components. The first, which we call specification uncertainty, is the expected reduction in this Bayes risk from revealing the intent. The second is the residual aleatoric uncertainty, i.e., the Bayes risk that remains once the intent is known, averaged over its possible values given . The distinction is important, as specification uncertainty informs the decision to request clarification before making a prediction. We build approximations of these components and connect them to existing uncertainty measures. Our experiments across different domains and modalities show distinct roles for the three components. Specification uncertainty is the most informative for detecting underspecified inputs, while residual aleatoric uncertainty is the most informative for detecting in-distribution prediction errors.
Figures & tables
| Gemma E2B | Gemma E4B | Heterogeneous | ||||
| Method | CoQAR-U | AmbigQA | CoQAR-U | AmbigQA | CoQAR-U | AmbigQA |
| 0.5373 | 0.5102 | 0.5760 | 0.5071 | 0.5404 | 0.5350 | |
| 0.5213 | 0.5442 | 0.5303 | 0.5461 | 0.5232 | 0.4396 | |
| 0.7807 | 0.6575 | 0.7709 | 0.6678 | 0.7873 | 0.6017 | |
| 0.7365 | 0.5169 | 0.7614 | 0.5164 | 0.6593 | 0.5250 | |
| BALD (without clarifications) | ||||||
| Gemma E2B | Gemma E4B | Heterogeneous | ||||
| Method | CoQAR-U | AmbigQA | CoQAR-U | AmbigQA | CoQAR-U | AmbigQA |
| 0.7285 | 0.8103 | 0.7535 | 0.7739 | 0.7077 | 0.7197 | |
| 0.6573 | 0.6827 | 0.6145 | 0.6797 | 0.6730 | 0.5431 | |
| 0.5627 | 0.5279 | 0.6600 | 0.5544 | 0.5456 | 0.7234 | |
| 0.7561 | 0.8062 | 0.7974 | 0.7710 | 0.7323 | 0.7412 | |
| BALD (without clarifications) | ||||||
| Underspecification | ID error | |||||
| VQS | ImageNet | VQS | ImageNet | |||
| Method | 30% vs 10% | 30% vs 10% | 10% | 30% | 10% | 30% |
| 0.5422 | 0.5289 | 0.8217 | 0.7850 | 0.7448 | 0.7715 | |
| 0.5073 | 0.4907 | 0.7830 | 0.7521 | 0.7275 | 0.7334 | |
| 0.6614 | 0.6903 | 0.7256 | 0.7006 | 0.5992 | 0.7172 | |
| 0.5486 | 0.5452 | 0.8289 | 0.7900 | 0.7458 | 0.7756 | |
| SEA | BALD (without clarification) | BALD (clarification posterior) | |
| Total | 0.5926 | 0.7427 | 0.5926 |
| Aleatoric | 0.5170 | 0.7229 | 0.5851 |
| Epistemic | 0.4937 | 0.6108 | 0.4937 |
| Specification | 0.7739 | – | – |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Estimate | Log score | Brier score | Zero-one score |
| Input | Gemma E2B | Gemma E4B | Heterogeneous | |
| CoQAR-U | ||||
| Original | 1,000 | 87.10 | 92.10 | 92.00 |
| Omitted | 1,000 | 33.60 | 31.20 | 36.40 |
| Clarified | 1,000 | 45.80 | 46.80 | 47.20 |
| AmbigQA: pooled | ||||
| Original | 1,971 | 22.02 | 28.21 | 33.64 |
| Qwen2.5 Omni | Gemma-4 E4B | Phi-4 | MiniCPM-o | |||||
| Measure | CoQAR-U | AmbigQA | CoQAR-U | AmbigQA | CoQAR-U | AmbigQA | CoQAR-U | AmbigQA |
| 0.65 | 0.52 | 0.72 | 0.55 | 0.63 | 0.53 | 0.68 | 0.54 | |
| 0.78 | 0.61 | 0.77 | 0.59 | 0.78 | 0.55 | 0.77 | 0.64 | |
| 0.55 | 0.51 | 0.53 | 0.54 | 0.55 | 0.53 | 0.54 | 0.54 | |
| 0.65 | 0.52 | 0.72 | 0.55 | 0.63 | 0.53 | 0.68 | 0.54 | |
| 0.70 | 0.49 | 0.72 | 0.51 | 0.73 | 0.50 | 0.70 | 0.55 | |
| Qwen2.5 Omni | Gemma-4 E4B | Phi-4 | MiniCPM-o | |||||
| Measure | VQS | ImageNet | VQS | ImageNet | VQS | ImageNet | VQS | ImageNet |
| 0.56 | 0.55 | 0.55 | 0.57 | 0.56 | 0.54 | 0.57 | 0.57 | |
| 0.61 | 0.65 | 0.61 | 0.65 | 0.65 | 0.64 | 0.63 | 0.64 | |
| 0.54 | 0.53 | 0.53 | 0.53 | 0.54 | 0.52 | 0.55 | 0.54 | |
| 0.56 | 0.55 | 0.55 | 0.57 | 0.56 | 0.54 | 0.57 | 0.57 | |
| 0.53 | 0.56 | 0.53 | 0.53 | 0.54 | 0.56 | 0.52 | 0.56 | |
| Underspecification | ID error | OOD | ||||||
| Method | 10% vs 50% | 10% vs 100% | 10% | 50% | 100% | 10% | 50% | 100% |
| 0.6234 | 0.8776 | 0.8634 | 0.7705 | 0.8212 | 0.8606 | 0.8093 | 0.4588 | |
| 0.5585 | 0.7467 | 0.8459 | 0.7552 | 0.7420 | 0.8557 | 0.8149 | 0.6397 | |
| 0.7998 | 0.9304 | 0.7604 | 0.7391 | 0.8209 | 0.7466 | 0.6467 | 0.5793 | |
| 0.6451 | 0.8954 | 0.8640 | 0.7772 | 0.8503 | 0.8617 | 0.8050 | 0.4870 | |
| Underspecification | MAESTRO ID error | MusicNet OOD (combined) | ||||||
| Method | 10% vs 30% | 30% vs 50% | 10% | 30% | 50% | 10% | 30% | 50% |
| 0.5994 | 0.5151 | 0.7545 | 0.7090 | 0.6570 | 0.4992 | 0.4779 | 0.3986 | |
| 0.3652 | 0.4228 | 0.6530 | 0.6209 | 0.6159 | 0.7147 | 0.7839 | 0.8040 | |
| 0.9703 | 0.8934 | 0.6875 | 0.5515 | 0.5670 | 0.6198 | 0.5066 | 0.5402 | |
| 0.6713 | 0.6666 | 0.7232 | 0.6806 | 0.6445 | 0.5808 | 0.5498 | 0.5046 | |
Uncertainty Decomposition
Uncertainty decomposition separates sources of predictive uncertainty, such as data ambiguity and model ignorance, to improve diagnosis and decisions.
See all 55 papers