Matchability-Aware Conformal Prediction for Open-Ended Language Model Generation
Organizations: University of Lausanne, Switzerland · BegooAI, Switzerland
Abstract
Large language models (LLMs) often do not produce calibrated uncertainty estimates for their answers. We propose CAELUM, a framework that builds semantic prediction sets while separating aleatoric from epistemic uncertainty. CAELUM samples answers using an ensemble of clarification prompts, clusters them by semantic equivalence, and estimates the aleatoric and epistemic components via kernel-based von Neumann entropy before calibrating nonconformity scores with split conformal prediction. We use a weight to control for the contribution of the epistemic uncertainty of each instance and fit a coefficient to minimize prediction set size. When the objective is nonflat and selection is stable across splits, could be a diagnostic, where large values may indicate epistemic dominance (retrieve, abstain) and aleatoric dominance (clarify). For individual queries, a log ratio compares the weighted epistemic component with the aleatoric component. Theoretically, we demonstrate that the coverage is capped by matchability, the fraction of items whose reference answer appears in any sampled cluster. Empirically, this matchability can be predicted from the original question features using a lightweight classifier. We use those predictions to define a stratum and derive a conformal coverage bound within it. Across 30 LLM and dataset combinations (4-32B), mean coverage is 0.74 in the learned stratum versus 0.65 overall, averaged over the 78 evaluations using the learned classifier. Project page: https://unco3892.github.io/caelum/
Figures & tables
| Support / CAELUM | Average set size | SC proxy | |||||||
| Dataset | Cov. | CAELUM | Vanilla | Equal/SE | Epi.-only | Cov. | Size | ||
| Qwen3-8B | |||||||||
| TriviaQA | 0.811 | 0.811 | 47.42 (14.5) | 1.82 | 1.84 | 1.84 | 1.81 | 0.778 | 1.02 † |
| AmbigQA | 0.667 | 0.667 | 7.54 (33.9) | 1.30 | 1.30 | 1.30 | 1.30 | 0.650 | 1.00 † |
| OpenNQ | 0.700 | 0.700 | 0.00 (0.0) | 1.20 | 1.20 | 1.20 | 1.20 | 0.700 | 1.00 |
| AmbigInst | 1.000 | 0.935 | 0.00 (0.0) | 1.00 | 1.00 | 1.00 | 1.00 | 0.935 | 1.05 |
| Trigger signal / rule | AUROC | Abstention rate | Abstention precision | Aug. coverage |
|---|---|---|---|---|
| CAELUM detector (ours) | 0.780 | 0.459 | 0.569 (28) | 0.924 |
| trigger [ 11 ] | 0.696 | 0.506 | 0.481 (28) | 0.918 |
| Good–Turing trigger [ 19 ] | 0.550 | 0.878 | 0.371 (28) | 0.989 |
| CPQ calibration module [ 19 ] | — | 0.848 | 0.384 (27) | 0.987 |
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| CLEAR | JUCAL | CAELUM | |
|---|---|---|---|
| Output space | Open-ended text | ||
| Output | Interval | Calibrated probabilities | Semantic clusters |
| Aleatoric proxy | Quantile regression | Temperature ( ) | Between-clarification entropy |
| Epistemic proxy | Ensemble diversity | Diversity adjustment ( ) | Within-clarification entropy |
| Calibration loss | Quantile loss | NLL | Set size (APSS) |
| Requires ensemble | Yes | Yes | No (single LLM) |
| Encoder | Match% | Coverage | APSS | AUROC | |
|---|---|---|---|---|---|
| TriviaQA | |||||
| all-mpnet-base-v2 | 78.0 | 47.4 | 0.811 | 1.82 | 0.709 |
| all-MiniLM-L12-v2 | 75.3 | 9.58 | 0.767 | 2.02 | 0.724 |
| bge-large-en-v1.5 | 93.7 | 17.1 | 0.933 | 1.28 | 0.519 |
| e5-large-v2 | 100.0 | 0.00 | 1.000 | 1.00 | n/a |
| AmbigQA | |||||
| CAELUM | Vanilla | Equal | Epi. only | SC proxy | SE-CP | |||||||
| Dataset | Cov. | APSS | Cov. | APSS | Cov. | APSS | Cov. | APSS | Cov. | APSS | Cov. | APSS |
| Qwen3-8B | ||||||||||||
| TriviaQA | 0.811 | 1.82 | 0.811 | 1.84 | 0.811 | 1.84 | 0.811 | 1.81 | 0.778 | 1.02 | 0.811 | 1.84 |
| AmbigQA | 0.667 | 1.30 | 0.667 | 1.30 | 0.667 | 1.30 | 0.667 | 1.30 | 0.650 | 1.00 | 0.667 | 1.30 |
| OpenNQ | 0.700 | 1.20 | 0.700 | 1.20 | 0.700 | 1.20 | 0.700 | 1.20 | 0.700 | 1.00 | 0.700 | 1.20 |
| AmbigInst | 0.935 | 1.00 | 0.935 | 1.00 | 0.935 | 1.00 | 0.935 | 1.00 | 0.935 | 1.05 | 0.935 | 1.00 |
| Average set size (mean split SD) | ||||||
| Dataset | Match% | CAELUM | Vanilla | SE-CP | TrueAPS | |
| Qwen3-8B | ||||||
| TriviaQA | 78.0 | 1.82 0.13 | 1.82 0.12 | 1.81 0.12 | 1.84 0.12 | 14.6 23.8 |
| AmbigQA | 67.0 | 1.30 0.08 | 1.30 0.08 | 1.30 0.08 | 1.30 0.08 | 13.4 8.7 |
| OpenNQ | 63.7 | 1.19 0.03 | 1.19 0.03 | 1.19 0.03 | 1.19 0.03 | 0.0 0.0 |
| AmbigInst | 100.0 | 1.66 0.05 | 1.67 0.05 | 1.00 0.00 | 1.59 0.07 | 0.0 0.0 |
| Dataset | Qwen3-8B | Ministral-3-8B | Gemma-4-E4B-it |
|---|---|---|---|
| Instruct-2512 | |||
| TriviaQA | 47.42 (14.5) | 26.96 (25.5) | 45.74 (13.6) |
| AmbigQA | 7.54 (33.9) | 18.41 (24.5) | 9.32 (5.2) |
| OpenNQ | 0.00 (0.0) | 6.19 (14.8) | 0.00 (15.0) |
| AmbigInst | 0.00 (0.0) | 0.00 (25.5) | 6.17 (18.9) |
| SimpleQA | 0.00 (12.2) | 0.32 (20.9) | 0.28 (0.1) |
| Average set size at matched coverage | |||||||
| Dataset | CAELUM | Vanilla | Equal | Epi. | SC proxy | SE-CP | |
| Qwen3-8B | |||||||
| TriviaQA | 0.77 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| AmbigQA | 0.67 | 1.20 | 1.07 | 1.10 | 1.20 | 1.07 | 1.10 |
| OpenNQ | 0.61 | 1.06 | 1.17 | 1.14 | 1.06 | 1.00 | 1.14 |
| AmbigInst | 0.90 | 1.00 | 1.00 | 1.00 | 1.00 | 1.01 | 1.00 |
| Support-failure AUROC | Difference vs. | ||||||
| Dataset | Match% | ||||||
| Qwen3-8B | |||||||
| TriviaQA | 78.0 | 0.658 | 0.709 | 0.571 | 0.730 | +0.050 | +0.072 |
| AmbigQA | 67.0 | 0.574 | 0.632 | 0.449 | 0.677 | +0.059 | +0.104 |
| OpenNQ | 63.7 | 0.629 | 0.650 | 0.581 | 0.419 | +0.021 | 0.210 |
| SimpleQA | 35.6 | 0.588 | 0.583 | 0.558 | 0.442 | 0.005 | 0.146 |
| Dataset (mean over 3 LLMs) | Random | Oracle | ||||
|---|---|---|---|---|---|---|
| TriviaQA | 0.865 | 0.917 | 0.923 | 0.915 | 0.963 | +0.049 |
| AmbigQA | 0.801 | 0.854 | 0.852 | 0.819 | 0.928 | +0.018 |
| OpenNQ | 0.834 | 0.868 | 0.871 | 0.827 | 0.936 | 0.007 |
| AmbigInst | 0.924 | 0.963 | 0.956 | 0.910 | 0.986 | 0.014 |
| SimpleQA | 0.626 | 0.645 | 0.647 | 0.631 | 0.728 | +0.005 |
| PopQA | 0.738 | 0.787 | 0.785 | 0.766 | 0.838 | +0.028 |
| Source LLM Target LLM | AUC ( CI) | AUC ( CI) | Agreement | ||
|---|---|---|---|---|---|
| Qwen3 Ministral | 2104 | 0.831 | |||
| Qwen3 Gemma | 2104 | 0.817 | |||
| Ministral Qwen3 | 2104 | 0.831 | |||
| Ministral Gemma | 2104 | 0.796 | |||
| Gemma Qwen3 | 2104 | 0.817 | |||
| Gemma Ministral | 2104 | 0.796 |
| Evidence | Questions / labels | Scores | Split / seeds |
|---|---|---|---|
| Main comparison and NCAPSS | 18 cells, heuristic | Deterministic APS | 40/30/30; 42, with SD over 42–45 |
| Sets across splits | Same generations | Randomized APS for CAELUM, Vanilla, Equal, Epi., TrueAPS; deterministic SE-CP | 40/30/30; 42–45 |
| Support failure ranking | 18 cells, heuristic | , , , . For , deterministic APS in Table 9 , randomized APS across splits | 40/30/30; 42 for Table 9 , 42–45 across splits |
| Selective answering | 18 cells, heuristic | with deterministic APS in Table 10 ; randomized APS for comparison with | 40/30/30; 42 |
| Learned stratum | 30 cells, heuristic | Randomized APS, | /test 30/30/40; 42–45 |
| Integrated stratum | 18 cells, heuristic | Randomized APS, fitted | Four 25% splits; 42 |
| Threshold | Fraction selected | Precision | Recall | |
|---|---|---|---|---|
| 0.3 | 0.966 | 0.671 | 0.970 | 0.329 |
| 0.5 | 0.578 | 0.735 | 0.636 | 0.265 |
| 0.7 | 0.192 | 0.839 | 0.207 | 0.161 |
| Model | Dataset | (%) | APSS | Tied grid (%) | |
|---|---|---|---|---|---|
| Qwen3-8B | TriviaQA | 120 | 11.7 | 0.092 | 10.8 |
| AmbigQA | 80 | 1.2 | 0.013 | 37.3 | |
| OpenNQ | 120 | 0.8 | 0.017 | 6.9 | |
| AmbigInst | 121 | 0.0 | 0.000 | 100.0 | |
| SimpleQA | 200 | 0.0 | 0.000 | 100.0 | |
| PopQA | 200 | 3.5 | 0.020 | 44.6 |
| Support / coverage | Support-failure AUROC | ||||||
| Dataset | Cov. | APSS | |||||
| Qwen3-14B | |||||||
| TriviaQA | 0.840 | 0.822 | 0.744 | 0.744 | -0.001 | 1.56 | 0.0 0.0 |
| AmbigQA | 0.730 | 0.700 | 0.756 | 0.681 | +0.075 | 1.45 | 0.0 0.0 |
| OpenNQ | 0.690 | 0.686 | 0.718 | 0.648 | +0.070 | 1.16 | 0.0 0.0 |
| AmbigInst | 0.918 | 0.910 | 0.801 | 0.702 | +0.098 | 1.92 | 0.0 0.0 |
| Source LLM | Target LLM | Source AUC | Transfer AUC (95% CI) | Agreement |
|---|---|---|---|---|
| Qwen3-8B | Ministral-3-8B-Instruct-2512 | 0.793 | 0.771 [0.750, 0.791] | 0.831 |
| Gemma-4-E4B-it | 0.793 | 0.797 [0.778, 0.815] | 0.817 | |
| Qwen3-14B | 0.793 | 0.752 [0.730, 0.772] | 0.854 | |
| Qwen3-32B | 0.793 | 0.763 [0.742, 0.782] | 0.846 | |
| Ministral-3-8B-Instruct-2512 | Qwen3-8B | 0.788 | 0.771 [0.750, 0.790] | 0.831 |
| Gemma-4-E4B-it | 0.788 | 0.782 [0.763, 0.801] | 0.796 |
| Support-failure score | AUROC | AUPRC | FPR at 90% TPR |
|---|---|---|---|
| CAELUM detector (ours) | 0.780 0.08 | 0.648 0.16 | 0.542 0.17 |
| Logistic regression (per cell) | 0.716 0.09 | 0.573 0.17 | 0.661 0.16 |
| Gradient boosting (per cell) | 0.709 0.10 | 0.566 0.17 | 0.659 0.16 |
| [ 11 ] | 0.696 0.10 | 0.529 0.17 | 0.636 0.15 |
| [ 11 ] | 0.674 0.10 | 0.510 0.16 | 0.678 0.16 |
| [ 11 ] | 0.604 0.09 | 0.457 0.16 | 0.749 0.14 |
| Dataset | Matched references / (rate) | (pp) | ||
|---|---|---|---|---|
| Qwen3-8B | Qwen3-32B | |||
| TriviaQA | 300 | ( 77.3% ) | ( 84.3% ) | |
| AmbigQA | 200 | ( 64.0% ) | ( 77.5% ) | |
| OpenNQ | 300 | ( 61.3% ) | ( 72.0% ) | |
| AmbigInst | 304 | ( 99.0% ) | ( 97.0% ) | |
| SimpleQA | 500 | ( 35.0% ) | ( 36.6% ) | |
| Policy | Escalation budget | Accuracy / coverage | vs base | Oracle |
|---|---|---|---|---|
| Answer top-1 | ||||
| LearnedNet-RF | acc. | |||
| CAELUM margin | acc. | |||
| Set coverage | ||||
| LearnedNet-RF | cov. | |||
| Learned logistic | cov. | |||
| Dataset | Match rate | Coverage | APSS | Abstention rate | ||
|---|---|---|---|---|---|---|
| Semantic | Augmented | Semantic | Augmented | |||
| OpenNQ | ||||||
| SimpleQA | ||||||
| PopQA | ||||||
| Dataset mean | ||||||
| 1. Reference in a single sampled cluster | |
|---|---|
| Source | Qwen3-8B / TriviaQA, question ID 251; zero-based cache row 251. |
| Question | In which city was Pablo Picasso born, in 1881? |
| Reference | Malaga |
| Clusters | (1.00) “Pablo Picasso was born in Malaga, Spain, in 1881.” |
| Signals | , , , . |
| 2. Reference absent from sampled clusters | |