stat.MLNov 7, 2026

Matchability-Aware Conformal Prediction for Open-Ended Language Model Generation

Authors: Ilia Azizi

Organizations: University of Lausanne, Switzerland · BegooAI, Switzerland

Abstract

Large language models (LLMs) often do not produce calibrated uncertainty estimates for their answers. We propose CAELUM, a framework that builds semantic prediction sets while separating aleatoric from epistemic uncertainty. CAELUM samples answers using an ensemble of clarification prompts, clusters them by semantic equivalence, and estimates the aleatoric and epistemic components via kernel-based von Neumann entropy before calibrating nonconformity scores with split conformal prediction. We use a weight wλ(x)=1+Uale(x)+λ Uepi(x)w_λ(x)=1+U_{\mathrm{ale}}(x)+λ\,U_{\mathrm{epi}}(x) to control for the contribution of the epistemic uncertainty of each instance and fit a coefficient λ⋆λ^\star to minimize prediction set size. When the objective is nonflat and selection is stable across splits, λ⋆λ^\star could be a diagnostic, where large values may indicate epistemic dominance (retrieve, abstain) and λ⋆→0λ^\star\to 0 aleatoric dominance (clarify). For individual queries, a log ratio R(x)R(x) compares the weighted epistemic component with the aleatoric component. Theoretically, we demonstrate that the coverage is capped by matchability, the fraction of items whose reference answer appears in any sampled cluster. Empirically, this matchability can be predicted from the original question features using a lightweight classifier. We use those predictions to define a stratum and derive a conformal coverage bound within it. Across 30 LLM and dataset combinations (4-32B), mean coverage is 0.74 in the learned stratum versus 0.65 overall, averaged over the 78 evaluations using the learned classifier. Project page: https://unco3892.github.io/caelum/

Figures & tables

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. LLMs Uncertainty Quantification via Adaptive Conformal Semantic Entropy

    May 5, 2026Hamed Karimi, Vaishali Meyappan, Reza SamaviSelective PredictionSemantic Entropy

  2. Beyond Surface Statistics: Robust Conformal Prediction for LLMs via Internal Representations

    Apr 17, 2026Yanli Wang, Peng Kuang, Xiaoyu Han +2Language Model RobustnessConformal Prediction

  3. Conformalized Large Language Models under Configuration Shift

    Aug 2, 2026Yuqicheng Zhu, Jialin Yu, Lin Li +7Conformal PredictionConformal Prediction for Language Models