cs.LGMay 22, 2026

Empirical Bayes Conformal Prediction for Vision and Language Models

Authors: Jiapeng ZengYogesh PrabhuZhanpeng ZengMichael A. NewtonVikas Singh

Organizations: University of Wisconsin–Madison · University of California San Diego · Xiamen University

Abstract

Conformal prediction (CP) gives distribution-free coverage for modern vision and language models, but it is often forced to make a ranking decision from a single unstable nonconformity score. Standard CP uses one realization, while average-then-calibrate variants smooth multiple realizations into a point estimate. Both options discard the inconsistency that can help identify whether a candidate is indeed stable. A weak answer can enter the conformal set even if the evidence is not strong, simply because one posterior sample or prompt phrasing made it look strong. But variability can help distinguish a stable signal from noise-driven fluctuations. We describe an empirical Bayes conformal prediction framework that uses rr-values to convert score variability into an uncertainty informed nonconformity score. The resulting rr-value estimates how likely a candidate's latent score belongs to the top-ranked group after accounting for both its mean score and its uncertainty. It admits both a closed-form Normal-Normal empirical Bayes estimator and a nonparametric posterior-sampling estimator. Using the rr-value as the nonconformity score preserves the target conformal coverage while provably reducing the inclusion of high variance false candidates under mild regularity conditions. Across image classification, CLIP-based VLM benchmarks, and LLMs, we show that rr-value conformal prediction preserves target coverage while improving ranking stability and reducing set size when variability is informative, and reverting to CP-like behavior when variability vanishes.

Explore similar work

CardsList
  1. Robust Bayes-Assisted Conformal Prediction

    Jul 5, 2026Kianoosh Ashouritaklimi, Stefano Cortinovis, François CaronConformal PredictionNonconformity Scores