Sample-Efficient Generative Conformal Prediction

Minxing Zheng; Shixiang Zhu

cs.LGOct 8, 2026

Sample-Efficient Generative Conformal Prediction

Authors: Minxing Zheng, Shixiang Zhu

Organizations: Carnegie Mellon University

Abstract

Generative conformal prediction builds uncertainty sets from samples of a conditional generator, which are efficient only when the samples represent the response distribution well. This can require many samples, each of which can be costly, as in large diffusion models and scientific simulators, so the sampling budget must be used efficiently. Existing methods draw the same number of samples at every input, wasting samples where the response distribution is simple and undersampling where it is complex, which inflates sets and leaves those inputs under-covered. We propose CASA (Conformal Adaptive Sample Allocation), which characterizes the marginal value of an additional sample and allocates samples across inputs to minimize the expected set size subject to marginal coverage and an average sampling budget. Theoretical analysis shows that adaptive allocation yields smaller sets than a fixed count at the same budget: a missed mode forces a radius that spans the gap between modes, and even oracle radius cannot compensate for it. On synthetic and real tasks, CASA produces substantially smaller sets at the same budget, often improves conditional coverage, and complements existing radius-adaptive methods.

Figures & tables

Panel of Figure 1 — Figure 1: Motivation for adaptive sampling. Moving three draws from $x_{1}$ to $x_{2}$ improves mode representation and balances conditional coverage at the same average budget.

Panel of Figure 2 — Figure 2: Mechanism of CASA with average budget $B=4$ on a conditional distribution whose modes separate as $x$ grows. (b) Dots mark the radius minimizing the priced cost $g$ in Theorem 1 for each count, and stars mark the selected count–radius pair. (c) The value of $k$ samples is $-\min_{r}g_{k}(x;r)$ relative to $k=2$ ; each input takes the count of highest value. (d) CASA moves samples from unimodal to multimodal inputs while keeping the same average budget as fixed PCP.

Figure 3: Synthetic results on a subset of DGPs with a

90\%

coverage target. (A) Marginal coverage and set size relative to fixed PCP at

B=5

. (B) Set size against realized mean samples with

B=3,\ldots,10

on the emerging-modes DGP; open circles mark fixed PCP at equal samples and at equal size. (C) Size reduction and MACE change of CASA at

B=5

. Error bars show 95% intervals across

100

independent runs.

Figure 4: Calibrated prediction sets on a synthetic DGP with $B=5$ . Gray points are test pairs; at each displayed input, colored segments are the set and dots the generated samples.

	Rare modes			Emerging modes			Compact to extended
Method	Size	$\Delta$ (%)	MACE	Size	$\Delta$ (%)	MACE	Size	$\Delta$ (%)	MACE
PCP	1.31 $\mkern-1.0mu\to\mkern-1.0mu$ 1.15	$-$ 12.0	14.6 $\mkern-1.0mu\to\mkern-1.0mu$ 12.9	3.29 $\mkern-1.0mu\to\mkern-1.0mu$ 1.54	$-$ 54.9	10.0 $\mkern-1.0mu\to\mkern-1.0mu$ 7.4	2.73 $\mkern-1.0mu\to\mkern-1.0mu$ 2.45	$-$ 10.2	9.6 $\mkern-1.0mu\to\mkern-1.0mu$ 9.3
CP 2	2.97 $\mkern-1.0mu\to\mkern-1.0mu$ 2.63	$-$ 11.5	1.6 $\mkern-1.0mu\to\mkern-1.0mu$ 2.3 †	4.90 $\mkern-1.0mu\to\mkern-1.0mu$ 4.13	$-$ 16.2	2.9 $\mkern-1.0mu\to\mkern-1.0mu$ 3.9 †	3.68 $\mkern-1.0mu\to\mkern-1.0mu$ 3.41	$-$ 7.6	1.5 $\mkern-1.0mu\to\mkern-1.0mu$ 2.1 †
RCP	2.00 $\mkern-1.0mu\to\mkern-1.0mu$ 1.86	$-$ 7.2	3.2 $\mkern-1.0mu\to\mkern-1.0mu$ 2.3	2.80 $\mkern-1.0mu\to\mkern-1.0mu$ 2.49	$-$ 11.9	3.9 $\mkern-1.0mu\to\mkern-1.0mu$ 2.5	2.59 $\mkern-1.0mu\to\mkern-1.0mu$ 2.46	$-$ 5.1	3.2 $\mkern-1.0mu\to\mkern-1.0mu$ 2.4
CP4Gen	1.36 $\mkern-1.0mu\to\mkern-1.0mu$ 1.18	$-$ 13.0	14.3 $\mkern-1.0mu\to\mkern-1.0mu$ 13.2	2.55 $\mkern-1.0mu\to\mkern-1.0mu$ 2.60	+1.5	4.8 $\mkern-1.0mu\to\mkern-1.0mu$ 4.9	2.38 $\mkern-1.0mu\to\mkern-1.0mu$ 2.43	+2.3 †	5.3 $\mkern-1.0mu\to\mkern-1.0mu$ 5.5 †
C-PCP ‡	2.08 $\mkern-1.0mu\to\mkern-1.0mu$ 1.99	$-$ 4.3	2.6 $\mkern-1.0mu\to\mkern-1.0mu$ 2.8	2.55 $\mkern-1.0mu\to\mkern-1.0mu$ 1.96	$-$ 23.2	3.1 $\mkern-1.0mu\to\mkern-1.0mu$ 2.9	2.62 $\mkern-1.0mu\to\mkern-1.0mu$ 2.57	$-$ 2.0	3.0 $\mkern-1.0mu\to\mkern-1.0mu$ 3.3

Table 1: CASA combined with other generative conformal methods at

B=5

and a

90\%

coverage target. Arrows compare each method’s fixed count with CASA count allocation. Size: mean set size;

\Delta

: paired geometric-mean size change; MACE in percentage points. Bold/ † : significant improvement/deterioration (paired

95\%

interval). ‡ C-PCP uses 20 extra auxiliary draws. Details are in Appendix B.2 .

Figure 5: Porto taxi routes at $B=5$ . (A) Mean allocated samples. (B) Samples along one trip. (C–D) More samples at a junction, fewer on a straight road. Stars mark true positions; dashed and filled circles are fixed PCP and CASA sets.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Fixed count versus CASA on rare modes for

B=3,\ldots,10

, with 95% intervals over 100 seeds. The horizontal axis counts realized samples, auxiliary draws included; panel titles give the mean size change.

Figure 7: Calibrated uncertainty sets on four synthetic DGPs for fixed PCP, CASA , and CASA (joint). (a–b) Gray points are test pairs, and colored segments are the sets. (c–d) Gray points are responses at one input, crosses are samples, and circles outline the sets. Headings give marginal coverage and mean samples.

Configuration	$B{=}3$	$5$	$10$	$5$	$10$	$3$	$5$	$10$	Mean $K$
	CASA , same cap			Equal samples		CASA (joint), same cap
20-NN, 300 m (main)	0.68	0.63	0.88	0.60	0.80	0.58	0.56	0.85	3.0/4.9/9.7
10-NN, 300 m	0.69	0.63	0.90	0.60	0.78	0.56	0.52	0.91	3.0/4.9/9.5
40-NN, 300 m	0.67	0.62	0.86	0.57	0.77	0.54	0.55	0.85	3.0/4.8/9.7
20-NN, 300 m, time split	0.67	0.64	0.90	0.60	0.84	0.51	0.54	0.85	2.9/4.9/9.5
20-NN, 200 m	0.62	0.73	0.94	0.66	0.84	0.52	0.66	0.89	2.9/4.8/8.7

Appendix

Table 2: Taxi routes, all configurations: mean set area relative to fixed PCP over ten splits, at the same budget and at equal realized samples. Bold: paired

t

-test

p<0.05

. Last column: CASA ’s mean samples at

B=3,5,10

Figure 8: Protein backbone angles at $B=5$ . (A) Mean samples CASA assigns, by position on the Ramachandran plane. (B) A helical residue: fewer samples give a smaller set. (C) A rare conformation that fixed PCP’s samples miss. (D) Set area against mean samples over ten splits.

Configuration	$B{=}3$	$5$	$10$	$5$	$10$	$3$	$5$	$10$	Mean $K$
	CASA , same cap			Equal samples		CASA (joint), same cap
20-NN, ESM-2 35M (main)	0.76	0.87	0.94	0.76	0.86	0.63	0.79	0.91	2.9/4.6/8.7
10-NN	0.71	0.88	0.97	0.80	0.89	0.62	0.81	0.94	3.0/4.7/8.7
40-NN	0.81	0.86	0.94	0.80	0.87	0.69	0.81	0.90	2.9/4.7/8.8
20-NN, ESM-2 150M	0.80	0.89	0.92	0.79	0.88	0.68	0.81	0.92	2.9/4.5/9.2

Appendix

Table 3: Protein backbone angles, all configurations: mean set area relative to fixed PCP over ten splits, at the same budget and at equal realized samples. Bold: paired

t

-test

p<0.05

. Last column: CASA ’s mean samples at

B=3,5,10

Figure 9: TextVQA with Qwen2.5-VL-3B at an $80\%$ target and $B=5$ . (A) Mean samples CASA assigns, by question accuracy. (B) An ambiguous question: CASA ’s extra samples find the answer. (C) A clear question: two samples suffice. (D) Answers per set against mean samples over ten splits.

Explore similar work

Conf-Gen: Conformal Uncertainty Quantification for Generative Models
May 27, 2026Gabriel Loaiza-Ganem, Kevin Zhang, Wei Cui +2Uncertainty Quantification Conformal Prediction
Layer 6 AI, Toronto, Canada
TRACE: Transport Alignment Conformal Prediction via Diffusion and Flow Matching Models
May 8, 2026Zhenhan Fang, Aixin Tan, Jian HuangConformal Prediction
Department of Statistics and Actuarial Science University of Iowa Iowa, IA 52242, USA · Departments of Data Science and AI, and Applied Mathematics The Hong Kong Polytechnic University Hong Kong, China
SPACR: Single-Pass Adaptive Training of Uncertainty-Aware Conformal Regressors
Jun 9, 2026Soundouss Messoudi, Sylvain Rousseau, Sébastien DesterckeConformal Prediction Adaptive Conformal Inference
Heudiasyc - UMR CNRS 7253, Université de Technologie de Compiègne

May 27, 2026, cs.LGJ/K move · Enter open · S save

Conf-Gen: Conformal Uncertainty Quantification for Generative Models

Gabriel Loaiza-Ganem, Kevin Zhang, Wei Cui, Marc T. Law+1

Layer 6 AI, Toronto, Canada

Uncertainty Quantification Conformal Prediction Generative Modeling Conformal Risk Control

Open paper AskPDF arXiv

Conformal prediction (CP) and its extension, conformal risk control (CRC), are established frameworks for quantifying uncertainty in supervised machine learning through formal guarantees. However, recent breakthroughs in artificial intelligence (AI) have been driven by unsupervised generative models, such as large language models (LLMs) and image generators, which are not directly compatible with CP or CRC. In this work we introduce conformal generation (Conf-Gen), a general framework adapting CRC to generative tasks while relaxing its theoretical assumptions. Conf-Gen unifies and generalizes previous attempts to apply CP to LLMs, and extends conformal methodology to entirely new domains. We demonstrate the flexibility of Conf-Gen through some novel applications, including obtaining conformal guarantees on: image generators producing non-memorized images, conversational AI systems having asked enough clarifying questions, and the output of AI agents being correct.

Abstract

Explore similar work

Conf-Gen: Conformal Uncertainty Quantification for Generative Models

TRACE: Transport Alignment Conformal Prediction via Diffusion and Flow Matching Models

SPACR: Single-Pass Adaptive Training of Uncertainty-Aware Conformal Regressors