ConfHit: Conformal Generative Design with Oracle Free Guarantees

Siddhartha Laghuvarapu; Ying Jin; Jimeng Sun

cs.LGMar 7, 2026

ConfHit: Conformal Generative Design with Oracle Free Guarantees

Authors: Siddhartha Laghuvarapu, Ying Jin, Jimeng Sun

Organizations: Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, IL, USA · Department of Statistics and Data Science, Wharton School, University of Pennysylvania, PA, USA

Abstract

The success of deep generative models in scientific discovery requires not only the ability to generate novel candidates but also reliable guarantees that these candidates indeed satisfy desired properties. Recent conformal-prediction methods offer a path to such guarantees, but its application to generative modeling in drug discovery is limited by budget constraints, lack of oracle access, and distribution shift. To this end, we introduce ConfHit, a distribution-free framework that provides validity guarantees under these conditions. ConfHit formalizes two central questions: (i) Certification: whether a generated batch can be guaranteed to contain at least one hit with a user-specified confidence level, and (ii) Design: whether the generation can be refined to a compact set without weakening this guarantee. ConfHit leverages weighted exchangeability between historical and generated samples to eliminate the need for an experimental oracle, constructs multiple-sample density-ratio weighted conformal p-value to quantify statistical confidence in hits, and proposes a nested testing procedure to certify and refine candidate sets of multiple generated samples while maintaining statistical guarantees. Across representative generative molecule design tasks and a broad range of methods, ConfHit consistently delivers valid coverage guarantees at multiple confidence levels while maintaining compact certified sets, establishing a principled and reliable framework for generative modeling.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Problem setup: given an input, certify and generate a set of candidates that contains at least one “hit” (green) with probability at least $1-\alpha$ . (b) ConfHit workflow. Given a nested sequence of candidate batches, we estimate the density ratio between labeled data and generated samples, compute a conformal p-value for each batch to quantify the confidence in it containing a hit, and return the smallest batch whose p-value falls below $\alpha$ .

Panel of Figure 2 — Figure 2: Certification results. Left : realized error rates at fixed $N$ for different models and error levels $\alpha$ in SBDD (upper) and CMO (lower). Middle : average error rates while varying budget $N$ . Right : power, i.e., the fraction of actives certified at various error level $\alpha$ and budget $N$ values. The dashed line denotes the ideal $y=x$ error bound. Our method consistently achieves valid coverage across scenarios. Results are averaged over 5 random runs; error bars and additional results for other values of $N$ are in Appendix E.2

Panel of Figure 3 — Figure 3: Design results. Error rate at fixed $N$ for different methods (left), mean set sizes averaged across methods at different values of $N$ (middle), and empty set percentage at different values of $N$ (right) across target levels $\alpha$ . The top row shows results for SBDD and the bottom for CMO. Dashed black line in the error plots indicates the ideal $y=x$ bound. ConfHit achieves tight error control while producing substantially smaller sets. Results are averaged over 5 random runs; additional results are provided in the Appendix E.2 .

Figure 4: Comparison of score statistics on TargetDiff ( SBDD , N=5). Left: rejection (power) at $\alpha=0.1,0.3$ . Right: error vs. $\alpha$ . All remain valid; the max statistic shows the highest power.

Panel of Figure 5 — Figure 5: (a) Distribution shift adjustment (left). Coverage violations when ConfHit is run without density correction. (b) Budget analysis (middle and right). Fraction of inputs with at least one hit under increasing generation budget. Solid lines: fraction of actual hits; dashed lines: predicted fraction of hits.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Error rate as a function of the number of Monte Carlo permutations B for DRD2 with HGraph2Graph (N = 5). The empirical error rate remains stable with minimal variance.

$N$	$\alpha$	Error Rate (%)	Certified Fraction (%)
3	0.1	$6.5\pm 1.1$	$22.6\pm 2.4$
3	0.2	$9.6\pm 1.4$	$33.2\pm 4.1$
3	0.3	$18.5\pm 1.9$	$39.8\pm 4.7$
3	0.4	$26.8\pm 2.1$	$47.6\pm 3.5$
3	0.5	$26.0\pm 2.0$	$57.5\pm 6.8$
5	0.1	$7.3\pm 2.0$	$43.0\pm 4.3$

Appendix

Table 1: DRD2_HGRAPH Certification Results

$N$	$\alpha$	Error Rate (%)	Certified Fraction (%)
3	0.1	$7.1\pm 0.4$	$19.1\pm 2.1$
3	0.2	$8.7\pm 1.7$	$21.5\pm 3.1$
3	0.3	$19.8\pm 1.2$	$28.2\pm 3.2$
3	0.4	$19.8\pm 1.6$	$39.3\pm 4.1$
3	0.5	$28.3\pm 3.3$	$42.5\pm 5.2$
5	0.1	$7.0\pm 1.0$	$22.8\pm 3.1$

Appendix

Table 2: DRD2_SELFEDIT Certification Results

$N$	$\alpha$	Error Rate (%)	Certified Fraction (%)
3	0.1	$7.8\pm 1.2$	$49.3\pm 2.0$
3	0.2	$13.4\pm 1.6$	$47.8\pm 1.8$
3	0.3	$16.3\pm 2.1$	$54.0\pm 1.0$
3	0.4	$19.4\pm 2.7$	$56.8\pm 1.0$
3	0.5	$21.8\pm 3.2$	$60.8\pm 3.4$
7	0.1	$7.9\pm 0.5$	$52.2\pm 3.2$

Appendix

Table 3: QED_HGRAPH Certification Results

$N$	$\alpha$	Error Rate (%)	Certified Fraction (%)
3	0.1	$7.6\pm 1.2$	$42.6\pm 6.1$
3	0.2	$10.4\pm 0.6$	$54.6\pm 3.9$
3	0.3	$14.2\pm 1.3$	$52.2\pm 2.7$
3	0.4	$15.7\pm 2.1$	$57.0\pm 3.6$
3	0.5	$17.2\pm 3.7$	$62.7\pm 4.1$
7	0.1	$11.9\pm 1.4$	$52.2\pm 3.5$

Appendix

Table 4: QED_SELFEDIT Certification Results

$N$	$\alpha$	Error Rate (%)	Certified Fraction (%)
5	0.1	$8.4\pm 2.6$	$42.9\pm 2.5$
5	0.2	$10.8\pm 3.0$	$52.1\pm 2.9$
5	0.3	$11.4\pm 2.3$	$57.7\pm 3.8$
5	0.4	$14.9\pm 4.4$	$64.3\pm 3.6$
5	0.5	$17.1\pm 3.0$	$68.5\pm 3.4$
10	0.1	$5.3\pm 2.5$	$57.8\pm 4.3$

Appendix

Table 5: TARGETDIFF Certification Results

$N$	$\alpha$	Error Rate (%)	Certified Fraction (%)
5	0.1	$7.0\pm 1.2$	$40.1\pm 2.7$
5	0.2	$11.0\pm 1.8$	$47.9\pm 3.5$
5	0.3	$12.7\pm 1.0$	$55.5\pm 4.9$
5	0.4	$15.1\pm 1.2$	$63.1\pm 4.9$
5	0.5	$16.6\pm 1.2$	$68.1\pm 3.8$
10	0.1	$5.2\pm 1.6$	$58.3\pm 5.8$

Appendix

Table 6: MOLCRAFT Certification Results

$N$	$\alpha$	Error Rate (%)	Certified Fraction (%)
5	0.1	$3.2\pm 2.2$	$56.6\pm 1.4$
5	0.2	$3.1\pm 1.9$	$63.5\pm 1.2$
5	0.3	$4.7\pm 2.0$	$69.0\pm 2.4$
5	0.4	$5.2\pm 2.4$	$73.4\pm 0.8$
5	0.5	$5.4\pm 2.6$	$77.0\pm 1.2$
10	0.1	$2.1\pm 1.6$	$74.4\pm 4.0$

Appendix

Table 7: DECOMPDIFF Certification Results

$N$	$\alpha$	Error Rate (%)	Mean Set Size	Empty Set (%)
3	0.1	$4.2\pm 0.3$	$1.9\pm 0.0$	$68.8\pm 1.3$
3	0.2	$6.6\pm 0.5$	$1.8\pm 0.0$	$46.1\pm 1.3$
3	0.3	$9.1\pm 0.6$	$1.8\pm 0.0$	$25.3\pm 1.8$
3	0.4	$11.0\pm 0.8$	$1.6\pm 0.0$	$15.0\pm 1.3$
3	0.5	$13.0\pm 0.8$	$1.5\pm 0.0$	$9.9\pm 0.8$
5	0.1	$5.4\pm 1.6$	$2.4\pm 0.7$	$47.9\pm 2.6$

Appendix

Table 8: DRD2_HGRAPH Design Results

$N$	$\alpha$	Error Rate (%)	Mean Set Size	Empty Set (%)
3	0.1	$9.8\pm 1.3$	$2.0\pm 0.0$	$64.5\pm 0.5$
3	0.2	$14.6\pm 2.2$	$1.9\pm 0.1$	$42.3\pm 0.2$
3	0.3	$19.6\pm 3.2$	$1.7\pm 0.0$	$24.0\pm 0.9$
3	0.4	$22.5\pm 3.2$	$1.6\pm 0.0$	$12.3\pm 0.1$
3	0.5	$24.5\pm 4.0$	$1.5\pm 0.0$	$7.3\pm 0.6$
5	0.1	$9.5\pm 1.2$	$2.6\pm 0.1$	$53.0\pm 1.5$

Appendix

Table 9: DRD2_SELFEDIT Design Results

$N$	$\alpha$	Error Rate (%)	Mean Set Size	Empty Set (%)
3	0.1	$10.1\pm 0.7$	$1.2\pm 0.0$	$70.7\pm 1.9$
3	0.2	$17.0\pm 0.6$	$1.3\pm 0.0$	$45.4\pm 1.6$
3	0.3	$21.2\pm 1.4$	$1.3\pm 0.1$	$26.8\pm 0.9$
3	0.4	$25.2\pm 1.5$	$1.3\pm 0.0$	$13.2\pm 1.2$
3	0.5	$28.2\pm 1.7$	$1.2\pm 0.0$	$6.9\pm 0.6$
5	0.1	$12.2\pm 0.7$	$2.6\pm 0.0$	$27.2\pm 0.2$

Appendix

Table 10: QED_HGRAPH Design Results

$N$	$\alpha$	Error Rate (%)	Mean Set Size	Empty Set (%)
3	0.1	$10.1\pm 0.4$	$1.2\pm 0.0$	$64.8\pm 1.8$
3	0.2	$14.4\pm 0.9$	$1.3\pm 0.1$	$45.2\pm 1.5$
3	0.3	$18.8\pm 1.4$	$1.3\pm 0.1$	$27.1\pm 1.8$
3	0.4	$21.5\pm 1.9$	$1.2\pm 0.1$	$13.3\pm 1.0$
3	0.5	$23.4\pm 1.9$	$1.2\pm 0.0$	$7.0\pm 1.0$
5	0.1	$11.9\pm 0.8$	$2.7\pm 0.1$	$26.0\pm 2.4$

Appendix

Table 11: QED_SELFEDIT Design Results

$N$	$\alpha$	Error Rate (%)	Mean Set Size	Empty Set (%)
5	0.1	$10.0\pm 1.5$	$2.5\pm 0.1$	$37.9\pm 2.6$
5	0.2	$17.4\pm 1.5$	$2.2\pm 0.1$	$21.4\pm 3.1$
5	0.3	$27.2\pm 3.5$	$2.0\pm 0.1$	$8.6\pm 0.0$
5	0.4	$34.2\pm 2.9$	$1.7\pm 0.1$	$2.1\pm 1.4$
5	0.5	$37.0\pm 1.2$	$1.6\pm 0.0$	$0.4\pm 0.7$
10	0.1	$9.4\pm 2.7$	$3.7\pm 0.1$	$21.9\pm 8.0$

Appendix

Table 12: TARGETDIFF Design Results

$N$	$\alpha$	Error Rate (%)	Mean Set Size	Empty Set (%)
5	0.1	$9.8\pm 1.2$	$2.6\pm 0.1$	$40.7\pm 1.9$
5	0.2	$18.7\pm 1.4$	$2.3\pm 0.2$	$20.7\pm 5.6$
5	0.3	$26.0\pm 3.1$	$2.0\pm 0.1$	$7.7\pm 0.7$
5	0.4	$29.3\pm 5.3$	$1.7\pm 0.0$	$2.4\pm 0.0$
5	0.5	$32.9\pm 4.2$	$1.6\pm 0.0$	$1.2\pm 1.2$
10	0.1	$9.9\pm 1.3$	$3.6\pm 0.3$	$22.2\pm 3.3$

Appendix

Table 13: MOLCRAFT Design Results

$N$	$\alpha$	Error Rate (%)	Mean Set Size	Empty Set (%)
5	0.1	$9.6\pm 2.6$	$2.6\pm 0.3$	$25.0\pm 2.2$
5	0.2	$16.7\pm 4.7$	$2.3\pm 0.2$	$11.7\pm 0.7$
5	0.3	$24.2\pm 0.7$	$2.0\pm 0.0$	$2.5\pm 2.2$
5	0.4	$26.7\pm 0.7$	$1.7\pm 0.1$	$1.2\pm 0.0$
5	0.5	$28.8\pm 1.2$	$1.6\pm 0.1$	$0.8\pm 0.7$
10	0.1	$11.2\pm 0.5$	$3.4\pm 0.1$	$13.5\pm 0.9$

Appendix

Table 14: DECOMPDIFF Design Results

Figure 7: Comparison of score statistics on DRD2 ( HGraph2Graph , N=3). Left: rejection (power) at $\alpha=0.1,0.3$ . Right: error vs. $\alpha$

Figure 8: Comparison of score statistics on DRD2 ( HGraph2Graph , N=5). Left: rejection (power) at $\alpha=0.1,0.3$ . Right: error vs. $\alpha$

Figure 9: Comparison of score statistics on DRD2 ( HGraph2Graph , N=7). Left: rejection (power) at $\alpha=0.1,0.3$ . Right: error vs. $\alpha$

Figure 10: Comparison of score statistics on SBDD ( Targetdiff , N=5). Left: rejection (power) at $\alpha=0.1,0.3$ . Right: error vs. $\alpha$

Figure 11: Comparison of score statistics on SBDD ( Targetdiff , N=10). Left: rejection (power) at $\alpha=0.1,0.3$ . Right: error vs. $\alpha$

Figure 12: Density of conformal nested p-values for QED using SelfEdit at set sizes $k=3,5,7$ .

Figure 13: Density of conformal nested p-values for DRD2 using Hgraph at set sizes $k=3,5,7$ .

Figure 14: Density of conformal nested p-values for SBDD using TargetDiff at set sizes $k=1,5,10$ .

Panel of Figure 15 — Figure 15: Effect of predictor quality on ConfHit ’s error rate on the CMO task. In both DRD2 (top) and QED (bottom) using the Hgraph model, noisy and inverse predictors exhibit reduced power but maintain valid error control across all values of $\alpha$ .

Figure 16: Results for heuristic baseline with predicted probabilities (while varying predictor quality), showing rejection rates and error rates for QED with $N=5$ using the Hgraph model. As predictor quality deteriorates, the performance quickly drops with severe violation of the error control.

Figure 17: Conformal LM baseline for molecule optimization. Error rate (left) and empty–set percentage (right) versus $\alpha$ for QED and DRD2 at N = 5 using the Hgraph model. The method breaks coverage and underperforms ConfHit despite relying on an oracle that is unavailable in real-world settings.

Panel of Figure 18 — Figure 18: Effect of power transformations $w^{\gamma}$ of likelihood-ratio weights on QED and DRD2 using the HGraph2Graph model at $N=5$ at different $\alpha$ . Across all transformations, ConfHit maintains a graceful degradation of error control with major changes in terms of the rejection rate.

Panel of Figure 19 — Figure 19: P-value distributions on scaffold-split calibration negatives for QED (left) and DRD2 (right). In both cases, calibration produces p-values that closely follow the uniform distribution, indicating that the density correction remains reliable under scaffold-based distribution shift.

Figure 20: Feature means for QED using the Hgraph model at $N=5$ . Density correction aligns the test features more closely with the calibration distribution.

Figure 21: Feature means for DRD2 using the Hgraph model at $N=5$ . Density correction reduces the discrepancy between calibration and test distributions.

Molecular Generation

Molecular generation develops computational methods to create chemical structures and conformations for discovery, optimization, and structure elucidation.

See all 235 papers

Conf-Gen: Conformal Uncertainty Quantification for Generative Models
May 27, 2026Gabriel Loaiza-Ganem, Kevin Zhang, Wei Cui +2Uncertainty Quantification Conformal Prediction
Layer 6 AI, Toronto, Canada
Closed-Loop Generative Selection: Convergence, Memory, and Noisy Oracles
Jul 24, 2026Konstantin Fackeldey, Christof SchütteStochastic Optimization Runtime Analysis of Evolutionary Algorithms
Zuse Institute Berlin, Takustraße 7, 14195 Berlin, Germany · Institute for Mathematics, Technical University Berlin, Fasanenstr 89,10623 Berlin, Germany · Department of Mathematics and Computer Science, Freie Universität Berlin, Arnimallee 6, 14195 Berlin, Germany
FlashMol: High-Quality Molecule Generation in as Few as Four Steps
May 7, 2026Xinyuan Wei, Zian Li, Shaoheng Yan +2Diffusion Distillation Few-Step Diffusion Sampling
Institute for Artificial Intelligence, Peking University · Yuanpei College, Peking University · School of Intelligence Science and Technology, Peking University +2

May 27, 2026J/K move · Enter open · S save

Conf-Gen: Conformal Uncertainty Quantification for Generative Models

Gabriel Loaiza-Ganem, Kevin Zhang, Wei Cui, Marc T. Law+1

Layer 6 AI, Toronto, Canada

Uncertainty Quantification Conformal Prediction Generative Modeling Conformal Risk Control Conformal Prediction for Language Models

Open paper AskPDF arXiv

Conformal prediction (CP) and its extension, conformal risk control (CRC), are established frameworks for quantifying uncertainty in supervised machine learning through formal guarantees. However, recent breakthroughs in artificial intelligence (AI) have been driven by unsupervised generative models, such as large language models (LLMs) and image generators, which are not directly compatible with CP or CRC. In this work we introduce conformal generation (Conf-Gen), a general framework adapting CRC to generative tasks while relaxing its theoretical assumptions. Conf-Gen unifies and generalizes previous attempts to apply CP to LLMs, and extends conformal methodology to entirely new domains. We demonstrate the flexibility of Conf-Gen through some novel applications, including obtaining conformal guarantees on: image generators producing non-memorized images, conversational AI systems having asked enough clarifying questions, and the output of AI agents being correct.

Abstract

Molecular Generation

Explore similar work

Conf-Gen: Conformal Uncertainty Quantification for Generative Models

Closed-Loop Generative Selection: Convergence, Memory, and Noisy Oracles

FlashMol: High-Quality Molecule Generation in as Few as Four Steps