Scientific Discovery under Validation Congestion via Multi-Fidelity Pairwise Rankings

Kevin Tirta Wijaya; Alston Lo; Michael Sun; Wojciech Matusik; Vahid Babaei

cs.LGOct 1, 2026

Scientific Discovery under Validation Congestion via Multi-Fidelity Pairwise Rankings

Authors: Kevin Tirta Wijaya, Alston Lo, Michael Sun, Wojciech Matusik, Vahid Babaei

Organizations: University of Bonn · Fraunhofer SCAI · Massachusetts Institute of Technology

Abstract

Modern computational methods can now propose candidate molecules, materials, and other scientific designs at an unprecedented scale, creating a validation congestion where candidates are abundant, but experimental capacity to physically evaluate them remains scarce. Discovering novel scientific designs has therefore become increasingly dependent on curation: selecting a small set of promising designs for slow and costly experiments. Existing curation methods typically rely on data-driven regression models that predict absolute scores, but training these models requires substantial experimental data to begin with. Yet, useful curation signals do not have to take the form of absolute measurements, as scientific design discovery is often comparative in nature. Here, we propose that curation can instead be primarily driven by expert pairwise rankings, which are substantially easier to gather. The expertise can come from computational tools or human input of multiple levels of fidelity, ranging from empirical rules of thumb to agentic workflows and experienced scientists. We introduce PRISMS, a framework that uses pairwise rankings from one or more experts, potentially spanning multiple levels of expertise, to identify the most promising candidates without relying on data-hungry regressors. When experts differ in fidelity and cost, PRISMS escalates pairwise queries from lower- to higher-fidelity rankers based on a Fisher-information criterion. In iterative screening that selects designs from fixed drug discovery libraries, PRISMS achieves 50% top-10 discovery recall in ~42% fewer rounds than regression-only active learning, and in ~15% fewer rounds than the ranking-based method with no selective escalation. In optimization that generates new designs without restriction to a predefined library, PRISMS achieves ~18.8% higher hypervolume than the Bayesian optimization baseline.

Figures & tables

Figure 1: Accelerating scientific discovery through curation. Conventional curation methods predict absolute scores from scarce experimental data, while PRISMS curates designs using selectively queried multi-fidelity pairwise rankers. Better curation sends more promising designs for experiments at each round, reducing the number of design-experiment loops needed for discovery.

Figure 2: Overview of PRISMS. (Left) A rank outcome graph connects unmeasured designs (circles) to all experimentally validated designs (squares) and some unmeasured designs. (Middle) Starting with low-fidelity rankings on all eligible edges, a regularized Bradley-Terry model jointly refines property estimates of all unmeasured designs. Select pairs (edges) are escalated to higher-fidelity rankers based on a Fisher information criterion, and the estimates are further refined after exhausting the budget of a ranker

g_{\ell}

. (Right) The highest-ranked designs are selected for experimental validation, subject to a budget. New measurements update the rank outcome graph for the next round.

Figure 3: Aggregated results of 12-round scientific design discovery campaigns on 14 drug discovery datasets (ADMET and binding affinity). We report the mean and standard error over 30 random seeds. Left. Top-10 discovery recall over 12 rounds. Right. AUC of Top-10% F1 score.

Figure 4: Normalized hypervolume over 12 campaign rounds and 30 random seeds for multi-objective optimization targeting JNK3 activity and solubility

Figure 5: Sensitivity to higher-fidelity ranker accuracy and query budget under three ranking-noise structures. (Top) : PRISMS’ AUDC@10. (Bottom) : AUDC@10 differences between PRISMS and RankRefine++ under matched settings.

Figure 6: Recovery of true top-10 designs across campaign rounds. Heatmaps show the distribution of normalized Earth Mover’s Distance ( Rubner et al., 1998 ) for top-10 designs ( $\mathrm{EMD}_{\text{norm}}@10$ , lower is better) over 12 rounds across 14 drug-discovery datasets and 30 random seeds. An $\mathrm{EMD}_{\text{norm}}@10=0.0$ places all true top-10 designs in the top-10 positions, whereas $\mathrm{EMD}_{\text{norm}}@10=1.0$ places them in the bottom-10 positions. Across rounds, PRISMS shifts true top-10 designs toward the highest ranks faster than competing methods.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Sensitivity of PRISMS to prior strength

\lambda_{2}

, query-query neighborhood size

k^{U}

, and low- and high-fidelity ranker weights

\lambda_{1}^{(1)}

and

\lambda_{1}^{(2)}

. Each parameter is varied while the others remain at their default values, indicated by vertical dashed lines. Points show mean AUDC@10.

Figure 8: Pairwise ranking accuracy of the LLM-based ranker as a function of the standardized true property gap

d_{ij}=|y_{i}-y_{j}|/s_{y}

, where

s_{y}

is the dataset-level standard deviation of the property values. Accuracy generally increases with property gap, indicating a difficulty-dependent error structure in which pairs with similar property values are more difficult for the LLM to rank correctly.

Method	12-round AUC				24-round RMST
	Recall	NDCG	Top-1% F1	Top-5% F1	to 50% Recall
Random Sampling	0.077 (0.005)	0.334 (0.057)	0.120 (0.015)	0.134 (0.005)	23.3
GP-EI	0.141 (0.015)	0.398 (0.057)	0.192 (0.027)	0.206 (0.010)	19.3
GP-EI Pairwise	0.227 (0.026)	0.416 (0.067)	0.313 (0.043)	0.260 (0.013)	17.9
HodgeRank	0.306 (0.042)	0.490 (0.055)	0.372 (0.055)	0.326 (0.024)	13.5
Active top-k	0.306 (0.043)	0.487 (0.055)	0.248 (0.022)	0.307 (0.017)	13.6

Appendix

Table 1: Experimental results for iterative screening in drug discovery datasets, repeated over 30 random seeds.

Figure 9: Paired difference in area under the top-10 discovery recall curve for RankRefine++ (left) and HodgeRank (right). Positive values indicate higher discovery performance for PRISMS.

Figure 10: Aggregated results of 12-round scientific design discovery campaigns on three engineering datasets. We report the mean and standard error over 10 random seeds. Left. Top-10 discovery recall over 12 rounds. Right. AUC of Top-10% F1 score.

Method	AUDC@10 $\uparrow$	Relative change
PRISMS 2 stage	$0.527\pm 0.049$	—
No unlabeled-unlabeled edges	$0.487\pm 0.048$	$-7.6\%$
No edge uncertainty	$0.502\pm 0.047$	$-4.6\%$
No outcome uncertainty	$0.514\pm 0.048$	$-2.3\%$
Random escalation	$0.486\pm 0.050$	$-7.7\%$
No lateral edges + random escalation	$0.481\pm 0.051$	$-8.7\%$

Appendix

Table 2: Ablation of PRISMS components and fidelity stages. We report the 24-round mean area under the discovery recall curve (AUDC) across 14 drug-discovery datasets, repeated for 10 random seeds, with standard error of the mean. Relative change is computed with respect to the default two-stage PRISMS configuration.

Dataset Name	Dataset Size	Target Property
Caco-2 efflux	$3777$	Caco-2 efflux ratio
Caco-2 Papp	$3773$	Caco-2 permeability
Human LM	$4541$	microsomal clearance
Kinetic solubility	$7298$	kinetic solubility
LogD	$7309$	LogD
Mouse LM	$5692$	microsomal clearance

Appendix

Table 3: Details of datasets used in the experiments.

Explore similar work

May 11, 2026cs.LG

Many Needles in a Haystack: Active Hit Discovery for Perturbation Experiments

High-throughput gene perturbation experiments can test several genetic interventions in parallel, yet experimental budgets remain limited. A central goal is hit discovery: identifying as many perturbations as possible whose phenotypic effect exceeds a predefined threshold. Pure exploration strategies are statistically inefficient, wasting budget on low-value regions. Bayesian optimization methods offer a principled alternative but target a single global optimum, over-exploiting dominant modes while neglecting other high-value regions. We formalize hit discovery as a sequential experimental design problem and propose Probability-of-Hit, an acquisition function that directly targets threshold exceedance by ranking candidates according to their posterior probability of being a hit. We prove asymptotic optimality of this approach and demonstrate strong empirical performance on both synthetic benchmarks and real biological immunology datasets, including up to 6.4% improvement over baselines on the Schmidt IL-2 dataset.

Andrea Rubbi, Arpit Merchant, Samuel Ogden +4

Department of Computer Science and Technology, University of Cambridge, Cambridge, UK · Cambridge Center for AI in Medicine, University of Cambridge, Cambridge, UK · MediaTek Research, Cambridge, UK +1

Bayesian Experimental Design Bayesian Optimization Search Algorithms False Discovery Rate Drug Discovery Discovery+1

Ask PDF

May 28, 2026cs.CL

From Blind Guess to Informed Judgment: Teaching LLMs to Evaluate Materials by Building Knowledge-Augmented Preference Signals

As candidate generation and high-throughput experimentation advance, the primary bottleneck in materials discovery is shifting from property prediction to making reliable evaluations among massive candidate sets. We propose a Knowledge-Augmented Preference Signals Framework, MaterEval, that automatically produces, for the same candidate, two evaluations: an informed judgment that follows expert rules and provides supporting evidence, and a rule-removed blind guess. By pairing the two evaluations as preference data, we guide general-purpose large language models (LLMs), originally lacking materials-specific criteria, from intuitive judgment toward reliable evaluation supported by explicit evidence. To balance throughput, cost, and reliability, we further introduce a fast-slow reasoning scheme that decouples large-scale rapid screening from in-depth review on a small subset. Using high-entropy alloy (HEA) assessment as a case study, we show that, without external retrieval and relying solely on internalized capabilities, small open-source LLMs achieve substantial gains in accuracy, conclusion consistency, and evidence discrimination, approaching the performance of rule-based closed-source LLMs. These results demonstrate that expert rules can be systematically transformed into learnable preference signals, enabling a low-cost and deployable evaluation module for autonomous materials discovery loops.

Yeyong Yu, Wenya Hu, Xing Wu +1

School of Computer Engineering & Science, Shanghai University, Shanghai, 200444, China. · Center of Materials Informatics and Data Science, Materials Genome2026 Institute, Shanghai University, Shanghai, 200444, China. · Key Laboratory of Silicate Cultural Relics Conservation (Shanghai University), Ministry of Education, China. +1

Data-Driven Materials Discovery Materials Preference Alignment Learning

Ask PDF

Jun 25, 2026cs.AI

Scientific discovery as meta-optimization: a combinatorial optimization case study

Scientific discovery is fundamentally an optimization problem, defined by a vast "state space" of theories and experiments, and an evaluation criterion based on quality, novelty, and validity. Large language models (LLMs) have enabled automated exploration of this space, but we argue that simultaneous modification of the evaluation criteria is equally important. Here, we propose formalizing research as meta-optimization, where the optimization objective itself is also being optimized. Our key contribution is "consensus objective aggregation," where LLM-generated objective functions are combined via correlation-weighted voting, yielding a stable, self-correcting evaluation criterion that evolves as understanding deepens. We apply this framework to algorithm discovery for 3-SAT problems based on digital MemComputing machines, reducing the baseline scaling with problem size

N

from

\sim N^{2.51}

\sim N^{1.33}

and delivering a

\sim 67\times

speedup on the largest instances tested. As a problem-agnostic framework, we hope this approach will considerably aid scientific discovery.

Yuan-Hang Zhang, Chesson Sipling, Massimiliano Di Ventra

Scientific Discovery Model Discovery Optimization Modeling Combinatorial Optimization

Ask PDF