Budget-Sensitive Discovery Scoring: A Formally Verified Framework for Evaluating AI-Guided Scientific Selection

Abhinaba Basu; Pavan Chakraborty

cs.LGMar 12, 2026

Budget-Sensitive Discovery Scoring: A Formally Verified Framework for Evaluating AI-Guided Scientific Selection

Authors: Abhinaba Basu, Pavan Chakraborty

Organizations: Indian Institute of Information Technology Allahabad (IIITA) and National Institute of Electronics & Information Technology (NIELIT). · Indian Institute of Information Technology Allahabad (IIITA).

Abstract

Scientific discovery increasingly relies on AI systems to select candidates for expensive experimental validation, yet no principled, budget-aware evaluation framework exists for comparing selection strategies -- a gap intensified by large language models (LLMs), which generate plausible scientific proposals without reliable downstream evaluation. We introduce the Budget-Sensitive Discovery Score (BSDS), a formally verified metric -- 20 theorems machine-checked by the Lean 4 proof assistant -- that jointly penalizes false discoveries (lambda-weighted FDR) and excessive abstention (gamma-weighted coverage gap) at each budget level. Its budget-averaged form, the Discovery Quality Score (DQS), provides a single summary statistic that no proposer can inflate by performing well at a cherry-picked budget. As a case study, we apply BSDS/DQS to: do LLMs add marginal value to an existing ML pipeline for drug discovery candidate selection? We evaluate 39 proposers -- 11 mechanistic variants, 14 zero-shot LLM configurations, and 14 few-shot LLM configurations -- using SMILES representations on MoleculeNet HIV (41,127 compounds, 3.5% active, 1,000 bootstrap replicates) under both random and scaffold splits. Three findings emerge. First, the simple RF-based Greedy-ML proposer achieves the best DQS (-0.046), outperforming all MLP variants and LLM configurations. Second, no LLM surpasses the Greedy-ML baseline under zero-shot or few-shot evaluation on HIV or Tox21, establishing that LLMs provide no marginal value over an existing trained classifier. Third, the proposer hierarchy generalizes across five MoleculeNet benchmarks spanning 0.18%-46.2% prevalence, a non-drug AV safety domain, and a 9x7 grid of penalty parameters (tau >= 0.636, mean tau = 0.863). The framework applies to any setting where candidates are selected under budget constraints and asymmetric error costs.

Figures & tables

Figure 1: Evaluation pipeline for 39 proposers across five categories: baselines (2), mechanistic ablations (5), direct optimization (BSDS-Recursive), ablation controls (3), and real LLMs (28). All proposers are evaluated by the formally verified

\mathrm{BSDS}

\mathrm{DQS}

metric with 1,000 bootstrap replicates. Multi-round proposers receive evaluation feedback (dashed arrow).

Mechanistic Ablations
Proposer	ML	Structure	Stochastic	Multi-round	Coverage
Random	No	No	Yes	No	Full
Greedy-ML	Yes	No	No	No	Full
Informed-Prior	Yes	Yes	No	No	Full
Retrieval	Yes	Yes	No	No	Full
Generative	Yes	No	Yes	No	Full

Table 1: Summary of the 39 proposer strategies. “ML” indicates use of pre-trained RF scores; “Structure” indicates use of fingerprints or descriptors; “Stochastic” indicates non-deterministic selection; “Multi-round” indicates iterative refinement with label feedback.

Proposer	$B/N=0.01$	$B/N=0.05$	$B/N=0.20$	$\mathrm{DQS}_{\mathrm{boot}}$ [95% CI]
Random	$-0.953$	$-0.895$	$-0.782$	$-0.819$ $[-0.834,-0.803]$
Greedy-ML	$\mathbf{-0.003}$	$\mathbf{-0.010}$	$\mathbf{-0.121}$	$\mathbf{-0.046}$ $[-0.076,-0.019]$
Informed-Prior	$-0.297$	$-0.232$	$-0.303$	$-0.178$ $[-0.210,-0.146]$
Retrieval	$-0.056$	$-0.033$	$-0.138$	$-0.091$ $[-0.121,-0.062]$
Generative	$-0.906$	$-0.822$	$-0.548$	$-0.670$ $[-0.691,-0.649]$
BSDS-Guided	$-0.236$	$-0.184$	$-0.248$	$-0.234$ $[-0.264,-0.202]$

Table 2: Per-budget

\mathrm{BSDS}

at seed 0 (point estimates) and bootstrap-averaged

\mathrm{DQS}

with 95% BCa confidence intervals (1,000 seeds). Best value per column in bold.

Variant	$\mathrm{DQS}$	$\mathrm{BSDS}_{0.05}$	What is ablated
BSDS-Recursive (full)	$-0.323$	$-0.298$	— (reference)
BSDS-NoAug	$-0.296$	$-0.269$	No recursive feature augmentation
BSDS-1Round	$-0.258$	$-0.223$	Single optimization round only
Greedy-MLP-NN	$-0.315$	$-0.281$	BCE loss (standard ML)
Greedy-ML (RF)	$\mathbf{-0.046}$	$\mathbf{-0.010}$	RF baseline

Table 3: Ablation analysis: isolating the contribution of each BSDS-Recursive component. All models share the same 5-fold CV protocol and 10-dimensional feature set.

Figure 2:

\mathrm{BSDS}

as a function of budget fraction

B/N

for all 11 mechanistic proposers (2 baselines, 5 ablations, BSDS-Recursive, and 3 ablation controls), with 95% bootstrap confidence bands. Greedy-ML dominates all proposers across budget levels.

Figure 3: Bootstrap distributions of

\mathrm{DQS}

for all mechanistic proposers (1,000 replicates). Greedy-ML has the lowest variance; Retrieval achieves the most favorable individual budget-level scores.

Proposer	$\mathrm{HR}$	$\mathrm{FDR}$	$\mathrm{PPV}$	TP	$\mathrm{BSDS}$
Random	0.062	0.957	0.043	89	$-0.895$
Greedy-ML	0.581	0.592	0.408	839	$\mathbf{-0.010}$
Informed-Prior	0.451	0.683	0.317	651	$-0.232$
Retrieval	0.568	0.601	0.399	820	$-0.033$
Generative	0.105	0.927	0.073	151	$-0.822$
BSDS-Guided	0.521	0.634	0.366	752	$-0.184$

Table 4: Component metrics at

B/N=0.05

(seed 0).

\mathrm{BSDS}=\mathrm{HR}-\lambda\,\mathrm{FDR}-\gamma\,(1-\mathrm{Cov})

, with

\lambda=1.0

\gamma=0.3

\mathrm{PPV}

= positive predictive value; TP = true positive count.

Model	Mode	$\mathrm{DQS}$	$\mathrm{BSDS}_{0.05}$
ChatGPT-5.2	Direct	$-0.683$	$-0.788$
ChatGPT-5.2	Rerank	$-0.159$	$-0.165$
Claude	Direct	$-0.678$	$-0.813$
Claude	Rerank	$-0.161$	$-0.170$
Gemini	Direct	$-0.585$	$-0.668$
Gemini	Rerank	— †	— †

Table 5: LLM proposer performance: budget-averaged

\mathrm{DQS}

(1,000 bootstrap seeds) and

\mathrm{BSDS}

B/N=0.05

. Direct = zero-shot LLM ranking; Rerank = LLM reranking of Greedy-ML shortlist. Best LLM value per column in bold.

Model	Mode	$\mathrm{DQS}$	$\mathrm{BSDS}_{0.05}$
ChatGPT-5.2	FewDirect	$-0.624$	$-0.720$
ChatGPT-5.2	FewRerank	$-0.138$	$-0.160$
Claude	FewDirect	$-0.547$	$-0.656$
Claude	FewRerank	$-0.141$	$-0.132$
Gemini	FewDirect	— †	— †
Gemini	FewRerank	— †	— †

Table 6: Few-shot LLM performance (

k{=}3

active +

k{=}3

inactive examples). FewDirect = few-shot without ML scores; FewRerank = few-shot with RF scores. Values in italics indicate improvement over the corresponding zero-shot mode (cf. Table 5 ). Best few-shot value per column in bold.

Model	Mode	HIV $\mathrm{DQS}$	Tox21 $\mathrm{DQS}$
DeepSeek-V3.1	Direct	$-0.774$	$-0.401$
	Rerank	$-0.318$	$-0.071$
	FewDirect	$-0.747$	$-0.383$
	FewRerank	$-0.236$	$-0.065$
Llama-4-Maverick	Direct	$-0.802$	$-0.363$
	Rerank	— †	$-0.042$

Table 7: LLM performance on Tox21 NR-AR-LBD (3.5% prevalence). Same protocol as HIV (Tables 5 – 6 ). Greedy-ML DQS on Tox21

=+0.086

. No LLM surpasses the RF baseline.

Model	$\mathrm{DQS}$	$\mathrm{BSDS}_{0.05}$	AUROC	EF@1%
Greedy-ML (ECFP4 + RF)	$-0.046$	$-0.010$	$0.854$	$22.1\times$
Greedy-MolFormer	$-0.150$	$-0.178$	$0.806$	$20.5\times$
Greedy-ChemBERTa	$-0.184$	$-0.200$	$0.785$	$19.6\times$
BSDS-Recursive (ECFP4)	$-0.323$	$-0.298$	$0.776$	$22.1\times$

Table 8: Foundation model baselines: frozen pretrained [CLS] embeddings + RF with greedy (top-

B

) selection. Same 5-fold CV protocol. Seed-0 (full dataset) values.

Figure 4:

\mathrm{DQS}

as a function of softmax temperature

\tau

for the Generative proposer. Performance degrades monotonically with increasing

\tau

; even the optimal

\tau^{*}=0.1

yields negative

\mathrm{DQS}

Figure 5: NL framing invariance. Dots show mean $\mathrm{DQS}$ across six requirement phrasings; horizontal bars show the min–max spread. All proposers except Informed-Prior exhibit zero spread ( $\Delta<0.01$ ), confirming that $\mathrm{DQS}$ is invariant to NL framing.

Proposer	$(0.1,0.1)$	$(1.0,0.3)$	$(2.0,0.5)$	$(5.0,1.0)$
Greedy-ML	$\mathbf{+0.515}$	$\mathbf{-0.046}$	$\mathbf{-0.670}$	$\mathbf{-2.540}$
Retrieval	$+0.484$	$-0.091$	$-0.729$	$-2.644$
Ensemble	$+0.396$	$-0.102$	$-0.649$	$-2.270$
Informed-Prior	$+0.429$	$-0.178$	$-0.851$	$-2.873$
BSDS-Guided	$+0.415$	$-0.234$	$-0.950$	$-3.077$

Table 9: DQS for the top-5 proposers at four representative

(\lambda,\gamma)

pairs spanning the sensitivity grid. Greedy-ML maintains the top position across all settings.

Figure 6: Parameter sensitivity analysis across a $9\times 7$ grid of $(\lambda,\gamma)$ values. (a) DQS of BSDS-Recursive: absolute performance varies with parameters (positive at low penalties, negative at high), but the star marks the default $(1.0,0.3)$ . (b) Kendall $\tau$ of the full proposer ranking vs. the default ranking: $\tau\geq 0.636$ across all 63 parameter pairs (mean $\tau=0.863$ ), confirming that the proposer hierarchy is robust to parameter choice.

Proposer	HIV	Tox21	ClinTox	MUV-466	SIDER-Ear
	3.5%	3.5%	7.5%	0.18%	46.2%
Random	$-0.819$	$-0.817$	$-0.776$	$-0.849$	$-0.390$
Greedy-ML	$\mathbf{-0.046}$	$+0.086$	$\mathbf{-0.278}$	$\mathbf{-0.628}$	$\mathbf{+0.019}$
Informed-Prior	$-0.178$	$-0.046$	$-0.428$	$-0.703$	$-0.097$
Retrieval	$-0.091$	$+0.058$	$-0.389$	$-0.698$	$-0.083$
Generative	$-0.670$	$-0.624$	$-0.701$	$-0.842$	$-0.330$

Table 10: Cross-dataset DQS for all 11 mechanistic proposers across five MoleculeNet benchmarks spanning 0.18%–46.2% prevalence. Greedy-ML consistently ranks among the top proposers across prevalence regimes. All MLP variants perform worse than Greedy-ML on every dataset.

Figure 7: Cross-dataset DQS comparison for 11 mechanistic proposers across five MoleculeNet benchmarks and AV Safety (non-drug domain). Error bars show 95% bootstrap CIs. Greedy-ML consistently ranks among the top proposers across prevalence regimes from 0.18% to 46.2%. Proposers unavailable for AV Safety (requiring molecular features) are omitted. Horizontal line at DQS

=0

Proposer	AV Safety DQS
Random	$-0.819$
Greedy-ML	$+0.179$
Generative	$+0.018$
BSDS-Recursive	$+0.145$
BSDS-NoAug	$+0.101$
BSDS-1Round	$+0.192$

Table 11: DQS for 7 generic proposers on AV Safety triage (30,000 scenarios, 3.41% safety-critical). Greedy-MLP-NN achieves the highest DQS, diverging from the drug-discovery hierarchy where Greedy-ML leads.

Strategy	$B$	Hits	Hit Rate	Cost ($K)	ROI
Random	50	2	3.4%	250	$-$ 66%
RF (Greedy-ML)	50	48	96.0%	250	860%
BSDS-Recursive	50	31	62.0%	250	520%
Greedy-MLP-NN	50	39	78.0%	250	680%
Random	200	7	3.5%	1000	$-$ 65%
RF (Greedy-ML)	200	172	86.0%	1000	760%

Table 12: Deployment simulation on MoleculeNet HIV: hits, hit rate, and return on investment (ROI) for four ranking strategies across experimental budgets

B

Proposer	Random DQS	Scaffold DQS
Greedy-ML	$\mathbf{-0.046}$	$\mathbf{-0.116}$
Retrieval	$-0.091$	$-0.163$
Ensemble	$-0.102$	$-0.182$
Informed-Prior	$-0.178$	$-0.260$
BSDS-Guided	$-0.234$	$-0.310$
BSDS-1Round	$-0.258$	$-0.321$

Table 13: Random vs. scaffold split comparison on MoleculeNet HIV. DQS values are budget-averaged over 1,000 bootstrap replicates. Greedy-ML remains the best proposer under both split types.

Large Language Model-Guided Scientific Discovery

Large language model-guided scientific discovery uses AI to propose, test, and refine hypotheses, models, experiments, and designs.

See all 274 papers

Jul 29, 2026

Budget-Aware LLM Discovery via Cost-Calibrated Frontier Utility

Large language models increasingly support scientific and algorithmic discovery through inference-time search over evaluated candidates. Existing adaptive discovery controllers assign credit based only on score progress, even though prompt length, retries, and guidance calls cause search actions to incur different token costs. We prove that cost-blind credit can forfeit all but a vanishing fraction of attainable quality as frontiers multiply and costs diverge. Under a fixed search-side token budget, the controller must decide which frontier is improving and whether its gain justifies the realized cost before the budget is exhausted. We introduce CostAda, a cost-calibrated adaptive controller built around cost-calibrated frontier utility. The utility values frontier progress relative to realized action cost and conditions that credit on the remaining budget. CostAda uses this signal to control local exploration intensity, frontier allocation, and budgeted tactic intervention. Cost and remaining budget therefore shape the search rather than serving only as accounting variables or a stopping rule. CostAda reaches the strongest baseline's full-budget quality with at most half the budget on twelve of sixteen benchmark--backbone pairs while achieving the strongest mean final quality on all eight benchmarks under GLM-5 and GPT-5.4.

Yansen Zhang, Yilu Liu, Tianyu Liu +6

International Digital Economy Academy (IDEA) · City University of Hong Kong · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)

Inference-Time Search Automated Algorithm Discovery Cost-Aware Inference Large Language Model-Guided Scientific Discovery

Ask PDF

Oct 17, 2026

CADS: Measuring the Incremental Contribution of LLM Proposers in Sequential Biological Screens

Large language models are increasingly used to guide sequential biological screens, yet standard evaluations rely on raw hit counts that conflate a proposer's value with the discovery capacity of the underlying algorithmic harness. Adaptive sampling and corrupted experimental readouts also leave standard feedback comparisons without a false-discovery guarantee. We present Certified Agentic Discovery Screens (CADS), an error-controlled measurement framework that quantifies a proposer's incremental value against an identical LLM-free loop. By integrating capped likelihood-ratio bounds, e-processes, and complete-family e-BH, CADS controls the terminal false discovery rate under the stated null and corruption conditions despite arbitrary dependence, adaptive sampling, and optional stopping. In DepMap-backed simulations, CADS exposes two evaluation artifacts: the LLM-free harness alone clears the feedback gate, and raw origin counts overstate LLM contributions because LLM picks displace fallback discoveries. On the initial 20 seeds, supplying tested-candidate history to Qwen2.5-7B shifts its net contribution from +0.35, with a 95% interval covering zero, to +2.90, with the interval above zero. This initial net effect did not persist in a 50-seed matched-quota replication. These results support net gain over an LLM-free loop, measured between separately certified sets, as the appropriate measure of proposer contribution.

Guanxing Chen, Nihui Shao, Haohuai He +3

City University of Hong Kong (Dongguan) · The Hong Kong Polytechnic University · Tencent AI Lab +1

Large Language Model-Guided Scientific Discovery LLM Evaluation FDR Control

Ask PDF

May 28, 2026

SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?

Autonomous AI research agents aim to speed up scientific discovery by automating the research pipeline from hypothesis generation to peer review. However, existing benchmarks rarely test a fundamental bottleneck whether LLMs can judge the methodological viability of a research idea

\textbf{before}

investing time and computational resources. Without a reliable ``first-gate'' filter, autonomous agents risk scaling flawed methodology rather than accelerating meaningful science. We introduce

\textbf{SoundnessBench}

, a curated benchmark of 1,099 real research proposals, with source verified extraction, reviewer-derived proxy labels, and proposal-only expert judgments on a subset of 327 proposals. Across 12 frontier LLMs, our evaluation reveals a pervasive

\textbf{optimism bias}

: standard prompting yields a

74.0\%

mean false-positive rate. Aggressive prompting reduces false approvals but substantially increases false rejections. Additional analyses show that optimism persists under controls for public corpus contamination, and surface features, as well as in literature-grounded agent evaluation. These results suggest a limitation in judging scientific soundness, indicating that current LLMs are not yet reliable as standalone gatekeepers for scientific rigor.

Sy-Tuyen Ho, Minghui Liu, Huy Nghiem +6

University of Maryland · All Purpose AI

AI Agent Benchmarks LLM Evaluation LLM Reliability Automated Peer Review AI for Science Scientific Idea Evaluation+1

Ask PDF

Abstract

Large Language Model-Guided Scientific Discovery

Explore similar work

Budget-Aware LLM Discovery via Cost-Calibrated Frontier Utility

CADS: Measuring the Incremental Contribution of LLM Proposers in Sequential Biological Screens

SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?