SciDesignBench: Benchmarking and Improving Language Models for Scientific Inverse Design

David van Dijk; Ivan Vrkic

cs.LGMar 13, 2026

SciDesignBench: Benchmarking and Improving Language Models for Scientific Inverse Design

Authors: David van Dijk, Ivan Vrkic

Organizations: CellType Inc.

Abstract

Many of the most important problems in science and engineering are inverse problems: given a desired outcome, find a design that achieves it. Evaluating whether a candidate meets the spec is often routine; a binding energy can be computed, a reactor yield simulated, a pharmacokinetic profile predicted. But searching a combinatorial design space for inputs that satisfy those targets is fundamentally harder. We introduce SciDesignBench, a benchmark of 520 simulator-grounded tasks across 14 scientific domains and five settings spanning single-shot design, short-horizon feedback, long-horizon refinement, and seed-design optimization. On the 10-domain shared-core subset, the best zero-shot model reaches only 29.0% success despite substantially higher parse rates. Simulator feedback helps, but the leaderboard changes with horizon: Sonnet 4.5 is strongest in one-turn de novo design, whereas Opus 4.6 is strongest after 20 turns of simulator-grounded refinement. Providing a starting seed design reshuffles the leaderboard again, demonstrating that constrained modification requires a fundamentally different capability from unconstrained de novo generation. We then introduce RLSF, a simulator-feedback training recipe. An RLSF-tuned 8B model raises single-turn success rates by 8-17 percentage points across three domains. Together, these results position simulator-grounded inverse design as both a benchmark for scientific reasoning and a practical substrate for amortizing expensive test-time compute into model weights.

Figures & tables

Panel of Figure 1 — Figure 1: SciDesignBench overview. (a) Each benchmark task pairs a natural-language design goal with a ground-truth forward simulator. We evaluate frontier LLMs across five settings and train small models via RLSF . (b) Frontier LLMs achieve only 12–26% zero-shot; simulator feedback helps but saturates below 68%. (c) RLSF training improves an 8B model on three case studies: ADMET optimization (30% $\to$ 41%), PK/PD dosing (24% $\to$ 36%), and docking (42% $\to$ 59% with real AutoDock Vina).

Domain	Oracle (Time)	Design Space	Fidelity	De Novo Task	Optimization Task
Drug Design
ADMET	RDKit ( $<$ 1 ms)	SMILES molecule	Empirical	—	Modify a drug candidate to improve ADMET profile
PK/PD	SciPy ODE ( ${\sim}$ 25 ms)	Dose (mg), freq. (hr)	Model	Design a dosing regimen for target plasma concentrations	Adjust dosing when PK metrics are suboptimal
Docking	AutoDock Vina ( ${\sim}$ 1.5 s)	SMILES molecule	Model	Design a ligand for a protein binding pocket	Modify a lead compound to improve binding fit
Biology
FBA	COBRApy LP ( ${\sim}$ 70 ms)	Gene knockouts	Model	Engineer E. coli for target metabolite production	Modify a strain to redirect metabolic flux

Table 1: Overview of SciDesignBench domains grouped by scientific area. Each domain has 20 tasks (5 per difficulty level L1–L4) in both de novo (design from scratch) and optimization (modify existing design) settings. Fidelity : Exact = mathematically precise; Model = simplified scientific model solved exactly; Empirical = data-derived correlations. Perturbation is de novo only; ADMET is optimization only.

Model	PK/PD	Docking	FBA	SSA	RNA Design	Perturbation	Quantum	Thin Film	Controls	Signal Processing	Alloy	Reactor	Heat Exchanger	Score
	Drug Design		Biology				Physics		Engineering			Chemical Engineering
Sonnet 4.5	38.0 ±8	53.3 ±9	0.0 ±0	20.0 ±9	3.3 ±2	50.0 ±10	48.3 ±11	33.3 ±10	1.7 ±2	30.0 ±11	30.0 ±11	45.0 ±8	26.7 ±7	29.0
Opus 4.6	23.3 ±9	56.7 ±9	3.3 ±2	13.3 ±7	5.0 ±4	53.3 ±11	33.3 ±10	28.3 ±10	3.3 ±3	33.3 ±11	25.0 ±10	30.0 ±10	20.0 ±5	23.8
Sonnet 4.6	29.0 ±9	46.7 ±8	1.7 ±2	16.7 ±8	20.0 ±7	63.3 ±9	26.7 ±10	28.3 ±10	0.0 ±0	30.0 ±11	0.0 ±0	28.3 ±9	33.3 ±9	22.9
Gemini 3.1 Pro	48.0 ±9	31.7 ±10	1.7 ±2	13.3 ±7	23.3 ±7	70.0 ±10	15.0 ±7	18.3 ±8	8.3 ±6	31.7 ±10	23.3 ±9	0.0 ±0	28.3 ±8	24.0
GPT-5.2	38.0 ±10	—	18.3 ±5	18.3 ±7	—	55.0 ±10	33.3 ±10	—	0.0 ±0	30.0 ±11	13.3 ±6	31.7 ±10	18.3 ±8	25.6

Table 2: Frontier LLM evaluation on the 13 official de novo SciDesignBench v1 domains. Success rate (%)

\pm

standard error per domain (20 tasks each,

K{=}3

attempts). The score column is the manifest-defined shared-core score over 10 domains, enabling fair comparison across all seven models. Bold : best. Underline : second best.

Model	1-turn	5-turn	20-turn	$\Delta$ (1 $\to$ 5)	$\Delta$ (5 $\to$ 20)
Sonnet 4.5	29.0	66.5	69.5	37.5	3.0
Opus 4.6	23.8	65.5	76.0	41.7	10.5
Sonnet 4.6	22.9	66.5	71.5	43.6	5.0
Gemini 3.1 Pro	24.0	57.3	60.5	33.4	3.2
GPT-5.2	25.6	62.5	67.0	36.9	4.5
Gemini 2.0 Flash	19.5	43.2	60.5	23.7	17.3

Table 3: Shared-core comparison across the three de novo benchmark settings. All scores are computed over the 10 manifest-defined shared-core de novo domains. Bold : best per column. Underline : second best.

Figure 2: Aggregate success rate vs. turn number in the 20-turn long-horizon setting, averaged across the 10 manifest-defined de novo shared-core domains. Simulator feedback provides rapid initial gains (turns 1–5) but exhibits diminishing returns thereafter. Even at 20 turns, models plateau well below full benchmark saturation.

Figure 3: Success rate vs. turn number in the 20-turn long-horizon setting, per domain. Each line represents one model. Perturbation converges quickly; Controls and SSA show sustained gains through 20 turns; domains such as FBA remain highly model-dependent even with longer trajectories.

Figure 4: Feedback utilization is a distinct capability from static knowledge. Zero-shot success rate (x-axis) vs. feedback utilization (20-turn minus zero-shot, y-axis) for each model. Models with similar one-turn performance can have vastly different long-horizon gains—e.g., Sonnet 4.6 and Gemini 3.1 Pro start within a few points but Sonnet 4.6 realizes a much larger improvement under iterative simulator feedback.

Model	1-turn	5-turn	20-turn	Opt. (1-turn)	Opt. (20-turn)
Sonnet 4.5	29.0	66.5	69.5	30.5	61.5
Opus 4.6	23.8	65.5	76.0	34.5	67.5
Sonnet 4.6	22.9	66.5	71.5	31.2	63.0
Gemini 3.1 Pro	24.0	57.3	60.5	31.3	56.0
GPT-5.2	25.6	62.5	67.0	29.0	58.5
Gemini 2.0 Flash	19.5	43.2	60.5	22.2	48.0

Table 4: Evaluation across five settings. The first three columns use the 10 de novo shared-core domains; optimization columns use the 10 optimization shared-core domains. Bold : best per column. Underline : second best.

Model	20-turn (de novo)	20-turn (opt.)	$\Delta$
Sonnet 4.5	66.1	62.2	-3.9
Opus 4.6	73.3	68.3	-5.0
Sonnet 4.6	68.3	64.4	-3.9
Gemini 3.1 Pro	56.1	55.0	-1.1
GPT-5.2	63.9	59.4	-4.4
Gemini 2.0 Flash	57.8	49.4	-8.3

Table 5: Comparison of 20-turn de novo vs. 20-turn agentic optimization on the 9 domains shared by both manifest-defined shared-core sets.

\Delta

: optimization minus de novo. Bold : best per column. Underline : second best.

Figure 5: RLSF training across three scientific design tasks. (a) ADMET optimization : curriculum GRPO improves from 30% (SFT) to 41% (GRPO), narrowly exceeding the strongest frontier ADMET optimization benchmark result. (b) PK/PD: success improves from 24% to 36% in de novo design and from 32% to 47% in optimization, yielding a competitive optimization result on a coupled dosing-control task. (c) Docking optimization : real AutoDock Vina improves from 42% (SFT) to 59% (GRPO) in the artifact-backed run used in the paper. Dashed vertical lines mark curriculum phase transitions. Points show raw eval; curves are smoothed.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Drug Design
Domain	Model	1-turn	5-turn	20-turn	$\Delta$ (1 $\to$ 5)	$\Delta$ (5 $\to$ 20)
PK/PD	Sonnet 4.5	38.0	96.7	50.0	+58.7	$-46.7$
	Opus 4.6	23.3	50.0	50.0	+26.7	0.0
	Sonnet 4.6	29.0	100.0	50.0	+71.0	$-50.0$
	Gemini 3.1 Pro	48.0	98.3	35.0	+50.3	$-63.3$
	GPT-5.2	38.0	98.3	50.0	+60.3	$-48.3$

Appendix

Table 6: Per-domain success rate (%) across the three de novo evaluation modes for the 10 manifest-defined shared-core domains. Each row is one model–domain combination;

\Delta

columns show absolute improvement.

Domain	Model	De novo	Opt.	$\Delta$	Domain	Model	De novo	Opt.	$\Delta$
PK/PD	Sonnet 4.5	38.0	48.3	+10.3	Docking	Sonnet 4.5	53.3	36.7	$-16.7$
	Opus 4.6	23.3	43.3	+20.0		Opus 4.6	56.7	61.7	+5.0
	Sonnet 4.6	29.0	41.7	+12.7		Sonnet 4.6	46.7	51.7	+5.0
	Gemini 3.1 Pro	48.0	51.7	+3.7		Gemini 3.1 Pro	31.7	61.7	+30.0
	GPT-5.2	38.0	41.7	+3.7		GPT-5.2	—	38.3	—
	Gemini 2.0 Flash	15.0	43.3	+28.3		Gemini 2.0 Flash	10.0	25.0	+15.0

Appendix

Table 7: Per-domain comparison of de novo zero-shot vs. design optimization success rate (%) across the 12 benchmark domains shared by both settings. ADMET is excluded because it is benchmarked only in optimization, and Perturbation is excluded because it has no optimization tasks.

Model	DN	Opt	$\Delta$	DN	Opt	$\Delta$	DN	Opt	$\Delta$	DN	Opt	$\Delta$	DN	Opt	$\Delta$
	Alloy			Controls			FBA			Heat Exchanger			PK/PD
Sonnet 4.5	45.0	35.0	$-10.0$	40.0	60.0	+20.0	95.0	90.0	$-5.0$	100.0	40.0	$-60.0$	50.0	70.0	+20.0
Opus 4.6	50.0	35.0	$-15.0$	45.0	70.0	+25.0	100.0	90.0	$-10.0$	100.0	40.0	$-60.0$	50.0	75.0	+25.0
Sonnet 4.6	45.0	35.0	$-10.0$	50.0	65.0	+15.0	95.0	90.0	$-5.0$	95.0	40.0	$-55.0$	50.0	70.0	+20.0
Gemini 3.1 Pro	40.0	35.0	$-5.0$	45.0	55.0	+10.0	95.0	55.0	$-40.0$	100.0	40.0	$-60.0$	35.0	65.0	+30.0
GPT-5.2	40.0	35.0	$-5.0$	45.0	60.0	+15.0	95.0	85.0	$-10.0$	100.0	40.0	$-60.0$	50.0	60.0	+10.0

Appendix

Table 8: Per-domain success rate (%) for 20-turn de novo agentic vs. 20-turn agentic optimization on the 9 domains shared by both shared-core domain sets.

\Delta

: optimization minus de novo.

Model	DN 1-turn	DN 5-turn	DN 20-turn	Opt. 1-turn	Opt. 20-turn
Sonnet 4.5	$41.9\pm 15.0$	$115\pm 138$	$87.2\pm 95.0$	$39.1\pm 14.7$	$102\pm 250$
Opus 4.6	$52.5\pm 16.9$	$149\pm 209$	$100.0\pm 108$	$55.6\pm 14.3$	$109\pm 254$
Sonnet 4.6	$61.5\pm 171$	$114\pm 89.9$	$85.9\pm 95.3$	$43.9\pm 15.1$	$252\pm 2406$
Gemini 3.1 Pro	$52.5\pm 41.1$	$152\pm 86.0$	$151\pm 126$	$58.5\pm 10.0$	$185\pm 179$
GPT-5.2	$5.46\pm 5.25$	$35.0\pm 181$	$22.8\pm 42.3$	$6.85\pm 11.3$	$24.4\pm 53.8$
Gemini Flash	$7.12\pm 5.76$	$77.5\pm 837$	$24.7\pm 45.0$	$8.52\pm 7.83$	$22.9\pm 29.6$

Appendix

Table 9: Per-task active runtime (seconds) aggregated across the official benchmark tasks. Each cell reports mean

\pm

standard deviation of the logged generation time plus oracle execution time summed over all attempts and turns for that task. External scheduler idle time is excluded.

Figure 6: Average success rate by domain and evaluation setting, averaged across all seven models. ADMET (in optimization) and Perturbation (in de novo design) are among the most accessible domains; domains such as FBA remain challenging and uneven across models; feedback and optimization can unlock domains like Controls and SSA that are weak in zero-shot.

Figure 7: Parse rate vs. success rate for each model–domain combination in zero-shot and design optimization settings. Points cluster in the upper-left quadrant: high parse rates but low success rates, confirming that format compliance does not imply scientific correctness.

Figure 8: Success rate by difficulty level, averaged across all domains and models. Error bars show standard error across domains. Performance degrades from

\sim

45% at L1 to

\sim

12% at L4.

Figure 9: Per-domain change in success rate from de novo zero-shot to design optimization on the 12 domains shared by both settings. Positive values (right) indicate the starting design helps; negative values (left) indicate it hurts. Controls improves consistently, while Reactor and Quantum often degrade.

Figure 10: Manifest-defined shared-core scores across five evaluation settings. The grouped bars compare zero-shot de novo , 5-turn feedback, 20-turn long-horizon, zero-shot design optimization, and 20-turn long-horizon optimization. Leadership shifts across settings, underscoring that static design, feedback use, and optimization are distinct capabilities.

Figure 11: Per-domain success rate heatmap across five evaluation settings and all seven models. Cell values show success rate (%). Green indicates high success; red indicates low. The de novo panels use the 13 official SciDesignBench v1 de novo domains, while the optimization panels use the 13 official optimization domains; ADMET appears only in optimization and Perturbation only in de novo .

Figure 12: Per-domain comparison of de novo zero-shot (blue) vs. design optimization (purple) success rates on the 12 domains shared by both settings. Each subplot shows one domain with all evaluated models. Controls improves consistently, while Reactor and Quantum often degrade under optimization.

Figure 13: Design optimization success rate by difficulty level (L1–L4), averaged across 13 domains. Performance drops from

\sim

40% at L1 to

\sim

10% at L4 for the best models. The pattern mirrors de novo difficulty progression ( Figure 8 ).

Figure 14: Difficulty progression comparison: de novo zero-shot vs. design optimization, averaged across all models and domains. Both settings show similar degradation from L1 to L4, confirming that multi-target constraints are the primary difficulty driver regardless of evaluation mode.

Figure 15: Radar chart showing the top four models’ 20-turn long-horizon performance across the 10 de novo shared-core domains. The distinct shapes highlight complementary strengths across domains rather than a single model dominating every scientific setting.

Figure 16: Per-domain success rate vs. turn number in the 20-turn long-horizon optimization setting. Each line represents one model. Docking and RNA Design converge quickly (5–10 turns); SSA can improve substantially for top models; FBA remains highly model-dependent even with longer trajectories.

Figure 17: Aggregate success rate vs. turn number in the 20-turn long-horizon optimization setting, averaged across the 10 optimization shared-core domains. Compared to the de novo aggregate in Figure 2 , optimization starts from a higher turn-1 success rate but remains highly domain-dependent thereafter.

Figure 18: RLSF training: optimization task performance. (a) ADMET drug design (curriculum GRPO): optimization success improves from 30% (SFT) to 41% (GRPO peak at step 700). (b) PK/PD dosing design: optimization success improves from 32% (SFT) to 47% (GRPO peak at step 600). Points show raw eval; curves are smoothed.

Explore similar work

LEAP: Trajectory-Level Evaluation of LLMs in Iterative Scientific Design
May 14, 2026Marilyn Zhang, Tianfeng Chen, Fabián Barzuna +2LLM Evaluation AI Agent Benchmarks
MolDesignBench: Evaluating LLM-based Agent for Scenario-grounded Molecular Design
Sep 23, 2026Yongjun Jeong, Hanbum Ko, Ye Rin Kim +6Molecular Optimization LLM Agent Evaluation
Department of Artificial Intelligence, Korea University · LG AI Research · Department of Statistics, Korea University
InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis
Apr 14, 2026Oliver Bentham, Vivek SrikumarLLM Evaluation Scientific QA
Kahlert School of Computing University of Utah

May 14, 2026J/K move · Enter open · S save

LEAP: Trajectory-Level Evaluation of LLMs in Iterative Scientific Design

Marilyn Zhang, Tianfeng Chen, Fabián Barzuna, Ankita Rathod+1

LLM Evaluation AI Agent Benchmarks Automated Evaluation Autonomous Experimentation

Open paper AskPDF arXiv

LLMs are increasingly deployed in autonomous laboratories, under the assumption that their domain priors and reasoning over iterative feedback let them converge on good designs in fewer iterations than feedback-only baselines. Current iterative scientific design benchmarks, however, score only outcome snapshots at fixed horizons. This leaves the learning trajectory unmeasured, even though the trajectory is what captures learning efficiency, where each iteration saved is a real saving in cost and time. Motivated by this, we examine three evaluation choices that change the conclusions one draws about LLM learning efficiency in iterative scientific design: what to measure, what baseline to compare against, and what to ground against. We introduce LEAPBench, Learning Efficiency in Adaptive Processes, a 55-task framework that pairs a best-so-far area under the curve (AUC) trajectory metric with a classical Bayesian-optimization reference and an audit grounded in published literature. Applied to eight contemporary LLMs, switching from final-outcome to trajectory scoring changes the best-model decision on 53% of tasks at matched horizons, and exposes efficiency gains overlooked by outcome-based scoring. LLMs do not outperform a classical Bayesian baseline. On 16 biology tasks where the oracle's reward signal is aligned with configurations from the published-best design, domain-aware prompting leads to LLM choices that match the published-best's approximately 10 percentage points less often than domain-agnostic prompting at iteration 30. The pattern is sharpest on 6 tasks where the literature-typical and published-best configurations diverge, and domain-agnostic prompting matches the published-best more often on all 6. The trajectory metric also doubles as a tractable training target. Offline reinforcement learning with the metric as a reward improves performance on 14 of 21 held-out tasks.

Abstract

Explore similar work

LEAP: Trajectory-Level Evaluation of LLMs in Iterative Scientific Design

MolDesignBench: Evaluating LLM-based Agent for Scenario-grounded Molecular Design

InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis