RxnOptBench: Benchmarking LLMs for Reaction-Condition Optimization in Organic Methodology

Lingli Ge; Yubin Wang; Junyuan Gao; Jiahe Song; Jiaxing Sun; Boyu Zhu; Haote Yang; Jingchao Wang

physics.chem-phSep 29, 2026

RxnOptBench: Benchmarking LLMs for Reaction-Condition Optimization in Organic Methodology

Authors: Lingli Ge, Yubin Wang, Junyuan Gao, Jiahe Song, Jiaxing Sun, Boyu Zhu, Haote Yang, Jingchao Wang, +4 more

Organizations: Shanghai Jiao Tong University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China

Abstract

Chemical reaction-condition optimization -- choosing the catalyst, ligand, solvent, reagent, temperature, time, and atmosphere that jointly maximize yield and stereoselectivity -- is a central, judgement-laden subtask of organic methodology research that large language models are increasingly expected to support. Yet existing chemistry benchmarks evaluate reaction-class labelling, retrosynthesis, or SMILES manipulation, and do not ask models to read a real condition-screening table and pick the best set. We introduce RxnOptBench, a benchmark whose every option and precedent is a real wet-lab entry mined from the optimization tables of organic-methodology papers published in 2025, graded by a continuous relative score derived from a declared headline utility that combines reported yield with enantiomeric excess (ee), diastereomeric ratio (dr), and regioisomeric ratio (rr), and equipped with a paired precedents-vs-no-precedents design that isolates in-context use of literature evidence from parametric memorization. Across nine frontier LLMs and three Chemistry LLMs, even the best models leave substantial headroom: chemistry-specialized models fall to the random-baseline floor on multi-axis selection, while open-weight models have closed most of the gap to proprietary frontier models. We release the final human-reviewed benchmark test set and evaluation code.

Figures & tables

Figure 1: RxnOptBench overview. (A) Question Construction: each condition-screening table yields MC/FIB items in matched precedents (prior rows shown, top withheld) and no-precedents (no prior rows) modes. (B) Scoring Rules: every observed row carries a precomputed relative score fusing yield with ee/dr/rr. MC reports Accuracy and Score; FIB separately reports observed-row Hit, Strict Accuracy, and Score conditioned on Hit.

Figure 2: Headline accuracy on the no-precedents subsets. Per-model Accuracy on MC_single ( $n=1{,}039$ ), MC_all ( $n=552$ ), and FIB ( $n=552$ ); sectors group models by tier. FIB uses Strict Accuracy over submitted predictions. Numbers in Table 2 ; precedents comparisons use the remaining matched subsets.

Figure 3: Four-stage semi-automated extraction pipeline. PDF pages are screened, parsed, converted into row-level reaction records, and manually audited before question construction. The GTR-VL label in Stage 3 denotes the MinerU.Chem recognizer; details in § 3.2 .

Type	Sub-type	#Q	Description
MC	single_var	1,039	One target axis; all other varying axes locked.
MC	all_var	552	Choose one observed joint-condition row.
MC	single_var_precs	850	single_var with precedents.
MC	all_var_precs	310	all_var with precedents.
MC	single_var_precs_paired	850	Twin of single_var_precs without precedents.
MC	all_var_precs_paired	310	Twin of all_var_precs without precedents.

Table 1: The eight sub-types of RxnOptBench.

	Model	MC_single		MC_all		FIB
	Model	Acc.	Score	Acc.	Score	Hit	Strict	Score $\|$ Hit
Baseline	Uniform random	36.5%	0.669	17.8%	0.540	28.1%	5.9%	0.540
Chemistry LLMs	ChemLLM	36.1%	0.666	20.7%	0.545	25.2%	6.6%	0.554
	ChemDFM	36.7%	0.670	20.5%	0.547	33.5%	7.6%	0.557
	ChemVLM	41.4%	0.712	23.6%	0.602	27.7%	6.7%	0.610
Open Source LLMs	Intern-S1-Pro	43.4%	0.734	27.9%	0.644	41.7%	9.6%	0.576

Table 2: No-precedents results ( MC_single : 1,039; MC_all /FIB: 552 each). FIB Strict uses all submissions; Score is conditional on Hit. Cells are heat-mapped per column; models are ordered weakest

\to

strongest by mean Score within tier.

Figure 4: Precedents vs. no-precedents on MC. Solid = with precedents, hatched = no-precedents; bars annotated with $\Delta_{\text{precs}}$ . Top: MC_single ; bottom: MC_all .

Figure 5: Precedents vs. no-precedents on FIB. Solid = with precedents, hatched = no-precedents; bars annotated with $\Delta_{\text{precs}}$ . Observed-row Hit, Strict Accuracy (best observed row over all submitted predictions), and Score $\,\mid\,$ Hit are reported separately.

Model	rea	sol	cat	T	t	hv	atm	lig	ov
$n$	227	201	145	104	95	73	48	40	933
ChemLLM	33.5	30.8	29.7	41.3	40.0	35.6	47.9	35.0	34.8
ChemDFM	34.8	32.8	29.0	40.4	42.1	37.0	50.0	32.5	35.7
ChemVLM	43.6	34.8	37.2	41.3	45.3	50.7	56.2	37.5	41.6
Kimi-K2.6	40.1	42.3	34.5	44.2	42.1	43.8	43.8	27.5	40.3
Qwen3.5-397B-A17B	45.8	35.8	33.8	41.3	44.2	57.5	45.8	45.0	42.0

Table 3: Per-condition-key Accuracy (%) on no-precedents subsets, with 12 models grouped by tier ( Chemistry LLMs / Open Source LLMs / Proprietary LLMs ) and sorted ascending by overall within each tier (best at bottom). Column abbreviations: rea=reagents, sol=solvents, cat=catalysts, T=temperature, t=time, hv=light, atm=atmosphere, lig=ligand, ov=overall. Bold = best in column across all 12 models.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Benchmark composition across reaction and product dimensions . A , mutually exclusive primary transformation classes over 552 parent screening tables. B–C , multi-label energy inputs and catalyst classes; labels report table count and share of all parent tables. D , family sizes over 232 non-empty Bemis–Murcko scaffolds from 341 unique canonical target products.

Figure 7: Condition-space structure (

N=552

). A , fraction of parent tables in which each major condition family is varied/queried or held constant. B , candidate-set arity for varied fields (boxes show quartiles; whiskers show the non-outlier range). C , number of varied condition axes per parent table. D , number of experimentally observed condition combinations per source table.

Figure 8: Per-model predictions on four illustrative cases. Each card shows the target reaction, options (gold $\bigstar$ ), and the 12 models’ picks (green $=$ correct, red $=$ wrong). M1/M2: MC_single items; M3: paired MC_single / MC_all on the same parent reaction; M4: FIB without vs. with a sibling precedent.

Explore similar work

oMeBench: Towards Robust Benchmarking of LLMs in Organic Mechanism Elucidation and Reasoning
Oct 9, 2025Ruiling Xu, Yifan ZhangLLM Evaluation Benchmark Design
University of Illinois Urbana-Champaign
onepot-Bench 0: towards lab-aware in silico chemistry benchmarks
Aug 3, 2026Brandon Wang, Andrei S. Tyrin, Daniil A. BoikoLLM Safety Benchmarks LLM Evaluation
Onepot AI, Inc.
From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models
Jun 2, 2026Hongyu Guo, Hao Li, He Cao +2LLM Evaluation LLM Reasoning
Peking University, Shenzhen Graduate School · International Digital Economy Academy (IDEA)

Oct 9, 2025J/K move · Enter open · S save

oMeBench: Towards Robust Benchmarking of LLMs in Organic Mechanism Elucidation and Reasoning

Ruiling Xu, Yifan Zhang

University of Illinois Urbana-Champaign

LLM Evaluation Benchmark Design LLM Reasoning

Open paper AskPDF arXiv

Organic reaction mechanisms describe the step-wise elementary processes by which reactants transform into intermediates and products, and are fundamental to understanding chemical reactivity and guiding molecular and reaction de-sign. While large language models (LLMs) have shown promise on chemical tasks such as synthesis design, it remains unclear to what extent this reflects genuine chemical reasoning capabilities: the ability to generate chemically valid intermediates, maintain consistency across reaction steps, and follow logically coherent multi-step pathways. To investigate this, we introduce oMeBench, the first large-scale, expert-curated benchmark for organic mechanism reasoning, comprising over 10,000 annotated mechanistic steps with reaction type labels, intermediate structures, and difficulty ratings. To enable fine-grained evaluation, we further propose oMeS, a dynamic scoring framework that jointly assesses step-level logical consistency and chemical structural similarity. Systematic evaluation of state-of-the-art LLMs reveals that while current models exhibit promising chemical intuition, they struggle to produce correct and consistent reasoning across multi-step mechanisms. Notably, combining prompting strategies with fine-tuning enables smaller-scale models to achieve performance comparable to closed-source frontier models. We hope oMeBench will serve as a rigorous foundation for advancing AI systems toward genuine chemical reasoning.

Abstract

Explore similar work

oMeBench: Towards Robust Benchmarking of LLMs in Organic Mechanism Elucidation and Reasoning

onepot-Bench 0: towards lab-aware in silico chemistry benchmarks

From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models