LabOPBench: A Multimodal Benchmark for Evidence-Grounded Laboratory Decision Making

Jingyuan Li; Yujie Liu; Yijie Dai; Xiaoyu Yuan; Nian Ran; Huajie Niu; Ming Zhu; Shuwen Zhao

cs.AIOct 14, 2026

LabOPBench: A Multimodal Benchmark for Evidence-Grounded Laboratory Decision Making

Authors: Jingyuan Li, Yujie Liu, Yijie Dai, Xiaoyu Yuan, Nian Ran, Huajie Niu, Ming Zhu, Shuwen Zhao,

Organizations: Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Fudan University · XtalPi ·

Abstract

Laboratory experiments involve more than following predefined protocols; they require adaptive decision-making as the experimental state changes. Existing scientific benchmarks rarely assess such evidence-based decisions in laboratory settings. Here we present LabOPBench, a multimodal benchmark of 756 expert-authored, independently reviewed questions in organic synthesis and materials science. Each item pairs laboratory evidence with procedural context and an open-ended judgment, evaluated using a question-specific scoring rubric. Eight multimodal models obtain overall normalized scores of 28.8-45.8%, leaving substantial room for improvement. On a controlled 100-question subset, the strongest models outperform undergraduate participants on average but remain below the average performance of PhD participants. Expert analysis of responses from the three leading models identifies Evidence Misreading and Evidence-Decision Conflict as the dominant error patterns. These results reveal that current models still struggle to turn multimodal perception into reliable laboratory decision-making.

Figures & tables

Benchmark	Data source	Format	Focus
LAB-Bench Laurent et al. (2024)	Literature / protocols	MCQ	Biological research
MMMU-Pro Yue et al. (2025)	Academic exams	MCQ	Visual reasoning
MatCha Lai et al. (2025)	Published figures	MCQ	Materials characterization
MaCBench Alampara et al. (2025)	Figures / lab photos	MCQ + Numeric	Data & experiment analysis
LabSafety Bench Zhou et al. (2026)	Safety resources	MCQ + Open	Hazards & consequences
ChemReason-Bench Zhang et al. (2026)	Synthesis protocols	Mixed	Procedural reasoning

Table 1: Comparison of LabOPBench with related benchmarks by data source, format, and focus.

Figure 1: Composition of LabOPBench across laboratory task categories. a , Distribution of the 756 benchmark questions across the five major categories. b-e , Distribution of organic-synthesis questions across the subcategories of preparation, postprocessing, monitoring and characterization, respectively.

Figure 2: Workflow for constructing and validating a LabOPBench question. Experts draft and review each evidence-grounded item, while an LLM is used only for language standardization. Disagreements are adjudicated before the final question, multimodal evidence, reference options, and scoring anchors are compiled.

Model	Preparation	Monitoring	Postprocessing	Characterization	Materials Science	Overall
Intern-S1-Pro	24.2	40.0	25.6	29.6	41.7	28.8
Gemini 3.1 Flash-Lite	41.1	41.4	33.1	37.8	47.6	36.8
Grok 4.3	35.8	49.4	33.5	37.4	53.7	37.5
MiniMax-M3	47.3	54.8	33.3	41.2	55.6	39.6
Qwen3.6-Plus	37.3	54.8	37.2	44.8	52.8	41.2
Gemini 3.1 Pro	38.0	48.1	37.5	48.0	53.5	41.9

Table 2: Normalized scores (%) of eight multimodal models across the major LabOPBench categories. Bold and underlined values denote the highest and second-highest scores in each column, respectively.

Figure 3: Fine-grained performance of eight multimodal models across 16 organic-synthesis subcategories. Cells show the mean normalized scores (%) from two independent judges, with darker colors indicating higher scores. Bold values indicate the highest score in each subcategory, including ties.

Figure 4: Differences between automated and expert scoring across major laboratory categories. Bars show differences in category-level mean normalized scores (automated minus expert) for a stratified sample of Claude Opus 4.7 responses. Positive values indicate higher automated scores, whereas negative values indicate higher expert scores.

Figure 5: Performance on the controlled 100-question organic-synthesis subset. Bars show normalized scores for eight multimodal models and mean normalized scores for undergraduate and PhD participants.

Figure 6: Image-mismatch ablation on the 100-question organic-synthesis subset. Connected points show normalized scores with matched and mismatched images. Values in parentheses indicate the score difference (matched minus mismatched) in percentage points.

Case	Expert judgment	Model response	Failure mechanism
TLC	Spotting was standard; the developed pattern indicated unsuitable developing-solvent polarity.	Judged spotting as non-standard and attributed the pattern to excessive sample loading.	Evidence Misreading. Misinterpreted spotting evidence and selected an unsupported cause.
LC-MS sampling	Collect a representative solid-liquid aliquot while stirring, then dissolve in methanol.	Proposed centrifugation or filtration, then analysis of only the clear supernatant.	Evidence-Decision Conflict. Excluded the solid fraction required for representative sampling.

Table 3: Representative failures in model answers. The cases illustrate misinterpretation of experimental evidence and an operation inconsistent with the sampling objective.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

categories	Subcategories
Experiment Preparation	Experiment_plan; Experiment_Safety
Experiment Monitor	Experiment_Monitor; Experiment_notes
Experiment Postprocess	Extraction; Evaporation; Column; Recrystallization; Trituration; TLC; Lyophilization
Experiment Characterization	NMR; LCMS; Meltingpoint; Crystal; GCMS

Appendix

Table 4: Organic-synthesis categories and subcategories.

Anchor	Score	Definition
Score 3 anchor	3	Complete, scientifically correct, consistent with the supplied evidence, and covers the core judgment, causal explanation, and necessary operational recommendations required by the question.
Score 2 anchor	2	The core judgment is correct, but key evidence, verification steps, causal explanation, or corrective measures are omitted; no substantive error that would change the conclusion should be present.
Score 1 anchor	1	Identifies some relevant observations or proposes a directionally reasonable recommendation, but does not complete the core judgment, lacks the necessary evidence chain, or uses an approach insufficient to support the conclusion.
Score 0 anchor	0	The core judgment is incorrect, contradicts the supplied evidence, proposes clearly inappropriate or unsafe operations, or is irrelevant to the question.

Appendix

Table 5: Four-level scoring anchors for benchmark questions.

Position	Contributors
PhD researcher	4
Postdoctoral researcher	6
Laboratory specialist	8
Associate researcher or above	4
Total	22

Appendix

Table 6: Contributor composition by position.

Laboratory experience	Contributors	Proportion
3–5 years	2	9.1%
5–7 years	8	36.4%
7–10 years	4	18.2%
$>10$ years	8	36.4%
Total	22	100.0%

Appendix

Table 7: Contributor laboratory experience.

Scoring source	Preparation	Monitoring	Postprocess	Characterization	Material Science
GPT-5.5 judge	48.72%	63.89%	40.20%	47.62%	59.26%
Gemini 3.1 Pro judge	50.00%	63.89%	40.39%	48.21%	60.19%
Automated mean	49.36%	63.89%	40.29%	47.92%	59.72%
Expert rater 1	48.72%	61.11%	40.00%	48.21%	58.33%
Expert rater 2	47.44%	63.89%	40.20%	48.81%	59.26%
Expert mean	48.08%	62.50%	40.10%	48.51%	58.80%

Appendix

Table 8: Automated and expert scores by major category.

Model	Judge scores
Claude Opus 4.7	58.2% / 61.4%
Gemini 3.1 Pro	51.1% / 55.8%
Gemini 3.1 Flash-Lite	46.0% / 49.1%
GPT-5.5	52.4% / 53.3%
Grok 4.3	52.0% / 55.3%
Qwen3.6-Plus	50.2% / 55.3%

Appendix

Table 9: Model performance for materials-science tasks.

Model	Experiment_ plan ( $n=46$ )	Experiment_ Safety ( $n=21$ )
Claude Opus 4.7	54.7% / 54.7%	33.3% / 40.5%
Gemini 3.1 Pro	41.7% / 43.1%	29.4% / 27.0%
Gemini 3.1 Flash-Lite	44.9% / 47.5%	31.0% / 28.6%
GPT-5.5	50.4% / 49.6%	42.9% / 50.8%
Grok 4.3	37.3% / 39.1%	27.8% / 33.3%
Qwen3.6-Plus	40.2% / 41.7%	26.2% / 32.5%

Appendix

Table 10: Model performance for experiment-preparation subcategories.

Model	Experiment_ Monitor ( $n=16$ )	Experiment_ notes ( $n=14$ )
Claude Opus 4.7	44.8% / 47.9%	85.7% / 86.9%
Gemini 3.1 Pro	32.3% / 35.4%	64.3% / 64.3%
Gemini 3.1 Flash-Lite	31.3% / 31.3%	52.4% / 53.6%
GPT-5.5	43.8% / 41.7%	61.9% / 65.5%
Grok 4.3	33.3% / 41.7%	61.9% / 64.3%
Qwen3.6-Plus	36.5% / 36.5%	76.2% / 75.0%

Appendix

Table 11: Model performance for experiment-monitoring subcategories.

Model	Extraction ( $n=38$ )	Evaporation ( $n=46$ )	Column ( $n=75$ )	Recrystallization ( $n=16$ )
Claude Opus 4.7	48.2% / 49.6%	48.9% / 51.4%	44.9% / 46.2%	32.3% / 33.3%
Gemini 3.1 Pro	46.5% / 44.3%	39.1% / 40.6%	39.1% / 43.1%	31.3% / 36.5%
Gemini 3.1 Flash-Lite	43.0% / 42.1%	37.0% / 37.0%	29.6% / 32.7%	28.1% / 35.4%
GPT-5.5	43.4% / 46.5%	50.7% / 50.4%	46.9% / 46.7%	34.4% / 38.5%
Grok 4.3	46.1% / 44.7%	36.2% / 36.6%	40.4% / 41.3%	33.3% / 36.5%
Qwen3.6-Plus	46.5% / 46.9%	40.2% / 40.2%	36.7% / 42.2%	22.9% / 30.2%

Appendix

Table 12: Model performance for postprocessing subcategories.

Model	NMR ( $n=42$ )	LCMS ( $n=48$ )	Meltingpoint ( $n=8$ )	Crystal ( $n=29$ )	GCMS ( $n=15$ )
Claude Opus 4.7	41.3% / 41.7%	57.6% / 59.0%	8.3% / 6.3%	55.7% / 55.7%	34.4% / 33.3%
Gemini 3.1 Pro	47.2% / 48.8%	54.2% / 56.6%	18.8% / 25.0%	49.4% / 54.0%	31.1% / 31.1%
Gemini 3.1 Flash-Lite	38.1% / 42.1%	36.8% / 37.2%	16.7% / 14.6%	40.8% / 45.4%	31.1% / 40.0%
GPT-5.5	44.4% / 48.0%	54.2% / 53.8%	16.7% / 16.7%	47.1% / 50.6%	38.9% / 42.2%
Grok 4.3	31.0% / 31.3%	45.5% / 50.0%	12.5% / 14.6%	42.0% / 40.8%	24.4% / 28.9%
Qwen3.6-Plus	42.1% / 44.0%	55.6% / 54.9%	22.9% / 20.8%	36.8% / 37.9%	42.2% / 43.3%

Appendix

Table 13: Model performance for characterization subcategories.

Major category	Full benchmark $N$	Subset $n$	Fine-grained allocation Full $N\rightarrow$ subset $n$
Preparation	67	10	Experiment_plan: 46 $\rightarrow$ 7; Experiment_Safety: 21 $\rightarrow$ 3
Monitoring	30	5	Experiment_Monitor: 16 $\rightarrow$ 3; Experiment_notes: 14 $\rightarrow$ 2
Postprocessing	425	64	Extraction: 38 $\rightarrow$ 6; Evaporation: 46 $\rightarrow$ 7; Column: 75 $\rightarrow$ 11; TLC: 201 $\rightarrow$ 30; Recrystallization: 16 $\rightarrow$ 2; Trituration: 25 $\rightarrow$ 4; Lyophilization: 24 $\rightarrow$ 4
Characterization	142	21	NMR: 42 $\rightarrow$ 6; LCMS: 48 $\rightarrow$ 7; Meltingpoint: 8 $\rightarrow$ 1; Crystal: 29 $\rightarrow$ 5 ; GCMS: 15 $\rightarrow$ 2
Total	664	100	16 subcategories

Appendix

Table 14: Composition of the controlled 100-question subset.

Participant ID	Group	GPT-5.5 judge	Gemini 3.1 Pro judge	Mean score	Sample SD
U01	Undergraduate	26.7%	28.2%	27.4%	—
U02	Undergraduate	29.0%	31.3%	30.2%	—
U03	Undergraduate	33.8%	35.0%	34.4%	—
U04	Undergraduate	29.8%	30.8%	30.3%	—
U05	Undergraduate	32.0%	33.3%	32.7%	—
Group mean	Undergraduate, n = 5	30.3%	31.7%	31.0%	2.7%

Appendix

Table 15: Individual human participant scores.

Model	Matched scores	Matched mean	Mismatched scores	Mismatched mean	Score drop
Claude Opus 4.7	43.2% / 44.2%	43.7%	25.0% / 20.3%	22.7%	21.0 pp
Gemini 3.1 Pro	37.8% / 40.2%	39.0%	17.7% / 16.7%	17.2%	21.8 pp
Gemini 3.1 Flash-Lite	34.7% / 36.0%	35.4%	22.5% / 18.7%	20.6%	14.8 pp
GPT-5.5	41.7% / 45.2%	43.5%	21.0% / 18.0%	19.5%	24.0 pp
Grok 4.3	35.5% / 33.5%	34.5%	22.5% / 20.7%	21.6%	12.9 pp
Qwen3.6-Plus	38.2% / 39.7%	39.0%	20.5% / 18.3%	19.4%	19.6 pp

Appendix

Table 16: The cross-model score drop is the difference between the separately rounded condition means, rather than the mean of model-wise reductions.

Figure 7: Model-balanced shares of assigned error labels across Claude Opus 4.7, GPT-5.5, and Gemini 3.1 Pro on the controlled organic-synthesis subset. Percentages sum to 100%.

Error type	Definition	GPT- 5.5	Gemini 3.1 Pro	Claude Opus 4.7
Evidence Misreading	Misreading or omission of decision-relevant experimental evidence.	38	31	34
Evidence–Decision Conflict	A diagnosis or action inconsistent with the experimental evidence or rubric.	33	29	30
Chemical Signal Errors	Incorrect interpretation of chemical signals, peaks, species, or quantitative results.	6	5	9
Common Rule Misuse	Applying generic laboratory rules in place of case-specific evidence.	9	6	4
Mistaking Normal for Abnormal	Treating a normal or transient state as abnormal and recommending unnecessary intervention.	3	5	4
Experimental Method Errors	Independent errors in experimental steps, conditions, methods, or execution.	3	5	3

Appendix

Table 17: Definitions and raw counts of expert-assigned error labels across the three models.

Evidence-Grounded Reasoning

Evidence-grounded reasoning studies how AI systems connect conclusions to traceable, relevant evidence and recognize when support is incomplete.

See all 90 papers

onepot-Bench 0: towards lab-aware in silico chemistry benchmarks
Aug 3, 2026Brandon Wang, Andrei S. Tyrin, Daniil A. BoikoLLM Safety Benchmarks LLM Evaluation
Onepot AI, Inc.
BenchBench-Protocol: Evaluating Real-World Wet-Lab Protocol Reasoning and Modification
Aug 24, 2026Aditya Sivakumar, Ashu Singhal, Nicholas Larus-Stone +1LLM Evaluation Benchmark Design
Benchling
Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discovery
Aug 7, 2026Taolin Han, Yuchen Zhang, Jinghang Wang +22Scientific Reasoning Scientific Discovery
1Alibaba Group · University of Chinese Academy of Sciences · Taolin Han1,3* +28

Aug 3, 2026J/K move · Enter open · S save

onepot-Bench 0: towards lab-aware in silico chemistry benchmarks

Brandon Wang, Andrei S. Tyrin, Daniil A. Boiko

Onepot AI, Inc.

LLM Safety Benchmarks LLM Evaluation Benchmark Design

Open paper AskPDF arXiv

Language models are playing an increasingly important role in laboratory science, performing tasks such as experiment planning, execution, and post-hoc analysis. However, precisely measuring their abilities is difficult, as scientific capabilities require a mixture of both problem-solving skills and domain-specific intuition. Existing evaluations rarely measure the capabilities required to make reliable decisions in a physical laboratory and often rely on public data that may have appeared in model training corpora. We introduce onepot-Bench 0, a proprietary benchmark suite for evaluating language models on synthetic chemistry capabilities relevant to wet-lab execution. onepot-Bench 0 comprises three complementary evaluations: ChemAbacus measures tool-free cheminformatics literacy and numerical reasoning; SynthRefusal characterizes safety and refusal behavior across a variety of benign, controlled, and designer-drug targets; and SynthBench evaluates reaction-outcome prediction and catalyst selection using private experimental data generated in our laboratory. Together, these evaluations probe basic competency, reliability, and deeper knowledge, all skills which are required for reliable performance in the lab.

Abstract

Evidence-Grounded Reasoning

Explore similar work

onepot-Bench 0: towards lab-aware in silico chemistry benchmarks

BenchBench-Protocol: Evaluating Real-World Wet-Lab Protocol Reasoning and Modification

Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discovery