BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories

Fei Lin; Tengchao Zhang; Ziyang Gong; Xiaotong Yu; Bohan Zhang; Yifan Zhou; Qihao Yang; Ji Dai

cs.AINov 4, 2026

BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories

Authors: Fei Lin, Tengchao Zhang, Ziyang Gong, Xiaotong Yu, Bohan Zhang, Yifan Zhou, Qihao Yang, Ji Dai,

Organizations: HydroSophy · Macau University of Science and Technology · Shanghai Jiao Tong University · Institute of Process Engineering, Chinese Academy of Sciences ·

Abstract

Self-Driving Chemical Laboratories (SDCLs) are moving chemical experimentation toward embodied automation, where machines must perceive laboratory scenes, reason about experimental states, and act under physical, procedural, and safety constraints. Although Multimodal Large Language Models (MLLMs) are increasingly considered as embodied brains for such systems, existing laboratory benchmarks mainly focus on safety, anomaly detection, or general scientific understanding, leaving their execution-oriented embodied capabilities underexplored. To fill this gap, we introduce BEAKER, a Benchmark for evaluating Embodied Actionability, Knowledge, Experimentation, and Reasoning in SDCLs. BEAKER contains 1,000 expert-curated question-answer samples across 11 image and video task types, covering key judgments required for chemical laboratory execution, including affordance grounding, trajectory planning, equipment, state, and safety recognition, apparatus setup, operation and step reasoning, and temporal procedure reasoning. All questions and annotations are manually created and reviewed by chemistry and embodied Artificial Intelligence (AI) experts, without AI-assisted generation or annotation. We evaluate 28 general-purpose, science-oriented, chemistry-specialized, and embodied MLLMs in an offline, single-turn setting. Results show that current models can follow coarse experimental procedures in videos, but still struggle to ground actions into operable regions and feasible trajectories, to recognize fine-grained laboratory apparatus, states, and safety norms, and to order steps in mid-length procedures. BEAKER thus reveals a clear gap between coarse procedural understanding and the precise, action-grounded perception required for laboratory execution, providing a diagnostic benchmark for future MLLMs in SDCLs.

Figures & tables

Figure 1: Overview of BEAKER and key evaluation findings. The radar chart compares four representative models across 11 tasks grouped by capability. All axes use a 0–100 scale except AFF, which uses a 0–50 scale due to uniformly low scores across all models. AFF: Affordance Grounding; TRAJ: Trajectory Planning; ERI: Equipment and Reagent Identification; ESS: Experimental Safety and Standardization; SR: State Recognition; EAS: Experimental Apparatus Setup; EOP: Experimental Operations; SVOR: Short-Video Operation Reasoning; MVTS: Mid-Video Task-Step Reasoning; TSR: Temporal Sequence Reasoning; LVPR: Long-Video Experimental Procedure Reasoning; GT: Ground Truth.

Benchmark	Scale	SDCL	Real	Image	Video	Action.	Know.	Exper.	Reason.	Human
MaCBench [ 29 ]	1.1k+ QA	$\triangle$	✓	✓	✗	✗	✓	$\triangle$	✓	✓
ExpVid [ 30 ]	7.8k QA	$\triangle$	✓	✗	✓	✗	✓	✓	✓	$\triangle$
LabUtopia [ 64 ]	30 tasks	$\triangle$	✗	✓	✗	$\triangle$	✓	✓	✓	✓
AutoBio [ 65 ]	16 tasks	✗	✗	✓	✗	$\triangle$	✓	✓	$\triangle$	✓
Chemistry3D [ 66 ]	Sim. env.	$\triangle$	✗	✓	✗	$\triangle$	✓	✓	$\triangle$	✓
SDCL Visual Anomaly Dataset [ 26 ]	2.8k pairs	✓	✓	✓	✗	✗	$\triangle$	✓	✓	✓

Table 1: Comparison with related scientific and laboratory benchmarks. Scale denotes the primary evaluation unit of each work: QA for question-answering samples, tasks for embodied or robotic tasks, pairs for image-text pairs, imgs for images, and Sim. env. for simulation environments. ✓ , ✗ , and

\triangle

denote full, no, and partial coverage, respectively. Human indicates whether all questions and annotations are authored by human experts without LLM/AI-assisted generation or annotation.

Figure 2: Overview of BEAKER construction and task examples. Top: the four-stage expert-curated pipeline of data collection, expert curation, QA authoring, and quality control. Bottom: examples of the four capability dimensions with the input, question, Ground Truth (GT), and GPT-5.4 [ 72 ] prediction.

Figure 3: Statistics of BEAKER. Counts by input type, capability, task, data source, and modality.

Chemistry-specialized Multimodal Models
Model	Actionability		Knowledge			Experimentation				Reasoning		Overall
Model	AFF $\uparrow$	TRAJ $\uparrow$	ERI $\uparrow$	ESS $\uparrow$	SR $\uparrow$	EAS $\uparrow$	EOP $\uparrow$	SVOR $\uparrow$	MVTS $\uparrow$	TSR $\uparrow$	LVPR $\uparrow$	Overall
Human	88.0	91.0	96.7	95.0	93.3	93.3	96.7	94.4	92.0	95.0	96.0	93.3
Open-source Models
ChemVLM-8B [ 40 ]	2.1	21.0	33.3	40.0	21.7	36.7	33.3	36.0	40.0	37.9	24.0	28.4
ChemVLM-26B [ 40 ]	3.1	27.0	23.3	25.0	31.7	36.7	33.3	37.6	42.7	42.4	28.0	30.1

Table 2: Consolidated results on BEAKER. Task scores are grouped by the four capability dimensions, and Overall aggregates them as described in Section 4.1 . Model rankings within each column are color-coded: 1st , 2nd , and 3rd ; tied scores share the same rank.

Model	Actionability			Knowledge				Experimentation					Reasoning			Overall
Model	AFF	TRAJ	Avg.	ERI	ESS	SR	Avg.	EAS	EOP	SVOR	MVTS	Avg.	TSR	LVPR	Avg.	Overall
ChemVLM-8B [ 40 ]	-2.00 (13/16)	-5.78 (42/108)	-3.89	-11.67 (10/17)	-1.67 (12/13)	-35.00 (3/24)	-16.11	-8.33 (11/16)	-26.67 (6/22)	-39.20 (8/57)	-26.67 (9/29)	-25.22	-16.25 (16/44)	-26.17 (5/30)	-21.21	-18.67
Intern-S1-Mini [ 38 ]	-2.33 (7/17)	-4.63 (50/100)	-3.48	-1.67 (13/14)	-3.33 (9/11)	-13.33 (5/13)	-6.11	-15.00 (10/19)	-18.33 (11/22)	-28.80 (7/43)	-10.67 (5/13)	-18.20	-30.16 (15/57)	+2.50 (16/14)	-13.83	-11.84
RoboBrain2.5-8B [ 49 ]	-1.81 (8/11)	+4.08 (80/70)	+1.14	-3.33 (4/6)	-6.67 (5/9)	-8.33 (3/8)	-6.11	-1.67 (8/9)	-1.67 (6/7)	-0.80 (6/7)	-5.33 (5/9)	-2.37	+1.11 (14/13)	-2.67 (5/7)	-0.78	-1.76

Table 3: Specialization gains and losses relative to Qwen3-VL-8B [ 92 ] . Task cells report score differences with paired wins/losses below; ties are omitted. Avg. reports the difference in unweighted capability means (and thus differs from the 2:5-weighted Experimentation score in Table 12 ). Positive and negative differences are shown in green and red, respectively; shaded bold cells summarize capability and Overall differences.

Panel of Table 4 — Table 4: Capability profile of representative MLLMs on BEAKER. Avg. is averaged over the 13 models shown (all 28 in Table 12 ). Shading scales with the score; the best result in each row is in bold.

Panel of Table 5 — Table 5: Performance across question-format task groups. We compare representative models across three BEAKER QA task groups: Coordinate (Coord.) QA, MCQ, and Fill-in-the-Blank (FIB) QA. MCQ scores are computed as the unweighted macro-average across the eight MCQ task types. The last three columns report pairwise performance ratios between task groups, reflecting relative performance rather than format-specific robustness. Score cells are shaded by value, and ratio cells by their distance from 1.0.

Figure 4: Qualitative examples for three representative task types: Affordance Grounding (AFF) (Actionability, image), Equipment and Reagent Identification (ERI) (Knowledge, image MCQ), and Short-Video Operation Reasoning (SVOR) (Experimentation, video MCQ). Left panels show correct GPT-5.4 [ 72 ] predictions; right panels show incorrect ones. GT: Ground Truth.

Figure 5: Self-explanation examples of GPT-5.4 [ 72 ] on BEAKER. Representative correct and incorrect cases in Long-Video Experimental Procedure Reasoning (LVPR) show that GPT-5.4 can extract procedural evidence from experimental videos but may struggle to match visual observations with fine-grained chemical concepts. GT: Ground Truth.

Panel of Figure 6 — Figure 6: Visual input ablation on image MCQs. Accuracy is averaged over five tasks with 60 questions each. Image + text includes images, questions, and options; Text only removes the images. Visual input improves average accuracy for all eight models. The dashed line marks the random-choice baseline (Appendix K ). Gain denotes the increase in percentage points (pp).

Figure 7: Input modality ablation on video-based tasks. Performance of GPT-5.4, Claude Opus 4.7, and Qwen3.5-397B-A17B under video-plus-question, caption-plus-question, and caption-plus-video-plus-question settings. M-TSR: Mid-Video Temporal Sequence Reasoning; MVTS: Mid-Video Task-Step Reasoning; L-TSR: Long-Video Temporal Sequence Reasoning; LVPR: Long-Video Experimental Procedure Reasoning.

Panel of Table 6 — Table 6: Frame sampling ablation on video-based BEAKER tasks. We evaluate Qwen3.5-397B-A17B [ 93 ] under different frame sampling settings for short-, mid-, and long-video tasks. Frames are listed as short / mid / long; long-video frames are arranged into four-frame image grids (16, 32, and 48 grids). The best result in each column is in bold.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Content	#Raw
Image sources (12)
HeinSight 2.0 & 3.0 [ 95 , 96 ]	Vessel detection, liquid/solid phase classification	5,131
Roboflow [ 97 ]	Safety labels, apparatus, spills, operations (10 sets)	3,178
ChemEq25 [ 98 ]	25-class real-world apparatus across multiple labs	1,138
Chemical Apparatus [ 99 ]	Apparatus in complex backgrounds with 6 categories	629
CLAD [ 100 ]	21-class apparatus recognition and detection	488

Appendix

Table 7: Data sources for BEAKER. Roboflow encompasses 10 community-contributed datasets.

Panel of Table 8 — Table 8: License and usage information for external assets used in BEAKER construction. Badge colors mark permissive , use-restricted , and copyleft terms.

Panel of Table 9 — Table 9: Task-type allocation matrix and post-review retention statistics. Alloc./Grp. : number of questions assigned to each of the 10 annotation groups. Initial : total candidate questions before cross-group review ( $=\text{Alloc./Grp.}\times 10$ ). Final : questions retained after cross-group blind review and consensus adjudication. Ret. : retention rate. (s) / (c) : questions on simple/complex images. $\dagger$ Five groups contribute 8 questions; the other five contribute 7. Dots and bars follow the capability colors: Actionability, Knowledge, Experimentation, Reasoning.

Panel of Table 10 — Table 10: Cross-group annotation agreement on candidate and retained spatial questions. IoU and point-in-region agreement are percentages; trajectory agreement is scored on a 0–100 scale.

Panel of Table 11 — Table 11: Overview of evaluation prompt templates. Badges index Prompts 1–7 below and mark image ( ) or video ( ) input; task names follow the capability colors of Table 9 .

	Capability Dimension				Aggregate
Model	Action. $\uparrow$	Know. $\uparrow$	Exper. $\uparrow$	Reason. $\uparrow$	Overall $\uparrow$
Chemistry-specialized Multimodal Models (open-source)
ChemVLM-8B [ 40 ]	11.6	31.7	37.1	31.0	28.4
ChemVLM-26B [ 40 ]	15.1	26.7	38.7	35.2	30.1
Science-oriented Multimodal Models (open-source)
Intern-S1 [ 38 ]	14.0	37.8	55.1	48.1	40.5

Appendix

Table 12: Capability-level scores of all 28 MLLMs on BEAKER. Action., Know., Exper., and Reason. denote Actionability, Knowledge, Experimentation, and Reasoning, aggregated from the task scores in Table 2 as described in Section 4.1 ; Overall combines them with weights of 0.25, 0.15, 0.35, and 0.25. Cell shading is proportional to the score, and the last row averages each column over all 28 models. Rankings within each column are color-coded: 1st , 2nd , and 3rd .

Aggregation	Scheme	Spearman $\boldsymbol{\rho}$	Kendall $\boldsymbol{\tau}$	Top-5	Top-10
Equal-column macro	13 score columns, equal weights	0.994	0.952	5/5	10/10
Equal-group macro	3 task groups, equal weights	0.993	0.947	5/5	10/10
Sample-weighted micro	Weighted by question count	0.996	0.963	5/5	10/10
Rank-based	Mean within-column model rank	0.980	0.889	5/5	10/10

Appendix

Table 13: Ranking sensitivity to score aggregation. Top-

k

reports overlap with the official top-

k

set; average ranks are used for ties.

Panel of Table 14 — Table 14: Bootstrap score and rank uncertainty. CI denotes the 95% Confidence Interval from 10,000 paired, task-stratified resamples. Lower ranks indicate better performance. Dot colors mark general-purpose, science-oriented, chemistry-specialized, and embodied brain models; the strip plots each score (dot) and its 95% CI (bar).

	By Video Length			Aggregate
Model	S-TSR $\uparrow$	M-TSR $\uparrow$	L-TSR $\uparrow$	TSR $\uparrow$
Chemistry-specialized Multimodal Models (open-source)
ChemVLM-8B [ 40 ]	45.7	18.0	50.0	37.9

Appendix

Table 15: Temporal sequence reasoning across different video lengths. S-TSR, M-TSR, and L-TSR denote Temporal Sequence Reasoning on short, mid-length, and long videos, and TSR is their unweighted mean. Cell shading is proportional to the score, and the last row averages each column over all 28 models. Rankings within each column are color-coded: 1st , 2nd , and 3rd . Tied scores are assigned the same rank.

Embodied Agent Benchmarks

See all 51 papers

onepot-Bench 0: towards lab-aware in silico chemistry benchmarks
Aug 3, 2026Brandon Wang, Andrei S. Tyrin, Daniil A. BoikoLLM Safety Benchmarks LLM Evaluation
Onepot AI, Inc.
LABSHIELD: A Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratories
Mar 12, 2026Qianpu Sun, Xiaowei Chi, Yuhan Rui +5AI Agent Safety Benchmarks Multimodal Large Language Models
Tsinghua University, Beijing, China · The Hong Kong University of Science and Technology, Hong Kong, China · Southern University of Science and Technology, Shenzhen, China +2
Stress-testing large language model agents in a robotic chemistry laboratory
Jul 25, 2026Lulu Guo, Yingkai Sun, Xiaobo Li +13AI Agent Reliability AI Agents for Scientific Discovery
State Key Laboratory of Precision and Intelligent Chemistry, Hefei National Research Center for Physical Sciences at the Microscale, School of Chemistry and Materials Science, University of Science and Technology of China, Hefei, China · Center for Scientific Intelligence Innovation, Hefei, China · School of Chemistry, School of Computer Science, University of Birmingham, Birmingham, UK

Aug 3, 2026J/K move · Enter open · S save

onepot-Bench 0: towards lab-aware in silico chemistry benchmarks

Brandon Wang, Andrei S. Tyrin, Daniil A. Boiko

Onepot AI, Inc.

LLM Safety Benchmarks LLM Evaluation Benchmark Design

Open paper AskPDF arXiv

Language models are playing an increasingly important role in laboratory science, performing tasks such as experiment planning, execution, and post-hoc analysis. However, precisely measuring their abilities is difficult, as scientific capabilities require a mixture of both problem-solving skills and domain-specific intuition. Existing evaluations rarely measure the capabilities required to make reliable decisions in a physical laboratory and often rely on public data that may have appeared in model training corpora. We introduce onepot-Bench 0, a proprietary benchmark suite for evaluating language models on synthetic chemistry capabilities relevant to wet-lab execution. onepot-Bench 0 comprises three complementary evaluations: ChemAbacus measures tool-free cheminformatics literacy and numerical reasoning; SynthRefusal characterizes safety and refusal behavior across a variety of benign, controlled, and designer-drug targets; and SynthBench evaluates reaction-outcome prediction and catalyst selection using private experimental data generated in our laboratory. Together, these evaluations probe basic competency, reliability, and deeper knowledge, all skills which are required for reliable performance in the lab.

Abstract

Embodied Agent Benchmarks

Explore similar work

onepot-Bench 0: towards lab-aware in silico chemistry benchmarks

LABSHIELD: A Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratories

Stress-testing large language model agents in a robotic chemistry laboratory