cs.AINov 4, 2026

BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories

Authors: Fei Lin, Tengchao Zhang, Ziyang Gong, Xiaotong Yu, Bohan Zhang, Yifan Zhou, Qihao Yang, Ji Dai, 

Organizations: HydroSophy · Macau University of Science and Technology · Shanghai Jiao Tong University · Institute of Process Engineering, Chinese Academy of Sciences · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Self-Driving Chemical Laboratories (SDCLs) are moving chemical experimentation toward embodied automation, where machines must perceive laboratory scenes, reason about experimental states, and act under physical, procedural, and safety constraints. Although Multimodal Large Language Models (MLLMs) are increasingly considered as embodied brains for such systems, existing laboratory benchmarks mainly focus on safety, anomaly detection, or general scientific understanding, leaving their execution-oriented embodied capabilities underexplored. To fill this gap, we introduce BEAKER, a Benchmark for evaluating Embodied Actionability, Knowledge, Experimentation, and Reasoning in SDCLs. BEAKER contains 1,000 expert-curated question-answer samples across 11 image and video task types, covering key judgments required for chemical laboratory execution, including affordance grounding, trajectory planning, equipment, state, and safety recognition, apparatus setup, operation and step reasoning, and temporal procedure reasoning. All questions and annotations are manually created and reviewed by chemistry and embodied Artificial Intelligence (AI) experts, without AI-assisted generation or annotation. We evaluate 28 general-purpose, science-oriented, chemistry-specialized, and embodied MLLMs in an offline, single-turn setting. Results show that current models can follow coarse experimental procedures in videos, but still struggle to ground actions into operable regions and feasible trajectories, to recognize fine-grained laboratory apparatus, states, and safety norms, and to order steps in mid-length procedures. BEAKER thus reveals a clear gap between coarse procedural understanding and the precise, action-grounded perception required for laboratory execution, providing a diagnostic benchmark for future MLLMs in SDCLs.

Figures & tables

Figure 1: Overview of BEAKER and key evaluation findings. The radar chart compares four representative models across 11 tasks grouped by capability. All axes use a 0–100 scale except AFF, which uses a 0–50 scale due to uniformly low scores across all models. AFF: Affordance Grounding; TRAJ: Trajectory Planning; ERI: Equipment and Reagent Identification; ESS: Experimental Safety and Standardization; SR: State Recognition; EAS: Experimental Apparatus Setup; EOP: Experimental Operations; SVOR: Short-Video Operation Reasoning; MVTS: Mid-Video Task-Step Reasoning; TSR: Temporal Sequence Reasoning; LVPR: Long-Video Experimental Procedure Reasoning; GT: Ground Truth.
BenchmarkScaleSDCLRealImageVideoAction.Know.Exper.Reason.Human
MaCBench [ 29 ]1.1k+ QA△\triangle✓✓✗✗✓△\triangle✓✓
ExpVid [ 30 ]7.8k QA△\triangle✓✗✓✗✓✓✓△\triangle
LabUtopia [ 64 ]30 tasks△\triangle✗✓✗△\triangle✓✓✓✓
AutoBio [ 65 ]16 tasks✗✗✓✗△\triangle✓✓△\triangle✓
Chemistry3D [ 66 ]Sim. env.△\triangle✗✓✗△\triangle✓✓△\triangle✓
SDCL Visual Anomaly Dataset [ 26 ]2.8k pairs✓✓✓✗✗△\triangle✓✓✓
Table 1: Comparison with related scientific and laboratory benchmarks. Scale denotes the primary evaluation unit of each work: QA for question-answering samples, tasks for embodied or robotic tasks, pairs for image-text pairs, imgs for images, and Sim. env. for simulation environments. ✓ , ✗ , and △\triangle denote full, no, and partial coverage, respectively. Human indicates whether all questions and annotations are authored by human experts without LLM/AI-assisted generation or annotation.
Figure 2: Overview of BEAKER construction and task examples. Top: the four-stage expert-curated pipeline of data collection, expert curation, QA authoring, and quality control. Bottom: examples of the four capability dimensions with the input, question, Ground Truth (GT), and GPT-5.4 [ 72 ] prediction.
Figure 3: Statistics of BEAKER. Counts by input type, capability, task, data source, and modality.
ModelActionabilityKnowledgeExperimentationReasoningOverall
AFF ↑\uparrowTRAJ ↑\uparrowERI ↑\uparrowESS ↑\uparrowSR ↑\uparrowEAS ↑\uparrowEOP ↑\uparrowSVOR ↑\uparrowMVTS ↑\uparrowTSR ↑\uparrowLVPR ↑\uparrow
Human88.091.096.795.093.393.396.794.492.095.096.093.3
Chemistry-specialized Multimodal Models
Open-source Models            
ChemVLM-8B [ 40 ]2.121.033.340.021.736.733.336.040.037.924.028.4
ChemVLM-26B [ 40 ]3.127.023.325.031.736.733.337.642.742.428.030.1
Table 2: Consolidated results on BEAKER. Task scores are grouped by the four capability dimensions, and Overall aggregates them as described in Section 4.1 . Model rankings within each column are color-coded: 1st , 2nd , and 3rd ; tied scores share the same rank.
ModelActionabilityKnowledgeExperimentationReasoningOverall
AFFTRAJAvg.ERIESSSRAvg.EASEOPSVORMVTSAvg.TSRLVPRAvg.
ChemVLM-8B [ 40 ]-2.00 (13/16)-5.78 (42/108)-3.89-11.67 (10/17)-1.67 (12/13)-35.00 (3/24)-16.11-8.33 (11/16)-26.67 (6/22)-39.20 (8/57)-26.67 (9/29)-25.22-16.25 (16/44)-26.17 (5/30)-21.21-18.67
Intern-S1-Mini [ 38 ]-2.33 (7/17)-4.63 (50/100)-3.48-1.67 (13/14)-3.33 (9/11)-13.33 (5/13)-6.11-15.00 (10/19)-18.33 (11/22)-28.80 (7/43)-10.67 (5/13)-18.20-30.16 (15/57)+2.50 (16/14)-13.83-11.84
RoboBrain2.5-8B [ 49 ]-1.81 (8/11)+4.08 (80/70)+1.14-3.33 (4/6)-6.67 (5/9)-8.33 (3/8)-6.11-1.67 (8/9)-1.67 (6/7)-0.80 (6/7)-5.33 (5/9)-2.37+1.11 (14/13)-2.67 (5/7)-0.78-1.76
Table 3: Specialization gains and losses relative to Qwen3-VL-8B [ 92 ] . Task cells report score differences with paired wins/losses below; ties are omitted. Avg. reports the difference in unweighted capability means (and thus differs from the 2:5-weighted Experimentation score in Table 12 ). Positive and negative differences are shown in green and red, respectively; shaded bold cells summarize capability and Overall differences.
Table 4: Capability profile of representative MLLMs on BEAKER. Avg. is averaged over the 13 models shown (all 28 in Table 12 ). Shading scales with the score; the best result in each row is in bold.
Table 5: Performance across question-format task groups. We compare representative models across three BEAKER QA task groups: Coordinate (Coord.) QA, MCQ, and Fill-in-the-Blank (FIB) QA. MCQ scores are computed as the unweighted macro-average across the eight MCQ task types. The last three columns report pairwise performance ratios between task groups, reflecting relative performance rather than format-specific robustness. Score cells are shaded by value, and ratio cells by their distance from 1.0.
Figure 4: Qualitative examples for three representative task types: Affordance Grounding (AFF) (Actionability, image), Equipment and Reagent Identification (ERI) (Knowledge, image MCQ), and Short-Video Operation Reasoning (SVOR) (Experimentation, video MCQ). Left panels show correct GPT-5.4 [ 72 ] predictions; right panels show incorrect ones. GT: Ground Truth.
Figure 5: Self-explanation examples of GPT-5.4 [ 72 ] on BEAKER. Representative correct and incorrect cases in Long-Video Experimental Procedure Reasoning (LVPR) show that GPT-5.4 can extract procedural evidence from experimental videos but may struggle to match visual observations with fine-grained chemical concepts. GT: Ground Truth.
Figure 6: Visual input ablation on image MCQs. Accuracy is averaged over five tasks with 60 questions each. Image + text includes images, questions, and options; Text only removes the images. Visual input improves average accuracy for all eight models. The dashed line marks the random-choice baseline (Appendix K ). Gain denotes the increase in percentage points (pp).
Figure 7: Input modality ablation on video-based tasks. Performance of GPT-5.4, Claude Opus 4.7, and Qwen3.5-397B-A17B under video-plus-question, caption-plus-question, and caption-plus-video-plus-question settings. M-TSR: Mid-Video Temporal Sequence Reasoning; MVTS: Mid-Video Task-Step Reasoning; L-TSR: Long-Video Temporal Sequence Reasoning; LVPR: Long-Video Experimental Procedure Reasoning.
Table 6: Frame sampling ablation on video-based BEAKER tasks. We evaluate Qwen3.5-397B-A17B [ 93 ] under different frame sampling settings for short-, mid-, and long-video tasks. Frames are listed as short / mid / long; long-video frames are arranged into four-frame image grids (16, 32, and 48 grids). The best result in each column is in bold.
Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

SourceContent#Raw
Image sources (12)
HeinSight 2.0 & 3.0 [ 95 , 96 ]Vessel detection, liquid/solid phase classification5,131
Roboflow [ 97 ]Safety labels, apparatus, spills, operations (10 sets)3,178
ChemEq25 [ 98 ]25-class real-world apparatus across multiple labs1,138
Chemical Apparatus [ 99 ]Apparatus in complex backgrounds with 6 categories629
CLAD [ 100 ]21-class apparatus recognition and detection488
Appendix
Table 7: Data sources for BEAKER. Roboflow encompasses 10 community-contributed datasets.
Table 8: License and usage information for external assets used in BEAKER construction. Badge colors mark permissive , use-restricted , and copyleft terms.
Table 9: Task-type allocation matrix and post-review retention statistics. Alloc./Grp. : number of questions assigned to each of the 10 annotation groups. Initial : total candidate questions before cross-group review ( =Alloc./Grp.×10=\text{Alloc./Grp.}\times 10 ). Final : questions retained after cross-group blind review and consensus adjudication. Ret. : retention rate. (s) / (c) : questions on simple/complex images. †\dagger Five groups contribute 8 questions; the other five contribute 7. Dots and bars follow the capability colors: Actionability, Knowledge, Experimentation, Reasoning.
Table 10: Cross-group annotation agreement on candidate and retained spatial questions. IoU and point-in-region agreement are percentages; trajectory agreement is scored on a 0–100 scale.
Table 11: Overview of evaluation prompt templates. Badges index Prompts 1–7 below and mark image ( ) or video ( ) input; task names follow the capability colors of Table 9 .
 Capability Dimension Aggregate
ModelAction. ↑\uparrowKnow. ↑\uparrowExper. ↑\uparrowReason. ↑\uparrow Overall ↑\uparrow
Chemistry-specialized Multimodal Models (open-source)
ChemVLM-8B [ 40 ]11.631.737.131.0 28.4
ChemVLM-26B [ 40 ]15.126.738.735.2 30.1
Science-oriented Multimodal Models (open-source)
Intern-S1 [ 38 ]14.037.855.148.1 40.5
Appendix
Table 12: Capability-level scores of all 28 MLLMs on BEAKER. Action., Know., Exper., and Reason. denote Actionability, Knowledge, Experimentation, and Reasoning, aggregated from the task scores in Table 2 as described in Section 4.1 ; Overall combines them with weights of 0.25, 0.15, 0.35, and 0.25. Cell shading is proportional to the score, and the last row averages each column over all 28 models. Rankings within each column are color-coded: 1st , 2nd , and 3rd .
AggregationSchemeSpearman ρ\boldsymbol{\rho}Kendall τ\boldsymbol{\tau}Top-5Top-10
Equal-column macro13 score columns, equal weights0.9940.9525/510/10
Equal-group macro3 task groups, equal weights0.9930.9475/510/10
Sample-weighted microWeighted by question count0.9960.9635/510/10
Rank-basedMean within-column model rank0.9800.8895/510/10
Appendix
Table 13: Ranking sensitivity to score aggregation. Top- kk reports overlap with the official top- kk set; average ranks are used for ties.
Table 14: Bootstrap score and rank uncertainty. CI denotes the 95% Confidence Interval from 10,000 paired, task-stratified resamples. Lower ranks indicate better performance. Dot colors mark general-purpose, science-oriented, chemistry-specialized, and embodied brain models; the strip plots each score (dot) and its 95% CI (bar).
 By Video Length Aggregate
ModelS-TSR ↑\uparrowM-TSR ↑\uparrowL-TSR ↑\uparrow TSR ↑\uparrow
Chemistry-specialized Multimodal Models (open-source)
ChemVLM-8B [ 40 ]45.718.050.0 37.9
Appendix
Table 15: Temporal sequence reasoning across different video lengths. S-TSR, M-TSR, and L-TSR denote Temporal Sequence Reasoning on short, mid-length, and long videos, and TSR is their unweighted mean. Cell shading is proportional to the score, and the last row averages each column over all 28 models. Rankings within each column are color-coded: 1st , 2nd , and 3rd . Tied scores are assigned the same rank.
More in this topic

Embodied Agent Benchmarks

See all 51 papers

Explore similar work

CardsList
Aug 3, 2026

onepot-Bench 0: towards lab-aware in silico chemistry benchmarks

Language models are playing an increasingly important role in laboratory science, performing tasks such as experiment planning, execution, and post-hoc analysis. However, precisely measuring their abilities is difficult, as scientific capabilities require a mixture of both problem-solving skills and domain-specific intuition. Existing evaluations rarely measure the capabilities required to make reliable decisions in a physical laboratory and often rely on public data that may have appeared in model training corpora. We introduce onepot-Bench 0, a proprietary benchmark suite for evaluating language models on synthetic chemistry capabilities relevant to wet-lab execution. onepot-Bench 0 comprises three complementary evaluations: ChemAbacus measures tool-free cheminformatics literacy and numerical reasoning; SynthRefusal characterizes safety and refusal behavior across a variety of benign, controlled, and designer-drug targets; and SynthBench evaluates reaction-outcome prediction and catalyst selection using private experimental data generated in our laboratory. Together, these evaluations probe basic competency, reliability, and deeper knowledge, all skills which are required for reliable performance in the lab.

Brandon Wang, Andrei S. Tyrin, Daniil A. Boiko

Onepot AI, Inc.

LLM Safety BenchmarksLLM EvaluationBenchmark Design
AskPDF
Mar 12, 2026

LABSHIELD: A Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratories

Artificial intelligence is increasingly catalyzing scientific automation, with multimodal large language model (MLLM) agents evolving from lab assistants into self-driving lab operators. This transition imposes stringent safety requirements on laboratory environments, where fragile glassware, hazardous substances, and high-precision laboratory equipment render planning errors or misinterpreted risks potentially irreversible. However, the safety awareness and decision-making reliability of embodied agents in such high-stakes settings remain insufficiently defined and evaluated. To bridge this gap, we introduce LABSHIELD, a realistic multi-view benchmark designed to assess MLLMs in hazard identification and safety-critical reasoning. Grounded in U.S. Occupational Safety and Health Administration (OSHA) standards and the Globally Harmonized System (GHS), LABSHIELD establishes a rigorous safety taxonomy spanning 164 operational tasks with diverse manipulation complexities and risk profiles. We evaluate 20 proprietary models, 9 open-source models, and 3 embodied models under a dual-track evaluation framework. Our results reveal a systematic gap between general-domain MCQ accuracy and Semi-open QA safety performance, with models exhibiting an average drop of 32.0% in professional laboratory scenarios, particularly in hazard interpretation and safety-aware planning. These findings underscore the urgent necessity for safety-centric reasoning frameworks to ensure reliable autonomous scientific experimentation in embodied laboratory contexts. The full dataset will be released soon.

Qianpu Sun, Xiaowei Chi, Yuhan Rui +5

Tsinghua University, Beijing, China · The Hong Kong University of Science and Technology, Hong Kong, China · Southern University of Science and Technology, Shenzhen, China +2

AI Agent Safety BenchmarksMultimodal Large Language ModelsLaboratory Automation
AskPDF
Jul 25, 2026

Stress-testing large language model agents in a robotic chemistry laboratory

AI is evaluated through knowledge, reasoning and plan generation, yet scientific agency requires reliable physical action and adaptation to evidence. Here, we use a robotic chemistry laboratory as a physical-world testbed to make scientific agency measurable. Its 45 modular workstations exposed as machine-readable skills enabled 4,608 trials. Only 3.3% of trials produced expert-assessed executable workflows under laboratory constraints; even the best system achieved 28.1%. Long-horizon planning remained a challenge: only three executable workflows exceeded 30 operations, although the longest contained 44. Across five rounds, experimental feedback prompted local adjustments but no workflow-level replanning or analytical-method redesign. By making physical executability and evidence-driven replanning measurable, our study provides an evidence-based assessment of deployment readiness and a diagnostic framework to guide closed-loop improvements towards physically grounded autonomous research.

Lulu Guo, Yingkai Sun, Xiaobo Li +13

State Key Laboratory of Precision and Intelligent Chemistry, Hefei National Research Center for Physical Sciences at the Microscale, School of Chemistry and Materials Science, University of Science and Technology of China, Hefei, China · Center for Scientific Intelligence Innovation, Hefei, China · School of Chemistry, School of Computer Science, University of Birmingham, Birmingham, UK

AI Agent ReliabilityAI Agents for Scientific DiscoveryRobotic Laboratory AutomationAI Agent EvaluationRoboticsLaboratory Automation+2
AskPDF