RoboChemGym: A Protocol-Driven Generative Simulation Framework for Long-Horizon Chemical Manipulation

Chenxi Li; Haiyuan Wan; Rui Li; Jingyuan Li; Sha Zhang; Bohan Feng; Jianbao Cao; Zhangrui Zhao

cs.ROOct 2, 2026

RoboChemGym: A Protocol-Driven Generative Simulation Framework for Long-Horizon Chemical Manipulation

Authors: Chenxi Li, Haiyuan Wan, Rui Li, Jingyuan Li, Sha Zhang, Bohan Feng, Jianbao Cao, Zhangrui Zhao, +5 more

Organizations: Shanghai Artificial Intelligence Laboratory · Zhejiang University · Tsinghua University · Harbin Institute of Technology · Shanghai Jiao Tong University · The Chinese University of Hong Kong · Fudan University · Wuhan University · Beihang University · Renmin University of China

Abstract

Wet-lab experimentation serves as the gold standard for hypothesis verification in scientific discovery; yet it is inherently labor-intensive, costly, and safety-critical. Embodied agents hold the promise of automating these tedious workflows, but their development is hindered by the scarcity of real-world training data. While simulation offers a scalable alternative for producing demonstrations, current methods primarily target relatively short-horizon tasks with loosely structured interactions, failing to meet the strict procedural constraints and fine-grained manipulation demands of chemical experiments. To bridge this gap, we introduce RoboChemGym, a framework that autonomously generates high-fidelity manipulation demonstrations aligned with real-world experiment protocols, featuring a self-improving task synthesis mechanism to iteratively refine task execution and scene configurations, enabling the reliable generation of expert trajectories for complex, multi-object protocols exceeding 10 interaction steps. Furthermore, we introduce a hierarchical benchmark that systematically assesses performance across varying granularities, spanning from atomic operations to full-cycle experimental workflows. RoboChemGym sets a scalable paradigm for the automated data synthesis and capability evaluation of embodied agents in intricate chemical tasks, serving as a critical stepping stone toward fully intelligent laboratories.

Figures & tables

Figure 1: We illustrate a protocol-driven chemical manipulation task based on a real laboratory reaction. Our Goal is to support fine-grained, long-horizon manipulation involving multiple objects. Action Refinement corrects execution failures, while Scene Refinement resolves layout issues. Protocol-Aligned refers to manipulation trajectories generated by our framework for the given experimental protocol. Together, these components enable robust Sim2Real transfer.

Figure 2: RoboChemGym Pipeline. (a) Given a textual protocol, the framework constructs an initial scene and executable action sequence using asset and atomic action libraries. (b) Dual-loop optimization refines actions and layouts under continuous monitoring, correcting execution errors and resolving global feasibility issues. (c) The memory module accumulates refinement histories to guide subsequent iterations and enable self-improvement. (d) Lab-centric data scaling applies randomization to generate diverse expert trajectories for policy training.

Setting	Config	Level 1	Level 2	Level 3
Clean	Ours	89.2	73.4	52.3
	$-$ AM	75.8 $\downarrow$ 13.4	52.7 $\downarrow$ 20.7	27.4 $\downarrow$ 24.9
	$-$ SM	73.6 $\downarrow$ 15.6	56.4 $\downarrow$ 17.0	32.3 $\downarrow$ 20.0
Random	Ours	85.6	70.3	47.5
	$-$ AM	69.7 $\downarrow$ 15.9	48.9 $\downarrow$ 21.4	21.2 $\downarrow$ 26.3
	$-$ SM	64.1 $\downarrow$ 21.5	51.8 $\downarrow$ 18.5	25.4 $\downarrow$ 22.1

Table 1: Success rates (%) of the generative pipeline across configurations and task difficulty levels. Clean excludes lab-centric data scaling, while Random includes it. -AM and -SM denote removing action- and scene-level memory, respectively. The

\Delta

values indicate the corresponding decreases in success rate.

	Level-1 (3-step)				Level-2 (7-step)
Method	A1	A2	A3	Full Seq	A1	A2	A3	A4	A5	A6	A7	Full Seq
$\pi_{0}$	14.3	4.7	1.3	0.8	11.5	3.1	0.8	0.2	0	0	0	0
$\pi_{0}{+}\text{clean}$	90.2	82.5	75.6	55.4 ↑ 54.6	84.5	72.1	60.4	50.2	42.1	30.5	22.4	18.2 ↑ 18.2
$\pi_{0}{+}\text{random}$	94.1	88.6	82.3	68.5 ↑ 67.7	88.5	78.4	68.2	58.4	48.5	40.2	32.5	28.4 ↑ 28.4
ACT $+$ clean	72.0	62.5	50.4	22.5	65.5	52.3	40.1	32.5	24.8	15.2	8.5	5.2
ACT $+$ random	80.5	70.2	60.1	35.2	72.1	58.4	45.2	38.6	30.1	22.4	15.8	12.4

Table 2: Step-wise and full-sequence success rates (%) in long-horizon chemical manipulation tasks. Level-1 contains 3 actions and Level-2 contains 7 actions. ↑ is reported for

\pi_{0}{+}\text{clean}

and

\pi_{0}{+}\text{random}

, denoting absolute improvement over

\pi_{0}

on Full Seq metric.

	Level-3 (11-step)
Method	A1	A2	A3	A4	A5	A6	A7	A8	A9	A10	A11	Full Seq
$\pi_{0}$	9.5	2.1	0.5	0	0	0	0	0	0	0	0	0
$\pi_{0}{+}\text{clean}$	84.4	72.0	60.3	50.1	42.0	30.4	22.3	16.1	11.6	8.3	6.0	0.5 ↑ 0.5
$\pi_{0}{+}\text{random}$	88.4	78.3	68.1	58.3	48.4	40.1	32.4	23.3	16.8	12.1	8.7	1.7 ↑ 1.7

Table 3: Step-wise and full-sequence success rates (%) in ultra-long horizon chemical manipulation tasks. Level-3 contains 11 actions. ↑ is reported for

\pi_{0}{+}\text{clean}

and

\pi_{0}{+}\text{random}

, denoting absolute improvement over

\pi_{0}

on Full Seq metric.

Figure 3: Success rates of Level 1, Level 2, and Level 3 under clean and random settings as a function of the number of iterations.

Task	Position	Clutter	Clean Real	RoboChemGym	Combined
Solid Weighing	Seen-Position	No	19.8	20.5	41.8
		Yes	14.9	17.2	38.5
	Unseen-Position	No	13.2	16.8	36.9
		Yes	9.1	13.5	32.7
Pyridine Addition	Seen-Position	No	23.5	29.3	36.4
		Yes	18.1	22.4	33.6

Table 4: Real-world success rates (%) under different task configurations. Seen-Position and Unseen-Position indicate whether object locations are observed during training. Clutter indicates the presence of distractor objects. Numbers in parentheses denote absolute improvements.

Task Level	# Calls	Input Tokens / Task	Output Tokens / Task	Cost / Task (USD)	TTFW / Call (s)
Level 1	$\sim$ 15	Low: $\sim$ 20k Typical: $\sim$ 38k	Low: $\sim$ 21k Typical: $\sim$ 36k	Low: $\sim$ 0.29 Typical: $\sim$ 0.50	$\sim$ 15–20 $\sim$ 25–40
Level 2	$\sim$ 20	Low: $\sim$ 27k Typical: $\sim$ 50k	Low: $\sim$ 28k Typical: $\sim$ 48k	Low: $\sim$ 0.38 Typical: $\sim$ 0.66	$\sim$ 15–20 $\sim$ 25–40
Level 3	$\sim$ 30	Low: $\sim$ 40k Typical: $\sim$ 75k	Low: $\sim$ 42k Typical: $\sim$ 72k	Low: $\sim$ 0.57 Typical: $\sim$ 0.99	$\sim$ 15–20 $\sim$ 25–40

Table 5: Per-task LLM overhead for protocol-driven refinement. Each task triggers multiple API calls (Level 1/2/3:

\sim

15/20/30 calls). Low and Typical estimates are derived from the two most frequent call patterns observed in empirical logs (one record corresponds to one API call). TTFW is measured under non-streaming settings.

Figure 4: Representative visualizations of the lab-centric data scaling strategy. Each column illustrates variations in object spatial layouts, background distractors, workbench materials, and illumination conditions, while preserving operational feasibility and experimental validity.

Figure 5: Visual comparison of sim-to-real transfer. For two representative manipulation tasks, we compare the simulated environment (left) with the corresponding real-world execution (right), showing consistent object placement, robot configurations, and task progress

Figure 6: Sim-to-real evaluation of open-loop trajectory replay on a physical robot. Success rates are reported over 100 trajectories per difficulty level, comparing cases with and without reset correction actions inserted between pick-and-place executions.

Explore similar work

SciHorizon-eLab: An Agentic Protocol-to-Task Compiler for Scalable Benchmarking of Scientific Embodied Agents
Sep 25, 2026Maokai Qin, Chuan Qin, Qi Zhang +5Embodied AI AI Agent Benchmarks
Data Intelligence for Scientific Innovation Lab, Computer Network Information Center, Chinese Academy of Sciences · Beihang University
LabDex: A Hierarchical Benchmark for Dexterous Manipulation in Laboratories
Aug 19, 2026Zhipeng Tang, Sihang Chen, Sha Zhang +11Robot Manipulation Benchmarks Dexterous Manipulation
University of Science and Technology of China · The Chinese University of Hong Kong · iFLYTEK
AgentChemist: A Multi-Agent Experimental Robotic Platform Integrating Chemical Perception and Precise Control
Mar 25, 2026Xiangyi Wei, Fei Wang, Haotian Zhang +6Laboratory Automation Autonomous Experimentation
School of Computer Science and Technology, East China Normal University, Shanghai, China. · School of Chemistry and Molecular Engineering, East China Normal University, Shanghai, China. · School of Data Science and Engineering, East China Normal University, Shanghai, China. +1

Sep 25, 2026J/K move · Enter open · S save

SciHorizon-eLab: An Agentic Protocol-to-Task Compiler for Scalable Benchmarking of Scientific Embodied Agents

Maokai Qin, Chuan Qin, Qi Zhang, Dianyu Liu+4

Data Intelligence for Scientific Innovation Lab, Computer Network Information Center, Chinese Academy of Sciences · Beihang University

Embodied AI AI Agent Benchmarks Synthetic Benchmark Generation Scientific Workflow Automation Laboratory Automation

Open paper AskPDF arXiv

Embodied agents offer a promising route to automating scientific experimentation, yet their progress is constrained by the lack of reliable and systematic evaluation environments. Existing simulation-based laboratory benchmarks rely heavily on manual task engineering, making it challenging to systematically compile diverse scientific protocols into executable and verifiable embodied tasks at scale. To address this challenge, we introduce SciHorizon-eLab, an agentic protocol-to-task compiler that formulates scientific embodied task construction as a compilation problem. Given a natural-language protocol of scientific experiments, SciHorizon-eLab progressively compiles laboratory protocols into semantic-preserving embodied tasks through semantic grounding, executable task synthesis, and multi-stage simulation-based certification. The system generates semantically grounded environments, executable manipulation programs, and step-level success specifications, while enabling reproducible generation of expert demonstrations and execution traces. Using this pipeline, we further construct \BenchName, a ready-to-use benchmark comprising 300 certified tasks across diverse laboratory operations. It supports HIL task execution, reproducible expert-demonstration generation, and ordered step-level evaluation. Across representative tasks, the strongest policy attains an average success rate of only 49.7%, with further evaluations revealing pronounced weaknesses in human and embodied agent coordination. We publicly release the code, benchmark data, and evaluation toolkit at https://github.com/SciHorizon-elab/SciHorizon-elab.

Abstract

Explore similar work

SciHorizon-eLab: An Agentic Protocol-to-Task Compiler for Scalable Benchmarking of Scientific Embodied Agents

LabDex: A Hierarchical Benchmark for Dexterous Manipulation in Laboratories

AgentChemist: A Multi-Agent Experimental Robotic Platform Integrating Chemical Perception and Precise Control