RoboAtom: Large-scale Atomic-Action Supervision for Robot Learning

Xueyuan Wei; Xuanxu Wang; Xudong Li; Yuping He; Wanshui Gan; Jiaqi Peng; Wenqi Guo; Jianjun Zhou

cs.ROOct 29, 2026

RoboAtom: Large-scale Atomic-Action Supervision for Robot Learning

Authors: Xueyuan Wei, Xuanxu Wang, Xudong Li, Yuping He, Wanshui Gan, Jiaqi Peng, Wenqi Guo, Jianjun Zhou,

Organizations: Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Fudan University · Tsinghua University ·

Abstract

Fine-grained action annotations can support skill reuse and compositional generalization in robot policies. However, robot datasets typically provide task-level instructions without subtask annotations. Although some include subtask labels, inconsistent formats and conventions hinder cross-source joint training. We propose RoboAtom-Data, which adds unified, time-aligned atomic-action annotations to existing data from multiple sources, covering 1.45 million trajectories, 6.88 million skill-labeled segments, and 16,882.1 hours. Our pipeline combines a unified protocol, large-scale manual annotation, propagation across repeated executions with the same ordered skill sequence, and multi-stage quality control. We evaluate the dataset through automatic atomic-action annotation and robot control. We construct RoboAtom-Bench to standardize evaluation of skill recognition and temporal boundary localization. Trained on the annotated data, RoboAtom-Anno achieves better automatic annotation performance than all evaluated open- and closed-source models on both RoboAtom-Bench and the external WGO-Bench. For robot control, atomic-action supervision paired with hierarchical execution improves task success over task-level baselines on LIBERO-Long, LIBERO-Plus, and RoboTwin. We will release the data, benchmark, model, and construction pipeline to support further research on skill-level robot learning.

Figures & tables

Figure 1: Overview of RoboAtom: data and benchmark (top), atomic-action annotations (center), annotation performance (bottom left), and robot-control success rates (bottom right).

Figure 2: RoboAtom construction pipeline. Model-assisted annotation and human review produce seed annotations; PhysExtend and quality control scale them into RoboAtom-Data. Expert review establishes the evaluation annotations for RoboAtom-Bench (Section 4 ).

Dataset	Domain	Hours	Trajectories	Segments	Skills	Tasks	Embod.
AgiBot World Beta	Real	2,976.4	141,563 †	1,001,552	77 †	217	1
AgiBot World 2026	Real	591.6 †	28,331 †	327,355 †	43 †	37 †	1 †
Galaxea Open-World	Real	500.0	19,183 †	108,775 †	58	150	1
RoboInter-Data	Real	2,417.1 †	235,880	752,388	15	232	6
BEHAVIOR-1K 2026	Sim	1,952.9	20,000	406,341	35	100	1
RoboAtom-Data	Mixed	16,882.1 †	1,446,974 †	6,876,282 †	29	3,334 †	32 †

Table 1: Robot datasets with subtask annotations.

\dagger

denotes statistics recomputed by us; source versions and counting rules are in Appendix A.2 .

Figure 3: Skill-aware boundary evidence and extension performance. (a) Gripper-state signals aligned with Pick/Place boundaries. (b) Base-motion signals aligned with Move boundaries. (c) Skill-feature associations. (d) PhysExtend consistently outperforms Cortex across reference counts; shaded bands indicate $\pm$ 1 standard deviation.

Model	mIoU $\uparrow$	[email protected] $\uparrow$	Skill [email protected] $\uparrow$	EA (%) $\uparrow$	Cost $\downarrow$
GPT-4o	48.2	20.7	11.1	10	0.888
Gemini 3.1 Pro	61.1	44.9	39.7	39	2.877
GPT-5.6-sol	58.9	37.8	35.0	46	3.510
Qwen3.8-27B	46.1	25.4	19.1	25	0.051
Cortex	66.5	49.8	48.5	37	$4.8\times 10^{-4}$
PhysExtend	76.8	71.6	71.0	69	$\mathbf{2.8\times 10^{-5}}$

Table 2: Annotation extension performance and cost. Episode accuracy (EA) requires all subtasks in a trajectory to have correct semantics and segmentation, evaluated on 100 trajectories, one per task. Cost is in USD per source-video hour.

Figure 4: Scale and diversity of RoboAtom-Data. (a) Annotation word cloud. (b) Trajectory durations. (c) Segments per trajectory. (d) Dataset-scale comparison. (e–f) Task-instruction and action-description distributions.

Figure 5: (a–c) Illustrations of temporal mIoU/F1, Skill-F1, and Edit. Dark regions indicate unmatched predictions or edit operations. (d) Cumulative distributions of GT segment counts. (e) Ratios of total predicted to total GT segment counts across RoboAtom-Bench subsets.

Figure 6: Dual-system execution. System 2 uses a task-plan prior, visual observations, and execution history to select an atomic-action instruction. System 1 conditions $\pi_{0.5}$ on that instruction to produce robot actions; subsequent observations inform the next planning decision.

Model	RoboAtom-Bench					WGO-Bench Robot
Model	mIoU $\uparrow$	[email protected] $\uparrow$	Skill [email protected] $\uparrow$	Edit $\uparrow$	Cost $\downarrow$	mIoU $\uparrow$	[email protected] $\uparrow$	Cost $\downarrow$
GPT-4o	31.5	13.9	7.2	47.7	0.971	46.1	14.6	0.800
GPT-5.6-sol	44.0	28.0	21.4	63.4	4.330	41.9	31.3	3.375
Gemini 3.1 Pro	42.5	24.4	16.1	61.8	4.516	52.1	29.6	4.389
Qwen3.8-27B	38.4	20.1	10.9	56.6	0.103	51.3	24.3	0.096
Qwen3.5-9B	9.6	6.2	3.5	28.7	0.008	12.8	1.7	0.007

Table 3: Annotation performance and inference cost (USD per source-video hour; Appendix D.1 ).

Method	LIBERO	LIBERO-Plus					RoboTwin
Method	Long $\uparrow$	Spatial $\uparrow$	Object $\uparrow$	Goal $\uparrow$	Long $\uparrow$	Overall $\uparrow$	Clean $\uparrow$	Random $\uparrow$	Avg. $\uparrow$
$\pi_{0.5}$	95.00	93.01	85.23	79.93	78.56	84.05	80.70	48.04	64.37
Our method	98.20	92.46	91.86	86.34	82.29	88.18	82.02	50.32	66.17

Table 4: Downstream success rates (%) on LIBERO, LIBERO-Plus, and RoboTwin. LIBERO-Plus Overall is the success rate over all 10,030 tasks; RoboTwin Avg. is the Clean/Random mean.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1: Atomic-action annotation interface.

Figure A.2: Model-assisted review interface.

Dataset	Domain	Hours	Trajectories	Segments	Skills	Tasks	Embod.
Private	Real	54.7 †	3,187 †	20,088 †	10 †	7 †	2 †
ABC-130k	Real	3,585.6 †	130,740 †	2,353,804 †	17 †	198 †	1 †
EBench	Sim	182.9 †	6,600 †	42,459 †	12 †	26 †	1 †
Galaxea Open-World	Real	465.9 †	19,183 †	124,104 †	20 †	169 †	1 †
InternData-A1	Sim	7,073.8 †	621,633 †	1,740,880 †	13 †	1,148 †	1 †
LIBERO	Sim	3.9 †	1,712 †	3,910 †	8 †	40 †	1 †

Appendix

Table A.2: Per-source statistics of RoboAtom-Data.

\dagger

: local or recomputed statistics. Private is one merged source; segments require non-empty skill labels. Skills counts distinct non-empty labels per source and is non-additive. Galaxea includes retained source labels as well as core skills; the total of 29 denotes the core vocabulary size. Embod. counts distinct non-empty robot_type strings, deduplicated across sources for the total.

Category	Samples	Training tasks
Locomotion	17,185	Navigation actions, movement direction, and target keypoint prediction
Manipulation	954	Grasp/release actions and manipulation keypoint prediction
Robot QA	1,007,450	Question answering on robot state, scene, and action
Grounding	1,080,004	Localization of 2D points, target boxes, affordance regions, and manipulation trajectories
Planning	326,747	Task decomposition, action sequence, and next-action prediction
Spatial reasoning	409,602	Reasoning about distance, direction, relative position, and 3D relations

Appendix

Table A.4: Task composition of robot-domain mid-training.

Test subset	Trajectories	Share	GT segments	Mean GT segments	Median segments	Segment range	Total frames	Duration (hours)	Videos
Short	200	40.0%	411	2.06	2	1–4	74,214	0.732	200
Medium	100	20.0%	634	6.34	6	5–8	126,762	1.193	100
Long	100	20.0%	2,302	23.02	21	9–62	298,015	2.761	100
Multiview	100	20.0%	671	6.71	5	1–48	108,105	1.119	321
Total	500	100.0%	4,018	8.04	5	1–62	607,096	5.806	721

Appendix

Table A.5: Test-set distribution of RoboAtom-Bench. Multi-view durations are not multiplied by the number of cameras; trajectory and video counts are reported separately.

Figure A.3: Task-instruction distributions across datasets. Colored points highlight each dataset within the shared t-SNE projection.

Figure A.4: Action-description distributions across datasets, illustrating the broad atomic-action coverage of RoboAtom-Data.

Configuration	ViT-LoRA	visual.merger	mIoU	[email protected]	Skill [email protected]
① Language-only	Off	Frozen	44.3	28.7	21.7
② + merger	Off	Updated	50.0	28.7	21.4
③ + ViT-LoRA	On	Frozen	43.8	28.8	22.3
④ + ViT-LoRA + merger	On	Updated	48.7	29.5	22.7

Appendix

Table C.1: Ablation of the update scope of the visual modules. All configurations update the language-side LoRA for 500 steps on the same task-sampled RoboAtom subset.

Subset	Trajectories	Mean GT segments	Skill [email protected]
Short	200	2.06	71.1
Medium	100	6.34	43.7
Long	100	23.02	17.3
Multiview	100	6.71	33.9
Overall	500	8.04	30.1

Appendix

Table C.3: Skill [email protected] of RoboAtom-Anno-9B on each RoboAtom-Bench test subset.

Figure C.1: Boundary localization of Cortex and PhysExtend compared with ground truth (GT).

Figure C.2: Response of physical features at action boundaries versus within segments. (a) Relative evidence at the boundaries; (b) evidence of the same features within the segments (at the center).

Method	Spatial	Object	Goal	Long	Overall
$\pi_{0.5}$ (w/o aug)	97.00	99.00	97.50	95.00	97.125
$\pi_{0.5}$ (w/ aug)	98.80	98.20	98.00	92.40	96.85

Appendix

Table C.5(a): Image augmentation control of the

\pi_{0.5}

baseline on standard LIBERO (%).

Method	Spatial	Object	Goal	Long	Overall
$\pi_{0.5}$ (w/o aug)	93.01	85.23	79.93	78.56	84.05
$\pi_{0.5}$ (w/ aug)	91.63	89.36	79.08	79.48	84.77

Appendix

Table C.5(b): Image augmentation control of the

\pi_{0.5}

baseline on LIBERO-Plus (%).

Method	Clean	Random	Avg.
$\pi_{0.5}$ (w/ aug)	70.70	46.00	58.35
$\pi_{0.5}$ (w/o aug)	80.70	48.04	64.37

Appendix

Table C.6: Image augmentation control of the

\pi_{0.5}

baseline on RoboTwin (%).

Robot Manipulation Datasets

See all 58 papers

ATLAS: An Annotation Tool for Long-horizon Robotic Action Segmentation
Apr 29, 2026Sergej Stanovcic, Daniel Sliwowski, Dongheui LeeTemporal Action Segmentation Human-in-the-Loop Annotation
Autonomous Systems Lab, Technische Universität Wien (TU Wien), Vienna, Austria · Institute of Robotics and Mechatronics, German Aerospace Center (DLR), Wessling, Germany
ATOM-Bench: A Real-World Benchmark for Atomic Skills and Compositional Generalization in Manipulation Policies
Jun 15, 2026Zenan Wu, Bingqing Wei, Lu Liu +8Robot Skill Learning Language-Conditioned Robot Manipulation
1Beijing Academy of Artificial Intelligence · 2Peking University
Learning Semantic Atomic Skills for Multi-Task Robotic Manipulation
Dec 20, 2025Yihang Zhu, Weiqing Wang, Shijie Wu +2Long-Horizon Robotic Manipulation Robot Skill Learning
ShanghaiTech University, Shanghai, China · InstAdapt

Apr 29, 2026J/K move · Enter open · S save

ATLAS: An Annotation Tool for Long-horizon Robotic Action Segmentation

Sergej Stanovcic, Daniel Sliwowski, Dongheui Lee

Autonomous Systems Lab, Technische Universität Wien (TU Wien), Vienna, Austria · Institute of Robotics and Mechatronics, German Aerospace Center (DLR), Wessling, Germany

Temporal Action Segmentation Human-in-the-Loop Annotation

Open paper AskPDF arXiv

Annotating long-horizon robotic demonstrations with precise temporal action boundaries is crucial for training and evaluating action segmentation and manipulation policy learning methods. Existing annotation tools, however, are often limited: they are designed primarily for vision-only data, do not natively support synchronized visualization of robot-specific time-series signals (e.g., gripper state or force/torque), or require substantial effort to adapt to different dataset formats. In this paper, we introduce ATLAS, an annotation tool tailored for long-horizon robotic action segmentation. ATLAS provides time-synchronized visualization of multi-modal robotic data, including multi-view video and proprioceptive signals, and supports annotation of action boundaries, action labels, and task outcomes. The tool natively handles widely used robotics dataset formats such as ROS bags and the Reinforcement Learning Dataset (RLDS) format, and provides direct support for specific datasets such as REASSEMBLE. ATLAS can be easily extended to new formats via a modular dataset abstraction layer. Its keyboard-centric interface minimizes annotation effort and improves efficiency. In experiments on a contact-rich assembly task, ATLAS reduced the average per-action annotation time by at least 6% compared to ELAN, while the inclusion of time-series data improved temporal alignment with expert annotations by more than 2.8% and decreased boundary error fivefold compared to vision-only annotation tools.

Abstract

Robot Manipulation Datasets

Explore similar work

ATLAS: An Annotation Tool for Long-horizon Robotic Action Segmentation

ATOM-Bench: A Real-World Benchmark for Atomic Skills and Compositional Generalization in Manipulation Policies

Learning Semantic Atomic Skills for Multi-Task Robotic Manipulation