Organizations: Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Fudan University · Tsinghua University · Zhejiang University
Fine-grained action annotations can support skill reuse and compositional generalization in robot policies. However, robot datasets typically provide task-level instructions without subtask annotations. Although some include subtask labels, inconsistent formats and conventions hinder cross-source joint training. We propose RoboAtom-Data, which adds unified, time-aligned atomic-action annotations to existing data from multiple sources, covering 1.45 million trajectories, 6.88 million skill-labeled segments, and 16,882.1 hours. Our pipeline combines a unified protocol, large-scale manual annotation, propagation across repeated executions with the same ordered skill sequence, and multi-stage quality control. We evaluate the dataset through automatic atomic-action annotation and robot control. We construct RoboAtom-Bench to standardize evaluation of skill recognition and temporal boundary localization. Trained on the annotated data, RoboAtom-Anno achieves better automatic annotation performance than all evaluated open- and closed-source models on both RoboAtom-Bench and the external WGO-Bench. For robot control, atomic-action supervision paired with hierarchical execution improves task success over task-level baselines on LIBERO-Long, LIBERO-Plus, and RoboTwin. We will release the data, benchmark, model, and construction pipeline to support further research on skill-level robot learning.
Figures & tables
Figure 1: Overview of RoboAtom: data and benchmark (top), atomic-action annotations (center), annotation performance (bottom left), and robot-control success rates (bottom right).
Figure 2: RoboAtom construction pipeline. Model-assisted annotation and human review produce seed annotations; PhysExtend and quality control scale them into RoboAtom-Data. Expert review establishes the evaluation annotations for RoboAtom-Bench (Section 4 ).
Dataset
Domain
Hours
Trajectories
Segments
Skills
Tasks
Embod.
AgiBot World Beta
Real
2,976.4
141,563 †
1,001,552
77 †
217
1
AgiBot World 2026
Real
591.6 †
28,331 †
327,355 †
43 †
37 †
1 †
Galaxea Open-World
Real
500.0
19,183 †
108,775 †
58
150
1
RoboInter-Data
Real
2,417.1 †
235,880
752,388
15
232
6
BEHAVIOR-1K 2026
Sim
1,952.9
20,000
406,341
35
100
1
RoboAtom-Data
Mixed
16,882.1 †
1,446,974 †
6,876,282 †
29
3,334 †
32 †
Table 1: Robot datasets with subtask annotations. † denotes statistics recomputed by us; source versions and counting rules are in Appendix A.2 .
Figure 3: Skill-aware boundary evidence and extension performance. (a) Gripper-state signals aligned with Pick/Place boundaries. (b) Base-motion signals aligned with Move boundaries. (c) Skill-feature associations. (d) PhysExtend consistently outperforms Cortex across reference counts; shaded bands indicate ± 1 standard deviation.
Table 2: Annotation extension performance and cost. Episode accuracy (EA) requires all subtasks in a trajectory to have correct semantics and segmentation, evaluated on 100 trajectories, one per task. Cost is in USD per source-video hour.
Figure 4: Scale and diversity of RoboAtom-Data. (a) Annotation word cloud. (b) Trajectory durations. (c) Segments per trajectory. (d) Dataset-scale comparison. (e–f) Task-instruction and action-description distributions.
Figure 5: (a–c) Illustrations of temporal mIoU/F1, Skill-F1, and Edit. Dark regions indicate unmatched predictions or edit operations. (d) Cumulative distributions of GT segment counts. (e) Ratios of total predicted to total GT segment counts across RoboAtom-Bench subsets.
Figure 6: Dual-system execution. System 2 uses a task-plan prior, visual observations, and execution history to select an atomic-action instruction. System 1 conditions π0.5 on that instruction to produce robot actions; subsequent observations inform the next planning decision.
Table 3: Annotation performance and inference cost (USD per source-video hour; Appendix D.1 ).
Method
LIBERO
LIBERO-Plus
RoboTwin
Long ↑
Spatial ↑
Object ↑
Goal ↑
Long ↑
Overall ↑
Clean ↑
Random ↑
Avg. ↑
π0.5
95.00
93.01
85.23
79.93
78.56
84.05
80.70
48.04
64.37
Our method
98.20
92.46
91.86
86.34
82.29
88.18
82.02
50.32
66.17
Table 4: Downstream success rates (%) on LIBERO, LIBERO-Plus, and RoboTwin. LIBERO-Plus Overall is the success rate over all 10,030 tasks; RoboTwin Avg. is the Clean/Random mean.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: Atomic-action annotation interface.
Figure A.2: Model-assisted review interface.
Dataset
Domain
Hours
Trajectories
Segments
Skills
Tasks
Embod.
Private
Real
54.7 †
3,187 †
20,088 †
10 †
7 †
2 †
ABC-130k
Real
3,585.6 †
130,740 †
2,353,804 †
17 †
198 †
1 †
EBench
Sim
182.9 †
6,600 †
42,459 †
12 †
26 †
1 †
Galaxea Open-World
Real
465.9 †
19,183 †
124,104 †
20 †
169 †
1 †
InternData-A1
Sim
7,073.8 †
621,633 †
1,740,880 †
13 †
1,148 †
1 †
LIBERO
Sim
3.9 †
1,712 †
3,910 †
8 †
40 †
1 †
Appendix
Table A.2: Per-source statistics of RoboAtom-Data. † : local or recomputed statistics. Private is one merged source; segments require non-empty skill labels. Skills counts distinct non-empty labels per source and is non-additive. Galaxea includes retained source labels as well as core skills; the total of 29 denotes the core vocabulary size. Embod. counts distinct non-empty robot_type strings, deduplicated across sources for the total.
Category
Samples
Training tasks
Locomotion
17,185
Navigation actions, movement direction, and target keypoint prediction
Manipulation
954
Grasp/release actions and manipulation keypoint prediction
Robot QA
1,007,450
Question answering on robot state, scene, and action
Grounding
1,080,004
Localization of 2D points, target boxes, affordance regions, and manipulation trajectories
Planning
326,747
Task decomposition, action sequence, and next-action prediction
Spatial reasoning
409,602
Reasoning about distance, direction, relative position, and 3D relations
Appendix
Table A.4: Task composition of robot-domain mid-training.
Test subset
Trajectories
Share
GT segments
Mean GT segments
Median segments
Segment range
Total frames
Duration (hours)
Videos
Short
200
40.0%
411
2.06
2
1–4
74,214
0.732
200
Medium
100
20.0%
634
6.34
6
5–8
126,762
1.193
100
Long
100
20.0%
2,302
23.02
21
9–62
298,015
2.761
100
Multiview
100
20.0%
671
6.71
5
1–48
108,105
1.119
321
Total
500
100.0%
4,018
8.04
5
1–62
607,096
5.806
721
Appendix
Table A.5: Test-set distribution of RoboAtom-Bench. Multi-view durations are not multiplied by the number of cameras; trajectory and video counts are reported separately.
Figure A.3: Task-instruction distributions across datasets. Colored points highlight each dataset within the shared t-SNE projection.
Figure A.4: Action-description distributions across datasets, illustrating the broad atomic-action coverage of RoboAtom-Data.
Table C.1: Ablation of the update scope of the visual modules. All configurations update the language-side LoRA for 500 steps on the same task-sampled RoboAtom subset.
Table C.3: Skill [email protected] of RoboAtom-Anno-9B on each RoboAtom-Bench test subset.
Figure C.1: Boundary localization of Cortex and PhysExtend compared with ground truth (GT).
Figure C.2: Response of physical features at action boundaries versus within segments. (a) Relative evidence at the boundaries; (b) evidence of the same features within the segments (at the center).
Method
Spatial
Object
Goal
Long
Overall
π0.5 (w/o aug)
97.00
99.00
97.50
95.00
97.125
π0.5 (w/ aug)
98.80
98.20
98.00
92.40
96.85
Appendix
Table C.5(a): Image augmentation control of the π0.5 baseline on standard LIBERO (%).
Method
Spatial
Object
Goal
Long
Overall
π0.5 (w/o aug)
93.01
85.23
79.93
78.56
84.05
π0.5 (w/ aug)
91.63
89.36
79.08
79.48
84.77
Appendix
Table C.5(b): Image augmentation control of the π0.5 baseline on LIBERO-Plus (%).
Method
Clean
Random
Avg.
π0.5 (w/ aug)
70.70
46.00
58.35
π0.5 (w/o aug)
80.70
48.04
64.37
Appendix
Table C.6: Image augmentation control of the π0.5 baseline on RoboTwin (%).
Annotating long-horizon robotic demonstrations with precise temporal action boundaries is crucial for training and evaluating action segmentation and manipulation policy learning methods. Existing annotation tools, however, are often limited: they are designed primarily for vision-only data, do not natively support synchronized visualization of robot-specific time-series signals (e.g., gripper state or force/torque), or require substantial effort to adapt to different dataset formats. In this paper, we introduce ATLAS, an annotation tool tailored for long-horizon robotic action segmentation. ATLAS provides time-synchronized visualization of multi-modal robotic data, including multi-view video and proprioceptive signals, and supports annotation of action boundaries, action labels, and task outcomes. The tool natively handles widely used robotics dataset formats such as ROS bags and the Reinforcement Learning Dataset (RLDS) format, and provides direct support for specific datasets such as REASSEMBLE. ATLAS can be easily extended to new formats via a modular dataset abstraction layer. Its keyboard-centric interface minimizes annotation effort and improves efficiency. In experiments on a contact-rich assembly task, ATLAS reduced the average per-action annotation time by at least 6% compared to ELAN, while the inclusion of time-series data improved temporal alignment with expert annotations by more than 2.8% and decreased boundary error fivefold compared to vision-only annotation tools.
Sergej Stanovcic, Daniel Sliwowski, Dongheui Lee
Autonomous Systems Lab, Technische Universität Wien (TU Wien), Vienna, Austria · Institute of Robotics and Mechatronics, German Aerospace Center (DLR), Wessling, Germany
Generalist manipulation policies are increasingly presented as foundation models for robotic control, but their real-world generalization remains difficult to diagnose. A policy may succeed on demonstrated tasks while still failing to execute fine-grained atomic skills or recombine learned skills in new task structures. We introduce ATOM-Bench, a real-world benchmark for evaluating both atomic skills and compositional generalization in manipulation policies. ATOM-Bench factorizes tabletop manipulation into motor atoms and instruction atoms, and contains 30 atomic tasks and 24 held-out compositional tasks across paired single-arm and dual-arm robot tracks. We collect 3,000 human demonstrations for atomic fine-tuning and release both the demonstration data and evaluation rollout data to support reproducible real-world evaluation. Policies are fine-tuned on atomic tasks and evaluated on both atomic skill acquisition and held-out compositional tasks. We further introduce Atomic Score (AS) and Compositional Failure Share (CFS) to distinguish failures caused by weak atomic skills from failures caused by limited compositional reuse. Through 2,700 physical rollouts on five representative manipulation policies, we find that current policies can acquire simple instruction-grounding skills, but still struggle with fine-grained motor atoms, counting, and logical filtering. More importantly, strong atomic performance does not reliably transfer to held-out compositional tasks. ATOM-Bench provides a diagnostic testbed for studying whether failures arise from weak motor execution, poor instruction grounding, or limited compositional reuse.
Zenan Wu, Bingqing Wei, Lu Liu +8
1Beijing Academy of Artificial Intelligence · 2Peking University
Scaling imitation learning to diverse multi-task robot manipulation remains challenging due to suboptimal demonstrations, behavioral multi-modality, and destructive interference across tasks. While skill-based methods offer a promising direction by decomposing behaviors into reusable abstractions, existing approaches often learn skills that are either biased toward linguistic structure or lack semantic alignment across tasks, limiting generalization. In this work, we propose AtomSkill, a novel framework that learns a semantically aligned Atomic Skill Space from demonstrations and enables robust long-horizon execution through keypose imagination. Our method introduces: (1) semantic contrastive skill alignment, which partitions demonstrations into variable-length atomic skills and employs a contrastive objective to jointly enforce semantic consistency and temporal coherence, yielding a compact and reusable skill library; and (2) action decoding with keypose imagining, where the policy predicts both a skill's terminal keypose and immediate actions, thereby supporting progress-aware skill transitions. During inference, an atomic skill diffusion sampler generates plausible skill sequences, while predicted keyposes autonomously trigger smooth skill chaining. Extensive experiments in simulation and real-world settings show that AtomSkill consistently outperforms state-of-the-art imitation learning and skill-based baselines. Project page: https://atom-skill.github.io.
Yihang Zhu, Weiqing Wang, Shijie Wu +2
ShanghaiTech University, Shanghai, China · InstAdapt