AtomicRL: Atomic Reinforcement Learning for Long-Horizon Robotic Manipulation

Likui Zhang; Zhe Zhu; Zhihao Zhan; Yibo Bai; Hanlin Wang; Dongwei Li; Bin Wang; Tianle Zhang

cs.ROOct 21, 2026

AtomicRL: Atomic Reinforcement Learning for Long-Horizon Robotic Manipulation

Authors: Likui Zhang, Zhe Zhu, Zhihao Zhan, Yibo Bai, Hanlin Wang, Dongwei Li, Bin Wang, Tianle Zhang,

Organizations: Sun Yat-sen University · JD Explore Academy · The Hong Kong Polytechnic University

Abstract

Vision-language-action (VLA) models have shown promise in robotic manipulation, but imitation learning (IL) alone often struggles to master complex long-horizon skills. Reinforcement learning (RL) offers an effective avenue for further improving VLA policies. However, existing VLA-RL methods typically suffer from delayed and biased credit assignment due to trajectory-level sparse rewards, while imbalanced subtask mastery creates bottlenecks that limit overall performance. To address these challenges, we propose AtomicRL, an RL post-training framework that leverages atomic task structures for reliable credit assignment and adaptive exploration. AtomicRL introduces Semantic-Aware Policy Optimization (SAPO), which uses semantic-aware gating to constrain TD residuals from failed atomic subtasks from propagating across task boundaries, mitigating credit contamination while preserving intra-subtask credit assignment and value bootstrapping. In addition, Bottleneck-Adaptive Target Exploration (BATE) dynamically coordinates task sampling and local exploration incentives based on the mastery of individual atomic subtasks, directing learning toward the remaining bottlenecks. Finally, we develop a Real2Sim2Real pipeline that bridges online RL in simulation with real-world deployment, facilitating practical transfer to physical robots. On LIBERO-LONG, AtomicRL achieves absolute performance gains of 3.6% and 4.0% over PPO across two VLA architectures. On CALVIN ABC-D, it improves full-task performance by up to 9.5% over PPO. Combined with the Real2Sim2Real pipeline, AtomicRL further improves real-world task performance by an average of 21.3%. These results demonstrate that AtomicRL effectively improves the learning efficiency and manipulation performance of VLA policies on long-horizon tasks.

Figures & tables

Figure 1: Overview of AtomicRL. Unlike previous RL methods that use standard GAE which may suffer from credit contamination, AtomicRL uses SAPO to promote more reliable credit assignment. Together with BATE and the Real2Sim2Real pipeline, AtomicRL achieves strong performance on long-horizon manipulation tasks in both simulation and the real world.

Figure 2: Overall pipeline of AtomicRL. (a) Progress–Completion Critic: PCC identifies atomic task boundaries and estimates atomic task progress from interaction histories. (b) Semantic-Aware Policy Optimization: SAPO uses these signals to regulate cross-subtask advantage propagation based on subtask execution outcomes. (c) Bottleneck-Adaptive Targeted Exploration: BATE adapts task sampling and local exploration according to subtask mastery, focusing training on insufficiently mastered bottleneck subtasks. (d) Real2Sim2Real workflow: A complementary Real2Sim2Real pipeline enables parallel simulation-based training and transfer to real-world robots.

Backbone	Method	SR (%) $\uparrow$	$\Delta_{\mathrm{PPO}}$ (pp)
$\pi_{0.5}$	SFT	89.0	–
	+ PPO	95.4	–
	+ AtomicRL (Ours)	99.0	+3.6
OpenVLA-OFT	SFT	86.5	–
	+ PPO	94.8	–
	+ AtomicRL (Ours)	98.8	+4.0

Table 1: Evaluation on LIBERO-LONG (%).

\Delta_{\mathrm{PPO}}

denotes the improvement over PPO.

Method	Task	Tasks Completed in a Row					Avg. Len $\uparrow$
Method	Task	1	2	3	4	5	Avg. Len $\uparrow$
$\pi_{0.5}$ SFT	ABC $\rightarrow$ D	92.8	83.5	73.2	61.5	45.3	3.56
+ PPO	ABC $\rightarrow$ D	99.2	95.7	89.8	81.2	70.6	4.36
+ AtomicRL (Ours)	ABC $\rightarrow$ D	99.4	97.0	92.6	87.3	80.1	4.56

Table 2: Long-horizon Robotic Manipulation Evaluation on CALVIN Benchmark (%).

Method	BP	WD	TC	Avg. SR $\uparrow$
$\pi_{0.5}$ SFT	60	52	32	48.0
+ PPO	68	64	52	61.3
+ AtomicRL	76	72	60	69.3

Table 3: Real-world robot results(%).

Figure 3: Sim2Real Correlation.

Panel of Figure 4 — Figure 4: Training curves on LIBERO-LONG and CALVIN ABC-D and Real-World(BP).

Figure 5: Demonstrations show the execution process of SFT and AtomicRL.

Reward	PPO	SAPO
0/1	78.2	87.8
SubR	80.2	89.4
PBSR	81.8	89.0
Ours	–	89.6

Table 4: Ablations on LIBERO-LONG. (a) Reward shaping with and without SAPO. (b) Component contributions. All results are success rates (%).

Category	Metric	Value
Prediction quality	Accuracy	95.92%
	False detection rate	4.08%
	Recall	89.20%
Efficiency	Trainable Parameters	28.8 M
Efficiency	Inference latency	42.4 ms

Table 5: PCC performance and efficiency.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

$\kappa$	Success Rate
0.0	87.2%
0.3	87.8%
0.5	85.0%
0.8	81.2%
1.0	78.2%

Appendix

Table 6: Effect of the SAPO gating coefficient.

Panel of Figure 6 — Figure 6: Training curves on LIBERO-LONG and CALVIN ABC-D.

Panel of Figure 7 — Figure 7: Critic explained variance.

Panel of Figure 8 — Figure 8: Training curves on Real-world robot.

Figure 9: Automated Real2Sim2Real workflow with essential tool calls and a real-to-simulation scene example.

Figure 10: Demonstrations show the execution process of SFT and AtomicRL.

RL for Robotics

Reinforcement learning for robotics trains robots to act through interaction, from locomotion and navigation to dexterous manipulation.

See all 252 papers

AtomicVLA: Unlocking the Potential of Atomic Skill Learning in Robots
Mar 8, 2026Likui Zhang, Tao Tang, Zhihao Zhan +9Robot Skill Learning VLMs for Robotics
Sun Yat-sen University · Yinwang Intelligent Technology Co. Ltd. · Peng Cheng Laboratory
AtomVLA: Scalable Post-Training for Robotic Manipulation via Predictive Latent World Models
Mar 9, 2026Xiaoquan Sun, Zetian Xu, Chen Cao +9Vision-Language-Action Models Robotic Manipulation
INFIFORCE Intelligent Technology Co., Ltd. Hangzhou, China · Huazhong University of Science and Technology, Wuhan, China · The University of Hong Kong, Hong Kong SAR, China +1
Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agents
Apr 15, 2026Angelo Moroncelli, Roberto Zanetti, Marco Maccarini +1RL for Robotics Data-Efficient Robot Learning
University of Applied Science and Arts of Southern Switzerland, Department of Innovative Technologies, IDSIA-SUPSI, Lugano, Switzerland · Università della Svizzera Italiana, Faculty of Informatics, Lugano, Switzerland · Politecnico di Milano, Mechanical Department, Milano, Italy

Mar 8, 2026J/K move · Enter open · S save

AtomicVLA: Unlocking the Potential of Atomic Skill Learning in Robots

Likui Zhang, Tao Tang, Zhihao Zhan, Xiuwei Chen+8

Sun Yat-sen University · Yinwang Intelligent Technology Co. Ltd. · Peng Cheng Laboratory

Robot Skill Learning VLMs for Robotics Long-Horizon Robotic Manipulation Continual Robot Learning Continual Learning for VLA Models Mixture of Experts

Open paper AskPDF arXiv

Recent advances in Visual-Language-Action (VLA) models have shown promising potential for robotic manipulation tasks. However, real-world robotic tasks often involve long-horizon, multi-step problem-solving and require generalization for continual skill acquisition, extending beyond single actions or skills. These challenges present significant barriers for existing VLA models, which use monolithic action decoders trained on aggregated data, resulting in poor scalability. To address these challenges, we propose AtomicVLA, a unified planning-and-execution framework that jointly generates task-level plans, atomic skill abstractions, and fine-grained actions. AtomicVLA constructs a scalable atomic skill library through a Skill-Guided Mixture-of-Experts (SG-MoE), where each expert specializes in mastering generic yet precise atomic skills. Furthermore, we introduce a flexible routing encoder that automatically assigns dedicated atomic experts to new skills, enabling continual learning. We validate our approach through extensive experiments. In simulation, AtomicVLA outperforms

π_{0}

by 2.4% on LIBERO, 10% on LIBERO-LONG, and outperforms

π_{0}

and

π_{0.5}

by 0.22 and 0.25 in average task length on CALVIN. Additionally, our AtomicVLA consistently surpasses baselines by 18.3% and 21% in real-world long-horizon tasks and continual learning. These results highlight the effectiveness of atomic skill abstraction and dynamic expert composition for long-horizon and lifelong robotic tasks. The project page is here.

Abstract

RL for Robotics

Explore similar work

AtomicVLA: Unlocking the Potential of Atomic Skill Learning in Robots

AtomVLA: Scalable Post-Training for Robotic Manipulation via Predictive Latent World Models

Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agents