cs.LGMar 19, 2026
SaveAcceRL: A Distributed Asynchronous Reinforcement Learning and World Model Framework for Vision-Language-Action Models
Organizations: IROOTECH TECHNOLOGY Wolf 1069 b Lab, Sany Group
Abstract
Reinforcement learning (RL) for large-scale Vision-Language-Action (VLA) models is severely bottlenecked by synchronization barriers and the high cost of environment data acquisition. To overcome these challenges, we propose AcceRL, a distributed asynchronous RL framework that physically isolates environment rollouts, model inference, and gradient updates. By eliminating the cascading long-tail idle bubbles inherent in synchronous systems, AcceRL maximizes hardware utilization and ensures scalable throughput. Furthermore, AcceRL features a modular design that supports the integration of diverse, plug-and-play world models into its distributed pipeline. Extensive experiments demonstrate that the base framework achieves highly competitive performance across all four LIBERO~\cite{liu2023libero} task suites. Systematically, the asynchronous architecture delivers a throughput speedup over leading synchronous baselines. Algorithmically, by leveraging a world model pre-trained on 1,000 offline trajectories, AcceRL achieves up to a improvement in online sample efficiency on LIBERO-Spatial, establishing a robust framework that is both sample-efficient and time-efficient for embodied AI. Code is included in the supplementary material. Code is available at https://github.com/distanceLu/AcceRL.
Figures & tables
Figure 1: Timelines and GPU bubbles across three levels of synchronous RL framework (left) and the asynchronous framework AcceRL (right). The synchronous framework suffers from long-tail effects at the step, episode, and cluster levels, resulting in severe forced waiting bubbles (red hatched areas). By introducing a dual asynchronous mechanism, AcceRL substantially reduces synchronization barriers and long-tail idle periods, enabling more efficient computation.
Figure 2: The pipeline of AcceRL base backbone (left) and world-model-augmented mode (right). The framework isolates training, inference, and rollout into independent asynchronous streams. By integrating a plug-and-play world model, AcceRL bypasses physical simulation constraints through imagination rollouts to significantly enhance sample efficiency.
Figure 3: Scalability performance of the AcceRL framework with decoupled data labels. (a) Throughput scaling demonstrates near-linear performance up to 64 Rollout Workers. (b) Trainer scalability shows that the actual trainer SPS closely tracks the ideal marginal scaling curve when scaling up to 7 GPUs. Both configurations effectively maintain high hardware utilization without suffering from severe communication overhead.
| Framework | Peak Trainer | Throughput (SPS) | Throughput (SPS) | Est. Time to |
|---|---|---|---|---|
| GPU Util. | @ 90% Succ. | @ 99% Succ. | 2K Steps | |
| SimpleVLA (VeRL) | 30–40% | 21.4 | 28.5 | 16.7 hrs |
| RLinf | 50–60% | 25.2 | 35.6 | 15.8 hrs |
| AcceRL (Ours) | 94–95% | 42.4 | 42.4 | 6.5 hrs |
Table 1: Comparison of system efficiency and training throughput on 4 H200 GPUs. Under the same 4 H200 hardware setting, AcceRL achieves higher trainer GPU utilization and sampling throughput than the compared baselines, substantially reducing the estimated wall-clock time required to reach 2K training steps.
| Method | Spatial (%) | Object (%) | Goal (%) | Long (%) |
|---|---|---|---|---|
| Ours (AcceRL) | 99.6 | 100.0 | 98.8 | 99.1 |
| SimpleVLA-RL [ 33 ] | 99.4 | 99.8 | 99.2 | 98.5 |
| RLinf-VLA [ 19 ] | 99.4 | 99.8 | 98.8 | 94.0 |
| OpenVLA-OFT | 96.2 | 98.3 | 96.2 | 90.7 |
Table 2: Task performance comparison on the LIBERO benchmark. AcceRL achieves superior success rates across most categories. Notably, it attains a 99.1% success rate on Long tasks, demonstrating stability in long-horizon scenarios where supervised fine-tuning typically falters.
Figure 4: (a) On ManiSkill PickCube, AcceRL reaches a 90% peak success rate, overcoming the early plateaus of supervised baselines. (b) On LIBERO-Spatial, AcceRL-WM demonstrates substantially higher sample efficiency than the model-free AcceRL baseline by reaching a comparable average return with far fewer real environment interactions, corresponding to an approximately 200 reduction in required environment steps.(c) Replacing the DIAMOND world model with Cosmos, AcceRL-Cosmos successfully completes the closed-loop imagined rollout and policy-update pipeline, suggesting that AcceRL can accommodate alternative world-model backends with limited engineering changes.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Overview of AcceRL . (Top Left) The framework’s throughput exhibits super-linear scaling with the number of trainer GPUs, enabled by ZeRO [ 28 ] optimizations. (Bottom Left) Experimental results on the LIBERO benchmark demonstrate that AcceRL achieves highly competitive performance across all evaluation categories, attaining the best results among the compared methods on Spatial, Object, and Long tasks. (Middle) A conceptual illustration of the world model in reinforcement learning, representing the agent “learning in imagination” to significantly boost data utilization. (Right) Performance comparison between model-based (AcceRL with WM) and model-free (AcceRL) approaches. By leveraging a world model pre-trained on 1,000 offline trajectories, the model-based architecture improves online sample efficiency by .
Figure 6: Timeline of a synchronous RL framework (left) and the asynchronous framework AcceRL (right) . The inference forward blocks in AcceRL encompass both the batching wait time and the inference time. By eliminating the GPU bubbles, AcceRL maximizes hardware utilization and significantly enhances overall training efficiency.
| Category | Value |
|---|---|
| Environment & Task | |
| Simulator | ManiSkill (Task: PickCube-v1) |
| Observation Space | RGB Camera ( base_camera , 128 128) |
| Max Episode Steps | 100 |
| Model Architecture | |
| VLA Backbone | OpenVLA-OFT (based on Llama-2-7B) |
Table 3: Detailed Experimental Settings and Hyperparameters for AcceRL on ManiSkill tasks.
| Category | Value |
| Environment & Task | |
| Simulator / Benchmark | LIBERO |
| Observation Space | RGB Image ( num_images_in_input:2 ) |
| Model Architecture | |
| VLA Backbone | OpenVLA-OFT |
| Cosmos WM Actions | 16 |
Table 4: Detailed Experimental Settings and Hyperparameters for OpenVLA-OFT RL with Cosmos WM on LIBERO Object tasks.
| Category | Value |
|---|---|
| Environment & Task | |
| Simulator | LIBERO |
| Observation Space | RGB Image ( num_images_in_input:1 ) |
| Model Architecture | |
| VLA Backbone | OpenVLA-OFT (based on Llama-2-7B) |
| Action Vocabulary Size | 256 bins (Discrete Actions) |
Table 5: Detailed Experimental Settings and Hyperparameters for AcceRL WM on LIBERO tasks.
| Category | Value |
|---|---|
| Environment & Task | |
| Benchmark | LIBERO |
| Observation Space | RGB images + proprioceptive state |
| Number of Images in Input | 2 |
| Model Architecture | |
| VLA Backbone | OpenVLA-OFT |
Table 6: Detailed Experimental Settings and Hyperparameters for OpenVLA RL Training.
| GPUs | Throughput (SPS) | GPU Util. (%) |
|---|---|---|
| 1 | 14.13 | 96.45 |
| 2 | 28.82 | 97.17 |
| 3 | 42.42 | 94.22 |
| 4 | 60.33 | 98.36 |
| 5 | 75.95 | 96.72 |
| 6 | 90.78 | 96.63 |
Table 7: Scaling performance and hardware utilization of AcceRL. The data illustrates a super-linear increase in training throughput (SPS) while maintaining consistently high GPU utilization ( ) at different scale points.
Figure 7: Ablation study of the AcceRL framework regarding the value recomputation mechanism. The comparison highlights the critical role of recomputing values in maintaining training stability. AcceRL with Revalue (dark blue) effectively handles data staleness in low-rollout scenarios, while AcceRL without Revalue (light blue) exhibits significant performance drops and higher variance due to misaligned value targets.
Figure 8: Ablation study of the AcceRL framework regarding the GIPO algorithm. The curves highlight the stability of GIPO (dark blue) compared to standard PPO (light blue). GIPO maintains a robust, high-performance trajectory, whereas PPO exhibited oscillatory behavior and inferior performance due to policy lag.
| Method | Sync Latency | Infer Update Delay | Sample Policy Lag |
|---|---|---|---|
| NCCL with drain | 70–95 ms | 70–95 ms | 0.03–0.05 |
| NCCL without drain | 220–300 ms | 220–300 ms | 0.1–0.2 |
| PCIe / Host-mediated | 2–7 s | 3–12 s | 0.3–1.5 |
| Shared-storage / AReal-style | 12–53 s | 17–76 s | 2–12 |
Table 8: Comparison of synchronization overheads across different methods. By employing NCCL with an early drain strategy, our system significantly reduces synchronization latency, infer update delay, and sample policy lag compared to traditional host-mediated or shared-storage approaches.
| Rank | Algorithm | IQM | Mean Norm. |
|---|---|---|---|
| 1 | GIPO(0.2, 1.0) | 0.724 | 0.651 |
| 2 | PPO | 0.595 | 0.565 |
| 3 | GIPO(0.5, 1.0) | 0.578 | 0.582 |
| 4 | GIPO(1.0, 1.0) | 0.446 | 0.481 |
| 5 | SAPO | 0.315 | 0.366 |
Table 9: Aggregate Performance Scores. Evaluated across 196 independent runs in 10 MuJoCo environments. GIPO(0.2, 1.0) achieves a substantial lead in both IQM (robustness) and Mean Normalized score (overall average).