LoongForge-Embodied: A Unified Framework for Training Heterogeneous Embodied Models
Organizations: Baidu, Inc.
Abstract
Training Vision-Language-Action (VLA) models and World-Action Models (WAMs) is computationally expensive, while their heterogeneous architectures and fragmented training stacks make system optimizations difficult to reuse across models. To address these challenges, we present LoongForge-Embodied, a native-PyTorch framework that unifies and accelerates the training of heterogeneous embodied models. At its core, LoongForge-Embodied introduces a common abstraction layer that decouples model-specific implementations from shared training infrastructure, enabling cross-model reuse without constraining individual architectures or execution requirements. Buildingon this unified infrastructure, the framework combines reusable acceleration mechanisms with model-specific and hardware-aware optimization strategies across computation, execution, memory, data I/O, and distributed communication, including FP8-based communication optimizations for DDP and FSDP. Beyond training acceleration, LoongForge-Embodied integrates a unified data infrastructure with a pipeline for synthesizing robot data from egocentric data, and an efficient evaluation framework that supports concurrency and optimized computation in simulator. Experiments across representative VLA and WAM models demonstrate 1.47x-4.38x end-to-end training throughput improvements over their corresponding open-source baselines, as well as 4.1x-23.2x faster evaluation under the same protocols.
Figures & tables
| Model | Type | Dominant bottleneck | Key optimizations | Evaluated in |
|---|---|---|---|---|
| VLA | Activation memory, compilation granularity, gradient synchronization | FlexAttention, selective checkpointing, multi-group compilation, DDP bucket tuning | § 8.7 , App. A.1 | |
| GR00T N1.7 | VLA | Fragmented execution and host synchronization (78.2% of kernels s) | Metadata caching, operator fusion, CUDA Graph, cross-iteration pipelining | § 8.7 , App. A.2 |
| X-VLA | VLA | Host synchronization, fine-grained operators, data loading | Static-graph restructuring, torch.compile , data-loader tuning | § 8.7 , App. A.3 |
| Wall-OSS-0.5 | VLA | Unfused element-wise operators and kernel launches, exposed FSDP parameter communication | Bitwise-exact SwiGLU/RMSNorm kernels, execution-order prefetching, FSDP resharding | § 8.7 , App. A.4 |
| FastWAM | WAM | Bandwidth-limited gradient communication | ZeRO-1 DDP, FP8 AllToAll–AllGather, asynchronous overlap, fused kernels | § 8.8 , App. A.5 |
| LingBot-VA | WAM | Redundant FSDP parameter gathers, recomputed short kernels, rank load imbalance from variable-length samples | Accumulation-aware FSDP2 resharding, selective compilation, TE/Triton kernels, cost-balanced sampling | § 8.8 , App. A.6 |
| Metric | Unit | Dir. | What it measures |
| Kinematic feasibility | |||
| Velocity violation rate | % | Share of valid joint-velocity samples whose magnitude exceeds the reference limit of the target embodiment | |
| Acceleration violation rate | % | Share of valid joint-acceleration samples exceeding the reference limit, with accelerations differenced from the velocity samples | |
| Trajectory jerk P95 | unit/s 3 | 95th percentile of the unfiltered absolute third difference of the joint trajectory, aggregated across episodes | |
| IK rotation error | ∘ | Angular deviation between the achieved and the commanded end-effector orientation | |
| Visual quality (levels 0–2, 2 best) | |||
| Reference | Accelerated | Speedup | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Pool | Benchmark / model | Renderer | Episodes | E2E | Steady | E2E | Steady | E2E | Steady |
| Host CPU | LIBERO / FastWAM | OSMesa | 2,000 | 173.7 min | 170.5 min | 16.3 min | 13.2 min | 10.6 | 12.9 |
| GPU | LIBERO / FastWAM | EGL | 2,000 | 73.9 min | 70.8 min | 17.8 min | 14.7 min | 4.1 | 4.8 |
| GPU | LIBERO / | EGL | 2,000 | 80.2 min | 76.3 min | 16.3 min | 12.6 min | 4.9 | 6.1 |
| GPU | RoboTwin / | — | 5,000 | 41.03 h | 41.0 h | 1.77 h | 1.668 h | 23.2 | 24.6 |
| GPU | RoboTwin (clean) / FastWAM | — | 5,000 | 28.6 h | 28.6 h | 1.93 h | 1.84 h | 14.8 | 15.5 |
| Benchmark / model | Renderer | Reference | Accelerated | |
|---|---|---|---|---|
| LIBERO / FastWAM | OSMesa | 97.00 | 96.80 | |
| LIBERO / FastWAM | EGL | 96.75 | 97.40 | |
| LIBERO / | EGL | 98.60 | 98.40 | |
| RoboTwin / | — | 86.26 | 86.62 | |
| RoboTwin (clean) / FastWAM | — | 91.88 | 92.10 | |
| RoboTwin (random) / FastWAM | — | 91.78 | 92.12 |
| Version | Success rate (%) |
|---|---|
| Baseline (with clean data) | 75.76 |
| LoongForge (with Ego data) | 77.56 |
| Reference implementation | LoongForge-Embodied | ||||
| Metric | Value | 0/1/2 | Value | 0/1/2 | |
| Kinematic feasibility (lower is better) | |||||
| Velocity violation rate (%) | 3.938 | – | 3.886 | – | – |
| Acceleration violation rate (%) | 8.934 | – | 6.829 | – | – |
| Trajectory jerk P95 (unit/s 3 ) | 1370.9 | – | 213.5 | – | – |
| IK rotation error ( ∘ ) | 5.393 | – | 7.299 | – | – |
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Configuration | Thrpt. | Pearson | Mean err. |
|---|---|---|---|
| (samples/s/GPU) | (%) | ||
| BF16 | 24.1948 | – | – |
| FP8 (218 Linear) | 27.4301 | 0.99974017 | 0.5882 |
| FP8 (216 Linear) | 27.2991 | 0.99973545 | 0.5623 |
| Configuration | Samples/s/GPU | Speedup |
|---|---|---|
| Official Isaac-GR00T | 58.44 | 1.00 |
| + Data supply | 64.64 | 1.11 |
| + Backbone computation | 85.10 | 1.46 |
| + Training execution | 104.64 | 1.79 |
| Comparison | Measured scope | Reported check | Result |
|---|---|---|---|
| Official ZeRO-1 vs. LoongForge (deterministic settings) | Video loss; 300 steps 8 ranks | Maximum absolute difference | (bitwise equal) |
| Official ZeRO-1 vs. LoongForge (deterministic settings) | Action loss; 300 steps 8 ranks | Maximum absolute difference | (bitwise equal) |
| Official ZeRO-1 vs. LoongForge (deterministic settings) | 1,649 trainable parameter tensors at steps 1, 2, and 20 | torch.equal for all tensors | All equal |
| BF16 DDP vs. FP8 communication | Training loss comparison | Reported convergence alignment | Aligned; no raw trajectory provided |
| Configuration | Step (s) | MARE vs A0 (%) |
|---|---|---|
| Reference A0 | 0.980355 | 0.0000 |
| Delta-FP8 disabled | 0.740985 | 0.1686 |
| Delta-FP8 enabled | 0.640407 | 0.0616 |
| Optimization | Measured metric | Before | After |
|---|---|---|---|
| GPU preprocessing | get_batch share of iteration | 45% | 5% |
| torch.compile | Supported batch size | 2 | 4 |
| torch.compile | Cross-platform throughput ratio † | 0.90 | 1.13 |
| Selective FSDP sharding | Cross-platform throughput ratio † | 1.50 | 1.66 |
| FP8 computation | Normalized compute performance | 1.00 | 1.26 |
| Pool / model | Episodes/min | |
|---|---|---|
| Host CPU / FastWAM | 16 | 70.7 |
| 24 | 77.8 | |
| 48 | 81.8 | |
| 80 | 81.9 | |
| GPU / | 8 | 13.47 |
| 10 | 14.05 |