Thousand-GPU Large-Scale Training and Optimization Recipe for AI-Native Cloud Embodied Intelligence Infrastructure
Abstract
Embodied intelligence is a key step towards Artificial General Intelligence (AGI), yet its development faces multiple challenges including data, frameworks, infrastructure, and evaluation systems. To address these issues, we have, for the first time in the industry, launched a cloud-based, thousand-GPU distributed training platform for embodied intelligence, built upon the widely adopted LeRobot framework, and have systematically overcome bottlenecks across the entire pipeline. At the data layer, we have restructured the data pipeline to optimize the flow of embodied training data. In terms of training, for the GR00T-N1.5 model, utilizing thousand-GPU clusters and data at the scale of hundreds of millions, the single-round training time has been reduced from 15 hours to just 22 minutes, achieving a 40-fold speedup. At the model layer, by combining variable-length FlashAttention and Data Packing, we have moved from sample redundancy to sequence integration, resulting in a 188% speed increase; π-0.5 attention optimization has accelerated training by 165%; and FP8 quantization has delivered a 140% speedup. On the infrastructure side, relying on high-performance storage, a 3.2T RDMA network, and a Ray-driven elastic AI data lake, we have achieved deep synergy among data, storage, communication, and computation. We have also built an end-to-end evaluation system, creating a closed loop from training to simulation to assessment. This framework has already been fully validated on thousand-GPU clusters, laying a crucial technical foundation for the development and application of next-generation autonomous intelligent robots, and is expected to accelerate the arrival of the era of human-machine integration.
Figures & tables
| Strategy | Partitioned Object | Communication Pattern | Key Problem Solved | Scalability Limitation |
| DP | Training data | All-Reduce | Data throughput, compute acceleration | Per-device memory, global batch size |
| PP | Model layers | Point-to-point (adjacent stages) | Model depth (layers) | Pipeline bubbles, stage load balancing |
| TP | Intra-layer weight matrices | All-Reduce / All-Gather (within layer) | Model width (layer size) | Frequent intra-layer communication, bandwidth & latency sensitive |
| EP | Expert sub-networks | All-to-All (token routing) | Total parameter count | Load balancing, token-routing communication overhead |
| SP | Activation tensors (sequence dim) | All-Gather / Reduce-Scatter | Activation memory for long sequences | Communication overhead of sequence operations |
| No. | Task Name | Original | Optimized |
| 1 | pick up the black bowl between the plate and the ramekin and place it on the plate | 100% | 100% |
| 2 | pick up the black bowl next to the ramekin and place it on the plate | 100% | 100% |
| 3 | pick up the black bowl from table center and place it on the plate | 100% | 100% |
| 4 | pick up the black bowl on the cookie box and place it on the plate | 100% | 100% |
| 5 | pick up the black bowl in the top drawer of the wooden cabinet and place it on the plate | 94% (3 failures) | 92% (4 failures) |
| 6 | pick up the black bowl on the ramekin and place it on the plate | 98% (1 failure) | 100% |
| Configuration | LIBERO+ | LIBERO+GR00T N1.5 | ManiSkill+ | ||||||
| 8 GPUs | 16 GPUs | 32 GPUs | 8 GPUs | 16 GPUs | 32 GPUs | 8 GPUs | 16 GPUs | 32 GPUs | |
| Colocated | 289.23 | 547.55 | 703.85 | 371.80 | 680.46 | 1125.62 | 132.56 | 232.23 | 370.26 |
| Disaggregated (1:1) | 162.75 | 307.84 | 457.23 | 220.81 | 409.60 | 729.98 | 60.64 | 150.59 | 257.21 |
| Asynchronous improvements on Disaggregated (cumulative): | |||||||||
| + Train Async | 229.68 | 441.49 | 737.46 | 243.57 | 477.20 | 951.33 | 126.65 | 244.61 | 436.32 |
| + Rollout Async | 369.56 | 686.80 | 1041.36 | 434.20 | 816.48 | 1620.39 | 69.07 | 139.32 | 275.36 |
Distributed Training
Distributed training coordinates computation across multiple devices or machines to train models that exceed single-device limits.
See all 129 papers