EgoSwarm: Learning Decentralized Robot Swarm Intelligence from Human Swarm Demonstrations
Organizations: State Key Laboratory of Intelligent Manufacturing Equipment and Technology Huazhong University of Science and Technology
Abstract
Advancing from individual embodied intelligence to swarm embodied intelligence is a key step toward scalable, general-purpose embodied intelligence. Complex tasks often require multiple agents to divide responsibilities, coordinate handovers, and manipulate shared objects collaboratively. However, existing robot learning methods rely primarily on individual demonstrations, which provide limited supervision for modeling dependencies among agent actions, while collecting multi-robot demonstrations at scale remains difficult. Human egocentric demonstrations offer a scalable alternative, yet distributed observations from multiple participants cannot directly supervise decentralized policies. We introduce EgoSwarm, the first robot learning framework for learning decentralized multi-robot collaboration from human swarm demonstrations. EgoSwarm collects more than 50 hours of collaborative experience using a robot-free multi-person egocentric system. We introduce the Swarm Interaction Packet (SIP) and Swarm Interaction Token (SIT) to recover cross-agent interaction relations from distributed observations and transform this experience into local supervision consistent with deployment-time information constraints. We further propose SwarmVLA, which transfers knowledge of swarm behavior to decentralized policies conditioned on local observations, local history, and limited neighbor messages. SwarmVLA achieves a mean success rate of 72.2% across three categories of multi-robot collaborative tasks. Project page:https://xitong-c.github.io/EgoSwarm_Page/.
Figures & tables
| Method | MRFH | DCT | LSDOM | Mean |
|---|---|---|---|---|
| Local-Rule | 20.0 | 26.7 | 10.0 | 18.9 |
| Independent VLA | 43.3 | 46.7 | 36.7 | 42.2 |
| Dec-VLA-Comm | 63.3 | 56.7 | 30.0 | 50.0 |
| Centralized Joint VLA | 80.0 | 50.0 | 46.7 | 58.9 |
| EgoSwarm | 80.0 | 73.3 | 63.3 | 72.2 |
| Variant / condition | MRFH | DCT | LSDOM | Mean |
|---|---|---|---|---|
| Robot-only | 33.3 | 20.0 | 13.3 | 22.2 |
| 50% human swarm data | 63.3 | 46.7 | 50.0 | 53.3 |
| w/o SIT | 46.7 | 46.7 | 23.3 | 38.9 |
| w/o quality gate | 56.7 | 36.7 | 43.3 | 45.6 |
| 30% loss + 100 ms delay | 76.7 | 70.0 | 53.3 | 66.7 |
| Full SwarmVLA | 80.0 | 73.3 | 63.3 | 72.2 |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Value |
|---|---|
| Random seed | 42 |
| GPU configuration | NVIDIA A100 (80 GB each) |
| Numerical precision | bfloat16 mixed precision |
| Per-GPU micro-batch size | 2 |
| Gradient accumulation steps | 8 |
| Effective global batch size | 64 |
| Task | Task-level prompt |
|---|---|
| MRFH | Work together to sort, hand over, and place garments into baskets. Coordinate each handover with the receiving partner’s readiness, and place all target garments into their corresponding baskets. |
| DCT | Work together to load the box onto the cart and transport it to the designated area. Two robots jointly load the box, and a third robot takes over transport once loading is complete. |
| LSDOM | Grasp and support the wire harness at different locations. Coordinate the motion of the grasp points and adjust your actions to its deformation to move and place the harness in the designated target configuration. |
| Task | Role | Role-specific prompt |
|---|---|---|
| MRFH | A: Pick and hand over | Pick up a garment from the initial area and hand it to B. Release it only after B has established a stable grasp, then prepare the next garment. |
| B: Receive and relay | Receive the garment from A. After A releases it, pass it to C. Release it only after C has established a stable grasp. | |
| C: Receive and place | Receive the garment from B. After B releases it, place it in the corresponding basket, then prepare to receive the next garment. | |
| DCT | A: Load from one side | Grasp one side of the box and coordinate with B to lift, move, and place it. Release your grasp only after the box rests stably on the cart. |
| B: Load from the other side | Grasp the opposite side of the box. Adjust your motion according to A’s actions and the box pose, and release your grasp after jointly completing the loading operation. | |
| C: Take over transport | Keep the cart stable during loading. Once the box is stably placed and A and B have cleared the transport path, move the cart to the target area. |
| Task | Failure Mode | Behavioral Manifestation | Observation Challenge |
|---|---|---|---|
| MRFH | Premature release | The transferring robot releases the garment before the receiver establishes a stable grasp. | Grasp, release, and object-possession relations. |
| Prolonged waiting | Robots remain in the handover region without advancing to the next task stage. | Partner readiness and temporal coordination. | |
| DCT | Interrupted joint support | Asymmetric lifting or premature release causes box tilting or slippage. | Bilateral end-effector motion and object pose. |
| Mistimed role transition | Cart movement begins prematurely or remains stalled after loading. | Temporal relation between loading completion and transport initiation. | |
| LSDOM | Grasp loss or local slippage | Changing grasp-point configurations cause local slippage during harness transport. | Contact regions and local deformation. |
| Stalled coordinated manipulation | Harness crossings, repeated corrections, or interrupted overall transfer. | Multi-point motion and global object configuration. |
Multi-Robot Coordination
Multi-robot coordination studies how teams of robots plan, communicate, allocate tasks, and act together safely and efficiently.
See all 356 papers