Advancing from individual embodied intelligence to swarm embodied intelligence is a key step toward scalable, general-purpose embodied intelligence. Complex tasks often require multiple agents to divide responsibilities, coordinate handovers, and manipulate shared objects collaboratively. However, existing robot learning methods rely primarily on individual demonstrations, which provide limited supervision for modeling dependencies among agent actions, while collecting multi-robot demonstrations at scale remains difficult. Human egocentric demonstrations offer a scalable alternative, yet distributed observations from multiple participants cannot directly supervise decentralized policies. We introduce EgoSwarm, the first robot learning framework for learning decentralized multi-robot collaboration from human swarm demonstrations. EgoSwarm collects more than 50 hours of collaborative experience using a robot-free multi-person egocentric system. We introduce the Swarm Interaction Packet (SIP) and Swarm Interaction Token (SIT) to recover cross-agent interaction relations from distributed observations and transform this experience into local supervision consistent with deployment-time information constraints. We further propose SwarmVLA, which transfers knowledge of swarm behavior to decentralized policies conditioned on local observations, local history, and limited neighbor messages. SwarmVLA achieves a mean success rate of 72.2% across three categories of multi-robot collaborative tasks. Project page:https://xitong-c.github.io/EgoSwarm_Page/.
Figures & tables
Figure 1: Overview of EgoSwarm. EgoSwarm is the first robot learning framework for decentralized multi-robot collaboration from human swarm demonstrations. It reconstructs cross-agent interaction in the form of the Swarm Interaction Packet (SIP), encodes deployment-available interaction information into the Swarm Interaction Token (SIT), and learns locally executable collaboration policies through SwarmVLA.
Figure 2: EgoSwarm Capture and shared human–robot sensing interface. Human participants wear SwarmEgo sensing heads to record synchronized multi-person egocentric demonstrations. Each sensing head integrates task and scene cameras, infrared cameras, an IMU, and UWB sensing. The robot platform reuses the same sensing interface and UWB infrastructure, reducing the perception gap between human demonstrations and decentralized robot deployment.
Figure 3: EgoSwarm pipeline for decentralized multi-robot collaboration. (a) Multimodal egocentric recordings are temporally aligned and reconstructed into quality-annotated Swarm Interaction Packets (SIPs). (b) A causal CTDE teacher learns coordination representations and action flow fields from collective interaction supervision, which are distilled into the decentralized student policy. (c) During deployment, each robot constructs local Swarm Interaction Tokens (SITs), aggregates quality-aware neighbor information, and generates future actions from the resulting Action Context.
Figure 4: Composition of the SwarmVLA training data. The dataset consists of three complementary sources for decentralized multi-robot learning: human swarm data for cross-agent interaction experience, robot adaptation data for embodiment alignment, and joint robot teleoperation data for task-specific collaborative action supervision.
Figure 5: Real-world multi-robot collaborative task execution. The figure illustrates representative execution sequences of the three evaluation tasks: Multi-Robot Flexible Handover (MRFH, top), Dynamic Collaborative Transport (DCT, middle), and Large-Scale Deformable Object Manipulation (LSDOM, bottom). Frames progress from left to right; red arrows indicate robot motion directions, and dashed circles highlight key interaction regions.
Method
MRFH
DCT
LSDOM
Mean
Local-Rule
20.0
26.7
10.0
18.9
Independent VLA
43.3
46.7
36.7
42.2
Dec-VLA-Comm
63.3
56.7
30.0
50.0
Centralized Joint VLA
80.0
50.0
46.7
58.9
EgoSwarm
80.0
73.3
63.3
72.2
Table 1: Success rates (%) across three swarm manipulation tasks.
Variant / condition
MRFH
DCT
LSDOM
Mean
Robot-only
33.3
20.0
13.3
22.2
50% human swarm data
63.3
46.7
50.0
53.3
w/o SIT
46.7
46.7
23.3
38.9
w/o quality gate
56.7
36.7
43.3
45.6
30% loss + 100 ms delay
76.7
70.0
53.3
66.7
Full SwarmVLA
80.0
73.3
63.3
72.2
Table 2: Success rates (%) for ablation and robustness tests.
Figure 6: Evaluation of SIP relative-geometry reconstruction. (a) Optical motion capture provides reference poses for ground-truth evaluation. (b) Translation and rotation error distributions over 30 independent multi-person interaction segments. SIP achieves a translation RMSE of 4.8 mm and a rotation RMSE of 2.6 ∘ .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: Multi-person data collection setup and SwarmEgo sensing hardware. (a) The multi-person capture environment is equipped with distributed UWB anchors for inter-agent ranging, with each participant wearing an identical SwarmEgo sensing head. (b) The SwarmEgo device integrates RGB cameras, infrared cameras, an IMU, and UWB sensing to capture multimodal egocentric observations during natural human collaboration.
Figure A2: Multi-robot hardware platform. Each robot integrates two 7-DoF manipulators, grippers, an omnidirectional mobile base, and a vertical lifting mechanism with a 300 mm stroke. An identical EgoSwarm Head provides a shared sensing interface with human demonstrations. An NVIDIA AGX Thor and an STM32F407IGH6-based controller support onboard computation and low-level actuation, respectively, enabling decentralized robot execution.
Parameter
Value
Random seed
42
GPU configuration
4× NVIDIA A100 (80 GB each)
Numerical precision
bfloat16 mixed precision
Per-GPU micro-batch size
2
Gradient accumulation steps
8
Effective global batch size
64
Appendix
Table B1: Training hyperparameters and optimization settings for SwarmVLA. The learning-rate schedule is applied separately in each stage.
Task
Task-level prompt
MRFH
Work together to sort, hand over, and place garments into baskets. Coordinate each handover with the receiving partner’s readiness, and place all target garments into their corresponding baskets.
DCT
Work together to load the box onto the cart and transport it to the designated area. Two robots jointly load the box, and a third robot takes over transport once loading is complete.
LSDOM
Grasp and support the wire harness at different locations. Coordinate the motion of the grasp points and adjust your actions to its deformation to move and place the harness in the designated target configuration.
Appendix
Table B2: Example task-level prompts specifying shared objectives and completion conditions.
Task
Role
Role-specific prompt
MRFH
A: Pick and hand over
Pick up a garment from the initial area and hand it to B. Release it only after B has established a stable grasp, then prepare the next garment.
B: Receive and relay
Receive the garment from A. After A releases it, pass it to C. Release it only after C has established a stable grasp.
C: Receive and place
Receive the garment from B. After B releases it, place it in the corresponding basket, then prepare to receive the next garment.
DCT
A: Load from one side
Grasp one side of the box and coordinate with B to lift, move, and place it. Release your grasp only after the box rests stably on the cart.
B: Load from the other side
Grasp the opposite side of the box. Adjust your motion according to A’s actions and the box pose, and release your grasp after jointly completing the loading operation.
C: Take over transport
Keep the cart stable during loading. Once the box is stably placed and A and B have cleared the transport path, move the cart to the target area.
Appendix
Table B3: Example role-specific prompts for the three collaborative tasks. A, B, and C denote the participating robots.
Task
Failure Mode
Behavioral Manifestation
Observation Challenge
MRFH
Premature release
The transferring robot releases the garment before the receiver establishes a stable grasp.
Grasp, release, and object-possession relations.
Prolonged waiting
Robots remain in the handover region without advancing to the next task stage.
Partner readiness and temporal coordination.
DCT
Interrupted joint support
Asymmetric lifting or premature release causes box tilting or slippage.
Bilateral end-effector motion and object pose.
Mistimed role transition
Cart movement begins prematurely or remains stalled after loading.
Temporal relation between loading completion and transport initiation.
LSDOM
Grasp loss or local slippage
Changing grasp-point configurations cause local slippage during harness transport.
Contact regions and local deformation.
Stalled coordinated manipulation
Harness crossings, repeated corrections, or interrupted overall transfer.
Multi-point motion and global object configuration.
Appendix
Table C1: Failure modes and observation challenges in multi-robot collaboration. Representative coordination failures across the three evaluation tasks and the interaction information required to identify them.
School of Computer Science, The University of Sydney, Australia. · College of Computer Science and Technology, Zhejiang University of Technology, China.