RoboMoCap: A Large-Scale Whole-Body Motion Capture Dataset with Egocentric Vision for Robot Learning
Organizations: MAGIC Lab, National University of Singapore · BitRobot · Grounded Superintelligence
Abstract
Humanoid robots must coordinate locomotion and manipulation to navigate complex environments and perform everyday tasks. Learning such behaviours from human demonstrations requires accurate whole-body motion paired with egocentric vision, particularly for lower-body movements and foot placements that are frequently invisible from a first-person perspective. However, existing egocentric datasets often lack high-fidelity optical motion capture or systematic coverage of diverse loco-manipulation behaviours. To address this gap, we introduce RoboMoCap, a large-scale multimodal dataset combining egocentric vision with marker-based optical motion capture, specifically designed for locomotion-centric humanoid learning. RoboMoCap comprises over 200 hours of recordings from 14 participants, featuring six head-mounted cameras, wrist-mounted visual and inertial sensing, and optical full-body motion capture at 120 Hz. The dataset systematically covers diverse locomotion primitives, dynamic whole-body movements, and loco-manipulation tasks, including variable-speed walking, turning, constrained navigation, terrain traversal, object transportation, and manipulation during locomotion. It provides temporally aligned multimodal observations and metric whole-body motion representations, enabling the study of human movement beyond directly observable body regions. We systematically evaluate existing methods on three benchmark tasks: egocentric full-body motion reconstruction, foot contact and placement estimation, and human-to-humanoid motion retargeting. By pairing egocentric vision with high-fidelity human motion across diverse locomotion and interaction scenarios, RoboMoCap provides a foundation for egocentric motion understanding and humanoid learning from human demonstrations.
Figures & tables
| Dataset | Year | Hours | #Subj | Body GT | #Ego | Body-fac. | #Wrist | #Exo | IMU | Hand | Obj. 6D | Scene | Audio | Loco | L-M |
| Motion capture and pseudo-labelled motion | |||||||||||||||
| AMASS [ 38 ] | 2019 | 40+ | 344 | Opt | 0 | ✗ | ✗ | ✗ | ✗ | (✓) | ✗ | ✗ | ✗ | ✓ | (✓) |
| LAFAN1 [ 17 ] | 2020 | 4.6 | 5 | Opt | 0 | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ |
| GRAB [ 51 ] | 2020 | 1.6M fr | 10 | Opt | 0 | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ |
| 100STYLE [ 40 ] | 2022 | 22.1 | 1 | Opt | 0 | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ |
| OMOMO [ 29 ] | 2023 | 10 | 17 | Opt | 0 | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | (✓) | ✓ |
| Action category | # | Representative tasks | Task variations |
|---|---|---|---|
| Locomotion primitives | 44 | Variable-speed walking, turns and figure-eights, backward and sideways walking, narrow passages, ramps and stairs | Speed, acceleration, travel direction, path curvature, stance width, obstacles and terrain |
| Locomotion posture and dynamics | 60 | Sit-to-stand, squatting, kneeling, lying and rising, jumping, mimed sports and combat, dance, running and cutting | Posture, support configuration, movement intensity, in-place vs. travelling, gait transitions |
| Loco-manipulation | 86 | Height-paired transport, varied-load carries, carrying over terrain and stairs, manipulation while walking, bin loading, precision placement, deformable loads | Object weight and shape, carry position, pick and place height, route and terrain, concurrent manipulation |
| Static manipulation | 44 | Stationary pick-and-place, drawer, lid and latch operation, folding and pouring, tool use, reach-envelope mapping | Reach height and direction, articulation type, uni- vs. bimanual coordination, object rigidity |
| Long-horizon composite | 11 | Delivery shelving, order picking, restocking, palletising, sorting, kit assembly and room tidying | Task order, object selection, goal specification, start-state and resource perturbations |
| Dynamic and recovery | 15 | Recovery stepping, trips, surface changes, reduced friction, load perturbations, controlled falls and rising | Perturbation source and direction, support surface, carried load and recovery strategy |
| Joint Position | Rotation | Temporal Motion | Head | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Method | MPJPE (mm) | PA-MPJPE (mm) | W-MPJPE (mm) | WA-MPJPE (mm) | MPJRE ( ∘ ) | MPJVE (cm/s) | Accel. Error (cm/s 2 ) | Jitter (m/s 3 ) | Position (mm) | Orientation ( ∘ ) |
| AGRoL [ 12 ] | 82.3 | 51.2 | 95.4 | 64.9 | 28.2 | 15.1 | 285.7 | 119.31 | 30.4 | 23.0 |
| EgoPoser [ 24 ] | 88.8 | 56.1 | 105.8 | 71.4 | 28.2 | 17.1 | 251.5 | 96.48 | 30.4 | 23.0 |
| EgoEgo [ 28 ] | 130.9 | 97.2 | 224.6 | 118.8 | 33.5 | 27.5 | 426.7 | 157.73 | 138.2 | 22.9 |
| EgoAllo [ 58 ] | 80.8 | 69.2 | 103.9 | 79.3 | 30.3 | 14.5 | 214.5 | 72.06 | 74.8 | 22.9 |
| Matched Key Bodies | Feet | Hands/Wrists | Orientation | ||||
|---|---|---|---|---|---|---|---|
| Method | (mm) | (mm) | Global (mm) | Root-rel. (mm) | Global (mm) | Root-rel. (mm) | Geodesic (rad) |
| GMR [ 4 ] | 90.7 | 78.6 | 84.8 | 87.7 | 131.5 | 121.7 | 0.385 |
| PHC [ 33 ] | 173.9 | 64.9 | 160.6 | 62.7 | 186.1 | 88.8 | 0.435 |
| OmniRetarget [ 57 ] | 34.8 | 36.3 | 49.5 | 61.8 | 29.3 | 28.8 | 0.470 |
| Foot Skating | Ground Penetration | Floating | Contact F1 | Constraints | Success | ||||
|---|---|---|---|---|---|---|---|---|---|
| Method | Rate (%) | p99 speed (m/s) | Frames (%) | Max depth (cm) | Mean gap (cm) | Foot (%) | Joint-limit (%) | Self-coll. (%) | Clips (%) |
| GMR [ 4 ] | 12.3 | 0.36 | 72.5 | 6.8 | 1.5 | 83.0 | 87.2 | 10.7 | 7.4 |
| PHC [ 33 ] | 18.5 | 0.56 | 20.9 | 37.1 | 1.4 | 80.4 | 82.9 | 1.1 | 79.6 |
| OmniRetarget [ 57 ] | 2.2 | 0.26 | 0.0 | 0.2 | 0.1 | 87.7 | 71.8 | 10.0 | 53.6 |
| Contact | Foot placement | Plausibility | |||
|---|---|---|---|---|---|
| Method | F1 (%) | Footprint (mm) | Step (mm) | Skate (mm/s) | Penetr. (mm) |
| EgoEgo [ 28 ] | 74.3 | 220.1 | 234.7 | 291.1 | 36.0 |
| EgoAllo [ 58 ] | 90.5 | 85.1 | 154.1 | 54.8 | 8.5 |
| EgoPoser [ 24 ] | 68.1 | 140.3 | 266.3 | 314.4 | 64.1 |
| AGRoL [ 12 ] | 75.3 | 113.8 | 237.0 | 273.7 | 66.8 |
| GT SMPL-X | 93.6 | – | – | 43.6 | 0.0 |