Perceptive Humanoid Motion from Simulation
Organizations: Apple
Abstract
Humanoid robots have the potential to automate a wide range of physical tasks. Progress toward generalist humanoid autonomy, however, remains constrained by a data bottleneck: mapping pixels and free-form language to low-level joint commands requires paired supervision, and no corpus of such data exists at sufficiently large scale. Some approaches rely on human teleoperation and wearable capture devices, which require at least an hour of human time to produce an hour of data. We present HALO, a humanoid autonomy system powered by a pipeline that synthesizes vision-language-motion data across a thousand procedurally generated homes. The corpus scales with compute rather than human labor. HALO acquires a repertoire of physical skills, including sitting on chairs, navigating to named objects, stepping sideways through half-open doors, crouching and crawling beneath tables, and pushing boxes. It orchestrates these skills to solve composite, long-horizon tasks. Without ever seeing a real home during training, HALO transfers zero-shot to the physical world, achieving a 91% success rate across 360 trials in six unmapped real-world environments. See https://apple-aiml-research.github.io/ml-halo/ for code and videos.
Figures & tables
| ZeroWBC | VLK | GRAIL | TANGO | HALO | |
| [ 20 ] | [ 15 ] | [ 18 ] | [ 11 ] | (ours) | |
| Free-form language | ✓ | ✓ | ✓ | ||
| Object interaction | ✓ | ✓ | ✓ | ✓ | |
| Multi-step tasks | ✓ | ||||
| Perceptive tracker | ✓ | ||||
| Training envs. | n/a | 2 | 2 | 578 | 1,000 |
| Success rate (%) | |||
| Category | Task | Sim. | Real |
| Single-Task | Go-To | 98 | 92 |
| Sit | 84 | 85 | |
| Door | 91 | 93 | |
| Crouch | 95 | 100 | |
| Crawl | 84 | 90 | |
| Stage | Configuration | Success rate (%) | |
| Training scenes | 1,000 | 84 | |
| 100 | 81 | ||
| 10 | 64 | ||
| 1 | 43 | ||
| Real-time chunking | Training-time | 84 | |
| Inference-time | 66 |
| Skill | Motion | Keyframes from | Filter | Appearance | Episodes | Hours | Mean length |
| Sit | walk (best of ), turn, sit | G1 joint log | checks, keep best | image randomization | s | ||
| Go-To | one walk to an object | standing pose | target in view at start | random dome light | s | ||
| Door | one call, KF | hand-edited poses | none | image randomization | s | ||
| Crawl | one call, KF | phone video, GVHMR, GMR | checks, best of | random dome light | s | ||
| Crouch | one call, KF | phone video, GVHMR, GMR | checks, best of | random dome light | s | ||
| Push | walk (best of ) recorded push | motion capture | gait ranking | image randomization | s |
| Group | Parameter | Value |
| Encoder | backbone | Qwen3.5-2B (frozen) |
| image resolution | ||
| feature layer | last | |
| conditioning tokens | image all text ( ) | |
| Decoder | architecture | DiT [ 37 ] |
| blocks / width / heads | / / |
| Parameter | Phase 1 | Phase 2 | Phase 3 |
| Network architecture | |||
| History encoder (1D CNN) | |||
| Spatial input | height | depth | depth |
| Spatial encoder (2D CNN) | |||
| SimBa MLP [ 49 ] ( ) | |||
| Reference lookahead | ( s) | ( s) | ( s) |
| Reward Term | Expression | Weight |
| Motion tracking [ 53 ] | ||
| Pelvis position | ||
| Pelvis orientation | ||
| Link positions | ||
| Link orientations | ||
| Link linear velocities | ||
| Parameter | Phases 1–2 | Phase 3 |
| Actuators and robot inertia [ 53 , 44 ] | ||
| PD stiffness / damping scale | / | / |
| Motor armature scale | ||
| Joint torque limit scale | ||
| Default joint offset (rad) | ||
| Command latency (ms) | ||
| Stage | Latency |
| Motion generator, alone | |
| Motion generator, co-served with orchestrator transition | |
| Orchestrator transition, alone | |
| Orchestrator transition, co-served with generator |
| Subtask | Generator prompt |
| go_to_chair | walk to the chair |
| go_to_box | walk to the box |
| go_to_human | walk to the person |
| follow | walk to the person |
| sit | sit down on the chair |
| crouch_under | crouch under the table |
Humanoid Robotics
Humanoid robotics develops human-shaped robots that move, manipulate objects, and interact with people in human environments.
See all 163 papers