Organizations: University of Washington · Stanford University · University of Southern California · National University of Singapore · University of North Carolina at Chapel Hill · Allen Institute for AI
Humans effortlessly react to the world; they adeptly return a tennis shot, catch a falling cup, step aside from a passing cart, or pass a box tossed during truck unloading. These everyday actions require anticipating a moving target and reflexively responding with the whole body within about a second. Such visual reactive whole-body control remains challenging for humanoids, which must (1) coordinate whole-body control, (2) react to object dynamics in real time, and (3) ground perception in egocentric vision. We study catching thrown boxes as a representative task that demands all three capabilities; existing systems cover only part of it, relying on end-effectors, motion capture, or marked objects. We present Reflex, a three-stage framework that learns these capabilities stage by stage while connecting them through a compact representation of object dynamics. First, a state-based policy learns whole-body catching with reinforcement learning. Second, the policy learns to infer the dynamics of a moving box from delayed and incomplete observations. Third, an egocentric RGB-D encoder learns to recover this dynamics representation directly from visual observations while keeping the whole-body controller fixed. This decomposition provides each capability with a direct training signal and makes visual learning substantially more scalable. In simulation, Reflex achieves 85.7% catch success from egocentric RGB-D, compared with 87.9% when given privileged box observations. Temporal visual history is critical: removing it reduces success by 35.6 and 39.8 points on centered and laterally offset throws. On a real Unitree G1, Reflex catches human-thrown boxes with 65% and 40% success at 2.5 and 5 m, reaching 76% and 59% of human catchers' success at the same distances.
Figures & tables
Fig. 2: Training pipeline and policy architecture. Stage 1 learns whole-body control by training the state-policy teacher with PPO. Stage 2 learns the object dynamics: it initializes the privileged encoder and decoder latent-input weights by matching teacher actions, then jointly refines the encoder and decoder with PPO. Human motion examples supply reference tracking and an AMP style reward. Stage 3 learns visual grounding by predicting the latent from RGB-D histories and camera ego-motion, using box-state supervision and DAgger rollouts. The pretrained CNN and heatmap head stay fixed during GRU training; the privileged encoder and decoder remain fixed throughout Stage 3. The decoder also receives robot/reference observations. Because Stage 3 learns only the latent predictor, the visual controller reuses the PPO-trained decoder without modification.
Fig. 3: Throw and appearance variations. Simulation examples show changes in release distance, box size, color, and landing location. The box-size panel illustrates measured real boxes; simulated weight and distance tests use the 34 × 26 × 22 cm training box. Color augmentation is applied when training the visual component. These variations define the operating conditions evaluated in Tab. I .
Box size (cm)
Box weight (kg)
Throw distance (m)
Box color
Controller
Rate (%)
Small 22 × 21 × 15
Regular 34 × 26 × 22
Large 46 × 41 × 32
Avg.
0.5
1.0
2.0
Avg.
2.5
5.0
7.0
Avg.
White
Grey
Red
Green
Blue
Black
Avg.
Privileged box-state input
Stage 1 / GT state
Catch
19 . 5
89 . 4
43 . 1
50 . 7
88 . 8
85 . 2
73 . 1
82 . 4
88 . 1
79 . 7
58 . 3
75 . 4
88 . 9
88 . 9
88 . 9
88 . 9
88 . 9
88 . 9
88 . 9
Intercept
54 . 2
93 . 8
79 . 4
75 . 8
95 . 5
95 . 5
93 . 0
94 . 7
93 . 3
88 . 0
65 . 3
82 . 2
93 . 8
93 . 8
93 . 8
93 . 8
93 . 8
93 . 8
93 . 8
Stage 2 / privileged z
Catch
21 . 6
88 . 4
47 . 8
52 . 6
93 . 4
92 . 5
84 . 7
90 . 2
90 . 9
79 . 7
41 . 6
70 . 7
88 . 3
88 . 3
88 . 3
88 . 3
88 . 3
88 . 3
88 . 3
Intercept
46 . 6
96 . 2
86 . 7
76 . 5
97 . 0
96 . 7
93 . 9
95 . 9
97 . 0
90 . 6
49 . 5
79 . 0
96 . 2
96 . 2
96 . 2
96 . 2
96 . 2
96 . 2
96 . 2
TABLE I: Simulation robustness. Catch and interception rates (%); one checkpoint per controller, 640 throws per condition (320 for the detector). With the same detector, visual latent control exceeds detection plus filtering within its visual training range.
Box size (cm)
Box weight (kg)
Throw distance (m)
Catcher
Small 22 × 21 × 15
Regular 31 × 26 × 21
Large 46 × 41 × 32
Avg.
0.5
1.0
2.0
Avg.
2.5
5.0
7.0
Avg.
Real G1 (ours)
20 . 0
65 . 0
45 . 0
43 . 3
45 . 0
60 . 0
30 . 0
45 . 0
65 . 0
40 . 0
15 . 0
40 . 0
Human reference
75 . 0
85 . 0
78 . 3
79 . 4
85 . 0
86 . 7
76 . 7
82 . 8
85 . 0
68 . 3
28 . 3
60 . 6
TABLE II: Real-robot catching. Catch rates (%) over 20 G1 trials per condition, varying one factor at a time with the others fixed at about 2.5 m and the regular box; 95% Wilson intervals span up to about ± 20 points (e.g., 0.5 vs. 1 kg: 9/20 vs. 12/20, Fisher p=0.53 ). The G1 reaches 76% and 59% of human success (three people, 60 trials per condition) at 2.5 and 5 m.
Catch rate (%), lateral offset (m)
Training objective
0.0
0.6
Init. only (no PPO)
75 . 3
44 . 2
No Stage 1 (scratch) ‡
6 . 7
0 . 2
Full objective
88 . 6 ± 4.8
76 . 1 ± 4.8
Task rewards removed
w/o success bonus
87 . 8 ± 1.6
71 . 8 ± 2.6
TABLE III: Reward ablation. Catch rates (%, mean ± s.d. over three training seeds) after PPO. Removing all three task groups drops success to 33.7% and 2.7%. ∗ Foot-height termination; no episode reaches the throw. ‡ Stage 2 recipe trained by PPO from random initialization, one seed, 1000 iterations.
Fig. 4: Catching across lateral landing offsets. Catch rates for the ground-truth state-policy reference, the YOLO detector with the state policy, our CNN detector with the latent decoder, privileged latent input, and visual latent input. Shading indicates the training-offset region and its tail. Each point pools 640 throws for one policy checkpoint (320 throws for the YOLO curve). Visual latent input exceeds the ground-truth state policy by about 12 points at 0.6 and 0.9 m, whereas YOLO detection lowers the state policy’s catch rate at every reported offset.
Latency (ms)
Hardware measurement
p50
p95
p99
Camera exposure → YOLO input
50 . 1
69 . 0
72 . 0
YOLO inference
8 . 2
12 . 9
14 . 9
Camera exposure → control tick
69 . 1
90 . 2
94 . 1
Control period (nominal 20 ms)
20 . 0
20 . 4
20 . 9
TABLE IV: Latency. Hardware timings (top) and delay sensitivity (bottom). Latency randomization helps at every delay.
Fig. 5: Real-robot catching across throw distances, box sizes, payloads, and lateral offsets. Successful trials at 2.5, 5, and 7 m and with small, regular, and large boxes (left, each at release, during approach, and after the catch), with 0.5, 1, and 2 kg payloads (right; weighing views show the payload mass), and with lateral offsets of 0.9 m right and 0.7 m left (bottom). Insets show onboard camera views; timestamps indicate elapsed time within each sequence. White outlines mark box positions and dashed curves indicate flight direction. The same visual latent controller succeeds at every tested distance, box size, payload, and lateral offset; Tab. II reports catch rates for distance, size, and payload, and Sec. V-F the lateral-offset rate.
Catch rate (%), lateral offset (m)
Visual encoder
0.0
0.6
Privileged latent z (reference)
87 . 9 ± 3.5
74 . 3 ± 4.5
Full visual encoder
85 . 2 ± 4.2
66 . 6 ± 8.9
Inputs removed
w/o camera ego-motion
74 . 8 ± 3.8
44 . 0 ± 3.2
w/o temporal history (single frame)
49 . 6 ± 4.9
26 . 8 ± 2.4
TABLE V: Visual-encoder ablation. Catch rates (%, mean ± s.d. over five policies). Removing temporal history costs the most.
Catch rate (%)
Box appearance
Without color augmentation
With color augmentation
Cardboard
87 . 0
87 . 8
White
78 . 0
87 . 0
Grey
78 . 8
88 . 9
Red
87 . 3
88 . 8
Green
83 . 3
87 . 8
TABLE VI: Color augmentation. Catch rates (%), decoder fixed. Augmentation raises blue and black boxes to 88.0 and 88.8%.
Catch rate (%), latent input
Latent initialization
Stage 2 / privileged z
Stage 3 / visual z
Action matching (ours)
87 . 9 ± 3.2
85 . 7 ± 3.4
Weight mapping
82 . 8 ± 3.2
80 . 5 ± 2.9
TABLE VII: Stage 2 initialization. Catch rates (%, mean ± s.d. over six controllers). Action matching averages 5 points higher.
Visual loco-manipulation of arbitrary in-the-wild objects requires accurate end-effector (EE) control and a generalizable understanding of the scene from visual inputs (eg, RGB-D images). Existing imitation and sim2real methods jointly learn both these aspects via monolithic end-to-end learning and are thus hard to scale. In this work, we bring to bear the best tools for each of these problems -- large vision models for generalizable scene understanding and simulated training for accurate EE control -- leading to an overall modular loco-manipulation system that exhibits strong generalization. Our core technical innovation is HERO, an accurate residual-aware EE tracking policy made possible by combining classical robotics with machine learning. It uses a) inverse kinematics to convert residual end-effector targets into reference trajectories, b) a learned neural forward model for accurate forward kinematics, and c) goal adjustment and replanning. Together, these innovations reduce the end-effector tracking error to 2.44cm, outperforming the strongest prior method by 5.5x. Our overall system operates in diverse real-world environments, from offices to coffee shops, where the robot reliably grasps various everyday objects (eg, mugs, apples, toys) on surfaces ranging from 43cm to 92cm in height. Systematic modular and end-to-end tests demonstrate the effectiveness of our proposed design. We believe our advances open up new ways of training humanoids to interact with daily objects. Project page: https://hero-humanoid.github.io/
Current humanoid dexterous manipulation remains largely confined to end-effectors and simple grasps, falling short of humans' ability to engage the entire body by bracing and sliding bulky objects against the arms and torso. We introduce Whole-Body Dexterity (WBD), a general framework that uses the robot's body and appendages as contact surfaces to grasp and manipulate objects. Our sim-to-real pipeline first generates a dataset of feasible whole-body grasps, then trains reinforcement-learning experts on a general any-to-any repositioning objective by sampling initial and goal grasps from the dataset, and distills these experts into a single policy that uses proprioception and object geometry. We show that the distilled policy transfers zero-shot to a real Unitree G1, where it repositions a basketball and a cardboard box to regions around the torso and completes a long-horizon task that stows a box under one arm, carries it from a shelf to a table, and places it. The controller operates interactively, steered by coarse motion references from a human teleoperator. Across real-world whole-body manipulation tasks, the distilled controller completes 90% of trials, compared with 18% for direct retargeting and 4% for a generalist whole-body controller.
Conor Mc Gartoll, An Dang, Pranay Thangeda +5
Amazon · University of California, San Diego · University of Michigan
In this work we study if a robotic hand using proprioception alone can grasp diverse objects with no visual observation. We present a modular dexterous grasping architecture that separates global arm motion from local contact control. An independently controlled arm guides the hand toward the object, while a reinforcement learning policy grasps and stabilizes it using only hand proprioceptive feedback. We call this a blind grasp reflex: grasping without images, object poses, or geometric observations. A learned stable-grasp score determines when the object is securely held, allowing the arm to begin post-grasp manipulation. This separation makes grasping a reusable hand-level skill that can be combined with independently designed arm controllers for various manipulation tasks. Experiments in simulation and on hardware demonstrate robust blind grasping across diverse objects and seamless composition with a range of arm controllers. Moreover, despite never observing contact geometry, the learned grasp score closely aligns with an independent physics-based measure of grasp stability. The resulting approach follows a simple principle: see to reach, feel to grasp. Project page: https://blindgraspreflex.github.io.
Alexander Alexiev, Tzu-Yuan Lin, Sang Min Kim +3
Department of Mechanical Engineering, Massachusetts Institute of Technology