Organizations: University of Washington · Stanford University · University of Southern California · National University of Singapore · University of North Carolina at Chapel Hill · Allen Institute for AI
Humans effortlessly react to the world; they adeptly return a tennis shot, catch a falling cup, step aside from a passing cart, or pass a box tossed during truck unloading. These everyday actions require anticipating a moving target and reflexively responding with the whole body within about a second. Such visual reactive whole-body control remains challenging for humanoids, which must (1) coordinate whole-body control, (2) react to object dynamics in real time, and (3) ground perception in egocentric vision. We study catching thrown boxes as a representative task that demands all three capabilities; existing systems cover only part of it, relying on end-effectors, motion capture, or marked objects. We present Reflex, a three-stage framework that learns these capabilities stage by stage while connecting them through a compact representation of object dynamics. First, a state-based policy learns whole-body catching with reinforcement learning. Second, the policy learns to infer the dynamics of a moving box from delayed and incomplete observations. Third, an egocentric RGB-D encoder learns to recover this dynamics representation directly from visual observations while keeping the whole-body controller fixed. This decomposition provides each capability with a direct training signal and makes visual learning substantially more scalable. In simulation, Reflex achieves 85.7% catch success from egocentric RGB-D, compared with 87.9% when given privileged box observations. Temporal visual history is critical: removing it reduces success by 35.6 and 39.8 points on centered and laterally offset throws. On a real Unitree G1, Reflex catches human-thrown boxes with 65% and 40% success at 2.5 and 5 m, reaching 76% and 59% of human catchers' success at the same distances.
Figures & tables
Fig. 2: Training pipeline and policy architecture. Stage 1 learns whole-body control by training the state-policy teacher with PPO. Stage 2 learns the object dynamics: it initializes the privileged encoder and decoder latent-input weights by matching teacher actions, then jointly refines the encoder and decoder with PPO. Human motion examples supply reference tracking and an AMP style reward. Stage 3 learns visual grounding by predicting the latent from RGB-D histories and camera ego-motion, using box-state supervision and DAgger rollouts. The pretrained CNN and heatmap head stay fixed during GRU training; the privileged encoder and decoder remain fixed throughout Stage 3. The decoder also receives robot/reference observations. Because Stage 3 learns only the latent predictor, the visual controller reuses the PPO-trained decoder without modification.
Fig. 3: Throw and appearance variations. Simulation examples show changes in release distance, box size, color, and landing location. The box-size panel illustrates measured real boxes; simulated weight and distance tests use the 34 × 26 × 22 cm training box. Color augmentation is applied when training the visual component. These variations define the operating conditions evaluated in Tab. I .
Box size (cm)
Box weight (kg)
Throw distance (m)
Box color
Controller
Rate (%)
Small 22 × 21 × 15
Regular 34 × 26 × 22
Large 46 × 41 × 32
Avg.
0.5
1.0
2.0
Avg.
2.5
5.0
7.0
Avg.
White
Grey
Red
Green
Blue
Black
Avg.
Privileged box-state input
Stage 1 / GT state
Catch
19 . 5
89 . 4
43 . 1
50 . 7
88 . 8
85 . 2
73 . 1
82 . 4
88 . 1
79 . 7
58 . 3
75 . 4
88 . 9
88 . 9
88 . 9
88 . 9
88 . 9
88 . 9
88 . 9
Intercept
54 . 2
93 . 8
79 . 4
75 . 8
95 . 5
95 . 5
93 . 0
94 . 7
93 . 3
88 . 0
65 . 3
82 . 2
93 . 8
93 . 8
93 . 8
93 . 8
93 . 8
93 . 8
93 . 8
Stage 2 / privileged z
Catch
21 . 6
88 . 4
47 . 8
52 . 6
93 . 4
92 . 5
84 . 7
90 . 2
90 . 9
79 . 7
41 . 6
70 . 7
88 . 3
88 . 3
88 . 3
88 . 3
88 . 3
88 . 3
88 . 3
Intercept
46 . 6
96 . 2
86 . 7
76 . 5
97 . 0
96 . 7
93 . 9
95 . 9
97 . 0
90 . 6
49 . 5
79 . 0
96 . 2
96 . 2
96 . 2
96 . 2
96 . 2
96 . 2
96 . 2
TABLE I: Simulation robustness. Catch and interception rates (%); one checkpoint per controller, 640 throws per condition (320 for the detector). With the same detector, visual latent control exceeds detection plus filtering within its visual training range.
Box size (cm)
Box weight (kg)
Throw distance (m)
Catcher
Small 22 × 21 × 15
Regular 31 × 26 × 21
Large 46 × 41 × 32
Avg.
0.5
1.0
2.0
Avg.
2.5
5.0
7.0
Avg.
Real G1 (ours)
20 . 0
65 . 0
45 . 0
43 . 3
45 . 0
60 . 0
30 . 0
45 . 0
65 . 0
40 . 0
15 . 0
40 . 0
Human reference
75 . 0
85 . 0
78 . 3
79 . 4
85 . 0
86 . 7
76 . 7
82 . 8
85 . 0
68 . 3
28 . 3
60 . 6
TABLE II: Real-robot catching. Catch rates (%) over 20 G1 trials per condition, varying one factor at a time with the others fixed at about 2.5 m and the regular box; 95% Wilson intervals span up to about ± 20 points (e.g., 0.5 vs. 1 kg: 9/20 vs. 12/20, Fisher p=0.53 ). The G1 reaches 76% and 59% of human success (three people, 60 trials per condition) at 2.5 and 5 m.
Catch rate (%), lateral offset (m)
Training objective
0.0
0.6
Init. only (no PPO)
75 . 3
44 . 2
No Stage 1 (scratch) ‡
6 . 7
0 . 2
Full objective
88 . 6 ± 4.8
76 . 1 ± 4.8
Task rewards removed
w/o success bonus
87 . 8 ± 1.6
71 . 8 ± 2.6
TABLE III: Reward ablation. Catch rates (%, mean ± s.d. over three training seeds) after PPO. Removing all three task groups drops success to 33.7% and 2.7%. ∗ Foot-height termination; no episode reaches the throw. ‡ Stage 2 recipe trained by PPO from random initialization, one seed, 1000 iterations.
Fig. 4: Catching across lateral landing offsets. Catch rates for the ground-truth state-policy reference, the YOLO detector with the state policy, our CNN detector with the latent decoder, privileged latent input, and visual latent input. Shading indicates the training-offset region and its tail. Each point pools 640 throws for one policy checkpoint (320 throws for the YOLO curve). Visual latent input exceeds the ground-truth state policy by about 12 points at 0.6 and 0.9 m, whereas YOLO detection lowers the state policy’s catch rate at every reported offset.
Latency (ms)
Hardware measurement
p50
p95
p99
Camera exposure → YOLO input
50 . 1
69 . 0
72 . 0
YOLO inference
8 . 2
12 . 9
14 . 9
Camera exposure → control tick
69 . 1
90 . 2
94 . 1
Control period (nominal 20 ms)
20 . 0
20 . 4
20 . 9
TABLE IV: Latency. Hardware timings (top) and delay sensitivity (bottom). Latency randomization helps at every delay.
Fig. 5: Real-robot catching across throw distances, box sizes, payloads, and lateral offsets. Successful trials at 2.5, 5, and 7 m and with small, regular, and large boxes (left, each at release, during approach, and after the catch), with 0.5, 1, and 2 kg payloads (right; weighing views show the payload mass), and with lateral offsets of 0.9 m right and 0.7 m left (bottom). Insets show onboard camera views; timestamps indicate elapsed time within each sequence. White outlines mark box positions and dashed curves indicate flight direction. The same visual latent controller succeeds at every tested distance, box size, payload, and lateral offset; Tab. II reports catch rates for distance, size, and payload, and Sec. V-F the lateral-offset rate.
Catch rate (%), lateral offset (m)
Visual encoder
0.0
0.6
Privileged latent z (reference)
87 . 9 ± 3.5
74 . 3 ± 4.5
Full visual encoder
85 . 2 ± 4.2
66 . 6 ± 8.9
Inputs removed
w/o camera ego-motion
74 . 8 ± 3.8
44 . 0 ± 3.2
w/o temporal history (single frame)
49 . 6 ± 4.9
26 . 8 ± 2.4
TABLE V: Visual-encoder ablation. Catch rates (%, mean ± s.d. over five policies). Removing temporal history costs the most.
Catch rate (%)
Box appearance
Without color augmentation
With color augmentation
Cardboard
87 . 0
87 . 8
White
78 . 0
87 . 0
Grey
78 . 8
88 . 9
Red
87 . 3
88 . 8
Green
83 . 3
87 . 8
TABLE VI: Color augmentation. Catch rates (%), decoder fixed. Augmentation raises blue and black boxes to 88.0 and 88.8%.
Catch rate (%), latent input
Latent initialization
Stage 2 / privileged z
Stage 3 / visual z
Action matching (ours)
87 . 9 ± 3.2
85 . 7 ± 3.4
Weight mapping
82 . 8 ± 3.2
80 . 5 ± 2.9
TABLE VII: Stage 2 initialization. Catch rates (%, mean ± s.d. over six controllers). Action matching averages 5 points higher.