Catching arbitrary objects requires generalization to novel projectiles and real-time adaptation to unpredictable contact dynamics under noisy visual observations. This paper presents Catch It All!, a sim-to-real catching framework for humanoid robots that generalizes and adapts to unseen objects with variable physical properties for underhand and overhand catches, using only one egocentric camera. We learn policies which generalize to unseen objects with varying geometry, deformability, elasticity, and physical properties. Effective grasps which adapt real-time to object poses and shapes are learned through careful grasp reward design. We provide safety-aware curricula which enable safe, high-frequency action execution near hardware limits, including wrist torque limits as low as 5 Nm. We validate our system by deploying zero-shot on a Unitree G1. With only one egocentric camera, Catch It All! successfully performs underhand and overhand catches on 9 unseen objects not seen during training, including low-rebound plushes, semi-deformable toys and chip bags, and rigid-elastic objects with high rebound on contact.
Figures & tables
Fig. 2: Teacher-student training. Stage 1: An RL teacher policy is trained on ground-truth projectile position. Stage 2: A student policy imitates the frozen teacher via DAgger using noisy camera observations and a 50 -step (1 second) history.
Fig. 3: Qualitative Results. Examples of successful catches across 4 of the 9 real-world objects: Takis, turtle, strawberry, and sandbag. Top, overhand: policy catches short tosses by moving the hand forward, while it catches long tosses by moving the hand backwards mid-capture. Bottom, underhand: the hand tracks the object during the toss to intercept and grasp it.
Fig. 4: Simulation objects. Top: training uses only primitive objects with simple meshes to increase throughput. Middle: hold-out set of irregular objects with novel geometries, unseen during training. Bottom: hold-out set of long-axis objects with high angular velocity, unseen during training.
Fig. 5: Curricula in RL training. Penalty and reward curricula progression during RL training.
Fig. 6: Real-world tossing objects. Real-world projectiles unseen in training; we exert 1 pound of force to show variations in deformability, geometry, and physical properties.
Fig. 7: Robustness and generalization. We test generalization to new object geometries and robustness to mass, volume, and restitution randomization. We evaluate underhand and overhand variants of the MPC, RL-only, and teacher-student distillation methods. In RL, primitive objects are seen during training, while irregular and long-axis objects are novel and unseen during training. 95% CIs are shown via error bars and shaded regions; some CIs may be too small to see. Our method is highlighted blue in the legend. Best viewed zoomed.
Method
Sim SR (%) ↑
Sim-to-Sim SR (%) ↑
Underhand
Overhand
Underhand
Overhand
Ground Truth Object Position Obs
MPC
3.7 ± 0.2
1.3 ± 0.1
0.1 ± 0.4
0.7 ± 0.6
RL G.T. Obs w/o Torque Clip Curriculum
75.2 ± 0.5
51.4 ± 0.5
43.8 ± 2.7
8.8 ± 1.7
RL G.T. Obs (ours, teacher)
79.5 ± 0.4
60.0 ± 0.5
81.6 ± 2.2
22.1 ± 2.4
Realistic Object Position Obs (Noisy, Dropped Outside Cam Frame)
TABLE I: Sim and sim-to-sim evaluations. Success rate (%) of all policy types, 95% CIs. Highest success rate per column is bolded; second-highest is underlined; our method is highlighted blue.
Fig. 8: Joint mechanical power. We plot mechanical power summed across joints, averaged over 32,768 episodes per policy. 95% CIs are shown via shaded regions; some CIs may be too small to see. Our method is highlighted blue in the legend. Best viewed zoomed.
Fig. 9: Qualitative Grasp Examples. Examples of successful underhand and overhand grasps across different objects from real-world catching trials. Grasps adjust to object geometry and pose in-hand, yielding secure grasps.