HuG: Learning Humanoid Whole-Body Affordance-Aware Grasping with Active Perception

Xuanbin Peng; Lai Wei; Yuze Zhai; Qingyuan Yu; Keyi Wang; Jianglong Ye; Xiaolong Wang

cs.RONov 1, 2026

HuG: Learning Humanoid Whole-Body Affordance-Aware Grasping with Active Perception

Authors: Xuanbin Peng, Lai Wei, Yuze Zhai, Qingyuan Yu, Keyi Wang, Jianglong Ye, Xiaolong Wang

Organizations: UC San Diego

Abstract

Acquiring objects in human environments is hard for humanoids: targets are partially occluded under egocentric viewpoints, and reaching them demands coordinated whole-body motion that keeps the target visible and reachable while the base stays stable. We present HuG, a framework for whole-body affordance-aware grasping with active perception that couples perception and action through a Perceive-Align-Grasp pipeline: belief-driven active perception resolves occlusion by treating viewpoint as an explicit decision variable; a point-cloud-conditioned affordance policy proposes whole-body grasps; and an affordance-aware alignment stage repositions the body to the most executable configuration. A key insight is that, on a floating base, kinematic validity is not enough: we curate training data with controller-in-the-loop dynamics simulation, rejecting 73% of kinematically valid grasps that would destabilize the deployment controller, so the policy learns kinodynamically feasible affordances. Trained entirely in simulation with zero real demonstrations, HuG attains 87.5% real-world success on a multi-level shelf task, generalizes to novel objects, and holds at least 73% success at every tested height. Beyond generalizing within its training distribution, the policy extrapolates spatially: at scale in simulation it succeeds at table heights well below any seen during training, indicating that kinodynamically filtered synthetic data grounds affordances in object geometry and whole-body feasibility rather than in memorized configurations.

Figures & tables

Figure 1: Humanoid whole-body affordance-aware grasping. Top: our system runs zero-shot on a Unitree G1, coupling active perception with whole-body control to discover and grasp objects across diverse everyday scenes, from deep squats at low tables to full extension at high shelves. Bottom: the policy is trained entirely on large-scale synthetic data generated and filtered with kinodynamic feasibility.

Figure 2: System and training pipelines. The system pipeline (top) follows a Perceive–Align–Grasp structure, combining active perception, whole-body alignment, and robust grasp execution. The training pipeline (bottom) constructs large-scale synthetic data with kinodynamic filtering to learn affordance-aware whole-body grasping.

Panel of Figure 3 — Figure 3: Quantitative results. (a) Task 1 full-pipeline evaluation ( $N{=}40$ , 10 objects, 5 ID/5 OOD); Det. = object localized, Gr $|$ Det = grasp success given detection. (b) Task 2 spatial generalization across five heights (oracle perception, 10 objects; solid = ID, hatched = OOD, green = ACT/ViA demo heights): ours $\geq$ 73% everywhere, while ACT/ViA succeed only at their demonstrated heights (a demonstration-coverage effect; Sec. 4.3.2 ) and Dex1B/–Dyn. Filter degrade at the extremes. (c) Kinodynamic data filtering: 73% of kinematically valid grasps fail under the deployment whole-body controller (508 M SDF candidates $\to$ 18 K training grasps).

Panel of Figure 4 — Figure 4: Large-scale simulation evaluation ( ${\sim}$ 18,800 paired trials; Task 3 protocol in Sec. 4.1 ). (a) SR vs. table height on YCB (left) and RoboCasa (right). Training tables span 0.40–0.90 m (dashed line); the shaded band is extrapolation below the training range . (b) SR (%) aggregated over the extrapolation and in-range bands (95% Wilson CIs for the Full rows are within $\pm 4$ pts, e.g. YCB extrapolation $81.1\,[77.7,84.1]$ ). The full system loses only 6–8 pts below the training floor; removing any single component collapses extrapolation to 0–6% (–Align to ${\sim}$ 40–46%).

Figure 5: Whole-body spatial coverage. Adapting its entire body, our policy grasps in simulation across a continuous table-height range (0.10–0.90 m), from deep squats for low targets to full extension for high ones. Training tables spanned 0.40–0.90 m; the leftmost frames lie below the training range.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Parameter	Symbol	Value
Grasp Optimization (Simulated Annealing)
Iterations	Number of iterations	$N_{\mathrm{iter}}$	5,000
Batching	Grasp samples per scene	$N_{\mathrm{cand}}$	8,192
Contacts	Fingertip contact points	$N_{c}$	4
Energy	Contact-distance weight	$w_{\mathrm{dis}}$	100
Energy	Scene-penetration weight	$w_{\mathrm{pen}}$	100

Appendix

Table 1: Key hyperparameters for synthetic grasp generation (optimization and motion planning).

Module	Parameter	Symbol	Value
Policy (PointNet + CVAE)
Input	Policy input point cloud size	$N$	2,048
Action	Stages (pre/grasp/post)	$T$	3
Action	Action dimension	$D_{a}$	13
Action	Hand DoF in action	$D_{h}$	6
Encoder	PointNet feature dimension	$d_{f}$	512

Appendix

Table 2: Hyperparameters for policy learning, evaluation, and rendering.

Figure 6: Real-world rollouts, zero-shot from simulation. (a) Tabletop grasping of diverse objects; (b) long-reach grasping across a table; (c) deployment in varied unseen environments and heights (shelves, benches, tables).

Figure 7: Failure modes eliminated by our pipeline. (a) No dynamics filter: unstable jittering. (b) No WB co-optimization/alignment: twisted infeasible poses. (c) GraspNet: unnatural gripper-designed grasps. (d) No WB–hand co-optimization: hand–environment collisions.

Figure 8: Our full pipeline in fourteen real-world environments.

Figure 9: Data scaling (real world, oracle perception, 5 ID + 5 OOD objects; 120 trials per data scale, a separate protocol from Task 2). SR scales from 9.2% at 5k trajectories to

{\sim}

89% at 400k and plateaus thereafter, with OOD benefiting most.

Figure 10: Side-view success-rate field (Full pipeline): table height vs. forward reach $x$ , per catalog. The dashed line marks the 0.40 m training floor. Success stays high across the whole $0.10$ – $0.90\times 0.25$ – $0.55$ m envelope and degrades only at the far-reach column and the deepest squat row; per-cell $n$ varies with random placement (8–42), so the field should be read as a whole rather than cell-by-cell.

Figure 11: Top-view success-rate fields (plan view; robot at the bottom, grasping-hand side to the right): Full vs. –Dyn. Filter vs. –Compensator on both catalogs, aggregated over all heights. The Full envelope is broad and dense; removing the dynamics filter keeps coverage but at uniformly lower success density; removing the compensator degrades the whole field, worst at far cells.

Figure 12: Per-object success rate (Full pipeline,

n{=}136

trials/object, 95% Wilson CIs). The RoboCasa catalog’s broader shape distribution widens the outcome spread (51.5–93.4% vs. 67.7–94.1% for YCB); the hardest objects are thin and elongated (whisk, chicken drumstick, dish brush), whose narrow graspable surfaces demand centimeter-precise hand placement.

Robotic Grasping

Robotic grasping studies how robots perceive, plan, and execute stable, task-appropriate contacts with objects across varied shapes and conditions.

See all 324 papers

Human Universal Grasping
Jun 15, 2026Kevin Yuanbo Wu, Tianxing Zhou, Isaac Tu +5HOI Generation Robotic Grasping
New York University · Tsinghua University · University of Michigan
Reflex: Visually Grounded Reactive Humanoid Control
Oct 8, 2026Taoyang Jia, Weikai Huang, Linxin Song +6Robotic Catching Humanoid Robot Control
University of Washington · Stanford University · University of Southern California +3
Whole-Body Dexterity
Oct 15, 2026Conor Mc Gartoll, An Dang, Pranay Thangeda +5Whole-Body Robotic Manipulation Long-Horizon Robotic Manipulation
Amazon · University of California, San Diego · University of Michigan

Jun 15, 2026J/K move · Enter open · S save

Human Universal Grasping

Kevin Yuanbo Wu, Tianxing Zhou, Isaac Tu, Billy Yan+4

New York University · Tsinghua University · University of Michigan

HOI Generation Robotic Grasping Generalization in Robotic Manipulation

Open paper AskPDF arXiv

Humans can grasp objects effortlessly, whereas multi-fingered robots are far from this level of generality. We argue that the most natural source of robot grasping data is from humans, who pick up thousands of objects every day. We present HUG, a flow-matching model that generates diverse human grasps for any user-specified object in a single RGB-D image captured from a stereo camera. Using smart glasses, we first collect 1M-HUGs, an egocentric dataset of human grasps spanning 1M frames (27.8 hrs) and 6,707 object instances across 41 buildings. Next, to model the distribution of natural human grasps, our novel flow-matching model fuses RGB and depth observations to output a grasp parameterized by wrist translation, wrist rotation, and MANO hand pose. Predicted grasps can be retargeted to various robot hands, enabling zero-shot grasping in everyday scenes. To standardize evaluation, we build a new simulated benchmark, HUG-Bench, of 90 unseen objects from five geometric categories and various sizes, with metric-scale 3D meshes. We evaluate HUG in the real world on the 30-object test set of HUG-Bench across multiple stereo cameras, robot embodiments, and household environments. HUG outperforms the state-of-the-art grasping baselines by +23% and +34% on our challenging object set. Code, data, benchmark, checkpoints, and an interactive demo are released on our website: https://grasping.io/

Abstract

Robotic Grasping

Explore similar work

Human Universal Grasping

Reflex: Visually Grounded Reactive Humanoid Control

Whole-Body Dexterity