GRAZE: Text-Free Grasp Synthesis for Unseen Objects from Human Gaze
Organizations: Dankook University · University of Birmingham · Gachon University
Abstract
Text-conditioned hand-object interaction (HOI) generation provides intuitive semantic control, but language is an indirect way to specify where an interaction should occur on an object. Naming the part to grasp requires object-specific words, and naming the approach side requires a reference frame that a prompt lacks. We propose GRAZE (GRAsp from gaZE), which generates hand-object grasp motion from recorded gaze instead of object and part names. Its Geometry-Aware Gaze Encoding (GeoGaze) maps gaze rays to cues on object geometry, and cross-attention conditions a diffusion Transformer on these cues. We also introduce HOT3D-Grasp, a grasp-centered benchmark with object-disjoint folds built on HOT3D recordings. On unseen objects, GRAZE grasps the target part more often than the strongest text-conditioned baseline (49.3% vs. 43.6%), achieves the highest grasp success (66.3% vs. 46.6%), and transfers without fine-tuning to GazeHOI (45.3% vs. 33.8%). Gaze conveys the approach side: when gaze is rotated around the object, the approach turns with it in 59.1% of cases, against 20.5% for a text model retrained with clock words. On the same backbone, a text prompt names parts better, but gaze adds the approach side. Given both, GRAZE keeps the prompt's part accuracy and follows gaze.
Figures & tables
| Success | Part grounding | Physics | ||||
| Method | GSR | Part | PCP | ID | Sim | SimD |
| Text2HOI | 35.3 | 36.8 | 36.6 | 12.0 | 44.0 | 5.09 |
| DiffH2O | 17.9 | 31.7 | 31.0 | 0 8.9 | 27.8 | 7.09 |
| LatentHOI | 46.6 | 43.6 | 42.5 | 0 8.4 | 40.7 | 5.54 |
| OpenHOI | 25.8 | 31.5 | 31.0 | 11.9 | 31.7 | 6.16 |
| GRAZE (Ours) | 66.3 | 49.3 | 50.1 | 0 6.7 | 74.5 | 2.04 |
| Method | GSR | Sim | SimD | Contact | ID |
|---|---|---|---|---|---|
| Text2HOI | 18.4 | 27.1 | 7.50 | 46.8 | 11.2 |
| DiffH2O | 11.1 | 22.7 | 8.00 | 35.3 | 12.8 |
| LatentHOI | 33.8 | 48.6 | 4.80 | 72.6 | 11.6 |
| OpenHOI | 16.8 | 34.0 | 6.32 | 51.8 | 13.3 |
| GRAZE (Ours) | 45.3 | 67.2 | 2.97 | 92.2 | 10.3 |
| Gaze window | GSR | Part Acc. | PCP |
|---|---|---|---|
| First 10% | 61.9 | 47.9 | 48.6 |
| First 20% | 64.5 | 47.4 | 48.7 |
| First 30% | 63.8 | 46.5 | 47.8 |
| First 50% | 63.3 | 46.5 | 48.1 |
| Before contact | 63.2 | 46.1 | 47.9 |
| All frames | 63.8 | 48.6 ∗ | 49.7 ∗ |
| Protocol | Train clips | Test clips | Objects (train/test) |
|---|---|---|---|
| Object fold 1 | 611 | 301 | 21 / 10 (unseen) |
| Object fold 2 | 602 | 310 | 20 / 11 (unseen) |
| Object fold 3 | 611 | 301 | 21 / 10 (unseen) |
| Subject fold 1 | 627 | 285 | 31 / 31 |
| Subject fold 2 | 603 | 309 | 31 / 30 |
| Subject fold 3 | 594 | 318 | 31 / 31 |
| Method | Geometry | Text | Hand | Gaze | Length | Init. pose |
|---|---|---|---|---|---|---|
| Text2HOI | PointNet | flag | ||||
| DiffH2O | BPS | prompt | ||||
| LatentHOI | BPS | prompt | ||||
| OpenHOI | PointNet | flag | ||||
| GRAZE | BPS | flag |
| 2 objects | 3 objects | |||||
|---|---|---|---|---|---|---|
| Method | TSA | DCR | Held | TSA | DCR | Held |
| Text2HOI ∗ | 15.3 | 79.6 | 19.3 | 0 4.0 | 93.6 | 0 7.3 |
| + nearest-pt. | 15.0 | 80.3 | 19.7 | 0 3.9 | 94.0 | 0 7.5 |
| GRAZE (Ours) | 47.4 | 24.1 | 42.3 | 34.6 | 40.0 | 30.9 |
| Recorded (GT) | 95.6 | 0 4.4 | 99.9 | 93.3 | 0 6.7 | 99.7 |
| Metric | Support | Meaning |
|---|---|---|
| Grasp Success Rate (GSR) | Final five valid frames (contact), final valid frame (penetration, lift) | Contact in each of the final five frames, final-frame penetration depth at most 10 mm, and final lift of at least 5 mm along the world-up direction |
| Part Contact Precision (PCP) / Part Accuracy (Part Acc.) | Final valid frame | Target-part precision and dominant-part match |
| Interpenetration Depth (ID) | Penetrating valid frames | Maximum depth (mm) per frame, averaged over penetrating frames |
| Contact | Final five valid frames | Proportion meeting the contact condition of GSR |
| Sim / SimD | Final valid frame | Object held in physics simulation (displacement below 2 cm) / mean object displacement (cm) |
| Follow | Rotated input | Final-wrist azimuth about the object’s up axis turns with the input rotation within 45 ∘ |
| Method | Refinement | GSR | Part Acc. | PCP | ID |
|---|---|---|---|---|---|
| Text2HOI | own pipeline | 35.3 | 36.8 | 36.6 | 12.0 |
| + nearest-pt. | 40.3 | 37.8 | 38.0 | 10.2 | |
| DiffH2O | own pipeline | 17.9 | 31.7 | 31.0 | 8.9 |
| + nearest-pt. | 22.7 | 33.1 | 32.5 | 7.4 | |
| LatentHOI | own pipeline | 46.6 | 43.6 | 42.5 | 8.4 |
| + nearest-pt. | 55.6 | 44.8 | 44.1 | 7.2 |
| Object fold 1 | Object fold 2 | Object fold 3 | |||||||
| Row | GSR | Part Acc. | PCP | GSR | Part Acc. | PCP | GSR | Part Acc. | PCP |
| Table 1 of the main paper (own pipelines) | |||||||||
| Text2HOI | 23.2 | 43.8 | 43.0 | 41.4 | 37.6 | 37.9 | 41.4 | 29.0 | 29.0 |
| DiffH2O | 12.1 | 30.0 | 28.4 | 17.0 | 33.8 | 34.5 | 24.6 | 31.4 | 30.2 |
| LatentHOI | 35.6 | 42.5 | 41.7 | 51.9 | 50.5 | 48.3 | 52.3 | 37.9 | 37.4 |
| OpenHOI | 13.9 | 34.7 | 33.7 | 25.7 | 30.3 | 30.1 | 37.7 | 29.6 | 29.3 |
| Method | Refinement | GSR | Sim | Contact | ID |
|---|---|---|---|---|---|
| Text2HOI | own pipeline | 18.4 | 27.1 | 46.8 | 11.2 |
| + nearest-pt. | 24.9 | 31.9 | 51.9 | 9.5 | |
| DiffH2O | own pipeline | 11.1 | 22.7 | 35.3 | 12.8 |
| + nearest-pt. | 15.1 | 25.7 | 38.7 | 10.8 | |
| LatentHOI | own pipeline | 33.8 | 48.6 | 72.6 | 11.6 |
| + nearest-pt. | 40.0 | 52.3 | 76.3 | 10.3 |
| Condition | GSR | Part Acc. | PCP | Follow |
|---|---|---|---|---|
| Without gaze | ||||
| No condition | 59.0 | 43.7 | 43.6 | – |
| Part label | 60.8 | 48.8 | 48.4 | – |
| Text prompt | 64.8 | 52.2 | 50.6 | – |
| Derived from the recorded gaze | ||||
| Gaze point | 59.9 | 46.5 | 46.8 | 28.6 |
| Variant | GSR | Part Acc. | Follow |
|---|---|---|---|
| Ray closeness only | 57.0 | 46.7 | 37.9 |
| Alignment only | 68.3 | 49.2 | 53.6 |
| Both (Ours) | 66.3 | 49.3 | 59.1 |
| Corruption | GSR | Part Acc. |
|---|---|---|
| Angular noise, 5 ∘ | ||
| Angular noise, 10 ∘ | ||
| Frame drop, 20% | ||
| Frame drop, 50% |
| Part Acc. | PCP | |||||||
| Method | All | Micro | Obj. | Pair | All | Micro | Obj. | Pair |
| Own pipeline | ||||||||
| Text2HOI | 36.8 | 33.0 | 36.6 | 38.2 | 36.6 | 32.8 | 36.1 | 37.7 |
| DiffH2O | 31.7 | 28.8 | 30.5 | 29.9 | 31.0 | 28.0 | 29.6 | 28.9 |
| LatentHOI | 43.6 | 39.2 | 41.1 | 42.0 | 42.5 | 37.9 | 40.1 | 40.7 |
| OpenHOI | 31.5 | 29.5 | 31.8 | 34.5 | 31.0 | 28.9 | 31.1 | 34.1 |
| Method | GSR | Part Acc. | PCP | ID |
|---|---|---|---|---|
| Text2HOI | 56.8 3.9 | 71.7 2.7 | 69.4 2.1 | 8.0 0.8 |
| DiffH2O | 26.0 5.3 | 60.9 3.9 | 58.5 3.3 | 5.9 0.5 |
| LatentHOI | 43.1 14.2 | 69.7 6.6 | 67.4 5.5 | 6.2 0.1 |
| OpenHOI | 21.9 1.9 | 41.6 1.1 | 40.9 0.7 | 8.5 1.1 |
| GRAZE (Ours) | 69.7 3.1 | 59.2 3.3 | 59.7 3.0 | 6.2 0.2 |
HOI Generation
Human-object interaction generation creates 3D motion, grasps, and videos of people manipulating or coordinating with objects.
See all 87 papers