GVT3D: Gesture-Vision-Text Token Enrichment for 3D Embodied Reference Understanding
Organizations: BITS Pilani Goa campus, India. · Singapore-MIT Alliance for Research and Technology (SMART) Centre, Singapore. · Massachusetts Institute of Technology, USA. · Singapore Management University, Singapore. ·
Abstract
Natural human-robot interaction requires robots to interpret language and gestures within a 3D environment to support tasks such as pick-and-place and object retrieval. 3D Embodied Reference Understanding (3D-ERU) supports this by identifying the 3D location of a target object from a scene (point-cloud) containing a human pointing gesture and an accompanying language description. However, current 3D-ERU models lack alignment of cross-modal representations and incur high processing costs that increase with the point-cloud's size. To address this, we propose GVT3D, which employs an Embodied Fusion module that jointly models language, gesture, and vision inputs within a shared embedding space, promoting cross-modal alignment. We additionally introduce a foveation approach that selectively processes the point cloud within a coarse region around the estimated pointing direction to reduce computations. GVT3D achieves ~8% higher accuracy and foveation enables 2.95x faster execution while maintaining comparable accuracy, relative to prior 3D-ERU models.
Figures & tables
| Model | [email protected] | [email protected] | ||||
|---|---|---|---|---|---|---|
| Unique | Multiple | Overall | Unique | Multiple | Overall | |
| without Gestures | ||||||
| 3DVG-Transformer [ 27 ] | 71.56 | 31.35 | 39.17 | 50.66 | 21.54 | 27.20 |
| HAM [ 28 ] | 67.10 | 25.42 | 33.51 | 48.13 | 16.04 | 22.27 |
| 3DJCG [ 29 ] | 75.93 | 40.34 | 47.24 | 59.19 | 30.61 | 36.16 |
| RefMask3D [ 30 ] | 72.45 | 25.44 | 34.57 | 64.95 | 22.24 | 30.54 |
| Configuration | [email protected] | [email protected] | ||||
|---|---|---|---|---|---|---|
| Unique | Multiple | Overall | Unique | Multiple | Overall | |
| w/o Fusion | 80.1 | 59.0 | 63.2 | 68.0 | 47.8 | 51.6 |
| w/o Ges. | 65.9 | 27.3 | 34.8 | 57.4 | 22.2 | 29.0 |
| w. fixed lang. | 71.7 | 68.9 | 69.3 | 63.1 | 58.4 | 59.3 |
| Ges3ViG [ 6 ] (w. fixed lang) | 51.1 | 44.9 | 46.1 | 43.9 | 36.8 | 38.2 |
| GVT3D w/o Fov. (Ours) | 86.7 | 77.4 | 79.2 | 73.3 | 65.2 | 66.7 |
3D Visual Grounding
Three-dimensional visual grounding links language or visual queries to specific objects, parts, regions, and spatial relations in 3D scenes.
See all 149 papers