cs.CVNov 3, 2026

GVT3D: Gesture-Vision-Text Token Enrichment for 3D Embodied Reference Understanding

Authors: Vimarsh Shah, Dulanga Weerakoon, Ajinkya Deshpande, Saksham Agarwal, Nilesh Bhatia, Neel Kiran Naik, Parth Shah, Sravan Danda, 

Organizations: BITS Pilani Goa campus, India. · Singapore-MIT Alliance for Research and Technology (SMART) Centre, Singapore. · Massachusetts Institute of Technology, USA. · Singapore Management University, Singapore. · 

Abstract

Natural human-robot interaction requires robots to interpret language and gestures within a 3D environment to support tasks such as pick-and-place and object retrieval. 3D Embodied Reference Understanding (3D-ERU) supports this by identifying the 3D location of a target object from a scene (point-cloud) containing a human pointing gesture and an accompanying language description. However, current 3D-ERU models lack alignment of cross-modal representations and incur high processing costs that increase with the point-cloud's size. To address this, we propose GVT3D, which employs an Embodied Fusion module that jointly models language, gesture, and vision inputs within a shared embedding space, promoting cross-modal alignment. We additionally introduce a foveation approach that selectively processes the point cloud within a coarse region around the estimated pointing direction to reduce computations. GVT3D achieves ~8% higher accuracy and foveation enables 2.95x faster execution while maintaining comparable accuracy, relative to prior 3D-ERU models.

Figures & tables

Explore similar work

CardsList
  1. AgentGrounder: Zero-Shot 3D Visual Pointcloud Grounding using Multimodal Language Models

    May 25, 2026Cuong Huynh, Maxim Popov, Denis Gridusov +13D Visual Grounding

  2. UniGround: Universal 3D Visual Grounding via Training-Free Scene Parsing

    Mar 9, 2026Jiaxi Zhang, Yunheng Wang, Wei Lu +83D Visual Grounding3D VLMs

  3. PointACT: Vision-Language-Action Models with Multi-Scale Point-Action Interaction

    May 20, 2026Shizhe Chen, Paul Pacaud, Cordelia SchmidLanguage-Conditioned Robot ManipulationPoint Cloud Learning