Beyond Language: Grounding Referring Expressions with Hand Pointing in Egocentric Vision

Ling Li; Bowen Liu; Zinuo Zhan; Peng Jie; Jianhui Zhong; Kenglun Chang; Zhidong Deng

cs.CVMar 27, 2026

Beyond Language: Grounding Referring Expressions with Hand Pointing in Egocentric Vision

Authors: Ling Li, Bowen Liu, Zinuo Zhan, Peng Jie, Jianhui Zhong, Kenglun Chang, Zhidong Deng

Organizations: Tsinghua University, Beijing, China · Dalian University of Technology, Dalian, China · Northwestern Polytechnical University, Xi’an, China · Apple, USA

Abstract

Traditional Visual Grounding (VG) predominantly relies on textual descriptions to localize objects, a paradigm that inherently struggles with linguistic ambiguity and often ignores non-verbal deictic cues prevalent in real-world interactions. In natural egocentric engagements, hand-pointing combined with speech forms the most intuitive referring mechanism. To bridge this gap, we introduce EgoPoint-Ground, the first large-scale multimodal dataset dedicated to egocentric deictic visual grounding. Comprising over 15k interactive samples in complex scenes, the dataset provides rich, multi-grained annotations including hand-target bounding box pairs and dense semantic captions. We establish a comprehensive benchmark for hand-pointing referring expression resolution, evaluating a wide spectrum of mainstream Multimodal Large Language Models (MLLMs) and state-of-the-art VG architectures. Furthermore, we propose SV-CoT, a novel baseline framework that reformulates grounding as a structured inference process, synergizing gestural and linguistic cues through a Visual Chain-of-Thought paradigm. Extensive experiments demonstrate that SV-CoT achieves an $\textbf{11.7\%}$ absolute improvement over existing methods, effectively mitigating semantic ambiguity and advancing the capability of agents to comprehend multimodal physical intents. The dataset and code will be made publicly available.

Figures & tables

Figure 1: Our dataset features images captured from an egocentric perspective where a hand is pointing towards a target object. The dataset includes four core annotations: Hand Annotation, Target Object Annotation, Image Caption, and Visual QA Pairs. While conventional Referring Expression Comprehension (REC) tasks only support language-assisted localization, our proposed task aims to address referring disambiguation by utilizing physical hand information as opposed to traditional linguistic assistance for target grounding.

Figure 2: Statistical overview and distribution analysis of the Ego-Point Ground dataset. The dataset encompasses a broad spectrum of everyday objects, accompanied by semantically rich and lexically diverse textual annotations.

Dataset	Lang.	Gest.	Embo.	Ego/Exo	Pose	VQA	Source	No. images
PointAt [ 87 ]	✗	✓	✓	Ego	✗	✗	Lab	220
ReferAt [ 86 ]	✗	✓	✓	Exo	✗	✗	Lab	242
IPO [ 89 ]	✗	✓	✓	Exo	✗	✗	Lab	278
IMHF [ 90 ]	✗	✗	✓	Ego	✗	✗	Lab	1,716
RefIt [ 56 ]	✓	✗	✗	Exo	✗	✗	CLEF	19,894
YourRefIt [ 18 ]	✓	✓	✓	Exo	✗	✗	Crowd-Sourced	497,348

Table 1: Feature comparison of EgoPoint versus representative reference datasets.

Figure 3: Analysis of Cross-Modal Quality and Semantic Diversity. Figure (a) shows the visual-textual semantic alignment heatmap. The high diagonal similarity confirms strong semantic consistency between images and descriptions. Figure (b) displays the category semantic distribution scatter plot. Image description embeddings are visualized in 2D using PCA (color-coded by category). The points are uniformly distributed without significant clustering, demonstrating the diversity of the dataset’s semantic space.

Figure 4: Overview of the Ego-Point Ground Dataset Collection and Multi-Stage Annotation Pipeline. Stages 1–3 illustrate the raw data acquisition and fundamental annotation workflow. Stage 4 details the extraction of 2D keypoint pose information. Subsequently, Stage 5 focuses on generating high-quality Visual Question-Answering (VQA) pairs for the target objects.

Figure 5: Image Examples of Ego-Point Ground Dataset. We showcase example images and partial annotations from the dataset, including bounding boxes for the hand and the target object, as well as the corresponding caption for the target object. The first row specifically illustrates data collected from our real-world captures.

Figure 6: Architectural overview of the SV-CoT framework. The egocentric hand bounding box $\mathcal{B}_{hand}^{2D}$ is aligned into discrete spatial anchors $\mathcal{T}_{pos}$ . Zero-shot grounding is reformulated as a latent reasoning chain $\mathcal{R}$ that sequentially infers: (1) a deictic directional primitive $\vec{v}$ ; (2) a virtual ray $\mathcal{L}(t)$ to spatially prune distractors into a candidate set $\Omega$ ; and (3) a multi-modal semantic verification score $\Psi$ to either resolve the target $BBox_{\mathcal{O}}$ or explicitly reject it via threshold $\tau$ .

Model	EDG — Hybrid			EDG — Real-World
Model	$\mathcal{P}@0.3\uparrow$	$\mathcal{P}@0.5\uparrow$	$\mathcal{P}@0.7\uparrow$	$\mathcal{P}@0.3\uparrow$	$\mathcal{P}@0.5\uparrow$	$\mathcal{P}@0.7\uparrow$
Qwen3-VL-4B [ 110 ]	0.743	0.702	0.622	0.747	0.699	0.616
Qwen2.5-VL-3B [ 100 ]	0.731	0.671	0.592	0.738	0.632	0.488
Qwen3-VL-8B [ 110 ]	0.726	0.676	0.598	0.789	0.701	0.593
Qwen2.5-VL-7B [ 100 ]	0.721	0.653	0.563	0.776	0.688	0.533
LLaVA-OneVision-4B [ 3 ]	0.679	0.475	0.145	0.654	0.431	0.129

Table 2: Quantitative results of state-of-the-art models on the D-VQA task, evaluated across both Hybrid (mixed synthetic and captured) and Real-World (naturally captured) datasets. Shaded rows categorize the models into distinct performance tiers, while the Human Study establishes the theoretical upper bound.

Model	D-VQA (Hybrid)			D-VQA (Real-World)
Model	Acc. $\uparrow$	M-F1 $\uparrow$	Rec. $\uparrow$	Acc. $\uparrow$	M-F1 $\uparrow$	Rec. $\uparrow$
LLaVA-OneVision-8B [ 3 ]	0.531	0.519	0.548	0.544	0.525	0.526
LLaVA-OneVision-4B [ 3 ]	0.510	0.535	0.547	0.521	0.476	0.487
Qwen3-VL-8B [ 110 ]	0.494	0.484	0.490	0.517	0.510	0.503
Qwen3-VL-4B [ 110 ]	0.471	0.488	0.495	0.456	0.507	0.489
Qwen2.5-VL-7B [ 100 ]	0.457	0.427	0.445	0.444	0.428	0.411

Table 3: Performance of state-of-the-art models on the D-VQA task across Hybrid (mixed synthetic/captured) and Real-World (exclusively captured) datasets. The "Human Study" denotes the evaluation results from human participants.

Model	D-REC			POG
Model	$\mathcal{P}@0.3\uparrow$	$\mathcal{P}@0.5\uparrow$	$\mathcal{P}@0.7\uparrow$	$\mathcal{P}@0.3\uparrow$	$\mathcal{P}@0.5\uparrow$	$\mathcal{P}@0.7\uparrow$
Qwen3-VL-4B [ 110 ]	0.737 (+0.087)	0.684 (+0.118)	0.605 (+0.123)	0.671 (+0.021)	0.612 (+0.046)	0.532 (+0.050)
Qwen3-VL-8B [ 110 ]	0.701 (+0.111)	0.648 (+0.087)	0.561 (+0.077)	0.653 (+0.063)	0.595 (+0.034)	0.514 (+0.030)
Qwen2.5-VL-7B [ 100 ]	0.703 (+0.190)	0.638 (+0.160)	0.541 (+0.125)	0.670 (+0.157)	0.611 (+0.143)	0.515 (+0.099)
Qwen2.5-VL-3B [ 100 ]	0.665 (+0.169)	0.592 (+0.167)	0.491 (+0.139)	0.575 (+0.079)	0.501 (+0.076)	0.407 (+0.055)
LLaVA-OneVision-4B [ 3 ]	0.586 (+0.194)	0.347 (+0.186)	0.088 (+0.071)	0.525 (+0.133)	0.292 (+0.131)	0.065 (+0.048)

Table 4: Performance comparison on conventional REC and dedicated POG tasks. Absolute scores are reported with relative gains/losses compared to the baseline (Inputs are provided without either spatial or linguistic cues) in parentheses.

Figure 7: Overview of the Egopoint-ground Dataset and its Core Annotations. The Egopoint-ground dataset comprises first-person-perspective images captured via smart glasses, depicting a hand pointing towards a target object. The dataset features four primary annotations for each image: 1) Target Object Annotation: Includes the object’s bounding box, textual description, and essential attributes. 2) Fine-grained Description: Detailed linguistic descriptions of the overall scene and the deictic (referential) interaction. 3) High-Quality Visual QA Pairs: Question-and-answer pairs focusing on the target object and the interaction intent. 4) First-Person Hand Pose and Action: Provides the hand’s bounding box, 2D keypoint pose, and precise action description, serving as a key non-verbal spatial anchor.

Figure 8: Example of the Visual Question Answering Section for Object Pointing, Showcased via Random Sampling.

Figure 9: Display of Randomly Sampled Image Captions.

Source	Ratio
Real-World Data	39.35%
Synthetic Generation	38.96%
Image Editing	21.69%

Table 5: Data Source Composition

Parameter	Specification
Model	RayNeo V3
Sensor	Sony IMX681, 12MP
Lens	5P customized F2.2 large aperture 16mm ultra-wide-angle
Algorithm	ArcSoft customized algorithm
ISP	1st generation Snapdragon AR1 dual ISP

Table 6: Key Parameters of RayNeo V3

Model Name	Model Size	Inference Time (h:min)	Peak VRAM Usage	Model Source
Qwen3-VL-8B	8.78 B	1:47	16.6 GB	Open-Sourced Model
Florence-2-large	0.77 B	0:36	1.8 GB	Open-Sourced Model
Ferret	13.37 B	4:52	26.9 GB	Open-Sourced Model
LLaVA-OneVision-1.5-8B-Instruct	8.53 B	1:58	16.3 GB	Open-Sourced Model
GroundingGPT	7.90 B	1:52	16.5 GB	Open-Sourced Model
InternVL-3.5-8B	7.91 B	2:21	40.2 GB	Open-Sourced Model

Table 7: Model Specifications and Evaluation Metrics. (Test on Egocentric Deictic Visual Grounding - EDG)

Model Name	Input Paradigm
GroundingGPT	"[object caption]"
Qwen3-VL	"You are given an egocentric (first-person) image where a visible hand/finger is pointing to an object. Your task is to localize the pointed-at target object and output its bounding box. Use [x1, y1, x2, y2] coordinates in range [0, 1000] (relative to image size). The underspecified reference below is auxiliary text about the target. Underspecified reference: {underspecified_referent}. Return ONLY one JSON object in this format: {"bbox_2d": [x1, y1, x2, y2]}."
InternVL-3.5	"You are given an egocentric (first-person) image where a visible hand/finger is pointing to an object, predict the bounding box of <ref>{underspecified_referent}</ref> being pointed at in format [x1, y1, x2, y2] where coordinates are in range [0, 1000] (relative to image size). Return ONLY the bbox of the pointed object in format [x1, y1, x2, y2]. Give me the result directly, do not reason."
LLaVA-OneVision	"You are given an egocentric (first-person) image where a visible hand/finger is pointing to an object. Your task is to localize the pointed-at target object and output its bounding box, where each coordinate is normalized to [0, 1] relative to image width/height. Underspecified reference: {underspecified_referent}. Return ONLY the bbox in this exact format: [x1, y1, x2, y2]."
Ferret	"You are given an egocentric (first-person) image where a visible hand/finger is pointing to an object. Your task is to localize the pointed-at target object and output its bounding box in format [x1, y1, x2, y2], where coordinates are in range [0, 1000] (relative to image size). Underspecified reference: {underspecified_referent}. Return ONLY the bbox of the pointed object as [x1, y1, x2, y2]."
DeepSeek-VL2	"<\|grounding\|> You are given an egocentric (first-person) image where a visible hand/finger is pointing to an object. Your task is to localize the pointed-at target object and output its bounding box in format [x1, y1, x2, y2], where each coordinate is in [0, 1000] relative to image size. Target object’s underspecified reference: {underspecified_referent}. Return ONLY one bbox in [x1, y1, x2, y2]."

Table 8: Model Input Paradigms (Test on Language-Only Grounding (D-REC))

Model Name	Input Paradigm
GroundingGPT	"You are a pointed object bounding box detector. Given an image and a hand bbox, predict the bounding box of the object being pointed at in format [x1, y1, x2, y2]. Hand bbox (xyxy, normalized to [0,1]): {hand_input}. Return ONLY the bbox of the pointed object in this format: [x1, y1, x2, y2]."
Qwen3-VL	"You are a pointed object bounding box detector. Given an image and a hand bbox, predict the bounding box of the object being pointed at in format [x1, y1, x2, y2] where coordinates are in range [0, 1000] (relative to image size). Input hand bbox: {hand_input}. Return ONLY the bbox of the hand-pointed object in this format: {"bbox_2d": [x1, y1, x2, y2]}."
InternVL-3.5	"You are a pointed object bounding box detector. Given an image and a hand bbox, predict the bounding box of <ref>the object being pointed</ref> in format [x1, y1, x2, y2] where coordinates are in range [0, 1000] (relative to image size). Input hand bbox: {hand_input}. Return ONLY the bbox of the pointed object in format [x1, y1, x2, y2]."
LLaVA-OneVision	"You are a pointed object bounding box detector. Given an image and a hand bbox, predict the bounding box of the object being pointed at in format [x1, y1, x2, y2], where each coordinate is normalized to [0, 1] relative to image width/height. Input hand bbox (xyxy, normalized to [0, 1]): {hand_input}. Return ONLY the bbox in this exact JSON format: [x1, y1, x2, y2]."
Ferret	"You are a pointed object bounding box detector. Given an image and a hand bbox, predict the bounding box of the object being pointed at in format [x1, y1, x2, y2] where coordinates are in range [0, 1000] (relative to image size). Input hand bbox: {hand_input}. Return ONLY about the POINTED OBJECT, do not mention the hand, do not reason, return the result directly as [x1, y1, x2, y2]."
DeepSeek-VL2	"<\|grounding\|> You are a pointed object grounding model. Given an image and a hand bbox, predict the bbox of the object being pointed at in format [x1, y1, x2, y2] where coordinates are in [0, 1000] relative to image size. Input hand bbox: {hand_input}. Return ONLY one bbox in [x1, y1, x2, y2]."

Table 9: Model Input Paradigms (Test on Pointing Only Grounding - POG)

Category	ID	Category	ID	Category	ID	Category	ID	Category	ID	Category	ID
advertising board	0	airplane	1	alarm	2	ammeter	3	apple	4	artwork	5
ashtray	6	audio equipment	7	axe	8	backpack	9	bag	10	ball	11
ball of thread	12	ballpoint pen	13	banana	14	band-aid	15	baseball bat	16	basketball	17
bear	18	bed	19	beer	20	bell	21	belt	22	bench	23
beverage	24	bicycle	25	bike	26	bird	27	blackboard	29	blanket	30
board	31	boat	32	book	33	bookshelf	34	bottle	35	bouquet of flowers	36

Table 10: Detailed Mapping of Dataset Object Categories to Their Corresponding Integer IDs.

Multimodal Grounding

Multimodal grounding links language and other signals to the specific objects, regions, events, or actions that support an interpretation.

See all 185 papers

Do MLLMs Understand Pointing? Benchmarking and Enhancing Referential Reasoning in Egocentric Vision
Apr 23, 2026Chentao Li, Zirui Gao, Mingze Gao +3Multimodal QA Vision-Language Grounding
Department of Automation, Tsinghua University · Academy of Art & Design, Tsinghua University
Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answering
Mar 13, 2026Yura Choi, Roy Miles, Rolandos Alexandros Potamias +3Egocentric Video QA Video QA
Imperial College London
MM-Conv: A Multimodal Dataset and Benchmark for Context-Aware Grounding in 3D Dialogue
May 20, 2026Anna Deichler, Jim O'Regan, Fethiye Irmak Dogan +4Query Reformulation Visually Grounded Reasoning
KTH Royal Institute of Technology, Stockholm, Sweden

Apr 23, 2026J/K move · Enter open · S save

Do MLLMs Understand Pointing? Benchmarking and Enhancing Referential Reasoning in Egocentric Vision

Chentao Li, Zirui Gao, Mingze Gao, Yinglian Ren+2

Department of Automation, Tsinghua University · Academy of Art & Design, Tsinghua University

Multimodal QA Vision-Language Grounding

Open paper AskPDF arXiv

Egocentric AI agents, such as smart glasses, rely on pointing gestures to resolve referential ambiguities in natural language commands. However, despite advancements in Multimodal Large Language Models (MLLMs), current systems often fail to precisely ground the spatial semantics of pointing. Instead, they rely on spurious correlations with visual proximity or object saliency, a phenomenon we term "Referential Hallucination." To address this gap, we introduce EgoPoint-Bench, a comprehensive question-answering benchmark designed to evaluate and enhance multimodal pointing reasoning in egocentric views. Comprising over 11k high-fidelity simulated and real-world samples, the benchmark spans five evaluation dimensions and three levels of referential complexity. Extensive experiments demonstrate that while state-of-the-art proprietary and open-source models struggle with egocentric pointing, models fine-tuned on our synthetic data achieve significant performance gains and robust sim-to-real generalization. This work highlights the importance of spatially aware supervision and offers a scalable path toward precise egocentric AI assistants. Project page: https://guyyyug.github.io/EgoPoint-Bench/

Abstract

Multimodal Grounding

Explore similar work

Do MLLMs Understand Pointing? Benchmarking and Enhancing Referential Reasoning in Egocentric Vision

Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answering

MM-Conv: A Multimodal Dataset and Benchmark for Context-Aware Grounding in 3D Dialogue