Understanding and answering questions based on a user's pointing gesture is essential for next-generation egocentric AI assistants. However, current Multimodal Large Language Models (MLLMs) struggle with such tasks due to the lack of gesture-rich data and their limited ability to infer fine-grained pointing intent from egocentric video. To address this, we introduce EgoPointVQA, a dataset and benchmark for gesture-grounded egocentric question answering, comprising 4000 synthetic and 400 real-world videos across multiple deictic reasoning tasks. Built upon it, we further propose Hand Intent Tokens (HINT), which encodes tokens derived from 3D hand keypoints using an off-the-shelf reconstruction model and interleaves them with the model input to provide explicit spatial and temporal context for interpreting pointing intent. We show that our model outperforms others in different backbones and model sizes. In particular, HINT-14B achieves 68.1% accuracy, on average over 6 tasks, surpassing the state-of-the-art, InternVL3-14B, by 6.6%. To further facilitate the open research, we will release the code, model, and dataset. Project page: https://yuuraa.github.io/papers/choi2026egovqa
Figures & tables
Figure 2: Task taxonomy and examples from EgoPointVQA . EgoPointVQA includes six subsets of questions regarding the properties of a pointed object. Each example shows egocentric video frames and a question using deictic references. Tasks include reference (object identification), counting (number of same objects), spatial (location and relative depth), temporal (order of multiple gestures), attribute (object properties), and feedback (object function). All questions require resolving deictic references through visual grounding of pointing gestures. The pointed objects are highlighted with red circles for visualization purposes.
Figure 3: Visualization of synthetic videos in EgoPointVQA . Our synthetic data covers diverse indoor scenes with various lighting conditions.
Figure 4: EgoPointVQA generation pipeline. From a mixture of simulated and real egocentric videos, we automatically generate multiple-choice question answer pairs referring to the pointed objects in the video. The questions are made deictic so that the model should visually understand the pointing gesture to answer.
Figure 5: EgoPointVQA statistics. Distribution of task types (charts) and common object word clouds for the (Left) training set (N=18073) and the (Right) test set (N=672).
Figure 6: HINT overall architecture. HINT uses an additional adapter to model the 3D location and movement of the hand directly. Vt denotes a visual token, Kt the keypoint feature, and Ht the hand intent token extracted for each frame It .
Method
Size
LLM
Refer.
Temporal
Spatial
Count
Attr.
Feed.
Avg.
Random
-
-
20.0
20.0
27.0
20.0
20.0
50.0
26.2
Proprietary Models
GPT-5 [ 30 ]
-
-
75.6
53.6
62.3
50.0
56.1
77.8
62.6
GPT-4o [ 20 ]
-
-
56.1
29.5
43.1
44.8
41.5
65.7
46.8
Open-source MLLMs ≥ 32B
Qwen3-VL [ 37 ]
32B
Qwen3
63.7
67.9
65.8
66.7
63.4
77.2
67.5
Table 1: Performance of different MLLMs on the EgoPointVQA test set. We report multiple-choice accuracy (%). HINT (highlighted in light blue) consistently improves its corresponding open-source backbones and outperforms all compared baselines. Task categories are Reference (Refer.), Temporal, Spatial, Counting (Count), Attribute (Attr.), and Feedback (Feed.). The random baseline reflects the varying number of answer choices across tasks.
SFT
Hand Int.
Reference
Temporal
Spatial
Attribute
✗
✗
66.1
57.5
63.2
51.3
✓
✗
68.5
60.7
59.6
56.7
✓
✓
75.0
66.1
64.9
61.0
Table 2: Ablation of HINT components. ‘SFT’ denotes supervised fine-tuning on EgoPointVQA . ‘Hand Int.’ denotes use of our Hand Intent Token. Combining both yields the largest gains.
Real
Synthetic
Reference
Temporal
Spatial
Attribute
✗
✓
69.0
62.5
60.5
58.5
✓
✗
67.3
60.7
56.1
56.1
✓
✓
75.0
66.1
64.9
61.0
Table 3: Impact of training dataset. We vary the usage of synthetic and real videos in the training set. Using a synthetic dataset to complement a real dataset yields the best result.
Hand Intent Modeling
Reference
Temporal
Spatial
None
68.5
60.7
59.6
Visual Keypoints
57.1
60.7
61.4
Visual Arrow from Fingertip
70.2
60.7
62.3
3D Keypoints in Text
68.5
55.4
58.8
2D Keypoints in Text
69.0
57.1
59.6
Table 4: Different methods of hand intent modeling. We compare different methods to encode the user’s hand pose. Our HINT with learning keypoint adapter outperforms alternative representations, such as visual prompts and textual inputs.
τ
Reference
Temporal
Spatial
Count
Attribute
0.1
66.7
58.7
62.9
32.4
59.3
0.3
75.0
58.7
62.9
35.3
59.3
0.5
75.0
66.1
64.9
35.4
61.0
0.7
64.9
60.9
62.9
32.4
59.3
0.9
68.4
63.1
60.0
32.4
59.3
Table 5: Impact of hand detection confidence threshold τ on HINT . The threshold τ controls a trade-off between filtering noisy detections and retaining valid pointing gestures. A value of 0.5 achieves the best overall performance.
Method
Input Frames
Reference
Spatial
Attribute
InternVL3-8B
uniform 32
66.1
63.2
51.3
InternVL3-8B
keyframes
61.3
62.3
61.0
InternVL3-14B
uniform 32
63.1
61.4
58.5
InternVL3-14B
keyframes
60.1
56.1
61.2
Table 6: Impact of input frames. We compare uniform 32-frame input with keyframes manually selected as oracle, where each pointing gesture is most clearly visible (one frame per gesture). y Even with this oracle advantage, Reference drops by 4.8/3.0 pp (8B/14B) compared to uniform 32-frame input.
Backbone
r
a
lr
Resolution
InternVL3-8B
64
128
1e-5
448 × 448
InternVL3-14B
32
64
2e-5
448 × 448
LLaVA-OneVision-7B
32
64
1e-5
384 × 384
Table 7: Hyperparameters for HINT per backbone. This table summarizes the optimized training configurations. Here, ‘r’ denotes the LoRA rank, ‘a’ the LoRA scaling factor, and ‘lr’ the learning rate used during finetuning.
Method
Video-MME
MVBench
EgoSchema
EgoBlind
InternVL3-8B
64.2
73.2
67.2
52.8
HINT InternVL3-8B
64.6
73.2
67.1
57.5
Table 8: Performance on existing video understanding benchmarks. Comparison of HINT against the baseline InternVL3-8B on Video-MME [ 15 ] , MVBench [ 26 ] , EgoSchema [ 28 ] (MCQ), and EgoBlind [ 11 ] . The results indicate that finetuning on our dataset preserves the model’s general video understanding capabilities.
Ref.
Temp.
Spat.
Count
Attr.
Feed.
Human
98.2
94.3
93.3
92.8
96.5
100.0
Table 9: Human performance on EgoPointVQA . We report the average accuracy of 5 human participants, each evaluating the full test set (672 questions).
Video
Reference
Temporal
Spatial
Attribute
Original
75.0
66.1
64.9
61.0
w/o Hand
41.7
21.4
44.4
36.3
Table 10: Ablation study on the effect of hand gestures. Performance comparison on our dataset with and without the pointing hand visible in the video frames. The significant drop in performance across all tasks in the ‘w/o Hand’ setting confirms that the pointing gesture is essential for identifying the target.
Reference
Temporal
Spatial
Count
Attribute
Feedback
Random
20.0
20.0
27.0
20.0
20.0
50.0
Blind
16.7
25.5
32.0
6.3
22.5
54.9
Choices-only
20.8
25.5
20.8
10.4
21.6
54.1
Table 11: Dataset bias analysis. We evaluate two text-only baselines to detect potential shortcuts: “Blind” receives the question text without video, and “Choices-only” receives only the answer options without the question or video. Performance near random chance confirms that EgoPointVQA requires visual grounding.
Figure 7: Representative failure cases on EgoPointVQA . (a)-(b): baseline MLLM failures due to saliency bias and temporal confusion, respectively. (c)-(d): remaining HINT failures caused by unreliable hand keypoints and rapid viewpoint drift.
Data Split
# Vid.
# QA
Vid. Dur.(s)
# Obj.
Train(Real)
100
640
4.61
22.5
Train(Synthetic)
4,000
18,745
11.6
11.6
Test (Real)
300
672
5.05
16.5
Table 12: EgoPointVQA statistics. Statistics of the dataset across synthetic and real-world splits. Real-world clips generally feature higher object density (scene complexity) than synthetic clips.
Figure 8: Spatial distribution of target objects in EgoPointVQA . ‘x’ and ‘y’ axes refer to the horizontal and vertical pixel coordinates, respectively. Individual object locations are shown as scattered dots.
Figure 9: Prompt used for dense scene information extraction (Stage 1). Instructions given to the MLLM to extract a structured JSON list of all salient objects.
Figure 10: General template for QA pair generation (Stage 2-1). The base instruction template is used for all question types. It enforces that the ”correct answer” must be derived exclusively from the scene JSON provided in Stage 1 to prevent hallucinations.
Figure 11: Task-specific prompt for Reference QA generation. This task directs the model to generate questions that require identifying the specific description or category of the pointed-at object (e.g., “What is this?”).
Figure 12: Task-specific prompt for Attribute QA generation. This module focuses on fine-grained visual details, instructing the model to query properties such as color, shape, material, or state (e.g., “Is this sliced?”).
Figure 13: Task-specific prompt for Spatial QA generation. This task generates questions regarding the relative position or depth of the target object compared to other landmarks in the scene (e.g., “Is this to the left of the sink?”).
Figure 14: Task-specific prompt for Feedback QA generation. This task tests reasoning about user intent and affordance, generating questions about whether a specific object is suitable for a stated goal (e.g., “I am thirsty. Can I drink this?”).
Figure 15: Task-specific prompt for Counting QA generation. This task tests visual search capabilities by asking the model to enumerate instances of the pointed-at object category visible in the scene.
Figure 16: Task-specific prompt for Temporal QA generation. This task challenges the model’s ability to recall the chronological sequence of pointing gestures (e.g., “What is the second object I pointed at?”).
Figure 17: Prompt used for negative choices generation (Stage 2-2). Instructions for generating negative choices. The model is prompted to select attributes from neighboring objects or plausible but incorrect properties to create challenging multiple-choice options.
Figure 18: Prompt used for deictic question rephrasing (Stage 3). The final prompt in the pipeline, where GPT-4o acts as a linguistic expert to convert structured queries into natural, spoken questions. It replaces explicit object names with deictic pronouns (“this”, “that”) to ensure the question requires visual grounding to be answered.
Figure 19: Examples from real-world indoor scenes. Samples from the real-world split of the dataset, captured in indoor environments. Each row displays the sample video frames alongside the corresponding question-answer pair generated by our pipeline. The correct answer is bolded and underlined within the options.
Figure 20: Examples from real-world outdoor scenes. Samples from the real-world split of the dataset, captured in outdoor environments. Each row displays the sample video frames alongside the corresponding question-answer pair generated by our pipeline. The correct answer is bolded and underlined within the options.
Figure 21: Examples from synthetic training data. Samples from the large-scale synthetic training set generated via AI2-THOR.
Egocentric AI agents, such as smart glasses, rely on pointing gestures to resolve referential ambiguities in natural language commands. However, despite advancements in Multimodal Large Language Models (MLLMs), current systems often fail to precisely ground the spatial semantics of pointing. Instead, they rely on spurious correlations with visual proximity or object saliency, a phenomenon we term "Referential Hallucination." To address this gap, we introduce EgoPoint-Bench, a comprehensive question-answering benchmark designed to evaluate and enhance multimodal pointing reasoning in egocentric views. Comprising over 11k high-fidelity simulated and real-world samples, the benchmark spans five evaluation dimensions and three levels of referential complexity. Extensive experiments demonstrate that while state-of-the-art proprietary and open-source models struggle with egocentric pointing, models fine-tuned on our synthetic data achieve significant performance gains and robust sim-to-real generalization. This work highlights the importance of spatially aware supervision and offers a scalable path toward precise egocentric AI assistants. Project page: https://guyyyug.github.io/EgoPoint-Bench/
Chentao Li, Zirui Gao, Mingze Gao +3
Department of Automation, Tsinghua University · Academy of Art & Design, Tsinghua University
Traditional Visual Grounding (VG) predominantly relies on textual descriptions to localize objects, a paradigm that inherently struggles with linguistic ambiguity and often ignores non-verbal deictic cues prevalent in real-world interactions. In natural egocentric engagements, hand-pointing combined with speech forms the most intuitive referring mechanism. To bridge this gap, we introduce EgoPoint-Ground, the first large-scale multimodal dataset dedicated to egocentric deictic visual grounding. Comprising over 15k interactive samples in complex scenes, the dataset provides rich, multi-grained annotations including hand-target bounding box pairs and dense semantic captions. We establish a comprehensive benchmark for hand-pointing referring expression resolution, evaluating a wide spectrum of mainstream Multimodal Large Language Models (MLLMs) and state-of-the-art VG architectures. Furthermore, we propose SV-CoT, a novel baseline framework that reformulates grounding as a structured inference process, synergizing gestural and linguistic cues through a Visual Chain-of-Thought paradigm. Extensive experiments demonstrate that SV-CoT achieves an 11.7% absolute improvement over existing methods, effectively mitigating semantic ambiguity and advancing the capability of agents to comprehend multimodal physical intents. The dataset and code will be made publicly available.
Ling Li, Bowen Liu, Zinuo Zhan +4
Tsinghua University, Beijing, China · Dalian University of Technology, Dalian, China · Northwestern Polytechnical University, Xi’an, China +1
Grounding has become a fundamental capability of vision-language models (VLMs). Most existing VLMs point by generating coordinates as part of their text output, which requires learning a complicated coordinate system and results in a high token count. Instead, we propose a more intuitive pointing mechanism that directly selects the visual tokens that contain the target concept. Our model generates a special pointing token that cross-attends to the input image or video tokens and selects the appropriate one. To make this model more fine-grained, we follow these pointing tokens with an additional special token that selects a fine-grained subpatch within the initially selected region, and then a third token that specifies a location within that subpatch. We further show that performance improves by generating points sequentially in a consistent order, encoding the relative position of the previously selected point, and including a special no-more-points class when selecting visual tokens. Using this method, we set a new state-of-the-art on image pointing (70.7% on PointBench), set a new state-of-the-art among fully open models on GUI pointing (61.1% on ScreenSpotPro), and improve video pointing (59.1% human preference win rate vs. a text coordinate baseline) and tracking (+6.3% gain on Molmo2Track). We additionally show that our method achieves much higher sample efficiency and discuss the qualitative differences that emerge from this design change.