cs.CVMar 13, 2026
SaveDo You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answering
Organizations: Imperial College London
Abstract
Understanding and answering questions based on a user's pointing gesture is essential for next-generation egocentric AI assistants. However, current Multimodal Large Language Models (MLLMs) struggle with such tasks due to the lack of gesture-rich data and their limited ability to infer fine-grained pointing intent from egocentric video. To address this, we introduce EgoPointVQA, a dataset and benchmark for gesture-grounded egocentric question answering, comprising 4000 synthetic and 400 real-world videos across multiple deictic reasoning tasks. Built upon it, we further propose Hand Intent Tokens (HINT), which encodes tokens derived from 3D hand keypoints using an off-the-shelf reconstruction model and interleaves them with the model input to provide explicit spatial and temporal context for interpreting pointing intent. We show that our model outperforms others in different backbones and model sizes. In particular, HINT-14B achieves 68.1% accuracy, on average over 6 tasks, surpassing the state-of-the-art, InternVL3-14B, by 6.6%. To further facilitate the open research, we will release the code, model, and dataset. Project page: https://yuuraa.github.io/papers/choi2026egovqa
Figures & tables
Figure 2: Task taxonomy and examples from EgoPointVQA . EgoPointVQA includes six subsets of questions regarding the properties of a pointed object. Each example shows egocentric video frames and a question using deictic references. Tasks include reference (object identification), counting (number of same objects), spatial (location and relative depth), temporal (order of multiple gestures), attribute (object properties), and feedback (object function). All questions require resolving deictic references through visual grounding of pointing gestures. The pointed objects are highlighted with red circles for visualization purposes.
Figure 3: Visualization of synthetic videos in EgoPointVQA . Our synthetic data covers diverse indoor scenes with various lighting conditions.
Figure 4: EgoPointVQA generation pipeline. From a mixture of simulated and real egocentric videos, we automatically generate multiple-choice question answer pairs referring to the pointed objects in the video. The questions are made deictic so that the model should visually understand the pointing gesture to answer.
Figure 5: EgoPointVQA statistics. Distribution of task types (charts) and common object word clouds for the (Left) training set (N=18073) and the (Right) test set (N=672).
Figure 6: HINT overall architecture. HINT uses an additional adapter to model the 3D location and movement of the hand directly. denotes a visual token, the keypoint feature, and the hand intent token extracted for each frame .
| Method | Size | LLM | Refer. | Temporal | Spatial | Count | Attr. | Feed. | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Random | - | - | 20.0 | 20.0 | 27.0 | 20.0 | 20.0 | 50.0 | 26.2 |
| Proprietary Models | |||||||||
| GPT-5 [ 30 ] | - | - | 75.6 | 53.6 | 62.3 | 50.0 | 56.1 | 77.8 | 62.6 |
| GPT-4o [ 20 ] | - | - | 56.1 | 29.5 | 43.1 | 44.8 | 41.5 | 65.7 | 46.8 |
| Open-source MLLMs 32B | |||||||||
| Qwen3-VL [ 37 ] | 32B | Qwen3 | 63.7 | 67.9 | 65.8 | 66.7 | 63.4 | 77.2 | 67.5 |
Table 1: Performance of different MLLMs on the EgoPointVQA test set. We report multiple-choice accuracy (%). HINT (highlighted in light blue) consistently improves its corresponding open-source backbones and outperforms all compared baselines. Task categories are Reference (Refer.), Temporal, Spatial, Counting (Count), Attribute (Attr.), and Feedback (Feed.). The random baseline reflects the varying number of answer choices across tasks.
| SFT | Hand Int. | Reference | Temporal | Spatial | Attribute |
| ✗ | ✗ | 66.1 | 57.5 | 63.2 | 51.3 |
| ✓ | ✗ | 68.5 | 60.7 | 59.6 | 56.7 |
| ✓ | ✓ | 75.0 | 66.1 | 64.9 | 61.0 |
Table 2: Ablation of HINT components. ‘SFT’ denotes supervised fine-tuning on EgoPointVQA . ‘Hand Int.’ denotes use of our Hand Intent Token. Combining both yields the largest gains.
| Real | Synthetic | Reference | Temporal | Spatial | Attribute |
| ✗ | ✓ | 69.0 | 62.5 | 60.5 | 58.5 |
| ✓ | ✗ | 67.3 | 60.7 | 56.1 | 56.1 |
| ✓ | ✓ | 75.0 | 66.1 | 64.9 | 61.0 |
Table 3: Impact of training dataset. We vary the usage of synthetic and real videos in the training set. Using a synthetic dataset to complement a real dataset yields the best result.
| Hand Intent Modeling | Reference | Temporal | Spatial |
|---|---|---|---|
| None | 68.5 | 60.7 | 59.6 |
| Visual Keypoints | 57.1 | 60.7 | 61.4 |
| Visual Arrow from Fingertip | 70.2 | 60.7 | 62.3 |
| 3D Keypoints in Text | 68.5 | 55.4 | 58.8 |
| 2D Keypoints in Text | 69.0 | 57.1 | 59.6 |
Table 4: Different methods of hand intent modeling. We compare different methods to encode the user’s hand pose. Our HINT with learning keypoint adapter outperforms alternative representations, such as visual prompts and textual inputs.
| Reference | Temporal | Spatial | Count | Attribute | |
|---|---|---|---|---|---|
| 0.1 | 66.7 | 58.7 | 62.9 | 32.4 | 59.3 |
| 0.3 | 75.0 | 58.7 | 62.9 | 35.3 | 59.3 |
| 0.5 | 75.0 | 66.1 | 64.9 | 35.4 | 61.0 |
| 0.7 | 64.9 | 60.9 | 62.9 | 32.4 | 59.3 |
| 0.9 | 68.4 | 63.1 | 60.0 | 32.4 | 59.3 |
Table 5: Impact of hand detection confidence threshold on HINT . The threshold controls a trade-off between filtering noisy detections and retaining valid pointing gestures. A value of 0.5 achieves the best overall performance.
| Method | Input Frames | Reference | Spatial | Attribute |
|---|---|---|---|---|
| InternVL3-8B | uniform 32 | 66.1 | 63.2 | 51.3 |
| InternVL3-8B | keyframes | 61.3 | 62.3 | 61.0 |
| InternVL3-14B | uniform 32 | 63.1 | 61.4 | 58.5 |
| InternVL3-14B | keyframes | 60.1 | 56.1 | 61.2 |
Table 6: Impact of input frames. We compare uniform 32-frame input with keyframes manually selected as oracle, where each pointing gesture is most clearly visible (one frame per gesture). y Even with this oracle advantage, Reference drops by 4.8/3.0 pp (8B/14B) compared to uniform 32-frame input.
| Backbone | r | a | lr | Resolution |
|---|---|---|---|---|
| InternVL3-8B | 64 | 128 | 1e-5 | 448 448 |
| InternVL3-14B | 32 | 64 | 2e-5 | 448 448 |
| LLaVA-OneVision-7B | 32 | 64 | 1e-5 | 384 384 |
Table 7: Hyperparameters for HINT per backbone. This table summarizes the optimized training configurations. Here, ‘r’ denotes the LoRA rank, ‘a’ the LoRA scaling factor, and ‘lr’ the learning rate used during finetuning.
| Method | Video-MME | MVBench | EgoSchema | EgoBlind |
|---|---|---|---|---|
| InternVL3-8B | 64.2 | 73.2 | 67.2 | 52.8 |
| HINT | 64.6 | 73.2 | 67.1 | 57.5 |
Table 8: Performance on existing video understanding benchmarks. Comparison of HINT against the baseline InternVL3-8B on Video-MME [ 15 ] , MVBench [ 26 ] , EgoSchema [ 28 ] (MCQ), and EgoBlind [ 11 ] . The results indicate that finetuning on our dataset preserves the model’s general video understanding capabilities.
| Ref. | Temp. | Spat. | Count | Attr. | Feed. | |
| Human | 98.2 | 94.3 | 93.3 | 92.8 | 96.5 | 100.0 |
Table 9: Human performance on EgoPointVQA . We report the average accuracy of 5 human participants, each evaluating the full test set (672 questions).
| Video | Reference | Temporal | Spatial | Attribute |
|---|---|---|---|---|
| Original | 75.0 | 66.1 | 64.9 | 61.0 |
| w/o Hand | 41.7 | 21.4 | 44.4 | 36.3 |
Table 10: Ablation study on the effect of hand gestures. Performance comparison on our dataset with and without the pointing hand visible in the video frames. The significant drop in performance across all tasks in the ‘w/o Hand’ setting confirms that the pointing gesture is essential for identifying the target.
| Reference | Temporal | Spatial | Count | Attribute | Feedback | |
|---|---|---|---|---|---|---|
| Random | 20.0 | 20.0 | 27.0 | 20.0 | 20.0 | 50.0 |
| Blind | 16.7 | 25.5 | 32.0 | 6.3 | 22.5 | 54.9 |
| Choices-only | 20.8 | 25.5 | 20.8 | 10.4 | 21.6 | 54.1 |
Table 11: Dataset bias analysis. We evaluate two text-only baselines to detect potential shortcuts: “Blind” receives the question text without video, and “Choices-only” receives only the answer options without the question or video. Performance near random chance confirms that EgoPointVQA requires visual grounding.
Figure 7: Representative failure cases on EgoPointVQA . (a)-(b): baseline MLLM failures due to saliency bias and temporal confusion, respectively. (c)-(d): remaining HINT failures caused by unreliable hand keypoints and rapid viewpoint drift.
| Data Split | # Vid. | # QA | Vid. Dur.(s) | # Obj. |
|---|---|---|---|---|
| Train(Real) | 100 | 640 | 4.61 | 22.5 |
| Train(Synthetic) | 4,000 | 18,745 | 11.6 | 11.6 |
| Test (Real) | 300 | 672 | 5.05 | 16.5 |
Table 12: EgoPointVQA statistics. Statistics of the dataset across synthetic and real-world splits. Real-world clips generally feature higher object density (scene complexity) than synthetic clips.
Figure 8: Spatial distribution of target objects in EgoPointVQA . ‘x’ and ‘y’ axes refer to the horizontal and vertical pixel coordinates, respectively. Individual object locations are shown as scattered dots.
Figure 9: Prompt used for dense scene information extraction (Stage 1). Instructions given to the MLLM to extract a structured JSON list of all salient objects.
Figure 10: General template for QA pair generation (Stage 2-1). The base instruction template is used for all question types. It enforces that the ”correct answer” must be derived exclusively from the scene JSON provided in Stage 1 to prevent hallucinations.
Figure 11: Task-specific prompt for Reference QA generation. This task directs the model to generate questions that require identifying the specific description or category of the pointed-at object (e.g., “What is this?”).
Figure 12: Task-specific prompt for Attribute QA generation. This module focuses on fine-grained visual details, instructing the model to query properties such as color, shape, material, or state (e.g., “Is this sliced?”).
Figure 13: Task-specific prompt for Spatial QA generation. This task generates questions regarding the relative position or depth of the target object compared to other landmarks in the scene (e.g., “Is this to the left of the sink?”).
Figure 14: Task-specific prompt for Feedback QA generation. This task tests reasoning about user intent and affordance, generating questions about whether a specific object is suitable for a stated goal (e.g., “I am thirsty. Can I drink this?”).
Figure 15: Task-specific prompt for Counting QA generation. This task tests visual search capabilities by asking the model to enumerate instances of the pointed-at object category visible in the scene.
Figure 16: Task-specific prompt for Temporal QA generation. This task challenges the model’s ability to recall the chronological sequence of pointing gestures (e.g., “What is the second object I pointed at?”).
Figure 17: Prompt used for negative choices generation (Stage 2-2). Instructions for generating negative choices. The model is prompted to select attributes from neighboring objects or plausible but incorrect properties to create challenging multiple-choice options.
Figure 18: Prompt used for deictic question rephrasing (Stage 3). The final prompt in the pipeline, where GPT-4o acts as a linguistic expert to convert structured queries into natural, spoken questions. It replaces explicit object names with deictic pronouns (“this”, “that”) to ensure the question requires visual grounding to be answered.
Figure 19: Examples from real-world indoor scenes. Samples from the real-world split of the dataset, captured in indoor environments. Each row displays the sample video frames alongside the corresponding question-answer pair generated by our pipeline. The correct answer is bolded and underlined within the options.
Figure 20: Examples from real-world outdoor scenes. Samples from the real-world split of the dataset, captured in outdoor environments. Each row displays the sample video frames alongside the corresponding question-answer pair generated by our pipeline. The correct answer is bolded and underlined within the options.
Figure 21: Examples from synthetic training data. Samples from the large-scale synthetic training set generated via AI2-THOR.
More in this topic
Egocentric Video QA
Egocentric video question answering uses first-person camera footage to answer grounded questions about activities, objects, space, and time.
See all 46 papers