Listening with the Eyes: Benchmarking Egocentric Co-Speech Grounding across Space and Time
Authors: Weijie Zhou, Xuantang Xiong, Zhenlin Hu, Xiaomeng Zhu, Chaoyang Zhao, Honghui Dong, Zhengyou Zhang, Ming Tang, Jinqiao Wang
Organizations: Beijing Jiaotong University · Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences (CASIA) · Tencent Robotics X · Harbin Institute of Technology, Shenzhen · Department of Computer Science and Engineering, The Hong Kong University of Science and Technology (HKUST)
In situated collaboration, speakers often use intentionally underspecified deictic commands (e.g., ``pass me that''), whose referent becomes identifiable only by aligning speech with a brief co-speech pointing stroke. However, many embodied benchmarks admit language-only shortcuts, allowing MLLMs to perform well without learning the audio--visual alignment required by deictic interaction. To bridge this gap, we introduce Egocentric Co-Speech Grounding (EcoG), where grounding is executable only if an agent jointly predicts What, Where, and When. To operationalize this, we present EcoG-Bench, an evaluation-only bilingual (EN/ZH) diagnostic benchmark of 811 egocentric clips with dense spatial annotations and millisecond-level stroke supervision. It is organized under a Progressive Cognitive Evaluation protocol. Benchmarking state-of-the-art MLLMs reveals a severe executability gap: while human subjects achieve near-ceiling performance on EcoG-Bench (96.9% strict Eco-Accuracy), the best native video-audio setting remains low (Gemini-3-Pro: 17.0%). Moreover, in a diagnostic ablation, replacing the native video--audio interface with timestamped frame samples and externally verified ASR (with word-level timing) substantially improves the same model (17.0%→42.9%). Overall, EcoG-Bench provides a strict, executable testbed for event-level speech--gesture binding, and suggests that multimodal interfaces may bottleneck the observability of temporal alignment cues, independently of model reasoning.
Figures & tables
Figure 1: From text-sufficient grounding to deictic co-speech event binding. Left: In many existing embodied/grounding benchmarks, the instruction is semantically exhaustive (e.g., attributes and spatial relations), so the correct referent can be inferred from text alone and co-speech gesture video is largely optional. Right: EcoG models natural deictic collaboration, where utterances are intentionally underspecified (e.g., “put this in it ”) and become solvable only by aligning each deictic phrase to a brief co-speech pointing stroke on the video timeline. Successful EcoG grounding requires within-clip event assignment : binding each phrase to the correct stroke, then producing an executable intent for every step ( What target, Where actionable 2D point, and When stroke time).
Dataset
Egocentric
Audio
Deictic Dominance
Gesture Stroke
Task Annotation
RefCOCO/+/g [ 19 ]
✗
✗
✗
✗
Box/Mask
HC-STVG [ 36 ]
✗
✓
✗
✗
Time + Box
YouRefIt [ 8 ]
✓
✓
✗
✓ (Gesture, coarse)
Box/Mask
Ego4D (FHO) [ 16 ]
✓
✓
✗
✗
Time + Box
ToG-Bench [ 40 ]
✓
✗
✗
✗
Time + Box
EcoG (Ours)
✓
✓
✓
✓ (Stroke, ms-level)
Class + Mask + Time
Table 1: Comparison of EcoG with related grounding and interaction datasets. EcoG uniquely integrates egocentric vision, audio, deictic ambiguity, and precise gesture stroke annotations.
Figure 2: EcoG task overview. Given an egocentric video clip with synchronized audio, the model must ground each deictic referent in the instruction by outputting an ordered list of triplets: What (an index in a clip-specific closed-set of candidate options), Where (a 2D point on the last frame, ensuring an actionable “landing point”), and When (an integer timestamp in milliseconds from clip start that must fall inside the annotated gesture-stroke window that disambiguates the referent).
Figure 3: Progressive Cognitive Evaluation protocol and dataset composition. EcoG-Bench organizes 811 egocentric clips (EN/ZH) into four levels with increasing compositionality and event-assignment difficulty: L1 silent deictic pointing (K=1), L2 single-event co-speech binding (K=1), L3 dual-event deictic assignment (K=2), and L4 multi-event intent chaining (K=3–4). The figure illustrates the corresponding instruction templates and the increasing requirement to assign each deictic phrase to the correct within-clip gesture stroke.
Model
L1
L2
L3
L4
Overall
eco
seq
cls
eco
seq
cls
eco
seq
cls
eco
seq
cls
eco
seq
cls
Human Performance
98.2
98.2
99.1
97.5
97.5
98.8
97.2
96.7
98.5
96.4
95.2
98.1
96.9
96.2
97.3
Native Omni Models (Video-Omni: video+audio)
Gemini-3-Pro [ 10 ]
30.2
30.2
79.9
29.2
29.2
71.5
10.6
1.8
51.9
10.2
0.4
64.5
17.0
10.9
63.9
Gemini-3-Flash [ 10 ]
12.2
12.2
71.2
10.2
10.2
69.3
3.2
0.7
65.3
6.6
0.0
79.5
7.0
4.1
71.4
Qwen3-Omni-30B-A3B [ 39 ]
3.6
3.6
40.3
0.7
0.7
48.9
0.0
0.0
23.9
0.0
0.0
19.1
0.7
0.7
29.5
Table 2: EcoG-Bench results under each model class’s native interface. We report referent-level Eco-Accuracy ( Acceco ), clip-level sequence success ( Accseq ), and classification accuracy ( Acccls ) across levels L1–L4 and overall. Acceco is conjunctive: for all levels, a referent is correct only if What ∧ Where ∧ When are correct. Bold/underline indicate the best/second result per column within each model family (Omni vs. VL), excluding Human.
Figure 4: Qualitative results on EcoG-Bench. Examples of model predictions versus ground truth under strict What/Where/When evaluation. The shown cases highlight typical failure modes: correct recognition but inaccurate pointing on small/occluded objects, and mis-binding a deictic phrase to a nearby (but incorrect) stroke event—both of which render the output non-executable under conjunctive Eco-Accuracy.
Figure 5: Failure bottleneck analysis of EcoG . Breakdown of errors by which components of the executable grounding triplet fail (What, Where, When) and their combinations. Joint failures (e.g., Where+When) constitute a large portion of errors, indicating that EcoG difficulty is dominated by cross-modal event binding rather than isolated object classification alone.
Model
Acceco (%)
Accseq (%)
Gemini-3-Pro
17.0 → 42.9 (+25.9)
10.9 → 25.5 (+14.6)
Gemini-3-Flash
7.0 → 48.1 (+41.1)
4.1 → 30.8 (+26.7)
Qwen3-Omni-Flash
0.7 → 9.3 (+8.6)
0.6 → 6.5 (+5.9)
Table 3: Input-stack diagnosis (Overall): Images+ASR vs. native Video-Omni. We report strict executability metrics as Video-Omni → Images+ASR ( Δ ).
Model
Condition
L1
L2
L3
L4
Overall
t
eco
t
eco
t
eco
t
eco
t
eco
Full Anchors
94.2 (–)
66.9 (–)
91.2 (–)
65.7 (–)
77.1 (–)
40.0 (–)
75.0 (–)
37.3 (–)
81.8 (–)
48.1 (–)
Gemini-3-Flash
No Frame Timestamps
15.8 (-78.4)
9.4 (-57.5)
85.4 (-5.8)
56.2 (-9.5)
70.1 (-7.0)
30.1 (-9.9)
67.8 (-7.2)
28.4 (-8.9)
62.7 (-19.1)
30.4 (-17.7)
No Word-level ASR Timing
92.8 (-1.4)
66.9 (0.0)
84.7 (-6.5)
61.3 (-4.4)
75.2 (-1.9)
38.2 (-1.8)
66.7 (-8.3)
34.5 (-2.8)
77.2 (-4.6)
45.9 (-2.2)
Full Anchors
88.5 (–)
59.0 (–)
86.9 (–)
59.9 (–)
72.9 (–)
34.5 (–)
68.4 (–)
34.2 (–)
76.5 (–)
42.9 (–)
Gemini-3-Pro
No Frame Timestamps
20.9 (-67.6)
14.4 (-44.6)
82.5 (-4.4)
43.1 (-16.8)
67.8 (-5.1)
18.5 (-16.0)
68.2 (-0.2)
17.7 (-16.5)
63.0 (-13.5)
21.7 (-21.2)
Table 4: Temporal anchor ablations (diagnostic-only) under Images+ASR. We report Acct (temporal accuracy) and strict Acceco (executability) for each level and overall. Each cell shows absolute score (top) and Δ vs. Full Anchors (bottom), computed within the same model . Note that L1 contains no speech; thus ASR timing is unavailable and only frame timestamps can provide absolute temporal anchors.
Traditional Visual Grounding (VG) predominantly relies on textual descriptions to localize objects, a paradigm that inherently struggles with linguistic ambiguity and often ignores non-verbal deictic cues prevalent in real-world interactions. In natural egocentric engagements, hand-pointing combined with speech forms the most intuitive referring mechanism. To bridge this gap, we introduce EgoPoint-Ground, the first large-scale multimodal dataset dedicated to egocentric deictic visual grounding. Comprising over 15k interactive samples in complex scenes, the dataset provides rich, multi-grained annotations including hand-target bounding box pairs and dense semantic captions. We establish a comprehensive benchmark for hand-pointing referring expression resolution, evaluating a wide spectrum of mainstream Multimodal Large Language Models (MLLMs) and state-of-the-art VG architectures. Furthermore, we propose SV-CoT, a novel baseline framework that reformulates grounding as a structured inference process, synergizing gestural and linguistic cues through a Visual Chain-of-Thought paradigm. Extensive experiments demonstrate that SV-CoT achieves an 11.7% absolute improvement over existing methods, effectively mitigating semantic ambiguity and advancing the capability of agents to comprehend multimodal physical intents. The dataset and code will be made publicly available.
Ling Li, Bowen Liu, Zinuo Zhan +4
Tsinghua University, Beijing, China · Dalian University of Technology, Dalian, China · Northwestern Polytechnical University, Xi’an, China +1
The rapid development of Multimodal Large Language Models (MLLMs) has led to growing interest in egocentric video understanding, specifically the ability for MLLMs to recognize fine-grained hand-object interactions, track object state changes over time, and reason about manipulative processes in dynamic environments from a first-person perspective. However, existing egocentric video benchmarks suffer from limited grounded rationale evaluation, offering limited support for fine-grained operation-centric reasoning and rarely examining whether model rationales are grounded in explicit spatio-temporal evidence. To address this gap, we introduce EgoCoT-Bench, a fine-grained egocentric benchmark for grounded and verifiable operation-centric reasoning with explicit step-by-step rationale annotations. Overall, EgoCoT-Bench comprises 3,172 verifiable QA pairs over 351 egocentric videos separated into four task groups for a total of 12 sub-task groups, encompassing perception and retrospection, anticipation, and high-level reasoning. The benchmark is constructed through a spatio-temporal scene graphs (STSG) guided generation framework and is further refined by human annotators to ensure correctness, egocentric relevance and fine-grained quality. Experimental results show continuing difficulties with egocentric fine-grained reasoning and further reveal that many multimodal models produce explanations that are answer-correct, but have evidence that is inconsistent with the answer. We hope EgoCoT-Bench can serve as a useful testbed for grounded and verifiable reasoning in egocentric video understanding. Project page and supplementary materials are available at: https://dstardust.github.io/EgoCoT/.
While humans naturally gesture during speech, only a sparse subset of these movements are visually depictive and semantically linked to specific spoken words. Current multimodal models struggle to capture these semantic co-speech gestures, heavily bottlenecked by a lack of precisely annotated training data. To address this, we introduce the Gesture Recognition in the Wild (GRW) dataset, the first large-scale benchmark designed to map unconstrained human gestures to specific words with frame-accurate temporal boundaries. Comprising 156,688 manually annotated video clips, GRW spans a highly diverse 150-word taxonomy of physical actions, spatial descriptors, and abstract concepts. We leverage GRW to train video models to (a) classify gestures as semantic or not, (b) recognize the word corresponding to a co-speech gesture, and (c) temporally localize the gesture. We also use GRW to establish benchmarks for these three tasks.
Sindhu B Hegde, K R Prajwal, Andrew Zisserman
Visual Geometry Group, Dept. of Engineering Science, University of Oxford