Listening with the Eyes: Benchmarking Egocentric Co-Speech Grounding across Space and Time
Authors: Weijie Zhou, Xuantang Xiong, Zhenlin Hu, Xiaomeng Zhu, Chaoyang Zhao, Honghui Dong, Zhengyou Zhang, Ming Tang, Jinqiao Wang
Organizations: Beijing Jiaotong University · Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences (CASIA) · Tencent Robotics X · Harbin Institute of Technology, Shenzhen · Department of Computer Science and Engineering, The Hong Kong University of Science and Technology (HKUST)
In situated collaboration, speakers often use intentionally underspecified deictic commands (e.g., ``pass me that''), whose referent becomes identifiable only by aligning speech with a brief co-speech pointing stroke. However, many embodied benchmarks admit language-only shortcuts, allowing MLLMs to perform well without learning the audio--visual alignment required by deictic interaction. To bridge this gap, we introduce Egocentric Co-Speech Grounding (EcoG), where grounding is executable only if an agent jointly predicts What, Where, and When. To operationalize this, we present EcoG-Bench, an evaluation-only bilingual (EN/ZH) diagnostic benchmark of 811 egocentric clips with dense spatial annotations and millisecond-level stroke supervision. It is organized under a Progressive Cognitive Evaluation protocol. Benchmarking state-of-the-art MLLMs reveals a severe executability gap: while human subjects achieve near-ceiling performance on EcoG-Bench (96.9% strict Eco-Accuracy), the best native video-audio setting remains low (Gemini-3-Pro: 17.0%). Moreover, in a diagnostic ablation, replacing the native video--audio interface with timestamped frame samples and externally verified ASR (with word-level timing) substantially improves the same model (17.0%→42.9%). Overall, EcoG-Bench provides a strict, executable testbed for event-level speech--gesture binding, and suggests that multimodal interfaces may bottleneck the observability of temporal alignment cues, independently of model reasoning.
Figures & tables
Figure 1: From text-sufficient grounding to deictic co-speech event binding. Left: In many existing embodied/grounding benchmarks, the instruction is semantically exhaustive (e.g., attributes and spatial relations), so the correct referent can be inferred from text alone and co-speech gesture video is largely optional. Right: EcoG models natural deictic collaboration, where utterances are intentionally underspecified (e.g., “put this in it ”) and become solvable only by aligning each deictic phrase to a brief co-speech pointing stroke on the video timeline. Successful EcoG grounding requires within-clip event assignment : binding each phrase to the correct stroke, then producing an executable intent for every step ( What target, Where actionable 2D point, and When stroke time).
Dataset
Egocentric
Audio
Deictic Dominance
Gesture Stroke
Task Annotation
RefCOCO/+/g [ 19 ]
✗
✗
✗
✗
Box/Mask
HC-STVG [ 36 ]
✗
✓
✗
✗
Time + Box
YouRefIt [ 8 ]
✓
✓
✗
✓ (Gesture, coarse)
Box/Mask
Ego4D (FHO) [ 16 ]
✓
✓
✗
✗
Time + Box
ToG-Bench [ 40 ]
✓
✗
✗
✗
Time + Box
EcoG (Ours)
✓
✓
✓
✓ (Stroke, ms-level)
Class + Mask + Time
Table 1: Comparison of EcoG with related grounding and interaction datasets. EcoG uniquely integrates egocentric vision, audio, deictic ambiguity, and precise gesture stroke annotations.
Figure 2: EcoG task overview. Given an egocentric video clip with synchronized audio, the model must ground each deictic referent in the instruction by outputting an ordered list of triplets: What (an index in a clip-specific closed-set of candidate options), Where (a 2D point on the last frame, ensuring an actionable “landing point”), and When (an integer timestamp in milliseconds from clip start that must fall inside the annotated gesture-stroke window that disambiguates the referent).
Figure 3: Progressive Cognitive Evaluation protocol and dataset composition. EcoG-Bench organizes 811 egocentric clips (EN/ZH) into four levels with increasing compositionality and event-assignment difficulty: L1 silent deictic pointing (K=1), L2 single-event co-speech binding (K=1), L3 dual-event deictic assignment (K=2), and L4 multi-event intent chaining (K=3–4). The figure illustrates the corresponding instruction templates and the increasing requirement to assign each deictic phrase to the correct within-clip gesture stroke.
Model
L1
L2
L3
L4
Overall
eco
seq
cls
eco
seq
cls
eco
seq
cls
eco
seq
cls
eco
seq
cls
Human Performance
98.2
98.2
99.1
97.5
97.5
98.8
97.2
96.7
98.5
96.4
95.2
98.1
96.9
96.2
97.3
Native Omni Models (Video-Omni: video+audio)
Gemini-3-Pro [ 10 ]
30.2
30.2
79.9
29.2
29.2
71.5
10.6
1.8
51.9
10.2
0.4
64.5
17.0
10.9
63.9
Gemini-3-Flash [ 10 ]
12.2
12.2
71.2
10.2
10.2
69.3
3.2
0.7
65.3
6.6
0.0
79.5
7.0
4.1
71.4
Qwen3-Omni-30B-A3B [ 39 ]
3.6
3.6
40.3
0.7
0.7
48.9
0.0
0.0
23.9
0.0
0.0
19.1
0.7
0.7
29.5
Table 2: EcoG-Bench results under each model class’s native interface. We report referent-level Eco-Accuracy ( Acceco ), clip-level sequence success ( Accseq ), and classification accuracy ( Acccls ) across levels L1–L4 and overall. Acceco is conjunctive: for all levels, a referent is correct only if What ∧ Where ∧ When are correct. Bold/underline indicate the best/second result per column within each model family (Omni vs. VL), excluding Human.
Figure 4: Qualitative results on EcoG-Bench. Examples of model predictions versus ground truth under strict What/Where/When evaluation. The shown cases highlight typical failure modes: correct recognition but inaccurate pointing on small/occluded objects, and mis-binding a deictic phrase to a nearby (but incorrect) stroke event—both of which render the output non-executable under conjunctive Eco-Accuracy.
Figure 5: Failure bottleneck analysis of EcoG . Breakdown of errors by which components of the executable grounding triplet fail (What, Where, When) and their combinations. Joint failures (e.g., Where+When) constitute a large portion of errors, indicating that EcoG difficulty is dominated by cross-modal event binding rather than isolated object classification alone.
Model
Acceco (%)
Accseq (%)
Gemini-3-Pro
17.0 → 42.9 (+25.9)
10.9 → 25.5 (+14.6)
Gemini-3-Flash
7.0 → 48.1 (+41.1)
4.1 → 30.8 (+26.7)
Qwen3-Omni-Flash
0.7 → 9.3 (+8.6)
0.6 → 6.5 (+5.9)
Table 3: Input-stack diagnosis (Overall): Images+ASR vs. native Video-Omni. We report strict executability metrics as Video-Omni → Images+ASR ( Δ ).
Model
Condition
L1
L2
L3
L4
Overall
t
eco
t
eco
t
eco
t
eco
t
eco
Full Anchors
94.2 (–)
66.9 (–)
91.2 (–)
65.7 (–)
77.1 (–)
40.0 (–)
75.0 (–)
37.3 (–)
81.8 (–)
48.1 (–)
Gemini-3-Flash
No Frame Timestamps
15.8 (-78.4)
9.4 (-57.5)
85.4 (-5.8)
56.2 (-9.5)
70.1 (-7.0)
30.1 (-9.9)
67.8 (-7.2)
28.4 (-8.9)
62.7 (-19.1)
30.4 (-17.7)
No Word-level ASR Timing
92.8 (-1.4)
66.9 (0.0)
84.7 (-6.5)
61.3 (-4.4)
75.2 (-1.9)
38.2 (-1.8)
66.7 (-8.3)
34.5 (-2.8)
77.2 (-4.6)
45.9 (-2.2)
Full Anchors
88.5 (–)
59.0 (–)
86.9 (–)
59.9 (–)
72.9 (–)
34.5 (–)
68.4 (–)
34.2 (–)
76.5 (–)
42.9 (–)
Gemini-3-Pro
No Frame Timestamps
20.9 (-67.6)
14.4 (-44.6)
82.5 (-4.4)
43.1 (-16.8)
67.8 (-5.1)
18.5 (-16.0)
68.2 (-0.2)
17.7 (-16.5)
63.0 (-13.5)
21.7 (-21.2)
Table 4: Temporal anchor ablations (diagnostic-only) under Images+ASR. We report Acct (temporal accuracy) and strict Acceco (executability) for each level and overall. Each cell shows absolute score (top) and Δ vs. Full Anchors (bottom), computed within the same model . Note that L1 contains no speech; thus ASR timing is unavailable and only frame timestamps can provide absolute temporal anchors.