Organizations: Hong Kong University of Science and Technology (Guangzhou) · Sun Yat-sen University · King Abdullah University of Science and Technology · Great Bay University
Egocentric video provides a natural modality for studying human behavior, but conventional visual understanding captures mainly observable scenes, objects, and actions rather than the latent goals that organize them. Existing intent benchmarks typically focus on coarse event-level goals and overlook how intent evolves across procedural steps. We introduce EgoIntent, a step-level intent-understanding benchmark comprising 3,014 pre-outcome micro-steps from 32 egocentric videos across 15 indoor and outdoor daily-life scenarios. Each step is manually annotated along three complementary dimensions: Local Intent (What), the immediate goal; Procedural Intent (Why), the role of the step in the broader procedure; and Next-Plan (Next), the action most likely to follow. Multiple rounds of human review refine temporal boundaries and annotation quality. We evaluate 15 multimodal large language models using reference-based scores and complementary reference-free diagnostics. Controlled studies on four representative models show that only one model gains significantly from correct temporal order, while a single boundary frame outperforms the full ordered clip for three models. Step-only input performs best for all four models, and adding 15 seconds of history significantly degrades three. Revealing the current outcome improves Local Intent by 7.81 points, while revealing the following step improves Next-Plan by 13.17 points. These findings indicate that current models can achieve strong intent-prediction scores through static boundary cues without robustly exploiting temporal order or procedural history.
Figures & tables
Figure 1: Overall performance and intent-quality diagnostics across 15 MLLMs. Blue bars report the reference-based Overall Score (OS) against the left MLLMs Score axis (0–100). The four curves report reference-free Visual Grounding Faithfulness (GF), Hierarchical Intent Consistency (HIC), Temporal Progression Consistency (TPC), and Next-Plan Feasibility (NPF) against the right Intent Quality Score axis (0–90). Models are ordered by OS; the dashed red box marks the three highest-OS systems, and the endpoint annotations give the leading model’s scores. OS and the diagnostics use different Judges and evaluation targets, so they are complementary rather than directly comparable.
Figure 2: EgoIntent data construction pipeline and benchmark composition. The left panel follows five stages: (1) select coherent procedural source videos; (2) manually segment goal-directed micro-steps; (3) retain only the visible prefix Vi from step start tis to the pre-outcome cutoff tio , hiding the outcome-revealing tail and all later frames; (4) annotate Local Intent (What) , Procedural Intent (Why) , the observed Next-Plan (Next) , and plausible alternatives; and (5) conduct multi-round human review of temporal boundaries, label consistency, and ambiguous futures. The right panel summarizes the resulting benchmark: 3,014 micro-steps, each at most 10 seconds, from 32 source videos across 15 indoor and outdoor scenarios.
Figure 3: Scenario-level composition of EgoIntent. Each sector is proportional to the number of annotated micro-steps in one of 15 scenarios; labels report the count and share of all 3,014 micro-steps. Scenario size ranges from 90 steps (3.0%) to 459 (15.2%), and the four largest scenarios together comprise 49.3% of the benchmark.
Model
Reference-Based Evaluation
Reference-Free Diagnostics
Local
Procedural
Next
Overall
GF
HIC
TPC
NPF
Closed-Source MLLMs
Doubao-Seed-2.1-Turbo [ 4 ]
57.98
63.62
43.78
55.13
74.47
85.19
78.86
76.75
Qwen3.5-Plus [ 33 ]
57.99
57.76
42.38
52.71
73.30
81.62
76.97
75.52
Gemini-3.5-Flash [ 16 ]
43.34
48.31
32.72
41.45
58.79
70.24
62.66
60.66
Amazon-Nova-2-Lite-V1 [ 1 ]
37.98
42.96
22.84
34.59
55.05
63.75
56.77
54.93
Table 1: Full-benchmark performance of 15 MLLMs on EgoIntent ( N=3,014 ). The left block reports reference-based semantic scores, whereas the right block reports complementary reference-free diagnostics. The two blocks use different Judges and are not directly comparable. Overall averages Local , Procedural , and Next . Bold marks within-group column maxima.
A. Four observation conditions
Condition
Local
Proc.
Next
Overall
Early
45.59
51.86
35.55
44.33
Official
49.80
55.21
39.30
48.10
Outcome Visible
57.61
57.09
41.33
52.01
Next Visible
55.23
57.47
52.47
55.06
Table 2: Pre-outcome leakage analysis on 508 human-validated paired samples. Scores average four models. Panel A compares observation conditions; Panel B reports target-versus-nontarget gains (L/P/N: Local / Procedural / Next ). Bootstrap confidence intervals are clustered by source video.
Model
Temporal
Motion
Visual Evidence
Gain
95% CI
Gain
95% CI
Gain
95% CI
Doubao-Seed-2.1-Turbo
−0.31
[−2.34,+1.69]
−1.36
[−4.53,+2.00]
+40.96
[+36.63,+44.66]
Qwen3-VL-32B
+0.56
[−1.28,+2.46]
−7.41
[−9.45,−5.28]
+33.52
[+27.60,+39.50]
Molmo2-8B
+2.38
[+0.75,+4.05]
−2.70
[−4.95,−0.13]
+34.73
[+29.71,+39.03]
Qwen3.5-Plus
+1.43
[−0.68,+3.39]
−3.08
[−5.44,−0.62]
+35.75
[+29.94,+40.80]
Table 3: Paired shortcut diagnostics on the 640-step subset. Temporal, Motion, and Visual Evidence Gain denote Ordered − Shuffled, Ordered − Last Frame, and Ordered − Scene Only, respectively. Values include source-video-clustered 95% confidence intervals; bold denotes intervals excluding zero (effective paired N=639/639/598 ).
A. Human–model performance
Evaluator
Local
Proc.
Next
Overall
Gap
Human mean
78.3
73.1
60.8
70.7
–
Doubao
59.7
56.2
45.5
53.8
−16.9
Qwen3-VL-32B
48.8
44.3
36.5
43.2
−27.5
Table 4: Human performance and task validity on the balanced 320-step subset. Panel A compares the three-human mean with the strongest closed- and open-model baselines on identical steps. Panel B reports answerability (Ans.), chance-corrected Fleiss κ , mean confidence (Conf.; 1–5), and pairwise semantic agreement (Sem.; 0–1); higher is better.
Figure 4: Context-window effects on the 640-step diagnostic subset. Overall scores for four MLLMs under Step, Hist.-5s, Hist.-15s, and Prev.-Step inputs with a shared observation endpoint.
A. Human inter-rater reliability
Dimension
ICC (2,k)
95% CI
α
Local
0.87
[0.84,0.90]
0.78
Procedural
0.83
[0.79,0.87]
0.73
Next
0.78
[0.72,0.83]
0.69
Mean
0.83
–
0.73
Table 5: Human and automatic-Judge validation on 150 steps (300 triplets; three raters each). Panel A reports ICC (2,k) and Krippendorff’s α ; Panel B reports Spearman ρ , MAE, and Judge-minus-human bias against human consensus. Overall is computed per triplet.
Egocentric video understanding studies how AI interprets first-person video to recognize actions, track interactions, reason about space and time, and assist users.
Video reasoning models are a core component of egocentric and embodied agents. However, standard benchmarks for assessing models provide only evaluation of the output (e.g. the answer to a question), without evaluation of intermediate reasoning steps, and most provide answers only in the text domain. We introduce Minerva-Ego, a benchmark for evaluating complex egocentric visual reasoning. We extend recent high-quality video data sources recorded from egocentric / embodied settings with a set of challenging, multi-step multimodal questions and spatiotemporally-dense human-annotated reasoning traces. Benchmarking experiments show that state-of-the-art models still have a large gap to human performance. To investigate this gap in detail, we annotate each reasoning trace in the dataset with the objects of interest required to solve the question, as spatiotemporal mask annotations. Through extensive evaluations, we identify that prompting frontier models with hints of 'where' and 'when' to look yields substantial improvements in performance. Minerva-Ego can be downloaded at https://github.com/google-deepmind/neptune.
Egocentric video understanding spans diverse capabilities, modalities, and evaluation formats, yet these are often studied in isolation despite their tight coupling in real-world interaction. We propose the Ego-Action-World interaction loop as a unifying principle: the wearer perceives the world, forms goals, acts, changes the world, and uses feedback to guide subsequent behavior. Based on this view, we introduce EgoOmniWorld, a benchmark organized around four complementary interaction structures: Node captures ego, action, and world states; Edge captures directed relations among them; Chain captures linked dependencies; and Loop captures feedback-closed interactions. We develop an interaction-first, evidence-grounded data pipeline that extracts structured interactions from egocentric videos, grounds them in visual and auditory evidence, and generates questions according to their interaction structure. Required modalities and single- or multi-turn formats follow naturally from the supporting evidence and dependencies. EgoOmniWorld contains 24,166 QA groups, including 20,284 training and 3,882 test samples. Experiments on representative open- and closed-source multimodal large language models reveal substantial and uneven limitations across interaction structures. Models trained on EgoOmniWorld improve in-domain interaction understanding and generalize to existing egocentric benchmarks. EgoOmniWorld provides a unified foundation for evaluating and advancing egocentric video understanding. Data, code, and models will be released.
The rapid development of Multimodal Large Language Models (MLLMs) has led to growing interest in egocentric video understanding, specifically the ability for MLLMs to recognize fine-grained hand-object interactions, track object state changes over time, and reason about manipulative processes in dynamic environments from a first-person perspective. However, existing egocentric video benchmarks suffer from limited grounded rationale evaluation, offering limited support for fine-grained operation-centric reasoning and rarely examining whether model rationales are grounded in explicit spatio-temporal evidence. To address this gap, we introduce EgoCoT-Bench, a fine-grained egocentric benchmark for grounded and verifiable operation-centric reasoning with explicit step-by-step rationale annotations. Overall, EgoCoT-Bench comprises 3,172 verifiable QA pairs over 351 egocentric videos separated into four task groups for a total of 12 sub-task groups, encompassing perception and retrospection, anticipation, and high-level reasoning. The benchmark is constructed through a spatio-temporal scene graphs (STSG) guided generation framework and is further refined by human annotators to ensure correctness, egocentric relevance and fine-grained quality. Experimental results show continuing difficulties with egocentric fine-grained reasoning and further reveal that many multimodal models produce explanations that are answer-correct, but have evidence that is inconsistent with the answer. We hope EgoCoT-Bench can serve as a useful testbed for grounded and verifiable reasoning in egocentric video understanding. Project page and supplementary materials are available at: https://dstardust.github.io/EgoCoT/.