Organizations: Hong Kong University of Science and Technology (Guangzhou) · Sun Yat-sen University · King Abdullah University of Science and Technology · Great Bay University
Egocentric video provides a natural modality for studying human behavior, but conventional visual understanding captures mainly observable scenes, objects, and actions rather than the latent goals that organize them. Existing intent benchmarks typically focus on coarse event-level goals and overlook how intent evolves across procedural steps. We introduce EgoIntent, a step-level intent-understanding benchmark comprising 3,014 pre-outcome micro-steps from 32 egocentric videos across 15 indoor and outdoor daily-life scenarios. Each step is manually annotated along three complementary dimensions: Local Intent (What), the immediate goal; Procedural Intent (Why), the role of the step in the broader procedure; and Next-Plan (Next), the action most likely to follow. Multiple rounds of human review refine temporal boundaries and annotation quality. We evaluate 15 multimodal large language models using reference-based scores and complementary reference-free diagnostics. Controlled studies on four representative models show that only one model gains significantly from correct temporal order, while a single boundary frame outperforms the full ordered clip for three models. Step-only input performs best for all four models, and adding 15 seconds of history significantly degrades three. Revealing the current outcome improves Local Intent by 7.81 points, while revealing the following step improves Next-Plan by 13.17 points. These findings indicate that current models can achieve strong intent-prediction scores through static boundary cues without robustly exploiting temporal order or procedural history.
Figures & tables
Figure 1: Overall performance and intent-quality diagnostics across 15 MLLMs. Blue bars report the reference-based Overall Score (OS) against the left MLLMs Score axis (0–100). The four curves report reference-free Visual Grounding Faithfulness (GF), Hierarchical Intent Consistency (HIC), Temporal Progression Consistency (TPC), and Next-Plan Feasibility (NPF) against the right Intent Quality Score axis (0–90). Models are ordered by OS; the dashed red box marks the three highest-OS systems, and the endpoint annotations give the leading model’s scores. OS and the diagnostics use different Judges and evaluation targets, so they are complementary rather than directly comparable.
Figure 2: EgoIntent data construction pipeline and benchmark composition. The left panel follows five stages: (1) select coherent procedural source videos; (2) manually segment goal-directed micro-steps; (3) retain only the visible prefix Vi from step start tis to the pre-outcome cutoff tio , hiding the outcome-revealing tail and all later frames; (4) annotate Local Intent (What) , Procedural Intent (Why) , the observed Next-Plan (Next) , and plausible alternatives; and (5) conduct multi-round human review of temporal boundaries, label consistency, and ambiguous futures. The right panel summarizes the resulting benchmark: 3,014 micro-steps, each at most 10 seconds, from 32 source videos across 15 indoor and outdoor scenarios.
Figure 3: Scenario-level composition of EgoIntent. Each sector is proportional to the number of annotated micro-steps in one of 15 scenarios; labels report the count and share of all 3,014 micro-steps. Scenario size ranges from 90 steps (3.0%) to 459 (15.2%), and the four largest scenarios together comprise 49.3% of the benchmark.
Model
Reference-Based Evaluation
Reference-Free Diagnostics
Local
Procedural
Next
Overall
GF
HIC
TPC
NPF
Closed-Source MLLMs
Doubao-Seed-2.1-Turbo [ 4 ]
57.98
63.62
43.78
55.13
74.47
85.19
78.86
76.75
Qwen3.5-Plus [ 33 ]
57.99
57.76
42.38
52.71
73.30
81.62
76.97
75.52
Gemini-3.5-Flash [ 16 ]
43.34
48.31
32.72
41.45
58.79
70.24
62.66
60.66
Amazon-Nova-2-Lite-V1 [ 1 ]
37.98
42.96
22.84
34.59
55.05
63.75
56.77
54.93
Table 1: Full-benchmark performance of 15 MLLMs on EgoIntent ( N=3,014 ). The left block reports reference-based semantic scores, whereas the right block reports complementary reference-free diagnostics. The two blocks use different Judges and are not directly comparable. Overall averages Local , Procedural , and Next . Bold marks within-group column maxima.
A. Four observation conditions
Condition
Local
Proc.
Next
Overall
Early
45.59
51.86
35.55
44.33
Official
49.80
55.21
39.30
48.10
Outcome Visible
57.61
57.09
41.33
52.01
Next Visible
55.23
57.47
52.47
55.06
Table 2: Pre-outcome leakage analysis on 508 human-validated paired samples. Scores average four models. Panel A compares observation conditions; Panel B reports target-versus-nontarget gains (L/P/N: Local / Procedural / Next ). Bootstrap confidence intervals are clustered by source video.
Model
Temporal
Motion
Visual Evidence
Gain
95% CI
Gain
95% CI
Gain
95% CI
Doubao-Seed-2.1-Turbo
−0.31
[−2.34,+1.69]
−1.36
[−4.53,+2.00]
+40.96
[+36.63,+44.66]
Qwen3-VL-32B
+0.56
[−1.28,+2.46]
−7.41
[−9.45,−5.28]
+33.52
[+27.60,+39.50]
Molmo2-8B
+2.38
[+0.75,+4.05]
−2.70
[−4.95,−0.13]
+34.73
[+29.71,+39.03]
Qwen3.5-Plus
+1.43
[−0.68,+3.39]
−3.08
[−5.44,−0.62]
+35.75
[+29.94,+40.80]
Table 3: Paired shortcut diagnostics on the 640-step subset. Temporal, Motion, and Visual Evidence Gain denote Ordered − Shuffled, Ordered − Last Frame, and Ordered − Scene Only, respectively. Values include source-video-clustered 95% confidence intervals; bold denotes intervals excluding zero (effective paired N=639/639/598 ).
A. Human–model performance
Evaluator
Local
Proc.
Next
Overall
Gap
Human mean
78.3
73.1
60.8
70.7
–
Doubao
59.7
56.2
45.5
53.8
−16.9
Qwen3-VL-32B
48.8
44.3
36.5
43.2
−27.5
Table 4: Human performance and task validity on the balanced 320-step subset. Panel A compares the three-human mean with the strongest closed- and open-model baselines on identical steps. Panel B reports answerability (Ans.), chance-corrected Fleiss κ , mean confidence (Conf.; 1–5), and pairwise semantic agreement (Sem.; 0–1); higher is better.
Figure 4: Context-window effects on the 640-step diagnostic subset. Overall scores for four MLLMs under Step, Hist.-5s, Hist.-15s, and Prev.-Step inputs with a shared observation endpoint.
A. Human inter-rater reliability
Dimension
ICC (2,k)
95% CI
α
Local
0.87
[0.84,0.90]
0.78
Procedural
0.83
[0.79,0.87]
0.73
Next
0.78
[0.72,0.83]
0.69
Mean
0.83
–
0.73
Table 5: Human and automatic-Judge validation on 150 steps (300 triplets; three raters each). Panel A reports ICC (2,k) and Krippendorff’s α ; Panel B reports Spearman ρ , MAE, and Judge-minus-human bias against human consensus. Overall is computed per triplet.
Egocentric video understanding studies how AI interprets first-person video to recognize actions, track interactions, reason about space and time, and assist users.