Understanding an egocentric environment requires reasoning about how surrounding entities affect the observer's actions. Existing evaluations of multimodal foundation models (MFMs) examine observer localization, motion trajectories, and action feasibility, but leave the discovery and interpretation of evolving observer-entity interactions insufficiently assessed. We introduce EgoPass-Bench, a benchmark for observer-relative interaction understanding in real-world egocentric video. EgoPass-Bench comprises 284 manually reviewed evaluation clips recorded using Ray-Ban Meta (Gen 2) smart glasses and 2,201 distinct questions across five tasks assessing motion perception, interaction judgments about specified entities, and relevant-entity discovery without target cues. Our evaluation of eight frame-based and six native-video configurations reveals a pronounced discrepancy between motion perception and interaction understanding. The strongest evaluated model, GPT-6 Astra, correctly recognizes object motion in 91.37% of 394 comparisons involving the same targets, but correctly identifies changes in passage constraints in only 66.50%. Across all 574 entity-discovery questions, it trails the two-person human baseline by 23.65±7.45 percentage points (paired 95% CI), while exceeding the always-empty baseline (36.76%) by at least 18.9 percentage points. Further analysis reveals frequent inclusion of entities that neither pose a plausible physical conflict nor constrain passage, showing that accurate motion recognition does not ensure reliable relevant-entity discovery.
Figures & tables
Figure 1: From motion perception to passage understanding. EgoPass-Bench evaluates how wearer motion and external entities shape passage constraints. Models must separate observed states from their implications for the wearer and identify relevant entities without target cues.
Benchmark
Wearable
View–Path
Cued/Uncued
Rel. Δ
Uncertainty
Set–Label
M–R–D
Paired ctx.
General video
MVBench
△
×
×
×
△
×
×
×
Video-MME
×
×
×
×
×
×
×
×
TempCompass
×
△
×
×
×
×
×
×
TOMATO
×
△
×
×
×
△
×
×
MotionBench
×
△
×
×
×
×
×
×
Table 1: Evaluation design of representative benchmarks. Dimensions: 1 wearable video, 2 viewpoint–path distinction, 3 cued reasoning and uncued entity discovery, 4 changes in passage constraints, 5 insufficient evidence versus irrelevance, 6 separate entity-selection and label scoring, 7 motion–relation–discovery evaluation on shared clips, and 8 paired evaluation with and without visual history. ✓ , △ , and × denote explicit, partial or related, and no explicit coverage. Group colors indicate general video (blue), egocentric video (purple), and safety and proactive assistance (yellow).
Figure 2: Five tasks share the same visual evidence. Timestamped frames support judgments about wearer translation, turning, external object motion, passage relations for a specified entity, and autonomous discovery of all relevant entities.
Multiple choice
Free response
Model
Overall
Wearer translation
Turning
Object motion
Passage relation
Exact
Discovery
Labels
Baselines
Human (mean of two)
85.80
89.89
83.27
94.12
79.93
77.79
79.46
98.49
Always empty
—
—
—
—
—
36.76
36.76
—
Question-only model
GPT-5.6 Sol
45.54
49.81
0 8.46
78.35
37.48
—
—
—
Table 2: Performance on EgoPass-Bench without visual history (%). For visual models, bold and underline mark the best and second-best results. FFG denominators, group-resampled 95% confidence intervals and unresolved-identity bounds are in Appendix C .
Figure 3: Left. The approaching pedestrian yields, but six of eight frame-based models predict greater required attention. Right. Each family averages two frame-based models over 574 FFG questions. (a) Matched (TP) and extra (FP) entities. (b) Discovery on 211 empty and 363 nonempty references. Values are lower bounds; the rightmost “+” values give maximum additions from unresolved identities (entities in a, percentage points in b).
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Questions
Wearer translation
267
Turning
260
External-object motion
425
Specified-object passage relation
675
Relevant-entity discovery (FFG)
574
Total
2,201
Appendix
Table 3: Distribution of the 2,201 questions across the five tasks. All 574 FFG questions form one task, with 211 empty and 363 nonempty reference sets.
Model
Model identifier
Max output tokens
Temp.
Top-P
Top-K
Visual input
Thinking setting
Chronological frames: GPT
GPT-6 Astra ∗
gpt-6-astra
–
–
–
–
2 fps a
Medium
GPT-5.6 Sol
gpt-5.6-sol
–
–
–
–
2 fps a
Medium
Chronological frames: Gemini
Gemini 3.8 Flash
gemini-3.8-flash
16,384
1.0
0.95 c
64 c
2 fps a
Medium
Gemini 3.1 Pro (Preview)
gemini-3.1- pro-preview
16,384
1.0
0.95 c
64 c
2 fps a
Medium
Appendix
Table 4: Model configurations for evaluation. Settings are based on evaluation records and official model or provider documentation. Unset parameters use defaults where applicable. Temp. : temperature.
Model
MCQ overall
Wearer translation
Turning
Object motion
Passage relation
Exact
Discovery
Labels
Chronological frames
GPT-6 Astra
65.52
86.14
43.85
91.29
49.48
48.26
55.73
82.97
GPT-5.6 Sol
57.47
61.05
33.85
86.59
46.81
35.54
44.56
74.46
Gemini 3.8 Flash
60.97
66.67
45.77
87.29
48.00
43.80
52.11
76.09
Gemini 3.1 Pro
57.47
69.66
28.46
84.24
46.96
34.21
45.40
67.27
Llama 4 Maverick
47.20
39.33
38.08
80.00
33.19
14.44
29.46
38.46
Appendix
Table 5: Visual-model scores on the full 2,201-question benchmark (%). Values reproduce Table 2 ; FFG uses the same 574 questions. Point-estimate denominators and identity bounds are in Tables 9 and 8 .
Model
Wearer translation
Turning
Object motion
Passage relation
MCQ correct
FFG valid
All valid
Missing
Chronological frames
GPT-6 Astra
230/267
114/260
388/425
334/675
1066/1627
574/574
2201/2201
0
GPT-5.6 Sol
163/267
88/260
368/425
316/675
935/1627
574/574
2201/2201
0
Gemini 3.8 Flash
178/267
119/260
371/425
324/675
992/1627
574/574
2201/2201
0
Gemini 3.1 Pro
186/267
74/260
358/425
317/675
935/1627
574/574
2201/2201
0
Llama 4 Maverick
105/267
99/260
340/425
224/675
768/1627
573/574
2200/2201
1
Appendix
Table 6: Response counts for all 2,201 questions. Panel (a): correct/planned MCQs, valid/planned responses and unavailable responses. Entity counts continue in panel (b).
Model
TP
FP
FN
Correct labels
Labels (%)
Chronological frames
GPT-6 Astra
278–279
224–225
152–153
231
82.80–83.09
GPT-5.6 Sol
278–279
332–333
152–153
207–208
74.46–74.55
Gemini 3.8 Flash
184–190
132–138
241–247
140–143
74.47–76.47
Gemini 3.1 Pro
224–239
251–266
192–207
149–160
64.22–68.38
Llama 4 Maverick
130–157
305–332
274–301
50–63
33.56–44.06
Appendix
Table 6: Response counts (continued): FFG outcomes for 431 reference entities across 574 questions, including 211 empty and 363 nonempty references. Ranges retain unresolved identity matches; Labels is conditional accuracy (%).
Configuration
Overall
Wearer translation
Turning
Object motion
Passage relation
Chronological frames
GPT-6 Astra
65.5 ± 2.7
86.1 ± 5.1
43.8 ± 6.7
91.3 ± 3.1
49.5 ± 3.7
GPT-5.6 Sol
57.5 ± 2.9
61.0 ± 7.8
33.8 ± 5.9
86.6 ± 4.7
46.8 ± 4.1
Gemini 3.8 Flash
61.0 ± 3.1
66.7 ± 8.4
45.8 ± 6.5
87.3 ± 4.2
48.0 ± 3.9
Gemini 3.1 Pro
57.5 ± 2.3
69.7 ± 6.4
28.5 ± 6.4
84.2 ± 4.4
47.0 ± 4.3
Llama 4 Maverick
47.2 ± 2.7
39.3 ± 6.5
38.1 ± 7.0
80.0 ± 5.1
33.2 ± 3.7
Appendix
Table 7: MCQ accuracy with group-resampled 95% sampling intervals (%). Overall pools the four task denominators. Wearer translation, turning, object motion and passage relations contain 267, 260, 425 and 675 questions, respectively.
Configuration
Valid
Discovery
Exact
Labels
Empty
Nonempty
Chronological frames
GPT-6 Astra
2201
55.69–55.81
48.26
82.80–83.09
69.67
47.57–47.75
GPT-5.6 Sol
2201
44.48–44.57
35.54
74.46–74.55
46.92
43.07–43.20
Gemini 3.8 Flash
2201
51.74–52.24
43.73–43.90
74.47–76.47
72.04
39.94–40.73
Gemini 3.1 Pro
2201
44.45–46.08
33.97–34.67
64.22–68.38
46.45
43.30–45.87
Llama 4 Maverick
2200
28.08–31.59
14.29–15.33
33.56–44.06
19.91
32.83–38.38
Appendix
Table 8: FFG scores across all 574 questions (%). Ranges give the lowest and highest scores consistent with unresolved object identities. Valid counts scorable responses across the 2,201 benchmark questions.
Configuration
nD
Discovery
nE
Exact
nL
Labels
Chronological frames
GPT-6 Astra
573
55.7 ± 6.7
574
48.3 ± 6.9
276
83.0 ± 7.1
GPT-5.6 Sol
573
44.6 ± 6.1
574
35.5 ± 5.6
278
74.5 ± 6.6
Gemini 3.8 Flash
570
52.1 ± 5.8
573
43.8 ± 6.1
184
76.1 ± 8.9
Gemini 3.1 Pro
562
45.4 ± 5.7
570
34.2 ± 5.0
220
67.3 ± 7.2
Llama 4 Maverick
547
29.5 ± 5.6
568
14.4 ± 4.0
130
38.5 ± 5.8
Appendix
Table 9: FFG point estimates from Table 2 and their 95% confidence intervals from grouped resampling (%). nD and nE count determined question scores for Discovery and Exact. nL counts matched entities in the determined label ratio.
Configuration
Precision
Recall
F1
Chronological frames
GPT-6 Astra
55.27–55.47
64.50–64.73
59.53–59.74
GPT-5.6 Sol
45.50–45.66
64.50–64.73
53.36–53.55
Gemini 3.8 Flash
57.14–59.01
42.69–44.08
48.87–50.46
Gemini 3.1 Pro
45.71–48.78
51.97–55.45
48.64–51.90
Llama 4 Maverick
28.14–33.98
30.16–36.43
29.12–35.16
Appendix
Table 10: Entity-level micro-precision, recall and F1 (%) under feasible assignments. The 574 questions have 431 reference entities. An extra report contributes a false positive, including when the reference is empty.
Respondent
MCQ
Wearer translation
Turning
Object motion
Passage relation
Exact
Discovery
Labels
Participant A
85.00
89.14
84.62
94.35
77.63
75.96
77.38
99.36
Participant B
86.60
90.64
81.92
93.88
82.22
79.62
81.53
97.63
Two-person mean
85.80
89.89
83.27
94.12
79.93
77.79
79.46
98.49
Appendix
Table 11: Free-response human baseline on 2,201 questions (%). The 574 FFG questions use the same definitions as the models. Conditional Labels excludes omissions and extras.
Respondent
MCQ
Discovery
Exact
Labels
Empty
Nonempty
Participant A
85.0 ± 1.8
77.4 ± 3.6
76.0 ± 3.8
[98.5, 100.0]
82.5 ± 5.2
74.4 ± 5.0
Participant B
86.6 ± 1.6
81.5 ± 2.7
79.6 ± 2.9
97.6 ± 2.2
84.8 ± 4.9
79.6 ± 3.8
Two-person mean
85.8 ± 1.4
79.5 ± 3.0
77.8 ± 3.0
98.5 ± 1.1
83.6 ± 4.9
77.0 ± 4.1
Appendix
Table 12: Human estimates with group-resampled 95% sampling intervals (%). Empty and Nonempty refer to Discovery on 211 and 363 questions. Labels conditions on matched entities.
Comparison
Units
Agreement (%)
Cohen’s κ
Wearer translation
267
94.76
0.875
Turning sequence
260
82.69
0.745
Object motion
425
94.82
0.912
Passage relation
675
82.07
0.750
FFG empty/nonempty
574
93.38
0.866
FFG label on jointly matched entities
295
96.95
0.946
Appendix
Table 13: Agreement between independent respondents. Turning is compared using semantic event sequences (head/path, direction and event order). The final row conditions on objects identified by both respondents.
System
Input
MCQ
Wearer translation
Turning
Object motion
Passage relation
GPT-5.6 Sol
Question only
45.5 ± 2.8
49.8 ± 6.8
8.5 ± 3.3
78.4 ± 4.9
37.5 ± 4.5
GPT-5.6 Sol
With frames
57.5 ± 2.9
61.0 ± 7.8
33.8 ± 5.9
86.6 ± 4.7
46.8 ± 4.1
Appendix
Table 14: GPT-5.6 Sol accuracy with group-resampled 95% confidence intervals on the same 1,627 MCQs (%). Turning answer options vary across questions. Both input conditions exclude the same turning question whose corrected reference has no matching answer option.
System
Empty (211)
Nonempty (363)
All (574)
Always empty
100.00
0.00
36.76
GPT-6 Astra
69.67
47.57–47.75
55.69–55.81
Human mean
83.65
77.02
79.46
Appendix
Table 15: Discovery by reference-set status (%). Always-empty Discovery and Exact are 36.8 ± 6.6 (95% CI, percentage points). Model bounds reflect unresolved identity. The free-response human mean exceeds Astra’s upper bound by 23.65±7.45 percentage points (paired 95% CI).
Model
MCQ
Wearer translation
Turning
Object motion
Passage relation
FFG Exact
Discovery
Labels (%)
History: full coverage
GPT-6 Astra
195/239
45/55
42/53
61/64
47/67
38/54
41.50/54
78.95
Gemini 3.8 Flash
179/239
42/55
44/53
54/64
39/67
32/54
36.50/54
64.71
Gemini 3.1 Pro
147/239
34/55
23/53
56/64
34/67
25/54
28.50/54
76.47
GPT-5.6 Sol
159/239
38/55
36/53
60/64
25/67
23/54
26.00/54
70.00
Qwen3.8 Max
148/232
42/55
30/51
57/64
19/62
13/51
15.00/51
60.00
Appendix
Table 16: History results over each model’s scorable set. Score formats follow Table 17 ; label counts appear in Table 18 . The lower panel gives the auxiliary aggregate numerator (MCQ correct plus summed Discovery scores) and denominator.
Model
Scorable/all
Excluded
Auxiliary aggregate
GPT-6 Astra
293/293
0
236.50/293
Gemini 3.8 Flash
293/293
0
215.50/293
Gemini 3.1 Pro
293/293
0
175.50/293
GPT-5.6 Sol
293/293
0
185.00/293
Qwen3.8 Max
283/293
10
163.00/283
Qwen3.8 Flash
270/293
23
143.00/270
Appendix
Table 16: (continued)
Model
MCQ
Wearer translation
Turning
Object motion
Passage relation
FFG Exact
Discovery
Labels (%)
No History
GPT-6 Astra
174/218
46/53
34/45
54/62
40/58
24/39
27.50/39
69.23
Gemini 3.8 Flash
163/218
37/53
36/45
57/62
33/58
21/39
22.00/39
66.67
Gemini 3.1 Pro
146/218
36/53
26/45
55/62
29/58
16/39
18.50/39
69.23
GPT-5.6 Sol
143/218
33/53
30/45
57/62
23/58
12/39
14.00/39
66.67
Qwen3.8 Max
136/218
38/53
27/45
54/62
17/58
11/39
14.00/39
45.45
Appendix
Table 17: Paired results on 257 questions. MCQ and FFG Exact give correct/scorable counts; Discovery gives summed scores/scorable counts; Labels gives conditional accuracy (%; counts in Table 18 ). Bold/underlining mark best/second-best scores per condition.
Model
N
Empty/nonempty
∑s
Exact
TP/FP/FN
Labels
Paired No History (257 questions)
GPT-6 Astra
39
24/15
27.50
24
13/9/4
9/13
Gemini 3.8 Flash
39
24/15
22.00
21
12/12/5
8/12
Gemini 3.1 Pro
39
24/15
18.50
16
13/20/4
9/13
GPT-5.6 Sol
39
24/15
14.00
12
15/27/2
10/15
Qwen3.8 Max
39
24/15
14.00
11
11/28/6
5/11
Appendix
Table 18: FFG counts for the paired and full History evaluations. N is the number of scorable questions; empty/nonempty refer to reference sets. Discovery =100∑s/N , FFG Exact =100Exact/N , and Labels is 100 times the stated correct/assessable fraction.
Figure 4: Changes with visual history on the same 257 questions. Cells show History minus No History (percentage points): blue indicates improvement and orange decline. MCQ Overall and FFG Exact are outlined. Absolute scores and condition-specific references appear in Appendix C.3 .
Object type
Clips
Astra passage relation correct/total
Cars and other road vehicles
80
65/119
Riders with their vehicles
64
55/122
Pedestrians
50
42/98
Two-wheelers assessed separately
11
9/18
Curbs, steps and ground boundaries
68
49/102
Bollards, posts and cones
54
43/84
Appendix
Table 19: Coverage of eleven object types in the benchmark, with rare types combined in the last row. The last column counts correct passage-relation answers for Astra among questions targeting that type.
Interaction
Clips
Empty
Nonempty
Oncoming approach or passing
36
8
97
Crossing paths
23
7
62
Same-direction travel or following
12
3
31
Passing a stationary person, rider or vehicle
8
1
25
Space constrained by fixtures or vegetation
51
8
138
Negotiating a curb, step or ground boundary
32
6
82
Appendix
Table 20: Interaction coverage in the 145 evaluation clips with a nonempty FFG reference in at least one time window. For each identifiable interaction type, counts include all FFG windows in the clips where it occurs. Categories can overlap.
Target condition
Comparisons
Object motion correct
Passage relation correct
All linked targets
394
360
262
Stationary, constraint increases
67
67
46
Moving, irrelevant at both times
84
82
54
Appendix
Table 21: Astra on named targets with linked, independently answered questions. Links require identical target descriptions in the same clip and time order, with one unique question in each task. The two accuracies use the same targets in each row. Of the 394 comparisons, Astra answers only the motion question correctly in 107 and only the passage question correctly in nine.
Passage relation
Questions
Irrelevant at both times
268
Constraint increases
181
Constraint decreases
181
Insufficient evidence
45
Total
675
Appendix
Table 22: Reference answers for the 675 passage-relation questions. Insufficient visual evidence and irrelevance are distinct reference answers.
Figure 5: Entity selection across all 14 visual configurations. All panels use the same 574 FFG questions, including 211 empty and 363 nonempty references, with 431 reference entities. (a) Matched entities and extra reports, including duplicates. (b) Discovery on empty and nonempty references. Solid bars reach the lower feasible value; pale extensions and numerical ranges show the remaining identity uncertainty. Always returning an empty set scores 36.76% over all 574 questions.
Model
TP
FP
FN
Precision (%)
Recall (%)
Chronological frames
GPT-6 Astra
278–279
224–225
152–153
55.27–55.47
64.50–64.73
GPT-5.6 Sol
278–279
332–333
152–153
45.50–45.66
64.50–64.73
Gemini 3.8 Flash
184–190
132–138
241–247
57.14–59.01
42.69–44.08
Gemini 3.1 Pro
224–239
251–266
192–207
45.71–48.78
51.97–55.45
Llama 4 Maverick
130–157
305–332
274–301
28.14–33.98
30.16–36.43
Appendix
Table 23: Entity discovery over all 574 FFG questions and 431 reference entities. TP counts matched entities, FP extra reports and FN missed references. Micro-precision is 100TP/(TP+FP) and recall is 100TP/431 . Ranges reflect feasible identity matches. All 14 configurations use the same reference set.
Figure 6: Local correctness and incomplete coverage on all 574 FFG questions. Every bar accounts for the same 431 reference entities. Solid segments give the minimum feasible count of correctly labeled matches, incorrectly labeled matches and missed entities. The hatched remainder retains allocation that depends on unresolved identity; it is not assigned to an outcome. Unavailable responses contribute omissions. Conditional label accuracy uses matched entities only, with bounds from the same feasible matches. Extra reports appear in Figure 5 (a).
Model
Wearer translation
Passage relation
Overall
Chronological frames
GPT-6 Astra
1/27
2/45
3/72
GPT-5.6 Sol
0/27
0/45
0/72
Gemini 3.8 Flash
0/27
0/45
0/72
Gemini 3.1 Pro
0/27
0/45
0/72
Llama 4 Maverick
0/27
1/45
1/72
Appendix
Table 24: Recognizing insufficient evidence on all 72 such questions in the 2,201-question benchmark: 27 wearer-translation and 45 passage-relation questions. Entries are correct/planned counts; unavailable responses remain in the denominator.
Egocentric video understanding studies how AI interprets first-person video to recognize actions, track interactions, reason about space and time, and assist users.
Egocentric video understanding spans diverse capabilities, modalities, and evaluation formats, yet these are often studied in isolation despite their tight coupling in real-world interaction. We propose the Ego-Action-World interaction loop as a unifying principle: the wearer perceives the world, forms goals, acts, changes the world, and uses feedback to guide subsequent behavior. Based on this view, we introduce EgoOmniWorld, a benchmark organized around four complementary interaction structures: Node captures ego, action, and world states; Edge captures directed relations among them; Chain captures linked dependencies; and Loop captures feedback-closed interactions. We develop an interaction-first, evidence-grounded data pipeline that extracts structured interactions from egocentric videos, grounds them in visual and auditory evidence, and generates questions according to their interaction structure. Required modalities and single- or multi-turn formats follow naturally from the supporting evidence and dependencies. EgoOmniWorld contains 24,166 QA groups, including 20,284 training and 3,882 test samples. Experiments on representative open- and closed-source multimodal large language models reveal substantial and uneven limitations across interaction structures. Models trained on EgoOmniWorld improve in-domain interaction understanding and generalize to existing egocentric benchmarks. EgoOmniWorld provides a unified foundation for evaluating and advancing egocentric video understanding. Data, code, and models will be released.
Video reasoning models are a core component of egocentric and embodied agents. However, standard benchmarks for assessing models provide only evaluation of the output (e.g. the answer to a question), without evaluation of intermediate reasoning steps, and most provide answers only in the text domain. We introduce Minerva-Ego, a benchmark for evaluating complex egocentric visual reasoning. We extend recent high-quality video data sources recorded from egocentric / embodied settings with a set of challenging, multi-step multimodal questions and spatiotemporally-dense human-annotated reasoning traces. Benchmarking experiments show that state-of-the-art models still have a large gap to human performance. To investigate this gap in detail, we annotate each reasoning trace in the dataset with the objects of interest required to solve the question, as spatiotemporal mask annotations. Through extensive evaluations, we identify that prompting frontier models with hints of 'where' and 'when' to look yields substantial improvements in performance. Minerva-Ego can be downloaded at https://github.com/google-deepmind/neptune.
A core aspect of human perception is situated awareness, the ability to relate ourselves to the surrounding physical environment and reason over possible actions in context. However, most existing benchmarks for multimodal foundation models (MFMs) emphasize environment-centric spatial relations (relations among objects in a scene), while largely overlooking observer-centric relationships that require reasoning relative to agent's viewpoint, pose, and motion. To bridge this gap, we introduce SAW-Bench (Situated Awareness in the Real World), a novel benchmark for evaluating egocentric situated awareness using real-world videos. SAW-Bench comprises 786 self-recorded videos captured with Ray-Ban Meta (Gen 2) smart glasses spanning diverse indoor and outdoor environments, and over 2,071 human-annotated question-answer pairs. It probes a model's observer-centric understanding with six different awareness tasks. Our comprehensive evaluation reveals a human-model performance gap of 37.66%, even with the best-performing MFM, Gemini 3 Flash. Beyond this gap, our in-depth analysis uncovers several notable findings; for example, while models can exploit partial geometric cues in egocentric videos, they often fail to infer a coherent camera geometry, leading to systematic spatial reasoning errors. We position SAW-Bench as a benchmark for situated spatial intelligence, moving beyond passive observation to understanding physically grounded, observer-centric dynamics.
Chuhan Li, Rilyn Han, Joy Hsu +5
University of California, Santa Barbara · Yale University · Stanford University +3