Organizations: The Hong Kong University of Science and Technology · Tsinghua University · Southeast University · The Hong Kong Polytechnic University · Southwest Jiaotong University · RayNeo
Recalling where and when an object was last seen and who saw it requires recognizing it across days and people. Existing long-video benchmarks focus on events, object categories, or relative locations, leaving instance-level spatial memory largely untested in multimodal models. To address this gap, we present EgoWitness, a benchmark for instance-level spatial memory and reasoning in egocentric video. Its 8,453 human-verified questions span approximately 324 hours of week-long and short-horizon recordings and test seven abilities, from recognizing an object or room to recalling who saw it and reasoning about room visits. Over one million reviewed masks of 34 registered landmarks and 21 movable-object labels are linked through room timelines. Among twelve direct vision-language models (VLMs), open-weight models reach at most 46.97% accuracy, and none exceeds 35% on identifying who saw an object last. We therefore propose EgoClue, a spatial-memory agent that retrieves candidate intervals, verifies instance identity, and computes spatial and temporal relations with five tools. Two-stage training with supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) raises its 4B controller from 23.64% to 72.65% accuracy on 1,320 held-out questions. Evidence-trace synthesis matches the best direct VLM at 74.85% on week-long questions and achieves 80.56% in environments excluded from training.
Figures & tables
Figure 1: Instance-level spatial memory across egocentric streams. (A) Instance-reference comparison. (B) Historical observations with wearer, time, and room context. (C) Seven QA abilities in two groups: object identity and memory, and room identity and visit reasoning.
(a) Memory and spatial question answering
Benchmark
Scale
Span
Object identity
Place identity
Cross- wearer
QA evidence
EgoLifeQA [ 68 ]
300 h
7 days
Object recall
Location recall
∘
Evidence time
MA-EgoQA [ 29 ]
266 h
7 days
Object usage
Event context
∙
Context windows
HD-EPIC [ 45 ]
41 h
3 days
Object itineraries
Fixture identity
∘
∘
AMB / AMEGO [ 21 ]
22.7 h
14 min avg.
Interaction links
Activity zones
∘
∘
UCS-Bench [ 62 ]
170+ h
Seconds to hours
Trajectory recall
Relative places
∘
Evidence segments
Table 1: Related benchmarks and annotation resources. (a) Identity-related QA. (b) Object and environment annotations. ∙ : included in the reported protocol; ∘ : not specified. QA evidence: reference evidence per question; EgoWitness also scores it.
Figure 2: Room references, object annotations, and question coverage. a. The external room catalog contains 400 source-local reference entries across 81 scopes and 18 room types, separate from the 17 named zones in the shared home. b. The week-long component contains 34 registered landmark instances and 21 movable-object labels. c. Questions cover both components; Q07 is split into five subtypes.
Figure 3: From reviewed observations to evidence-linked questions. a. Room timelines and landmark locations supply spatial context. b. SAM 3.1 tracks Gemini- and human-checked keyframes. c. Annotators resolve instance identity using reference views and room context. d. Accepted observations fix question facts before wording. Full review interfaces appear in Fig. S1 .
Attribute
Week-long
Short-horizon
Recordings and spatial coverage
Video source
EgoLife
Ego4D-derived
Source hours
∼ 300
23.46
Recording units
42 wearer-days
43 videos
Room references
17 zones
400 local entries
Reviewed annotations and questions
Table 2: Source collection and annotation coverage. L/M denote landmark instances/movable labels. Room references retain their source-local scope.
Figure 4: EgoClue: finding, verifying, and linking evidence. The trained 4B controller selects tools and video intervals, then refines its search from returned observations. The optional EgoClue + Sol stage synthesizes an answer from the trace without new visual calls.
Model or system
Week-long answer accuracy
Week-long
Cross-env.
Q01
Q02
Q03
Q04
Q05
Q06
Q07
Macro
Micro
Δ
Micro
Uniform random, expected
33.33
25.00
25.00
25.00
25.00
33.33
25.00
27.38
26.52
–
27.12
Public-text ridge
42.50
28.33
27.50
27.50
25.00
43.33
30.67
32.12
31.59
–
27.06
Direct open-weight VLMs
Qwen3.8-27B [ 50 ]
38.33
50.00
27.50
57.50
52.50
62.50
45.67
47.71
46.97
0.00
58.07
VideoChat-Flash-7B [ 32 ]
62.50
26.67
25.00
32.50
63.33
30.83
43.83
40.67
41.82
-5.15
38.37
Table 3: Spatial memory and reasoning across model and system families. Accuracy in % on 1,320 week-long and 787 cross-environment questions. Direct VLMs use public references and up to 32 frames. Agents share the budget in Table S12 . Macro/Micro average abilities/questions. Bold/underline mark group-best/runner-up aggregates. Δ reports Micro differences in percentage points relative to direct-group leaders, 4B base for controllers, or SFT+GRPO for synthesis.
Configuration
All test questions
Full-invocation subset
Correct
Accuracy, %
Δ
Questions
Full
Ablated
Δ
Full
959
72.65
0.00
Without Video
620
46.97
-25.68
971
688
369
-32.85
Without Timeline
845
64.02
-8.64
178
151
103
-26.97
Without Topology
881
66.74
-5.91
107
83
56
-25.23
Without Memory
900
68.18
-4.47
111
44
31
-11.71
Table 4: Individual tool removal from EgoClue. Left: all 1,320 test questions. Right: questions on which Full invokes the removed tool, with Full and Ablated counting correct answers. Signed changes are percentage points relative to the Full evaluation results. Rows are ordered by overall loss. Supp. Sec. F.3 details the paired comparison.
Figure 5: EgoClue training dynamics. Thin traces show raw scalars and thick traces show EMA with coefficient 0.2. Supp. Sec. D.1 details both training schedules.
Appendix figures & tables39 assets
Supplementary material from the paper’s appendix.
Appendix
Resource
Reused input
Integration in EgoWitness
Layer
Scope / sampling
Week-long recordings
EgoLife [ 68 ]
Videos, house assets
Six wearers, seven days
New room intervals, instance links, and masks
Short-horizon recordings and annotations
Ego4D [ 25 ]
Videos
43 original videos
Shared video IDs and time axes
EgoMask / EgoTracks [ 33 , 57 ]
Masks, track lineage
1 Hz retained
Boxes and local tracks aligned to rooms
Appendix
Table S1: Source resources and their integration. Rows distinguish original media from inherited annotations. Sampling rates describe the annotations retained in EgoWitness. The shared Ego4D videos are counted once across resources.
Quantity
EgoMask
RefEgo
Total
Source media
Original videos
37
10
43
Source clips
70
13
83
Tracks and object observations
Source-local tracks
157
13
170
Observation candidates
159
13
172
Appendix
Table S2: Selected short-horizon data. Counts from EgoMask-Train and RefEgo. Totals deduplicate videos. Tracks retain source-local scope. “Matched” denotes an unambiguous association with the camera wearer’s occupied room.
Figure S1: Annotation workspaces and review operations. A. The Online Room Curator System supports boundary editing, room-instance assignment, landmark registration, and visual verification. The DAY1/A1 example contains 14 reviewed intervals. Shared room IDs and aligned intervals supply context to B , where annotators compare landmark and movable-object instances, inspect logical conflicts, and correct associations before export. A5, A6, and A4 show one shared instance. Room-boundary conflicts return to A for interval correction.
Figure S2: Spatial reference for the week-long environment. A. The official EgoLife house model, a courtyard observation, and bedroom ownership. B and C. Upper- and ground-floor plans paired with real video observations of all 16 indoor room instances; the courtyard is the seventeenth named zone. Numbered pins match the photo cards, which retain the room IDs and colors used in Fig. S3 . Geometry and textures come from the official EgoLife assets [ 68 ] . The pins are schematic labels.
Figure S3: Room observations across wearers and days. A. Each row represents one wearer and each column one day, with a common 09:00 to 24:00 day-clock axis. Colors identify 17 named rooms and zones, with separate colors for outdoor, transition, and unresolved-room states. White gaps indicate missing recordings. B. The DAY6 window from 15:00 to 16:00, outlined in A, expands the room intervals for all six wearers. Plotted intervals are clipped to the media manifest, preserving their boundaries and recording gaps.
Rule
Action
Boxes
Movable candidates 406,644 boxes
Filtering
Below class confidence
Reject
357,116
Below minimum persistence
Reject
5,729
Duplicate box
Suppress
7,342
Wearer outdoors
Exclude
16,001
Appendix
Table S4: Context and quality rules by action. Counts describe boxes affected by each rule among the 406,644 movable candidates. Held candidates proceed to visual review.
Figure S4: Recurrence of representative targets across the shared week. a. Four human-verified movable instances. b. Registered landmark instances. Thumbnails are unedited source-frame crops around one accepted mask of each target, labeled with its wearer and day. Each day spans 09:30 to 23:30 in 30-minute bins. A bin’s shade gives the largest number of wearers who see the target label within one wall-clock minute of that bin. Dots give each wearer’s share of seen seconds, and the note under each timeline gives the most frequent wearer and wearer room. Room labels indicate wearer occupancy.
Table S5: Target recurrence across days, wearers, and rooms. Rows marked g are group-level labels. Landmark rows are registered instances, and registered rooms follow each name. Numbers separate distinct instances sharing a name. For landmarks, the last column names a room only when it differs from the registered room. Seen time counts distinct wearer-seconds with an accepted mask, which removes the 5 Hz/1 Hz sampling difference. Rooms are occupied wearer rooms from the Online Room Curator System. Same minute gives the wall-clock minutes in which at least two wearers see the target label, over all minutes with a sighting, and the largest number of wearers in one minute. Cell shading follows each wearer’s share. Summary rows give totals, medians for days, wearers, and rooms, and pooled shares. Object icons are schematic.
Component
Train
Val.
Test
Total
Week-long
5,750
383
1,533
7,666
Short-horizon
0
0
787
787
All sources
5,750
383
2,320
8,453
Appendix
Table S6: Question inventory. Unique questions by partition. All short-horizon questions are evaluation-only.
Table S7: Question templates for the seven spatial-memory abilities. Colored slots vary with the query’s public references, wearers, and time scope. Q07 expands into five visit-reasoning subtypes.
Property
Questions
Candidate scope
Candidates on multiple dates
118 / 120
Candidates from multiple wearers
117 / 120
Target-positive support
At least one positive window older than 24 h
103 / 120
Positive windows on multiple dates
78 / 120
Appendix
Table S8: Historical support in the 120 evaluated Q04 questions. Positive windows are those selected by the frozen correct answer. Cross-day and cross-wearer rows are separate counts.
Window
Wearer
Day
Time interval
Age, h
Same target
W1
A4
DAY3
20:04:31.297 to 20:14:31.297
93.76
∙
W2
A3
DAY5
19:11:33.909 to 19:21:33.909
46.64
∙
W3
A6
DAY1
13:55:56.799 to 14:05:56.799
147.90
∙
W4
A6
DAY7
14:23:37.783 to 14:33:37.783
3.44
∘
Appendix
Table S9: One Q04 query links three historical wearers and dates. The query time is DAY7 18:00:06.000. Each candidate spans ten minutes. Positive labels come from correct option B, which selects W1 + W2 + W3. Age is measured from each window’s end to the query time.
Figure S5: Question and answer vocabulary in EgoWitness. The panels show questions, correct answers, and their combined vocabulary for 7,666 week-long and 787 short-horizon questions. Font sizes reflect relative token frequencies within each panel. The corpus uses question text and correct-answer text only.
Figure S6: Two-pass QA review. The top row links model-assisted review, expert verification, and versioned decisions. a. Cropped question and answer view. b. Cropped multi-frame references and source video for identity verification. c. Redrawn room context and textual cues aligned with the selected interval. Timecodes use HH:MM:SS.FF at 20 FPS. The example compares kitchens observed by A4 on DAY2 and A3 on DAY4.
Item
Value
Controller
Qwen3.5-4B
RL algorithm / rollout group
GRPO / 4
Prompt batch
18
RL learning rate
10−6
KL coefficient / clip
0.001 / 0.2
Training temperature / top- p
1.0 / 1.0
Appendix
Table S10: Training configuration. The visual and retrieval services remain fixed across the Base, SFT, and GRPO controller comparison.
Table S11: Evaluated direct models and API controllers. Each entry gives the model and its checkpoint or API identifier. The shared-tool Base, SFT, and GRPO conditions use Qwen3.5-4B.
Attribute
Shared setting
Public inputs and shared services
Wearers / query time
Question-defined
Visual references
Public instance and room catalogs
Private GT / demonstrations
Excluded / 0
Controller observations
Textual tool outputs
Visual backend
Gemini 3.6 Flash
Appendix
Table S12: Agent access and execution limits. Shared settings for Base, SFT, and SFT+GRPO on the week-long questions.
Tool
Inputs
Returns
Implementation and access limits
Retrieval and visual inspection with fixed external models
Memory search_memory
Text query, level of day, hour, or 10-min segment, optional wearer and time scope
Relevance note, selected records, and video locators for segment hits
No vector index is built. The text model GPT-5.3 Chat selects record IDs from batches of at most 30 prebuilt records, newest first, and IDs outside the batch are rejected. Records come from dense captions, transcripts, and coarse annotation categories. Instance IDs, attributes, boxes, exact sighting times, and answers are removed.
Video video_qa_updated
Prompt and sources such as public references, wearer intervals, or subclips
Text observation, evidence IDs, and sampled time points
Gemini 3.6 Flash receives one silent MP4 per request at 1 FPS with source timestamps. Subclips of at most 40 s use 5 FPS, and object checks of at most 2 s use 20 FPS. Room IDs appear as permuted labels, and the room graph is withheld. Media outside the question scope or after the query time are rejected.
Image image_qa
Prompt, reference images, and frames at given times
Text observation and evidence IDs
The same Gemini model compares target references and frames in one request of at most 50 images. Access limits match Video.
Deterministic spatial and temporal computation
Topology query_topology
Start and end room IDs
All tied shortest paths, or none when unreachable
Breadth-first search on the unweighted room adjacency graph. It receives no visual input, so Video or Image must first identify the endpoint rooms.
Appendix
Table S13: EgoClue tool implementations. Each tool returns text to the controller. Shaded bands separate tools that call fixed external models from tools that compute their results deterministically.
Table 28
Answers
Execution and evidence
Controller
Correct
Micro
Macro
Format
Eg
Without SFT
Base Qwen3.5-4B [ 49 ]
312
23.64
23.52
25.76
0.099
Direct RL, step 414
537
40.68
39.55
99.24
0.123
SFT initializations, before and after RL
SFT/4
686
51.97
52.14
81.52
0.414
Appendix
Table S16: Post-training configurations on 1,320 questions. Micro, Macro, and Format are percentages. Eg is the resolved runtime-grounded evidence mean. SFT/2 and SFT/4 stop SFT at 0.05 and 0.025 epoch. Shading marks EgoClue-SFT+GRPO (step 259), and bold marks the best value in each column. Metric definitions appear in Section E.3 .
Figure S7: Post-training across initializations and abilities. Left compares micro accuracy before and after RL at three SFT initializations under their recorded training schedules. Right gives the SFT and step-259 SFT+RL comparison by ability.
Figure S8: Execution quality and answers. Format validity and answer accuracy on 1,320 questions.
Removed tool
Subset
Questions
Full
Removed
Δ pp
Image
All questions
1,320
959
907
−3.94
Memory
All questions
1,320
959
900
−4.47
Topology
All questions
1,320
959
881
−5.91
Timeline
All questions
1,320
959
845
−8.64
Video
All questions
1,320
959
620
−25.68
Topology
Q05
120
88
65
−19.17
Appendix
Table S17: Paired tool-removal outcomes. Full-invocation subsets are defined from full-agent execution traces. Task-applicable subsets use question types independently of observed calls. Full and Removed give correct-answer counts. The final column is Removed minus Full accuracy in percentage points.
Figure S9: Tool removal on all questions and invocation-selected subsets. The left panel uses the same 1,320 questions for each removal. The right panel uses a different Full-invocation subset for each tool. Subsets can overlap and include unsuccessful calls.
Controller
Micro, %
Turns
Tool calls
Controller tokens
Tool-backend tokens
Proprietary total
Proprietary per correct
EgoClue-SFT+GRPO, 4B
72.65
3.30
2.31
18.3k †
38.1k
38.1k
52.4k
GPT-5.3 Chat [ 43 ]
65.76
4.34
2.55
25.0k
38.3k
63.3k
96.3k
Gemini 4 Argon [ 24 ]
46.14
4.83
2.78
32.1k
28.9k
61.0k
132.2k
Gemini 3.5 Flash-Lite [ 22 ]
37.58
7.06
1.15
32.6k
11.7k
44.3k
117.8k
Claude Haiku 5.5 [ 3 ]
32.20
8.07
2.22
69.4k
27.9k
97.4k
302.4k
Appendix
Table S18: Per-question cost of shared-tool controllers. All controllers answer the same 1,320 week-long questions with the same tools, backends, and budget. Turns count controller generations. Token counts include Gemini thinking. † Local 4B controller, excluded from proprietary totals. Bold marks the best accuracy, turns, controller tokens, and totals.
Figure 35
Week-long
Cross-env.
Model
Parsed
A
Parsed
A
Qwen3.8-27B [ 50 ]
92.80
46.97
99.75
58.07
VideoChat-Flash-7B [ 32 ]
100.00
41.82
100.00
38.37
MiMo-VL-7B-RL [ 66 ]
99.39
40.68
99.75
46.25
VideoLLaMA3-7B [ 71 ]
100.00
40.23
100.00
47.78
InternVideo3-8B [ 67 ]
100.00
39.85
100.00
44.09
Appendix
Table S19: Direct-output normalization. Parsed is the percentage assigned a choice by the common normalizer. A is normalized accuracy on all 1,320 week-long and 787 cross-environment questions, as reported in the main model comparison and Table S24 .
Ability
Questions
Majority
Ridge
Q01
120
30.83
42.50
Q02
120
18.33
28.33
Q03
120
22.50
27.50
Q04
120
25.00
27.50
Q05
120
25.00
25.00
Q06
120
35.00
43.33
Appendix
Table S20: Nonvisual baselines on the week-long setting. Majority selects the most frequent training answer per detailed type. Ridge uses public text and validation-selected regularization. Q07 pools five subtypes. These scores use the same question sets as the main model comparison.
Identifier type
Shared atoms
Affected questions
Core group
0
0
Exact public input
0
0
Object episode
503
364
Object event
209
445
Reference
566
562
Room anchor
229
227
Appendix
Table S21: Train/test identifier overlap. Recomputed from assignment atoms. Shared atoms are distinct identifiers. Question counts refer to the evaluated 1,320 and may overlap across rows.
Unshared identifiers
N
SFT
GRPO
Δ
Reference
114
35.09
56.14
+21.05
Object event or episode
32
31.25
65.63
+34.38
Room event
81
76.54
88.89
+12.35
All recorded atoms
113
63.72
82.30
+18.58
Appendix
Table S22: Training-overlap sensitivity. Each row retains questions with no training-shared atom of the indicated kind, within its applicable domain. Checkpoints and inputs are unchanged. Values are micro accuracy and the GRPO gain in percentage points. Rows can overlap.
Continuous joint J
Strict semantic success H
Type
Applicable
SFT
GRPO
+ Sol
SFT
GRPO
+ Sol
Q02
95
34.74–49.47
33.68–47.37
43.16–61.05
34.74–49.47
33.68–47.37
43.16–61.05
Q03
120
0.00–8.33
0.00–17.50
0.00–20.00
0.00–8.33
0.00–17.50
0.00–20.00
Q05
120
40.00
52.08
54.17
29.17
37.50
39.17
Q06
120
56.67
62.92
60.00
45.00
52.50
51.67
Q07A
120
61.66
69.79
68.95
55.83
61.67
62.50
Appendix
Table S23: Semantic joint correctness by week-long type. Values are percentages on the fixed applicable subset. Ranges bound unresolved scores. Q01 and Q04 are answer-only under the frozen object contract. Q02 excludes 25 evidence-inapplicable cases and Q07E excludes 34, identically across runs. Scores include Q03 exact-citation normalization, applied by the same rule to SFT, GRPO, and synthesis with the frozen scorer.
Model or system
Q01
Q04
Q06
Q07
Macro
Micro
Direct visual reasoning, ordered by Micro
Gemini 4 Argon [ 24 ]
90.00
95.27
83.75
82.06
87.77
86.28
Claude Opus 5.5 [ 4 ]
77.50
94.08
80.00
76.56
82.03
80.81
GPT-6.1 Sol [ 44 ]
70.83
73.96
56.25
66.27
66.83
67.60
Qwen3.8-27B [ 50 ]
53.33
76.33
62.50
51.20
60.84
58.07
Gemma 4 12B [ 20 ]
49.17
55.62
53.75
46.89
51.36
49.81
Appendix
Table S24: Cross-environment model and system results. All 787 questions are evaluation-only. Q01, Q04, Q06, and Q07 contain 120, 169, 80, and 418 questions. Macro weights these four abilities equally. Micro weights all questions equally. Direct models answer from their frame inputs, and all tool systems use the week-long execution budget. All direct rows use the same normalizer. Bold marks the best Micro within each group.
Table S26: Video-cluster uncertainty for synthesis. Changes are percentage points relative to the 4B controller. CI denotes the 95% paired video-bootstrap interval. Evidence metrics use their applicable subset, with each metric’s denominator listed below.
Figure S12: Evidence-only synthesis. (a) Answer accuracy. (b) Incorrect-to-correct and correct-to-incorrect answer changes. (c) Accuracy change by week-long ability. (d) Semantic evidence E , strict success H , and evidence-submission validity on the same 498 cross-environment questions. Accuracy comparisons use 1,320 week-long and 787 cross-environment questions under their respective controller settings. Orange denotes the controller and red denotes synthesis. Gains in (a), (c), and (d) are percentage points.
Figure S13: Representative failure cases. Six abilities grouped by error type. Chips report the three frontier VLMs and EgoClue (green when correct), and the red line repeats Gemini 4 Argon’s option. Green and red frame outlines mark gold evidence and the frame behind a wrong answer; yellow boxes are released masks at the same timestamp. All timelines share the legend below: room bands come from reviewed room intervals, and Q03 ticks are released screen masks in 1-min bins. Q06 references show one of five views.
Type
Err. (%)
Dominant error direction
Object-instance memory
Q01
31.1
different categories 44%; merge 36%
Q02
19.7
over-report 89%; under-report 11%
Q03
69.4
–
Q04
29.3
omit a positive window 56%
Room and visit memory
Appendix
Table S27: Frontier-VLM error patterns by question type. Error pools the stored direct predictions of Gemini 4 Argon, Claude Opus 5.5, and GPT-6.1 Sol (358–360 answered per type). Shares are fractions of errors.
Figure S14: Evidence-annotated examples: object-instance memory. Evidence windows come from the gold records. Yellow boxes are released masks at the shown timestamp. Frame labels give wearer, day, time, and the wearer’s reviewed room. In Q04, W1 and W2 frames illustrate the negative windows. Chips report the three frontier VLMs and EgoClue (green when correct).
Figure S16: Evidence-annotated examples: room and visit memory. Timelines show reviewed room intervals over each question’s interval with the shared legend; the outlined segments are the queried room. Q05 and Q06 references show one of five views, and the Q05 graph highlights the shortest path on the supplied room graph. Q07E enlarges the first 30 s of its interval, where the first visit lasts 7.8 s. Chips report the three frontier VLMs and EgoClue.
Next-generation visual assistants, such as smart glasses, embodied agents, and always-on life-logging systems, must reason over an entire day or more of continuous visual experience. In ultra-long video settings, relevant information is sparsely distributed across hours or days, making memory a fundamental challenge: models must accumulate information over time, recall prior states, track temporal order, and abstract recurring patterns. However, existing week-long video benchmarks are primarily designed for perception and recognition, such as moment localization or global summarization, rather than reasoning that requires integrating evidence across multiple days. To address this gap, we introduce EgoMemReason, a comprehensive benchmark that systematically evaluates week-long egocentric video understanding through memory-driven reasoning. EgoMemReason evaluates three complementary memory types: entity memory, tracking how object states evolve and change across days; event memory, recalling and ordering activities separated by hours or days; and behavior memory, abstracting recurring patterns from sparse, repeated observations over the whole week period. EgoMemReason comprises 500 questions across three memory types and six core challenges, with an average of 5.1 video segments of evidence per question and 25.9 hours of memory backtracking. We evaluate EgoMemReason on 17 methods across MLLMs and agentic frameworks, revealing that even the best model achieves only 39.6% overall accuracy. Further analysis shows that the three memory types fail for distinct reasons and that performance degrades as evidence spans longer temporal horizons, revealing that long-horizon memory remains far from solved. We believe EgoMemReason establishes a strong foundation for evaluating and advancing long-context, memory-aware multimodal systems.
We introduce UCS-Bench, a dataset spanning 170+ hours of egocentric visual observations with 8.1K+ timestamped questions for diagnosing User-Centric Continual Spatial intelligence in egocentric video streams. UCS-Bench targets a new problem that emphasizes dynamic spatial reasoning, long-term memory, and their alignment with users' real-time locations. We propose DirectMe, a framework that incrementally constructs and maintains a structured spatial memory from streaming egocentric observations. DirectMe enables robust tracking and recall of object locations, all relative to the user's movement over time. By tightly coupling visual perception with memory updates and spatial reasoning, our approach supports long-horizon queries that require recalling interactions, resolving viewpoint-induced ambiguities, and adapting to dynamic scenes. Our experiments show that DirectMe significantly improves the spatial reasoning of leading multimodal LLMs; it also surpasses many spatially aware and long-form streaming video models. We hope our benchmark and solution will advance spatial intelligence research for egocentric AI assistants. Data and code are available at https://github.com/cocowy1/UCS-Bench.
Yun Wang, Junbin Xiao, Han Lyu +6
Department of Computer Science, City University of Hong Kong · University of Science and Technology of China · National University of Singapore +4
Video reasoning models are a core component of egocentric and embodied agents. However, standard benchmarks for assessing models provide only evaluation of the output (e.g. the answer to a question), without evaluation of intermediate reasoning steps, and most provide answers only in the text domain. We introduce Minerva-Ego, a benchmark for evaluating complex egocentric visual reasoning. We extend recent high-quality video data sources recorded from egocentric / embodied settings with a set of challenging, multi-step multimodal questions and spatiotemporally-dense human-annotated reasoning traces. Benchmarking experiments show that state-of-the-art models still have a large gap to human performance. To investigate this gap in detail, we annotate each reasoning trace in the dataset with the objects of interest required to solve the question, as spatiotemporal mask annotations. Through extensive evaluations, we identify that prompting frontier models with hints of 'where' and 'when' to look yields substantial improvements in performance. Minerva-Ego can be downloaded at https://github.com/google-deepmind/neptune.