EgoWitness: Benchmarking Instance-Level Spatial Memory and Reasoning in Egocentric Video Streams
Organizations: The Hong Kong University of Science and Technology · Tsinghua University · Southeast University · The Hong Kong Polytechnic University ·
Abstract
Recalling where and when an object was last seen and who saw it requires recognizing it across days and people. Existing long-video benchmarks focus on events, object categories, or relative locations, leaving instance-level spatial memory largely untested in multimodal models. To address this gap, we present EgoWitness, a benchmark for instance-level spatial memory and reasoning in egocentric video. Its 8,453 human-verified questions span approximately 324 hours of week-long and short-horizon recordings and test seven abilities, from recognizing an object or room to recalling who saw it and reasoning about room visits. Over one million reviewed masks of 34 registered landmarks and 21 movable-object labels are linked through room timelines. Among twelve direct vision-language models (VLMs), open-weight models reach at most 46.97% accuracy, and none exceeds 35% on identifying who saw an object last. We therefore propose EgoClue, a spatial-memory agent that retrieves candidate intervals, verifies instance identity, and computes spatial and temporal relations with five tools. Two-stage training with supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) raises its 4B controller from 23.64% to 72.65% accuracy on 1,320 held-out questions. Evidence-trace synthesis matches the best direct VLM at 74.85% on week-long questions and achieves 80.56% in environments excluded from training.
Figures & tables
| (a) Memory and spatial question answering | ||||||
| Benchmark | Scale | Span | Object identity | Place identity | Cross- wearer | QA evidence |
| EgoLifeQA [ 68 ] | 300 h | 7 days | Object recall | Location recall | Evidence time | |
| MA-EgoQA [ 29 ] | 266 h | 7 days | Object usage | Event context | Context windows | |
| HD-EPIC [ 45 ] | 41 h | 3 days | Object itineraries | Fixture identity | ||
| AMB / AMEGO [ 21 ] | 22.7 h | 14 min avg. | Interaction links | Activity zones | ||
| UCS-Bench [ 62 ] | h | Seconds to hours | Trajectory recall | Relative places | Evidence segments | |
| Attribute | Week-long | Short-horizon |
| Recordings and spatial coverage | ||
| Video source | EgoLife | Ego4D-derived |
| Source hours | 300 | 23.46 |
| Recording units | 42 wearer-days | 43 videos |
| Room references | 17 zones | 400 local entries |
| Reviewed annotations and questions | ||
| Model or system | Week-long answer accuracy | Week-long | Cross-env. | ||||||||
| Q01 | Q02 | Q03 | Q04 | Q05 | Q06 | Q07 | Macro | Micro | Micro | ||
| Uniform random, expected | 33.33 | 25.00 | 25.00 | 25.00 | 25.00 | 33.33 | 25.00 | 27.38 | 26.52 | – | 27.12 |
| Public-text ridge | 42.50 | 28.33 | 27.50 | 27.50 | 25.00 | 43.33 | 30.67 | 32.12 | 31.59 | – | 27.06 |
| Direct open-weight VLMs | |||||||||||
| Qwen3.8-27B [ 50 ] | 38.33 | 50.00 | 27.50 | 57.50 | 52.50 | 62.50 | 45.67 | 47.71 | 46.97 | 0.00 | 58.07 |
| VideoChat-Flash-7B [ 32 ] | 62.50 | 26.67 | 25.00 | 32.50 | 63.33 | 30.83 | 43.83 | 40.67 | 41.82 | -5.15 | 38.37 |
| Configuration | All test questions | Full-invocation subset | |||||
| Correct | Accuracy, % | Questions | Full | Ablated | |||
| Full | 959 | 72.65 | 0.00 | ||||
| Without Video | 620 | 46.97 | -25.68 | 971 | 688 | 369 | -32.85 |
| Without Timeline | 845 | 64.02 | -8.64 | 178 | 151 | 103 | -26.97 |
| Without Topology | 881 | 66.74 | -5.91 | 107 | 83 | 56 | -25.23 |
| Without Memory | 900 | 68.18 | -4.47 | 111 | 44 | 31 | -11.71 |
Appendix figures & tables39 assets
Supplementary material from the paper’s appendix.
Appendix
| Resource | Reused input | Integration in EgoWitness | |
| Layer | Scope / sampling | ||
| Week-long recordings | |||
| EgoLife [ 68 ] | Videos, house assets | Six wearers, seven days | New room intervals, instance links, and masks |
| Short-horizon recordings and annotations | |||
| Ego4D [ 25 ] | Videos | 43 original videos | Shared video IDs and time axes |
| EgoMask / EgoTracks [ 33 , 57 ] | Masks, track lineage | 1 Hz retained | Boxes and local tracks aligned to rooms |
| Quantity | EgoMask | RefEgo | Total |
| Source media | |||
| Original videos | 37 | 10 | 43 |
| Source clips | 70 | 13 | 83 |
| Tracks and object observations | |||
| Source-local tracks | 157 | 13 | 170 |
| Observation candidates | 159 | 13 | 172 |
| Rule | Action | Boxes |
| Movable candidates 406,644 boxes | ||
| Filtering | ||
| Below class confidence | Reject | 357,116 |
| Below minimum persistence | Reject | 5,729 |
| Duplicate box | Suppress | 7,342 |
| Wearer outdoors | Exclude | 16,001 |
| Component | Train | Val. | Test | Total |
| Week-long | 5,750 | 383 | 1,533 | 7,666 |
| Short-horizon | 0 | 0 | 787 | 787 |
| All sources | 5,750 | 383 | 2,320 | 8,453 |
| Property | Questions |
| Candidate scope | |
| Candidates on multiple dates | 118 / 120 |
| Candidates from multiple wearers | 117 / 120 |
| Target-positive support | |
| At least one positive window older than 24 h | 103 / 120 |
| Positive windows on multiple dates | 78 / 120 |
| Window | Wearer | Day | Time interval | Age, h | Same target |
| W1 | A4 | DAY3 | 20:04:31.297 to 20:14:31.297 | 93.76 | |
| W2 | A3 | DAY5 | 19:11:33.909 to 19:21:33.909 | 46.64 | |
| W3 | A6 | DAY1 | 13:55:56.799 to 14:05:56.799 | 147.90 | |
| W4 | A6 | DAY7 | 14:23:37.783 to 14:33:37.783 | 3.44 |
| Item | Value |
| Controller | Qwen3.5-4B |
| RL algorithm / rollout group | GRPO / 4 |
| Prompt batch | 18 |
| RL learning rate | |
| KL coefficient / clip | 0.001 / 0.2 |
| Training temperature / top- | 1.0 / 1.0 |
| Open-weight models | ||
| Qwen3.5-4B [ 49 ] Qwen/ Qwen3.5-4B | Gemma 4 12B [ 20 ] google/ gemma-4-12B-it | Qwen3.8-27B [ 50 ] Qwen/ Qwen3.8-27B |
| InternVL3.5-8B [ 59 ] OpenGVLab/ InternVL3_5-8B | MiMo-VL-7B-RL [ 66 ] XiaomiMiMo/ MiMo-VL-7B-RL | VideoLLaMA3-7B [ 71 ] DAMO-NLP-SG/ VideoLLaMA3-7B |
| InternVideo3-8B [ 67 ] yanziang/ InternVideo3-8B-Instruct | VideoChat-Flash-7B [ 32 ] OpenGVLab/ VideoChat-Flash-Qwen2-7B_res448 | LongVILA-R1-7B [ 10 ] Efficient-Large-Model/ LongVILA-R1-7B |
| Closed-source models | ||
| GPT-6.1 Sol [ 44 ] gpt-6.1-sol | Gemini 4 Argon [ 24 ] gemini-4-argon | Claude Opus 5.5 [ 4 ] claude-opus-5-5 |
| Prompted API controllers | ||
| Attribute | Shared setting |
| Public inputs and shared services | |
| Wearers / query time | Question-defined |
| Visual references | Public instance and room catalogs |
| Private GT / demonstrations | Excluded / 0 |
| Controller observations | Textual tool outputs |
| Visual backend | Gemini 3.6 Flash |
| Tool | Inputs | Returns | Implementation and access limits |
| Retrieval and visual inspection with fixed external models | |||
| Memory search_memory | Text query, level of day, hour, or 10-min segment, optional wearer and time scope | Relevance note, selected records, and video locators for segment hits | No vector index is built. The text model GPT-5.3 Chat selects record IDs from batches of at most 30 prebuilt records, newest first, and IDs outside the batch are rejected. Records come from dense captions, transcripts, and coarse annotation categories. Instance IDs, attributes, boxes, exact sighting times, and answers are removed. |
| Video video_qa_updated | Prompt and sources such as public references, wearer intervals, or subclips | Text observation, evidence IDs, and sampled time points | Gemini 3.6 Flash receives one silent MP4 per request at 1 FPS with source timestamps. Subclips of at most 40 s use 5 FPS, and object checks of at most 2 s use 20 FPS. Room IDs appear as permuted labels, and the room graph is withheld. Media outside the question scope or after the query time are rejected. |
| Image image_qa | Prompt, reference images, and frames at given times | Text observation and evidence IDs | The same Gemini model compares target references and frames in one request of at most 50 images. Access limits match Video. |
| Deterministic spatial and temporal computation | |||
| Topology query_topology | Start and end room IDs | All tied shortest paths, or none when unreachable | Breadth-first search on the unweighted room adjacency graph. It receives no visual input, so Video or Image must first identify the endpoint rooms. |
| Answers | Execution and evidence | ||||
| Controller | Correct | Micro | Macro | Format | |
| Without SFT | |||||
| Base Qwen3.5-4B [ 49 ] | 312 | 23.64 | 23.52 | 25.76 | 0.099 |
| Direct RL, step 414 | 537 | 40.68 | 39.55 | 99.24 | 0.123 |
| SFT initializations, before and after RL | |||||
| SFT/4 | 686 | 51.97 | 52.14 | 81.52 | 0.414 |
| Removed tool | Subset | Questions | Full | Removed | pp |
| Image | All questions | 1,320 | 959 | 907 | |
| Memory | All questions | 1,320 | 959 | 900 | |
| Topology | All questions | 1,320 | 959 | 881 | |
| Timeline | All questions | 1,320 | 959 | 845 | |
| Video | All questions | 1,320 | 959 | 620 | |
| Topology | Q05 | 120 | 88 | 65 |
| Controller | Micro, % | Turns | Tool calls | Controller tokens | Tool-backend tokens | Proprietary total | Proprietary per correct |
| EgoClue-SFT+GRPO, 4B | 72.65 | 3.30 | 2.31 | 18.3k † | 38.1k | 38.1k | 52.4k |
| GPT-5.3 Chat [ 43 ] | 65.76 | 4.34 | 2.55 | 25.0k | 38.3k | 63.3k | 96.3k |
| Gemini 4 Argon [ 24 ] | 46.14 | 4.83 | 2.78 | 32.1k | 28.9k | 61.0k | 132.2k |
| Gemini 3.5 Flash-Lite [ 22 ] | 37.58 | 7.06 | 1.15 | 32.6k | 11.7k | 44.3k | 117.8k |
| Claude Haiku 5.5 [ 3 ] | 32.20 | 8.07 | 2.22 | 69.4k | 27.9k | 97.4k | 302.4k |
| Week-long | Cross-env. | |||
| Model | Parsed | Parsed | ||
| Qwen3.8-27B [ 50 ] | 92.80 | 46.97 | 99.75 | 58.07 |
| VideoChat-Flash-7B [ 32 ] | 100.00 | 41.82 | 100.00 | 38.37 |
| MiMo-VL-7B-RL [ 66 ] | 99.39 | 40.68 | 99.75 | 46.25 |
| VideoLLaMA3-7B [ 71 ] | 100.00 | 40.23 | 100.00 | 47.78 |
| InternVideo3-8B [ 67 ] | 100.00 | 39.85 | 100.00 | 44.09 |
| Ability | Questions | Majority | Ridge |
| Q01 | 120 | 30.83 | 42.50 |
| Q02 | 120 | 18.33 | 28.33 |
| Q03 | 120 | 22.50 | 27.50 |
| Q04 | 120 | 25.00 | 27.50 |
| Q05 | 120 | 25.00 | 25.00 |
| Q06 | 120 | 35.00 | 43.33 |
| Identifier type | Shared atoms | Affected questions |
| Core group | 0 | 0 |
| Exact public input | 0 | 0 |
| Object episode | 503 | 364 |
| Object event | 209 | 445 |
| Reference | 566 | 562 |
| Room anchor | 229 | 227 |
| Unshared identifiers | SFT | GRPO | ||
| Reference | 114 | 35.09 | 56.14 | +21.05 |
| Object event or episode | 32 | 31.25 | 65.63 | +34.38 |
| Room event | 81 | 76.54 | 88.89 | +12.35 |
| All recorded atoms | 113 | 63.72 | 82.30 | +18.58 |
| Continuous joint | Strict semantic success | ||||||
| Type | Applicable | SFT | GRPO | + Sol | SFT | GRPO | + Sol |
| Q02 | 95 | 34.74–49.47 | 33.68–47.37 | 43.16–61.05 | 34.74–49.47 | 33.68–47.37 | 43.16–61.05 |
| Q03 | 120 | 0.00–8.33 | 0.00–17.50 | 0.00–20.00 | 0.00–8.33 | 0.00–17.50 | 0.00–20.00 |
| Q05 | 120 | 40.00 | 52.08 | 54.17 | 29.17 | 37.50 | 39.17 |
| Q06 | 120 | 56.67 | 62.92 | 60.00 | 45.00 | 52.50 | 51.67 |
| Q07A | 120 | 61.66 | 69.79 | 68.95 | 55.83 | 61.67 | 62.50 |
| Model or system | Q01 | Q04 | Q06 | Q07 | Macro | Micro |
| Direct visual reasoning, ordered by Micro | ||||||
| Gemini 4 Argon [ 24 ] | 90.00 | 95.27 | 83.75 | 82.06 | 87.77 | 86.28 |
| Claude Opus 5.5 [ 4 ] | 77.50 | 94.08 | 80.00 | 76.56 | 82.03 | 80.81 |
| GPT-6.1 Sol [ 44 ] | 70.83 | 73.96 | 56.25 | 66.27 | 66.83 | 67.60 |
| Qwen3.8-27B [ 50 ] | 53.33 | 76.33 | 62.50 | 51.20 | 60.84 | 58.07 |
| Gemma 4 12B [ 20 ] | 49.17 | 55.62 | 53.75 | 46.89 | 51.36 | 49.81 |
| Evaluation scope | Questions | 4B, % | + Sol, % | , pp | Corrected | Regressed |
| Week-long | 1,320 | 72.65 | 74.85 | +2.20 | 75 | 46 |
| Cross-environment | 787 | 78.40 | 80.56 | +2.16 | 41 | 24 |
| Metric | Questions | 95% CI | |
| Answer | 787 | +2.16 | |
| Semantic | 498 | ||
| Joint | 498 | +1.09 | |
| Strict | 498 | +0.80 |
| Type | Err. (%) | Dominant error direction |
| Object-instance memory | ||
| Q01 | 31.1 | different categories 44%; merge 36% |
| Q02 | 19.7 | over-report 89%; under-report 11% |
| Q03 | 69.4 | – |
| Q04 | 29.3 | omit a positive window 56% |
| Room and visit memory | ||
Spatial Reasoning Benchmarks
Spatial reasoning benchmarks test whether AI systems can perceive, represent, and act on geometric, relational, and viewpoint-dependent information.
See all 182 papers