EgoProbe: Benchmarking Long-Range and Exact Memory in Streaming Egocentric Videos

Haokai Pang; Yuqian Fu; Yifei Wang; Yujiao Shen; Tianwen Qian; Mohamed Elhoseiny; Hongbo Fu

cs.CVOct 12, 2026

EgoProbe: Benchmarking Long-Range and Exact Memory in Streaming Egocentric Videos

Authors: Haokai Pang, Yuqian Fu, Yifei Wang, Yujiao Shen, Tianwen Qian, Mohamed Elhoseiny, Hongbo Fu

Organizations: HKUST · KAUST · ECNU · NTU

Abstract

Egocentric video provides a continuous first-person record of everyday activities, making it a natural foundation for wearable AI assistants. Unlike offline video understanding, streaming systems must process observations as they arrive, while queries may refer to experiences from the distant past. Recent work has begun to explore whether information can be recalled across long temporal gaps. Yet remembering how far is only part of the challenge: recurring objects, actions, and locations also require models to remember exactly which detail belongs to which event. We introduce EgoProbe, a benchmark that evaluates streaming egocentric memory along these two complementary dimensions: long-range retention, i.e., how far, and event-specific recall under interference, i.e., how exact. Built on long-form egocentric videos, EgoProbe uses an event-grounded data engine to extend recall intervals, identify related events, and construct event-grounded distractors, followed by human verification. It contains two subsets: EgoProbe-Base, which evaluates long-range recall without intervening related events, and EgoProbe-Echo, which challenges models with similar intervening events and event-grounded distractors. EgoProbe-Echo adopts coarse-to-fine multi-turn probing: the first turn evaluates coarse-grained event recall, while the second tests whether the model can retrieve the target-specific detail associated with the designated event. Extensive evaluation across diverse models reveals substantial limitations in long-range recall, with further challenges in recovering target-event details among related experiences. Even when models correctly answer coarse-grained questions, they often fail to recall the specific details of the designated event. EgoProbe exposes this underexplored failure mode and provides a testbed for advancing long-range and exact memory in streaming egocentric video understanding.

Figures & tables

Benchmark	Egocentric	Streaming	Memory Recall	Event-grounded Distractors	Multi-turn Dialogue	Max. Video Span	Max. Recall Duration
EgoTempo Plizzari et al. [2025]	✓	✗	✗	✗	✗	140 s	—
EgoLife / EgoLifeQA Yang et al. [2025]	✓	✗	✓	✗	✗	7 days	$>24$ h †
OVO-Bench Niu et al. [2025]	$\triangle$	✓	✓	✗	✗	2.51 h	—
RIVER Shi et al. [2026]	$\triangle$	✓	✓	✗	✗	4.11 h	1 h
StreamingBench Lin et al. [2026a]	✗	✓	✓	✗	✓	24 min	—
EgoStream Forte et al. [2026]	✓	✓	✓	✗	✗	7 days ∗	45.3 h

Table 1: Comparison with related benchmarks.

\triangle

denotes partial egocentric coverage; — denotes unavailable comparable statistics. Event-grounded distractors link options to evidence from other related events. Video span includes recording gaps. ∗ EgoStream’s span refers to its EgoLife source. † EgoLife reports recall intervals exceeding 24 h without specifying the exact maximum.

Figure 1: EgoProbe data engine. Four automatic stages transform source QA into streaming memory probes: related-event discovery, recall interval extension, query disambiguation, and multiple-choice and dialogue generation. The resulting QA items are then reviewed by human annotators.

Figure 2: EgoProbe statistics : (a) recall intervals and (b) queried information types across EgoProbe; (c) related events and (d) event-grounded distractors per question across EgoProbe-Echo.

		EgoProbe-Base	EgoProbe-Echo
Model	Backbone	Single-turn	Multi-turn		Single-turn
		Acc. $\uparrow$	R1 Acc. $\uparrow$	R2 Acc. $\uparrow$	Acc. $\uparrow$
Proprietary MLLMs
Gemini-3.5-Flash	–	51.10	79.25	33.77	43.69
GPT-5.6-Sol	–	55.32	89.75	43.69	43.37
Open-source General-purpose MLLMs

Table 2: Overall accuracy (%) on EgoProbe. Bold indicates best within each model category. Row colors indicate offline evaluation ( ) and streaming approaches based on recent windows ( ), KV-cache management ( ), event-centric memory ( ), object-centric memory ( ), or streaming reasoning and interaction ( ).

	1 event			2 events			$\geq$ 3 events
Model	Acc. $\uparrow$	Event err. $\downarrow$	Other err. $\downarrow$	Acc. $\uparrow$	Event err. $\downarrow$	Other err. $\downarrow$	Acc. $\uparrow$	Event err. $\downarrow$	Other err. $\downarrow$
Random choice	25.00	25.00	50.00	25.00	36.93	38.07	25.00	53.01	21.99
StreamingVLM	37.08	35.47	27.45	22.41	52.28	25.31	27.12	63.56	9.32
HERMES	36.44	36.44	27.13	31.54	46.89	21.58	32.05	60.55	7.40
StreamForest	43.18	38.04	18.78	39.83	44.40	15.77	30.96	61.64	7.40
OASIS	42.54	34.19	23.27	29.88	56.02	14.11	29.59	65.21	5.21

Table 4: Single-turn answer outcomes (%) on EgoProbe-Echo. Event err. and Other err. denote event-linked and other-option error rates, respectively. Bold marks the highest accuracy and lowest error rates among models within each event group. The gray row denotes random choice.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: EgoProbe-Base single-turn example. The model must recall the earlier location of the shoes after approximately 17 minutes.

Figure 5: EgoProbe-Echo example with one event-grounded distractor. R1 asks about shared event content, while the single-turn question and R2 require the target-specific ingredient.

Figure 6: EgoProbe-Echo example with two event-grounded distractors. The wooden-plank and pocket locations are supported by later related events but are incorrect for the event preceding use of the gauge.

Role	Instruction and constraint
Event assessment	Determine whether the candidate is a distinct related event. Extract an answer only when its own descriptions support the queried detail; retrieval similarity is not evidence.
Clarity detector	Apply the full question to the supplied pre-query events. Identify alternative plausible referents without answering or rewriting the question.
Question editor	Add minimal target-grounded cues that distinguish the target. Preserve the queried fact and correct answer, and do not reveal that answer.
Revision verifier	Check target identification, evidence for added cues, and answer leakage. Return specific feedback when a revision fails.
Option construction	Prioritize safely supported related-event answers, fill the remaining slots, and reject equivalent options or additional correct answers.
Dialogue construction	Identify shared content supported by the selected events, ask a coarse yes/no question, and adapt the original detail question into a follow-up.

Appendix

Table 5: Core instructions used by the construction roles.

Figure 7: Human-review interface. Annotators navigate timestamped evidence clips, check QA and event-linked options, and record decisions with reasons and suggested edits. Labels shown in the interface are subject to verification.

Model	Input and principal settings
Gemini / GPT	128 prefix frames; maximum side 448 pixels; high image detail.
Qwen3.5	128 prefix frames; per-frame pixel budget 200,704; non-thinking.
InternVL3.5	128 prefix frames; one $448\times 448$ patch per frame.
LLaVA-OneVision-2	128 prefix frames; frame input rather than codec input.
Ego-R1	1 FPS; 30-frame caption chunks; agent limit of seven turns; temporal video retrieval.
EgoRAG	1 FPS; 30-frame caption chunks; Qwen3-32B text retrieval/answering.

Appendix

Table 6: Input and memory settings recovered from evaluation records. Frame budgets describe the available input, not the full duration of the source stream.

Figure 8: EgoProbe-Base accuracy (%) across recall intervals for the nine models in Table 3 .

Model / Input	Base Acc. $\uparrow$	Echo Acc. $\uparrow$
Qwen3.5 — Text-only	25.51	24.49
Qwen3.5 — Video + Text	45.91	35.31

Appendix

Table 7: Qwen3.5 single-turn accuracy (%) with and without video on EgoProbe-Base and EgoProbe-Echo single turn QAs.

	1 event		2 events		3 events		4 events		$\geq$ 5 events
Model	Acc. $\uparrow$	Event err. $\downarrow$	Acc. $\uparrow$	Event err. $\downarrow$	Acc. $\uparrow$	Event err. $\downarrow$	Acc. $\uparrow$	Event err. $\downarrow$	Acc. $\uparrow$	Event err. $\downarrow$
StreamingVLM	37.08	35.47	22.41	52.28	29.60	60.00	26.87	58.21	25.43	68.21
HERMES	36.44	36.44	31.54	46.89	37.60	52.80	25.37	64.18	30.64	64.74
StreamForest	43.18	38.04	39.83	44.40	31.20	60.00	28.36	59.70	31.79	63.58
OASIS	42.54	34.19	29.88	56.02	40.00	54.40	16.42	76.12	27.17	68.79
EventMemAgent	44.46	31.94	33.61	46.89	28.00	62.40	20.90	68.66	32.95	63.01

Appendix

Table 8: Fine-grained EgoProbe-Echo single-turn results (%). Groups count annotated related events supporting different answers. Event err. denotes event-linked error rate. Bold indicates the highest accuracy and lowest error rate among reported models in each group.

Egocentric Video Understanding

Egocentric video understanding studies how AI interprets first-person video to recognize actions, track interactions, reason about space and time, and assist users.

See all 200 papers

EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision
May 29, 2026Rosario Forte, Giuseppe Lando, Antonino FurnariVLM Evaluation Streaming Video Understanding
Department of Mathematics and Computer Science University of Catania
ProactEgoMem: Benchmarking Proactive Memory in Streaming Egocentric Video
Oct 21, 2026Ruotong Liao, Meikel Kokowski, Xiaoxiao Sun +6Agent Memory Memory Agent Benchmarks
LMU Munich · Technical University of Munich · Stanford University +1
S-EMBER: A Large-Scale Benchmark for Streaming Egocentric Memory Retrieval
Jul 2, 2026Xiaodong Wang, Xuanyi Zhao, Pedro Rodriguez +7Temporal Video Grounding Episodic Memory
1FAIR, Meta · 2Reality Labs, Meta

May 29, 2026J/K move · Enter open · S save

EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision

Rosario Forte, Giuseppe Lando, Antonino Furnari

Department of Mathematics and Computer Science University of Catania

VLM Evaluation Streaming Video Understanding Episodic Memory Egocentric Video Understanding Memory Agent Benchmarks Agent Memory Management

Open paper AskPDF arXiv

Continuous episodic memory is a core capability for autonomous agents operating in dynamic, real-world environments, yet current streaming video benchmarks provide limited tools for diagnosing what models remember and for how long. We introduce Egostream, a diagnostic benchmark for streaming episodic memory evaluation in egocentric vision. \egostream organizes 2,250 curated questions along seven cognitive dimensions: detail, spatial, temporal, event, social, causal, and prospective memory. We introduce the Answer Validity Window (AVW), which specifies the temporal span an answer remains valid as the observed scene evolves. This allows us to expand the questions into 8,528 recall-conditioned evaluations, enabling controlled testing from instant to ultra-long-term recall while separating genuine model forgetting from natural world-state changes. We rigorously establish baseline performance through a unified streaming MLLM framework that compares several state-of-the-art memory-management mechanisms, covering sliding windows, attention sinks, KV-cache pruning, merging, and offloading. Experiments within a unified Qwen3-VL backbone reveal that comparable aggregate accuracies mask starkly different memory profiles. For instance, token pruning preserves fine-grained details and temporal structure significantly better than token merging, while quantized offloading rescues ultra-long-term recall. Ultimately, all mechanisms operate well below real-time (>1s per frame), and top performing methods ceil at about 45% accuracy, exposing critical gaps in current architectures. Egostream provides the diagnostic testbed needed to close these gaps. Project website, news and updates at: https://saroo25.github.io/Egostream/

Abstract

Egocentric Video Understanding

Explore similar work

EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision

ProactEgoMem: Benchmarking Proactive Memory in Streaming Egocentric Video

S-EMBER: A Large-Scale Benchmark for Streaming Egocentric Memory Retrieval