EgoProbe: Benchmarking Long-Range and Exact Memory in Streaming Egocentric Videos

Haokai Pang; Yuqian Fu; Yifei Wang; Yujiao Shen; Tianwen Qian; Mohamed Elhoseiny; Hongbo Fu

cs.CVOct 12, 2026

EgoProbe: Benchmarking Long-Range and Exact Memory in Streaming Egocentric Videos

Authors: Haokai Pang, Yuqian Fu, Yifei Wang, Yujiao Shen, Tianwen Qian, Mohamed Elhoseiny, Hongbo Fu

Organizations: HKUST · KAUST · ECNU · NTU

Abstract

Egocentric video provides a continuous first-person record of everyday activities, making it a natural foundation for wearable AI assistants. Unlike offline video understanding, streaming systems must process observations as they arrive, while queries may refer to experiences from the distant past. Recent work has begun to explore whether information can be recalled across long temporal gaps. Yet remembering how far is only part of the challenge: recurring objects, actions, and locations also require models to remember exactly which detail belongs to which event. We introduce EgoProbe, a benchmark that evaluates streaming egocentric memory along these two complementary dimensions: long-range retention, i.e., how far, and event-specific recall under interference, i.e., how exact. Built on long-form egocentric videos, EgoProbe uses an event-grounded data engine to extend recall intervals, identify related events, and construct event-grounded distractors, followed by human verification. It contains two subsets: EgoProbe-Base, which evaluates long-range recall without intervening related events, and EgoProbe-Echo, which challenges models with similar intervening events and event-grounded distractors. EgoProbe-Echo adopts coarse-to-fine multi-turn probing: the first turn evaluates coarse-grained event recall, while the second tests whether the model can retrieve the target-specific detail associated with the designated event. Extensive evaluation across diverse models reveals substantial limitations in long-range recall, with further challenges in recovering target-event details among related experiences. Even when models correctly answer coarse-grained questions, they often fail to recall the specific details of the designated event. EgoProbe exposes this underexplored failure mode and provides a testbed for advancing long-range and exact memory in streaming egocentric video understanding.

Figures & tables

Benchmark	Egocentric	Streaming	Memory Recall	Event-grounded Distractors	Multi-turn Dialogue	Max. Video Span	Max. Recall Duration
EgoTempo Plizzari et al. [2025]	✓	✗	✗	✗	✗	140 s	—
EgoLife / EgoLifeQA Yang et al. [2025]	✓	✗	✓	✗	✗	7 days	$>24$ h †
OVO-Bench Niu et al. [2025]	$\triangle$	✓	✓	✗	✗	2.51 h	—
RIVER Shi et al. [2026]	$\triangle$	✓	✓	✗	✗	4.11 h	1 h
StreamingBench Lin et al. [2026a]	✗	✓	✓	✗	✓	24 min	—
EgoStream Forte et al. [2026]	✓	✓	✓	✗	✗	7 days ∗	45.3 h

Table 1: Comparison with related benchmarks.

\triangle

denotes partial egocentric coverage; — denotes unavailable comparable statistics. Event-grounded distractors link options to evidence from other related events. Video span includes recording gaps. ∗ EgoStream’s span refers to its EgoLife source. † EgoLife reports recall intervals exceeding 24 h without specifying the exact maximum.

Figure 1: EgoProbe data engine. Four automatic stages transform source QA into streaming memory probes: related-event discovery, recall interval extension, query disambiguation, and multiple-choice and dialogue generation. The resulting QA items are then reviewed by human annotators.

Figure 2: EgoProbe statistics : (a) recall intervals and (b) queried information types across EgoProbe; (c) related events and (d) event-grounded distractors per question across EgoProbe-Echo.

		EgoProbe-Base	EgoProbe-Echo
Model	Backbone	Single-turn	Multi-turn		Single-turn
		Acc. $\uparrow$	R1 Acc. $\uparrow$	R2 Acc. $\uparrow$	Acc. $\uparrow$
Proprietary MLLMs
Gemini-3.5-Flash	–	51.10	79.25	33.77	43.69
GPT-5.6-Sol	–	55.32	89.75	43.69	43.37
Open-source General-purpose MLLMs

Table 2: Overall accuracy (%) on EgoProbe. Bold indicates best within each model category. Row colors indicate offline evaluation ( ) and streaming approaches based on recent windows ( ), KV-cache management ( ), event-centric memory ( ), object-centric memory ( ), or streaming reasoning and interaction ( ).

	1 event			2 events			$\geq$ 3 events
Model	Acc. $\uparrow$	Event err. $\downarrow$	Other err. $\downarrow$	Acc. $\uparrow$	Event err. $\downarrow$	Other err. $\downarrow$	Acc. $\uparrow$	Event err. $\downarrow$	Other err. $\downarrow$
Random choice	25.00	25.00	50.00	25.00	36.93	38.07	25.00	53.01	21.99
StreamingVLM	37.08	35.47	27.45	22.41	52.28	25.31	27.12	63.56	9.32
HERMES	36.44	36.44	27.13	31.54	46.89	21.58	32.05	60.55	7.40
StreamForest	43.18	38.04	18.78	39.83	44.40	15.77	30.96	61.64	7.40
OASIS	42.54	34.19	23.27	29.88	56.02	14.11	29.59	65.21	5.21

Table 4: Single-turn answer outcomes (%) on EgoProbe-Echo. Event err. and Other err. denote event-linked and other-option error rates, respectively. Bold marks the highest accuracy and lowest error rates among models within each event group. The gray row denotes random choice.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: EgoProbe-Base single-turn example. The model must recall the earlier location of the shoes after approximately 17 minutes.

Figure 5: EgoProbe-Echo example with one event-grounded distractor. R1 asks about shared event content, while the single-turn question and R2 require the target-specific ingredient.

Figure 6: EgoProbe-Echo example with two event-grounded distractors. The wooden-plank and pocket locations are supported by later related events but are incorrect for the event preceding use of the gauge.

Role	Instruction and constraint
Event assessment	Determine whether the candidate is a distinct related event. Extract an answer only when its own descriptions support the queried detail; retrieval similarity is not evidence.
Clarity detector	Apply the full question to the supplied pre-query events. Identify alternative plausible referents without answering or rewriting the question.
Question editor	Add minimal target-grounded cues that distinguish the target. Preserve the queried fact and correct answer, and do not reveal that answer.
Revision verifier	Check target identification, evidence for added cues, and answer leakage. Return specific feedback when a revision fails.
Option construction	Prioritize safely supported related-event answers, fill the remaining slots, and reject equivalent options or additional correct answers.
Dialogue construction	Identify shared content supported by the selected events, ask a coarse yes/no question, and adapt the original detail question into a follow-up.

Appendix

Table 5: Core instructions used by the construction roles.

Figure 7: Human-review interface. Annotators navigate timestamped evidence clips, check QA and event-linked options, and record decisions with reasons and suggested edits. Labels shown in the interface are subject to verification.

Model	Input and principal settings
Gemini / GPT	128 prefix frames; maximum side 448 pixels; high image detail.
Qwen3.5	128 prefix frames; per-frame pixel budget 200,704; non-thinking.
InternVL3.5	128 prefix frames; one $448\times 448$ patch per frame.
LLaVA-OneVision-2	128 prefix frames; frame input rather than codec input.
Ego-R1	1 FPS; 30-frame caption chunks; agent limit of seven turns; temporal video retrieval.
EgoRAG	1 FPS; 30-frame caption chunks; Qwen3-32B text retrieval/answering.

Appendix

Table 6: Input and memory settings recovered from evaluation records. Frame budgets describe the available input, not the full duration of the source stream.

Figure 8: EgoProbe-Base accuracy (%) across recall intervals for the nine models in Table 3 .

Model / Input	Base Acc. $\uparrow$	Echo Acc. $\uparrow$
Qwen3.5 — Text-only	25.51	24.49
Qwen3.5 — Video + Text	45.91	35.31

Appendix

Table 7: Qwen3.5 single-turn accuracy (%) with and without video on EgoProbe-Base and EgoProbe-Echo single turn QAs.

	1 event		2 events		3 events		4 events		$\geq$ 5 events
Model	Acc. $\uparrow$	Event err. $\downarrow$	Acc. $\uparrow$	Event err. $\downarrow$	Acc. $\uparrow$	Event err. $\downarrow$	Acc. $\uparrow$	Event err. $\downarrow$	Acc. $\uparrow$	Event err. $\downarrow$
StreamingVLM	37.08	35.47	22.41	52.28	29.60	60.00	26.87	58.21	25.43	68.21
HERMES	36.44	36.44	31.54	46.89	37.60	52.80	25.37	64.18	30.64	64.74
StreamForest	43.18	38.04	39.83	44.40	31.20	60.00	28.36	59.70	31.79	63.58
OASIS	42.54	34.19	29.88	56.02	40.00	54.40	16.42	76.12	27.17	68.79
EventMemAgent	44.46	31.94	33.61	46.89	28.00	62.40	20.90	68.66	32.95	63.01

Appendix

Table 8: Fine-grained EgoProbe-Echo single-turn results (%). Groups count annotated related events supporting different answers. Event err. denotes event-linked error rate. Bold indicates the highest accuracy and lowest error rate among reported models in each group.

Egocentric Video Understanding

Egocentric video understanding studies how AI interprets first-person video to recognize actions, track interactions, reason about space and time, and assist users.

See all 200 papers

May 29, 2026

EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision

Continuous episodic memory is a core capability for autonomous agents operating in dynamic, real-world environments, yet current streaming video benchmarks provide limited tools for diagnosing what models remember and for how long. We introduce Egostream, a diagnostic benchmark for streaming episodic memory evaluation in egocentric vision. \egostream organizes 2,250 curated questions along seven cognitive dimensions: detail, spatial, temporal, event, social, causal, and prospective memory. We introduce the Answer Validity Window (AVW), which specifies the temporal span an answer remains valid as the observed scene evolves. This allows us to expand the questions into 8,528 recall-conditioned evaluations, enabling controlled testing from instant to ultra-long-term recall while separating genuine model forgetting from natural world-state changes. We rigorously establish baseline performance through a unified streaming MLLM framework that compares several state-of-the-art memory-management mechanisms, covering sliding windows, attention sinks, KV-cache pruning, merging, and offloading. Experiments within a unified Qwen3-VL backbone reveal that comparable aggregate accuracies mask starkly different memory profiles. For instance, token pruning preserves fine-grained details and temporal structure significantly better than token merging, while quantized offloading rescues ultra-long-term recall. Ultimately, all mechanisms operate well below real-time (>1s per frame), and top performing methods ceil at about 45% accuracy, exposing critical gaps in current architectures. Egostream provides the diagnostic testbed needed to close these gaps. Project website, news and updates at: https://saroo25.github.io/Egostream/

Rosario Forte, Giuseppe Lando, Antonino Furnari

Department of Mathematics and Computer Science University of Catania

VLM Evaluation Streaming Video Understanding Episodic Memory Egocentric Video Understanding Memory Agent Benchmarks Agent Memory Management

Ask PDF

Oct 21, 2026

ProactEgoMem: Benchmarking Proactive Memory in Streaming Egocentric Video

Today's multimodal assistants are typically prompted to interpret the visual evidence presented to them. To offload the cognitive burden of remembering across long horizons, wearable assistants should instead proactively acquire and maintain useful memory as the user's surroundings evolve. Doing so on the user's behalf requires identifying what matters, understanding the scene in real time, and initiating memory maintenance in the background while the user stays focused on their activities. We introduce ProactEgoMem, a benchmark that evaluates proactive memory in streaming egocentric video. Three tasks ground the evaluation in ambient environmental needs: Information Capture (IC) preserves transient yet dense visual information, State Tracking (ST) follows changing object states, and Location Tracking (LT) records object movements. The benchmark defines its protocol at two levels of proactiveness: the Notification Level asks for a single timely notification that satisfies the user's intent, while the deeper Intervention Level asks the agent to maintain memory on the user's behalf through an active start-to-stop tool lifecycle. Parallel ProactEgoMem-Daily and ProactEgoMem-Sci tracks contain 300/100 clips covering daily and scientific use cases, paired with 1000/500 questions in ProactEgoMem-QA, which measures the quality of the proactively acquired memory. Across proprietary, open-source, video-language, and streaming baselines, even the strongest models fail to capture memory both timely and accurately. ProactEgoMem provides a rigorous evaluation foundation for proactive multimodal assistance and for video agents that co-exist with their users.

Ruotong Liao, Meikel Kokowski, Xiaoxiao Sun +6

LMU Munich · Technical University of Munich · Stanford University +1

Agent Memory Memory Agent Benchmarks Egocentric Video Understanding Streaming Video Understanding

Ask PDF

Jul 2, 2026

S-EMBER: A Large-Scale Benchmark for Streaming Egocentric Memory Retrieval

As wearable devices enable continuous first-person recording, AI assistants must reason across long time horizons to recall past experiences-a capability known as episodic memory. Current benchmarks often rely on offline evaluation with access to entire video files, failing to simulate the streaming reality of wearable intelligence. We introduce S-EMBER (Streaming Egocentric Memory Benchmark for Episodic Retrieval), a large-scale benchmark comprising 3,141 videos totaling 388 hours of organic activity captured via Ray-Ban Meta smart glasses. S-EMBER formalizes grounded streaming episodic retrieval, a paradigm shift from global offline search to causal, active recall triggered by visual events in a continuous stream. We provide 9,448 QA pairs requiring manual visual proof through precise temporal localization and supporting flexible response lengths to simulate natural human-AI interaction. Our extensive benchmarking of frontier models uncovers a localization paradox: while semantic reasoning improves with parameter scale, temporal grounding precision remains a stagnant architectural bottleneck that does not benefit from brute-force increases in model size, resolution, or frame density. S-EMBER establishes a hardware-authentic foundation for developing grounded, reliable episodic memory in the next generation of wearable AI agents.

Xiaodong Wang, Xuanyi Zhao, Pedro Rodriguez +7

1FAIR, Meta · 2Reality Labs, Meta

Temporal Video Grounding Episodic Memory Egocentric Video Understanding AI Agent Benchmarks Memory Agent Benchmarks

Ask PDF

Abstract

Egocentric Video Understanding

Explore similar work

EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision

ProactEgoMem: Benchmarking Proactive Memory in Streaming Egocentric Video

S-EMBER: A Large-Scale Benchmark for Streaming Egocentric Memory Retrieval