ProactEgoMem: Benchmarking Proactive Memory in Streaming Egocentric Video

Ruotong Liao; Meikel Kokowski; Xiaoxiao Sun; Yongqin Xian; Laura Bravo Sánchez; Shiye Su; Federico Tombari; Serena Yeung-Levy

cs.CVOct 21, 2026

ProactEgoMem: Benchmarking Proactive Memory in Streaming Egocentric Video

Authors: Ruotong Liao, Meikel Kokowski, Xiaoxiao Sun, Yongqin Xian, Laura Bravo Sánchez, Shiye Su, Federico Tombari, Serena Yeung-Levy,

Organizations: LMU Munich · Technical University of Munich · Stanford University · Google

Abstract

Today's multimodal assistants are typically prompted to interpret the visual evidence presented to them. To offload the cognitive burden of remembering across long horizons, wearable assistants should instead proactively acquire and maintain useful memory as the user's surroundings evolve. Doing so on the user's behalf requires identifying what matters, understanding the scene in real time, and initiating memory maintenance in the background while the user stays focused on their activities. We introduce ProactEgoMem, a benchmark that evaluates proactive memory in streaming egocentric video. Three tasks ground the evaluation in ambient environmental needs: Information Capture (IC) preserves transient yet dense visual information, State Tracking (ST) follows changing object states, and Location Tracking (LT) records object movements. The benchmark defines its protocol at two levels of proactiveness: the Notification Level asks for a single timely notification that satisfies the user's intent, while the deeper Intervention Level asks the agent to maintain memory on the user's behalf through an active start-to-stop tool lifecycle. Parallel ProactEgoMem-Daily and ProactEgoMem-Sci tracks contain 300/100 clips covering daily and scientific use cases, paired with 1000/500 questions in ProactEgoMem-QA, which measures the quality of the proactively acquired memory. Across proprietary, open-source, video-language, and streaming baselines, even the strongest models fail to capture memory both timely and accurately. ProactEgoMem provides a rigorous evaluation foundation for proactive multimodal assistance and for video agents that co-exist with their users.

Figures & tables

Figure 2: ProactEgoMem measures proactive memory as lifecycle behavior in egocentric streams. At the Notification Level the agent sends one timely message; at the Intervention Level it starts, maintains, and ends a memory tool, and releases a message when the task is completed.

Panel of Figure 3 — Figure 3: Two- level proactive modes .

Panel of Table 1 — Table 1: Illustrative examples of proactive messages for each task and mode.

Figure 4: ProactEgoMem composition and target objects. Left : composition by scenario (inner ring), track (middle), and task (outer). Centre : the ten most frequent target objects per track, counted in annotated events. Right : the full object vocabulary, sized by annotated events.

Task	Category	Question and candidates
IC	Text	What were the Rental Center hours on Sunday? a. 10a–4p b. 1p–6p c. 12p–5p d. 10a–6p
	Sequence	What was the first packaged product likely to be? a. Bacon strips b. Olive oil c. Ham d. Mushrooms
	Attribute	What kind of oil did I pour into the frying pan? a. Sesame oil b. Canola oil c. Olive oil d. Sunflower oil
	Price	How much was the Pop Mandalorian item? a. $20 b.$ 11 c. $9.50 d.$ 10
	Legibility	How readable were the Dominoes rules? a. Barely readable b. Written in large print c. Completely unreadable d. Clearly readable
	Direction	Which way did the sign point for Outdoor Center Rentals? a. To the left b. Back behind you c. To the right d. Straight ahead

Table 2: ProactEgoMem-QA question types and benchmark examples

Statistic	Value	Statistic	Value
Unique clips	300/100	closed-ended	924/1218
Source videos	239/100	open-ended	318/159
Total clip len.	16.00/6.47 h	Mean event len.	31.82/13.33 s
Mean clip len.	192.0/233.0 s	Median event len.	13.47/4.77 s
Annotated events	1242/1377	Distinct objects	292/89

Table 3: ProactEgoMem data statistics and inter-annotator agreement. Paired entries read Daily/Sci.

N

: reference-accepted GT events; agreement is averaged over three annotators.

Task	Group	Model	Notification Level						Intervention Level
			Timing	Abs Error		Detection			Overlap	Timing	Abs Error		Detection			QA
			[email protected] $\uparrow$	$\|\Delta_{s}\|{\downarrow}$	$\sigma{\downarrow}$	Prec $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	mIoU $\uparrow$	[email protected] $\uparrow$	$\|\Delta_{s}\|{\downarrow}$	$\sigma{\downarrow}$	Prec $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	Acc $\uparrow$
IC	Proprietary	Gemini 2.5 Flash	14.2	2.09	2.04	3.4	66.2	6.5	3.4	16.7	3.34	2.88	3.9	10.8	5.8	54.1
	Proprietary	GPT-5.4-mini	17.0	1.70	2.09	1.3	65.6	2.6	2.0	5.3	3.83	2.22	5.1	6.9	5.9	52.5
	Open Source	Gemma4-31B	11.8	2.16	2.25	6.1	62.8	11.1	4.5	8.6	3.01	2.71	3.0	37.9	5.5	66.5
		Qwen3.8-27B	14.9	1.70	1.85	3.2	74.8	6.1	5.6	20.0	2.50	2.64	5.9	10.8	7.6	62.7

Table 4: ProactEgoMem-Daily main results for notification and intervention levels. • [email protected] : matched actions within

0.2

s of the ground-truth start. • mIoU : interval IoU over all opportunities (unmatched scored

0

). • Error columns are seconds, all others percentages; bold : best in a task.

Figure 5: Proactive competence by model family , averaged over the three tasks: proprietary, open-source, video-language, and streaming models on the notification and intervention axes.

Panel of Figure 6 — Figure 6: QA accuracy by task . The No-Memory-to-Oracle gap is the acquisition headroom.

Panel of Figure 7 — Figure 7: From daily life to the laboratory : (a) the two tracks on the radar axes of Figure 5 ; (b) notification recall and QA acquisition gap per task.

Panel of Figure 10 — Figure 10: Failure modes on LT clips (Intervention Level): (a) MMDuet, $39$ calls for $2$ GT events; (b) Gemma4-31B, $111$ one-second tracks for $7$ events; (c) VideoLLaMA3-7B, tracks never stopped.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Video Domain	Setting	Proactive	Multi-Level	Tool Lifecycle	Mem. QA	Ave. Dur. (s)	Response Type
Offline video QA
Video-MME ( Fu et al., 2025 )	General	Offline	✗	✗	✗	✗	1018	Multi-choice
LongVideoBench ( Wu et al., 2024 )	General	Offline	✗	✗	✗	✗	473	Multi-choice
MLVU ( Zhou et al., 2025 )	General	Offline	✗	✗	✗	✗	930	Multi-choice
EgoSchema ( Mangalam et al., 2023 )	Egocentric	Offline	✗	✗	✗	✗	180	Multi-choice
HourVideo ( Chandrasegaran et al., 2024 )	Egocentric	Offline	✗	✗	✗	✗	2742	Multi-choice

Appendix

Table 5: Benchmark comparison . Proactive : acting without an explicit query; Multi-Level : two proactive depths; Tool Lifecycle : a maintained background lifecycle; Mem. QA : questions verify self-acquired memory. (✓) marks a partial case; ProactEgoMem marks all four.

Panel of Figure 12 — Figure 12: Group-level mean Start@ $k$ and intervention IoU@ $\tau$ over tasks and models as each criterion is relaxed. Note the collapsed scale of the rightmost panel: the best group mean stays below 6%.

Figure 13: Predicted start-error distributions by model, pooled across the three tasks, for notification and intervention levels. Points are individual matched starts; the dashed vertical line marks the ground-truth boundary, with positive values late and negative values early. Empty intervention rows indicate that no valid matched intervention was available for that model.

Panel of Figure 14 — Figure 14: Daily-to-Science transfer over the expanded paired results: $39$ model–task cells for notification and intervention, and $36$ for QA (VideoLLM-Online excluded). (a) Mean change per metric: only opportunity coverage moves substantially. (b) The same change split by task: the direction reverses between tasks for strict timing, intervention F1, and QA. (c) Notification -level F1 by model family: the open-source lead in daily life does not survive the move to the laboratory. (d) Mean gap between oracle and proactively acquired memory: acquisition costs more in the laboratory on IC and ST , less on LT .

Panel of Figure 15 — Figure 15: Daily vs. Science, best value across models per metric. Moving to the laboratory lowers notification F1, intervention mIoU, notification coverage, and memory-backed QA. The higher best intervention recall and F1 come from two different single-model LT spikes.

Task	Group	Model	Notification Level						Intervention Level
			Timing	Abs Error		Detection			Overlap	Timing	Abs Error		Detection			QA
			[email protected] $\uparrow$	$\|\Delta_{s}\|{\downarrow}$	$\sigma{\downarrow}$	Prec $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	mIoU $\uparrow$	[email protected] $\uparrow$	$\|\Delta_{s}\|{\downarrow}$	$\sigma{\downarrow}$	Prec $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	Acc $\uparrow$
IC	Proprietary	Gemini 2.5 Flash	31.6	1.23	1.92	2.2	43.4	4.2	2.7	36.4	1.39	2.11	2.5	6.3	3.6	41.8
	Proprietary	GPT-5.4-mini	30.1	1.28	1.94	1.6	47.4	3.2	1.9	20.0	3.41	3.56	4.6	5.7	5.1	46.5
	Open Source	Gemma4-31B	21.4	1.36	1.71	4.0	48.0	7.4	5.4	13.6	2.43	2.75	2.0	25.1	3.8	57.2
		Qwen3.8-27B	22.8	1.39	1.63	2.4	52.6	4.7	3.7	46.2	0.93	1.49	5.9	7.4	6.6	55.9

Appendix

Table 6: ProactEgoMem-Sci main results for both levels, in the layout of Table 4 . • [email protected] , mIoU and Acc are defined as in Table 4 . • Error columns are seconds, all others percentages; bold : best in a task.

Panel of Figure 20 — Figure 20: ProactEgoMem-Sci group-level mean Start@ $k$ and intervention IoU@ $\tau$ over tasks and models as each criterion is relaxed. Note the collapsed scale of the rightmost panel: the best group mean stays below 6%.

Figure 21: Notification- and intervention-level radar summaries for ProactEgoMem-Sci , task-averaged per model family. Axes and normalization follow Figure 5 .

Panel of Figure 22 — Figure 22: ProactEgoMem-Sci ProactEgoMem-QA performance by task. Each model is shown as a horizontal accuracy bar: black marks No Memory, brown marks Proact Memory, and gray marks Oracle Memory.

	ST	IC
Task line	You must decide whether to start tracking a state, stop tracking a state, or do nothing.	You must decide whether to start recording, stop recording, or do nothing.
Concurrency	Several state trackers may be active at the same time.	At most one recording may be active at a time.
Start tool	start_state_tracking(name: a short, specific name for the thing being tracked; state: a concise description of the state to track)	start_recording(name: a short, specific name for the information source; description: a concise description of what the recording should preserve)
Stop tool	stop_state_tracking(id: the tracking id of the active state tracker to stop)	stop_recording(): Arguments: {} -- the tool call has no arguments.
Policy	The user wants help avoiding accidentally leaving things in an unintended state during everyday activity. ... track relevant object states with the state-tracking tool.	The user seeks help in remembering information from their surroundings that may be useful in the future. ... build an ambient memory of the surrounding environment. Maintain broad coverage of relevant objects and their associated information. You should avoid both overly brief recordings and indiscriminate continuous recording.
Message style	"I decided to start/stop tracking a state because ..."	"I decided to start/stop recording because ..."

Appendix

Table 7: Lines of the Intervention-Level prompt that differ from LT . Elsewhere ST reads “state” for “location”; IC drops the instance_id paragraph, since a single recording needs no id.

	ST	IC
Policy	The user wants help avoiding accidentally leaving things in an unintended state during everyday activity. ... notify the user about relevant object states. But do not notify about every single object or state unless it is relevant to the user intent.	The user seeks help in remembering information from their surroundings that may be useful in the future. ... notify the user about relevant objects and their associated information. But do not notify about every single object unless it is relevant to the user intent.
Message style	"You just ..."	"I can see ..."

Appendix

Table 8: Lines of the Notification-Level prompt that differ from LT .

Agent Memory

Agent memory lets AI systems retain, update, and use facts, experiences, and skills across sessions and long-horizon tasks.

See all 190 papers

EgoProbe: Benchmarking Long-Range and Exact Memory in Streaming Egocentric Videos
Oct 12, 2026Haokai Pang, Yuqian Fu, Yifei Wang +4Egocentric Video Understanding Streaming Video Understanding
HKUST · KAUST · ECNU +1
Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos
Jul 13, 2026Gong Sitong, Tianyu Yan, Caixin Kang +6Egocentric Video Understanding Proactive Assistance
Dalian University of Technology · Alaya Lab · The University of Tokyo
GROVE: Growing and Reasoning over Temporally Stratified Memory from Streaming Video Experience
Aug 3, 2026Sitong Gong, Caixin Kang, Tianyu Yan +7Streaming Video Understanding Video QA
1Dalian University of Technology · 2Alaya Lab · 3The University of Tokyo +1

Oct 12, 2026J/K move · Enter open · S save

EgoProbe: Benchmarking Long-Range and Exact Memory in Streaming Egocentric Videos

Haokai Pang, Yuqian Fu, Yifei Wang, Yujiao Shen+3

HKUST · KAUST · ECNU+1

Egocentric Video Understanding Streaming Video Understanding Episodic Memory

Open paper AskPDF arXiv

Egocentric video provides a continuous first-person record of everyday activities, making it a natural foundation for wearable AI assistants. Unlike offline video understanding, streaming systems must process observations as they arrive, while queries may refer to experiences from the distant past. Recent work has begun to explore whether information can be recalled across long temporal gaps. Yet remembering how far is only part of the challenge: recurring objects, actions, and locations also require models to remember exactly which detail belongs to which event. We introduce EgoProbe, a benchmark that evaluates streaming egocentric memory along these two complementary dimensions: long-range retention, i.e., how far, and event-specific recall under interference, i.e., how exact. Built on long-form egocentric videos, EgoProbe uses an event-grounded data engine to extend recall intervals, identify related events, and construct event-grounded distractors, followed by human verification. It contains two subsets: EgoProbe-Base, which evaluates long-range recall without intervening related events, and EgoProbe-Echo, which challenges models with similar intervening events and event-grounded distractors. EgoProbe-Echo adopts coarse-to-fine multi-turn probing: the first turn evaluates coarse-grained event recall, while the second tests whether the model can retrieve the target-specific detail associated with the designated event. Extensive evaluation across diverse models reveals substantial limitations in long-range recall, with further challenges in recovering target-event details among related experiences. Even when models correctly answer coarse-grained questions, they often fail to recall the specific details of the designated event. EgoProbe exposes this underexplored failure mode and provides a testbed for advancing long-range and exact memory in streaming egocentric video understanding.

Abstract

Agent Memory

Explore similar work

EgoProbe: Benchmarking Long-Range and Exact Memory in Streaming Egocentric Videos

Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos

GROVE: Growing and Reasoning over Temporally Stratified Memory from Streaming Video Experience