SPACE: Sparse Predictive Attractor via Counterfactual Eviction for Streaming Video Memory

Hongjin Niu; Weizhan Zhang; Shuo Bao; Jiahao Wang; Muyan Jiao; Kairui Wen; Yong-Jin Liu

cs.CVSep 26, 2026

SPACE: Sparse Predictive Attractor via Counterfactual Eviction for Streaming Video Memory

Authors: Hongjin Niu, Weizhan Zhang, Shuo Bao, Jiahao Wang, Muyan Jiao, Kairui Wen, Yong-Jin Liu

Organizations: School of Computer Science and Technology, Xi’an Jiaotong University · School of Advanced Manufacturing and Robotics, Peking University · Department of Computer Science and Technology, Tsinghua University

Abstract

Fixed-capacity streaming video memory requires repeated eviction decisions whose effects accumulate over time. Yet existing policies are evaluated primarily in terms of retained information or downstream accuracy, leaving how repeated updates alter the futures supported by memory largely unexamined. We define a memory's predictive state as the future representations supported by its retained history and formulate eviction as counterfactual control over transitions in this space. We introduce SPACE (Sparse Predictive Attractor via Counterfactual Eviction), which uses a frozen multi-horizon JEPA to predict the future representations induced by alternative eviction actions. Counterfactual utility identifies future-useful alternatives, while slow predictive-basin geometry determines when to correct avoidable drift and when to adapt to sustained predictive change, without online parameter updates. We further introduce MABS-Bench, which evaluates future-task sufficiency, within-regime predictive stability, transition responsiveness, and perturbation recovery under matched causal streams and memory budgets. Across multiple video datasets, SPACE yields consistent improvements in dataset-native task performance while reducing predictive-state drift.

Figures & tables

Figure 1: Predictive stability of SPACE. Unlike conventional streaming memory, which drifts under repeated updates, SPACE forms a stable yet adaptive predictive attractor, improving long-horizon future alignment while preserving broad memory capabilities. In (c), the six metrics are independently min–max normalized to $[0,1]$ across all methods, with directions aligned so that higher is better; radial rings are spaced by 0.2. Predicted futures are nearest-neighbor RGB visualizations of future latents from a participant-disjoint RGB retrieval bank, not generated pixels.

Figure 2: SPACE overview. Candidate eviction actions are evaluated by a frozen multi-horizon JEPA and selected using complementary future-utility and predictive-basin signals.

Method	Action NLL $\downarrow$	mAP $\uparrow$	LRAP $\uparrow$	Recall@5 $\uparrow$	H64 gain $\uparrow$	Basin drift ( $\times 10^{-5}$ ) $\downarrow$	Boundary selectivity $\uparrow$
FIFO	$3.6820\pm 0.0091$	$0.1339\pm 0.0005$	$0.4743\pm 0.0017$	$0.7030\pm 0.0012$	$-0.0561\pm 0.0246$	$8.7180\pm 2.343$	$0.8229\pm 0.0220$
Reservoir	$\underline{3.6315\pm 0.0209}$	$\underline{0.1466\pm 0.0012}$	$\underline{0.4875\pm 0.0021}$	$\underline{0.7123\pm 0.0026}$	$0.0000\pm 0.0000$	$8.9420\pm 2.429$	$0.8177\pm 0.0206$
Surprise	$3.6633\pm 0.0218$	$0.1443\pm 0.0008$	$0.4797\pm 0.0026$	$0.7080\pm 0.0019$	$-0.0590\pm 0.0075$	$9.3870\pm 2.486$	$0.8135\pm 0.0180$
Similarity	$3.6451\pm 0.0201$	$0.1459\pm 0.0010$	$0.4841\pm 0.0022$	$0.7109\pm 0.0020$	$-0.0179\pm 0.0022$	$8.8410\pm 2.488$	$0.8178\pm 0.0232$
Attention	$3.6632\pm 0.0264$	$0.1375\pm 0.0022$	$0.4793\pm 0.0019$	$0.7060\pm 0.0012$	$-0.0296\pm 0.0162$	$\mathbf{8.1840\pm 2.344}$	$0.8254\pm 0.0187$
Self-Influence	$3.6391\pm 0.0218$	$\mathbf{0.1468\pm 0.0013}$	$0.4870\pm 0.0021$	$0.7115\pm 0.0011$	$\underline{0.0033\pm 0.0022}$	$8.3530\pm 2.492$	$0.8155\pm 0.0203$

Table 1: MABS-Bench results on Assembly101 ( Sener et al., 2022 ) over three complete predictor-and-policy seeds. Values are mean

\pm

sample standard deviation. H64 gain is paired against Reservoir within the same seed. Best mean values are bold and second-best mean values are underlined .

Corruption	Method	Predictive AUC $\uparrow$	H16 recovery $\uparrow$	Memory AUC $\uparrow$	Task AUC $\uparrow$
$25\%$	Reservoir	$\mathbf{0.0328}$	$\mathbf{0.1209}$	$0.0417$	$\underline{0.0399}$
	Utility-only	$-0.3954$	$-0.8692$	$0.0191$	$-0.2098$
	State-only	$-0.3740$	$-0.8425$	$\mathbf{0.0533}$	$-0.3587$
	SPACE	$\underline{0.0309}$	$\underline{0.0727}$	$\underline{0.0426}$	$\mathbf{0.0517}$
$50\%$	Reservoir	$\underline{0.0376}$	$\underline{0.0770}$	$\underline{0.0239}$	$\underline{0.0308}$
	Utility-only	$-0.4916$	$-0.7216$	$0.0082$	$-0.3305$

Table 2: Recovery from controlled memory perturbations on Assembly101. Predictive recovery measures contraction toward the paired clean trajectory, and recovery AUC summarizes offsets H1–H16. Best values within each perturbation level are bold and second-best values are underlined .

Figure 3: Predictive-state recovery after

50\%

perturbations. Positive values denote contraction toward the paired clean trajectory; shading shows 95% participant-bootstrap confidence intervals.

Figure 4: Future-semantic visualization on EPIC-KITCHENS-100. The left column shows the observed context, followed by predictions at H1, H4, H16, and H64. The rows show the ground-truth future and the nearest training frames retrieved using future representations predicted from Reservoir and SPACE memories.

Variant	Action NLL $\downarrow$	H1 gain ( $\times 10^{-2}$ ) $\uparrow$	H4 gain ( $\times 10^{-2}$ ) $\uparrow$	H16 gain ( $\times 10^{-2}$ ) $\uparrow$	H64 gain ( $\times 10^{-2}$ ) $\uparrow$	Drift gain ( $\times 10^{-6}$ ) $\uparrow$	Boundary selectivity $\uparrow$
SPACE	$\mathbf{3.6291\pm 0.0209}$	$\mathbf{0.0189\pm 0.0131}$	$\mathbf{-0.0007\pm 0.0015}$	$\mathbf{0.2574\pm 0.0029}$	$0.6844\pm 0.1626$	$2.5203\pm 0.0804$	$\mathbf{0.8270\pm 0.0219}$
w/o Departure gate	$3.6434\pm 0.0153$	$-2.1477\pm 0.0318$	$-2.2982\pm 1.6290$	$-1.7154\pm 0.0896$	$\mathbf{1.4013\pm 0.7037}$	$0.7356\pm 0.1854$	$0.8249\pm 0.0003$
w/o Utility gate	$3.6522\pm 0.0299$	$-1.6018\pm 0.5118$	$-2.6354\pm 0.7932$	$-2.3207\pm 0.2242$	$-1.7221\pm 2.2370$	$\mathbf{6.0203\pm 0.8612}$	$0.8263\pm 0.0172$
w/o Reachability gate	$3.6314\pm 0.0207$	$-0.2293\pm 0.0042$	$-0.2801\pm 0.0028$	$0.0713\pm 0.0120$	$0.4780\pm 0.4729$	$0.5623\pm 0.0147$	$0.8199\pm 0.0164$
Full-latent predictive state	$3.6431\pm 0.0210$	$-1.4174\pm 0.0020$	$-1.4993\pm 0.0873$	$-1.1734\pm 0.0199$	$-0.5559\pm 0.0716$	$0.5138\pm 0.0085$	$0.8261\pm 0.0097$
w/o Boundary release	$3.6437\pm 0.0209$	$-1.4370\pm 0.0057$	$-1.5063\pm 0.0991$	$-1.1458\pm 0.0380$	$-0.7908\pm 0.0203$	$0.3192\pm 0.0084$	$0.8268\pm 0.0172$

Table 3: Component ablation on Assembly101 over predictor-and-policy seeds. Within each seed, all variants share the same stream, predictor, utility bank, memory capacity, and downstream decoder. Horizon-specific NLL and drift gains are paired against Reservoir. Values are mean

\pm

sample standard deviation. Best mean values are bold and second-best mean values are underlined .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Parameter	Value
JEPA predictor	Hidden dimension	$512$
	Transformer layers / heads	$4/8$
	Feed-forward dimension	$2048$
	Recent-context length	$8$
	Prediction horizons $\mathcal{H}$	$\{1,4,16,64\}$
	Horizon weights $w_{h}$	uniform ( $1/4$ )

Appendix

Table 4: Default implementation configuration. Unless stated otherwise, the same settings are used across datasets.

Setting	$\Delta$ NLL $\uparrow$	H16 gain $\uparrow$	H64 gain $\uparrow$	Drift gain $\uparrow$
$K=8$	$0.000818$	$0.000811$	$0.002413$	$8.833{\times}10^{-8}$
$K=16$	$\mathbf{0.002387}$	$\mathbf{0.002552}$	$\mathbf{0.007412}$	$\mathbf{2.5101{\times}10^{-6}}$
$K=32$	$0.001384$	$0.001633$	$0.004260$	$4.303{\times}10^{-7}$
Default $(\tau_{U},\tau_{G},\tau_{R})=(0.05,0.5,3.0)$	$\mathbf{0.002387}$	$\mathbf{0.002552}$	$0.007412$	$\mathbf{2.5101{\times}10^{-6}}$
$\tau_{U}=0.00$	$0.002239$	$0.002413$	$\mathbf{0.008183}$	$7.153{\times}10^{-7}$
$\tau_{U}=0.10$	$0.001963$	$0.001522$	$0.006591$	$4.254{\times}10^{-7}$

Appendix

Table 5: Single-seed capacity and one-factor threshold sensitivity on Assembly101. Gains are paired against Reservoir with the same memory capacity. Best values within each block are bold.

Metric	Mean gain	Positive	95% bootstrap CI	$p$
Action NLL	$0.002400$	$8/10$	$[0.00105,0.00392]$	$0.0059$
H16 NLL	$0.002574$	$8/10$	$[0.00115,0.00422]$	$0.0059$
H64 NLL	$0.006844$	$9/10$	$[0.00296,0.01190]$	$0.0029$
Basin drift	$2.5203\times 10^{-6}$	$10/10$	$[1.97{\times}10^{-6},3.56{\times}10^{-6}]$	$0.0010$

Appendix

Table 6: Participant-level paired inference for SPACE relative to Reservoir on Assembly101. Gains are averaged across seeds within each participant, and positive values favor SPACE.

Corruption	Method	Median [5th, 95th]	Minimum	Fraction with $d_{0}<10^{-5}$
25%	Reservoir	$1.36{\times}10^{-5}$ [ $2.15{\times}10^{-6}$ , $6.73{\times}10^{-5}$ ]	$1.07{\times}10^{-6}$	35.5%
25%	Utility-only	$1.22{\times}10^{-5}$ [ $2.40{\times}10^{-6}$ , $7.78{\times}10^{-5}$ ]	$1.01{\times}10^{-6}$	33.9%
25%	State-only	$9.30{\times}10^{-6}$ [ $1.14{\times}10^{-6}$ , $6.81{\times}10^{-5}$ ]	$8.94{\times}10^{-7}$	51.6%
25%	SPACE	$1.37{\times}10^{-5}$ [ $2.39{\times}10^{-6}$ , $7.48{\times}10^{-5}$ ]	$1.07{\times}10^{-6}$	35.5%
50%	Reservoir	$2.74{\times}10^{-5}$ [ $3.22{\times}10^{-6}$ , $9.89{\times}10^{-5}$ ]	$1.67{\times}10^{-6}$	13.6%
50%	Utility-only	$2.36{\times}10^{-5}$ [ $2.82{\times}10^{-6}$ , $9.56{\times}10^{-5}$ ]	$1.61{\times}10^{-6}$	22.0%

Appendix

Table 7: Distribution of the intervention-level initial predictive distance

d_{0}

. Intervals denote the 5th–95th percentiles.

Method	Action NLL $\downarrow$	Top-1 $\uparrow$	Top-5 $\uparrow$	H64 gain $\uparrow$	Basin drift ( $\times 10^{-3}$ ) $\downarrow$	Boundary selectivity $\uparrow$
FIFO	$2.6320$	$0.3151$	$0.6256$	$0.0057$	$\underline{2.7040}$	$\underline{0.3538}$
Reservoir	$2.6474$	$0.3173$	$0.6152$	$0.0000$	$3.2163$	$0.2991$
Surprise	$2.7044$	$0.3046$	$0.6007$	$-0.1049$	$3.6405$	$0.3293$
Similarity	$2.6827$	$0.3063$	$0.6049$	$-0.0686$	$3.3748$	$0.3335$
Attention	$\mathbf{2.6126}$	$0.3212$	$0.6305$	$\mathbf{0.0571}$	$2.8566$	$0.3160$
Self-Influence	$2.6332$	$0.3145$	$0.6261$	$\underline{0.0342}$	$2.7257$	$0.3426$

Appendix

Table 8: Results on Breakfast Actions under the matched causal-stream protocol. H64 gain is paired against Reservoir. Best mean values are bold and second-best mean values are underlined .

Method	Action NLL $\downarrow$	Top-1 $\uparrow$	Top-5 $\uparrow$	H64 gain $\uparrow$	Basin drift ( $\times 10^{-3}$ ) $\downarrow$	Boundary selectivity $\uparrow$
FIFO	$2.5686$	$0.3176$	$\mathbf{0.6489}$	$0.0062$	$1.0576$	$0.1914$
Reservoir	$2.5786$	$0.3146$	$0.6482$	$0.0000$	$1.0782$	$\underline{0.2718}$
Surprise	$2.6211$	$0.2883$	$0.6332$	$-0.0591$	$1.1446$	$0.2654$
Similarity	$2.6365$	$0.2686$	$0.6269$	$-0.0696$	$1.0874$	$0.2688$
Attention	$\underline{2.5206}$	$\underline{0.3506}$	$0.6412$	$\underline{0.0549}$	$1.0085$	$0.2133$
Self-Influence	$2.5947$	$0.3078$	$0.6445$	$-0.0341$	$\underline{0.9963}$	$0.2439$

Appendix

Table 9: Results on the IKEA ASM sealed test split under the matched causal-stream protocol. H64 gain is paired against Reservoir. Best mean values are bold and second-best mean values are underlined .

Method	Action NLL $\downarrow$	mAP $\uparrow$	LRAP $\uparrow$	Recall@5 $\uparrow$	H64 gain $\uparrow$	Basin drift ( $\times 10^{-3}$ ) $\downarrow$	Boundary selectivity $\uparrow$
FIFO	$4.3424$	$0.2715$	$\underline{0.4478}$	$0.6198$	$0.0125$	$8.5610$	$0.6106$
Reservoir	$4.3469$	$0.2749$	$0.4473$	$0.6247$	$0.0000$	$9.2249$	$0.5950$
Surprise	$4.3844$	$0.2753$	$0.4426$	$0.6172$	$-0.0879$	$10.6138$	$0.5888$
Similarity	$4.3832$	$0.2746$	$0.4432$	$0.6147$	$-0.0821$	$9.6320$	$0.5975$
Attention	$4.3448$	$0.2714$	$0.4467$	$\mathbf{0.6261}$	$0.0162$	$\underline{7.4006}$	$\mathbf{0.6337}$
Self-Influence	$4.3519$	$0.2720$	$0.4458$	$0.6228$	$0.0037$	$7.5461$	$0.6257$

Appendix

Table 10: Results on EPIC-KITCHENS-100 under the matched causal-stream protocol. H64 gain is paired against Reservoir. Best mean values are bold and second-best mean values are underlined .

Method	Candidates	Calls	Wall time (ms) $\downarrow$	$\times$ Res.	$\Delta$ GPU (MB) $\downarrow$	Peak GPU (MB) $\downarrow$
FIFO	0	0	$1.686\pm 0.003$	1.67	0.78	925.68
Reservoir	0	0	$1.006\pm 0.006$	1.00	0.65	925.55
Attention	2	2	$9.866\pm 1.337$	9.89	4.28	931.89
Self-Influence	18	18	$61.698\pm 0.068$	63.05	4.63	932.24
Utility-only	17	1	$8.950\pm 0.042$	9.15	828.82	1756.43
State-only	17	1	$5.367\pm 0.188$	5.47	12.63	940.24

Appendix

Table 11: Deployment-time cost on Assembly101 with

K=16

. Wall time is the participant-balanced mean

\pm

sample standard deviation per full-memory update. “Candidates” counts candidate-equivalent evaluations, whereas “calls” counts predictor forward passes after batching.

\times

Res. is computed relative to Reservoir within each matched participant and seed before aggregation; it therefore need not equal the ratio of the displayed mean wall times.

\Delta

GPU denotes the peak memory allocated during the update above the loaded pre-update state.

Estimator ordering
Metric	Observed	Reference
Pairwise sign accuracy $\uparrow$	$61.98\%$	$51.29\%$ majority
Hit@1 $\uparrow$	$13.53\%$	$5.88\%$ random
Hit@3 $\uparrow$	$28.29\%$	$17.65\%$ random
Executed interventions
Positive-gain overrides $\uparrow$	$58.85\%$	$50.00\%$ parity

Appendix

Table 12: Retrospective utility validation on Assembly101. Estimator metrics use 960 full-memory states with 17 candidate actions, and intervention metrics use 209 SPACE overrides with complete oracle targets. Hit@1 and Hit@3 are compared with uniform ranking. Combined gain is reported in train-standardized cost units.

Method	Audit Action NLL $\downarrow$	H16 gain ( $\times 10^{-4}$ ) $\uparrow$	H64 gain ( $\times 10^{-4}$ ) $\uparrow$	Audit drift gain ( $\times 10^{-6}$ ) $\uparrow$	Boundary selectivity $\uparrow$
Reservoir	$\underline{3.5763\pm 0.1073}$	$0.0000\pm 0.0000$	$0.0000\pm 0.0000$	$0.0000\pm 0.0000$	$\underline{0.8568\pm 0.0864}$
Utility-only	$3.5768\pm 0.1082$	$\underline{2.5895\pm 10.7770}$	$5.0917\pm 10.9680$	$2.0749\pm 1.1209$	$0.8564\pm 0.0868$
State-only	$3.5777\pm 0.1081$	$-0.1472\pm 9.1888$	$\underline{5.2427\pm 11.9840}$	$\mathbf{22.2150\pm 14.2020}$	$0.8563\pm 0.0881$
SPACE	$\mathbf{3.5761\pm 0.1070}$	$\mathbf{4.6237\pm 4.6086}$	$\mathbf{6.0321\pm 4.5149}$	$\underline{3.0900\pm 4.5634}$	$\mathbf{0.8569\pm 0.0872}$

Appendix

Table 13: Evaluation with an independently trained audit predictor on Assembly101. Values are participant-balanced means

\pm

sample standard deviations across the ten participant-level means after averaging policy seeds. Gains are paired with Reservoir on the same recording. Best values are bold , and second-best values are underlined .

Evaluation space	Mean drift gain $\uparrow$	95% CI	Positive
Slow	$0.4958$	$[0.2818,\,0.7513]$	$5/5$
Pre-slow	$3.6922$	$[2.4701,\,5.5429]$	$5/5$
Raw JEPA output	$7.8954$	$[5.5802,\,11.5720]$	$5/5$

Appendix

Table 14: SPACE drift gains beyond the slow predictive space. Gain is defined as

D_{\mathrm{within}}^{\mathrm{Reservoir}}-D_{\mathrm{within}}^{\mathrm{SPACE}}

; positive values indicate lower within-regime drift. Means and 95% participant-level confidence intervals are reported in units of

10^{-6}

. Positive counts indicate participants with positive paired gains.

Memory-Augmented Video Understanding

Memory-augmented video understanding helps models retain and retrieve useful visual history for reasoning over long videos and live streams.

See all 85 papers

MemEvo: Automatic Discovery of Streaming Video Memory Mechanisms
Sep 29, 2026Guohong Liu, Jialei Ye, Shanhui Zhao +2Memory-Augmented VLMs Language Model-Guided Search Heuristics
Institute for AI Industry Research, Tsinghua University · Peking University
StreamEMS: Streaming Video Understanding with Self-Evolving Memory Scheme for Vision-Language Models
Aug 28, 2026Yuxin Liu, Peiqin Zhuang, Yali WangStreaming Video Understanding Memory-Augmented Video Understanding
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China · Shanghai AI Laboratory, Shanghai, China
Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding
Jun 24, 2026Baiyang Song, Yuli Lin, Qiong Wu +5Streaming Video Understanding Multimodal Large Language Models
1Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China

Sep 29, 2026J/K move · Enter open · S save

MemEvo: Automatic Discovery of Streaming Video Memory Mechanisms

Guohong Liu, Jialei Ye, Shanhui Zhao, Yunxin Liu+1

Institute for AI Industry Research, Tsinghua University · Peking University

Memory-Augmented VLMs Language Model-Guided Search Heuristics Streaming Video Understanding Program Synthesis Memory-Augmented Video Understanding Streaming Inference

Open paper AskPDF arXiv

Query-agnostic streaming video understanding requires vision-language models to continuously compress an indefinitely growing visual stream into a bounded memory before future queries are known. The performance depends critically on the memory mechanism--what observations to preserve, how to represent and consolidate them, and what information to retrieve when a query eventually arrives. Rather than designing a single memory architecture by hand, we formulate memory design as a search problem over executable memory programs. We introduce a lightweight domain-specific language that expresses memory mechanisms through structured primitives for representation, admission, retention, consolidation, budgeting, and retrieval, while enforcing causal and bounded-memory constraints. Although structured, the derived program space remains large and contains heterogeneous, conditionally dependent design choices whose effects can only be assessed via downstream execution. We therefore propose MemEvo, an LLM-driven auto-research framework that uses pretrained LLM as a semantics-aware proposal model to iteratively generate and refine candidate memory programs based on accumulated experimental feedback. At runtime, a deterministic evaluation pipeline validates and evaluates each candidate, while the underlying vision-language model remains frozen throughout discovery. We finally produce a training-free, bounded-memory mechanism. Extensive experiments on StreamingBench and OVO-Bench demonstrate strong streaming video understanding performance together with substantial context and inference efficiency.

Abstract

Memory-Augmented Video Understanding

Explore similar work

MemEvo: Automatic Discovery of Streaming Video Memory Mechanisms

StreamEMS: Streaming Video Understanding with Self-Evolving Memory Scheme for Vision-Language Models

Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding