Memory as Weights: Internalizing Long-Term History for Streaming Videos

Zhuoming Liu; Jinhong Lin; Lin Zhang; Yiwu Zhong; Yin Li

cs.CVOct 15, 2026

Memory as Weights: Internalizing Long-Term History for Streaming Videos

Authors: Zhuoming Liu, Jinhong Lin, Lin Zhang, Yiwu Zhong, Yin Li

Organizations: University of Wisconsin–Madison · Peking University

Abstract

Streaming vision-language models (VLMs) must continuously integrate information from the past while remaining responsive to current observations and user queries. Existing memory mechanisms typically compress, retrieve, or cache historical observations and reintroduce them into the input context, forcing historical memory and current observations to compete for limited context. We propose MaLoW, a streaming memory framework that instead internalizes historical context into evolving lightweight parameter updates. MaLoW uses a recurrent HyperNetwork to incrementally consolidate incoming video observations into fixed-size LoRA weights, allowing historical information to be carried forward without revisiting the full video stream. In doing so, MaLoW represents history as weights, while keeping the current observation as input. To support training, we further construct MaLoW-Instruct, a large-scale streaming video dataset containing 381K question-answer pairs over 56K videos. We conduct extensive experiments to demonstrate that MaLoW achieves strong performance on OVO-Bench, StreamingBench, and StreamArena. On the backward split of OVO-Bench, MaLoW outperforms prior memory approaches by 3.9% to 11.8% across different video backbones while incurring only modest latency overhead. Beyond streaming video understanding, MaLoW also generalizes to vision-language-action models, improving the action success rate on Simpler WidowX by 9.3%.

Figures & tables

Figure 1: (1) : MaLoW internalizes the history context into time-evolving, lightweight parameter updates (LoRA) of a streaming VLM. At each step, the streaming VLM takes the current observation and user query as input, and performs inference using the updated LoRA weights. (2) : By separating memory from the input, MaLoW preserves the performance on real-time perception while significantly improves the performance of memory-critical, backward tracing tasks.

Figure 2: Overview of MaLoW . (1) MaLoW internalizes history into LoRA weights applied to a base VLM that processes only the current observation. (2) Context encoding compresses visual tokens $V_{t-1}$ and timestamp tokens $T_{t-1}$ into a compact representation $q_{t-1}$ using a learnable query $Q$ . (3) Hidden state updates $m_{t}$ that captures local dynamics and $s_{t}$ that preserves global semantics. (4) LoRA weights are decoded from the hidden states. (5) $s_{t}$ is updated through gated summation.

Figure 3: Overview of the MaLoW data pipeline and statistics of MaLoW -Instruct . (1) : Our data pipeline consists of a Question Generator, an Answer Generator, and a Verifier. (2) : The distribution of the streaming question timing, the question type, and video source in MaLoW -Instruct.

Model	#Frames	Streaming Bench (R)	OVO-Bench
			Real-Time Visual Perception							Backward Tracing				Avg.
			OCR	ACR	ATR	STU	FPD	OJR	Avg.	EPM	ASI	HLD	Avg.	Avg.
Offline VLMs
LLaVA-Video-7B ( Zhang et al., 2025b )	64	–	69.1	58.7	68.8	49.4	74.3	59.8	63.5	56.2	57.4	7.5	40.4	52.0
LongVU-7B ( Shen et al., 2025 )	1 fps	–	55.7	49.5	59.5	48.3	68.3	63.0	57.4	43.1	66.2	9.1	39.5	48.5
Qwen3-VL-4B ( Bai et al., 2025 )	1 fps	78.4	85.2	60.6	74.1	51.1	76.2	67.9	69.2	57.9	61.5	16.7	45.4	57.3

Table 1: Main results in streaming video understanding. *VideoChat3 uses 2 fps on OVO-Bench and 4 fps on StreamingBench. Gray numbers denote results reported in the original paper that differ from our re-evaluation using the official implementation and reported evaluation hyperparameters. Please refer to Baselines Re-evaluation in sec C.1 for more information. MaLoW (MOSS-VL-Real-11B) and MaLoW (Qwen3-VL-4B) achieve a new state-of-the-art score of 74.5 and 84.8 on OVO-Bench and StreamingBench, respectively, while MaLoW (Qwen3.5-9B) attains the highest EPM score of 63.6 among all compared methods.

Method	OVO-Bench ( Niu et al., 2025 )				StreamingBench ( Lin et al., 2026 )
Method	Back	Real	Forw	Avg.	Real	Omni	Pro	SQA	Avg.
Base model with 1 fps, full-context baseline
Gemma4-E4B ( Gemma Team, 2026 )	56.8	63.1	37.9	52.6	68.6	37.0	24.4	50.0	54.6
Qwen3.5-0.8B	34.2	60.8	48.0	47.7	69.1	32.1	22.4	39.6	52.5
Qwen3.5-2B	44.6	62.5	46.3	51.2	75.1	38.1	6.4	42.8	57.2
Qwen3.5-4B	60.1	66.9	51.2	59.4	80.4	40.7	54.8	46.0	63.8

Table 4: Results across base model scales and families . Green: comparison to full-context baselines on real-time perception tasks; Blue: comparison to local-window baselines on memory-intensive tasks.

Figure 5: Visualization of results. Top row (1-3): results from StreamArena (HR) split. Bottom row (4-6): results from OVO-Bench (EPM) task. We show results from MaLoW (Qwen3.5-9B), VideoChat3-4B, and MOSS-VL-11B.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Underlying source	TimeLens	LLaVA-Video	ET-Instruct	Others	Unique Wideos	Weight
CosMo-Cap ( Wang et al., 2024a )	9,524	–	–	–	9,524	17.05%
YouCook2 ( Zhou et al., 2018 )	–	7,451	–	1,704	9,155	16.39%
Charades ( Gao et al., 2017 )	–	8,692	–	–	8,692	15.56%
ActivityNet ( Krishna et al., 2017 )	–	7,543	–	–	7,543	13.50%
InternVid-VTime ( Wang et al., 2024b )	4,644	–	–	–	4,644	8.31%
NextQA ( Xiao et al., 2021 )	–	3,704	–	–	3,704	6.63%

Appendix

Table 5: Distribution of training videos across underlying datasets and data sources.

Table 9

Figure 6: The distribution of questions, answers, and video durations in the dataset.

	Min	Max	Mean
Question tokens	33	263	70.4
Answer tokens	1	129	3.8
Question timestamp (s)	0.07	1,015.8	57.2
Full video length (s)	5.04	1,673.4	89.5

Appendix

Table 8: Statistics of questions, answers, and video durations in the dataset.

Table 12

Table 13

Table 14

Could be paralleled with the base VLMs prefilling and decoding
Component	Latency (ms)	Percentage
Encode new frames and update state $h_{t}$	12.11	7.91%
Generate LoRA tensors $A_{t+1}$ and $B_{t+1}$	3.27	2.14%
Combine LoRAs and generate $\Delta\theta_{t+1}$	5.69	3.72%
Can not be paralleled with the base VLMs prefilling and decoding
Apply LoRA $\Delta\theta_{t}$	1.13	0.74%

Appendix

Table 15: Breakdown of the MaLoW (Qwen3.5-9B) latency. Percentages are relative to the total latency of 152.89 ms.

Figure 7: The V-JEPA-2 surprise score versus the t-SNE map of the LoRA hidden state.

Figure 8: Effect of Memory on Real-Time Perception. Markers on subplot 1 show bin medians. We evaluate the accuracy of MaLoW on real-time questions across varying history lengths. MaLoW maintains stable real-time perception despite accumulating long-term memory.

Figure 9: The additional qualitative results.

Memory-Augmented VLMs

Memory-augmented vision-language models use persistent or retrieved context to improve reasoning across long inputs and interactions.

See all 158 papers

FlashBack: Knowing When to Remember in Streaming Vision-Language Models
Oct 1, 2026Yi Chen, MingMing Yu, Rui-Qi Wang +5Memory-Augmented VLMs Vision-Language Models
Rightly Robotics, A4X
StreamFlow: Dynamic Memory Flows for Streaming Video Understanding
Aug 11, 2026Muxin Fu, Yifan Zhang, Wentao Zhang +5Memory-Augmented VLMs Streaming Video Understanding
Tongji University · Nanyang Technological University · University of Michigan +2
Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding
Sep 3, 2026Hongyu Qu, Guangming Yao, Ling Xing +7Memory-Augmented VLMs Video Understanding
Nanjing University of Science and Technology · Ant Group · National University of Singapore +1

Oct 1, 2026J/K move · Enter open · S save

FlashBack: Knowing When to Remember in Streaming Vision-Language Models

Yi Chen, MingMing Yu, Rui-Qi Wang, Boran Wang+4

Rightly Robotics, A4X

Memory-Augmented VLMs Vision-Language Models Persistent Memory for Language Models Temporal Video Understanding Streaming Inference

Open paper AskPDF arXiv

Streaming vision-language models must process continuously growing video streams under a bounded compute budget, creating a persistent tension between real-time perception and long-term memory. Retrieving historical information provides a natural remedy, yet historical recall is not uniformly beneficial: unnecessary history may introduce irrelevant context into current reasoning and interfere with native real-time perception. Effective streaming memory should therefore address not only what to remember, but also when and how to access it. To this end, we introduce FlashBack, a training-free framework for selective, multi-level memory in streaming vision-language models. Before retrieving history, FlashBack draws on the semantic understanding of the frozen streaming VLM to infer whether a query calls for historical evidence. This assessment determines whether inference remains on the Native trajectory or invokes an isolated Recall trajectory. The Recall trajectory combines recent context with retrieved long-term memory through a query-local Side-KV pathway, preserving local temporal continuity without modifying the persistent Native state. We instantiate FlashBack on StreamingVLM and Mage-VL-4B and evaluate it on OVO-Bench and StreamingBench. The results show improvements on several long-horizon and memory-dependent tasks while largely preserving real-time perception, with performance competitive with strong training-based streaming methods despite requiring no additional training. Our code will be announced later.

Abstract

Memory-Augmented VLMs

Explore similar work

FlashBack: Knowing When to Remember in Streaming Vision-Language Models

StreamFlow: Dynamic Memory Flows for Streaming Video Understanding

Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding