Streaming vision-language models (VLMs) must continuously integrate information from the past while remaining responsive to current observations and user queries. Existing memory mechanisms typically compress, retrieve, or cache historical observations and reintroduce them into the input context, forcing historical memory and current observations to compete for limited context. We propose MaLoW, a streaming memory framework that instead internalizes historical context into evolving lightweight parameter updates. MaLoW uses a recurrent HyperNetwork to incrementally consolidate incoming video observations into fixed-size LoRA weights, allowing historical information to be carried forward without revisiting the full video stream. In doing so, MaLoW represents history as weights, while keeping the current observation as input. To support training, we further construct MaLoW-Instruct, a large-scale streaming video dataset containing 381K question-answer pairs over 56K videos. We conduct extensive experiments to demonstrate that MaLoW achieves strong performance on OVO-Bench, StreamingBench, and StreamArena. On the backward split of OVO-Bench, MaLoW outperforms prior memory approaches by 3.9% to 11.8% across different video backbones while incurring only modest latency overhead. Beyond streaming video understanding, MaLoW also generalizes to vision-language-action models, improving the action success rate on Simpler WidowX by 9.3%.
Figures & tables
Figure 1: (1) : MaLoW internalizes the history context into time-evolving, lightweight parameter updates (LoRA) of a streaming VLM. At each step, the streaming VLM takes the current observation and user query as input, and performs inference using the updated LoRA weights. (2) : By separating memory from the input, MaLoW preserves the performance on real-time perception while significantly improves the performance of memory-critical, backward tracing tasks.
Figure 2: Overview of MaLoW . (1) MaLoW internalizes history into LoRA weights applied to a base VLM that processes only the current observation. (2) Context encoding compresses visual tokens Vt−1 and timestamp tokens Tt−1 into a compact representation qt−1 using a learnable query Q . (3) Hidden state updates mt that captures local dynamics and st that preserves global semantics. (4) LoRA weights are decoded from the hidden states. (5) st is updated through gated summation.
Figure 3: Overview of the MaLoW data pipeline and statistics of MaLoW -Instruct . (1) : Our data pipeline consists of a Question Generator, an Answer Generator, and a Verifier. (2) : The distribution of the streaming question timing, the question type, and video source in MaLoW -Instruct.
Model
#Frames
Streaming Bench (R)
OVO-Bench
Real-Time Visual Perception
Backward Tracing
Avg.
OCR
ACR
ATR
STU
FPD
OJR
Avg.
EPM
ASI
HLD
Avg.
Offline VLMs
LLaVA-Video-7B ( Zhang et al., 2025b )
64
–
69.1
58.7
68.8
49.4
74.3
59.8
63.5
56.2
57.4
7.5
40.4
52.0
LongVU-7B ( Shen et al., 2025 )
1 fps
–
55.7
49.5
59.5
48.3
68.3
63.0
57.4
43.1
66.2
9.1
39.5
48.5
Qwen3-VL-4B ( Bai et al., 2025 )
1 fps
78.4
85.2
60.6
74.1
51.1
76.2
67.9
69.2
57.9
61.5
16.7
45.4
57.3
Table 1: Main results in streaming video understanding. *VideoChat3 uses 2 fps on OVO-Bench and 4 fps on StreamingBench. Gray numbers denote results reported in the original paper that differ from our re-evaluation using the official implementation and reported evaluation hyperparameters. Please refer to Baselines Re-evaluation in sec C.1 for more information. MaLoW (MOSS-VL-Real-11B) and MaLoW (Qwen3-VL-4B) achieve a new state-of-the-art score of 74.5 and 84.8 on OVO-Bench and StreamingBench, respectively, while MaLoW (Qwen3.5-9B) attains the highest EPM score of 63.6 among all compared methods.
Figure 5
Method
OVO-Bench ( Niu et al., 2025 )
StreamingBench ( Lin et al., 2026 )
Back
Real
Forw
Avg.
Real
Omni
Pro
SQA
Avg.
Base model with 1 fps, full-context baseline
Gemma4-E4B ( Gemma Team, 2026 )
56.8
63.1
37.9
52.6
68.6
37.0
24.4
50.0
54.6
Qwen3.5-0.8B
34.2
60.8
48.0
47.7
69.1
32.1
22.4
39.6
52.5
Qwen3.5-2B
44.6
62.5
46.3
51.2
75.1
38.1
6.4
42.8
57.2
Qwen3.5-4B
60.1
66.9
51.2
59.4
80.4
40.7
54.8
46.0
63.8
Table 4: Results across base model scales and families . Green: comparison to full-context baselines on real-time perception tasks; Blue: comparison to local-window baselines on memory-intensive tasks.
Figure 5: Visualization of results. Top row (1-3): results from StreamArena (HR) split. Bottom row (4-6): results from OVO-Bench (EPM) task. We show results from MaLoW (Qwen3.5-9B), VideoChat3-4B, and MOSS-VL-11B.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Underlying source
TimeLens
LLaVA-Video
ET-Instruct
Others
Unique Wideos
Weight
CosMo-Cap ( Wang et al., 2024a )
9,524
–
–
–
9,524
17.05%
YouCook2 ( Zhou et al., 2018 )
–
7,451
–
1,704
9,155
16.39%
Charades ( Gao et al., 2017 )
–
8,692
–
–
8,692
15.56%
ActivityNet ( Krishna et al., 2017 )
–
7,543
–
–
7,543
13.50%
InternVid-VTime ( Wang et al., 2024b )
4,644
–
–
–
4,644
8.31%
NextQA ( Xiao et al., 2021 )
–
3,704
–
–
3,704
6.63%
Appendix
Table 5: Distribution of training videos across underlying datasets and data sources.
Table 9
Figure 6: The distribution of questions, answers, and video durations in the dataset.
Min
Max
Mean
Question tokens
33
263
70.4
Answer tokens
1
129
3.8
Question timestamp (s)
0.07
1,015.8
57.2
Full video length (s)
5.04
1,673.4
89.5
Appendix
Table 8: Statistics of questions, answers, and video durations in the dataset.
Table 12Table 13Table 14
Component
Latency (ms)
Percentage
Could be paralleled with the base VLMs prefilling and decoding
Encode new frames and update state ht
12.11
7.91%
Generate LoRA tensors At+1 and Bt+1
3.27
2.14%
Combine LoRAs and generate Δθt+1
5.69
3.72%
Can not be paralleled with the base VLMs prefilling and decoding
Apply LoRA Δθt
1.13
0.74%
Appendix
Table 15: Breakdown of the MaLoW (Qwen3.5-9B) latency. Percentages are relative to the total latency of 152.89 ms.
Figure 7: The V-JEPA-2 surprise score versus the t-SNE map of the LoRA hidden state.
Figure 8: Effect of Memory on Real-Time Perception. Markers on subplot 1 show bin medians. We evaluate the accuracy of MaLoW on real-time questions across varying history lengths. MaLoW maintains stable real-time perception despite accumulating long-term memory.
Streaming vision-language models must process continuously growing video streams under a bounded compute budget, creating a persistent tension between real-time perception and long-term memory. Retrieving historical information provides a natural remedy, yet historical recall is not uniformly beneficial: unnecessary history may introduce irrelevant context into current reasoning and interfere with native real-time perception. Effective streaming memory should therefore address not only what to remember, but also when and how to access it. To this end, we introduce FlashBack, a training-free framework for selective, multi-level memory in streaming vision-language models. Before retrieving history, FlashBack draws on the semantic understanding of the frozen streaming VLM to infer whether a query calls for historical evidence. This assessment determines whether inference remains on the Native trajectory or invokes an isolated Recall trajectory. The Recall trajectory combines recent context with retrieved long-term memory through a query-local Side-KV pathway, preserving local temporal continuity without modifying the persistent Native state. We instantiate FlashBack on StreamingVLM and Mage-VL-4B and evaluate it on OVO-Bench and StreamingBench. The results show improvements on several long-horizon and memory-dependent tasks while largely preserving real-time perception, with performance competitive with strong training-based streaming methods despite requiring no additional training. Our code will be announced later.
Streaming video understanding requires multimodal large language models (MLLMs) to preserve relevant evidence from continuously evolving streams under strict causality and bounded memory. Yet existing paradigms remain limited: model-based methods require intrusive backbone updates, while memory-based methods expend substantial visual-encoding computation on temporally redundant content and rely on rigid access to visual history. To address these limitations, we introduce StreamFlow, an efficient visual memory framework that enables dynamic, on-demand access to historical visual information. StreamFlow combines a lightweight, dynamics-aware mid-term memory that filters temporal redundancy before visual encoding with a latent long-term memory that consolidates historical video content into visual latents accessible to subsequent reasoning. During generation, an attention-guided retrieval mechanism injects relevant visual latents when the model's reliance on visual evidence weakens. StreamFlow achieves state-of-the-art streaming video understanding performance, reaching 67.73% overall accuracy on StreamingBench, while also delivering strong performance on offline long-video benchmarks. Relative to the vanilla setting, it improves the visual attention score (VAS) by 59.1% while reducing end-to-end latency and peak memory by 50.4% and 21.1%, respectively, enabling more visually grounded and efficient reasoning.
Muxin Fu, Yifan Zhang, Wentao Zhang +5
Tongji University · Nanyang Technological University · University of Michigan +2
Streaming video understanding requires multimodal large language models (MLLMs) to process continuous visual inputs and respond to user queries under strict causality and bounded memory. Existing approaches typically compress historical observations into an external memory bank and retrieve query-relevant evidence as additional visual context. Though effective, this store-and-retrieve paradigm keeps historical evidence as external visual context, preventing it from being internalized into a compact, evolving latent memory that can continuously guide streaming reasoning. To bridge this gap, we introduce LatentStream, a progressive latent working memory framework that shifts streaming memory from store-and-retrieve to retrieve-and-internalize. Specifically, LatentStream comprises three coordinated components. First, Query-agnostic Hierarchical Streaming Memory organizes visual history into short-, mid-, and long-term levels under a fixed memory budget through Jenks-guided adaptive consolidation. Once a query arrives, Hierarchical Latent Memory Evolution equips groups of latent memory tokens with progressively expanding memory receptive fields, enabling them to iteratively retrieve historical evidence from their corresponding scopes and internalize it into a compact, fixed-length latent memory. Finally, Progressive Confidence-guided Latent Memory Optimization constructs a hierarchical progression reward from group-wise predictive entropy and jointly refines the latent memory tokens and retrieved evidence, encouraging increasingly confident streaming reasoning. Extensive experiments demonstrate that LatentStream achieves new state-of-the-art results on existing online and offline video benchmarks.
Hongyu Qu, Guangming Yao, Ling Xing +7
Nanjing University of Science and Technology · Ant Group · National University of Singapore +1