Recently, many streaming video understanding methods have been proposed by constructing an external memory to store historical data for computational reduction. Most methods focus on optimizing the injection procedure of current data (write) and retrieving informative historical data (read) from memory, while overlooking the opportunity to further enhancing the representational capability of memory itself. In this work, we present StreamEMS, a general mechanism for improving streaming video understanding by re-structuring the historical data stored in memory through self-evolving memory scheme, enabling more informative and robust memory representations. Specifically, we first introduce a Semantic Evolution Module to evolve the memory into more information-dense representations by exploiting informative memory entities discovered via progressively shrinking semantic scales from coarse to fine. In addition, we further introduce a Prior-informed Evolution Module to evolve memory into more robust representations by leveraging prior memory distributions to refine the current memory state. We validate the effectiveness of our proposed designs on widely-used streaming video understanding datasets, i.e., OVO-Bench and StreamingBench, and the results showcase that our method performs better than other methods. Moreover, the advantage of our method becomes consistently evident even under high token usage drop rate settings, indicating the effectiveness and robustness of our method in unleashing the potential of the memory itself.
Figures & tables
Figure 1: An overview of our framework . We propose to exploit the memory itself with self-evolving memory scheme. In the Semantic Evolution Module, we perform semantic evolution by using a hierarchy of semantic thresholds to select highly related memory entities and aggregate information from them, making the memory more compact. After that, we employ the Prior-informed Evolution Module to recalibrate the original similarity distribution using the previous memory distribution in the Exponential Moving Average (EMA) manner.
Table 3: Performance comparison with various models on offline video understanding benchmarks. “↓80%” denotes an approximate token reduction rate of 80%.
Method
Accuracy
Our Baseline
48.46
Our Baseline + PEM
50.56
Our Baseline + SEM
51.22
Our Baseline + PEM + SEM
51.41
Table 4: Investigation on the effectiveness of the self-evolving schemes. SEM denotes Semantic Evolution Module, while PEM refers to Prior-informed Evolution Module.
Semantic Threshold θ
Accuracy
0.85
51.17
(0.85, 0.9)
51.19
(0.85, 0.9, 0.96)
51.41
(0.85, 0.9, 0.96, 0.98)
51.38
Table 5: Investigation on the hierarchy of semantic threshold θ . We propose to perform progressive semantic enhancement through the hierarchical selection of θ at different semantic scales. The threshold values are determined by analyzing the similarity distribution and selecting the corresponding values at specific quantile points.
Figure 7Figure 8
Figure 6: The average count of memory entities of our method across different video durations. The metrics are obtained under the same setting as Table 6 .
Item
Method
Video Duration
0 ∼ 10 min
10 ∼ 30 min
30 ∼ 60 min
>= 60 min
Multimodal Token VRAM (MB) (↓)
TCO [ 8 ]
82.7
132.1
133.4
188.8
Ours
27.1
38.3
36.5
57.9
Token Reduction Rate (%) (↑)
TCO [ 8 ]
88.3%
82.0%
81.4%
74.1%
Ours
95.7%
94.8%
94.9%
92.7%
Average FPS (↑)
TCO [ 8 ]
23.0
24.2
27.2
25.5
Table 6: Performance comparison of our method and the baseline (TimeChat-Online, TCO) on MLVU videos of varying durations. All metrics are averaged over 20 randomly sampled videos per duration bin (80 videos total). ↓ indicates lower is better, ↑ indicates higher is better.
Figure 7: Visualization of a successful case . Compared with the baseline model TimeChat-Online, our method makes the correct prediction by focusing on highly semantically related scenarios with the assistance of self-evolving memory schemes. On the one hand, the progressive semantic evolution strategy enables the model to gradually aggregate distinct information from highly similar memory entities. On the other hand, the prior-informed evolution strategy allows the model to recalibrate the original similarity distribution using the previous memory distribution.
Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.
Streaming video understanding requires models to process unbounded visual streams while preserving rich visual semantics across vast temporal horizons, posing a fundamental challenge for memory modeling. Existing approaches primarily focus on increasing memory capacity, either by compressing historical information into fixed-size representations or by extending storage beyond GPU memory. However, these methods largely rely on global or coarse-grained representations, inevitably losing fine-grained visual information. In this work, we argue that streaming video memory should explicitly encode structured and semantically meaningful representations, particularly at the entity level. To this end, we propose MEMO, a novel framework that models streaming video through multi-level, entity-aware structured memory. MEMO performs multi-level perception to jointly capture global semantics, entity dynamics, and spatial structures, partitioning streaming video into semantically coherent chunks. Each chunk is organized into a structured memory, where lightweight global and entity-level representations serve as retrieval indices, while the corresponding high-resolution visual content is retained separately for on-demand access. At inference time, MEMO performs query-specific retrieval over the structured memory and selectively recalls relevant visual evidence for downstream reasoning. Notably, MEMO is training-free and plug-and-play with existing multimodal large language models. Extensive experiments on StreamingBench and OVO-Bench demonstrate that MEMO consistently improves multiple base models and achieves state-of-the-art performance. The code is available at https://github.com/Cecilia384/MEMO .
Yinying Li, Yuqian Fu, Yulin Dai +3
East China Normal University · King Abdullah University of Science and Technology
Query-agnostic streaming video understanding requires vision-language models to continuously compress an indefinitely growing visual stream into a bounded memory before future queries are known. The performance depends critically on the memory mechanism--what observations to preserve, how to represent and consolidate them, and what information to retrieve when a query eventually arrives. Rather than designing a single memory architecture by hand, we formulate memory design as a search problem over executable memory programs. We introduce a lightweight domain-specific language that expresses memory mechanisms through structured primitives for representation, admission, retention, consolidation, budgeting, and retrieval, while enforcing causal and bounded-memory constraints. Although structured, the derived program space remains large and contains heterogeneous, conditionally dependent design choices whose effects can only be assessed via downstream execution. We therefore propose MemEvo, an LLM-driven auto-research framework that uses pretrained LLM as a semantics-aware proposal model to iteratively generate and refine candidate memory programs based on accumulated experimental feedback. At runtime, a deterministic evaluation pipeline validates and evaluates each candidate, while the underlying vision-language model remains frozen throughout discovery. We finally produce a training-free, bounded-memory mechanism. Extensive experiments on StreamingBench and OVO-Bench demonstrate strong streaming video understanding performance together with substantial context and inference efficiency.
Guohong Liu, Jialei Ye, Shanhui Zhao +2
Institute for AI Industry Research, Tsinghua University · Peking University
Streaming video understanding requires multimodal large language models (MLLMs) to process continuous visual inputs and respond to user queries under strict causality and bounded memory. Existing approaches typically compress historical observations into an external memory bank and retrieve query-relevant evidence as additional visual context. Though effective, this store-and-retrieve paradigm keeps historical evidence as external visual context, preventing it from being internalized into a compact, evolving latent memory that can continuously guide streaming reasoning. To bridge this gap, we introduce LatentStream, a progressive latent working memory framework that shifts streaming memory from store-and-retrieve to retrieve-and-internalize. Specifically, LatentStream comprises three coordinated components. First, Query-agnostic Hierarchical Streaming Memory organizes visual history into short-, mid-, and long-term levels under a fixed memory budget through Jenks-guided adaptive consolidation. Once a query arrives, Hierarchical Latent Memory Evolution equips groups of latent memory tokens with progressively expanding memory receptive fields, enabling them to iteratively retrieve historical evidence from their corresponding scopes and internalize it into a compact, fixed-length latent memory. Finally, Progressive Confidence-guided Latent Memory Optimization constructs a hierarchical progression reward from group-wise predictive entropy and jointly refines the latent memory tokens and retrieved evidence, encouraging increasingly confident streaming reasoning. Extensive experiments demonstrate that LatentStream achieves new state-of-the-art results on existing online and offline video benchmarks.
Hongyu Qu, Guangming Yao, Ling Xing +7
Nanjing University of Science and Technology · Ant Group · National University of Singapore +1