CurveStream: Boosting Streaming Video Understanding in MLLMs via Curvature-Aware Hierarchical Visual Memory Management

Chao Wang; Xudong Tan; Jianjian Cao; Kangcong Li; Tao Chen

cs.CVMar 20, 2026

CurveStream: Boosting Streaming Video Understanding in MLLMs via Curvature-Aware Hierarchical Visual Memory Management

Authors: Chao Wang, Xudong Tan, Jianjian Cao, Kangcong Li, Tao Chen

Organizations: College of Future Information Technology, Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China

Abstract

Multimodal Large Language Models have achieved significant success in offline video understanding, yet their application to streaming videos is severely limited by the linear explosion of visual tokens, which often leads to Out-of-Memory (OOM) errors or catastrophic forgetting. Existing visual retention and memory management methods typically rely on uniform sampling, low-level physical metrics, or passive cache eviction. However, these strategies often lack intrinsic semantic awareness, potentially disrupting contextual coherence and blurring transient yet critical semantic transitions. To address these limitations, we propose CurveStream, a training-free, curvature-aware hierarchical visual memory management framework. Our approach is motivated by the key observation that high-curvature regions along continuous feature trajectories closely align with critical global semantic transitions. Based on this geometric insight, CurveStream evaluates real-time semantic intensity via a Curvature Score and integrates an online K-Sigma dynamic threshold to adaptively route frames into clear and fuzzy memory states under a strict token budget. Evaluations across diverse temporal scales confirm that this lightweight framework, CurveStream, consistently yields absolute performance gains of over 10% (e.g., 10.69% on StreamingBench and 13.58% on OVOBench) over respective baselines, establishing new state-of-the-art results for streaming video perception.The code will be released at https://github.com/streamingvideos/CurveStream.

Figures & tables

Fig. 1: Performance and mechanism of CurveStream. (a) CurveStream achieves state-of-the-art on OVOBench among training-free paradigms, boosting performance by 13.6% over the Qwen2.5-VL-7B baseline. (b) Curvature-aware memory management over infinite streams ( $t\rightarrow\infty$ ). By evaluating real-time semantic intensity (blue curve) against a K-Sigma dynamic threshold (pink dashed line), it adaptively filters redundant Low-Semantic Frames. Critical High-Semantic Frames (yellow dots) at curvature peaks are preserved, ensuring optimal visual context retention under strict token limits.

Fig. 2: Overview of the CurveStream framework. This training-free vision encoder enables infinite streaming video understanding by replacing traditional sampling with a dynamic-retention perception layer designed to prevent Out-of-Memory (OOM) errors in long-term sequences. The Curvature-Aware Scorer (CAS) evaluates semantic transition intensity by fusing first-order motion variation and second-order trajectory curvature within the latent feature manifold, while the Hierarchical Visual Memory Management (HVMM) module dynamically routes incoming tokens into a fixed-capacity ( $N_{\max}$ ) queue. By utilizing temporally adaptive $K$ -Sigma thresholds, the encoder adaptively categorizes visual information into Clear, Blurred, or Discard states based on the intensity of semantic shifts, thereby ensuring a constant memory footprint while preserving critical visual anchors for long-term multimodal reasoning.

Proprietary MLLMs
Method	Frame	StreamingBench [ 18 ]	OVOBench [ 19 ]
Human	-	91.46	93.20
Gemini 1.5 Pro [ 41 ]	1fps	75.69	69.32
GPT-4o [ 42 ]	64	73.28	64.46
Open-source Offline MLLMs
Qwen2-VL-7B [ 2 ]	64	69.04	60.65

TABLE I: Quantitative comparison of average accuracy across 10 Real-Time Visual Understanding sub-tasks in StreamingBench and 6 Real-Time Visual Perception sub-tasks in OVOBench. Best results are highlighted in bold, with absolute gains over respective baselines in red. Our frame count (10-20) reflects the dynamically changing size of the adaptive memory queue.

Proprietary MLLMs
Method	Frame	MVBench [ 20 ]	EgoSchema [ 21 ]	VideoMME [ 22 ]
GPT4-V [ 47 ]	1fps	43.7	55.6	60.7
GPT-4o [ 42 ]	64	64.6	72.2	77.2
Open-source Offline MLLMs
LLaVA-NeXT-Video [ 5 ]	32	33.7	43.9	46.5
Qwen2-VL-7B [ 2 ]	64	67.0	66.70	69.0

TABLE II: Quantitative comparison of the average accuracy on MVBench (20 sub-tasks), EgoSchema, and VideoMME benchmarks. The best results are highlighted in bold, with absolute performance gains over the respective baselines indicated in red.

Panel of Figure 5 — Fig. 3: Scalability and memory allocation analysis. (a) CurveStream consistently delivers significant performance gains across varying model capacities (4B, 8B, 32B) of the Qwen3-VL series. (b) Impact of the clear memory (High-Res) retention ratio on overall accuracy and token cost. An adaptive $\sim$ 50% ratio achieves the optimal trade-off between semantic integrity and computational overhead.

Fig. 4: Ablation on K-Sigma dual thresholds. CurveStream exhibits strong hyperparameter robustness across various

k_{1}

and

k_{2}

configurations on OVOBench. The dynamic mechanism effectively balances memory allocation between High-Res and Low-Res frames, ensuring an optimal accuracy-efficiency trade-off without tedious tuning.

Table 7

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Fig. 5: Action Recognition in dynamic virtual environments. Fast-paced viewpoint shifts often cause baseline models to lose track of transient actions, resulting in severe hallucinations (e.g., misinterpreting the action as setting up a camera). CurveStream captures the sharp curvature peak during the “drinking” animation, preserving it as a key semantic node to deliver an accurate response.

Panel of Figure 9 — Fig. 7: Attribute Recognition requiring fine-grained spatial details. Standard memory limits often force base models to downsample past frames uniformly, blurring complex textures. CurveStream dynamically assigns high-resolution Clear Memory to informative frames where the pot’s pattern is unobscured, allowing it to correctly identify the nested diamond shapes.

Proprietary MLLMs
Model	Frame	OP	CR	CS	ATP	EU	TR	PR	SU	ACP	CT	Avg.
Human	-	89.47	92.00	93.60	91.47	95.65	92.52	88.00	88.75	89.74	91.30	91.46
Gemini 1.5 Pro [ 41 ]	1 fps	79.02	80.47	83.54	79.67	80.00	84.74	77.78	64.23	71.95	48.70	75.69
GPT-4o [ 42 ]	64	77.11	80.47	83.91	76.47	70.19	83.80	66.67	62.19	69.12	49.22	73.28
Claude 3.5 Sonnet [ 49 ]	20	73.33	80.47	84.09	82.02	75.39	79.53	61.11	61.79	69.32	43.09	72.44
Open-source Offline MLLMs

Appendix

TABLE V: Comprehensive evaluation results on the StreamingBench benchmark for real-time visual understanding. Our method consistently improves the performance of base MLLMs across all contextual reasoning and event understanding metrics, as indicated by the “

\uparrow

” symbol. The highest scores in each column are marked in bold .

Proprietary MLLMs
Model	Frame	OCR	ACR	ATR	STU	FPD	OJR	Avg.
Human	-	93.96	92.57	94.83	92.70	91.09	94.02	93.20
Gemini 1.5 Pro [ 41 ]	1 fps	85.91	66.97	79.31	58.43	63.37	61.96	69.32
GPT-4o [ 42 ]	64	69.80	64.22	71.55	51.12	70.30	59.78	64.46
Open-source Offline MLLMs
LLaVA-Video-7B [ 23 ]	64	69.80	59.63	66.38	50.56	72.28	61.41	63.34

Appendix

TABLE VI: Detailed performance comparison on the OVOBench dataset across various real-time visual perception sub-tasks. We report the evaluation metric ( e.g. , Accuracy %) for both the standard base models and our proposed CurveStream. The “

\uparrow

” denotes the absolute performance gain achieved by integrating our curvature-aware memory management into the respective base models. Best results are highlighted in bold .

Proprietary MLLMs
Model	Frame	AS	HAC	SAD	MAD	CM	NSM	Avg.
Gemini-1.5-Pro [ 41 ]	1 fps*	49.22	53.73	48.80	54.85	41.58	56.25	49.87
GPT-4o [ 42 ]	1 fps*	40.65	45.10	42.84	45.48	36.00	48.44	42.09
Claude-3.7-Sonnet [ 50 ]	1 fps*	45.20	43.02	41.82	48.05	39.07	46.88	43.73
Open-source MLLMs
Video-LLaVA-7B [ 51 ]	8 frms	24.91	21.54	25.45	30.54	26.23	21.88	25.37

Appendix

TABLE VII: Detailed performance comparison on the FavorBench dataset. ”

\uparrow

” indicates the performance improvement of our method compared to the base model.

Model	Avg.	Action Antonym	Action Count	Episodic Reasoning	Action Localization	Action Prediction	Action Sequence	Character Order	Counterfactual Inference	Egocentric Navigation
Qwen3-VL-8B	60.17	84.00	37.50	51.50	34.50	57.49	65.95	61.50	65.50	38.00
+ Ours (CurveStream)	63.60 ( $\uparrow$ 3.43 )	68.50	50.50	54.00	39.50	79.00	70.50	77.50	60.50	37.50

Appendix

TABLE VIII: Detailed performance comparison on the MVBench dataset across 19 fine-grained sub-tasks. Due to space constraints, the results are split into two blocks. ”

\uparrow

” indicates the performance improvement of our method.

Model Configuration	CAS	HVMM	OP	CR	CS	ATP	EU	TR	PR	SU	ACP	CT	Avg.
Qwen2-VL-7B [ 2 ]			77.38	76.56	73.19	75.08	75.00	67.91	73.15	65.04	66.57	35.75	69.04
w/ CAS	✓		85.29	80.47	89.59	87.58	74.53	82.24	76.85	70.73	73.94	43.52	78.16 ( $\uparrow$ 9.12)
w/ HVMM		✓	87.19	76.56	89.59	86.93	75.16	86.92	76.85	70.33	74.50	43.01	78.80 ( $\uparrow$ 9.76)
CurveStream	✓	✓	88.56	77.34	88.61	89.84	76.25	92.52	76.85	76.83	76.70	45.31	81.04 ( $\uparrow$ 12.00)

Appendix

TABLE IX: Ablation study on StreamingBench . We evaluate the individual and combined effects of CAS and HVMM. Red arrows specifically denote the absolute average performance improvements achieved over the respective base models.

Model Configuration	CAS	HVMM	OCR	ACR	ATR	STU	FPD	OJR	Avg.
Qwen-3VL-8B [ 1 ]			71.14	65.14	75.86	64.61	75.25	70.65	70.10
w/ CAS	✓		87.92	83.49	81.90	64.04	76.24	80.98	78.49 ( $\uparrow$ 8.39)
w/ HVMM		✓	87.25	71.56	78.45	66.29	74.26	72.83	74.79 ( $\uparrow$ 4.69)
CurveStream	✓	✓	93.96	82.57	83.62	68.54	78.22	80.43	80.76 ( $\uparrow$ 10.66)

Appendix

TABLE X: Ablation study on OVO-Bench . We report the performance across various real-time visual perception sub-tasks to validate the synergistic effect between the proposed memory modules.

Hyperparameter	Value
Queue Size ( $N_{max}$ )	20
Curvature score weight ( $\lambda$ )	0.2
TRANSITION_SIZE	224
K_SIGMA_TRANS ( $k_{1}$ )	0.0
K_SIGMA_KEY ( $k_{2}$ )	1.0

Appendix

TABLE XI: Detailed core experimental hyperparameters utilized by the CurveStream framework throughout the entire inference phase.

Streaming Video Understanding

Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.

See all 132 papers

Aug 11, 2026

StreamFlow: Dynamic Memory Flows for Streaming Video Understanding

Streaming video understanding requires multimodal large language models (MLLMs) to preserve relevant evidence from continuously evolving streams under strict causality and bounded memory. Yet existing paradigms remain limited: model-based methods require intrusive backbone updates, while memory-based methods expend substantial visual-encoding computation on temporally redundant content and rely on rigid access to visual history. To address these limitations, we introduce StreamFlow, an efficient visual memory framework that enables dynamic, on-demand access to historical visual information. StreamFlow combines a lightweight, dynamics-aware mid-term memory that filters temporal redundancy before visual encoding with a latent long-term memory that consolidates historical video content into visual latents accessible to subsequent reasoning. During generation, an attention-guided retrieval mechanism injects relevant visual latents when the model's reliance on visual evidence weakens. StreamFlow achieves state-of-the-art streaming video understanding performance, reaching 67.73% overall accuracy on StreamingBench, while also delivering strong performance on offline long-video benchmarks. Relative to the vanilla setting, it improves the visual attention score (VAS) by 59.1% while reducing end-to-end latency and peak memory by 50.4% and 21.1%, respectively, enabling more visually grounded and efficient reasoning.

Muxin Fu, Yifan Zhang, Wentao Zhang +5

Tongji University · Nanyang Technological University · University of Michigan +2

Memory-Augmented VLMs Streaming Video Understanding Efficient VLM Inference Memory-Augmented Video Understanding

Ask PDF

Feb 2, 2026

FreshMem: Brain-Inspired Frequency-Space Hybrid Memory for Streaming Video Understanding

Transitioning Multimodal Large Language Models (MLLMs) from offline to online streaming video understanding is essential for continuous perception. However, existing methods lack flexible adaptivity, leading to irreversible detail loss and context fragmentation. To resolve this, we propose FreshMem, a Frequency-Space Hybrid Memory network inspired by the brain's logarithmic perception and memory consolidation. FreshMem reconciles short-term fidelity with long-term coherence through two synergistic modules: Multi-scale Frequency Memory (MFM), which projects overflowing frames into representative frequency coefficients, complemented by residual details to reconstruct a global historical "gist"; and Space Thumbnail Memory (STM), which discretizes the continuous stream into episodic clusters by employing an adaptive compression strategy to distill them into high-density space thumbnails. Extensive experiments show that FreshMem significantly boosts the Qwen2-VL baseline, yielding gains of 5.20%, 4.52%, and 2.34% on StreamingBench, OV-Bench, and OVO-Bench, respectively. As a training-free solution, FreshMem outperforms several fully fine-tuned methods, offering a highly efficient paradigm for long-horizon streaming video understanding.

Kangcong Li, Peng Ye, Lin Zhang +4

College of Future Information Technology, Fudan University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · The Chinese University of Hong Kong, Hong Kong, China +2

Streaming Video Understanding Memory-Augmented Video Understanding Multimodal Large Language Models

Ask PDF

May 8, 2026

Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding

Online streaming video understanding requires models to process continuous visual inputs and respond to user queries in real time, where the unbounded stream and unpredictable query timing turn memory management into a central challenge. Existing methods typically compress visual tokens via visual similarity heuristics, or augment compression with KV-cache-level retrieval. However, compression decisions rarely incorporate semantic signals, and retrieval is often added after compression is finalized, making the two stages hard to coordinate. We present SAVEMem, a training-free dual-stage framework that brings semantic awareness into memory generation and lets the retrieval scope adapt per query. In Stage1, SAVEMem builds a three-tier streaming memory online under a constant memory budget. A fixed pseudo-question bank provides a lightweight semantic prior, so that long-term retention is shaped by semantic salience rather than visual similarity alone. In Stage2, SAVEMem performs query-aware retrieval over this memory. An anchor-conditioned recency gate adapts the retrieval scope from short-term to mid- and long-term memory based on whether the query targets the present or the distant past. Within this scope, late interaction between query and memory tokens selects candidate frames for answering. Applied to Qwen2.5-VL without training, SAVEMem improves the OVO-Bench overall score from 52.27 to 62.69 and yields consistent gains on StreamingBench and ODV-Bench, while reducing peak GPU memory by 48% at 128 frames over the backbone.

Hang Wu, Sherin Mary Mathews, Yujun Cai +2

University of California, Merced · US Bank · University of Queensland

Streaming Video Understanding Efficient VLM Inference Video QA Memory-Augmented Video Understanding Visual Token Compression

Ask PDF

Abstract

Streaming Video Understanding

Explore similar work

StreamFlow: Dynamic Memory Flows for Streaming Video Understanding

FreshMem: Brain-Inspired Frequency-Space Hybrid Memory for Streaming Video Understanding

Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding