Video-MemComp: Taylor-Guided Sensitivity-Aware Memory Management for Streaming Video Understanding

Yonghui Niu; Yuhao Ji; Chang Nie; Jiao Zhang; Yilong Li; Junlan Feng; Ran He; Caifeng Shan

cs.CVOct 23, 2026

Video-MemComp: Taylor-Guided Sensitivity-Aware Memory Management for Streaming Video Understanding

Authors: Yonghui Niu, Yuhao Ji, Chang Nie, Jiao Zhang, Yilong Li, Junlan Feng, Ran He, Caifeng Shan,

Organizations: NJU · Jiutian AI · CASIA

Abstract

The rapid growth of video data has created an urgent need for efficient video understanding systems that can handle long and continuous visual inputs, posing significant challenges for Multimodal Large Language Models (MLLMs) due to severe memory bottlenecks. Moreover, existing compression methods often overlook the explosive accumulation of approximation errors. To address this issue, we propose Video-MemComp, a training-free and sensitivity-aware context management framework for memory-constrained streaming video understanding. Our method theoretically decomposes compression error into orthogonal local approximation error and layer-wise structural sensitivity via a first-order Taylor expansion. To minimize local approximation error, we perform token aggregation in a pre-RoPE safe zone with an attention-conserving logit bias, mitigating geometric distortions caused by RoPE rotations. To manage layer-wise structural sensitivity, we dynamically allocate the KV cache budget based on offline gradient priors and online context entropy, prioritizing high-sensitivity layers. Video-MemComp outperforms or achieves performance comparable to full-cache and state-of-the-art compression methods on benchmarks including MLVU, VideoMME, and EgoSchema. Notably, it improves the overall accuracy of Qwen2.5-VL-3B from 62.3% to 64.1% while reducing the KV cache to an equivalent of 6,000 tokens per layer, saving over 90% of KV memory.

Figures & tables

Figure 1: Our architecture . While standard Video-LLM models suffer from linear $O(T)$ memory growth, Video-MemComp maintains strictly $O(1)$ memory via a Taylor-expansion-guided hybrid strategy, enabling efficient streaming video understanding. Specifically, we decompose the error via Taylor expansion into local approximation error $\mathcal{E}_{l}$ and layer-wise structural sensitivity $\mathcal{S}_{l}$ . To minimize $\mathcal{E}_{l}$ , we adopt Attention-Preserved Pre-RoPE Aggregation and Intra-layer Sparsity-based KV pruning. Meanwhile, $\mathcal{S}_{l}$ is used to guide layer-wise budget allocation.

Figure 2: Performance on MLVU with Qwen2-VL-7B. Our method outperforms strong baselines across all memory budgets, and even surpasses the Full KV baseline at 6K, 12K, and 24K KV budgets, showing that Video-MemComp removes redundancy while preserving key semantics. See Section E.1 in the supplementary material for details.

Figure 3: Overview of our Video-MemComp Framework. Guided by Taylor expansion (top), we split global memory compression into two objectives. 1) Minimizing Local Approximation Error $\mathcal{E}_{l}$ (middle): Pre-RoPE Aggregation merges redundant tokens based on inter-frame filtering and intra-frame merging. Intra-layer KV Pruning selects key-value pairs using a dialogue-aware Proxy Query $\mathcal{Q}_{\text{proxy}}$ . 2) Managing Structural Sensitivity $\mathcal{S}_{l}$ (bottom): A Tri-Hybrid Budget Allocation dynamically assigns the total memory budget $B_{\text{total}}$ using offline structural sensitivity, online information density, and uniform regularization to protect critical layers from aggressive pruning.

Figure 4: Statistical Visualization on Qwen2.5-VL-3B. Layer-wise budget allocation (stacked bars) is shown under an average budget of 500 KV tokens per layer. Gradient Norm (

S_{l}

), Entropy (

\mathcal{H}

), and Feature Magnitude (

\overline{\|V\|}

) are normalized for comparison. The strategy effectively assigns higher budgets to shallow layers, while more aggressively pruning deeper layers. See Appendix E.3 for details.

Method	Size	# Frames	Avg Budget	EgoSchema	MLVU	VideoMME			ALL
Max Duration			$\|M\|$	3 min	120 min	Medium	Long	Avg	–
GPT4-V	–	1fps	–	55.6	–	–	–	60.7	–
GPT4-o	–	1fps	–	72.2	66.2	–	–	77.2	71.9
LongVU	7B	1fps	8K	67.6	65.4	–	–	60.6	64.5
LongVU	3B	1fps	8K	59.1	55.9	–	–	51.5	55.5
Qwen-2-VL*	7B	768	100K	65.2	65.8	63.9	55.1	63.7	64.9

Table 1: Performance Comparison on Offline Video Understanding Benchmarks. We evaluate Video-MemComp against state-of-the-art compression methods and full-cache baselines. Avg Budget indicates the KV cache token limit; bold and underlined values mark the best and second-best results, respectively. Entries marked with * denotes reproduced results, others are from original papers. Rows in light blue show that our method meets or exceeds full-cache performance under strict memory constraints.

	RVS-Ego		RVS-Movie		Avg
Method	Acc (%)	Score	Acc (%)	Score	Acc (%)	Score
ReKV	63.7	4.0	54.4	3.6	59.0	3.8
ReKV w/o off.	55.8	3.3	50.8	3.4	53.3	3.4
Flash-VStream	57.0	4.0	53.1	3.3	55.0	3.6
InfiniPot-V	57.9	3.5	51.4	3.5	54.6	3.5
StreamMem	57.6	3.8	52.7	3.4	55.2	3.6

Table 2: Quantitative Results on RVS Streaming Benchmarks. We compare our method with state-of-the-art streaming baselines, grouped into Reference (Top) using CPU offloading for infinite history, and Memory-Constrained (Bottom) within GPU limits. Video-MemComp achieves the highest average accuracy among memory-constrained methods, especially in the egocentric domain.

Method	Peak Memory	Video Enc.	TTFT	Throughput
	(GB) $\downarrow$	(ms/frame) $\downarrow$	(s) $\downarrow$	(tokens/s) $\uparrow$
LLaVA-OV (Baseline)	OOM	220.48	0.0600	6.45
ReKV	22.95	82.75	0.0846	35.61
ReKV w/o off.	22.95	90.49	0.0663	36.93
Video-MemComp (Ours)	16.74	64.92	0.0302	27.58

Table 3: System efficiency on LLaVA-OneVision-7B. Metrics include peak memory, visual encoder latency per frame, and time to first token (TTFT). Video-MemComp reduces memory and latency compared to retrieval-based baselines (ReKV), enabling OOM-free consumer-grade inference.

Figure 5: Accuracy-Memory trade-offs on Qwen2.5-VL-3B (VideoMME Long Video). Token Compression (Blue) excels in high-budget regimes, notably surpassing the dense baseline by mitigating attention dilution. Conversely, KV Compression (Orange) demonstrates robustness under strict memory constraints. Our Hybrid Method (Green) achieves the optimal Pareto frontier across all budgets, balancing redundancy removal with attention preservation. See E.4 in the supplementary material for details.

	VideoMME
Method	S	M	L	Avg	MLVU
Inter-frame Only	70.7	58.6	52.3	60.5	67.68
Inter + Intra-frame	70.3	59.3	52.8	60.8	68.19
$\Delta$	-0.4	+0.7	+0.5	+0.3	+0.51

Table 4: Ablation study on intra-frame token compression. The inter-frame only baseline applies temporal filtering (

\theta_{\text{inter}}=0.3

) with standard KV pruning. Adding intra-frame compression improves performance on longer contexts by densifying spatial semantics, with a minor trade-off on short clips.

Threshold	Strategy	Short	Medium	Long	All
0.75	Static	70.0	60.3	53.2	61.17
0.75	Dynamic	70.5	60.0	53.2	61.23
0.90	Static	70.0	60.7	52.3	61.00
0.90	Dynamic	71.0	59.9	53.0	61.53

Table 5: Ablation study on static vs. dynamic budget allocation on VideoMME. Our sensitivity-aware dynamic strategy consistently improves aggregate performance across similarity thresholds (

\tau=0.75,0.90

), validating the benefit of prioritizing structurally sensitive and high-entropy layers over a uniform distribution.

Figure 6: Hyperparameter landscape of budget allocation. OVO-Bench accuracy is shown as a contour over uniform prior ( $\lambda_{u}$ ) and structural sensitivity ( $\lambda_{s}$ ); $\lambda_{e}=1-\lambda_{u}-\lambda_{s}$ represents the dynamic entropy weight. The global optimum(star), revealing that a dominant structural prior combined with subtle dynamic adjustment yields superior performance compared to uniform allocation. See F.3 in the supplementary material for details.

Figure 7: Joint Impact of Spatio-Temporal Compression Thresholds. The heatmap plots accuracy across varying Inter-frame ( $\theta_{\text{inter}}$ ) and Intra-frame ( $\theta_{\text{intra}}$ ) similarity thresholds. The landscape reveals a convex optimization zone, where the peak represents the optimal trade-off between aggressive redundancy filtering and semantic preservation. Notably, overly aggressive temporal pruning (low $\theta_{\text{inter}}$ ) causes irreversible performance degradation. See F.4 in the supplementary material for details.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

KV Budget	Inter-Frame ( $\tau_{\text{inter}}$ )	Intra-Frame ( $\tau_{\text{intra}}$ )
3000 (3K)	0.15	0.45
6000 (6K)	0.20	0.55
12000 (12K)	0.25	0.55
24000 (24K)	0.30	0.60

Appendix

Table 6: Hyperparameter settings for different KV budgets. To achieve optimal performance under stricter budgets, we adopt slightly more aggressive filtering thresholds.

Method	KV Budget				Avg.
Method	3K	6K	12K	24K	Avg.
Baselines
Uniform	55.59	57.00	60.36	64.18	59.28
FastV Chen et al. (2024b)	55.01	57.60	-	-	-
SnapKV Li et al. (2024b)	58.99	60.66	61.31	64.37	61.33
InfiniPot Kim et al. (2024)	58.36	59.11	60.70	63.71	60.47

Appendix

Table 7: Performance comparison on MLVU (Qwen2-VL-7B) across different KV budgets. We report the average accuracy (%). Full KV represents the performance of the original model without compression. Bold indicates the best performance under compression.

KV Budget	Token Comp.	KV Cache Comp.	Video-MemComp (Ours) (Two-Stage)	InfiniPot-V	StreamMem
500	45.56	45.89	47.78	47.00	44.00
3000	49.33	49.44	50.33	48.00	44.60
6000	51.22	50.78	53.22	48.00	49.10
12000	52.89	51.11	53.44	49.67	50.00

Appendix

Table 8: Comparison of Compression Strategy Performance on VideoMME. We report model accuracy (%) under different KV budgets. The baseline performance of the original model (Qwen2.5-VL-3B without compression) is 50.56%.

Layer	Entropy ( $H$ )	Log Feature Norm	Layer	Entropy ( $H$ )	Log Feature Norm
0	0.701	1.44	18	$4.08\times 10^{-27}$	2.14
1	1.104	0.97	19	$1.48\times 10^{-24}$	2.09
2	0.766	1.04	20	$5.45\times 10^{-23}$	2.17
3	$2.82\times 10^{-31}$	1.80	21	$1.32\times 10^{-21}$	2.19
4	$1.49\times 10^{-35}$	1.65	22	$1.07\times 10^{-20}$	2.19
5	$3.94\times 10^{-36}$	1.73	23	$3.62\times 10^{-28}$	2.09

Appendix

Table 9: Distribution of Dynamic Metrics for Qwen2.5-VL-3B during Online Inference. Data is sampled from VideoMME streams.

H

denotes the Shannon entropy of the attention probability distribution, and

\log\|V\|

is the log-norm of the value vectors. Layers 3-28 exhibit extremely low entropy values, constituting the primary compression space.

Layer	Mean ( $S_{l}$ )	Std	RSD (%)	Layer	Mean ( $S_{l}$ )	Std	RSD (%)
0	2.2740e-04	7.0700e-05	31.09	18	1.2093e-04	2.7035e-05	22.36
1	1.6284e-04	5.2962e-05	32.52	19	1.1845e-04	2.5720e-05	21.71
2	1.5621e-04	5.1101e-05	32.71	20	1.1420e-04	2.4370e-05	21.34
3	1.5206e-04	4.9749e-05	32.72	21	1.1110e-04	2.2826e-05	20.54
4	1.4615e-04	4.7366e-05	32.41	22	1.0948e-04	2.1954e-05	20.05
5	1.4100e-04	4.4595e-05	31.63	23	1.0638e-04	2.0570e-05	19.34

Appendix

Table 10: Statistical Analysis of Structural Sensitivity (

S_{l}

) for Qwen2.5-VL-3B Bai et al. (2025b) . Data is calculated based on LLaVA-Pretrain Liu et al. (2023) . A higher Mean value indicates the layer is more sensitive to pruning errors. RSD (Relative Standard Deviation) reflects the fluctuation of importance across different samples.

Model	Benchmarks	$\tau_{\text{inter}}$	$\tau_{\text{intra}}$	Weights ( $\lambda_{u},\lambda_{s},\lambda_{e}$ )	Configuration Notes
Qwen2-VL-7B	VideoMME, MLVU	0.20	0.55	0.8, 0.166, 0.034	Balanced setting for standard duration.
Qwen2-VL-7B	EgoSchema	0.30	0.75	0.8, 0.166, 0.034	Higher thresholds adopted for shorter duration.
Qwen2.5-VL-3B	VideoMME, MLVU	0.30	0.75	0.8, 0.166, 0.034	Standard configuration.
Qwen2.5-VL-3B	EgoSchema	0.50	0.75	0.8, 0.166, 0.034	Aggressive inter-frame filtering for redundancy.
LLaVA-OV-7B	RVS, EgoSchema	0.75	0.80	0.4, 0.575, 0.025	YaRN scaling enabled (factor=5.5).
	VideoMME, MLVU	Dynamic Adjustment		0.4, 0.575, 0.025	Duration-Adaptive Thresholds ( $T$ ):

Appendix

Table 11: Detailed hyperparameter configurations for different models across benchmarks.

\tau_{\text{inter}}

and

\tau_{\text{intra}}

denote the thresholds for inter-frame filtering and intra-frame merging, respectively. The budget allocation weights are denoted by

\lambda_{u}

(uniform),

\lambda_{s}

(static/structure), and

\lambda_{e}

(dynamic/entropy).

Model	Frames	AVG KV Size	Real-Time Visual Perception							Backward Tracing				Forward Active Responding				Overall
Model	Frames		OCR	ACR	ATR	STU	FPD	OJR	Avg.	EPM	ASI	HLD	Avg.	REC	SSR	CRR	Avg.	Avg.
	Methods w/o Compression
LLaVA-NeXT-7B Zhang et al. (2024b)	64	$\approx$ 10K	69.8	59.6	66.4	50.6	72.3	61.4	63.3	51.2	64.2	9.7	41.7	34.1	67.6	60.8	54.2	53.1
LLaVA-OneVision-7B Li et al. (2024a)	64	$\approx$ 10K	67.1	58.7	69.8	49.4	71.3	60.3	62.8	52.5	58.8	23.7	45.0	24.8	66.9	60.8	50.9	52.9
Qwen2-VL-7B Wang et al. (2024a)	64	$\approx$ 10K	69.1	53.2	63.8	50.6	66.3	60.9	60.7	44.4	66.9	34.4	48.6	30.1	65.7	50.8	48.9	52.7
InternVL-V2-8B Chen et al. (2024c)	64	$\approx$ 10K	68.5	58.7	69.0	44.9	67.3	56.0	60.7	43.1	61.5	27.4	44.0	25.8	57.6	52.9	45.4	50.1

Appendix

Table 12: Evaluation results on OVO-Bench. The table compares offline models and online streaming models separately. Bold indicates the best performance and underline indicates the second best within each group.

Hyperparams			Backward Tracing				Real-time Perception							Forward Active				Total
$\lambda_{u}$	$\lambda_{s}$	$\lambda_{e}$	EPM	HLD	ASI	Avg	STU	OJR	ATR	FPD	ACR	OCR	Avg	REC	SSR	CRR	Avg	Avg
0.00	0.00	1.00	48.82	34.95	62.16	48.64	49.44	55.98	62.93	66.34	55.96	69.13	59.96	37.48	42.44	42.08	40.67	49.76
0.00	1.00	0.00	49.16	31.72	62.16	47.68	45.51	58.15	63.79	66.34	54.13	71.81	59.95	36.31	41.85	40.83	39.66	49.10
0.05	0.90	0.05	48.82	31.72	62.84	47.79	47.75	57.61	59.48	63.37	55.96	71.14	59.22	36.89	42.03	39.17	39.36	48.79
0.10	0.90	0.00	48.82	31.72	62.16	47.57	47.19	61.41	65.52	69.31	55.05	71.81	61.71	36.21	36.99	43.75	38.99	49.42
0.10	0.85	0.05	48.82	30.11	63.51	47.48	48.88	59.24	61.21	71.29	51.38	72.48	60.74	36.35	37.27	44.58	39.40	49.21

Appendix

Table 13: Comprehensive Ablation Study on Budget Allocation Hyperparameters. We report the accuracy (%) for all sub-tasks on OVO-Bench. The configurations are sorted by the Uniform Prior weight (

\lambda_{u}

). The extreme cases (pure strategies) are located at the top and bottom. The new global optimal configuration is highlighted in bold with a gray background.

Thresholds		Backward Tracing				Real-time Perception							Forward Active				Total
$\tau_{\text{inter}}$	$\tau_{\text{intra}}$	EPM	HLD	ASI	Avg	STU	OJR	ATR	FPD	ACR	OCR	Avg	REC	SSR	CRR	Avg	Avg
0.25	0.75	43.77	31.72	58.78	44.76	43.26	54.89	62.07	68.32	55.96	65.10	58.27	31.09	41.65	45.42	39.38	47.47
0.30	0.60	45.12	29.57	57.43	44.04	42.13	53.26	62.93	68.32	51.38	65.77	57.30	29.39	41.49	54.58	41.82	47.72
0.30	0.75	46.46	29.57	58.11	44.71	43.82	54.89	62.07	67.33	55.05	67.11	58.38	36.73	40.97	44.17	40.62	47.91
0.30	0.95	45.45	31.18	60.14	45.59	42.70	53.26	59.48	66.34	54.13	67.79	57.28	33.07	41.99	40.00	38.35	47.08
0.50	0.75	46.13	29.03	60.81	45.32	46.63	54.89	58.62	70.30	57.80	63.09	58.55	33.34	42.74	47.66	41.25	48.37

Appendix

Table 14: Ablation Study of Token Compression Thresholds on OVO-Bench. We report accuracy (%) for Backward, Real-time, and Forward tasks. The best configuration is highlighted in bold and gray background.

Streaming Video Understanding

Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.

See all 132 papers

OmniMem: Perturbation-aware Memory Compression for Streaming Audio-Visual LLMs
May 26, 2026Guangzhi Sun, Yixuan Li, Yudong Yang +1Streaming Video Understanding Multimodal Large Language Models
ByteDance · Department of Engineering, University of Cambridge · Tsinghua University
FluxMem: Adaptive Hierarchical Memory for Streaming Video Understanding
Mar 2, 2026Yiweng Xie, Bo He, Junke Wang +3Streaming Video Understanding Memory-Augmented Video Understanding
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Innovation Institute · Shanghai Key Laboratory of Multimodal Embodied AI +1
An Efficient Streaming Video Understanding Framework with Agentic Control
May 18, 2026Jinming Liu, Jianguo Huang, Zhaoyang Jia +7Visual Token Compression Adaptive Model Routing
1Shanghai Jiao Tong University · 2Eastern Institute of Technology, Ningbo, China · 3Microsoft Research Asia

May 26, 2026J/K move · Enter open · S save

OmniMem: Perturbation-aware Memory Compression for Streaming Audio-Visual LLMs

Guangzhi Sun, Yixuan Li, Yudong Yang, Chao Zhang

ByteDance · Department of Engineering, University of Cambridge · Tsinghua University

Streaming Video Understanding Multimodal Large Language Models Multimodal Token Compression KV-Cache Compression Video Token Compression

Open paper AskPDF arXiv

Audio-visual large language models (LLMs) hold strong promise for long-form video understanding, yet their long-video inference is fundamentally limited by the linear growth of video tokens and key-value (KV) caches. We present OmniMem, a memory-efficient streaming framework designed specifically for audio-visual LLMs. Unlike existing compression methods that treat all tokens uniformly, OmniMem introduces a modality-aware memory allocation strategy that separately manages visual and audio contexts, addressing the severe token imbalance between the two modalities. OmniMem further preserves informative and non-redundant KV states through perturbation-aware memory selection, enabling compact memory without sacrificing long-range understanding. To strengthen compression under realistic deployment constraints, we also explore budget-aware fine-tuning, which encourages the model to consolidate useful information into retained memory. Experiments on VideoMME Long, LVBench, and LVOmniBench with video-SALMONN 2+ and Qwen-2.5-Omni show that OmniMem consistently improves over strong training-free compression baselines by 2-4% absolute accuracy under the same memory budgets, with an additional 1-2% gain after fine-tuning.

Abstract

Streaming Video Understanding

Explore similar work

OmniMem: Perturbation-aware Memory Compression for Streaming Audio-Visual LLMs

FluxMem: Adaptive Hierarchical Memory for Streaming Video Understanding

An Efficient Streaming Video Understanding Framework with Agentic Control