AdaSpark: Adaptive Sparsity for Efficient Long-Video Understanding
Authors: Handong Li, Zikang Liu, Longteng Guo, Tongtian Yue, Yepeng Tang, Xinxin Zhu, Chuanyang Zheng, Ziming Wang, Zhibin Wang, Jun Song, Cheng Yu, Bo Zheng, Jing Liu
Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Alibaba Group Holding Limited · Future Living Lab of Alibaba
Processing long-form videos with Video Large Language Models (Video-LLMs) is computationally prohibitive. Current efficiency methods often compromise fine-grained perception through irreversible information disposal or inhibit long-range temporal modeling via rigid, predefined sparse patterns. This paper introduces AdaSpark, an adaptive sparsity framework designed to address these limitations. AdaSpark first partitions video inputs into 3D spatio-temporal cubes. It then employs two co-designed, context-aware components: (1) Adaptive Cube-Selective Attention (AdaS-Attn), which adaptively selects a subset of relevant video cubes to attend for each query token, and (2) Adaptive Token-Selective FFN (AdaS-FFN), which selectively processes only the most salient tokens within each cube. An entropy-based (Top-p) selection mechanism adaptively allocates computational resources based on input complexity. Experiments demonstrate that AdaSpark significantly reduces computational load by up to 57% FLOPs while maintaining comparable performance to dense models and preserving fine-grained, long-range dependencies, as validated on challenging hour-scale video benchmarks.
Figures & tables
Figure 1: Preliminary analysis. We analyzed internal distributions within the video-LLM layers. The upper figure shows text-to-visual attention score distributions, marking the 0.7 cumulative probability point per layer with a star. The lower figure displays L2 norm changes across modalities after the FFN, quantified as the post-to-pre norm ratio.
Figure 2: Framework illustration of AdaSpark. We process long-duration videos at their native resolution and subsequently apply video cube partitioning. Within the AdaS-Attn layer, each token query performs adaptive selection based on relevance scores computed over preceding Cubes. Upon entering the AdaS-FFN, visual tokens within each Cube are adaptively selected to pass through the FFN, while the transformations for the remaining tokens are estimated via Mean Compensation. Text tokens pass densely through the AdaS-FFN.
Model
Extra Long
Long-form
Short-form
Spatial
Grounding
VideoNIAH
MLVU Dev
VideoMME
LongVideo
LVBench
MVBench
VsiBench
CharadesSTA
Small Size Models
LongVU-3B [ 33 ]
-
55.9
51.5
-
-
60.9
-
-
InternVL2.5-2B [ 8 ]
-
61.4
51.9
52.0
-
68.8
-
-
VideoChat-Flash-2B [ 22 ]
92.0
65.7
57.0
58.3
42.9
70.0
-
45.2
Qwen-2.5-VL-3B [ 3 ]
86.5
65.3
61.5
54.2
43.3
65.7
32.4
42.6
Table 1: Performance on zero-shot video-language benchmarks. We evaluate AdaSpark on 1 extra long video benchmarks, 4 long-form video benchmarks, 1 short video benchmark, 1 spatial perception video benchmark and 1 video grounding benchmark. The results in bold and underline values indicate the best and second-best performance among encoder-free models, respectively.
Figure 3: Video Needle in A Haystack Results. We compare AdaSpark against existing high-efficiency models and methods.
Module
TFLOPs ↓
MLVU
VideoMME
LongVideo
LVBench
Charades STA
AdaS-Attn
AdaS-FFN
M-avg
w/o sub
acc
acc
mIoU
✗
✗
299.5
65.2
60.3
53.6
43.1
43.6
✗
✓
216.9 (-28%)
63.9
62.8
55.6
44.1
41.7
✓
✗
213.1 (-29%)
66.0
61.2
54.5
43.4
43.7
✓
✓
128.5 (-57%)
65.4
62.1
55.3
44.3
44.2
Table 2: Effects of AdaS-Attn and AdaS-FFN .
Model
Factors
Charades STA
VideoMME
MLVU
Cube Shape
(a)
1×1×64
35.3
52.7
51.3
(b)
4×4×4
35.5
54.0
52.1
(c)
8×8×1
34.5
54.2
49.4
Cube Size
(d)
64
35.5
54.0
52.1
Table 3: Ablation on design choices of AdaSpark .
Figure 4: Analysis of adaptive selection within AdaSpark. The left figure illustrates the number of cubes selected by each query per layer in AdaS-Attn. The middle figure details the average token keep ratio per cube for each layer in AdaS-FFN. The right figure demonstrates the impact of parameter choices on the dynamic selection mechanism.
Figure 5: Illustration of adaptive selection in a case study. AdaSpark adaptively selects visual cubes that exhibit high relevance to the posed query token.
Selection Strategy
Charades STA
VideoMME
MLVU
Uniform Sampling
31.5
50.2
53.8
Top-K
34.2
52.6
55.1
I-Frame
34.0
52.9
55.7
Top-P
36.1
55.7
57.5
Table 4: Ablation on more selection strategy.
Hyperparameter
Value
Model Configuration
Backbone
Qwen2.5-VL-3B / 7B
Visual Encoder Status
Frozen
Max Visual Sequence Length
48k
Max Context Length
64k
Cube Size ( H×W×T )
8×8×4
Table 5: Hyperparameter configuration for AdaSpark training. The model undergoes a single-stage post-training protocol.
Benchmark
FPS
# F (Max Frames)
VideoNIAH
1
4096
MLVU (Dev)
1
512
VideoMME
1
256
LongVideo
1
1024
LVBench
1
1024
MVBench
2
256
Table 6: Evaluation settings summary for each benchmark. For all benchmarks, we set the temperature, top p , and number of beams to 0, 0, and 1, respectively. FPS denotes the sampling frames per second, and # F represents the maximum number of sampling frames allowed.