Multimodal Large Language Models have demonstrated remarkable capabilities in video understanding, yet face prohibitive computational costs and performance degradation from ''context rot'' due to massive visual token redundancy. Existing compression strategies typically rely on heuristics or fixed transformations that are often decoupled from the downstream task objectives, limiting their adaptability and effectiveness. To address this, we propose SCORE (Surprise-augmented token COmpression via REinforcement learning), a unified framework that learns an adaptive token compression policy. SCORE introduces a lightweight policy network conditioned on a surprise-augmented state representation that incorporates inter-frame residuals to explicitly capture temporal dynamics and motion saliency. We optimize this policy using a group-wise reinforcement learning scheme with a split-advantage estimator, stabilized by a two-stage curriculum transferring from static pseudo-videos to real dynamic videos. Extensive experiments on diverse video understanding benchmarks demonstrate that SCORE significantly outperforms state-of-the-art baselines. Notably, SCORE achieves a 16x prefill speedup while preserving 99.5% of original performance at a 10% retention ratio, offering a scalable solution for efficient long-form video understanding.
Figures & tables
Figure 1: (a) Visual token redundancy induces context rot in video MLLMs. (b) A comparision of existing token compression methods and SCORE.
Figure 2: SCORE pipeline. A lightweight visual compressor is inserted between the frozen vision encoder and the frozen LLM. During training, we optimize a surprise-augmented Bernoulli gating policy with group rollouts and an accuracy–sparsity reward. During inference, we use deterministic global top- K selection to meet a target budget.
Figure 3: Asymmetric advantage for group rollouts. We separate rollouts into a safe zone ( ΔCE>0 ) and a penalty zone ( ΔCE≤0 ). Successful rollouts are ranked by sparsity via normalized advantages, while violating rollouts receive sparsity-weighted penalties to recover performance.
Figure 4: Pseudo-video samples for curriculum warm-up. We construct short clips by repeating each sampled image for several frames and concatenate their captions into a single training target. This synthesis produces near-zero inter-frame residuals within repeated segments and sharp changes at image boundaries, offering high-contrast temporal signals for learning redundancy-aware token pruning.
VideoMME
Method
Retention Ratio R
Max Frames
LVBench
MLVU
Overall
Short
Medium
Long
Avg Score
Acc. (%)
Duration
1 ∼ 120min
3 ∼ 120min
1 ∼ 60min
1 ∼ 3min
3 ∼ 30min
30 ∼ 60min
Vanilla
100%
128
38.3
69.7
63.8
74.4
64.2
52.8
57.3
100.0
DyCoke (CVPR’25)
40%
128
39.8
70.0
64.9
74.3
65.3
55.0
58.2
101.7
HoliTom (NeurIPS’25)
40%
128
39.1
69.9
61.6
73.1
59.9
51.8
56.9
99.3
VidCom 2 (EMNLP’25)
40%
128
40.2
70.6
64.7
74.8
64.7
54.6
58.5
102.1
Table 1: Main results on Qwen2.5-VL [ 2 ] across three video understanding benchmarks. We compare SCORE with the Vanilla baseline and state-of-the-art token reduction methods at varying retention ratios ( R ). The best results in each block are marked in bold . Rows highlighted in green denote our method.
Method
Retention Ratio R
Max Frames
Short
Medium
Long
Overall
Acc. (%)
Vanilla
100%
64
73.4
58.9
49.9
60.7
100
FastVID
40%
64
72.7
59.2
48.4
60.1
99.0
SCORE
40%
64
73.2
58.0
49.7
60.3
99.3
FastVID
25%
64
71.9
58.1
47.8
59.3
97.7
SCORE
25%
64
72.6
57.8
48.8
59.7
98.4
FastVID
10%
64
68.3
55.1
45.8
56.4
92.9
Table 2: Generalization analysis on Video-MME [ 6 ] with the LLaVA-Video [ 40 ] architecture.
Method
# Tokens (Visual / Total)
Prefill Time (ms)
Acc.
Compressor
LLM Fwd.
Total
Vanilla
42566 / 42657
–
8539±2231
8539±2231 (1.0 × )
63.8 (100.0%)
SCORE ( R =40%)
17117 / 17208
18.6±7.5
2441±583
2473±587 (3.5 × )
65.0 (101.9%)
SCORE ( R =25%)
10733 / 10824
18.6±8.9
1386±318
1417±326 (6.0 × )
65.3 (102.4%)
SCORE ( R =10%)
4348 / 4439
18.5±8.5
504±107
527±110 (16.2 × )
63.6 (99.7%)
Table 3: Efficiency Comparison on Qwen2.5VL-7B [ 2 ] with VideoMME [ 6 ] . We report the exact number of visual tokens, total tokens (including system/user text), and prefill latency.
Method
Training Data
R
LVBench
MLVU
VideoMME
Avg
Vanilla
-
100%
38.3
69.7
63.8
57.3
SCORE
Pseudo
40%
40.5
70.1
64.9
58.5
Pseudo + Real
40.6
70.6
65.0
58.7
SCORE
Pseudo
25%
40.4
69.1
65.2
58.2
Pseudo + Real
41.2
70.1
65.3
58.9
SCORE
Pseudo
10%
38.3
65.7
63.2
55.7
Table 4: Ablation study on the training data composition across different retention ratios R . We compare the performance of the model trained solely on Pseudo-Videos versus the model further trained on Real Videos . The Vanilla baseline ( R=100% ) is provided for reference.
Method
R
Short
Medium
Long
Overall
Vanilla
100%
74.4
64.2
52.8
63.8
w/o Residual
40%
74.4
65.3
54.3
64.7
SCORE (w/ Residual)
75.0
65.0
55.1
65.0
w/o Residual
25%
74.8
64.9
54.8
64.5
SCORE (w/ Residual)
74.8
65.4
55.7
65.3
w/o Residual
10%
71.4
62.0
52.6
62.0
Table 5: Ablation study on the impact of the Surprise-Augmented State Encoding . We compare the performance on VideoMME [ 6 ] (Short, Medium, Long) and the Overall score across different retention ratios.
Figure 5: Qualitative visualization of SCORE token masks. Top two rows: six pseudo-video inputs constructed from static images, where SCORE concentrates kept tokens on salient objects and action-related regions while pruning static backgrounds. Bottom two rows: two real-video examples, showing temporally adaptive masks that shift as motion and scene content evolve.