Token pruning is essential for enhancing the computational efficiency of vision-language models (VLMs), particularly for video-based tasks where temporal redundancy is prevalent. Prior approaches typically prune tokens either (1) within the vision transformer (ViT) exclusively for unimodal perception tasks such as action recognition and object segmentation, without adapting to downstream vision-language tasks; or (2) only within the LLM while leaving the ViT output intact, often requiring complex text-conditioned token selection mechanisms. In this paper, we introduce Spatio-Temporal Token Scoring (STTS), a simple and lightweight module that prunes vision tokens across both the ViT and the LLM without text conditioning or token merging, and is fully compatible with end-to-end training. By learning how to score temporally via an auxiliary loss and spatially via LLM downstream gradients, aided by our efficient packing algorithm, STTS prunes 50% of vision tokens throughout the entire architecture, resulting in a 62% improvement in efficiency during both training and inference with only a 0.7% drop in average performance across 13 short and long video QA tasks. Efficiency gains increase with more sampled frames per video. Applying test-time scaling for long-video QA further yields performance gains of 0.5-1% compared to the baseline. Overall, STTS represents a novel, simple yet effective technique for unified, architecture-wide vision token pruning.
Figures & tables
Figure 1: (Left) Token pruning with our STTS (purple box) vs. a cosine-similarity-based heuristic. STTS learns that background patches are less important, while the heuristic prunes all tokens equally. (Right) QA performance under increasing vision token pruning ratios ( k% ). STTS (pink squares) consistently demonstrates a flatter, more robust degradation curve compared to the Random baseline (blue circles) across all metrics.
Figure 2: Overall workflow of using STTS within the VLM. Numbered vision tokens here are 3x3 grids. After ViT layer l , STTS prunes vision tokens permanently from the entire architecture. We pad tokens during packing for ViT batch computation.
Figure 3: Architectural and procedural overview of STTS. We use 9x9 tokens per frame for illustration. Vision features after ViT layer l are first downsampled via pooling then scored. The scores are injected as attention bias for layer l+1 before the pruning algorithm is applied to allow for spatial pruning. The scores are also aligned with neighboring-frame per-patch cosine similarity for temporal pruning.
Figure 4: Visualization of the packing algorithm. (a) Before pruning, the scoring mechanism identifies the bottom- k% importance tokens (in dotted squares) to be removed. (b) To reduce tensor sparsity, the remaining tokens from Frame 2 (green) and Frame 4 (red) are consolidated into a single packed batch entry. Because Frame 1 is always untouched and Frame 3 retains high token counts, they remain independent.
Model
NextQA test [ 45 ]
Perception-Test test [ 30 ]
MVBench test [ 19 ]
Tomato test [ 32 ]
MotionBench val [ 13 ]
Temp-Compass test [ 22 ]
VideoMME test [ 11 ]
VideoMME-Sub test [ 11 ]
LongVideo val [ 42 ]
LongVideo-Sub val [ 42 ]
MLVU val MCQ [ 49 ]
LVBench test [ 40 ]
VideoEvalPro test [ 25 ]
Short avg.
Long avg.
Average
Qwen3-VL-4B [ 46 ]
81.4
70.7
68.9
31.8
58.6
70.8
69.3
74.0
62.8
-
58.4
56.2
49.8
63.7
61.8
62.7
PLM-8B [ 6 ]
84.1
82.7
77.1
33.2
61.4
72.7
58.3
65.4
56.9
-
52.6
44.5
47.2
68.5
54.2
61.3
InternVL3.5-8B [ 41 ]
81.7
72.7
72.1
24.6
56.6
70.3
66.0
68.6
62.1
-
53.2
43.4
48.1
63.0
56.9
60.0
Baseline (ours)
83.9
78.7
72.6
36.5
61.0
69.9
62.8
67.6
61.5
60.9
70.3
42.0
47.6
67.0
59.0
63.0
+ STTS 30%
84.1
79.0
72.7
35.6
59.2
69.6
63.4
68.5
61.1
59.2
69.5
42.6
47.7
66.7
58.9
62.8
+ STTS 40%
83.6
77.3
71.8
34.6
59.2
69.3
62.4
67.4
61.4
60.2
67.5
41.1
47.2
66.0
58.2
62.1
Table 1: Effect of pruning strength on video benchmarks. + STTS k% means k% pruning. Values that improve upon or remain within 0.5 points of the baseline are bolded and values within 1.0 point are underlined .
Figure 5: Comparison of efficiency gains during training and inference across different pruning ratios ( k ) . As k increases, speedups greatly increase and become significantly larger when sampling more frames. See Supp. Sec. B for more details.
Figure 7
k%
TTS
# Fr
VideoMME test [ 11 ]
VideoMME-Sub test [ 11 ]
LongVideo val [ 42 ]
LongVideo-Sub val [ 42 ]
MLVU val MCQ [ 49 ]
LVBench test [ 40 ]
VideoEvalPro test [ 25 ]
Long avg.
0%
✗
64
62.8
67.6
61.5
60.9
70.3
42.0
47.6
59.0
30%
✗
64
63.4
68.5
61.1
59.2
69.5
42.6
47.7
58.9
✓
92
62.9
69.1
62.7
60.8
70.5
44.9
49.6
60.1
40%
✗
64
62.4
67.4
61.4
60.2
67.5
41.1
47.2
58.2
✓
107
62.0
68.1
62.7
60.6
68.6
42.5
48.6
59.0
50%
✗
64
62.4
67.2
61.0
60.1
68.4
40.5
46.0
58.4
Table 3: Impact of Test-Time Scaling (TTS) on Long Video Benchmarks. Performance comparison when increasing number of frames sampled (# Fr) only during inference . Significant improvements over the baseline (0%) are bolded .
Figure 7: Visualizations of STTS (purple box) vs. the heuristic. STTS attempts to keep important information despite temporal redundancy, while the heuristic prunes them regardless due to the lack of knowledge.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
k%
AI2D test [ 17 ]
ChartQA test [ 26 ]
DocVQA test [ 28 ]
InfoQA test [ 27 ]
TextVQA val [ 35 ]
VQA v2.0 val [ 12 ]
RWQA [ 44 ]
MMMU val [ 48 ]
MathVista testmini [ 23 ]
CountBench [ 2 ]
PixMoCount test [ 9 ]
MuirBench [ 39 ]
MMIU [ 29 ]
Img avg.
MultiImg avg.
Average
0%
95.0
84.1
91.9
77.2
85.5
86.0
74.6
49.6
57.4
94.3
91.1
61.2
54.2
80.6
57.7
77.1
50%
94.3
83.9
91.3
77.3
85.7
86.2
75.4
50.0
57.4
94.9
89.6
62.0
54.9
80.5
58.4
77.3
Appendix
Table 4: Image benchmark results between a different version of Molmo2 trained with slightly different data and our STTS variant. In this attempt, both models are trained on the exact same full mixture with both video and image data. We can see that even though our model applies pruning on videos, the overall image performance did not degrade.
Method
Short avg.
Long avg.
Average
Heuristic [Inference Only]
63.1
55.1
59.1
ToMe [Inference Only]
62.4
55.9
59.2
ToMe
65.6
56.6
61.1
STTS
66.1
58.4
62.3
Appendix
Table 5: Performance comparison at a 50% pruning rate on video QA tasks. STTS significantly outperforms inference-only baselines (Spatial Heuristic, ToMe) and a fully trained ToMe model.
k%
# Fr
Toks/Inst.
Throughput
Speedup
0%
128
15670
0.1932
1x
30%
12560
0.2478
1.28x
40%
11524
0.2786
1.44x
50%
10486
0.3130
1.62x
0%
256
25307
0.0549
1x
30%
19087
0.0811
1.48x
Appendix
Table 6: Comparison of training speed between baseline and different pruning setup k ’s. The number of tokens per instance decreases as k increases, while throughput (batches per second) and speedup increases. Efficiency gains grew larger as we increase max number of frames sampled.
k%
# Fr
Throughput
Speedup
0%
128
1.0186
1x
30%
1.1651
1.14x
40%
1.3270
1.30x
50%
1.6439
1.61x
0%
256
0.2641
1x
30%
0.3836
1.45x
Appendix
Table 7: Comparison of inference speed between baseline and different pruning setup k ’s on MLVU. We observe identical trends during training.
k
Method
Short avg.
Long avg.
Average
50
Random
65.3
57.5
61.4
STTS
66.1
58.4
62.3
60
Random
63.9
56.3
60.1
STTS
65.3
57.9
61.6
70
Random
63.2
55.0
59.1
STTS
65.0
56.4
60.7
Appendix
Table 8: Comparing numerical performance values between Random and STTS from k=50 to k=90 . STTS consistently outperforms Random. Text-only baseline k=100 provided as lower bound.
University of Science and Technology of China He Fei, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center He Fei, China