Recent advances in large vision-language models (LVLMs) have significantly improved video understanding capabilities, yet processing long videos remains challenging due to the substantial computational cost of extensive visual inputs. Frame selection offers an efficient alternative, but many existing region-based methods rely on fixed or predefined temporal partitions, which may result in suboptimal temporal regions, compromising subsequent budget allocation and frame selection. In this work, we propose Adaptive Partitioning-Based Frame Selection (APS), a training-free framework that adapts temporal partitions to local variations in query-frame relevance. Specifically, APS formulates temporal partitioning as a complexity-regularized fitting problem, jointly optimizing partition boundaries and local models through dynamic programming. The resulting adaptive temporal partitions guide frame budget allocation based on query relevance and temporal extent, enabling informative frame selection under limited visual budgets. Extensive experiments on Video-MME, MLVU, and LongVideoBench demonstrate that APS consistently improves performance across diverse LVLM backbones, achieving the highest average gains over uniform sampling among competing methods. Further analyses validate the effectiveness of adaptive partitioning and demonstrate its generalizability across model scales, similarity encoders, and advanced LVLMs.
Figures & tables
Figure 1: Qualitative comparison of uniform sampling and APS under the same frame budget. (a) Uniform sampling misses critical visual evidence, leading to an incorrect response. (b) APS preserves the relevant evidence through adaptive temporal partitioning, enabling the correct answer. Red boxes highlight the critical visual evidence.
Figure 2: Overview of APS. Given a video and a textual query, APS constructs a temporal relevance sequence, determines adaptive temporal partitions through complexity-regularized dynamic programming, and allocates the frame budget across partitions for keyframe selection.
Model
Method
Frames
Video-MME
MLVU
LongVideoBench
Avg. Δ
Acc.
Δ
Acc.
Δ
Acc.
Δ
Qwen2.5-VL-7B
Uniform
32
59.7
–
60.7
–
58.0
–
–
AKS [CVPR’25]
32
62.0
+2.3
65.2
+4.5
63.2
+5.2
+4.00
DIG [CVPR’26]
32
60.4
+0.7
68.2
+7.5
61.5
+3.5
+3.90
QCA [ECCV’26]
32
62.1
+2.4
66.9
+6.2
61.5
+3.5
+4.03
FOCUS [ICLR’26]
32
60.4
+0.7
68.0
+7.3
61.2
+3.2
+3.73
Table 1: Comparison with training-free frame selection methods on Video-MME, MLVU, and LongVideoBench. Δ is the accuracy gain (percentage points) over uniform sampling, averaged across datasets in the final column. Best and second-best accuracies are shown in bold and underlined , respectively. The highest gains are also highlighted in bold .
Variant
Video-MME
LVB
Equal Partition
61.59
62.23
Constant-Only
61.93
63.05
Linear-Only
62.07
62.83
w/o Complexity Penalty
61.81
62.83
Relevance-only Allocation
61.93
63.50
APS (Full)
62.33
63.80
Table 2: Ablation results on Video-MME and LongVideoBench using Qwen2.5-VL-7B with 32 frames. Equal Partition uses uniform boundaries while preserving the partition count. LVB denotes LongVideoBench.
η
Video-MME
MLVU
LVB
0.25
66.5
75.9
63.5
0.50
66.9
75.2
63.6
0.75
67.0
75.0
63.9
1.00
66.4
74.9
63.6
1.25
66.6
74.7
63.8
Table 3: Sensitivity to the complexity regularization coefficient η using InternVL3-8B with 32 frames.
Figure 3: Parameter sensitivity and frame budget analysis of APS. (a)–(b) examine the sensitivity to budget allocation parameters α and β . (c)–(d) compare APS with uniform sampling under different frame budgets. All experiments use Qwen2.5-VL-7B with BLIP similarity scores.
Figure 4: Qualitative comparison of APS and four ablation variants on Video-MME under a fixed 16-frame budget. Timelines show temporal partitions, allocated budgets, and selected frames. The highlighted region contains the critical evidence at 76s, while hatched regions indicate zero-budget segments. Only APS preserves the critical evidence and answers correctly.
Encoder
Video-MME
LongVideoBench
Acc.
Δ
Acc.
Δ
Uniform
59.74
–
57.97
–
BLIP
62.33
+2.59
63.80
+5.83
CLIP
62.26
+2.52
62.23
+4.26
BLIP-2
61.19
+1.45
63.13
+5.16
Table 4: Effect of query–frame similarity encoders with Qwen2.5-VL-7B and 32 frames. Uniform sampling is shared across encoders; Δ is the gain over uniform.
Scale
Method
Video-MME
MLVU
Avg.
3B
Uniform
57.74
59.43
58.59
APS
59.30
68.77
64.04
7B
Uniform
59.74
60.67
60.21
APS
62.33
69.64
65.99
32B
Uniform
62.78
61.09
61.94
APS
63.30
72.31
67.81
Table 5: Effect of backbone scale with Qwen2.5-VL, 32 frames, and BLIP scores. Avg. is the mean accuracy across Video-MME and MLVU.
Method
Feature
Partition
Allocation
Selection
Total
AKS
13.743
<0.001
–
<0.001
13.744
QCA
14.280
<0.001
0.002
0.009
14.291
APS
13.743
0.045
<0.001
<0.001
13.789
Table 6: Average runtime per sample (s) on LongVideoBench at a budget of 32 frames, decomposed by processing stage.
Model
Duration
Uniform
APS
Δ
GLM-5.3-Flash
0–10 min
68.35
73.28
+4.93
10–60 min
53.53
61.66
+8.13
Overall
62.08
68.36
+6.28
DeepSeek-V4.1-Flash
0–10 min
67.83
71.73
+3.90
10–60 min
54.42
60.42
+6.00
Overall
62.15
66.94
+4.79
Table 7: Generalization to advanced LVLMs on LongVideoBench with a 16-frame budget. Δ denotes the absolute accuracy improvement over uniform sampling in percentage points.
Video frame selection chooses a compact set of frames that preserves important events, context, or task-specific evidence while reducing processing costs.