Efficiently understanding long-form videos remains a fundamental challenge for multimodal large language models (MLLMs). In this paper, we present MLLM-Sampler Joint Evolution (MSJoE), a novel framework that jointly evolves the MLLM and a lightweight key-frame sampler for efficient long-form video understanding. MSJoE builds upon a key assumption that only a small subset of key-frames is truly informative for answering each question to a video. Specifically, MSJoE first reasons out several queries, which describe diverse visual perspectives relevant to the question. Then, these queries interact with a frozen CLIP model to produce a query-frame similarity matrix. Finally, a lightweight sampler predicts key-frame sampling weights from this matrix, selecting a compact set of informative frames, which are then fed into the MLLM for answer generation. Both the MLLM and sampler are jointly optimized through reinforcement learning, enabling co-adaptation of query-reasoning, frame-sampling, and key-frame understanding. A new long-video QA dataset containing 2.8K videos with 7K question-answer pairs is collected to support the training process. Extensive experiments on VideoMME, LongVideoBench, LVBench, and MLVU show that MSJoE achieves 8.0% accuracy gain upon the base MLLM, and 1.1% higher accuracy than strongest baseline method.
Figures & tables
Figure 1: A direct comparison among static key-frame sampling algorithms, trainable key-frame sampler, and our proposed MLLM-Sampler Joint Evolution framework (MSJoE).
Figure 2: Our dataset construction pipeline. QA pairs with low difficulty or poor quality are removed during multi-stage filtering.
Figure 3: The proposed MSJoE framework. Given a video and question, MSJoE generates reasoning-based queries from a sparse preview, matches them against dense frames via CLIP to create a similarity matrix, and uses a lightweight U-Sampler to select informative frames. The MLLM then processes these key frames at high resolution for answer generation. The entire framework is jointly optimized through end-to-end reinforcement learning.
Method
#Frames
MLVU
LongVideoBench
Video-MME
LVBench
Long
Avg.
Close-Sourse MLLMs
GPT-4o
-
66.7
64.6
65.3
71.9
-
Gemini-1.5-pro
-
64.0
-
67.4
75.0
33.1
Open-Source MLLMs
VideoMind-7B
-
-
64.4
49.2
58.2
40.8
Table 1: Comparison of MLLMs and baseline methods against our method MSJoE on four benchmarks. We bold the best results and highlight performance gain over the base-MLLM. The performance of baseline methods are reported according to published papers.
Figure 4: Ablation studies on varying input frames (x-axis). Four methods are evaluated: MSJoE in light violet, Top- k in red, and Uniform Sampling uniform sampling in gray.
M
S
MLVU
LoVi
VLong
LVB
Ours
T
PT-T
69.3
60.1
54.1
46.4
(i)
T
T
69.1
59.8
53.8
46.2
(ii)
T
PT-F
68.8
58.1
52.4
45.5
(iii)
F
PT-T*
68.7
58.2
52.1
44.8
(iv)
F
PT-F
68.2
56.8
52.2
44.3
(v)
F
top-k
64.5
55.1
45.4
41.8
Table 2: Ablation studies on the MLLM (M) and Sampler (S) module with a fixed input budget of 32 frames. The module settings T, PT, and F denote Training, Pre-Trained, Frozen respectively. Setting iii (PT-T*) denotes we feed the same frames (reasoned and sampled) as Ours to a frozen MLLM. Setting vi (F*) denotes using a Frozen MLLM w/ question-as-query. LoVi, VLong, and LVB denote the benchmarks LongVideoBench, VideoMME-Long, and LVBench, respectively.
Figure 5: Three frame sets from a publicly available video generated by different sampling strategies. Question: What motivated her to change dietary habits? (A) Family and friends (B) Diabetes (C) Tooth decay (D) Anemia .
Figure 6: Distribution of question–frame similarity for top-k (left) and query–frame similarity for MSJoE (right). Blue markers denote the frames selected by each method.
IR
DR
MLVU
LoVi
VLong
LVB
✓
✓
69.3
60.1
54.1
46.4
✗
✓
69.1
59.8
54.1
46.2
-
✓
68.2
56.8
52.2
44.3
-
✗
62.5
55.4
49.2
38.9
Table 3: Ablation studies on the Informativeness Reward (IR) and Difficulty-aware Reward (DR) with a fixed input budget of 32 frames. LoVi, VLong, and LVB denote the benchmarks LongVideoBench, VideoMME-Long, and LVBench, respectively.
Figure 7: Average accuracy on the four benchmarks under different number of input frames as a preview.
Query
CLIP
UNet
QA
ALL
Uniform
2.809
0.029
0.038
3.353
6.226
3.723
Table 4: Average time (s) cost by the four inference steps on 1,000 samples.
Figure 8: Smoothed reward curves across different training configurations.