Video understanding with multimodal large language models (MLLMs) remains challenging due to the long token sequences of videos, which contain extensive temporal dependencies and redundant frames. Existing approaches typically treat MLLMs as passive recognizers, processing entire videos or uniformly sampled frames without adaptive reasoning. Recent agent-based methods introduce external tools, yet still depend on manually designed workflows and perception-first strategies, resulting in inefficiency on long videos. We present EVA, an Efficient Reinforcement Learning framework for End-to-End Video Agent, which enables planning-before-perception through iterative summary-plan-action-reflection reasoning. EVA autonomously decides what to watch, when to watch, and how to watch, achieving query-driven and efficient video understanding. To train such agents, we design a simple yet effective three-stage learning pipeline - comprising supervised fine-tuning (SFT), Kahneman-Tversky Optimization (KTO), and Group Relative Policy Optimization (GRPO) - that bridges supervised imitation and reinforcement learning. We further construct high-quality datasets for each stage, supporting stable and reproducible training. We evaluate EVA on six video understanding benchmarks, demonstrating its comprehensive capabilities. Compared with existing baselines, EVA achieves a substantial improvement of 6-12% over general MLLM baselines and a further 1-3% gain over prior adaptive agent methods.
Figures & tables
Figure 1: Given a question that requires the MLLM to figure out action sequences from an extremely long video (over 6600 seconds), the traditional uniformed sampling method is limited by the content length of the MLLM, and it is extremely hard for it to sample all the keyframes to answer the question correctly. As for the Traditional Agentic Method, the agent will also be given the uniformly sampled frames along with the video, which already occupy a lot of context. Although the agent can call tools to extract frames from a specific time range, the tool is rigid and the agent cannot adjust the fps and resolution, which leads to potential information loss. However, in EVA, the agent can arrange the tokens wisely. It can first watch the whole video with low resolution and high fps to get an overview of the video without costing too many visual tokens. After it finds the key time range, it will extract frames with high fps and high resolution, which leads to the correct answer.
Figure 2: Data Pipeline and Training Stage of EVA. The base model is first fine-tuned on synthetic dataset with certain reasoning and tool-calling pattern. Then we use KTO to help the model learn from typical failures. Finally, we introduce a Data-Enhanced Multi-Stage GRPO training pipeline, where we collect the failure cases of current policy and employ an teacher MLLM to generate new open-ended video QA dataset.
Figure 3: Distribution of the Training Dataset
Method
Frames
Visual Token
Acc (%)
Close Source Model
Gemini-2.0-Flash [ 34 ]
2700
696.6k
56.2
Open Source Model
256
36.9k
31.3
512
73.7k
33.0
LongVA [ 45 ]
1024
147.5k
32.5
Table 1: Performance on sampling dilemma bench. We report different stages of EVA model’s performance on LSDbench. SOTA model performance are directly from [ 17 ] . The visual token are roughly measured by 258, 144 ,256 and 650 token per frame for gemini, LongVA, LongVila and Qwen-VL family models.
LongVideoBench
MLVU
VideoMME-Long/Overall
LVBench
Model
Frame
Acc
Frame
Acc
Frame
Acc
Frame
Acc
Close Source Models
GPT-4o [ 19 ]
32
58.2
0.5 fps
64.6
384
65.3/71.9
60
48.9
Gemini-1.5-Pro [ 34 ]
32
55.2
-
-
0.5fps
67.4/75.0
3600
33.1
Static Frame Sampling
ShareGPT4Video [ 4 ]
16
39.7
16
46.4
16
35.0/39.9
-
-
Table 2: Main performance on multiple video understanding benchmark. Baseline results are directly cited from [ 17 ] .The number of frames for EVA, which is indicated by *, is estimated by assuming 650 visual tokens per frame for fair comparison; the actual number of frames may vary depending on the resolution determined by the model adaptively.
Model
Frame
SR
IMC
TCI
TA
MHR
PAR
CTI
Overall
Close Source Models
GPT-4o [ 19 ]
32
50.0
49.6
38.8
30.0
44.0
39.2
37.0
42.0
Gemini-2.0-Flash [ 34 ]
-
41.8
33.7
23.1
20.5
30.1
26.8
33.7
30.6
Open Source Model
InternVL2.5-8B [ 7 ]
32
28.0
32.2
21.5
7.7
25.7
23.8
22.6
23.8
InternVL3-8B [ 49 ]
32
29.5
40.7
37.9
35.1
24.6
38.9
24.1
32.3
Table 3: Zero-shot performance on video reasoning benchmark: Video-Holmes [ 8 ] , where SR stands for Social Reasoning; IMC stands for Intention & Motive Chaining; TCI stands for Temporal Causal Inference; TA Timeline Analysis; MHR stands for Multimodal Hint Reasoning; PAR stands for Physical Anomaly Reasoning; CTI stands for Core Theme Inference.
Figure 4: Distribution of Rounds and Visual Token cross Models and Benchmarks
Figure 5: Ablation study on the GRPO training dataset. The comparison between multi-choice (MC) only, open-ended (OE) only, and mixed (MC+OE) data shows that mixed data provides a more effective learning environment for the agent, which leads to better performance on VideoMME.
Figure 6: Diverse Workflows generated by EVA.
Figure 7: The advantage of plan before perception matters
Figure 8: Cold Start Data Pipeline
Figure 9: EVA statistics across rounds.
Models
LLM size
Visual Details
Object
Action
Declarative Content
Avg Acc↑
Avg Diff.↓
SAH Ratio↓
In.
Out.
Diff.
In.
Out.
Diff.
In.
Out.
Diff.
In.
Out.
Diff.
Closed Source Models
GPT-4o
/
8.5
14.2
5.7
16.3
17.5
1.2
13
15.2
2.2
13.3
12.2
-1.1
13.7
2
2.2
Open Source Models
Qwen2.5VL-3B
3B
2.2
10.5
8.3
7.7
13.8
6.1
5
8
3
6
6
0
7.4
4.3
4.5
LLaVA-OV-7b
7B
4.5
16.5
12
6.75
11.5
4.75
3.5
9.5
6
6
7
1
8.1
5.9
6.2
Table 4: Main results on ELV-Halluc. Diff. denotes the gap between in-video and out-video accuracy. All accuracies are shown as percentages.
Video understanding develops methods for recognizing actions, tracking events, grounding evidence in time, and reasoning across visual and audio streams.
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China