Organizations: Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Tencent Hunyuan · School of Computer Science and Communication Engineering, Jiangsu University of Science and Technology · AICOMO · College of Computer Science, Inner Mongolia University
Video understanding agents rely on a harness to acquire evidence, use tools, and reason, but improving it requires substantial effort. Recent self evolution methods seek to optimize harnesses through execution feedback to reduce human intervention. However, existing methods often rely on annotated video questions and answers or focus on updating model parameters, leaving harness evolution from unlabeled videos underexplored. VideoEvoHarness jointly evolves a Questioner and a Solver harness from unlabeled videos with frozen model parameters. The Questioner constructs questions and computes reference answers directly from video memory records. Question quality assessments and Solver execution trajectories guide question generation policy revisions. The Solver supports evolving skills, strategies, hyperparameters, and tools using execution feedback. Executable strategies define workflows in code, reducing repeated interpretation of natural language instructions. Each round, candidate harnesses are compared on identical validation questions, with validated improvements retained for later use. Experiments on benchmarks and a dedicated benchmark question set, using videos disjoint from evolution videos, show higher overall accuracy than initially and transfer gains across backbones. Overall accuracy on this set also improves progressively across rounds. This supports our foundational framework for harness evolution from unlabeled videos, opening a promising direction for self evolving video agents. Code is available at https://github.com/Wei-Xin-Xu/VideoEvoHarness.
Figures & tables
Figure 1: Comparison of video agent evolution paradigms. (a) Agent code evolution revises agent code through review and diagnosis, and relies on previously annotated questions, answers, and evidence. (b) Model parameter evolution builds supervision from unlabeled video through Questioner and Solver interaction, but updates model parameters and leaves the agent harness unchanged. (c) VideoEvoHarness derives questions, reference answers, and source intervals from video memory, and uses validated feedback to support Questioner refinement and Solver updates across skills, strategy code, hyperparameters, and tools, while the model stays frozen.
Figure 2: Overview of the VideoEvoHarness framework. The Questioner constructs and filters tasks from structured video memory, while the Solver acquires visual evidence through frame sampling or memory retrieval. Execution feedback supports revisions to the question generation policy and the Solver’s skills, strategy code, hyperparameters, and tool operations. Validated updates enable both agents to evolve from unlabeled videos while keeping model parameters fixed.
Method
L1-Perception
L2-Relation
Overall
S2E
S2O
S2A
E2O
O2E
T2E
T2O
T2A
Avg.
E3E
O3O
SSS
SOS
SAA
T3E
T3O
TOS
TAA
Avg.
Qwen3.7 Plus
Initial harness
69.9
72.2
85.1
75.4
65.5
64.5
72.0
65.3
71.4
74.7
74.2
55.8
72.8
59.7
52.1
59.5
31.9
53.7
59.6
65.1
VideoEvoHarness (R1)
72.0
70.8
86.2
73.8
70.1
62.9
78.7
64.0
72.7
75.8
77.3
64.2
75.3
58.3
60.3
63.5
31.9
43.9
61.5
66.7
VideoEvoHarness (R3)
73.1
66.7
87.4
80.0
65.5
67.7
76.0
68.0
73.2
78.0
80.3
61.1
82.7
65.3
53.4
54.1
34.7
56.1
63.2
67.9
Gemini 3.8 Flash
Table 1: Per-category results on LongVideoBench with frozen backbones. Question categories follow the official taxonomy and are grouped into L1-Perception and L2-Relation. Avg. reports accuracy over all questions within each group, and Overall reports accuracy over all questions. Scores are accuracy in percent, and the best result for each backbone is in bold.
Method
MMVU
V-Holmes
Sci.
Health.
Hum.
Eng.
Overall
Qwen3.7 Plus
Initial harness
76.5
79.3
87.0
80.9
80.0
56.5
VideoEvoHarness (R1)
78.4
82.6
85.9
81.9
81.4
57.1
VideoEvoHarness (R3)
79.8
85.1
83.7
82.9
82.4
58.0
Gemini 3.8 Flash
Table 2: Results on MMVU and Video-Holmes.
Figure 5Table 6
Harness
Perc.
Temp.
Reas.
All
Full harness
72.5
65.5
67.8
69.5
Strategy
68.0
62.7
67.8
66.5
Hyperparams.
69.5
60.0
66.7
66.3
Initial harness
67.5
58.2
72.2
66.0
Table 5: Ablation of Solver evolution. Column maxima are in bold.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Memory construction
Scene detection rate
0.25 fps
Minimum / maximum segment duration
30 / 300 s
Subgraph sampling rate / frame cap per segment
0.5 fps / 250
Question generation and quality
Number of answer options K
4
Appendix
Table 6: Key hyperparameters. Sampling settings for memory construction and Solver execution are listed separately. Model-call entries specify temperature / maximum output tokens.
Group
Question type
LVBench
MLVU
LongVB
V-Holmes
MMVU
V-MME
Total
Share
Perception (200)
Counting
3
8
2
0
1
6
20
5.00
Entities and attributes
15
13
25
4
6
17
80
20.00
Actions and events
10
15
12
6
5
12
60
15.00
Spatial context
7
6
13
3
4
7
40
10.00
Temporal (110)
Temporal order
10
15
14
7
5
9
60
15.00
Retrieval and localization
10
12
8
5
3
12
50
12.50
Appendix
Table 7: Composition of the dedicated benchmark question set used in Figure 4 and Section 4.4 . Question types are divided into three reporting groups, with the group size given in parentheses. Each column reports the number of questions contributed by one benchmark, and Share is the proportion of the full set. LongVB, V-Holmes, and V-MME denote LongVideoBench, Video-Holmes, and Video-MME.
Table 10
Figure 5: Generated action-recognition question. The micro-event record supports answer A: spraying textured paint onto a plate.
Figure 7: Comparison of the initial and R3 Solver harnesses on Video-Holmes with Qwen3.7 Plus. Compared with the initial harness, the R3 harness supplements inspection of the clapping interval with later evidence, leading to answer E rather than A. The comparison illustrates a change in evidence acquisition while the backbone remains frozen.
Figure 8: Illustrative strategy code change during evolution. The generated routine combines keyword-conditioned retrieval with semantic search and a summary-based fallback.
Figure 9: Illustrative configuration changes during evolution: enabling the generated retrieval strategy and reducing the replay-window cap from 60 to 15 seconds.