Video evaluators play an important role in video generation, yet they miss motion defects, limiting the quality of generated videos. Recursive self-improvement (RSI) has made strong progress in mathematics and coding, where explicit verifiers provide reliable feedback on a single goal. Video evaluation poses two difficulties for RSI: it has no verifier, and it must judge many dimensions at once, such as physics, smoothness and dynamics, where a gain on one can cost another. We introduce EVA-RSI, a multi-agent framework for exploring RSI under these two constraints. We build a fixed proxy exam with construction-defined targets by applying controlled changes to videos, such as temporal reversal and frame dropping. These edits supply supervision without human annotation. A small set of human-labeled real videos anchors the exam to human judgment. Each dimension has an LLM agent that proposes improvements. A leader agent coordinates the proposals, and a Nash bargaining rule selects updates that balance progress across dimensions. Humans design the exam, review candidate changes and revise the curriculum between stages. Starting from Qwen3-VL-8B, EVA-RSI improves motion evaluation, with the largest gain on smoothness, and the gains carry over to defect types never seen during the run. On the dynamics subsets of public benchmarks, our evaluator achieves the highest observed scores among 21 tested models. As an RL reward, it increases generated motion and earns human preference. A data-selection pipeline incorporating it also increases motion.
Figures & tables
Figure 2: The EVA-RSI system. ① A team of LLM agents proposes, implements and trains candidate evaluators from the current champion. ② A frozen verifier scores each candidate on an exam with fixed targets, and a Nash bargaining rule accepts or rejects it. An accepted candidate becomes the champion of the next round; every outcome enters a shared memory that the proposers read. Humans design the verifier and redesign the exam between stages.
Safeguard
Targeted failure
Two banks, 64 question types, both orders, ratings
a shortcut on one bank or format
Floors and family guards
trading one skill for the mean
Hidden test bank
overfitting the development banks
12 public benchmarks
overfitting the exam as a whole
Frozen scorer and rule, diff filter
editing the judge, training on banks
Reviewer agent, human inspection
violations the program misses
Table 1: Controls on evaluator search.
Figure 3: RSI improves the evaluator over rounds. (a) Normalized score of each champion on the hidden test bank. (b) Scored candidates (grey) and champions (blue) on the development banks, which change at the start of stage 2. (c) Mean rank score on 12 public benchmarks, against all evaluators except VideoReward ( Appendix D ). The development banks determine the final champion.
Figure 4: Test-bank pairs before and after RSI. B is the correct answer in each pair. P is the probability that each model gives to B. Left: close-up of four consecutive frames; dashed lines mark a fixed position. Middle: objects suddenly change size. Right: people and boxes vanish and reappear.
Dimension
Qwen3-VL-8B
After stage 1
EVA-RSI
Gain
Physics
0.739
0.881
0.858
+0.119
Consistency
0.877
0.977
0.997
+0.120
Smoothness
0.461
0.841
0.888
+0.427
Dynamics
0.769
0.904
0.923
+0.154
Real videos
0.684
0.755
0.863
+0.179
Semantics
0.988
0.992
0.992
+0.004
Table 2: Normalized scores on the hidden test bank. Top: main dimensions. Bottom: guard dimensions. Chance is 0.5 for pairwise questions.
Stage
Champion
Role
Round
Recipe in one sentence
Test gain
1
c007
dynamics
5
A weight average (“soup”) of three earlier candidates, which keeps their gains with less drift in human preference.
+0.068
1
c009
physics
5
A weight soup plus a short LoRA polish with graded time-reversal labels and new physics and consistency defect families; proposed in the same round as c007 and judged against it after c007 was accepted.
+0.142
1
c022
physics
14
A direction-aware vision tower: a time-antisymmetric change to the patch embedding, trained on motion and direction labels, language model frozen.
+0.153
1
c029
semantics
17
Composes c022’s vision tower with the language model of c021, at a higher resolution.
+0.158
1
c046
human
26
The best 32-frame vision tower plus a language-model LoRA, with a repair step for human preference and semantics.
+0.166
2
c001
human
1
Fine-tuning on real generated videos with human labels, with base-model anchors for text alignment and replay of earlier defect families.
+0.190
Table 3: The eight champions found by RSI. Role: the agent that proposed the candidate. Test gain: main mean minus the start model on the hidden test bank.
Model
VideoScore-Bench
VBench
EVA-RSI (ours)
0.611
0.906
After stage 1 (ours)
0.564
0.781
Qwen3-VL-8B (start)
0.538
0.812
Qwen3-VL-32B
0.495
0.875
Qwen3.8-27B (thinking)
0.453
0.750
InternVL3.5-8B [ 53 ]
0.433
0.750
Table 4: Video dynamics on public benchmarks. Spearman correlation with human dynamics scores (VideoScore-Bench) and preference accuracy on dynamics pairs (VBench). 80 videos and 16 pairs from 6 prompts. Bold: highest observed score among 21 models ( Tab. S7 ).
Outcome
Stage 1
Stage 2
All
Accepted
5
3
8
Scored and rejected
41
38
79
main gain too small
31
32
63
dimension below floor
10
5
15
a family guard
–
1
1
Not scored
34
49
83
Table 5: Outcomes of the 170 candidates. Each rejected candidate counts under the first rule that it failed.
Reward
Upd.
Dyn. ↑
Flow ↑
Obj. ↑
Adh. ↑
Elem. ↑
None (start)
0
0.579
13.77
1.43
3.668
0.745
Qwen3-VL-8B, seed 1
50
0.790
15.64
1.67
3.741
0.762
Qwen3-VL-8B, seed 2
50
0.749
14.97
1.49
3.684
0.751
EVA-RSI, seed 1
50
0.848
17.64
1.76
3.730
0.761
EVA-RSI, seed 2
50
0.892
19.81
2.03
3.714
0.757
Qwen3-VL-8B, seed 1
100
0.770
14.93
1.37
3.754
0.761
Table 6: RL with different rewards (630 test videos). Dyn.: dynamic degree. Obj.: object motion. Adh. and Elem.: judge scores. VideoReward (fair): same penalties and settings as EVA-RSI. VideoReward: stress test without penalties.
Figure 5: Same prompt and seed; only the reward changes (update 50). EVA-RSI keeps the scene and gives the most motion. VideoReward: visual and motion quality, without penalties. Corner: optical flow.
Question
vs. Qwen3-VL-8B
vs. VideoReward (fair)
More motion
0.77 [0.68,0.85]
0.85 [0.75,0.94]
Better overall
0.73 [0.65,0.80]
0.69 [0.55,0.82]
Visual quality
0.71 [0.63,0.79]
0.66 [0.51,0.79]
Prompt adherence
0.73 [0.65,0.80]
0.71 [0.58,0.82]
Table 7: Human study at update 50 (training seed 1) , share of answers for EVA-RSI (0.5: no preference). Six raters; 48 and 24 pairs.
Picked − random
Step 1,500
Step 3,000
Dynamic degree ↑
+0.143∗
+0.277∗
Optical flow ↑
+3.47∗
+7.38∗
Adherence ↑
+0.10
+0.03
Elements ↑
+0.022
−0.001
Dyn. degree, picked / random
0.867 / 0.723
0.873 / 0.597
Table 8: Selection pipeline incorporating EVA-RSI versus random data (300 prompts). ∗ : 95% interval excludes zero.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Start
Rounds
Candidates
Champions
1
Qwen3-VL-8B
49
80
5
2
stage-1 champion
39
90
3
Total
88
170
8
Appendix
Table S1 : The two stages of EVA-RSI.
Step
Gain in main mean
95% interval
Holm
Start → c007
+0.068
[+0.054,+0.082]
✓
c007 → c009
+0.074
[+0.056,+0.091]
✓
c009 → c022
+0.011
[+0.001,+0.022]
c022 → c029
+0.005
[−0.008,+0.019]
c029 → c046
+0.007
[−0.002,+0.017]
c046 → c001
+0.024
[+0.012,+0.037]
✓
Appendix
Table S2 : Steps between champions (stage-2 hidden test bank). Holm: significant after correction over the eight steps.
Stage 1
Stage 2
Dimension
Base
c007
c009
c022
c029
c046
c001
c033
c071 (EVA-RSI)
Proposed in round
–
5
5
14
17
26
1
16
30
Frames
32 †
32
32
32
32
32
32
32
54
Pixels per frame (k)
147 †
147
147
147
230
230
230
230
230
Physics
0.739
0.857
0.874
0.868
0.893
0.881
0.886
0.868
0.858
Consistency
0.877
0.943
0.960
0.968
0.977
0.977
0.995
0.998
0.997
Appendix
Table S3 : All eight champions on the hidden test bank. Main: mean of the five main dimensions. † Default settings.
Questions
Start
EVA-RSI
Gain [95% int.]
Seen families
0.662
0.909
+0.247
Unseen families
0.754
0.913
+0.159 [0.116, 0.200]
physics
0.849
0.943
+0.094
consistency
0.985
1.000
+0.015
smoothness
0.428
0.795
+0.367 [0.252, 0.479]
Main mean, all
0.706
0.906
+0.200 [0.181, 0.219]
Appendix
Table S4 : Unseen defects and shared questions on the hidden test bank. Families: mean over physics, consistency and smoothness. Not shared: without the questions that the stage-1 test bank also contains. Intervals: bootstrap over clip clusters.
Cand.
Parent
Decision
Dev gain
Test main
Δ main
Δ human
c010
c001
Nash
+0.011
0.896
+0.000
−0.007
c035
c001
Nash
+0.011
0.894
−0.002
−0.003
c056
c033
Nash
+0.013
0.905
+0.003
−0.006
c033 (54 frames)
c033
family drop
+0.011
0.904
+0.002
+0.002
c001
stage-1 champ.
accepted
+0.021
0.896
+0.024
+0.016
c033
c001
accepted
+0.010
0.902
+0.006
+0.000
Appendix
Table S5 : Acceptance replay on the stage-2 hidden test bank. Top: rejected candidates with a development gain of at least 0.01 . Bottom: champions. Δ : test score minus the parent.
Model
Frames
Phys.
Cons.
Smooth.
Dyn.
Real
Sem.
Qual.
Human
Gen.
Main
Public
GPU h
Rule
Qwen3-VL-8B
32
0.739
0.877
0.461
0.769
0.684
0.988
0.968
0.624
0.685
0.706
0.39
–
–
Union SFT
54
0.859
0.977
0.844
0.904
0.859
0.992
0.984
0.558
0.726
0.889
0.53
384
human floor
Real-only SFT
54
0.844
0.907
0.608
0.751
0.862
0.992
0.960
0.598
0.715
0.794
0.57
6.2
human floor
Union + anchors, seed 1
54
0.867
0.980
0.683
0.853
0.856
0.992
0.978
0.647
0.715
0.848
0.66
352
accept
Union + anchors, seed 2
54
0.873
0.969
0.679
0.847
0.854
0.992
0.964
0.654
0.723
0.844
0.60
352
accept
c033 recipe, seed 1
32
0.850
0.915
0.514
0.772
0.733
0.992
0.965
0.632
0.704
0.757
0.46
7.2
general floor
Appendix
Table S6 : Baselines without the loop on the hidden test bank. Main: mean of the five main dimensions. Public: mean rank score as in Fig. 3 c. Rule: decision on the development banks ( Sec. A.6 ). Bold: best.
Figure S1 : Outcomes of the candidates. (a–c) Outcome per stage. In stage 3 ( Appendix A ), no candidate met the frozen rule, two passed only a looser rule (teal), and three training jobs failed (unlabelled slices). (d) Why scored candidates were rejected, stages 1 and 2. (e) Candidates by proposing agent, stages 1 and 2; the dark part of each bar counts the accepted ones; the semantics and quality roles of stage 1 are counted as general.
Mean rank score
Dynamics
Selected metrics
Model
Type
Eval. (15)
Underst. (2)
VSB dyn.
VBench dyn.
VideoPhy-2 phys.
VideoScore-Bench
VBench pref.
T2VQA-DB
AIGVE align.
VSB-v2 text
EVA-RSI (ours)
open, 8B
0.721
0.421
0.611
0.906
0.317
0.598
0.717
0.554
0.656
0.548
After stage 1 (ours)
open, 8B
0.434
0.447
0.564
0.781
0.092
0.295
0.707
0.309
0.518
0.556
Qwen3-VL-8B (start)
open, 8B
0.407
0.447
0.538
0.812
0.108
0.327
0.641
0.263
0.599
0.570
Qwen3-VL-32B
open
0.542
0.711
0.495
0.875
0.156
0.347
0.685
0.334
0.630
0.674
Qwen3.8-27B (thinking)
open
0.628
0.711
0.453
0.750
0.221
0.389
0.601
0.444
0.696
0.663
Appendix
Table S7: Comparison on 12 public benchmarks (14 of 21 models; the other 7 are listed below). Mean rank score: rank on each metric mapped to [0,1] , then averaged. Bold: best per column.
Statistic
EVA-RSI
Claude Opus 5.5
GPT-5.6 Sol
Muse Spark 1.3
Main result (21 models, 15 metrics)
0.721
0.826
0.775
0.730
90% interval
[0.61,0.76]
[0.71,0.84]
[0.67,0.80]
[0.61,0.75]
Appendix
Table S8 : Mean rank scores and bootstrap intervals.
Reward
Updates
GPU hours
EVA-RSI, seed 1
342
498.6
EVA-RSI, seed 2
100
145.0
Qwen3-VL-8B, seed 1
346
498.0
Qwen3-VL-8B, seed 2
100
145.6
Stage-1 champion
100
144.4
VideoReward
130 ( Tab. 6 : 120)
188.7
Appendix
Table S9 : RL runs from Wan2.1 T2V-1.3B. The two long runs have a budget of 500 GPU hours.
Reward
Upd.
Dyn. ↑
Flow ↑
Adh. ↑
Elem. ↑
Stage-1 champion
50
0.857
18.77
3.659
0.748
Qwen3-VL-8B, seed 1
100
0.770
14.93
3.754
0.761
Qwen3-VL-8B, seed 2
100
0.700
13.48
3.671
0.748
Stage-1 champion
100
0.803
15.29
3.648
0.743
EVA-RSI, seed 1
100
0.913
20.53
3.675
0.750
EVA-RSI, seed 2
100
0.892
17.59
3.519
0.712
Appendix
Table S10 : More RL checkpoints (630 test videos).
630 test videos
300 general prompts
Seed
1
2
1
2
1
2
1
2
Update
50
50
100
100
50
50
100
100
Motion ↑
Dynamic degree
+0.06∗
+0.14∗
+0.14∗
+0.19∗
+0.07∗
+0.10∗
+0.13∗
+0.14∗
Optical flow
+2.0∗
+4.8∗
+5.6∗
+4.1∗
+2.7∗
+6.0∗
+5.4∗
+4.3∗
Object motion
+0.09∗
+0.54∗
+0.51∗
+0.27∗
+0.17∗
+0.74∗
+0.54∗
+0.37∗
Appendix
Table S11 : EVA-RSI minus Qwen3-VL-8B as the RL reward , same seed and update. ∗ : 95% interval excludes zero (whole prompts resampled). Dynamic degree and optical flow form the Holm family.
Figure S2 : Single questions through the chain of champions. Bars: probability of the correct answer. ✓: correct video.
Figure S3 : More prompts where EVA-RSI moves most (update 50). VideoReward: stress-test run.
Figure S4 : More examples (update 50).
Figure S5 : Largest motion gains of SFT on data picked by EVA-RSI.