EVA-RSI: Recursive Self-Improvement of Video Evaluators for Judging Temporal Quality
Organizations: Roblox · Purdue University · University of Texas at Austin
Abstract
Video evaluators play an important role in video generation, yet they miss motion defects, limiting the quality of generated videos. Recursive self-improvement (RSI) has made strong progress in mathematics and coding, where explicit verifiers provide reliable feedback on a single goal. Video evaluation poses two difficulties for RSI: it has no verifier, and it must judge many dimensions at once, such as physics, smoothness and dynamics, where a gain on one can cost another. We introduce EVA-RSI, a multi-agent framework for exploring RSI under these two constraints. We build a fixed proxy exam with construction-defined targets by applying controlled changes to videos, such as temporal reversal and frame dropping. These edits supply supervision without human annotation. A small set of human-labeled real videos anchors the exam to human judgment. Each dimension has an LLM agent that proposes improvements. A leader agent coordinates the proposals, and a Nash bargaining rule selects updates that balance progress across dimensions. Humans design the exam, review candidate changes and revise the curriculum between stages. Starting from Qwen3-VL-8B, EVA-RSI improves motion evaluation, with the largest gain on smoothness, and the gains carry over to defect types never seen during the run. On the dynamics subsets of public benchmarks, our evaluator achieves the highest observed scores among 21 tested models. As an RL reward, it increases generated motion and earns human preference. A data-selection pipeline incorporating it also increases motion.
Figures & tables
| Safeguard | Targeted failure |
| Two banks, 64 question types, both orders, ratings | a shortcut on one bank or format |
| Floors and family guards | trading one skill for the mean |
| Hidden test bank | overfitting the development banks |
| 12 public benchmarks | overfitting the exam as a whole |
| Frozen scorer and rule, diff filter | editing the judge, training on banks |
| Reviewer agent, human inspection | violations the program misses |
| Dimension | Qwen3-VL-8B | After stage 1 | EVA-RSI | Gain |
| Physics | 0.739 | 0.881 | 0.858 | |
| Consistency | 0.877 | 0.977 | 0.997 | |
| Smoothness | 0.461 | 0.841 | 0.888 | |
| Dynamics | 0.769 | 0.904 | 0.923 | |
| Real videos | 0.684 | 0.755 | 0.863 | |
| Semantics | 0.988 | 0.992 | 0.992 |
| Stage | Champion | Role | Round | Recipe in one sentence | Test gain |
| 1 | c007 | dynamics | 5 | A weight average (“soup”) of three earlier candidates, which keeps their gains with less drift in human preference. | |
| 1 | c009 | physics | 5 | A weight soup plus a short LoRA polish with graded time-reversal labels and new physics and consistency defect families; proposed in the same round as c007 and judged against it after c007 was accepted. | |
| 1 | c022 | physics | 14 | A direction-aware vision tower: a time-antisymmetric change to the patch embedding, trained on motion and direction labels, language model frozen. | |
| 1 | c029 | semantics | 17 | Composes c022’s vision tower with the language model of c021, at a higher resolution. | |
| 1 | c046 | human | 26 | The best 32-frame vision tower plus a language-model LoRA, with a repair step for human preference and semantics. | |
| 2 | c001 | human | 1 | Fine-tuning on real generated videos with human labels, with base-model anchors for text alignment and replay of earlier defect families. |
| Model | VideoScore-Bench | VBench |
| EVA-RSI (ours) | 0.611 | 0.906 |
| After stage 1 (ours) | 0.564 | 0.781 |
| Qwen3-VL-8B (start) | 0.538 | 0.812 |
| Qwen3-VL-32B | 0.495 | 0.875 |
| Qwen3.8-27B (thinking) | 0.453 | 0.750 |
| InternVL3.5-8B [ 53 ] | 0.433 | 0.750 |
| Outcome | Stage 1 | Stage 2 | All |
| Accepted | 5 | 3 | 8 |
| Scored and rejected | 41 | 38 | 79 |
| main gain too small | 31 | 32 | 63 |
| dimension below floor | 10 | 5 | 15 |
| a family guard | – | 1 | 1 |
| Not scored | 34 | 49 | 83 |
| Reward | Upd. | Dyn. | Flow | Obj. | Adh. | Elem. |
| None (start) | 0 | 0.579 | 13.77 | 1.43 | 3.668 | 0.745 |
| Qwen3-VL-8B, seed 1 | 50 | 0.790 | 15.64 | 1.67 | 3.741 | 0.762 |
| Qwen3-VL-8B, seed 2 | 50 | 0.749 | 14.97 | 1.49 | 3.684 | 0.751 |
| EVA-RSI, seed 1 | 50 | 0.848 | 17.64 | 1.76 | 3.730 | 0.761 |
| EVA-RSI, seed 2 | 50 | 0.892 | 19.81 | 2.03 | 3.714 | 0.757 |
| Qwen3-VL-8B, seed 1 | 100 | 0.770 | 14.93 | 1.37 | 3.754 | 0.761 |
| Question | vs. Qwen3-VL-8B | vs. VideoReward (fair) |
| More motion | 0.77 | 0.85 |
| Better overall | 0.73 | 0.69 |
| Visual quality | 0.71 | 0.66 |
| Prompt adherence | 0.73 | 0.71 |
| Picked random | Step 1,500 | Step 3,000 |
| Dynamic degree | ||
| Optical flow | ||
| Adherence | ||
| Elements | ||
| Dyn. degree, picked / random | 0.867 / 0.723 | 0.873 / 0.597 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Start | Rounds | Candidates | Champions |
| 1 | Qwen3-VL-8B | 49 | 80 | 5 |
| 2 | stage-1 champion | 39 | 90 | 3 |
| Total | 88 | 170 | 8 |
| Step | Gain in main mean | 95% interval | Holm |
| Start c007 | ✓ | ||
| c007 c009 | ✓ | ||
| c009 c022 | |||
| c022 c029 | |||
| c029 c046 | |||
| c046 c001 | ✓ |
| Stage 1 | Stage 2 | ||||||||
| Dimension | Base | c007 | c009 | c022 | c029 | c046 | c001 | c033 | c071 (EVA-RSI) |
| Proposed in round | – | 5 | 5 | 14 | 17 | 26 | 1 | 16 | 30 |
| Frames | 32 † | 32 | 32 | 32 | 32 | 32 | 32 | 32 | 54 |
| Pixels per frame (k) | 147 † | 147 | 147 | 147 | 230 | 230 | 230 | 230 | 230 |
| Physics | 0.739 | 0.857 | 0.874 | 0.868 | 0.893 | 0.881 | 0.886 | 0.868 | 0.858 |
| Consistency | 0.877 | 0.943 | 0.960 | 0.968 | 0.977 | 0.977 | 0.995 | 0.998 | 0.997 |
| Questions | Start | EVA-RSI | Gain [95% int.] |
| Seen families | 0.662 | 0.909 | |
| Unseen families | 0.754 | 0.913 | [0.116, 0.200] |
| physics | 0.849 | 0.943 | |
| consistency | 0.985 | 1.000 | |
| smoothness | 0.428 | 0.795 | [0.252, 0.479] |
| Main mean, all | 0.706 | 0.906 | [0.181, 0.219] |
| Cand. | Parent | Decision | Dev gain | Test main | main | human |
| c010 | c001 | Nash | 0.896 | |||
| c035 | c001 | Nash | 0.894 | |||
| c056 | c033 | Nash | 0.905 | |||
| c033 (54 frames) | c033 | family drop | 0.904 | |||
| c001 | stage-1 champ. | accepted | 0.896 | |||
| c033 | c001 | accepted | 0.902 |
| Model | Frames | Phys. | Cons. | Smooth. | Dyn. | Real | Sem. | Qual. | Human | Gen. | Main | Public | GPU h | Rule |
| Qwen3-VL-8B | 32 | 0.739 | 0.877 | 0.461 | 0.769 | 0.684 | 0.988 | 0.968 | 0.624 | 0.685 | 0.706 | 0.39 | – | – |
| Union SFT | 54 | 0.859 | 0.977 | 0.844 | 0.904 | 0.859 | 0.992 | 0.984 | 0.558 | 0.726 | 0.889 | 0.53 | 384 | human floor |
| Real-only SFT | 54 | 0.844 | 0.907 | 0.608 | 0.751 | 0.862 | 0.992 | 0.960 | 0.598 | 0.715 | 0.794 | 0.57 | 6.2 | human floor |
| Union + anchors, seed 1 | 54 | 0.867 | 0.980 | 0.683 | 0.853 | 0.856 | 0.992 | 0.978 | 0.647 | 0.715 | 0.848 | 0.66 | 352 | accept |
| Union + anchors, seed 2 | 54 | 0.873 | 0.969 | 0.679 | 0.847 | 0.854 | 0.992 | 0.964 | 0.654 | 0.723 | 0.844 | 0.60 | 352 | accept |
| c033 recipe, seed 1 | 32 | 0.850 | 0.915 | 0.514 | 0.772 | 0.733 | 0.992 | 0.965 | 0.632 | 0.704 | 0.757 | 0.46 | 7.2 | general floor |
| Mean rank score | Dynamics | Selected metrics | |||||||||
| Model | Type | Eval. (15) | Underst. (2) | VSB dyn. | VBench dyn. | VideoPhy-2 phys. | VideoScore-Bench | VBench pref. | T2VQA-DB | AIGVE align. | VSB-v2 text |
| EVA-RSI (ours) | open, 8B | 0.721 | 0.421 | 0.611 | 0.906 | 0.317 | 0.598 | 0.717 | 0.554 | 0.656 | 0.548 |
| After stage 1 (ours) | open, 8B | 0.434 | 0.447 | 0.564 | 0.781 | 0.092 | 0.295 | 0.707 | 0.309 | 0.518 | 0.556 |
| Qwen3-VL-8B (start) | open, 8B | 0.407 | 0.447 | 0.538 | 0.812 | 0.108 | 0.327 | 0.641 | 0.263 | 0.599 | 0.570 |
| Qwen3-VL-32B | open | 0.542 | 0.711 | 0.495 | 0.875 | 0.156 | 0.347 | 0.685 | 0.334 | 0.630 | 0.674 |
| Qwen3.8-27B (thinking) | open | 0.628 | 0.711 | 0.453 | 0.750 | 0.221 | 0.389 | 0.601 | 0.444 | 0.696 | 0.663 |
| Statistic | EVA-RSI | Claude Opus 5.5 | GPT-5.6 Sol | Muse Spark 1.3 |
| Main result (21 models, 15 metrics) | 0.721 | 0.826 | 0.775 | 0.730 |
| 90% interval |
| Reward | Updates | GPU hours |
| EVA-RSI, seed 1 | 342 | 498.6 |
| EVA-RSI, seed 2 | 100 | 145.0 |
| Qwen3-VL-8B, seed 1 | 346 | 498.0 |
| Qwen3-VL-8B, seed 2 | 100 | 145.6 |
| Stage-1 champion | 100 | 144.4 |
| VideoReward | 130 ( Tab. 6 : 120) | 188.7 |
| Reward | Upd. | Dyn. | Flow | Adh. | Elem. |
| Stage-1 champion | 50 | 0.857 | 18.77 | 3.659 | 0.748 |
| Qwen3-VL-8B, seed 1 | 100 | 0.770 | 14.93 | 3.754 | 0.761 |
| Qwen3-VL-8B, seed 2 | 100 | 0.700 | 13.48 | 3.671 | 0.748 |
| Stage-1 champion | 100 | 0.803 | 15.29 | 3.648 | 0.743 |
| EVA-RSI, seed 1 | 100 | 0.913 | 20.53 | 3.675 | 0.750 |
| EVA-RSI, seed 2 | 100 | 0.892 | 17.59 | 3.519 | 0.712 |
| 630 test videos | 300 general prompts | ||||||||
| Seed | 1 | 2 | 1 | 2 | 1 | 2 | 1 | 2 | |
| Update | 50 | 50 | 100 | 100 | 50 | 50 | 100 | 100 | |
| Motion | |||||||||
| Dynamic degree | |||||||||
| Optical flow | |||||||||
| Object motion | |||||||||
Video Generation Evaluation
Video generation evaluation measures visual quality, prompt adherence, temporal consistency, physical plausibility, and task-specific behavior in synthesized videos.
See all 205 papers