RoboJudge: Multimodal Language Models as Judges for Embodied Video Generation

Siyuan Ma; Yu Qi; Cheng Ye; Ruotian Peng; Xinyi Xu; Zitiantao Lin; Zedong Cai; Yijian Huang

cs.CVOct 20, 2026

RoboJudge: Multimodal Language Models as Judges for Embodied Video Generation

Authors: Siyuan Ma, Yu Qi, Cheng Ye, Ruotian Peng, Xinyi Xu, Zitiantao Lin, Zedong Cai, Yijian Huang,

Organizations: The Chinese University of Hong Kong · Northeastern University · University of California, San Diego · Westlake University ·

Abstract

Video generation models are promising foundations for embodied world models, but only if their generated rollouts are physically valid and faithful to task instructions. Multimodal large language models (MLLMs) are increasingly used to judge these properties, yet their agreement with human judgments on embodied videos has not been systematically evaluated. To address this gap, we introduce RoboJudge-Bench, a benchmark for evaluating MLLMs as judges of embodied video generation. The benchmark measures two fundamental dimensions, Physical Adherence and Instruction Alignment, across 800 human-annotated videos from 8 source corpora and 10 video generators, spanning grippers, human hands, and dexterous hands. Experiments reveal uneven alignment between existing judges and human ratings: general-purpose MLLMs show lower agreement on physical adherence, while specialized video judges designed for general-domain physical reasoning do not consistently transfer to embodied scenarios. Motivated by these findings, we construct RoboJudge-Train, comprising 12K videos annotated with 25K human-verified scores and model-assisted rationales for Physical Adherence and Instruction Alignment. We fine-tune a 9B MLLM on this dataset with supervised learning followed by reinforcement learning, yielding RoboJudge-9B. Among the model judges evaluated on RoboJudge-Bench, RoboJudge-9B achieves the highest overall agreement, reaching a Pearson correlation of 0.719 with a statistically significant 0.089 gain over Gemini-3.7-Flash, the best-performing proprietary baseline in our evaluation. Training on RoboJudge-Train also yields gains on external benchmarks, including VideoPhy2 and Physion-Eval. We further use RoboJudge-9B as a downstream evaluator to benchmark video generation models according to Physical Adherence and Instruction Alignment.

Figures & tables

Project page :

https://spherelab.ai/RoboJudge

Table 1

Figure 1: Overview of RoboJudge . (A) We propose RoboJudge-Bench , which evaluates MLLMs as judges of embodied videos. (B) Physical Adherence and Instruction Alignment serve as the two core evaluation dimensions for embodied judge models. (C) We benchmark 29 existing model judges, (D) and construct RoboJudge-Train , a 12K-video dataset for judge training, and fine-tune RoboJudge-9B , which achieves the strongest overall agreement among model judges.

Figure 2: Examples from RoboJudge-Bench . Each example shows the task instruction, sampled video frames, and human ratings with supporting evidence for Physical Adherence (PA) and Instruction Alignment (IA). Left: a gripper-robot video preserves most physical properties but fails to complete the packing task (PA: 4; IA: 2). Right: a human-hand video completes the requested insertion despite object-consistency and interaction artifacts (PA: 3; IA: 5). The examples illustrate why physical plausibility and task completion require separate assessment.

Dataset	Bench Size	Training Size	Domain	Real	Sim.	Embodiments	Data Sources	Video Generators	Embodied Rubrics
VideoPhy2	3,397 (11)	3K	General	–	–	–	–	7	✗
Physion-Eval	880 (204)	11k	General	✓	✗	Human	1	5	✗
RoboGaze	382 (382)	–	Embodied	✓	✓	Dex.	3	1	✓
RoboJudge	800 (800)	12K	Embodied	✓	✓	Human&Dex.&Grip.	8	10	✓

Table 1: Comparison of representative datasets for evaluating MLLMs as world model judges. Numbers in parentheses indicate the number of embodied videos.

Panel of Figure 3 — Figure 3: Statistics of RoboJudge-Bench . (a) Key statistics of the test set. (b) Videos by embodiment (inner ring) and source corpus (outer ring). (c) Judge model ranking by pooled Overall Pearson correlation across PA and IA.

Figure 4: Qualitative failure cases of existing judges on RoboJudge-Bench . Each panel compares human and judge Physical Adherence (PA) and Instruction Alignment (IA) scores with sampled frames and judge reasoning. (A) GPT-5.5 misses duplicated stickers and extra hands. (B) MiMo-Embodied-7B misses abrupt cloth deformation and vanishing manipulators. (C) VideoScore2 credits task completion although the refrigerator door stays closed. (D) Qwen3-VL-32B reports successful placement although the bottle lands beside the target pad. These failures span entity consistency, interaction realism, action completion, and goal-state verification.

Closed-Source MLLMs
	Physical Adherence			Instruction Alignment			Overall
Judge	$r\uparrow$	Acc $\uparrow$	MSE $\downarrow$	$r\uparrow$	Acc $\uparrow$	MSE $\downarrow$	$r\uparrow$	Acc $\uparrow$	MSE $\downarrow$
Random Baseline	0.000	0.200	3.349	0.000	0.200	3.631	0.000	0.200	3.490
Gemini-3.7-Flash ( Google DeepMind, 2026c )	0.611	0.245	1.949	0.709	0.341	0.916	0.630	0.293	1.433
Gemini-3.5-Flash ( Google DeepMind, 2026b )	0.594	0.304	1.527	0.535	0.384	1.400	0.562	0.344	1.463
Gemini-3.1-Pro ( Google DeepMind, 2026a )	0.451	0.296	1.505	0.584	0.321	1.395	0.529	0.309	1.450

Table 2: Judge performance on RoboJudge-Bench .

r

is Pearson correlation with human ratings, and Acc is exact-match accuracy. Overall

r

and MSE pool PA and IA as axis-level score predictions, while Overall Acc is the average of PA and IA Acc. The best value in each column is shown in bold .

Model	PA			IA			Overall
	$r\uparrow$	Acc $\uparrow$	MSE $\downarrow$	$r\uparrow$	Acc $\uparrow$	MSE $\downarrow$	$r\uparrow$	Acc $\uparrow$	MSE $\downarrow$
Qwen3.5-9B	0.379	0.223	3.011	0.461	0.347	2.314	0.401	0.285	2.663
+ SFT	0.624	0.405	0.913	0.766	0.471	0.671	0.700	0.438	0.792
+ SFT + RL (RoboJudge-9B)	0.645	0.434	0.806	0.775	0.512	0.688	0.719	0.473	0.747

Table 3: Training-stage ablation of RoboJudge-9B . Acc is exact-match accuracy.

Model	PC $r$ $\uparrow$	SA $r$ $\uparrow$	$J$ $\uparrow$
	VideoPhy2		Physion-Eval
Qwen3.5-9B	0.179	0.412	0.056
+ SFT	0.299	0.552	0.105
+ SFT + RL (Ours)	0.346	0.572	0.093

Table 4: OOD evaluation on VideoPhy2 and Physion-Eval. Training on RoboJudge-Train improves performance across all evaluated OOD metrics over the Qwen3.5-9B base model.

J

is Youden’s

J

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Annotation interface. Annotators score each generated video on physical adherence and instruction alignment, complete the corresponding diagnostic checklists, and write evidence for their decisions.

Round	Exact	Within $\bm{\pm 1}$	Mean abs. diff.
1	32.8%	76.2%	0.98
2	39.0%	78.1%	0.97
3	38.3%	81.2%	0.85
4	52.9%	92.1%	0.58

Appendix

Table 5: Annotator alignment across four training rounds, each based on 100 practice videos. Metrics aggregate pairwise comparisons of annotators’ primary PA and IA scores.

Video Generation Model / Source	Train	Test
CogVideo I2V	2,099	96
GigaWorld	1,550	105
WoW Wan14B	1,533	122
ABOT PhysWorld	1,473	94
Cosmos3-Nano	1,033	107
Wan2.2 TI2V-5B	971	110

Appendix

Table 6: Video sources of RoboJudge-Train and RoboJudge-Bench . The training set contains 11,520 generated videos from 14 generators and 831 verified successful real demonstrations paired with their corresponding instructions; the latter are assigned PA=5 and IA=5. The test set contains 800 videos from 10 of these generators; “–” indicates a generator not included in the test set.

Figure 8: Directional scoring bias relative to human ratings. RoboJudge is balanced around human scores, while baselines tend to over-score or under-score.

Metric	Strongest baseline	$\bm{\Delta r}$	95% CI
PA	Gemini-3.7-Flash	$+0.034$	$[-0.011,\ 0.078]$
IA	GPT-5.5	$+0.044$	$[0.010,\ 0.078]$
Overall	Gemini-3.7-Flash	$+0.089$	$[0.061,\ 0.119]$

Appendix

Table 7: Paired cluster-bootstrap comparisons between RoboJudge-9B and the strongest model baseline for each metric.

\Delta r

denotes RoboJudge-9B minus the comparison model.

Video Generation Model	PA $\uparrow$	IA $\uparrow$	Overall $\uparrow$
HappyHorse	3.821	4.313	4.067
Wan2.6 Flash	3.827	4.027	3.927
PF Wan14B	3.680	3.860	3.770
Vidu Q3	3.480	3.940	3.710
Kling2.2	3.420	3.940	3.680
Veo 3.1 Lite	3.580	3.780	3.680

Appendix

Table 8: RoboJudge-9B evaluation results on generated videos. PA denotes Physical Adherence, IA denotes Instruction Alignment, and Overall is the average of PA and IA.

Figure 9: Judge agreement with human ratings on RoboJudge-Bench .

Judge	FPS	PA			IA			Overall
Judge	FPS	$r\uparrow$	Acc $\uparrow$	MSE $\downarrow$	$r\uparrow$	Acc $\uparrow$	MSE $\downarrow$	$r\uparrow$	Acc $\uparrow$	MSE $\downarrow$
GPT-5.5	2	0.483	0.344	1.339	0.735	0.484	0.792	0.620	0.414	1.066
	4	0.489	0.334	1.359	0.733	0.481	0.786	0.618	0.407	1.073
	8	0.483	0.351	1.335	0.738	0.500	0.775	0.620	0.426	1.055
Qwen3.5-9B	2	0.316	0.214	3.422	0.429	0.355	2.522	0.363	0.285	2.972
	4	0.379	0.223	3.011	0.461	0.347	2.314	0.401	0.285	2.663

Appendix

Table 9: Frame-rate (FPS) ablation on RoboJudge-Bench . Acc is exact-match accuracy. The 4 FPS rows reuse the corresponding main-table results because 4 FPS is the default evaluation setting.

Vision-Language Model-as-a-Judge

Vision-language model-as-a-judge uses multimodal models to assess images, video, and visual-agent outputs against instructions, rubrics, or human preferences.

See all 59 papers

VibeVideoBench:Evaluating Multimodal Agents on Code-Driven Video Generation
Oct 26, 2026Muxi Diao, Wenbin Xu, Lele Yang +21Video Generation Evaluation Multimodal Agent Evaluation
RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis
Jun 22, 2026Minh-Loi Nguyen, Nghiem T. Diep, Hung K. Nguyen +13VLM Evaluation Robot Manipulation Benchmarks
Center for AI Research, VinUniversity · VinRobotics · DFKI +3
MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation
Dec 16, 2025Zefan Cai, Haoyi Qiu, Tianyi Ma +13Embodied Navigation Multimodal Reasoning
University of Wisconsin–Madison · University of California, Los Angeles · Michigan State University +8

Oct 26, 2026J/K move · Enter open · S save

VibeVideoBench:Evaluating Multimodal Agents on Code-Driven Video Generation

Muxi Diao, Wenbin Xu, Lele Yang, Xuyang Ma+20

Video Generation Evaluation Multimodal Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

Diffusion video models produce striking clips, but they cannot reliably render an exact formula, a correct chart, a ruleconsistent game or narration synchronized to on-screen events. A different route is emerging: multimodal agents that write code to render every frame and synthesize every sound. We introduce VibeVideoBench, a benchmark for evaluating agents on code-driven video generation. Its 239 self-contained tasks in English and Chinese span 12 categories, from data stories and music-driven pieces to classroom lessons and paper explainers; each asks an agent, running in its native harness inside a sandbox, to deliver a finished MP4 with sound within one hour. Evaluation is hierarchical and evidence-based: program checks of the delivery spec, about 13k task-specific rubric checks traced to the requirements they enforce, and five quality dimensions anchored per task, judged by three VLM judge panels and validated with blind human preferences. Across nine frontier agent systems and 2,151 scored rollouts, the benchmark is far from solved: GPT-6 Astra and Claude Opus 5.5 lead clearly, yet even the best system fails a core requirement on about a quarter of tasks. Whether a video arrives at all matters as much as its quality, and three quarters of the lost videos are final renders started too late, run too slowly or never started. The judges agree with each other and with human preferences on system rankings, but only moderately on individual videos.

Abstract

Vision-Language Model-as-a-Judge

Explore similar work

VibeVideoBench:Evaluating Multimodal Agents on Code-Driven Video Generation

RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis

MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation