RoboJudge: Multimodal Language Models as Judges for Embodied Video Generation

Siyuan Ma; Yu Qi; Cheng Ye; Ruotian Peng; Xinyi Xu; Zitiantao Lin; Zedong Cai; Yijian Huang

cs.CVOct 20, 2026

RoboJudge: Multimodal Language Models as Judges for Embodied Video Generation

Authors: Siyuan Ma, Yu Qi, Cheng Ye, Ruotian Peng, Xinyi Xu, Zitiantao Lin, Zedong Cai, Yijian Huang,

Organizations: The Chinese University of Hong Kong · Northeastern University · University of California, San Diego · Westlake University ·

Abstract

Video generation models are promising foundations for embodied world models, but only if their generated rollouts are physically valid and faithful to task instructions. Multimodal large language models (MLLMs) are increasingly used to judge these properties, yet their agreement with human judgments on embodied videos has not been systematically evaluated. To address this gap, we introduce RoboJudge-Bench, a benchmark for evaluating MLLMs as judges of embodied video generation. The benchmark measures two fundamental dimensions, Physical Adherence and Instruction Alignment, across 800 human-annotated videos from 8 source corpora and 10 video generators, spanning grippers, human hands, and dexterous hands. Experiments reveal uneven alignment between existing judges and human ratings: general-purpose MLLMs show lower agreement on physical adherence, while specialized video judges designed for general-domain physical reasoning do not consistently transfer to embodied scenarios. Motivated by these findings, we construct RoboJudge-Train, comprising 12K videos annotated with 25K human-verified scores and model-assisted rationales for Physical Adherence and Instruction Alignment. We fine-tune a 9B MLLM on this dataset with supervised learning followed by reinforcement learning, yielding RoboJudge-9B. Among the model judges evaluated on RoboJudge-Bench, RoboJudge-9B achieves the highest overall agreement, reaching a Pearson correlation of 0.719 with a statistically significant 0.089 gain over Gemini-3.7-Flash, the best-performing proprietary baseline in our evaluation. Training on RoboJudge-Train also yields gains on external benchmarks, including VideoPhy2 and Physion-Eval. We further use RoboJudge-9B as a downstream evaluator to benchmark video generation models according to Physical Adherence and Instruction Alignment.

Figures & tables

Project page :

https://spherelab.ai/RoboJudge

Table 1

Figure 1: Overview of RoboJudge . (A) We propose RoboJudge-Bench , which evaluates MLLMs as judges of embodied videos. (B) Physical Adherence and Instruction Alignment serve as the two core evaluation dimensions for embodied judge models. (C) We benchmark 29 existing model judges, (D) and construct RoboJudge-Train , a 12K-video dataset for judge training, and fine-tune RoboJudge-9B , which achieves the strongest overall agreement among model judges.

Figure 2: Examples from RoboJudge-Bench . Each example shows the task instruction, sampled video frames, and human ratings with supporting evidence for Physical Adherence (PA) and Instruction Alignment (IA). Left: a gripper-robot video preserves most physical properties but fails to complete the packing task (PA: 4; IA: 2). Right: a human-hand video completes the requested insertion despite object-consistency and interaction artifacts (PA: 3; IA: 5). The examples illustrate why physical plausibility and task completion require separate assessment.

Dataset	Bench Size	Training Size	Domain	Real	Sim.	Embodiments	Data Sources	Video Generators	Embodied Rubrics
VideoPhy2	3,397 (11)	3K	General	–	–	–	–	7	✗
Physion-Eval	880 (204)	11k	General	✓	✗	Human	1	5	✗
RoboGaze	382 (382)	–	Embodied	✓	✓	Dex.	3	1	✓
RoboJudge	800 (800)	12K	Embodied	✓	✓	Human&Dex.&Grip.	8	10	✓

Table 1: Comparison of representative datasets for evaluating MLLMs as world model judges. Numbers in parentheses indicate the number of embodied videos.

Panel of Figure 3 — Figure 3: Statistics of RoboJudge-Bench . (a) Key statistics of the test set. (b) Videos by embodiment (inner ring) and source corpus (outer ring). (c) Judge model ranking by pooled Overall Pearson correlation across PA and IA.

Figure 4: Qualitative failure cases of existing judges on RoboJudge-Bench . Each panel compares human and judge Physical Adherence (PA) and Instruction Alignment (IA) scores with sampled frames and judge reasoning. (A) GPT-5.5 misses duplicated stickers and extra hands. (B) MiMo-Embodied-7B misses abrupt cloth deformation and vanishing manipulators. (C) VideoScore2 credits task completion although the refrigerator door stays closed. (D) Qwen3-VL-32B reports successful placement although the bottle lands beside the target pad. These failures span entity consistency, interaction realism, action completion, and goal-state verification.

Closed-Source MLLMs
	Physical Adherence			Instruction Alignment			Overall
Judge	$r\uparrow$	Acc $\uparrow$	MSE $\downarrow$	$r\uparrow$	Acc $\uparrow$	MSE $\downarrow$	$r\uparrow$	Acc $\uparrow$	MSE $\downarrow$
Random Baseline	0.000	0.200	3.349	0.000	0.200	3.631	0.000	0.200	3.490
Gemini-3.7-Flash ( Google DeepMind, 2026c )	0.611	0.245	1.949	0.709	0.341	0.916	0.630	0.293	1.433
Gemini-3.5-Flash ( Google DeepMind, 2026b )	0.594	0.304	1.527	0.535	0.384	1.400	0.562	0.344	1.463
Gemini-3.1-Pro ( Google DeepMind, 2026a )	0.451	0.296	1.505	0.584	0.321	1.395	0.529	0.309	1.450

Table 2: Judge performance on RoboJudge-Bench .

r

is Pearson correlation with human ratings, and Acc is exact-match accuracy. Overall

r

and MSE pool PA and IA as axis-level score predictions, while Overall Acc is the average of PA and IA Acc. The best value in each column is shown in bold .

Model	PA			IA			Overall
	$r\uparrow$	Acc $\uparrow$	MSE $\downarrow$	$r\uparrow$	Acc $\uparrow$	MSE $\downarrow$	$r\uparrow$	Acc $\uparrow$	MSE $\downarrow$
Qwen3.5-9B	0.379	0.223	3.011	0.461	0.347	2.314	0.401	0.285	2.663
+ SFT	0.624	0.405	0.913	0.766	0.471	0.671	0.700	0.438	0.792
+ SFT + RL (RoboJudge-9B)	0.645	0.434	0.806	0.775	0.512	0.688	0.719	0.473	0.747

Table 3: Training-stage ablation of RoboJudge-9B . Acc is exact-match accuracy.

Model	PC $r$ $\uparrow$	SA $r$ $\uparrow$	$J$ $\uparrow$
	VideoPhy2		Physion-Eval
Qwen3.5-9B	0.179	0.412	0.056
+ SFT	0.299	0.552	0.105
+ SFT + RL (Ours)	0.346	0.572	0.093

Table 4: OOD evaluation on VideoPhy2 and Physion-Eval. Training on RoboJudge-Train improves performance across all evaluated OOD metrics over the Qwen3.5-9B base model.

J

is Youden’s

J

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Annotation interface. Annotators score each generated video on physical adherence and instruction alignment, complete the corresponding diagnostic checklists, and write evidence for their decisions.

Round	Exact	Within $\bm{\pm 1}$	Mean abs. diff.
1	32.8%	76.2%	0.98
2	39.0%	78.1%	0.97
3	38.3%	81.2%	0.85
4	52.9%	92.1%	0.58

Appendix

Table 5: Annotator alignment across four training rounds, each based on 100 practice videos. Metrics aggregate pairwise comparisons of annotators’ primary PA and IA scores.

Video Generation Model / Source	Train	Test
CogVideo I2V	2,099	96
GigaWorld	1,550	105
WoW Wan14B	1,533	122
ABOT PhysWorld	1,473	94
Cosmos3-Nano	1,033	107
Wan2.2 TI2V-5B	971	110

Appendix

Table 6: Video sources of RoboJudge-Train and RoboJudge-Bench . The training set contains 11,520 generated videos from 14 generators and 831 verified successful real demonstrations paired with their corresponding instructions; the latter are assigned PA=5 and IA=5. The test set contains 800 videos from 10 of these generators; “–” indicates a generator not included in the test set.

Figure 8: Directional scoring bias relative to human ratings. RoboJudge is balanced around human scores, while baselines tend to over-score or under-score.

Metric	Strongest baseline	$\bm{\Delta r}$	95% CI
PA	Gemini-3.7-Flash	$+0.034$	$[-0.011,\ 0.078]$
IA	GPT-5.5	$+0.044$	$[0.010,\ 0.078]$
Overall	Gemini-3.7-Flash	$+0.089$	$[0.061,\ 0.119]$

Appendix

Table 7: Paired cluster-bootstrap comparisons between RoboJudge-9B and the strongest model baseline for each metric.

\Delta r

denotes RoboJudge-9B minus the comparison model.

Video Generation Model	PA $\uparrow$	IA $\uparrow$	Overall $\uparrow$
HappyHorse	3.821	4.313	4.067
Wan2.6 Flash	3.827	4.027	3.927
PF Wan14B	3.680	3.860	3.770
Vidu Q3	3.480	3.940	3.710
Kling2.2	3.420	3.940	3.680
Veo 3.1 Lite	3.580	3.780	3.680

Appendix

Table 8: RoboJudge-9B evaluation results on generated videos. PA denotes Physical Adherence, IA denotes Instruction Alignment, and Overall is the average of PA and IA.

Figure 9: Judge agreement with human ratings on RoboJudge-Bench .

Judge	FPS	PA			IA			Overall
Judge	FPS	$r\uparrow$	Acc $\uparrow$	MSE $\downarrow$	$r\uparrow$	Acc $\uparrow$	MSE $\downarrow$	$r\uparrow$	Acc $\uparrow$	MSE $\downarrow$
GPT-5.5	2	0.483	0.344	1.339	0.735	0.484	0.792	0.620	0.414	1.066
	4	0.489	0.334	1.359	0.733	0.481	0.786	0.618	0.407	1.073
	8	0.483	0.351	1.335	0.738	0.500	0.775	0.620	0.426	1.055
Qwen3.5-9B	2	0.316	0.214	3.422	0.429	0.355	2.522	0.363	0.285	2.972
	4	0.379	0.223	3.011	0.461	0.347	2.314	0.401	0.285	2.663

Appendix

Table 9: Frame-rate (FPS) ablation on RoboJudge-Bench . Acc is exact-match accuracy. The 4 FPS rows reuse the corresponding main-table results because 4 FPS is the default evaluation setting.

Vision-Language Model-as-a-Judge

Vision-language model-as-a-judge uses multimodal models to assess images, video, and visual-agent outputs against instructions, rubrics, or human preferences.

See all 59 papers

Oct 26, 2026

VibeVideoBench:Evaluating Multimodal Agents on Code-Driven Video Generation

Diffusion video models produce striking clips, but they cannot reliably render an exact formula, a correct chart, a ruleconsistent game or narration synchronized to on-screen events. A different route is emerging: multimodal agents that write code to render every frame and synthesize every sound. We introduce VibeVideoBench, a benchmark for evaluating agents on code-driven video generation. Its 239 self-contained tasks in English and Chinese span 12 categories, from data stories and music-driven pieces to classroom lessons and paper explainers; each asks an agent, running in its native harness inside a sandbox, to deliver a finished MP4 with sound within one hour. Evaluation is hierarchical and evidence-based: program checks of the delivery spec, about 13k task-specific rubric checks traced to the requirements they enforce, and five quality dimensions anchored per task, judged by three VLM judge panels and validated with blind human preferences. Across nine frontier agent systems and 2,151 scored rollouts, the benchmark is far from solved: GPT-6 Astra and Claude Opus 5.5 lead clearly, yet even the best system fails a core requirement on about a quarter of tasks. Whether a video arrives at all matters as much as its quality, and three quarters of the lost videos are final renders started too late, run too slowly or never started. The judges agree with each other and with human preferences on system rankings, but only moderately on individual videos.

Muxi Diao, Wenbin Xu, Lele Yang +21

Video Generation Evaluation Multimodal Agent Evaluation AI Agent Benchmarks

Ask PDF

Jun 22, 2026

RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis

Robot world models now generate synthetic manipulation videos, but evaluating them is hard: realistic-looking outputs often violate physical laws, temporal consistency or task logic, while conventional metrics and monolithic Vision-Language Model (VLM) judges return scores that do not say what failed. We present RoboGaze, a training-free multi-agent VLM evaluator that grounds the task and scene, routes each subtask to dimension-specific specialists and verifies their findings with a critic, producing temporally localized diagnostic reports that state what failed, when, why and how severely, under a schema of 6 dimensions and 30 failure types whose annotation reliability and coverage we measure. To benchmark it, we introduce RoboGazeBench, 914 human-annotated clips from five generators, conditioned on simulated and real-world data and running from 4.9 to 148.9 s. Across eight open-source and proprietary VLM backbones, RoboGaze outperforms every baseline; even a published agentic detector given the same schema and a slightly larger inference budget trails it by 10.1-12.3 description-F1 points, a margin that blinded human raters confirm. On the strongest backbone it closes 83% of the gap between chain-of-thought prompting and human agreement, and its critic curbs the "cry-wolf" false positives of standard VLMs, lifting mean clean-clip accuracy from 17% to 85%. The advantage is largest on held-out generators and long clips, and carries over to selecting generated demonstrations (80% vs. 71% precision).

Minh-Loi Nguyen, Nghiem T. Diep, Hung K. Nguyen +13

Center for AI Research, VinUniversity · VinRobotics · DFKI +3

VLM Evaluation Robot Manipulation Benchmarks World Model Evaluation Temporal Consistency in Video Generation

Ask PDF

Dec 16, 2025

MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation

Modern multimodal generative models can synthesize visually compelling images and videos, but it remains unclear whether this visual fluency reflects genuine reasoning: when prompted to generate a solution, can a model preserve the physical, logical, spatial, and temporal constraints a task requires, or does it merely produce plausible-looking media? To answer this question, we introduce MMGR (Multi-Modal Generative Reasoning Benchmark and Evaluation), a benchmark for evaluating generative reasoning across video, image, and language-based systems. MMGR covers 10 tasks from three domains (Abstract Reasoning, Embodied Navigation, and Physical Commonsense) and probes five reasoning abilities: Physical, Logical, 2D Spatial, 3D Spatial, and Temporal. Its evaluation emphasizes answer-verifiable tasks and, for video generation, process-aware chain-of-frame reasoning, where intermediate frames must form valid steps toward the target outcome rather than visually smooth but incorrect transitions. Evaluating state-of-the-art video generators, image generators, and LLM/VLM baselines reveals a sharp gap between visual quality and reasoning correctness: video models perform best on Physical Commonsense, but remain weak on symbolic tasks such as Sudoku, ARC, and Math, and brittle in cross-view embodied navigation. Image generators often outperform video generators on embodied navigation despite lacking temporal outputs, showing that longer visual generation does not automatically yield stronger reasoning. MMGR reframes evaluation of multimodal generation from whether outputs look realistic to whether they solve the underlying reasoning problem.

Zefan Cai, Haoyi Qiu, Tianyi Ma +13

University of Wisconsin–Madison · University of California, Los Angeles · Michigan State University +8

Embodied Navigation Multimodal Reasoning Multimodal Model Evaluation Video Generation Evaluation

Ask PDF

Abstract

Vision-Language Model-as-a-Judge

Explore similar work

VibeVideoBench:Evaluating Multimodal Agents on Code-Driven Video Generation

RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis

MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation