RoboWorM: Evaluating Movement Realism, Consistency and Acceleration Affects in Embodied Video World Models

Doanh Le Thien; Viet-Thanh Nguyen; Pham Tri Quang; Nhi Pham; Christopher Wewer; Minh-Loi Nguyen; Hoang H. Tran; Tho Le

cs.ROOct 24, 2026

RoboWorM: Evaluating Movement Realism, Consistency and Acceleration Affects in Embodied Video World Models

Authors: Doanh Le Thien, Viet-Thanh Nguyen, Pham Tri Quang, Nhi Pham, Christopher Wewer, Minh-Loi Nguyen, Hoang H. Tran, Tho Le,

Organizations: Center for AI Research, VinUniversity · VinRobotics · MPI-INF · KAIST ·

Abstract

Video world models (VWMs) are rapidly emerging as scalable neural simulators for robotic manipulation. However, most benchmarks only evaluate them as open-world video generators, using Vision-Language Models (VLMs) that lack strict 3D spatial and kinematic reasoning. To bridge this gap, we introduce RoboWorM, a comprehensive, deployment-aware benchmark that leverages known hardware priors such as robot kinematic trees and fixed camera configurations, to systematically examine structural and spatiotemporal fidelity. RoboWorM curates 1,605 episodes across single-arm, bimanual, and humanoid morphologies, spanning both Markovian and long-horizon non-Markovian tasks under single- and multi-view setups. To explicitly measure physical consistency, we propose two deterministic, VLM-free metrics: KIVA for intra-view kinematic stability, and RIGIS for multi-view geometric consistency. By exploiting intrinsic robotic invariants, KIVA improves hallucination detection alignment with human annotations by 25.8% relative to the strongest frontier VLMs, while RIGIS achieves a 130% higher Spearman correlation with human preferences. Furthermore, since real-world deployment requires low inference latency, we utilize RoboWorM to conduct the first large-scale audit of the quality-efficiency trade-off. Evaluating 9 state-of-the-art acceleration techniques across 3 leading world models (over 29,000 generated rollouts), we demonstrate how these acceleration methods can retain superficial visual quality while silently compromising deployment-critical physical constraints. Using our metrics, we achieve 2.3 $\times$ maximum inference speed-up with minimal quality loss.

Figures & tables

	Data Diversity					Evaluation Dimensions
Benchmark	# Samples	Data Sources	Robot Morphologies	Multi-View Setup	Long-Horizon Tasks	Evaluator Paradigm	Cross-View Geometry	Speedup Audit
General Video Evaluation
VBench	800	N/A	N/A	✗	✗	Hybrid	✗	✗
VBench2.0	1260	N/A	N/A	✓	✗	Hybrid	✓	✗
VLM-Based Robotics Evaluation
WorldModelBench	350	3	1	✗	✗	Hybrid	✗	✗

Table 1: Comparison of Video World Model Evaluation Benchmarks . RoboWorM derives deterministic, deployment-aware metrics from dataset-native hardware priors , scaling across multi-view setups, complex morphologies, and long-horizon tasks while auditing cross-view geometry and inference acceleration. Prior benchmarks rely on external VLMs or off-the-shelf models that do not exploit priors such as camera calibration, robot proprioception, and actions. Off. Alg. denotes off-the-shelf algorithms; Hybrid combines Off. Alg. and VLM ; IK / Sim denotes Inverse Kinematics and Simulation; “ Prior Alg. ” denotes deterministic algorithms driven by dataset priors.

Figure 2: Benchmark Composition and Statistical Breakdown of RoboWorM . Data diversity spans a balanced mix of Markovian (short-horizon) and non-Markovian (long-horizon) task dynamics, task skills, and an even distribution across three robot morphologies. We benchmark 9 state-of-the-art acceleration methods on 29,000 $+$ generated rollouts from three leading world models.

Figure 3: We introduce KIVA for intra-view kinematics stability and RIGIS for multi-view consistency in deployment-aware setting. (a) KIVA. A shared video-conditioned reader learns from forward-kinematics targets using Huber loss. At inference, five denoised candidates are averaged before measuring rigidity and jerk, which are then calibrated against real-video references. The hand examples and numerical values, including the reference thresholds, are illustrative rather than empirical results. (b) RIGIS. Multi-view inputs are processed via VGGT- $\Omega$ to evaluate geometric consistency $S_{\mathrm{geo}}$ and camera pose stability $S_{\mathrm{pose}}$ , yielding the unified metric $S$ .

Figure 4: Our metrics KIVA (top) and RIGIS (bottom) shows significantly higher human alignment correlation and accuracy, while achieving better processing latency.

Figure 5: Acceleration mechanisms exhibit distinct speed–quality trade-offs. Caching achieves the largest speedups with near-zero degradation; token pruning degrades most. Bars show speedup, the middle axis physical audit, and dots mean NR. See Fig. C3 , C2 for other embodiments.

Figure 6: RoboWorM identifies optimal speedup configurations without quality trade-offs where standard metrics fail . (a) KIVA and RIGIS accurately evaluate embodied hallucinations and geometry. (b) RoboWorM systematically profiles the speedup-quality frontier. (c) Correlation between KIVA degradation and increased video chunk across Markovian and Non-Markovian tasks.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Embodiment	Track	Regime	Robot	Views	FPS	Resolution	Eps.	Pool	Instr.
Single-arm	single	Mk.	RoboChallenge	1	15	640 $\times$ 480	120	120	2
Single-arm	single	Non-mk.	RoboChallenge	1	15	640 $\times$ 480	120	120	2
Single-arm	multi	Mk.	Franka	3	15	320 $\times$ 180	128	95,600	126
Single-arm	multi	Non-mk.	Franka	3	15	320 $\times$ 180	129	95,600	128
Bimanual	single	Mk.	ALOHA	1	30	640 $\times$ 480	150	150	34
Bimanual	single	Non-mk.	ALOHA	1	30	640 $\times$ 480	150	150	17

Appendix

Table A1: Composition of the twelve RoboWorM evaluation settings, recomputed from the source metadata. Eps. is the number of episodes we evaluate and Pool the number available upstream; Views is the synchronised cameras per episode and Instr. the distinct instructions. Frame rate, resolution and view count are reported as found in the source, before the per-generator resampling and grid composition of Appendix A.2 . Bimanual single-view is the head camera of the same episodes as bimanual multi-view, hence the identical counts.

Figure A1: Instruction and horizon diversity. (a) The

509

distinct instructions of RoboWorM , grouped by leading verb into the

14

manipulation primitives of Appendix A.3 . The long tail is the point: pick & place alone covers

306

instructions while six primitives have fewer than

10

, so the taxonomy documents breadth and must not be read as a balanced factor. (b) Episode duration per embodiment over all

1{,}608

evaluation conditions, computed as decoded frame count divided by source frame rate (

n

annotated per embodiment). Boxes span the interquartile range and outliers are hidden, so the

0.2

–

142

s span in the panel title is wider than the whiskers. Horizons vary by an order of magnitude within every embodiment, which is why every clip is generated to its own ground-truth length (Appendix A.2 ) and all metrics are computed per episode rather than over a fixed-length window.

Embodiment	Regime	Instruction	Eps.
Bimanual	Markovian	close cardboard box	1
		close toolbox	7
		find hole and insert into gear	4
		find insert large gear shaft	6
		find insert small gear shaft	4
		fit large gear shaft	1

Appendix

Table A2: Complete instruction set for benchmark settings whose instructions are defined at the task level, quoted verbatim from the source corpus. The two bimanual camera tracks share the same tasks and are therefore listed once. The remaining settings (DROID, RoboChallenge, and GR1) use episode-specific instructions and are summarized in Table A1 .

Figure A2: Distribution of generated rollouts. (a) Dense rollouts across embodiment, camera track, and world model, with each cell divided into Markovian and non-Markovian subsets.Hatched settings indicate model–camera combinations that are not supported: Ctrl-World is evaluated on multi-view tracks, DreamDojo on single-view tracks, and DreamGen on all six tracks, resulting in

12

valid configurations. (b) Coverage of the nine acceleration methods over the

12

configurations. Crosses indicate unsupported backbone–method combinations.

Figure B1: What Visual Integrity reacts to. A low- and high-scoring clip are shown with five evenly spaced frames each; the value on the left is the score reported in the tables. Both examples are from DreamGen humanoid (low: ITM, static ; high: dense, head ). Because this metric fails the ground-truth-is-ceiling audit, it should not be used to rank models. The comparison is therefore intended only to illustrate the type of visual degradation captured by the metric.

Figure B2: What Text Alignment reacts to. A low- and high-scoring clip are shown, each evaluated against its own task clause printed below the frame strip. The two rows are not directly comparable because CLIPScore exhibits substantial caption-dependent bias. Accordingly, all ranking claims for this metric use paired deltas under the same prompt. Both examples are humanoid static views (low: Ctrl-World 4-view with SiTo; high: DreamGen dense).

Figure B3: FVD sensitivity to sample size. (a) Dense settings re-scored against the same reference while varying the number of windows

n

. (b) Matching

n

changes the ranking of

14/24

settings, indicating substantial estimator sensitivity at small sample sizes.

Figure C1: Effect of finetuning on matched episodes. Ground truth, unfine-tuned, and finetuned rollouts are shown at matched time points with all camera views. The unfine-tuned model retains plausible appearance but progressively loses task-relevant structure, including object position and cross-view consistency. Finetuning mitigates these failures and therefore defines the dense reference used for quality-retention evaluation. Reported PSNR values correspond to the full episode.

Model	Cell	Size	Adaptation	Steps	$g$	Seed	$n$
Ctrl-World	single-arm MV	1.5B	Released	50	1.0	unseeded	257
	bimanual MV	1.5B	Fine-tuned	50	1.0	unseeded	300
	bimanual MV (4-view)	1.5B	Fine-tuned	50	1.0	unseeded	300
	humanoid MV	1.5B	Fine-tuned	50	1.0	unseeded	248
	humanoid MV (4-view)	1.5B	Fine-tuned	50	1.0	unseeded	248
DreamGen	single-arm MV	14B	Released	35	7	0	92

Appendix

Table C1: Generation configurations of the

14

dense benchmark settings. Each row defines the dense baseline against which acceleration methods in the corresponding cell are evaluated. Size denotes the backbone parameter count, Adaptation indicates whether the model is finetuned for the corresponding robot setting or used from a released checkpoint, Steps is the number of denoising steps,

g

is the guidance scale, and

n

is the number of released rollouts. Seed handling differs across model families: Ctrl-World is unseeded, DreamGen uses seed

0

(offset by segment index for autoregressive multi-view rollouts), and DreamDojo uses a per-chunk seed.

Corpus	Stage	fps	Clips	Usable	% usable	Median fr.
humanoid MV	$2{\times}2$ grid stitch	$10$	$248$	$132$	$53$	$97$
	segment	$10$	$416$	$300$	$72$	$100$
	re-encode 16 fps	$16$	$373$	$373$	$100$	$160$
bimanual MV	$2{\times}2$ grid stitch	$30$	$300$	$300$	$100$	$1000$
	segment	$30$	$1694$	$1694$	$100$	$300$
	re-encode 16 fps	$16$	$1694$	$1694$	$100$	$162$

Appendix

Table C2: Training corpus construction for the finetuned DreamGen checkpoints. Each block traces a released checkpoint through its preprocessing pipeline. Clips denotes generated video files, while usable counts those with at least the

93

frames required for training. The humanoid corpus retains only

132/248

usable clips at

10

fps, whereas the bimanual corpus retains all clips at

30

fps. Overlapping

10

s windows with a

5

s stride and re-encoding to

16

fps increase usable samples by

2.8\times

and

5.6\times

, respectively, while matching the base checkpoint frame rate.

Configuration	Ctrl-World	DreamGen	DreamDojo
Model
Backbone	SVD image-to-video U-Net	Cosmos-Predict2 2B / 14B DiT	Cosmos-Predict2 2B DiT
Adaptation	Full finetuning	Full finetuning	Full finetuning
Conditioning	12-D actions + embodiment ID	Text prompt	Latent actions
Optimization
Optimizer	AdamW	AdamW	FusedAdamW

Appendix

Table C3: Finetuning configurations of the three base world models. We retain each model family’s original training recipe rather than enforcing matched hyperparameters across architectures. All adaptations use full-model finetuning without parameter-efficient adapters.

			Depth $C$		KIVA (%)
Embodiment	Model	$\Delta$	Mak.	Non-mak.	Mak.	Non-mak.	$\Delta$
Humanoid	DreamGen	92	3 [2–3]	2 [2–3]	4.65	12.30	$+7.65$
Humanoid	DreamDojo	49	2 [1–14]	2 [1–29]	13.04	3.31	$-9.73$
Bimanual	DreamDojo	12	34 [10–124]	41 [20–124]	6.11	0.00	$-6.11$
Bimanual	DreamGen	92	7 [2–18]	11 [3–18]	23.63	23.18	$-0.45$
Single-arm	DreamGen	92	3 [2–3]	5 [4–7]	59.40	70.65	$+11.25$

Appendix

Table C4: Median rollout depth

C

per clip (range in brackets) against KIVA, the percentage of invalid

16

-frame segments, ordered by the non-Markovian/Markovian depth ratio.

Figure C2: Acceleration trade-offs on humanoid morphology. Constructed as in Fig. 5 : bars show speedup over the corresponding dense rollout, the middle axis reports the view-specific physical audit, and boxed values denote aggregate retention

Q

(

Q=1

for dense). On multi-view Ctrl-World, all sparse-attention methods reduce RIGIS from

0.879

0.680

–

0.686

, whereas caching retains

0.875

–

0.878

and token pruning

0.822

–

0.835

. On single-view DreamDojo, all accelerated variants improve KIVA relative to dense (

13.0\%

), with FasterCache reaching

Q=1.13

Figure C3: Acceleration trade-offs on single-arm morphology. Constructed as in Fig. 5 ; see Fig. C2 for the companion embodiment. On multi-view, ITM produces the largest geometry degradation, reducing RIGIS to

0.810

on DreamGen and

0.721

on Ctrl-World, compared with

0.838

–

0.915

for the remaining methods. On single-view, the dense baseline already exhibits high KIVA (

59.4\%

versus

10.1\%

for real video), so the

55

–

80\%

range observed under acceleration should not be interpreted as acceleration-induced degradation. Missing entries indicate unmeasured configurations rather than zero.

Table D1: Data distribution of metric evaluation sets stratified by robot embodiment and task dynamics.

Panel of Figure 22 — Figure D1: Human evaluation interfaces. The web-based annotation dashboard showing tailored interfaces for multi-view consistency (side-by-side synchronized views with video-level scoring) and embodied physics (single-camera clips segmented into 3-second intervals with pause-and-prompt checks). Instructions, scoring rubrics, and reference examples remain persistently pinned on-screen to guide raters during evaluation.

Panel of Figure 23 — Figure D2: Human evaluation analysis. (a) Embodied physics confusion matrix. (b) Multi-view consistency confusion matrix. (c) Exact agreement rate between two raters across evaluation criteria.

Setting	Value
API backend	OpenRouter
Evaluated Models	Closed-source: Gemini 3.6 Flash, Qwen 3.6-Plus Open-source: Gemma 4 31B, Qwen 3.6-35B-A3B
Prompt variants	Vanilla; +Chain-of-Thought
Turns per video	3
Aggregation	Majority vote over turns
Score scale	{Bad , Medium , Good}

Appendix

Table D2: VLM judge configuration and model variants.

Method	Embodied Physics			Multi-view Consistency
Method	Agree (%) $\uparrow$	Score Std $\downarrow$	Latency $\downarrow$	Agree (%) $\uparrow$	Score Std $\downarrow$	Latency $\downarrow$
Vanilla Zero-Shot
Gemini 3.6 Flash	87.2	0.093	2.781	63.0	0.240	0.784
Qwen 3.6 Plus	90.3	0.076	4.396	77.0	0.191	3.121
Gemma 4 31B	84.9	0.097	2.000	57.7	0.291	1.054
Qwen 3.6 35B	80.7	0.138	0.750	38.7	0.471	1.154

Appendix

Table D3: Agreement rate, score standard deviation, and processing latency of VLM judges. Agree denotes the percentage of videos where a model yields identical scores across evaluation turns, and Score Std measures the average score standard deviation per video. Latency is measured as the average time in seconds required to process one video second. The bottom row reports the latency of our proposed RIGIS and KIVA metrics evaluated under identical settings.

Figure D3: KIVA scores against human ratings. Each panel shows KIVA’s score (log scale, lower is better) for every labelled 16-frame segment, grouped by the human label. Boxes span the quartiles and whiskers the 5th to 95th percentile of all segments at that level; dots are 40 random segments per level. Dashed lines are the calibrated band cuts. Each panel reports Spearman’s $\rho$ on the continuous score and the calibrated overlap.

(a) Spearman correlation ( $\rho\uparrow$ )
Method	Strategy	Bimanual	Humanoid	Single Arm	Macro	Micro
Gemini 3.6 Flash	Vanilla	0.465	0.396	0.565	0.475	0.480
Qwen 3.6 Plus		0.380	0.472	0.589	0.481	0.447
Gemma 4 31B		0.399	0.494	0.628	0.507	0.478
Qwen 3.6 35B		0.343	0.334	0.516	0.398	0.372
Gemini 3.6 Flash	CoT	0.438	0.377	0.527	0.447	0.450

Appendix

Table D4: Agreement of KIVA and VLM judges with human labels across robot morphologies. Spearman’s

\rho

is computed per 16-frame segment between the human label (Bad 1, Medium 2, Good 3) and each method’s score: the mean level over three runs for a VLM, the continuous score for KIVA. Overlap is the percentage of segments whose three-level verdict matches the human label; a VLM’s verdict is the majority over its runs. KIVA’s two band cuts are calibrated on nine videos per embodiment (three per quality level) and applied to the remaining videos; its overlap is the mean over 1000 random draws of the calibration videos. Macro is the mean over the three embodiments; Micro pools all segments. Bold: best; italic: best VLM.

Figure D4: Alignment of RIGIS scores with human preference ratings. Distribution of RIGIS scores across human ratings for Bimanual, Humanoid, and Single-Arm morphologies.

Table D5: Agreement of RIGIS and VLM judges with human labels across robot morphologies. Spearman

\rho

and pairwise rank accuracy

RankAcc

are computed per embodiment between human labels and each method’s scores, utilizing the majority verdict over three runs for VLMs and raw continuous scores for RIGIS, and subsequently averaged across embodiment. Pairwise rank accuracy specifically measures the fraction of human-ordered video pairs whose relative order is preserved by the evaluation method, with prediction ties treated as errors.

Figure D5: Qualitative comparison of multi-view consistency evaluation between RIGIS, VLM judges, and human consensus. Each panel displays synchronized multi-view video frames alongside temporal trajectories, where the synchronized trajectories denote the geometry and rig stability quality. The VLM’s score is averaged across all methods, turns, and prompting strategies. These examples highlight notable discrepancies and failure modes where VLM judges misalign with human assessments and automated metrics.

Figure E1: The three scaling dimensions for RoboWorM . Path A integrates a new world model $M_{\text{new}}$ fine-tuned on the released corpus, evaluated across the standard grid of embodiment, task dynamics, and camera setup, and scored with identical metrics to populate a new table row. Path B incorporates a new embodiment or dataset $E_{\text{new}}$ by adding a table column, registering camera mounts and task dynamics, formatting data into native representations, fine-tuning across three backbones, and instituting physical audits including a KIVA keypoint reader and a RIGIS camera setup. Path C introduces a new acceleration method $A_{\text{new}}$ as an intervention family applied to existing checkpoints without retraining, evaluated against dense rollouts on the standard grid, and plotted on the speed and quality plane alongside previously audited methods. Only Path B requires training, whereas Paths A and C reuse released episodes, conditioning, and checkpoints.

World Model Evaluation

World Model Evaluation measures whether learned simulators predict consistent, physically plausible, action-responsive futures that help agents plan and act.

See all 210 papers

Apr 21, 2026

RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulation

Recent advances in large-scale video world models have enabled increasingly realistic future prediction, raising the prospect of using generated videos as scalable supervision for robot learning. However, for embodied manipulation, perceptual realism alone is not sufficient: generated interactions must also be physically consistent and executable by robotic agents. Existing benchmarks provide valuable assessments of visual quality and physical plausibility, but they do not systematically evaluate whether predicted behaviors can be translated into executable actions that complete manipulation tasks. We introduce RoboWM-Bench, a manipulation-centric benchmark for embodiment-grounded evaluation of video world models. RoboWM-Bench converts generated human-hand and robotic manipulation videos into embodied action sequences and validates them through execution in physically grounded simulation environments. Built on real-to-sim scene reconstruction and diverse manipulation tasks, RoboWM-Bench enables standardized, reproducible, and scalable evaluation of physical executability. Using RoboWM-Bench, we evaluate state-of-the-art video world models and observe that visual plausibility and embodied executability are not always aligned. Our analysis highlights several recurring factors that affect execution performance, including spatial reasoning, contact prediction, and non-physical geometric distortions, particularly in complex and long-horizon interactions. These findings provide a more fine-grained view of current model capabilities and underscore the value of embodiment-aware evaluation for guiding physically grounded world modeling in robotic manipulation.

Feng Jiang, Yang Chen, Kyle Xu +8

Peking University · Tsinghua University · Lightwheel

Robotic Manipulation Robot Manipulation Benchmarks World Model Evaluation Video World Models

Ask PDF

Jul 1, 2026

RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation

Video world models are emerging as a scalable alternative for evaluating generalist robot policies, bypassing the physical constraints and engineering burdens of real-world deployment. However, evaluating policies with video world models remains challenging, as world-model errors can make generated rollouts unreliable and slow inference limits large-scale throughput. We introduce RoboWorld, an automated evaluation pipeline that pairs a fast autoregressive video world model with a task-progress-aware vision-language model scoring. To enable reliable long-horizon autoregressive world-model rollouts, we propose Step Forcing, which combines anchored and one-step self-forwarded contexts to reduce train-test mismatch while preserving action-observation dynamics. Together, these components enable RoboWorld to align strongly with real-world robot evaluation across tasks and environments, achieving Pearson's r = 0.989 and Spearman's

ρ

= 0.970.

Byeongguk Jeon, Seonghyeon Ye, JaeHyeok Doo +4

1KAIST · 2Config

World Model Learning Robotic Policy Evaluation World Model Evaluation Video World Models

Ask PDF

Aug 28, 2026

RoboPhys-3D: A Comprehensive Embodied World Model Evaluation via 3D Reconstruction

Video world models increasingly serve as data engines, action planners, and simulators for embodied AI, but conventional embodied world model (EWM) benchmarks lack a unified 3D-grounded protocol for establishing whether generated rollouts preserve the underlying 3D scene state or translate into executable actions. We introduce RoboPhys-3D, a 3D-grounded EWM benchmark built on RoboTwin 2.0, covering 50 manipulation tasks across four regimes, with 5,000 episodes and 25,000 multi-view ground-truth videos. A defining feature of RoboPhys-3D is that generated and ground-truth videos are processed through the same 3D reconstruction pipeline, enabling reconstruction-induced error to be distinguished from generation-induced error. The RoboPhys-3D benchmark organizes 50 complementary metrics into 18 sub-dimensions across four levels: pixel-level fidelity, 3D geometry consistency, state-level understanding, and task-level completeness. We further introduce Average Full Score, a hierarchical score averaging all 50 metrics for comprehensive evaluation, and RoboPhyscore, a compact task-aligned score averaging the metrics most strongly correlated with task success. Among the four representative video world models, Cosmos 3 achieves the highest RoboPhyscore (0.6330, 92.7% of ground truth), while state- and execution-grounded metrics reveal substantial failures that perceptual and vision-language model-based judgments fail to capture. RoboPhyscore further exhibits strong agreement with human evaluation (Pearson r = 0.9761 and Spearman \r{ho} = 0.8962), demonstrating the importance of grounded, execution-aware evaluation for EWM capability.

Tianyi Wang, Jiazhou Chen, Yiming Xu +7

The University of Texas at Austin · Purdue University · Futurewei Technologies, Inc.

World Model Evaluation Embodied World Models Video World Models Robot Manipulation Benchmarks

Ask PDF

Abstract

World Model Evaluation

Explore similar work

RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulation

RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation

RoboPhys-3D: A Comprehensive Embodied World Model Evaluation via 3D Reconstruction