ReWorld: Multi-Dimensional Reward Modeling for Embodied Video Generation

Baorui Peng; Wenyao Zhang; Liang Xu; Bohan Li; Hu Zhu; Zekun Qi; Hongsi Liu; Jiazhao Zhang

cs.ROJan 18, 2026

ReWorld: Multi-Dimensional Reward Modeling for Embodied Video Generation

Authors: Baorui Peng, Wenyao Zhang, Liang Xu, Bohan Li, Hu Zhu, Zekun Qi, Hongsi Liu, Jiazhao Zhang,

Organizations: Eastern Institute of Technology, Ningbo · Georgia Institute of Technology · Shanghai Jiao Tong University · Tsinghua University ·

Abstract

Current embodied video generation models primarily emphasize visual generative quality while overlooking physical fidelity, dynamic consistency, and task logic, especially for contact-rich manipulation tasks, which limits their applicability to downstream tasks. In this paper, we introduce ReWorld, a framework aimed to employ reinforcement learning to align the video generation with physical realism, task completion capability, embodiment plausibility and visual quality. Specifically, we first construct a large-scale (~235K) video preference dataset and employ it to train a hierarchical reward model designed to capture multi-dimensional reward consistent with human preferences. We further propose a practical alignment algorithm that post-trains flow-based video generators using this reward through a computationally efficient PPO-style algorithm. Comprehensive experiments and theoretical analysis demonstrate that ReWorld significantly improves the physical fidelity, logical coherence, embodiment and visual quality of generated rollouts, outperforming previous methods.

Figures & tables

Figure 1: ReWorld bridges the gap of embodied video generator with physical realism, task completion capability, embodiment plausibility, and visual quality. Each of the four dimensions shown is rated on a scale of 1 (poor) to 6 (excellent), where higher scores indicate better performance.

Figure 2: Overview of the ReWorld framework . (a) We employ a VLM-driven annotation system to generate the 4-dimensional embodied preference dataset. (b) Building upon this dataset, we train the multi-dimensional reward model HERO based on the hierarchical feature space of InternVideo2 [ 61 ] . (c) We detail the reinforcement learning pipeline HERO-FPO to refine the generative policy with the learned multi-dimensional reward signal. (d) We introduce ReWorldBench as a specialized benchmark to evaluate embodied video generation models.

	Video Quality					4D Embodied Dimensions (1-10 Scale)				Overall
Model	FVD ( $\downarrow$ )	SSIM ( $\uparrow$ )	DINO Similarity ( $\uparrow$ )	PSNR ( $\uparrow$ )	DreamSim ( $\uparrow$ )	$S_{phys}$ ( $\uparrow$ )	$S_{embod}$ ( $\uparrow$ )	$S_{task}$ ( $\uparrow$ )	$S_{vis}$ ( $\uparrow$ )	$S_{ReWorld}$ ( $\uparrow$ )
CogVideoX [ 69 ]	720	0.70	0.56	14.6	0.74	3.1	2.8	3.4	7.1	35.3
Wan2.1 [ 54 ]	510	0.61	0.59	14.1	0.74	3.4	3.1	3.8	7.3	38.6
Cosmos-Base [ 2 ]	281	0.63	0.62	15	0.74	4.2	3.9	4.4	7.0	44.7
Cosmos-SFT (Ours)	240	0.68	0.65	16.1	0.78	5.1	4.2	6.1	7.2	54.4
ReWorld (Ours)	190	0.66	0.71	15.2	0.82	5.9	5.6	6.5	7.3	61.9

Table 1: Quantitative comparison on ReWorldBench. The evaluation covers both visual fidelity and performance on embodied tasks. We highlight the best results in bold .

Overall Performance		Per-Dimension Accuracy (Specialization)
Metric	Value	Head	Accuracy	Feature Source	Rank
Accuracy	85.3%	$R_{embod}$	87.2%	Mid-layer (L16)	1 (Best)
AUC	0.901	$R_{vis}$	84.6%	Final-pooled (L39)	2
Spearman ( $\rho$ )	0.787	$R_{task}$	81.3%	Deep-layer (L32)	3
Kendall’s ( $\tau$ )	0.689	$R_{phys}$	79.1%	Early-layer (L8)	4 (Hardest)

Table 2: HERO reward model performance.

Figure 3: Qualitative comparisons on ReWorldBench. Our proposed ReWorld model achieves the best generative results for all the multi-dimensional metrics compared with the baseline video generation models.

Table 6

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Pipeline for constructing the 4D Embodied Preference Dataset. The process consists of three stages: (1) VLM-Driven Annotation (Left): We employ GPT-4o with role-playing prompts to act as four specialized experts, generating a dense 4D score vector for each video. (2) Dimension Isolation Sampling (Middle): A constrained optimization algorithm selects video pairs that maximize difference in a target dimension (Condition 1) while minimizing variance in others (Condition 2), ensuring signal purity. (3) Final Composition (Right): The resulting dataset contains $\sim$ 235K pairs, structured into dimension-isolated subsets for specialized learning and contrastive subsets for global calibration.

Preference Pair Type	Approx. Quantity	Purpose in HERO Training
Dimension Isolation ( $R_{\text{phys}}$ )	$\sim\!55{,}000$	Trains $R_{\text{phys}}$ head (via $\mathcal{L}_{D}$ )
Dimension Isolation ( $R_{\text{task}}$ )	$\sim\!60{,}000$	Trains $R_{\text{task}}$ head (via $\mathcal{L}_{D}$ )
Dimension Isolation ( $R_{\text{embod}}$ )	$\sim\!50{,}000$	Trains $R_{\text{embod}}$ head (via $\mathcal{L}_{D}$ )
Dimension Isolation ( $R_{\text{vis}}$ )	$\sim\!30{,}000$	Trains $R_{\text{vis}}$ head (via $\mathcal{L}_{D}$ )
Extreme Contrast Pairs	$\sim\!20{,}000$	Trains $\mathcal{L}_{O}$ (establishes score anchors)
Nuance Pairs (high similarity)	$\sim\!20{,}000$	Trains $\mathcal{L}_{O}$ (improves fine-grained discrimination)

Appendix

Table 5: Composition of the 4D Embodied Preference Dataset.

Reward Head	Eval. dim.	IV2 block	Feat. dim.	Rationale
$R_{\text{phys}}$	Physical Realism	Early-layer (Block 8)	3200	Captures fine-grained motion, edges, and geometric cues essential for collision/penetration detection.
$R_{\text{embod}}$	Embodiment Plausibility	Mid-layer (Block 16)	3200	Encodes complex spatio-temporal motion patterns, ideal for assessing kinematic smoothness and coordination.
$R_{\text{task}}$	Task Completion	Deep-layer (Block 32)	1408	Represents high-level semantic concepts and logical state, necessary for understanding goal completion.
$R_{\text{vis}}$	Visual Quality	Final pooled output	768	Represents global aesthetics and frame-level coherence for overall fidelity assessment.

Appendix

Table 6: Hierarchical Feature Mapping in HERO.

Module	Input	Network architecture (PyTorch-style)
$R_{\text{phys}}$ head	(B, 3200)	Linear(3200, 512) $\rightarrow$ ReLU $\rightarrow$ Dropout(0.2) $\rightarrow$ Linear(512, 1)
$R_{\text{embod}}$ head	(B, 3200)	Linear(3200, 512) $\rightarrow$ ReLU $\rightarrow$ Dropout(0.2) $\rightarrow$ Linear(512, 1)
$R_{\text{task}}$ head	(B, 1408)	Linear(1408, 512) $\rightarrow$ ReLU $\rightarrow$ Dropout(0.2) $\rightarrow$ Linear(512, 1)
$R_{\text{vis}}$ head	(B, 768)	Linear(768, 256) $\rightarrow$ ReLU $\rightarrow$ Dropout(0.2) $\rightarrow$ Linear(256, 1)
$V_{\psi}$ critic	(B, 3, 16, 224, 224)	Conv3d(3, 64, k=3, s=2) $\rightarrow$ ReLU $\rightarrow$ BatchNorm3d(64)
		Conv3d(64, 128, k=3, s=2) $\rightarrow$ ReLU $\rightarrow$ BatchNorm3d(128)

Appendix

Table 7: Network architectures for HERO reward heads and the FPO critic

V_{\psi}

Dimension	Diagnostic Task Prompt	Targeted Failure Mode
$S_{phys}$ (Collision)	“Grasp the red block and firmly push it through the blue block.”	Adversarial Test : Checks if the model hallucinates object penetration instead of simulating physical resistance/blocking.
$S_{phys}$ (Gravity)	“Pick up the apple, lift it high, and then release it in mid-air.”	Dynamics Test : Checks for "floating object" hallucinations vs. correct gravitational acceleration.
$S_{task}$ (Logic)	“ First , pick up the sponge, then wipe the counter, and finally place it in the sink.”	Memory Test : Probes long-horizon temporal consistency and instruction following.
$S_{task}$ (Semantics)	“There are two cups. Pick up the blue cup, ignoring the red one.”	Negation Test : Evaluates fine-grained concept binding and resistance to semantic distractors.
$S_{embod}$ (Kinematics)	“ Slowly and smoothly reach for the fragile wine glass.”	Control Test : Penalizes high-jerk, unnatural "teleporting" motions common in video generation.
$S_{embod}$ (Grasping)	“Use a pincer grasp to pick up the small screw.”	Fine-Motor Test : Checks for plausible hand-object contact points vs. generic "magnetic" grasping.

Appendix

Table 8: ReWorldBench Diagnostic Matrix. Representative examples of how prompt design targets specific embodied failure modes.

Parameter	Value
Model Architecture
Video Backbone	InternVideo2-1B
Backbone Parameters	Frozen
Head Architecture	See Table 7
Head Dropout	0.2
Training & Optimization

Appendix

Table 9: Hyperparameters for HERO Reward Model Training.

Parameter	Value
Models & Hardware
Actor ( $\pi_{\theta}$ )	Cosmos-2B (SFT on Bridge V2)
Reward ( $R_{HERO}$ )	HERO (InternVideo2-1B, Frozen)
Critic ( $V_{\psi}$ )	VideoValueNetwork (See Table 7 )
Hardware	8 $\times$ NVIDIA A100 (80GB)
Training & Optimization

Appendix

Table 10: Hyperparameters for HERO-FPO Policy Alignment Training.

Figure 7: Qualitative comparison against SOTA baselines on the ReWorldBench diagnostic suite. We visualize generated rollouts for three distinct task primitives: (Top Row) Rigid Body Interaction: Testing contact physics and friction using a “Pushing” task. Baselines often exhibit “ghost touches” or sliding without friction. (Middle Row) Articulated Object Manipulation: Testing kinematic constraints using a “Drawer Opening” task. Baselines struggle to maintain the rigid connection between the handle and the drawer. (Bottom Row) Sequential Logic: Testing long-horizon causality using a “Plug & Switch” task. Baselines fail the negative constraint, often lighting up the lamp before the switch is pressed. ReWorld (Rightmost Column) consistently adheres to physical, kinematic, and logical constraints specified in the diagnostic prompts.

Figure 8: Gallery of diverse embodied skills generated by ReWorld. The figure showcases the model’s generalization across eight distinct physical domains, ranging from complex material properties (fluids, soft bodies) to dynamic laws (rotation, projectile motion) and geometric constraints (topology change, high-precision assembly). Detailed diagnostic prompts are provided below each sequence.

Reward Modeling for Video Generation

See all 8 papers

PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulation
Jun 26, 2026Peiwen Zhang, Yufan Deng, Shangkun Sun +11Physical Consistency in Video Generation Video Diffusion Models
Peking University, NVIDIA
ReactWorld: Structuring Interaction Responses for Embodied Video Generation
Nov 4, 2026Hanwei Liu, Huijia Zhang, Hanzhang Wang +6Embodied World Models Action-Conditioned Video Generation
Huawei Technologies Co., Ltd. · Tongji University · Zhejiang University
CreFlow: Corrective Reflow for Sparse-Reward Embodied Video Diffusion RL
May 14, 2026Zhenyang Ni, Yijiang Li, Ruochen Jiao +7Video Diffusion Models Reinforcement Learning
Northwestern University · University of California, San Diego · University of Oxford

Jun 26, 2026J/K move · Enter open · S save

PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulation

Peiwen Zhang, Yufan Deng, Shangkun Sun, Juncheng Ma+10

Peking University, NVIDIA

Physical Consistency in Video Generation Video Diffusion Models Robotic Manipulation Embodied World Models

Open paper AskPDF arXiv

Video generation models have emerged as a promising paradigm for embodied world simulation. However, both general-domain video generators and robot-specific data fine-tuned models can still produce physically implausible manipulations, including discontinuous motion trajectories and inconsistent robot-object interactions, which limits their reliability as world simulators. Through extensive experiments, we find that such physical instability mainly arises from two factors: deformation of moving objects and implausible spatio-temporal correlations among interacting entities, particularly during contact. Building on this observation, we propose PhysisForcing, a scalable training framework that strengthens physical consistency by focusing supervision on physics-informative regions through joint optimization of pixel-level and semantic-level features. The framework consists of a pixel-level trajectory alignment loss, which supervises DiT features using reference point trajectories, and a semantic-level relational alignment loss, which aligns DiT features with inter-region relations extracted from a frozen video understanding encoder. Extensive experiments on R-Bench, PAI-Bench, and EZS-Bench show that PhysisForcing consistently improves embodied video generation over strong baselines, improving the Wan2.2-I2V-A14B and Cosmos3-Nano base models on R-Bench by 22.3% and 9.2% (7.1% and 3.7% over vanilla finetuning), with the Cosmos3-Nano variant attaining the best overall score. Beyond generation, as a world model under the WorldArena action-planner protocol it raises the closed-loop success rate from 16.0% to 24.0% and further improves downstream policy success, indicating that physically aligned video models yield stronger representations for robotic manipulation.

Abstract

Reward Modeling for Video Generation

Explore similar work

PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulation

ReactWorld: Structuring Interaction Responses for Embodied Video Generation

CreFlow: Corrective Reflow for Sparse-Reward Embodied Video Diffusion RL