Reinforcement learning has gained attention for improving action-conditioned world models (ACWMs), as it enables reward-driven optimization of visual quality (VQ) and action-following fidelity (AF) beyond flow-matching optimization. Realizing this potential requires reward signals that reliably rank videos by both VQ and AF. However, existing rewards capture only one dimension: metric-based rewards assess VQ but require ground-truth videos and do not directly assess AF, while inverse dynamics model (IDM) rewards assess AF without references but do not guarantee VQ. Moreover, IDMs, typically trained on real videos, may provide unreliable rewards for generated videos containing visual artifacts and physical inconsistencies. In this work, we introduce the Action and Video Mismatch Benchmark (AVM-Eval), comprising 67.2k pairs that evaluate how reliably IDMs assess AF on real and generated videos. We then propose ACWM-RM, a reward model that jointly assesses VQ and AF without ground-truth videos during reward computation. To train it, we first construct the Action and Video Mismatch Preference Dataset (AVM-Pref) with 506.3k pairs providing complementary supervision: action-mismatch pairs supervise AF, while video-mismatch pairs supervise VQ and expose the model to generated videos. We then pretrain an action-conditioned predictor on real videos to learn forward dynamics before jointly training it with a patch-weighted score head on AVM-Pref to transfer these dynamics to AF assessment on generated videos. Evaluation on AVM-Eval reveals the limited reliability of IDMs on generated videos, while downstream RL experiments show that ACWM-RM outperforms reward baselines in both criteria.
Figures & tables
Table 1: Statistics of AVM-Eval and AVM-Pref. Abbreviations: N-P/S-P, numerical/semantic perturbations; Intra/Cross, intra-model and cross-model pairs. Video source: Rel., real-world; Rec., simulator-recorded; Gen., generated. Criteria: AF, action-following fidelity; VQ, visual quality.
Figure 1: Overview of (a,b) action-mismatch and video-mismatch pair construction for AVM-Eval and AVM-Pref, and (c) ACWM-RM reward computation without reference videos.
Table 3
Table 5: Visual quality and physical consistency of DreamDojo on AgiC after Flow-GRPO with different reward signals, compared with the pre-RL model (Origin). LPIPS, SSIM, and MSE are reported ×100 , and physical consistency scores range from 1 to 5. Better Worse than Origin.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Clip trajectory spread. Each panel histograms the log10 spread statistic of Eq. 8 over the extracted clips, with one column per arm and one row per channel type. The red dashed line marks the threshold calibrated for that channel, placed slightly to the right of the valley separating them.
Figure 4: Distributions of absolute nDTW score differences ( ×100 ) for intra-model (left) and cross-model (right) candidate pairs from 5,760 rollouts conditioned on clips from AgiAtest and AgiB . The horizontal axes use a logarithmic scale. Black vertical lines mark the respective 5th-percentile thresholds, 0.03 and 0.12; pairs below the corresponding threshold are discarded.
Figure 5: Visual-quality gap distributions for intra-model (top) and cross-model (bottom) candidate pairs from AgiBotWorld. Black dashed lines mark the 2nd-percentile thresholds.
EVA-IDM
GR00T-IDM
TAP-IDM
J-IDM
Visual backbone
ResNet-50
SigLIP2-L/16
Qwen2.5-VL-3B
VGGT
Backbone initialization
pretrained
pretrained
pretrained
pretrained
Input resolution
512×512
224×224
224×224
128×128
Frames/prediction
3
2
2
2
Action dim for reward
(14)
(14)
(14)
(14)
Objective
smooth- L1
flow matching
CE
Charbonnier flow
Appendix
Table 8: Architectures and observation-action interfaces. Frame counts are per prediction within a 13-frame training clip.
EVA-IDM
GR00T-IDM
TAP-IDM
J-IDM
Batch / GPU
32
16
16
16
GPUs
8
8
8
8
Global batch
256
128
128
128
Learning rate
3.0×10−4
1.0×10−4
5.0×10−5
5.0×10−5
Steps
27.8k
55.6k
55.6k
55.6k
Clip draws
7.12M
7.12M
7.12M
7.12M
Appendix
Table 9: IDM optimization settings. Each model uses 8 GPUs; batch sizes count complete 13-frame clips. Optimizer steps are set to target a nominal budget of approximately 7.12M clip draws per model. Step and draw counts are rounded.
Action component
Unit
Left arm σj
Right arm σj
X translation
m
0.003746
0.004671
Y translation
m
0.003482
0.003662
Z translation
m
0.003115
0.003379
X rotation
rad
0.014987
0.013658
Y rotation
rad
0.013690
0.014198
Z rotation
rad
0.015895
0.015860
Appendix
Table 10: Per-dimension action standard deviations used for IDM reward computation, estimated from AgiBotWorld.
Table 11: Prompt used by Qwen3-VL to compute rewards during Flow-GRPO and by GPT-5.6 sol to evaluate the physical consistency of videos generated by DreamDojo after Flow-GRPO fine-tuning.
Figure 6: Distribution of numerical perturbation counts per action-mismatch pair. AgiAtrain shows AVM-Pref; all other panels show AVM-Eval. N is the number of distinct numerical-perturbation pairs, and kˉ is the mean number of distinct perturbation types per pair.
Figure 7: Video-mismatch pair statistics for AVM-Eval on AgiAtest , with 3,000 intra-model and 3,000 cross-model pairs. Panels (a) and (b) show counts by ACWM and ACWM pair, respectively; (c) and (d) show the corresponding task distributions.
Figure 8: Statistics of video-mismatch pairs in AVM-Pref constructed from AgiAtrain . Panels (a) and (b) show intra-model pair counts by ACWM and cross-model pair counts by ACWM combination, respectively; panels (c) and (d) show the corresponding counts by source task. We aim to balance pair counts across models and tasks. However, task-level balance is limited by the availability of candidate pairs that satisfy all four quality criteria. Cross-model pairing additionally requires rollouts from both ACWMs under the same input condition, further restricting the available candidates for some tasks. So, tasks with insufficient eligible pairs cannot meet their target quotas, and the remaining quotas are redistributed to tasks with sufficient candidates.
Figure 9: Statistics of video-mismatch pairs in AVM-Eval on AgiB , comprising 4,341 intra-model and 3,000 cross-model pairs. Panels (a) and (b) show counts by ACWM and ACWM pair, respectively; panels (c) and (d) show the corresponding task distributions.
Figure 10: Action-mismatch pair distances in AVM-Pref on AgiAtrain , computed using Eq. 48 . Panels (a) and (b) show numerical and semantic perturbations, respectively. The horizontal axes use a logarithmic scale.
Figure 11: Video-mismatch nDTW gaps in AVM-Eval (Eq. 49 ). Rows: AgiAtest and AgiB ; columns: intra-model and cross-model pairs. Horizontal axes are logarithmic.
Figure 12: Action-mismatch pair distances in AVM-Eval (Eq. 48 ). Rows: five source datasets; columns: numerical and semantic perturbations. Horizontal axes are logarithmic.
Figure 13: Video-mismatch pair distances in AVM-Pref on AgiAtrain (Eq. 50 ). Panels (a) and (b): intra-model and cross-model pairs. Horizontal axes are logarithmic.
Camera-conditioned world models generate interactive videos in which commanded actions should induce the expected scene changes while appearance, geometry, and temporal dynamics remain coherent. Existing rewards assess these requirements separately: geometry-based rewards estimate trajectory execution but cannot judge the visual quality of the executed motion, whereas image-based rewards measure frame quality without capturing action execution or temporal dynamics. We posit that a vision-language model (VLM) offers a shared reasoning space for relating actions to their visual outcomes. However, judging a complete long video against its full action sequence creates a lengthy, noisy context in which short-lived local action evidence can be missed or diluted. We present WorldReward, a VLM-based pairwise preference reward model that unifies action-consistency and visual-quality evaluation for camera-conditioned world models. WorldReward decomposes paired videos into action-aligned chunks, organizes each chunk into structured visual evidence, and aggregates chunk-level decisions by voting into separate video-level action and visual-quality preferences. To train it, we construct a large-scale reasoning-augmented preference dataset using structured judgments generated by a frontier VLM and refined through tool-based agent auditing and targeted human review. We further introduce WorldReward-Bench, a human-annotated benchmark measuring reward-model agreement with human preferences across action consistency, appearance quality, and motion quality. WorldReward achieves the highest agreement on all three dimensions, exceeding GPT-5.5 by 3.42, 1.45, and 3.56 percentage points, respectively. When used for RL post-training of HY-WorldPlay 1.5, it consistently improves both action execution and visual quality across short- to long-term horizons.
Yibin Wang, Zehan Wang, Junshu Tang +13
Fudan University · Tencent Hunyuan · Shanghai Innovation Institute +3
Video reward models are trained and evaluated on human preferences between videos that often differ simultaneously on properties such as prompt alignment, visual fidelity, motion, and generator identity. As a result, aggregate accuracy does not reveal which video properties a reward model can reliably distinguish. We construct preference pairs by controlled interventions that change one property of a generated video while holding the rest fixed. The resulting pairs, by construction, provide preferences for how the videos differ along a specific dimension, without requiring any human annotation. We propose IsoVRBench, a video generation reward modeling benchmark of 8,000 such pairs across eight interventions, and show that state-of-the-art models separate broad quality differences between generators but miss important features such as temporal order. The best reward model reaches only 60.8% on reordered frames against 100% for humans, and rankings on existing benchmarks do not predict this failure. Using the synthetic isolated intervation pairs as training data leads to 1) IsoVRM, a reward model outperforming its counterpart trained with a larger human labeled preference dataset by 3.8 points on VBench-2.0, 2) IsoWan, the video generator post-trained with IsoVRM, outperforming its human-data counterpart in human preferences, and 3) gains that stack when combined with human data. Because intervention pairs are automatically generated rather than collected, the benchmark and training data can be easily extended as generators evolve.
Shuyue Stella Li, Jacqueline He, Jiahui Chen +5
University of Washington · 2Meta Superintelligence Labs
Multimodal reward models have advanced substantially in text and image domains, yet progress in video understanding reward modeling remains severely limited by the lack of robust evaluation benchmarks and high-quality preference data. To address this, we propose a unified framework spanning benchmark design, data construction, and reward model training. We introduce Video Understanding Reward Bench (VURB), a benchmark featuring 2,100 preference pairs with long chain-of-thought reasoning traces (averaging 1,143 tokens) and majority voting evaluation across general, long, and reasoning-oriented video tasks. We further construct Video Understanding Preference Dataset (VUP-35K) via a fully automated pipeline, providing large-scale high-quality supervision for video reward training. Building on the data, we train VideoDRM and VideoGRM, a discriminative and a generative reward model, both achieving state-of-the-art performance on VURB and VideoRewardBench. Further analysis confirms that VUP-35K enhances both reward performance and model reasoning capability, while VideoDRM and VideoGRM yield significant gains under best-of-N test-time scaling.
Yuancheng Wei, Linli Yao, Lei Li +4
South China University of Technology · Peking University · The University of Hong Kong +1