ACWM-RM: A Reference-Free Reward Model from Complementary Mismatch Preferences

Shibo Hong; Siyu Ren; FengJiao Chen; Zhongyuan Peng; Wei Shi; Yixin Cao

cs.CVOct 23, 2026

ACWM-RM: A Reference-Free Reward Model from Complementary Mismatch Preferences

Authors: Shibo Hong, Siyu Ren, FengJiao Chen, Zhongyuan Peng, Wei Shi, Yixin Cao

Organizations: College of Computer Science and Artificial Intelligence, Fudan University · Meituan

Abstract

Reinforcement learning has gained attention for improving action-conditioned world models (ACWMs), as it enables reward-driven optimization of visual quality (VQ) and action-following fidelity (AF) beyond flow-matching optimization. Realizing this potential requires reward signals that reliably rank videos by both VQ and AF. However, existing rewards capture only one dimension: metric-based rewards assess VQ but require ground-truth videos and do not directly assess AF, while inverse dynamics model (IDM) rewards assess AF without references but do not guarantee VQ. Moreover, IDMs, typically trained on real videos, may provide unreliable rewards for generated videos containing visual artifacts and physical inconsistencies. In this work, we introduce the Action and Video Mismatch Benchmark (AVM-Eval), comprising 67.2k pairs that evaluate how reliably IDMs assess AF on real and generated videos. We then propose ACWM-RM, a reward model that jointly assesses VQ and AF without ground-truth videos during reward computation. To train it, we first construct the Action and Video Mismatch Preference Dataset (AVM-Pref) with 506.3k pairs providing complementary supervision: action-mismatch pairs supervise AF, while video-mismatch pairs supervise VQ and expose the model to generated videos. We then pretrain an action-conditioned predictor on real videos to learn forward dynamics before jointly training it with a patch-weighted score head on AVM-Pref to transfer these dynamics to AF assessment on generated videos. Evaluation on AVM-Eval reveals the limited reliability of IDMs on generated videos, while downstream RL experiments show that ACWM-RM outperforms reward baselines in both criteria.

Figures & tables

Panel of Table 1 — Table 1: Statistics of AVM-Eval and AVM-Pref. Abbreviations: N-P/S-P, numerical/semantic perturbations; Intra/Cross, intra-model and cross-model pairs. Video source: Rel., real-world; Rec., simulator-recorded; Gen., generated. Criteria: AF, action-following fidelity; VQ, visual quality.

Figure 1: Overview of (a,b) action-mismatch and video-mismatch pair construction for AVM-Eval and AVM-Pref, and (c) ACWM-RM reward computation without reference videos.

Table 3

Panel of Table 5 — Table 5: Visual quality and physical consistency of DreamDojo on $\mathrm{Agi_{C}}$ after Flow-GRPO with different reward signals, compared with the pre-RL model (Origin). LPIPS, SSIM, and MSE are reported $\times 100$ , and physical consistency scores range from 1 to 5. Better Worse than Origin.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Clip trajectory spread. Each panel histograms the $\log_{10}$ spread statistic of Eq. 8 over the extracted clips, with one column per arm and one row per channel type. The red dashed line marks the threshold calibrated for that channel, placed slightly to the right of the valley separating them.

Figure 4: Distributions of absolute nDTW score differences (

\times 100

) for intra-model (left) and cross-model (right) candidate pairs from 5,760 rollouts conditioned on clips from

\mathrm{Agi_{A}^{test}}

and

\mathrm{Agi_{B}}

. The horizontal axes use a logarithmic scale. Black vertical lines mark the respective 5th-percentile thresholds, 0.03 and 0.12; pairs below the corresponding threshold are discarded.

Figure 5: Visual-quality gap distributions for intra-model (top) and cross-model (bottom) candidate pairs from AgiBotWorld. Black dashed lines mark the 2nd-percentile thresholds.

	EVA-IDM	GR00T-IDM	TAP-IDM	J-IDM
Visual backbone	ResNet-50	SigLIP2-L/16	Qwen2.5-VL-3B	VGGT
Backbone initialization	pretrained	pretrained	pretrained	pretrained
Input resolution	$512\times 512$	$224\times 224$	$224\times 224$	$128\times 128$
Frames/prediction	3	2	2	2
Action dim for reward	(14)	(14)	(14)	(14)
Objective	smooth- $L_{1}$	flow matching	CE	Charbonnier flow

Appendix

Table 8: Architectures and observation-action interfaces. Frame counts are per prediction within a 13-frame training clip.

	EVA-IDM	GR00T-IDM	TAP-IDM	J-IDM
Batch / GPU	32	16	16	16
GPUs	8	8	8	8
Global batch	256	128	128	128
Learning rate	$3.0\!\times\!10^{-4}$	$1.0\!\times\!10^{-4}$	$5.0\!\times\!10^{-5}$	$5.0\!\times\!10^{-5}$
Steps	27.8k	55.6k	55.6k	55.6k
Clip draws	7.12M	7.12M	7.12M	7.12M

Appendix

Table 9: IDM optimization settings. Each model uses 8 GPUs; batch sizes count complete 13-frame clips. Optimizer steps are set to target a nominal budget of approximately 7.12M clip draws per model. Step and draw counts are rounded.

Action component	Unit	Left arm $\sigma_{j}$	Right arm $\sigma_{j}$
X translation	m	0.003746	0.004671
Y translation	m	0.003482	0.003662
Z translation	m	0.003115	0.003379
X rotation	rad	0.014987	0.013658
Y rotation	rad	0.013690	0.014198
Z rotation	rad	0.015895	0.015860

Appendix

Table 10: Per-dimension action standard deviations used for IDM reward computation, estimated from AgiBotWorld.

Panel of Table 11 — Table 11: Prompt used by Qwen3-VL to compute rewards during Flow-GRPO and by GPT-5.6 sol to evaluate the physical consistency of videos generated by DreamDojo after Flow-GRPO fine-tuning.

Figure 6: Distribution of numerical perturbation counts per action-mismatch pair.

\mathrm{Agi_{A}^{train}}

shows AVM-Pref; all other panels show AVM-Eval.

N

is the number of distinct numerical-perturbation pairs, and

\bar{k}

is the mean number of distinct perturbation types per pair.

Figure 7: Video-mismatch pair statistics for AVM-Eval on

\mathrm{Agi_{A}^{test}}

, with 3,000 intra-model and 3,000 cross-model pairs. Panels (a) and (b) show counts by ACWM and ACWM pair, respectively; (c) and (d) show the corresponding task distributions.

Figure 8: Statistics of video-mismatch pairs in AVM-Pref constructed from

\mathrm{Agi_{A}^{train}}

. Panels (a) and (b) show intra-model pair counts by ACWM and cross-model pair counts by ACWM combination, respectively; panels (c) and (d) show the corresponding counts by source task. We aim to balance pair counts across models and tasks. However, task-level balance is limited by the availability of candidate pairs that satisfy all four quality criteria. Cross-model pairing additionally requires rollouts from both ACWMs under the same input condition, further restricting the available candidates for some tasks. So, tasks with insufficient eligible pairs cannot meet their target quotas, and the remaining quotas are redistributed to tasks with sufficient candidates.

Figure 9: Statistics of video-mismatch pairs in AVM-Eval on

\mathrm{Agi_{B}}

, comprising 4,341 intra-model and 3,000 cross-model pairs. Panels (a) and (b) show counts by ACWM and ACWM pair, respectively; panels (c) and (d) show the corresponding task distributions.

Figure 10: Action-mismatch pair distances in AVM-Pref on

\mathrm{Agi_{A}^{train}}

, computed using Eq. 48 . Panels (a) and (b) show numerical and semantic perturbations, respectively. The horizontal axes use a logarithmic scale.

Figure 11: Video-mismatch nDTW gaps in AVM-Eval (Eq. 49 ). Rows:

\mathrm{Agi_{A}^{test}}

and

\mathrm{Agi_{B}}

; columns: intra-model and cross-model pairs. Horizontal axes are logarithmic.

Figure 12: Action-mismatch pair distances in AVM-Eval (Eq. 48 ). Rows: five source datasets; columns: numerical and semantic perturbations. Horizontal axes are logarithmic.

Figure 13: Video-mismatch pair distances in AVM-Pref on

\mathrm{Agi_{A}^{train}}

(Eq. 50 ). Panels (a) and (b): intra-model and cross-model pairs. Horizontal axes are logarithmic.

Multimodal Reward Modeling

Multimodal reward modeling builds evaluators that score images, video, audio, and agent behavior against human preferences or task goals.

See all 92 papers

WorldReward: Reward Modeling for Camera-Conditioned World Models
Sep 3, 2026Yibin Wang, Zehan Wang, Junshu Tang +13Reward Modeling Camera-Conditioned Video Generation
Fudan University · Tencent Hunyuan · Shanghai Innovation Institute +3
IsoVRBench: Video Reward Modeling Without Human Feedback
Oct 12, 2026Shuyue Stella Li, Jacqueline He, Jiahui Chen +5Reward Modeling Video Generation Evaluation
University of Washington · 2Meta Superintelligence Labs
Video Understanding Reward Modeling: A Robust Benchmark and Performant Reward Models
May 8, 2026Yuancheng Wei, Linli Yao, Lei Li +4Reward Modeling Multimodal Reward Modeling
South China University of Technology · Peking University · The University of Hong Kong +1

Sep 3, 2026J/K move · Enter open · S save

WorldReward: Reward Modeling for Camera-Conditioned World Models

Yibin Wang, Zehan Wang, Junshu Tang, Zhimin Li+12

Fudan University · Tencent Hunyuan · Shanghai Innovation Institute+3

Reward Modeling Camera-Conditioned Video Generation RL for Video Generation Multimodal Reward Modeling World Model Evaluation

Open paper AskPDF arXiv

Camera-conditioned world models generate interactive videos in which commanded actions should induce the expected scene changes while appearance, geometry, and temporal dynamics remain coherent. Existing rewards assess these requirements separately: geometry-based rewards estimate trajectory execution but cannot judge the visual quality of the executed motion, whereas image-based rewards measure frame quality without capturing action execution or temporal dynamics. We posit that a vision-language model (VLM) offers a shared reasoning space for relating actions to their visual outcomes. However, judging a complete long video against its full action sequence creates a lengthy, noisy context in which short-lived local action evidence can be missed or diluted. We present WorldReward, a VLM-based pairwise preference reward model that unifies action-consistency and visual-quality evaluation for camera-conditioned world models. WorldReward decomposes paired videos into action-aligned chunks, organizes each chunk into structured visual evidence, and aggregates chunk-level decisions by voting into separate video-level action and visual-quality preferences. To train it, we construct a large-scale reasoning-augmented preference dataset using structured judgments generated by a frontier VLM and refined through tool-based agent auditing and targeted human review. We further introduce WorldReward-Bench, a human-annotated benchmark measuring reward-model agreement with human preferences across action consistency, appearance quality, and motion quality. WorldReward achieves the highest agreement on all three dimensions, exceeding GPT-5.5 by 3.42, 1.45, and 3.56 percentage points, respectively. When used for RL post-training of HY-WorldPlay 1.5, it consistently improves both action execution and visual quality across short- to long-term horizons.

Abstract

Multimodal Reward Modeling

Explore similar work

WorldReward: Reward Modeling for Camera-Conditioned World Models

IsoVRBench: Video Reward Modeling Without Human Feedback

Video Understanding Reward Modeling: A Robust Benchmark and Performant Reward Models