cs.CVNov 8, 2026

GenRM-Flow: Generators are Process-aware Reward Models in Flow Matching

Authors: Siming Fu, Ruizhe He, Zheming Fu, Zeyue Xue, Fengkai Liu, Yufei Jiao, Jie Huang, Xiaoxiao Ma, 

Organizations: Zhejiang University · Joy Future Academy · Xiamen University · Tsinghua University

Abstract

Aligning video generation models with human preference depends critically on the quality of the reward model that scores generated videos. Current video reward models, however, source their reward signal from a model's representational rather than generative capacity: they either repurpose vision-language models as pixel-space scorers, or train auxiliary scoring heads on top of frozen generator features. The generator's actual ability to predict velocities, denoise latents, and produce videos plays no role in scoring its own generations. We argue that this overlooks a structural property of offline preference learning. Any preference optimization objective that approximates an offline preference dataset mathematically targets a strictly defined KL-constrained optimal policy. By isolating the reward term from this analytical solution and mapping the exact generation log-likelihood to its flow-matching surrogate, the empirical reward emerges deterministically as the relative difference in continuous-time velocity-prediction errors. We formalize this equivalence into GenRM-Flow, a framework where a preference-tuned video generator inherently operates as a process-aware reward model directly in the noisy latent space. We further show theoretically that distinct fine-tuning objectives, whether contrastive like DPO and IPO or non-contrastive like RWR, are alternative optimization pathways toward the same optimal policy, each inducing a reward readout from the generator's velocity-prediction errors. Furthermore, the extracted process reward seamlessly substitutes external VLM scorers in FlowAWR. Operating purely on native latent outputs without auxiliary reward networks, this generative reward drives measurable improvements in downstream video generation.

Figures & tables

Explore similar work

CardsList
  1. Aligning Human Sense: Calibrated Distributional Reward Learning for Video Generation

    Aug 16, 2026Nai-Xin Zhai, Weihua Cheng, Dexu Yu +9Reward Modeling for Video GenerationRL for Video Generation

  2. IsoVRBench: Video Reward Modeling Without Human Feedback

    Oct 12, 2026Shuyue Stella Li, Jacqueline He, Jiahui Chen +5Video Reward ModelingReward Modeling

  3. RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling

    Sep 19, 2026Zhenchen Tang, Yang Li, Songlin Yang +6Reward ModelingMultimodal Reward Modeling