Group Relative Policy Optimization (GRPO) has advanced generative model post-training in verifiable domains such as code and mathematics, but evaluating video quality, especially emotional expression, often depends on subjective aesthetics and has no uniquely correct target. Existing approaches typically rely on fixed rewards for visual quality, motion quality, and text alignment that poorly capture emotional intent and can invite reward hacking, while RLHF is difficult to scale and subject to individual aesthetic differences that complicate the calibration of facial, body motion, and chromatic cues across narrative contexts. We introduce EmoReward, a new self-evolving training mechanism that distills evidence reliability from reward traces collected during an actual training run to adaptively guide multi-reward weight allocation in subsequent training by determining which emotional evidence to trust and how much to weight it. EmoReward first plans specific grounding cues from the emotional prompt to guide evaluation, then uses heterogeneous signal primitives to score the generated video during the act stage. Reflective Reward Gating calibrates their contributions through instance-level gates for phase, face confidence, conflict, and subtlety, then renormalizes weights over the active set to produce the sample's fused reward. RewardMemory then estimates each primitive's reliability from rollout scores and outcomes within each emotional regime in the finished training run, then fixes bounded multipliers that calibrate fusion weights in the next training run without updating reward models. Experiments on HunyuanVideo 1.5 show improvements across multiple RL algorithms and compare EmoReward with other agentic reward methods under SAGE-GRPO. Ablations and transfer experiments on SD 3.5 and FLUX further support adaptive reward calibration for visual generation.
Figures & tables
Figure 1: From fixed scoring to contextual reward calibration. The reward design progresses from a fixed evaluator to a heterogeneous provider pool and then to EmoReward. Reflective Reward Gating selects relevant evidence for the current video; RewardMemory calibrates provider trust from experience. All three stages ultimately return a scalar reward. The reward providers remain frozen while RL updates the video generation model.
Figure 2: EmoReward calibrates emotional evidence within videos and across training runs. Plan translates intent into evaluation criteria, and Act obtains provider scores and evidence validity. Reflect combines current-video gates with memory-derived multipliers and normalizes over active providers. Evolve estimates contextual reliability from a completed reference run to construct the fixed multipliers used in subsequent training. The calibrated emotion reward complements the original video reward; GRPO updates the video generation model while the providers and memory table remain fixed.
Figure 3: Emotional expression in generated videos. Eight clips are shown at three moments each. (Top) A performer looks up from a letter. (Middle rows) Gaze, gesture, and expression convey emotion in scenes such as a violin performance and a rooftop piano scene. (Bottom) Laughter fades into a quiet gaze.
Figure 4
Emotion Metrics
VideoAlign
Visual Metrics
Method
DEA
TERS
Overall
VQ
MQ
TA
PickScore
CLIPScore
GT (reference)
0.655 ± 0.029
0.7904 ± 0.0145
-1.1938 ± 0.0520
-0.5899 ± 0.0255
-0.4289 ± 0.0220
-0.1750 ± 0.0363
0.7059 ± 0.0018
0.5623 ± 0.0037
DanceGRPO
0.605 ± 0.030
0.7640 ± 0.0153
-1.2812 ± 0.0513
-0.6392 ± 0.0230
-0.3867 ± 0.0235
-0.2554 ± 0.0357
0.7177 ± 0.0018
0.5675 ± 0.0036
+ EmoReward
0.632 ± 0.030
0.7824 ± 0.0149
-1.3930 ± 0.0559
-0.6279 ± 0.0284
-0.3833 ± 0.0243
-0.3818 ± 0.0340
0.6986 ± 0.0017
0.5599 ± 0.0037
FlowGRPO
0.621 ± 0.030
0.7862 ± 0.0144
-0.7719 ± 0.0569
-0.3567 ± 0.0290
-0.2753 ± 0.0271
-0.1398 ± 0.0361
0.7061 ± 0.0019
0.5634 ± 0.0036
+ EmoReward
0.670 ± 0.029
0.8018 ± 0.0142
-0.5376 ± 0.0556
-0.3398 ± 0.0291
-0.2302 ± 0.0275
0.0323 ± 0.0356
0.7115 ± 0.0018
0.5605 ± 0.0037
Table 2: EmoReward across video RL algorithms. DEA uses the 261 prompts with a usable opposite emotion. TERS ranks the target emotion on all 302 prompts. All metrics report mean ± SEM across prompts. GT is the pretrained no-RL reference. Bold marks an increase over its corresponding RL baseline.
Figure 6
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Interface for ordinal human evaluation. The interface presents the prompt and generated video with independent ratings of emotional fidelity, temporal coherence, and visual quality, each using three ordered categories: Bad, Normal, and Good. This reconstruction illustrates the reported rating protocol rather than the original data-collection interface.
Emotion fidelity
Temporal coherence
Visual quality
Method
Bad
Normal
Good
Bad
Normal
Good
Bad
Normal
Good
GT
118
188
234
127
204
209
136
208
196
SAGE-GRPO
170
84
286
156
106
278
167
130
243
SAGE-GRPO + EmoReward
83
133
324
109
138
293
86
145
309
Appendix
Table 4: Aggregate ordinal counts. Each method–criterion block contains 540 ratings. The counts reproduce the percentages in main-paper Table 3 .
Family
Provider
Estimator / signal
Eq.
Facial
AU matching
Facial action-unit head
12
Facial
CLIP-AUTT
Temporal AU–text similarity
13
Semantic
R1-Omni
Valence–arousal estimation
14
Semantic
Macro
Alignment and AU compliance
15
Body
Body cues
Body-marker satisfaction
16
Temporal
Peak alignment
AU peak versus target window
18
Appendix
Table 5: Primitive family, provider, backing estimator, and defining equation. The two appearance channels (chromatic, kinetic) are always-on lightweight signals; the remaining seven form the memory support whose per-provider reliability is reported in the calibration analysis (Table 8 ).
CLIP ViT-H/14
CLIP ViT-L/14
DEA
TERS
DEA
TERS
RL algorithm
Base
+EmoReward
Base
+EmoReward
Base
+EmoReward
Base
+EmoReward
DanceGRPO
0.605
0.632
0.764
0.782
0.625
0.632
0.763
0.770
FlowGRPO
0.621
0.670
0.786
0.802
0.625
0.659
0.782
0.802
SAGE-GRPO
0.617
0.674
0.788
0.819
0.613
0.670
0.774
0.808
Discordant wins
90/145 (62.1%)
–
106/186 (57.0%)
–
Appendix
Table 6: Emotion metrics across frozen encoders. DEA uses 261 prompts and TERS uses all 302. The last two rows summarize discordant algorithm–prompt pairs and prompt-blocked permutation tests for DEA.
Configuration
Overall
VQ
MQ
TA
ImageReward
VideoAlign
-1.614
-0.505
-0.313
-0.797
-0.866
+Semantic emotion
-1.363
—
-0.014
-0.991
-0.922
+Motion naturalness
-1.453
-0.505
0.075
-1.024
-1.293
Balanced tool pool
-1.178
-0.297
0.074
-0.954
-1.113
Appendix
Table 7: Provider-pool configurations. Higher is better. TA denotes text alignment. The reward configurations use different weights and curricula; missing VQ is not imputed.
Provider
Macro
Motion
CLIP
R1-Omni
Micro
Body
Temporal
ρk
0.429
0.332
0.073
0.047
0.000
0.000
0.000
mk
1.833
1.510
0.899
0.854
0.778
0.778
0.778
Appendix
Table 8: RewardMemory calibration. Provider reliability and unnormalized multipliers, estimated from 3,232 reflective reward traces ( τ=0.5 , c=1 ). Macro includes VideoAlign information, so its association is partly shared with the outcome.
Figure 8: From completed-run evidence to calibrated fusion. Recorded observations (A) supply provider scores and outcomes; RewardMemory (B) supplies fixed multipliers estimated from the complete reference bank, rather than the four displayed clips. A controlled replay (C) applies these multipliers to illustrative initial weights for rollout #2. The resulting 0.578→0.563 calculation demonstrates reallocation with fixed scores; it is distinct from the logged reward 0.721 and does not represent a second training run.
Figure 9: Qualitative comparison of reward calibration. Columns show the baseline, the addition of Reflect, and the complete EmoReward configuration for SD3.5 with DanceGRPO. Prompt (b) is spelling-normalized for readability; the images retain the original perturbed input. Budgets and seed limitations appear in Appendix K.1 .
Figure 10: Image transfer across SD3.5 and FLUX. Five backbone–algorithm settings share four prompts; each panel compares HPSv3 RL, Reflective gating, and EmoReward. Rows show the same curated prompts as the preceding figure. The first two FLUX baseline rows use earlier saved outputs; see Appendix K.1 for budgets and seed limitations.
Reinforcement learning for video generation fine-tunes video models with reward feedback to improve quality, motion, controllability, and physical consistency.
Reinforcement learning (RL) is vital for optimizing video generation models, with a robust reward model (RM) serving as the cornerstone. However, existing video reward models often produce unstable scalar scores because they directly map complex, subjective video quality into a single score without explicit evaluation criteria. This leads to scalar drift, where the scoring scale collapses or shifts across different prompts, making the reward unreliable for RL. Drawing inspiration from professional human annotation engineering, we address this problem with RewardVerse, a rubric-based video reward framework that introduces a dynamic rubric as an intermediate representation between the evaluation query and the scorer. Instead of unconstrained direct scoring, RewardVerse first generates explicit evaluation criteria and then performs rubric-guided scoring, providing a stable semantic anchor that mitigates scalar drift. To efficiently optimize this collaborative pipeline, we propose Rubric-Guided Policy Optimization (RGPO), a two-stage training algorithm. RGPO first warms up the scorer using self-evolving seed rubrics and then jointly optimizes the rubric generator to produce query-adaptive evaluation criteria while continuously aligning the scorer with human ratings. Extensive experiments on the 16-dimensional EvalVerse benchmark and external datasets demonstrate that RewardVerse mitigates scalar drift, achieves state-of-the-art performance on both pointwise and pairwise evaluation, and provides a robust and interpretable reward signal for RL in video generation.
Zhenchen Tang, Yang Li, Songlin Yang +6
New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tencent +1
Video generation is central to AI-powered content creation. Aligning generated videos with human preferences is a key criterion for evaluating generation quality. Despite significant progress in visual quality, three key challenges remain. First, the reliability of reward signals is constrained by the quality of human preference data, which is often affected by subjective noise and bias. Second, standard scalar reward models collapse multi-aspect human preferences into a single value, leading to the loss of dynamic trade-offs across multiple preference dimensions. Third, in policy optimization, the widely adopted KL divergence imposes primarily local constraints and may fail to capture the global structure of human preferences. To address these challenges, we propose a unified preference-aware learning framework for video generation. First, we introduce elite-guided filtering to calibrate preference data and construct reliable supervision for reward model training. We then model video quality as a multidimensional reward distribution to capture the uncertainty inherent in human preferences, and use the Wasserstein distance to align the learned reward distribution with the empirical human preference distribution. Finally, we introduce Wasserstein-based distributional alignment into GRPO, guiding policy optimization to better match the global structure of human preferences over videos. Experiments on reward modeling and video generation demonstrate that our approach improves the reliability of reward signals and the perceptual consistency of generated videos. Our code is available at https://github.com/alignhs26/ahs.
Nai-Xin Zhai, Weihua Cheng, Dexu Yu +9
City University of Hong Kong · DeciLix Lab · ShanghaiTech University +7
Reinforcement learning (RL) for video generation usually assigns one scalar reward to an entire sampled video. Yet a video is not uniformly flawed: some visual tokens may already satisfy the prompt, whereas others require correction. A scalar reward cannot localize errors, causing optimization to perturb satisfactory tokens while under-targeting the tokens that actually need to change. We introduce Token-Level Video Reinforcement Learning, TVRL, a framework that derives token-level credit from the reward being optimized. Our key insight is that the answer likelihood of a frozen vision-language model provides both signals: its outputs contribute to the video-level reward, while magnitudes of its video-input gradients reveal which generated video tokens most affect that score. We instantiate TVRL in Group Relative Policy Optimization by averaging prompt-derived question rewards into one group-relative advantage and using detached, question-conditioned token-credit maps to reweight dense denoising-transition log-probabilities inside the clipped policy ratio. On VBench-2.0, TVRL achieves an Overall score of 57.69, outperforming the base model by 3.60 points. TVRL also improves matched GRPO baselines across three SDE samplers (SAGE, Flow, and Dance) by 2.68--3.15 points and across four reward models (VideoAlign, VideoScore2, UnifiedReward2, and Qwen3.5-9B) by 1.33--3.15 points.