Group Relative Policy Optimization (GRPO) has advanced generative model post-training in verifiable domains such as code and mathematics, but evaluating video quality, especially emotional expression, often depends on subjective aesthetics and has no uniquely correct target. Existing approaches typically rely on fixed rewards for visual quality, motion quality, and text alignment that poorly capture emotional intent and can invite reward hacking, while RLHF is difficult to scale and subject to individual aesthetic differences that complicate the calibration of facial, body motion, and chromatic cues across narrative contexts. We introduce EmoReward, a new self-evolving training mechanism that distills evidence reliability from reward traces collected during an actual training run to adaptively guide multi-reward weight allocation in subsequent training by determining which emotional evidence to trust and how much to weight it. EmoReward first plans specific grounding cues from the emotional prompt to guide evaluation, then uses heterogeneous signal primitives to score the generated video during the act stage. Reflective Reward Gating calibrates their contributions through instance-level gates for phase, face confidence, conflict, and subtlety, then renormalizes weights over the active set to produce the sample's fused reward. RewardMemory then estimates each primitive's reliability from rollout scores and outcomes within each emotional regime in the finished training run, then fixes bounded multipliers that calibrate fusion weights in the next training run without updating reward models. Experiments on HunyuanVideo 1.5 show improvements across multiple RL algorithms and compare EmoReward with other agentic reward methods under SAGE-GRPO. Ablations and transfer experiments on SD 3.5 and FLUX further support adaptive reward calibration for visual generation.
Figures & tables
Figure 1: From fixed scoring to contextual reward calibration. The reward design progresses from a fixed evaluator to a heterogeneous provider pool and then to EmoReward. Reflective Reward Gating selects relevant evidence for the current video; RewardMemory calibrates provider trust from experience. All three stages ultimately return a scalar reward. The reward providers remain frozen while RL updates the video generation model.
Figure 2: EmoReward calibrates emotional evidence within videos and across training runs. Plan translates intent into evaluation criteria, and Act obtains provider scores and evidence validity. Reflect combines current-video gates with memory-derived multipliers and normalizes over active providers. Evolve estimates contextual reliability from a completed reference run to construct the fixed multipliers used in subsequent training. The calibrated emotion reward complements the original video reward; GRPO updates the video generation model while the providers and memory table remain fixed.
Figure 3: Emotional expression in generated videos. Eight clips are shown at three moments each. (Top) A performer looks up from a letter. (Middle rows) Gaze, gesture, and expression convey emotion in scenes such as a violin performance and a rooftop piano scene. (Bottom) Laughter fades into a quiet gaze.
Figure 4
Emotion Metrics
VideoAlign
Visual Metrics
Method
DEA
TERS
Overall
VQ
MQ
TA
PickScore
CLIPScore
GT (reference)
0.655 ± 0.029
0.7904 ± 0.0145
-1.1938 ± 0.0520
-0.5899 ± 0.0255
-0.4289 ± 0.0220
-0.1750 ± 0.0363
0.7059 ± 0.0018
0.5623 ± 0.0037
DanceGRPO
0.605 ± 0.030
0.7640 ± 0.0153
-1.2812 ± 0.0513
-0.6392 ± 0.0230
-0.3867 ± 0.0235
-0.2554 ± 0.0357
0.7177 ± 0.0018
0.5675 ± 0.0036
+ EmoReward
0.632 ± 0.030
0.7824 ± 0.0149
-1.3930 ± 0.0559
-0.6279 ± 0.0284
-0.3833 ± 0.0243
-0.3818 ± 0.0340
0.6986 ± 0.0017
0.5599 ± 0.0037
FlowGRPO
0.621 ± 0.030
0.7862 ± 0.0144
-0.7719 ± 0.0569
-0.3567 ± 0.0290
-0.2753 ± 0.0271
-0.1398 ± 0.0361
0.7061 ± 0.0019
0.5634 ± 0.0036
+ EmoReward
0.670 ± 0.029
0.8018 ± 0.0142
-0.5376 ± 0.0556
-0.3398 ± 0.0291
-0.2302 ± 0.0275
0.0323 ± 0.0356
0.7115 ± 0.0018
0.5605 ± 0.0037
Table 2: EmoReward across video RL algorithms. DEA uses the 261 prompts with a usable opposite emotion. TERS ranks the target emotion on all 302 prompts. All metrics report mean ± SEM across prompts. GT is the pretrained no-RL reference. Bold marks an increase over its corresponding RL baseline.
Figure 6
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Interface for ordinal human evaluation. The interface presents the prompt and generated video with independent ratings of emotional fidelity, temporal coherence, and visual quality, each using three ordered categories: Bad, Normal, and Good. This reconstruction illustrates the reported rating protocol rather than the original data-collection interface.
Emotion fidelity
Temporal coherence
Visual quality
Method
Bad
Normal
Good
Bad
Normal
Good
Bad
Normal
Good
GT
118
188
234
127
204
209
136
208
196
SAGE-GRPO
170
84
286
156
106
278
167
130
243
SAGE-GRPO + EmoReward
83
133
324
109
138
293
86
145
309
Appendix
Table 4: Aggregate ordinal counts. Each method–criterion block contains 540 ratings. The counts reproduce the percentages in main-paper Table 3 .
Family
Provider
Estimator / signal
Eq.
Facial
AU matching
Facial action-unit head
12
Facial
CLIP-AUTT
Temporal AU–text similarity
13
Semantic
R1-Omni
Valence–arousal estimation
14
Semantic
Macro
Alignment and AU compliance
15
Body
Body cues
Body-marker satisfaction
16
Temporal
Peak alignment
AU peak versus target window
18
Appendix
Table 5: Primitive family, provider, backing estimator, and defining equation. The two appearance channels (chromatic, kinetic) are always-on lightweight signals; the remaining seven form the memory support whose per-provider reliability is reported in the calibration analysis (Table 8 ).
CLIP ViT-H/14
CLIP ViT-L/14
DEA
TERS
DEA
TERS
RL algorithm
Base
+EmoReward
Base
+EmoReward
Base
+EmoReward
Base
+EmoReward
DanceGRPO
0.605
0.632
0.764
0.782
0.625
0.632
0.763
0.770
FlowGRPO
0.621
0.670
0.786
0.802
0.625
0.659
0.782
0.802
SAGE-GRPO
0.617
0.674
0.788
0.819
0.613
0.670
0.774
0.808
Discordant wins
90/145 (62.1%)
–
106/186 (57.0%)
–
Appendix
Table 6: Emotion metrics across frozen encoders. DEA uses 261 prompts and TERS uses all 302. The last two rows summarize discordant algorithm–prompt pairs and prompt-blocked permutation tests for DEA.
Configuration
Overall
VQ
MQ
TA
ImageReward
VideoAlign
-1.614
-0.505
-0.313
-0.797
-0.866
+Semantic emotion
-1.363
—
-0.014
-0.991
-0.922
+Motion naturalness
-1.453
-0.505
0.075
-1.024
-1.293
Balanced tool pool
-1.178
-0.297
0.074
-0.954
-1.113
Appendix
Table 7: Provider-pool configurations. Higher is better. TA denotes text alignment. The reward configurations use different weights and curricula; missing VQ is not imputed.
Provider
Macro
Motion
CLIP
R1-Omni
Micro
Body
Temporal
ρk
0.429
0.332
0.073
0.047
0.000
0.000
0.000
mk
1.833
1.510
0.899
0.854
0.778
0.778
0.778
Appendix
Table 8: RewardMemory calibration. Provider reliability and unnormalized multipliers, estimated from 3,232 reflective reward traces ( τ=0.5 , c=1 ). Macro includes VideoAlign information, so its association is partly shared with the outcome.
Figure 8: From completed-run evidence to calibrated fusion. Recorded observations (A) supply provider scores and outcomes; RewardMemory (B) supplies fixed multipliers estimated from the complete reference bank, rather than the four displayed clips. A controlled replay (C) applies these multipliers to illustrative initial weights for rollout #2. The resulting 0.578→0.563 calculation demonstrates reallocation with fixed scores; it is distinct from the logged reward 0.721 and does not represent a second training run.
Figure 9: Qualitative comparison of reward calibration. Columns show the baseline, the addition of Reflect, and the complete EmoReward configuration for SD3.5 with DanceGRPO. Prompt (b) is spelling-normalized for readability; the images retain the original perturbed input. Budgets and seed limitations appear in Appendix K.1 .
Figure 10: Image transfer across SD3.5 and FLUX. Five backbone–algorithm settings share four prompts; each panel compares HPSv3 RL, Reflective gating, and EmoReward. Rows show the same curated prompts as the preceding figure. The first two FLUX baseline rows use earlier saved outputs; see Appendix K.1 for budgets and seed limitations.
Reinforcement learning for video generation fine-tunes video models with reward feedback to improve quality, motion, controllability, and physical consistency.
New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tencent +1