Post-training aims to enhance desired attributes in visual generative models while preserving their general generation capabilities. Reward-guided post-training faces two challenges: real training data may mismatch the generator's output distribution, and attribute supervision may be confounded by unrelated content. Unbounded reward maximization can also cause overoptimization and degrade generation quality. We propose a framework combining Generator-Aligned Reward Modeling (GARM) and Target-Aware Bounded Optimization (TABO). GARM mitigates distribution mismatch through generator-support matching and reduces content confounding using content-matched real contrasts, yielding generator-relevant, attribute-focused reward supervision. TABO maps the learned reward to a smooth, upper-bounded function calibrated from real target-positive examples. This transformation is independent of the optimization algorithm and supports both direct reward-gradient and policy-based optimization. We further introduce Target-Aware GRPO (TA-GRPO), an adaptation of Flow-GRPO that combines fixed stage-wise advantage scaling, a generator-to-target curriculum, and timestep-balanced credit assignment for flow-based video optimization. Experiments show that GARM improves target-attribute expression while maintaining competitive generation quality. TABO better balances attribute enhancement and visual quality than unconstrained reward maximization, while TA-GRPO improves target enhancement over vanilla Flow-GRPO and reduces unintended attribute leakage on neutral prompts. These results highlight the importance of generator-aligned reward supervision and controlled, data-calibrated optimization for targeted visual generation.
Figures & tables
Figure 1: Targeted attribute enhancement with largely preserved general generation performance. Top: Base vs. Ours across four attributes (Vintage Faded, Oil Painting, Muscular, and 2D Japanese anime animation). Bottom: radar comparisons of Base (gray dashed, normalized to 100) and Ours.
Figure 2: Overview of our framework. GARM aligns real supervision with the generator-relevant regions defined by paired Base/Target anchors and introduces content-matched negatives to form an attribute-focused decision boundary. TABO maps the learned reward to a smooth, upper-bounded reward function, preserving sensitivity below a target level calibrated from real positives while reducing the marginal benefit of further score increases near and above it. Because this reward transformation is independent of the update rule, it can be used with differentiable optimization (VADER) and policy optimization (TA-GRPO).
Task
Matched positive
Matched negative
Content-matched negative
Vintage Faded
1,110
38,498
7,092
Oil Painting
585
39,117
4,169
Muscular
616
37,502
4,328
Anime video
7,737
52,975
7,933
Table 1: Core GARM pools used in the image and video experiments. The full pool definitions and sampling ratios are provided in the appendix.
Method
Mean ↑
P20 ↑
HPSv3 ↑
Q-Align Q. ↑
Q-Align Aes. ↑
HPSv2 ↑
PickScore ↑
VQA ↑
Muscular
Base (ref.)
51.838
24.835
9.469
4.826
3.881
0.307
20.582
0.706
Baseline
57.341
29.196
9.340
4.753
3.778
0.308
20.554
0.684
Generated-only
55.034
31.043
8.983
4.568
3.534
0.305
20.507
0.680
Content-Matched-only
57.445
29.880
9.427
4.812
3.915
0.310
20.564
0.700
Support-only
59.173
31.807
9.551
4.703
3.754
0.310
20.602
0.697
Table 2: Post-target results under different reward-supervision constructions. Target strength is measured by percentile statistics calibrated on the RGB Target-val anchors. Base is shown only as a reference; bold denotes the best post-trained result and underlining denotes the second-best post-trained result for each metric.
Method
Leakage ↓
HPSv3 ↑
Q-Align Q. ↑
Q-Align Aes. ↑
HPSv2 ↑
PickScore ↑
VQA ↑
Muscular
Base (ref.)
0.000
10.847
4.751
3.765
0.305
20.356
0.786
Baseline
0.011
10.678
4.696
3.667
0.306
20.328
0.783
Generated-only
0.002
10.584
4.618
3.568
0.304
20.307
0.788
Content-Matched-only
0.015
10.779
4.743
3.770
0.308
20.323
0.785
Support-only
0.020
10.823
4.653
3.628
0.306
20.354
0.786
Table 3: Post-neutral preservation under different reward-supervision constructions. Leakage measures the increase in target-tag probability relative to Base and is lower-is-better. Base is shown only as a reference; bold denotes the best post-trained result and underlining denotes the second-best post-trained result for each metric.
Figure 3: Qualitative comparison of Base, SFT, Baseline, Generated-only, and GARM on Muscular, Oil Painting, and Vintage Faded. The examples illustrate target-attribute expression and preservation of scene content and overall appearance.
(a) Post-target enhancement
Attribute
VBench
VideoScore2
Method
Mean ↑
P20 ↑
SC ↑
DD ↑
Aes. ↑
Mean ↑
VQ ↑
TA ↑
PC ↑
Mean ↑
Base (ref.)
59.263
43.430
0.9725
0.2117
0.6655
0.6166
3.9720
3.8826
3.7742
3.8763
Baseline
68.579
55.417
0.9717
0.1350
0.6362
0.5810
3.9586
4.1870
3.7643
3.9700
Generated-only
68.079
53.974
0.9684
0.1150
0.6288
0.5707
3.9713
3.9709
3.7016
3.8813
Content-Matched-only
65.589
51.195
0.9770
0.1400
0.6444
0.5871
3.9384
4.0143
3.7400
3.8976
Table 4: Post-target enhancement and post-neutral preservation on Anime video (Wan2.1-T2V-14B). Mean and P20 are RGB target-support percentiles; Leakage is the mean RGB logit increase relative to Base on neutral prompts. SC, DD, and Aes. denote VBench subject consistency, dynamic degree, and aesthetic quality; VQ, TA, and PC denote VideoScore2 visual quality, text alignment, and physical consistency. VBench Mean and VideoScore2 Mean are arithmetic means of the three corresponding metrics. Base is a reference; bold and underlining indicate the best and second-best post-trained values, respectively, with ties determined at the displayed precision.
Figure 4: Qualitative comparison of video generation methods for the target attribute 2D Japanese anime animation using Wan2.1-T2V-14B. GARM enhances the target animation style while preserving semantic content and temporal coherence.
Muscular
Oil Painting
Vintage Faded
Anime (video)
Method
Median ↑
Mean ↑
P20 ↑
Median ↑
Mean ↑
P20 ↑
Median ↑
Mean ↑
P20 ↑
Median ↑
Mean ↑
P20 ↑
Base (ref.)
48.184
51.838
24.835
48.018
51.282
26.436
49.264
50.733
22.606
60.778
59.263
43.430
SFT
47.776
50.460
24.319
64.091
60.614
33.835
57.300
56.889
26.660
65.087
66.774
50.822
Baseline
57.867
57.341
29.196
80.128
74.455
55.177
77.205
72.862
54.527
68.066
68.579
55.417
GARM
65.802
62.862
36.896
82.459
76.102
58.436
80.903
74.425
54.624
69.045
68.722
54.267
Table 5: SFT versus reward-guided post-training. We report RGB target-strength percentiles for three image attributes and Anime video generated with Wan2.1-T2V-14B. SFT applies LoRA updates using only real positive examples, while Baseline uses the corresponding real supervision to train a reward model for reward-guided post-training. Base is shown only as a reference; bold denotes the best post-trained result and underlining denotes the second-best post-trained result for each metric.
(a) VADER
Attribute
VBench
VideoScore2
Method
Mean ↑
P20 ↑
SC ↑
DD ↑
Aes. ↑
Mean ↑
VQ ↑
TA ↑
PC ↑
Mean ↑
Base (ref.)
59.263
43.430
0.9725
0.2117
0.6655
0.6166
3.9720
3.8826
3.7742
3.8763
Raw Reward Maximization
69.543
56.695
0.9763
0.1200
0.6359
0.5774
3.9897
3.9715
3.7782
3.9131
TABO
68.722
54.267
0.9772
0.1450
0.6434
0.5885
4.0182
4.1391
3.8467
4.0013
(b) FlowGRPO
Table 6: Post-target enhancement for target-aware optimization. The metrics follow Table 4 : target strength is reported by RGB percentile mean and P20, VBench uses subject consistency (SC), dynamic degree (DD), aesthetic quality (Aes.), and their arithmetic mean, and VideoScore2 uses visual quality (VQ), text alignment (TA), physical consistency (PC), and their arithmetic mean. Results are compared within each optimizer and backbone. VADER uses Wan2.1-T2V-14B and FlowGRPO uses Wan2.1-T2V-1.3B. A dash denotes an experiment that is not yet available. Base is a reference only; bold marks the best post-trained value within each panel.
Figure 5: Image-space diagnostic of TABO on Vintage Faded and Oil Painting. Bounded reward optimization is compared with direct raw-reward maximization using training reward, HPSv3, Q-Align aesthetic quality, and Q-Align quality.
Figure 6: Qualitative comparison between raw-reward maximization and TABO on 2D Japanese anime. Raw-reward maximization exhibits more pronounced visual degradation, while TABO better preserves subject appearance and scene content. Top: Raw Reward; Bottom: Ours (TABO).
Figure 7: Qualitative comparison of Base, Flow-GRPO, and our TA-GRPO for 2D Japanese anime animation. TA-GRPO strengthens style alignment while preserving scene semantics and composition.
Table 11: Video RGB evaluator validation results for Anime.
(a) Post-target enhancement
VBench
VideoReward
Method
SC ↑
BC ↑
MS ↑
DD ↑
Aes. ↑
IQ ↑
Mean ↑
VQ ↑
MQ ↑
TA ↑
Mean ↑
Base (ref.)
0.9725
0.9797
0.9932
0.2117
0.6655
0.6888
0.7519
-0.0836
-0.4643
0.1657
-0.1274
Baseline
0.9717
0.9763
0.9929
0.1350
0.6362
0.6824
0.7324
-0.1735
-0.5249
0.1437
-0.1849
Generated-only
0.9684
0.9763
0.9941
0.1150
0.6288
0.6877
0.7284
-0.2787
-0.4970
0.1179
-0.2193
Graph-only
0.9770
0.9803
0.9940
0.1400
0.6444
0.7218
0.7429
-0.1349
-0.5329
0.1712
-0.1656
Appendix
Table 12: Complete post-target and post-neutral VBench and VideoReward results on Anime video (Wan2.1-T2V-14B). SC, BC, MS, DD, Aes., and IQ denote VBench subject consistency, background consistency, motion smoothness, dynamic degree, aesthetic quality, and imaging quality; VQ, MQ, and TA denote VideoReward visual quality, motion quality, and text alignment. Base is a reference; bold and underlining indicate the best and second-best post-trained values within each panel, respectively, with ties determined at the displayed precision.
(a) VADER
Attribute
VBench
VideoScore2
Method
Leakage ↓
SC ↑
DD ↑
Aes. ↑
Mean ↑
VQ ↑
TA ↑
PC ↑
Mean ↑
Base (ref.)
0.0000
0.9706
0.3467
0.6689
0.6621
4.3207
4.4130
4.1911
4.3083
Raw Reward Maximization
0.0341
0.9742
0.3050
0.6526
0.6439
4.3217
4.3756
4.1806
4.2926
TABO
0.0101
0.9738
0.3050
0.6546
0.6445
4.3854
4.4665
4.2143
4.3554
(b) Flow-GRPO
Appendix
Table 13: Post-neutral preservation for target-aware optimization. The metrics follow Table 4 : leakage is the mean increase in the attribute-specific RGB logit relative to Base and is lower-is-better; VBench and VideoScore2 report the same component metrics and arithmetic means as in the post-target table. Results are compared within each optimizer and backbone.
Modality / tag
Pair count
Preference rate
Target–Base AUROC
Image / Vintage Faded
2,000
0.9265
0.823734
Image / Oil Painting
2,000
0.9395
0.841797
Image / Muscular
2,000
0.7455
0.686098
Video / Anime
250
0.9200
0.871304
Appendix
Table 14: Latent reward-model validation and checkpoint-selection statistics.
Reinforcement learning, particularly Group Relative Policy Optimization (GRPO), has emerged as an effective framework for post-training visual generative models with human preference signals. However, its effectiveness is fundamentally limited by coarse reward credit assignment. In modern visual generation, multiple reward models are often used to capture heterogeneous objectives, such as visual quality, motion consistency, and text alignment. Existing GRPO pipelines typically collapse these rewards into a single static scalar and propagate it uniformly across the entire diffusion trajectory. This design ignores the stage-specific roles of different denoising steps and produces mistimed or incompatible optimization signals. To address this issue, we propose Objective-aware Trajectory Credit Assignment (OTCA), a structured framework for fine-grained GRPO training. OTCA consists of two key components. Trajectory-Level Credit Decomposition estimates the relative importance of different denoising steps. Multi-Objective Credit Allocation adaptively weights and combines multiple reward signals throughout the denoising process. By jointly modeling temporal credit and objective-level credit, OTCA converts coarse reward supervision into a structured, timestep-aware training signal that better matches the iterative nature of diffusion-based generation. Extensive experiments show that OTCA consistently improves both image and video generation quality across evaluation metrics.
Rui Li, Ke Hao, Yuanzhi Liang +4
University of Science and Technology of China HeFei, China · Shanghai Jiao Tong University ShangHai, China · TeleAI ShangHai, China
Visual reward models are essential for evaluating and improving visual generation models, yet existing approaches typically map task conditions and candidate outputs directly to scalar rewards, leaving implicit what should be evaluated for each individual case. We introduce Think Before You Score, a paradigm that explicitly determines what matters for each case before judging how well the candidate performs. Following this principle, we propose the Thinking Reward Model (TRM), which formulates case-adaptive rubrics, performs rubric-guided assessment, and produces fine-grained pointwise rewards. We further observe that conventional pairwise preference optimization can induce score polarization, and introduce Pairwise Dual-Group Relative Policy Optimization (PD-GRPO), which leverages pairwise supervision to improve reward discrimination while preserving fine-grained pointwise scoring. Extensive experiments on image generation and editing reward-modeling benchmarks demonstrate that TRM achieves state-of-the-art performance among open-source reward models while remaining highly competitive with proprietary alternatives. Moreover, using TRM as a reward for reinforcement learning consistently improves diverse visual generation models, demonstrating that its fine-grained, case-adaptive rewards translate into effective optimization signals for visual generation.
Conventional reinforcement learning strategies for visual generation typically employ sample-wise reward functions, yet this practice frequently results in reward hacking that degrades image diversity and introduces visual anomalies. To address these limitations, we present a novel framework that finetunes generative models using distribution-wise rewards, ensuring better alignment with real-world data distributions. Unlike rewards that evaluate samples individually, distribution-wise reward accounts for the data distribution of the samples, mitigating the mode collapse problem that occurs when all samples optimize towards the same direction independently. To overcome the prohibitive computational cost of estimating these rewards, we introduce a subset-replace strategy that efficiently provides reward signals by updating only a small subset of a generated reference set. Additionally, we apply RL to optimize post-hoc model merging coefficients, potentially mitigating the train-inference inconsistency caused by introducing stochastic differential equation (SDE) in regular RL practices. Extensive experiments show our approach significantly improves FID-50K across various base models, from 8.30 to 5.77 for SiT and from 3.74 to 3.52 for EDM2. Qualitative evaluation also confirms that our method enhances perceptual quality while preserving sample diversity.
Ruihang Li, Mengde Xu, Shuyang Gu +4
University of Science and Technology of China · Shanghai Innovation Institute · Hunyuan Frontier Lab, Tencent +1