Practical image generation requires both human preference alignment and few-step efficiency, yet combining reinforcement learning (RL) with distillation is often multi-stage and may erode RL gains We observe that few-step RL progressively improves structural coherence, even with specialized rewards, but leaves low-level artifacts that these rewards may fail to penalize. To address this gap, we propose AdvFidelity, which integrates artifact-sensitive fidelity supervision into RL through an online adversarial reward. This enables joint reward alignment and high-fidelity few-step generation within a single RL stage, without distribution matching distillation (DMD) or other explicit distillation objectives. We train a discriminator to rank high-quality references (positives) above content-matched reconstructions (negatives) produced by the current generator. Reconstructing moderately noised references largely preserves style and scene layout while exposing low-level degradations. This encourages the discriminator to assess artifacts rather than content differences. With negatives reconstructed online, alternating updates adapt fidelity supervision to the generator's evolving failure modes. References can come from existing target-domain data or full-step model samples. Built on LeapAlign, AdvFidelity backpropagates gradients from both rewards through the few-step sampling trajectories used at inference. Across two model backbones and three task settings, AdvFidelity matches full-step LeapAlign in alignment, task performance, and overall generation quality and remains highly competitive with other evaluated full-step RL methods, using only 4 or 8 steps without CFG.
Figures & tables
Figure 1: Motivation of AdvFidelity.
Figure 2: Framework of AdvFidelity. (a) Joint RL update: the few-step generator Gθ ( πθ ) rolls out a trajectory from Gaussian noise x1 to the generated image x0 , and is updated with both the task reward rspec and the fidelity reward Dϕ . (b) Content-matched pair: a high-quality reference image x+ is perturbed with moderate noise and reconstructed by the current Gθ to form a matched negative x− . (c) Online update: the discriminator Dϕ is trained on these pairs with a BT loss and supplies the fidelity reward for subsequent generator updates. Each training iteration performs (a) once and then repeats pair construction and the discriminator update in (b)–(c).
In-Domain
Out-of-Domain
Methods
NFE
HPSv2
PickScore
ImgRwd
UR-Align
UR-IQ
HPSv3 †
Pre-trained Models
FLUX.1-dev
50
0.3083
22.6970
1.0210
3.4428
3.4792
13.3406
RL Methods
DanceGRPO
50
0.3451
23.1186
1.2347
3.4660
3.6199
14.8336
MixGRPO
50
0.3692
23.5184
1.6155
3.4393
3.6241
14.7530
Table 1: Quantitative comparison of preference alignment on FLUX.1-dev. Bold and underlined values denote the best and second best results. Higher is better. † HPSv3 is shaded and reported for reference only, as it assigns unreliable scores to artifact-laden images. ∗ Officially released RTDMD models; unmarked RTDMD rows are retrained under the same protocol as AdvFidelity.
Methods
NFE
Overall
Single Obj
Two Obj
Count
Color
Pos
AttrB
Pre-trained Models
FLUX.1-dev
50
0.6481
99.38
85.10
66.56
76.06
19.00
42.75
RL Methods
DanceGRPO
50
0.6775
99.38
90.15
69.38
76.33
22.25
49.00
MixGRPO
50
0.7232
99.69
93.69
80.00
80.05
24.25
56.25
ReFL
50
0.7011
99.38
92.68
69.06
75.80
26.75
57.00
Table 2: Compositional generation results on FLUX.1-dev, with HPSv2 as the training reward. Bold and underlined values denote the best and second best results. Higher is better. ∗ Officially released RTDMD models; unmarked RTDMD rows are retrained under the same protocol as AdvFidelity.
Figure 3: Qualitative comparisons on (a) HPDv2 preference-alignment prompts and (b) GenEval compositional prompts. Full-step FLUX.1-dev and LeapAlign are compared with 4- and 8-step RTDMD and AdvFidelity. Please zoom in for artifact details
In-Domain
Out-of-Domain
Methods
CFG
NFE
DINO
Realism
Consistency
Alignment
Coherence
Qwen-Image-Edit-2511
✓
40
0.7426
5.96 ±0.86
8.60 ±0.49
8.77 ±0.45
7.91 ±0.51
LeapAlign
×
40
0.7793
8.06 ±0.62
9.35 ±0.40
9.03 ±0.44
8.19 ±0.49
LeapAlign
✓
40
0.7874
8.33 ±0.42
9.28 ±0.42
9.06 ±0.39
8.30 ±0.46
AdvFidelity
×
8
0.7929
8.45 ±0.36
9.54 ±0.27
9.11 ±0.45
8.32 ±0.42
Table 3: Quantitative comparison on virtual dressing with Qwen-Image-Edit-2511.
Figure 4: Qualitative comparison on virtual dressing with Qwen-Image-Edit-2511. AdvFidelity uses 8 steps without CFG; the full-step comparisons use 40 steps, with CFG settings indicated above each column. Please zoom in for artifact details
Methods
Paired
Anchor
rfid
HPSv2
PickScore
ImgRwd
UR-Align
UR-IQ
FLUX.1-dev
–
–
–
0.2929
22.3912
0.9532
3.4227
3.7024
LeapAlign + DMD
–
–
–
0.3978
22.9654
1.3976
3.4259
3.7164
Vanilla
×
Base
Frozen
0.3416
22.6735
1.1650
3.4404
3.7123
+Adaptive rfid
×
Base
✓
0.3558
22.8637
1.2171
3.4494
3.7215
+Target-anchored positives
×
Target
✓
0.3920
23.2588
1.3567
3.4456
3.7141
+Paired Data (Full)
✓
Target
✓
0.4125
23.7685
1.5255
3.5024
3.7509
Table 4: Ablation study at 8-step inference under HPDv2 setting. Bold values denote the best results.
Figure 5: Qualitative ablation comparison on FLUX.1-dev with HPDv2 prompts at 8-step inference. Components are added cumulatively from the vanilla configuration to the full method.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Iterations
Effective batch size
Resolution
LoRA rank
HPDv2
450
64
720×720
256
GenEval
500
64
720×720
256
Virtual dressing
750
16
1024×1024
128
Appendix
Table 5: Training configuration of each experimental setting. Batch size is the effective global value (per-device batch size × gradient accumulation × 8 GPUs); iterations are counted after gradient accumulation.
Figure 6: Score distributions of six automated metrics on reference images and their short reconstructions by the original FLUX.1-dev model, from 1,000 content-matched pairs.
Metric
Mean Δ
Positive win rate (%)
HPSv3
−0.1748
38.30
HPSv2
−0.0087
79.90
Aesthetic Score
+0.2955
95.60
PickScore
+0.5341
83.30
ImageReward
+0.0070
48.20
CLIPScore
−0.0152
40.30
Appendix
Table 6: Pairwise comparison on 1,000 content-matched positive–negative pairs. Mean Δ is the reference score minus the reconstruction score, averaged over pairs; positive win rate is the percentage of pairs in which the reference scores higher.
Efficient text-to-image generation requires both reinforcement-learning (RL)-based reward alignment and few-step distillation, yet these procedures are typically performed sequentially, increasing training cost and risking the loss of reward gains during compression. We take an RL-native perspective: diffusion RL already generates reward-scored finite-step trajectories, whose intermediate states provide distillation supervision. Based on this insight, we propose REST (Reward-Enhanced Scored-Trajectory Distillation), a single-stage co-training framework in which a decoupled student learns from the evolving RL teacher's trajectories without changing teacher optimization. Advantage-Modulated Distillation (AMD) transforms rollout advantages into signed weights, strengthening imitation of preferred trajectories and aligning distillation priorities with task value. The resulting framework is general and lightweight, requires no extra image rollouts, no separate distillation dataset, and no adversarial training. Experiments on compositional generation, visual text rendering, and human-preference alignment demonstrate competitive few-step, CFG-free generation with RAM or DiffusionNFT teachers. With only four sampling steps, REST-RAM achieves a DrawBench PickScore of 23.97, outperforming both the 40-step RAM teacher (23.95) and RTDMD (23.71).
Yuhan Li, Fangao Zeng, Sicong Kang +7
Shanghai Jiao Tong University · Taobao & Tmall group of Alibaba
Recent advances in few-step diffusion distillation have enabled efficient image generation, yet aligning these models with human preferences remains challenging. We propose Reward-Tilted Distribution Matching Distillation (RTDMD), a two-stage framework that unifies distribution matching distillation with reward-guided reinforcement learning for few-step flow generators. We show that minimizing the KL divergence to a reward-tilted teacher distribution naturally decomposes into a distribution matching term and a reward maximization term. In the first stage, we introduce Ambient-Consistent Distribution Matching Distillation (AC-DMD), which performs subinterval-wise distribution matching and augments the fake score objective with a consistency regularizer to help the fake score model track the shifting generator distribution under limited updates. In the second stage, we jointly optimize both terms: for the reward maximization term, we derive a hybrid policy gradient that combines a GRPO-style estimator for the stochastic intermediate transitions with direct reward backpropagation through the deterministic final step, and further introduce step-subset GRPO (SubGRPO) to reduce variance. Experiments on SD3, SD3.5, and FLUX.2 demonstrate that RTDMD establishes new state-of-the-art results across preference, aesthetic, and compositional metrics with only 4 inference steps, outperforming previous few-step text-to-image generation methods. Code and models are available at https://github.com/Harahan/RTDMD.
Yushi Huang, Xiangxin Zhou, Ruoyu Wang +3
Tencent Hunyuan · Hong Kong University of Science and Technology · Westlake University
Diffusion models offer superior generation quality at the expense of extensive sampling steps. Distillation methods, with Distribution Matching Distillation (DMD) as a popular example, can mitigate this issue, but performance degradation remains pronounced when sampling steps are limited. Reinforcement learning (RL) has been leveraged to improve the few-step generation quality during distillation, with the potential to even surpass the performance of the teacher model. However, existing approaches are combinatorial in nature, merely integrating an RL process with the distillation process, which introduces unnecessary complexities. To address this gap, we propose AdvDMD, a method that seamlessly unifies DMD distillation and RL. Specifically, AdvDMD employs the adversarially trained discriminator from DMD2 as the reward model, which assigns low scores to generated images and high scores to real ones. It is trained on both intermediate and final states of the denoising process and updated online with the distilled model, enabling a holistic supervision of the sampling trajectories and mitigating reward hacking. We adopt a unified SDE backward simulation and a different training schedule for DMD and RL to enable a more stable and efficient training. Experimental results demonstrate that the 4-step AdvDMD outperforms the original 40-step model for SD3.5 on DPG-Bench, while achieving significant performance gains for SD3 on the GenEval. On Qwen-Image, our 2-step AdvDMD achieves superior performance over TwinFlow.