AdvFidelity: Unifying Preference Alignment and Few-Step Image Generation within Reinforcement Learning

Sicong Kang; Fangao Zeng; Yuhan Li; Hao Zhou; Mengfei Xu; Wenxiang Shang; Wei Li; Bingbing Ni

cs.CVOct 21, 2026

AdvFidelity: Unifying Preference Alignment and Few-Step Image Generation within Reinforcement Learning

Authors: Sicong Kang, Fangao Zeng, Yuhan Li, Hao Zhou, Mengfei Xu, Wenxiang Shang, Wei Li, Bingbing Ni,

Organizations: Taobao & Tmall group of Alibaba · Shanghai Jiao Tong University

Abstract

Practical image generation requires both human preference alignment and few-step efficiency, yet combining reinforcement learning (RL) with distillation is often multi-stage and may erode RL gains We observe that few-step RL progressively improves structural coherence, even with specialized rewards, but leaves low-level artifacts that these rewards may fail to penalize. To address this gap, we propose AdvFidelity, which integrates artifact-sensitive fidelity supervision into RL through an online adversarial reward. This enables joint reward alignment and high-fidelity few-step generation within a single RL stage, without distribution matching distillation (DMD) or other explicit distillation objectives. We train a discriminator to rank high-quality references (positives) above content-matched reconstructions (negatives) produced by the current generator. Reconstructing moderately noised references largely preserves style and scene layout while exposing low-level degradations. This encourages the discriminator to assess artifacts rather than content differences. With negatives reconstructed online, alternating updates adapt fidelity supervision to the generator's evolving failure modes. References can come from existing target-domain data or full-step model samples. Built on LeapAlign, AdvFidelity backpropagates gradients from both rewards through the few-step sampling trajectories used at inference. Across two model backbones and three task settings, AdvFidelity matches full-step LeapAlign in alignment, task performance, and overall generation quality and remains highly competitive with other evaluated full-step RL methods, using only 4 or 8 steps without CFG.

Figures & tables

Panel of Figure 1 — Figure 1: Motivation of AdvFidelity.

Figure 2: Framework of AdvFidelity. (a) Joint RL update: the few-step generator $G_{\theta}$ ( $\pi_{\theta}$ ) rolls out a trajectory from Gaussian noise $x_{1}$ to the generated image $x_{0}$ , and is updated with both the task reward $r_{\mathrm{spec}}$ and the fidelity reward $D_{\phi}$ . (b) Content-matched pair: a high-quality reference image $x^{+}$ is perturbed with moderate noise and reconstructed by the current $G_{\theta}$ to form a matched negative $x^{-}$ . (c) Online update: the discriminator $D_{\phi}$ is trained on these pairs with a BT loss and supplies the fidelity reward for subsequent generator updates. Each training iteration performs (a) once and then repeats pair construction and the discriminator update in (b)–(c).

		In-Domain	Out-of-Domain
Methods	NFE	HPSv2	PickScore	ImgRwd	UR-Align	UR-IQ	HPSv3 †
Pre-trained Models
FLUX.1-dev	50	0.3083	22.6970	1.0210	3.4428	3.4792	13.3406
RL Methods
DanceGRPO	50	0.3451	23.1186	1.2347	3.4660	3.6199	14.8336
MixGRPO	50	0.3692	23.5184	1.6155	3.4393	3.6241	14.7530

Table 1: Quantitative comparison of preference alignment on FLUX.1-dev. Bold and underlined values denote the best and second best results. Higher is better. † HPSv3 is shaded and reported for reference only, as it assigns unreliable scores to artifact-laden images. ∗ Officially released RTDMD models; unmarked RTDMD rows are retrained under the same protocol as AdvFidelity.

Methods	NFE	Overall	Single Obj	Two Obj	Count	Color	Pos	AttrB
Pre-trained Models
FLUX.1-dev	50	0.6481	99.38	85.10	66.56	76.06	19.00	42.75
RL Methods
DanceGRPO	50	0.6775	99.38	90.15	69.38	76.33	22.25	49.00
MixGRPO	50	0.7232	99.69	93.69	80.00	80.05	24.25	56.25
ReFL	50	0.7011	99.38	92.68	69.06	75.80	26.75	57.00

Table 2: Compositional generation results on FLUX.1-dev, with HPSv2 as the training reward. Bold and underlined values denote the best and second best results. Higher is better. ∗ Officially released RTDMD models; unmarked RTDMD rows are retrained under the same protocol as AdvFidelity.

Figure 3: Qualitative comparisons on (a) HPDv2 preference-alignment prompts and (b) GenEval compositional prompts. Full-step FLUX.1-dev and LeapAlign are compared with 4- and 8-step RTDMD and AdvFidelity. Please zoom in for artifact details

			In-Domain	Out-of-Domain
Methods	CFG	NFE	DINO	Realism	Consistency	Alignment	Coherence
Qwen-Image-Edit-2511	$\checkmark$	40	0.7426	5.96 ${\scriptstyle\pm 0.86}$	8.60 ${\scriptstyle\pm 0.49}$	8.77 ${\scriptstyle\pm 0.45}$	7.91 ${\scriptstyle\pm 0.51}$
LeapAlign	$\times$	40	0.7793	8.06 ${\scriptstyle\pm 0.62}$	9.35 ${\scriptstyle\pm 0.40}$	9.03 ${\scriptstyle\pm 0.44}$	8.19 ${\scriptstyle\pm 0.49}$
LeapAlign	$\checkmark$	40	0.7874	8.33 ${\scriptstyle\pm 0.42}$	9.28 ${\scriptstyle\pm 0.42}$	9.06 ${\scriptstyle\pm 0.39}$	8.30 ${\scriptstyle\pm 0.46}$
AdvFidelity	$\times$	8	0.7929	8.45 ${\scriptstyle\pm 0.36}$	9.54 ${\scriptstyle\pm 0.27}$	9.11 ${\scriptstyle\pm 0.45}$	8.32 ${\scriptstyle\pm 0.42}$

Table 3: Quantitative comparison on virtual dressing with Qwen-Image-Edit-2511.

Figure 4: Qualitative comparison on virtual dressing with Qwen-Image-Edit-2511. AdvFidelity uses 8 steps without CFG; the full-step comparisons use 40 steps, with CFG settings indicated above each column. Please zoom in for artifact details

Methods	Paired	Anchor	$r_{fid}$	HPSv2	PickScore	ImgRwd	UR-Align	UR-IQ
FLUX.1-dev	–	–	–	0.2929	22.3912	0.9532	3.4227	3.7024
LeapAlign + DMD	–	–	–	0.3978	22.9654	1.3976	3.4259	3.7164
Vanilla	$\times$	Base	Frozen	0.3416	22.6735	1.1650	3.4404	3.7123
+Adaptive $r_{\mathrm{fid}}$	$\times$	Base	$\checkmark$	0.3558	22.8637	1.2171	3.4494	3.7215
+Target-anchored positives	$\times$	Target	$\checkmark$	0.3920	23.2588	1.3567	3.4456	3.7141
+Paired Data (Full)	$\checkmark$	Target	$\checkmark$	0.4125	23.7685	1.5255	3.5024	3.7509

Table 4: Ablation study at 8-step inference under HPDv2 setting. Bold values denote the best results.

Figure 5: Qualitative ablation comparison on FLUX.1-dev with HPDv2 prompts at 8-step inference. Components are added cumulatively from the vanilla configuration to the full method.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Iterations	Effective batch size	Resolution	LoRA rank
HPDv2	450	64	$720\times 720$	256
GenEval	500	64	$720\times 720$	256
Virtual dressing	750	16	$1024\times 1024$	128

Appendix

Table 5: Training configuration of each experimental setting. Batch size is the effective global value (per-device batch size

\times

gradient accumulation

\times

8 GPUs); iterations are counted after gradient accumulation.

Figure 6: Score distributions of six automated metrics on reference images and their short reconstructions by the original FLUX.1-dev model, from 1,000 content-matched pairs.

Metric	Mean $\Delta$	Positive win rate (%)
HPSv3	$-0.1748$	38.30
HPSv2	$-0.0087$	79.90
Aesthetic Score	$+0.2955$	95.60
PickScore	$+0.5341$	83.30
ImageReward	$+0.0070$	48.20
CLIPScore	$-0.0152$	40.30

Appendix

Table 6: Pairwise comparison on 1,000 content-matched positive–negative pairs. Mean

\Delta

is the reference score minus the reconstruction score, averaged over pairs; positive win rate is the percentage of pairs in which the reference scores higher.

Figure 7: Additional qualitative results on HPDv2 prompts (left: 8-step, right: 4-step).

Figure 8: Additional qualitative results on GenEval prompts (left: 8-step, right: 4-step).

Figure 9: Additional qualitative results on virtual dressing (left: prompt, mid: input garment, right: 8-step).

RL for Image Generation

Reinforcement learning for image generation trains visual models with reward feedback to improve prompt alignment, quality, and control.

See all 93 papers

RL-Native Distillation: Exploiting Scored Trajectories for Few-Step Image Generation
Aug 10, 2026Yuhan Li, Fangao Zeng, Sicong Kang +7Diffusion Model Distillation Few-Step Diffusion Sampling
Shanghai Jiao Tong University · Taobao & Tmall group of Alibaba
Reinforcing Few-step Generators via Reward-Tilted Distribution Matching
May 25, 2026Yushi Huang, Xiangxin Zhou, Ruoyu Wang +3Diffusion Model Alignment Diffusion Model Distillation
Tencent Hunyuan · Hong Kong University of Science and Technology · Westlake University
AdvDMD: Adversarial Reward Meets DMD For High-Quality Few-Step Generation
Apr 29, 2026Xu Wang, Zexian Li, Litong Gong +2Diffusion Model Distillation Few-Step Diffusion Sampling
Shanghai Jiao Tong University · Alimama Tech

Aug 10, 2026J/K move · Enter open · S save

RL-Native Distillation: Exploiting Scored Trajectories for Few-Step Image Generation

Yuhan Li, Fangao Zeng, Sicong Kang, Mengfei Xu+6

Shanghai Jiao Tong University · Taobao & Tmall group of Alibaba

Diffusion Model Distillation Few-Step Diffusion Sampling RL for Diffusion Models

Open paper AskPDF arXiv

Efficient text-to-image generation requires both reinforcement-learning (RL)-based reward alignment and few-step distillation, yet these procedures are typically performed sequentially, increasing training cost and risking the loss of reward gains during compression. We take an RL-native perspective: diffusion RL already generates reward-scored finite-step trajectories, whose intermediate states provide distillation supervision. Based on this insight, we propose REST (Reward-Enhanced Scored-Trajectory Distillation), a single-stage co-training framework in which a decoupled student learns from the evolving RL teacher's trajectories without changing teacher optimization. Advantage-Modulated Distillation (AMD) transforms rollout advantages into signed weights, strengthening imitation of preferred trajectories and aligning distillation priorities with task value. The resulting framework is general and lightweight, requires no extra image rollouts, no separate distillation dataset, and no adversarial training. Experiments on compositional generation, visual text rendering, and human-preference alignment demonstrate competitive few-step, CFG-free generation with RAM or DiffusionNFT teachers. With only four sampling steps, REST-RAM achieves a DrawBench PickScore of 23.97, outperforming both the 40-step RAM teacher (23.95) and RTDMD (23.71).

Abstract

RL for Image Generation

Explore similar work

RL-Native Distillation: Exploiting Scored Trajectories for Few-Step Image Generation

Reinforcing Few-step Generators via Reward-Tilted Distribution Matching

AdvDMD: Adversarial Reward Meets DMD For High-Quality Few-Step Generation