FewStepRL: Fine-tuning Few-step Diffusion Models with Reinforcement Learning

Haitao Lin; Minsi Ren; Yang Li; Xingtong Ge; Zheyuan Hu; Peiyan Hu; Tailin Wu

cs.LGNov 5, 2026

FewStepRL: Fine-tuning Few-step Diffusion Models with Reinforcement Learning

Authors: Haitao Lin, Minsi Ren, Yang Li, Xingtong Ge, Zheyuan Hu, Peiyan Hu, Tailin Wu

Organizations: Tencent Hunyuan · Westlake University · University of Chinese Academy of Sciences

Abstract

Reinforcement learning (RL) has become a standard post-training method for diffusion models, but most RL objectives roll out the current policy at every iteration, and with a many-step sampler these rollouts typically dominate the cost of training. Few-step models cut this cost, yet they model an average velocity over an interval, whereas existing RL objectives are built for an instantaneous velocity with a many-step sampler. We adapt RL objectives to few-step models systematically in three phases, namely a transformation, a decomposition and an adaptation. The transformation to $x$ -space expresses each objective we consider through a common policy prediction, at the clean end by default and at an explicitly marked level for transition or flow-map objectives, so that objectives stated over densities, velocities or rewards become comparable. The decomposition writes the gradient of each of these objectives as the sum of a self-regression toward a reference and a reward augmentation, so that a method reduces, at the gradient level, to a choice of anchors and weights. The adaptation computes each anchor from a few-step model at the end of its transport or along its tangent, with the instantaneous velocity recovered from the network itself or from an auxiliary network. This yields few-step versions of FlowGRPO, AWM, DiffusionNFT, DiffusionOPSD and ReFL in one modular codebase. In our experiments, reward-gradient-based methods are the most effective when the reward is differentiable and its gradient is affordable; otherwise DiffusionNFT-style methods generally reach high rewards in relatively few iterations despite a sometimes higher per-iteration cost. Extra inference steps show no systematic aggregate benefit in our experiments, although selected outputs visibly improve with two to four times as many sampling steps as in training.

Figures & tables

Panel of Figure 1 — Figure 1: Held-out GenEval2 against the RL iteration in the mixed-reward setting of § 6.1 , from the shared pre-RL few-step checkpoint. Dashed: the original objective applied to the distilled model with the average velocity substituted for the instantaneous velocity (FlowGRPO, AWM, DiffusionNFT). Solid: the corresponding few-step adaptation of this paper (FewStepGRPO, MeanFlowAWM, MeanFlowNFT; § 5 ). Each annotated gap is the adaptation minus the baseline, at the best checkpoint of the baseline and at iteration $400$ ; the two coincide when the baseline is best at iteration $400$ .

Figure 2: Overview of the framework. (a) RL increases the relative mass of the high-reward mode under the rollout distribution $p_{\mathrm{old}}$ of the old policy. The default objective is evaluated at the clean end, on the policy prediction (renoise-and-denoise sample) $\tilde{x}_{\theta}=X\big[\tilde{x}_{t},\,v_{\theta},\,0\big]=\tilde{x}_{t}-t\,v_{\theta}$ , whose reward augmentation attracts it toward high-reward rollout samples and repels it from low-reward ones; in the figure, $\hat{x}^{+}$ and $\hat{x}^{-}$ denote a high-reward and a low-reward rollout sample $\hat{x}^{(i)}$ , respectively. (b) The RL-loss gradient can be decoupled into a self-regression term, visualised as a regression force toward the regression anchor $\tilde{x}_{\mathrm{reg}}$ , and a reward-augmentation term, visualised as a reward force along the displacement $\tilde{x}_{\mathrm{rwd}}-\tilde{x}_{\mathrm{base}}$ from the baseline anchor $\tilde{x}_{\mathrm{base}}$ , which lies between the two modes, to the rewarded anchor $\tilde{x}_{\mathrm{rwd}}$ . A high-reward sample produces attraction along that displacement (blue); a low-reward sample produces repulsion against it (orange), so both updates favour higher reward (§ 4.2 ). (c) For a few-step model, the prediction and the anchors $\{\tilde{x}_{\mathrm{rwd}},\tilde{x}_{\mathrm{base}},\tilde{x}_{\mathrm{reg}}\}$ are computed either at the end of the transport curve, $\tilde{x}_{t}-t\,u$ (u-form), or along the tangent at $\tilde{x}_{t}$ , $\tilde{x}_{t}-t\,v$ (v-form). The instantaneous velocity is recovered from the network itself for MeanFlow, $v=u+(t-r)\,\operatorname{sg}(\partial_{t}u)$ with the detached correction evaluated on the old policy in the implemented methods ( Equation ( 56 )), and from an auxiliary network for DMD, $v=v_{\phi}$ , fitted so that $v_{\phi}(\tilde{x}_{t},t)\approx\mathbb{E}\big[\epsilon-\hat{x}\mid\tilde{x}_{t}\big]$ (§ 4.3 ).

	u-form	v-form, shared- $J$ ( v-ind )	v-form, auxiliary ( v-aux )
$\tilde{x}_{\theta}$	$X\big[\tilde{x}_{t},\,u_{\theta}(\tilde{x}_{t},t,0),\,0\big]$	$X\big[\tilde{x}_{t},\,u_{\theta}(\tilde{x}_{t},t,r)+J,\,0\big]$	$X\big[\tilde{x}_{t},\,u_{\theta}(\tilde{x}_{t},t,r),\,0\big]$
$\tilde{x}_{\mathrm{reg}}$	$X\big[\tilde{x}_{t},\,u_{\mathrm{old}}(\tilde{x}_{t},t,0),\,0\big]$	$X\big[\tilde{x}_{t},\,v_{\mathrm{ind}},\,0\big]$	$X\big[\tilde{x}_{t},\,u_{\mathrm{old}}(\tilde{x}_{t},t,r),\,0\big]$
$\tilde{x}_{\mathrm{base}}$	$X\big[\tilde{x}_{t},\,u_{\mathrm{old}}(\tilde{x}_{t},t,0),\,0\big]$	$X\big[\tilde{x}_{t},\,v_{\mathrm{ind}},\,0\big]$	$X\big[\tilde{x}_{t},\,v_{\phi},\,0\big]$
$\tilde{x}_{\mathrm{rwd}}$ (rollout)	$X\big[\tilde{x}_{t},\,v_{\mathrm{cond}},\,0\big]$	$X\big[\tilde{x}_{t},\,v_{\mathrm{cond}},\,0\big]$	$X\big[\tilde{x}_{t},\,v_{\mathrm{cond}},\,0\big]$
trained $(t,r)$	$(t,0)$	$(t,r)$ with $0\leq r\leq t$	$(t,r)$ ; $v_{\phi}$ queried at $(\tilde{x}_{t},t)$ only

Table 1: The three few-step forms of the prediction and the anchors, with

J=J\big[u_{\mathrm{old}},\,t,\,r\big]

v_{\mathrm{ind}}=u_{\mathrm{old}}+J

and

X\big[\tilde{x}_{t},\,v_{\mathrm{cond}},\,0\big]=\hat{x}

. The labels v-ind and v-aux are used in § 5 .

		FewStepGRPO	FewStepAWM		FewStepNFT		FewStepOPSD
stage	quantity	FewStepGRPO	MeanFlowAWM	DMDAWM	MeanFlowNFT	DMDNFT	-general †	ReFL
rollout		SDE	ODE	ODE	ODE	ODE	ODE with $\tilde{x}_{\pm}$	ODE with $\tilde{x}_{+}$
advantage		—	—	—	—	—	—	$\hat{r}\equiv 1$
queries		$(t,s,r)$ , $s$ drawn as $r$	$(t,r)$	$(t,r)$	$(t,r)$	$(t,r)$	$(t,r)$	$(t,r)$
anchors	$\tilde{x}_{\mathrm{reg}}$	$\varnothing$	$\varnothing$	$\varnothing$	$\tilde{x}_{\mathrm{ind}}$	${\tilde{x}_{\mathrm{ind}}\|_{J=0}}$ ‡	${\tilde{x}_{\mathrm{ind}}\|_{J=0}}$ ‡	${\tilde{x}_{\mathrm{ind}}\|_{J=0}}$ ‡
	$\tilde{x}_{\mathrm{base}}$	$\operatorname{sg}(\mu_{\theta})$	$\operatorname{sg}(\tilde{x}_{\theta})$	$\tilde{x}_{\mathrm{aux}}$	$\tilde{x}_{\mathrm{ind}}$	$\tilde{x}_{\mathrm{aux}}$	${\tilde{x}_{\mathrm{ind}}\|_{J=0}}$ ‡	${\tilde{x}_{\mathrm{ind}}\|_{J=0}}$ ‡

Table 2: Specific implementation of stages of Section 5.1 for each trainer of § 6 . The method-specific quantities

\mu_{\theta}

\bar{\rho}_{\mathrm{GRPO}}

\rho_{\mathrm{AWM}}

\chi_{\mathrm{GRPO}}

\chi_{\mathrm{AWM}}

\sigma_{\mathrm{k}}

\tau_{\mathrm{AWM}}

a_{\pm}

\gamma_{\pm}

and

c_{\mathrm{adv}}

are defined in §§ 5.2 – 5.5 .

anchors	$\tilde{x}_{\theta}$	$\tilde{x}_{\mathrm{reg}}$ , $\tilde{x}_{\mathrm{base}}$
v-ind	$\tilde{x}_{\theta,\mathrm{v}}=X\big[\tilde{x}_{t},\,u_{\theta}(\tilde{x}_{t},t,r)+J,\,0\big]$	$\tilde{x}_{\mathrm{reg}}=\tilde{x}_{\mathrm{base}}=\tilde{x}_{\mathrm{ind}}$
v-aux	$X\big[\tilde{x}_{t},\,u_{\theta}(\tilde{x}_{t},t,r),\,0\big]$	$\tilde{x}_{\mathrm{reg}}={\tilde{x}_{\mathrm{ind}}\|_{J=0}}$ , $\tilde{x}_{\mathrm{base}}=\tilde{x}_{\mathrm{aux}}$

Table 3: Prediction and anchors of the two FewStepNFT variants;

J=J\big[u_{\mathrm{old}},\,t,\,r\big]

Trainers	$n_{\mathrm{it}}$	$c_{\theta}$	$c_{\mathrm{old}}$	$c_{J}$	$c_{\phi}$	$c_{\mathrm{ref}}$	$c_{\mathrm{extra}}$
§ 5.3 , FewStepGRPO	$M_{\mathrm{step}}$	1	1	2	—	1	—
§ 5.4 , MeanFlowAWM	$n_{\mathrm{ts}}$	1	1	2	—	1	—
§ 5.4 , DMDAWM	$n_{\mathrm{ts}}$	1	1	0	1	1	$n_{\mathrm{fm}}$ forwards and backwards of $v_{\phi}$
§ 5.5 , MeanFlowNFT	$n_{\mathrm{ts}}$	1	1	2	—	1	—
§ 5.5 , DMDNFT	$n_{\mathrm{ts}}$	1	1	0	1	1	$n_{\mathrm{fm}}$ forwards and backwards of $v_{\phi}$
§ 5.6 , FewStepOPSD	1	1	1	$0^{\dagger}$	—	1	$2M_{\mathrm{tgt}}$ reward backwards, 2 no-grad forwards of $u_{\mathrm{old}}$

Table 4: Network evaluations per inner iteration.

n_{\mathrm{it}}

: number of inner iterations per outer iteration, each querying every member once;

c_{\theta}

: forward–backward passes of

u_{\theta}

;

c_{\mathrm{old}}

: no-grad forwards of

u_{\mathrm{old}}

for the old-policy anchor;

c_{J}

: maximum extra no-grad forwards of

u_{\mathrm{old}}

for the directional-difference estimate of

J

( Equation ( 87 ) or Equation ( 109 );

0

where the estimator uses no

J

; † with

J

dropped because it cancels, which leaves the loss and its gradient unchanged by Appendix C );

c_{\phi}

: forwards with no gradient of the auxiliary network

v_{\phi}

;

c_{\mathrm{ref}}

: forwards of the frozen reference for the velocity penalty;

c_{\mathrm{extra}}

: extra cost per outer iteration, where

n_{\mathrm{fm}}

is the number of flow-matching steps that fit

v_{\phi}

. The cost of one outer iteration is thus

C=C_{\mathrm{rollout}}+c_{\mathrm{extra}}+n_{\mathrm{it}}\,(c_{\theta}+c_{\mathrm{old}}+c_{J}+c_{\phi}+c_{\mathrm{ref}})

, where

C_{\mathrm{rollout}}

is the plain rollout of Stage 1.

class	method	reward signal	query of $u_{\theta}$	$\tilde{x}_{\mathrm{base}}$	instantaneous field
Group-adv.	FewStepGRPO, § 5.3	$A$	$(t,s)$ , replayed	$\operatorname{sg}(\mu_{\theta})$	Eq. ( 109 )
	MeanFlowAWM, § 5.4	$A$	$(t,r)$	$\operatorname{sg}(\tilde{x}_{\theta})$	Eq. ( 87 )
	DMDAWM, § 5.4	$A$	$(t,r)$	$\tilde{x}_{\mathrm{aux}}$	$v_{\phi}$ baseline; $J=0$
	MeanFlowNFT, § 5.5	$\hat{r}$	$(t,r)$	$\tilde{x}_{\mathrm{ind}}$	Eq. ( 87 )
	DMDNFT, § 5.5	$\hat{r}$	$(t,r)$	$\tilde{x}_{\mathrm{aux}}$	$v_{\phi}$
Reward-grad.	FewStepOPSD, § 5.6	$\nabla R$ , $\hat{r}$	$(t,r)$	$\tilde{x}_{\mathrm{ind}}$	$J=0$ (App. C )

Table 5: The methods compared; ReFL is Example 9 . Class: the first five methods compute scalar rewards for the members of each prompt group and form per-sample advantages using the per-prompt mean and current global-batch standard deviation of Equation ( 83 ), so they are group-advantage-based. The final two obtain their clean-sample update direction from

\nabla R

and are reward-gradient-based; FewStepOPSD still uses

\hat{r}

only to split the ascent/descent weights. Reward signal: the advantage as a signed multiplier (

A

), as a mixing weight (

\hat{r}

, Stage 2), or the reward gradient (

\nabla R

, Equation ( 62 )). Every method except FewStepGRPO uses the deterministic sampler of Equation ( 5 ) in Stage 1. Appendix C establishes when dropping

J

(last column) is admissible.

		OCR accuracy at iteration				first it. with	at the first saved checkpoint with OCR accuracy $>0.82$
trainer	s/iter	50	100	150	200	OCR $\geq 0.85$ (h)	it.	OCR	aesthetic	HPSv3	ImageReward	GenEval2
AnyFlow Init.	—	—	—	—	—	—–	0	0.498	5.24	12.86	1.233	0.363
FewStepGRPO	451	0.599	0.645	0.708	0.776	— a	—	—	—	—	—	—
MeanFlowAWM	496	0.575	0.661	0.755	0.802	— a	—	—	—	—	—	—
DMDAWM	342	0.698	0.820	0.902	0.936	150 (14.3)	200	0.936	4.49	5.05	0.092	0.244
MeanFlowNFT	483	0.785	0.850	0.945	0.934	100 (13.4)	100	0.850	4.99	10.64	0.887	0.306

Table 6: Main results. (a) OCR setting, restricted to the

200

-iteration horizon of Figure 4 . (b) Mixed-reward setting at iteration

400

: s/iter and wall-clock hours to iteration

400

; the training reward and its three summands; the held-out evaluators; GenEval2 at iteration

400

, averaged over iterations

100

–

400

, and at the best checkpoint. Bold: best among the trained methods per comparable column.

Figure 3: OCR setting: the evaluated dimensions against the iteration, over the first

200

iterations. Top row: online evaluation every

50

iterations. Bottom row: offline evaluation at saved checkpoints, every

100

iterations; HPSv3 and ImageReward on the OCR prompts, GenEval2 on its own prompts.

Figure 5: Mixed-reward setting: GenEval2 of each trainer’s best checkpoint within

400

iterations. (a) Left: against the wall-clock time to reach it, s/iter times the iteration, on

8

GPUs. (b) Right: against s/iter. Both panels show the same checkpoints, so they share the vertical axis and the legend, and a trainer moves only horizontally from (a) to (b). The number beside each marker is the iteration of that checkpoint.

Panel of Figure 6 — Figure 6: Mixed-reward setting: the baselines of § 6.1 against the few-step adaptations of this paper (solid), from the shared pre-RL checkpoint. Panels as in Figure 4 . Each annotated gap is the adaptation minus the baseline, at the best checkpoint of the baseline and at iteration $400$ ; the two coincide when the baseline is best at iteration $400$ . GenEval2 for the same runs is shown in Figure 1 .

trainer	s/iter	sampling (s)	update (s)	$n_{\mathrm{it}}$	forwards for $J$	$c_{\phi}$	$n_{\mathrm{fm}}$
FewStepGRPO	423	78.7	344.2	4	8	0	0
MeanFlowAWM	425	78.3	344.5	4	8	0	0
DMDAWM	304	78.2	223.4	4	0	1	5
MeanFlowNFT	435	78.8	354.6	4	8	0	0
DMDNFT	326	78.4	244.7	4	0	1	5
FewStepOPSD	268	224.7	43.3	1	0	0	0

Table 7: Cost of one outer iteration in the mixed-reward setting, split into the sampling time (rollout, reward and, for FewStepOPSD and ReFL, the reward-coupled construction of Equation ( 82 )) and the update time; medians over the timers in the training logs, on

8

H20 GPUs. The small residual between the sum of the two times and s/iter covers the refresh of

u_{\mathrm{old}}

and logging.

				training reward					VBench (held-out)
method	GPUs	s/iter	GPU-h	sum	HPSv3-G	HPSv3-P	MQ	TA	overall	quality	semantic
AnyFlow Init.	—	—	—	16.54	6.42	9.43	2.64	1.85	83.68	85.36	76.98
FewStepGRPO	32	735	1633	15.89	6.24	9.38	2.40	1.23	83.62	85.27	76.98
MeanFlowAWM	32	737	1637	16.66	6.49	9.46	2.68	1.84	83.70	85.36	77.04
DMDAWM	32	821	1824	16.78	6.51	9.53	2.74	1.82	83.64	85.35	76.76
MeanFlowNFT	32	734	1631	16.80	6.53	9.51	2.76	1.80	83.62	85.34	76.75

Table 8: Main results on video generation at iteration

250

. Left: cost. Middle: the aggregate training reward and its four components on

256

held-out prompts. Right: VBench, which no method optimises. Bold: best per comparable column; s/iter is compared only among the 32-GPU rows.

Figure 7: Absolute value of each evaluated dimension against

M_{\mathrm{step}}

, one line per fine-tuned model, with the AnyFlow initialisation (dashed) and the pretrained SD3.5-Medium sampled with classifier-free guidance (dotted) as references. For the fine-tuned models, the three training-reward summands (CLIPScore, HPSv2 and PickScore) show little systematic gain from additional steps, while the held-out HPSv3 and aesthetic score more often decline. Trainer labels are as in Table 5 ; the checkpoint is given in Table S3 . Figure S1 and Figure S2 in Appendix D show the same results as percentage changes from the four-step value, grouped by evaluator and by model, respectively.

Figure 8: Two selected mild-degradation cases; each panel shows the six evaluated scores of that image (CLIPScore, HPSv2, PickScore, HPSv3, aesthetic score and ImageReward). Top, ReFL on the three-cube prompt, where HPSv3 gradually falls from $15.63$ to $14.78$ . Bottom, MeanFlowAWM on “A panda making latte art”, where HPSv3 first rises from $11.34$ to $11.59$ at $M_{\mathrm{step}}=8$ , then falls to $10.77$ at $32$ as an extra cup appears.

Panel of Figure 9 — Figure 9: Two selected OCR-sensitive text-dominance cases under test-time scaling: DMDNFT (top) and MeanFlowNFT (bottom). Within each row, the prompt and seed are fixed and only $M_{\mathrm{step}}$ changes; the values below each image are that image’s rewards. These are selected qualitative failure cases rather than population averages.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$c$	text prompt (conditioning); omitted from an argument list when it is fixed or irrelevant. A bare $c$ always has this meaning; subscripted quantities such as $c_{\mathrm{adv}}$ and $c_{\theta}$ are distinct
$\epsilon\mathchar 24891\penalty\ \epsilon^{\prime}$	standard Gaussian noise, $\epsilon,\epsilon^{\prime}\sim\mathcal{N}(0,I)$ ; a prime marks a fresh rollout-time renoise draw
$t\mathchar 24891\penalty r$	flow-map endpoints, $0\leq r\leq t\leq 1$ ; generation runs from noise at $t=1$ to data at $t=0$ . The unhatted letter $r$ always names a noise level and never a reward; the hatted $\hat{r}$ below is the mixing weight. Individual objectives state any stricter query domain required by divisions by $t$ or $t-r$
$\sigma$	a generic level on the same scale, $\sigma\in[0,1]$ , used as the time of the SDEs of Appendix B
$K$	positive-integer group size : the number of rollouts sharing one prompt, $K\in\mathbb{N}_{\geq 1}$ ; members carry the index $(i)$
$x$	a clean sample . Without a noise-level subscript, an $x$ -symbol denotes a quantity at $\sigma=0$ ; a subscript $t$ , $r$ or $t_{j}$ indicates its level, while $\theta$ , $\mathrm{old}$ , $\mathrm{real}$ or a role label indicates its source. Method-specific quantities have the level of the object they represent; in particular, the MeanFlow quantities of Equation ( 43 ) and the FewStepGRPO prediction and anchors of Equation ( 97 ) are defined at level $r$

Appendix

Table 17

trainer	it.	training reward	HPSv2	PickScore	CLIPScore	aesthetic	OCR
FewStepOPSD	100	1.5692 / 1.5704	0.3520 / 0.3515	0.9205 / 0.9230	0.2968 / 0.2959	6.092 / 6.093	0.400 / 0.405
	200	1.6052 / 1.6055	0.3643 / 0.3633	0.9338 / 0.9356	0.3071 / 0.3066	6.068 / 6.054	0.433 / 0.441
	300	1.6258 / 1.6243	0.3705 / 0.3698	0.9410 / 0.9416	0.3144 / 0.3129	5.984 / 5.983	0.474 / 0.486
	400	1.6406 / 1.6409	0.3733 / 0.3734	0.9464 / 0.9458	0.3208 / 0.3217	5.929 / 5.906	0.550 / 0.493
ReFL	400	1.6166 / 1.6225	0.3651 / 0.3675	0.9355 / 0.9378	0.3160 / 0.3172	5.893 / 5.891	0.468 / 0.546
		s/iter	sampling (s)	update (s)

Appendix

Table S1 : Comparison of retaining

J

and setting

J=0

in FewStepOPSD and in ReFL with

\nabla R

of the full reward. For each trainer, the two runs differ only in whether Equation ( 87 ) is evaluated. Each cell reads “

J

retained /

J=0

”; the runs with

J=0

are those of § 6.3 , and their entries are those of Table 6 , Table S4 and Figure 4 . Top: the runs with

J

retained are evaluated offline (

256

images, seed

12345

, DrawBench prompts, OCR prompts for OCR accuracy), with the same protocol as the runs with

J=0

at iteration

400

. Bottom: cost per iteration, medians over the training logs on

8

GPUs; the runs with

J

retained were not timed on the same machine as those with

J=0

		HPSv3 at $M_{\mathrm{step}}$				% change at $M_{\mathrm{step}}=32$ against $M_{\mathrm{step}}=4$
trainer	it.	4	8	16	32	HPSv3	ImgRwd	GenEval2	Aesth.	CLIP	HPSv2	Pick
FewStepGRPO	400	11.58	11.15	10.49	10.09	$-12.85$	$-4.26$	$-2.89$	$-1.76$	$+0.58$	$-3.82$	$-0.85$
MeanFlowAWM	100	12.30	12.36	12.17	11.85	$-3.71$	$-0.58$	$-6.46$	$-2.98$	$-0.08$	$+0.57$	$+0.07$
DMDAWM	400	13.10	13.24	12.98	12.77	$-2.53$	$+1.50$	$+6.02$	$-2.87$	$-2.47$	$+0.20$	$+0.19$
MeanFlowNFT	300	13.26	13.18	12.96	12.85	$-3.08$	$-1.67$	$+6.81$	$-1.27$	$-0.11$	$+0.27$	$+0.15$
DMDNFT	400	13.49	13.49	13.34	13.13	$-2.67$	$+0.08$	$+5.59$	$-2.09$	$-0.66$	$-0.24$	$-0.05$

Appendix

Table S3 : Test-time scaling in the mixed-reward setting. Left: HPSv3, a held-out evaluator with a clear signal, at each step count. Right: the percentage change of every evaluated dimension between

M_{\mathrm{step}}=4

and

32

. CLIPScore, HPSv2 and PickScore are the three summands of the training reward; HPSv3, ImageReward, GenEval2 and the aesthetic score are held out. The GenEval2 column is reported for completeness only, since its largest change from

4

32

steps (

0.028

) and its largest intermediate change (

0.032

) are within or at the

{\sim}0.03

resolution stated in § 6.1 for

256

images. Trainer labels are as in Table 5 ; “it.” is the checkpoint used.

Figure S1 : Percentage change of each evaluated dimension from its four-step value against

M_{\mathrm{step}}

, one panel per evaluator and one line per model, including the AnyFlow initialisation and the pretrained SD3.5-Medium. Values of SD3.5-Medium outside the plotted range are printed at the top of each panel. The shaded band in the GenEval2 panel marks the

{\sim}0.03

resolution of § 6.1 ; the changes of the fine-tuned models stay within or at this resolution, whereas both references move beyond it.

Figure S2 : The percentage changes of Figure S1 , one panel per model and one line per evaluator. Solid lines are held-out evaluators, and dashed lines are the summands of the training reward.

MeanFlowNFT, weights $w_{\mathrm{CLIP}}{:}w_{\mathrm{HPS}}{:}w_{\mathrm{Pick}}$
	CLIPScore at iteration
method	0	100	200	300	400
FewStepOPSD, $\nabla R$	0.2886	0.2959	0.3066	0.3129	0.3217
ReFL, $\nabla$ of $R$	0.2886	0.2967	0.3076	0.3127	0.3172
ReFL, $\nabla$ of HPSv2	0.2886	0.2845	0.2844	0.2863	0.2883
$1{:}1{:}1$ (default), $K=24$	0.2886	0.2800	0.2815	0.2806	0.2814

Appendix

Table S4 : CLIPScore against the iteration in the mixed-reward setting, the summand of the training reward on which the two groups of trainers disagree. Upper block: the update direction contains

\nabla R

. Lower block: the reward enters as the scalar

A

\hat{r}

. Evaluation protocol as in § 6.1 ; the HPSv2-only ReFL row uses a different training reward. Iteration

0

is the shared pre-RL checkpoint, CLIPScore

0.2886

Diffusion Model Fine-Tuning

Diffusion model fine-tuning adapts pretrained generative models to new concepts, tasks, domains, and preferences through targeted updates.

See all 188 papers

Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View
Aug 14, 2026Yixian Xu, Yuanrui Zhang, Shengjie Luo +3Diffusion Model Alignment Policy Gradient Methods
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Huazhong University of Science and Technology
ReDiF: Resource-Efficient Few-Step Diffusion Distillation via Reinforcement Learning
Dec 28, 2025Amirhossein Tighkhorshid, Zahra Dehghanian, Hamid R. RabieeDiffusion Distillation RL for Diffusion Models
Computer Engineering Sharif University of Technology
TDM-R1: Reinforcing Few-Step Diffusion Models with Non-Differentiable Reward
Mar 8, 2026Yihong Luo, Tianyang Hu, Weijian Luo +1RL for Diffusion Models Reinforcement Learning
Hong Kong University of Science and Technology · The Chinese University of Hong Kong, Shenzhen · hi-Lab, Xiaohongshu Inc +1

Aug 14, 2026J/K move · Enter open · S save

Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View

Yixian Xu, Yuanrui Zhang, Shengjie Luo, Kaiyuan Gao+2

State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Huazhong University of Science and Technology

Diffusion Model Alignment Policy Gradient Methods RL for Diffusion Models Variance Reduction

Open paper AskPDF arXiv

Reinforcement learning (RL) post-training provides a direct way to align diffusion models with human preferences and task-specific rewards. However, current RL algorithms for diffusion models remain fragmented: reverse-trajectory methods rely on discretized likelihood ratios, whereas forward-matching methods train on reward-labeled noising versions of the rollout samples. This paper shows that these seemingly different losses arise from a single path-space principle. Starting from the regularized diffusion-RL objective, we use importance sampling between sampling SDEs to obtain an explicit policy-gradient estimator on trajectory space. The estimator contains the stochastic Itô integral underlying Flow-GRPO-type updates; we derive an equivalent variance-reduced value-gradient form that recovers the forward-matching structure of AWM and DiffusionNFT. This identifies the empirical gap between these method families as a variance-reduction effect rather than a difference in RL principle. The derivation yields a unified design space organized by value-gradient estimation, weight functions, and sampling choices. Within this space, we propose a multi-sample KDE value-gradient estimator that reuses rollout groups, together with scale-bounded weight families that retain stable existing recipes while excluding singular ones. Experiments on SD3.5-M and Qwen-Image models validate the variance-reduction explanation and show that the resulting recipe improves over prior diffusion-RL baselines.

Abstract

Diffusion Model Fine-Tuning

Explore similar work

Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View

ReDiF: Resource-Efficient Few-Step Diffusion Distillation via Reinforcement Learning

TDM-R1: Reinforcing Few-Step Diffusion Models with Non-Differentiable Reward