RiDEx: Reward-Informed Directional Exploration for Diffusion Reinforcement Learning
Authors: Song Yan, Chenfeng Wang, Wei Zhai, Xingchen Wang, Tao Zhang, Yawen Shao, Shi Pan, Min Li, Zheng-Jun Zha
Organizations: University of Science and Technology of China · Li Auto Inc. · Huazhong University of Science and Technology · Tongyi Lab · Xi’an High-tech Research Institute
Reward-based diffusion post-training updates model parameters from task rewards. While existing approaches improve sampling and policy optimization, they still rely largely on isotropic random perturbations in a high-dimensional latent space for exploration. Such variation often spends the perturbation budget on changes with little influence on the target reward, making exploration inefficient. We study where diffusion models should explore and how to learn from that exploration. We introduce Reward-Informed Directional Exploration (RiDEx), which concentrates local exploration in reward-relevant subspaces of the denoising velocity field. Conditional residuals or differentiable reward gradients propose the subspace, and equal-and-opposite perturbations at the same generation state measure its terminal reward response. The reward difference supplies signed local supervision in the explored coordinates, while all generated images contribute to global DiffusionNFT learning. RiDEx thereby couples exploration-space selection with feedback about the changes it produces. On Stable Diffusion 3.5 Medium, RiDEx attains the highest peak rewards among the evaluated methods on four alignment and preference tasks. Its gradient-direction instantiation achieves 15x and 29x target-attainment speedups over DiffusionNFT on HPSv2 and PickScore, measured in H200 GPU hours. Controlled experiments confirm that reward-aligned directions elicit stronger responses at equal perturbation magnitude, and that applying their measured credit improves curve-average reward across all four tasks.
Figures & tables
Figure 1: Reward-informed exploration turns targeted changes into learning signals. Left: At equal perturbation norm, unrelated variations leave the chair blue, whereas a reward-relevant change targets the requested color. Center: RiDEx proposes directions, scores paired endpoints, and supervises their originating state. Right: Preference target attainment in H200 GPU hours. The schematic depicts RiDEx-Grad; Eqs. 3 , 7 , and 11 define the interventions, normalized credit, and CFG update.
Figure 2: Reward-efficient learning in H200 GPU hours. Dashed lines mark the shared target in each panel, with attainment measured at the first checkpoint reaching that score. Table 2 gives the corresponding GPU times. RiDEx denotes RiDEx-CFG on GenEval and OCR and RiDEx-Grad on HPSv2 and PickScore.
Figure 3: Qualitative comparison across alignment and preference benchmarks. Columns span GenEval (counting and color binding), OCR (text legibility in natural scenes), HPSv2, and PickScore prompts, comparing pretrained SD3.5-M, DiffusionNFT, and RiDEx.
Method
GenEval
OCR
HPSv2
PickScore
Flow-GRPO [ 24 ]
0.9500
0.9200
31.6000
23.5000
ReFL [ 43 ]
—
—
35.8073
23.9211
ExploreNet-GRPO [ 21 ]
—
—
30.1798
22.1234
DiffusionNFT [ 44 ]
0.9501
0.9285
34.2538
22.6407
RiDEx
0.9733
0.9683
38.5154
24.0847
Table 1: Reward scores on SD3.5 Medium. Bold marks the highest peak reward for each task. HPSv2 is scaled by 100; PickScore is reported in logit units.
Task
Target
NFT (h)
RiDEx (h)
Speedup
GenEval
0.9501
31.39
12.07
2.6×
OCR
0.9285
16.44
8.22
2.0×
HPSv2
34.2538
114.63
7.64
15.0×
PickScore
22.6407
222.52
7.67
29.0×
Table 2: H200 GPU hours to matched reward targets. Approximate times are read from the target-attaining Figure 2 checkpoints; speedups use unrounded coordinates. The same target is applied to both methods.
Figure 4: The exploration space must match the target reward. Seven directions are compared with same-state CFG at equal radius and query count. Zero marks the CFG response. Points and bars give the prompt-cluster mean and its 95% interval.
Figure 5: Local learning depends on direction, state, and credit application. (a,b) Held-out branch advantage when changing the space or the source of its coefficients (95% bootstrap intervals). (c–f) Learning curves comparing full credit (circles), measurement only (squares), and paired collection (triangles) across training epochs, summarized in Table 3 .
Figure 6: Reward relevance is visible at the endpoint. (a–d) Rotation toward an orthogonal direction at fixed radius eliminates reward gain. (e–h) Rewards of 1,024 branch pairs relative to their clean endpoint R0 . Bars show 95% prompt-cluster intervals.
Arm
GenEval
OCR
HPSv2
PickScore
Paired collection
0.9123
0.7640
36.3102
23.2071
Measurement only
0.9206
0.9100
37.0169
23.4473
Full credit
0.9260
0.9305
37.7607
23.6770
Table 3: Mean AUC for the credit ablation. For each task, the trapezoidal area under the reward curve in Fig. 5 is divided by the length of the evaluation window shared by the three arms.
Target
Direction
Odd oh
Even eh
Win+ (%)
HPSv2
Own
1.5624
-0.0240
100.0
CFG
0.0674
-0.0017
87.5
Random
0.0023
-0.0005
55.1
Shuffled
0.0017
-0.0008
52.0
PickScore
Own
0.3916
-0.0004
100.0
CFG
0.0215
0.0011
81.2
Table 4: Reward-matched positive probes improve every tested preference endpoint. Means over 256 states per target. Win+ is Pr(R+>R0) under the target evaluator. The odd term measures directional contrast, the even term measures the shared clean-relative response, and their sum gives the positive-branch gain.
Figure 7: Reward relevance vs. readout sensitivity. Continuing Fig. 6 (a–h), panels (i,j) compare full ( g ), readout-parallel ( ∥ ), complement ( ⊥ ), and random directions at equal radius for CLIP (circles) and DINOv2 (squares).
Figure 8: Energy and rank do not replace target alignment. Task reward gains across intervention steps (a,d,g,j), radii (b,e,h,k), and retained basis ranks (c,f,i,l) for CLIP (circles) and DINOv2 (squares), with 95% prompt-cluster intervals.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Task / comparison
Exploration / local update
Endpoints per epoch
NFT updates per epoch
HPSv2 / PickScore
RiDEx-Grad: rank-one, direct update
1,152 for both
1 / 1
GenEval learning
RiDEx-CFG: rank-two, projected update
576 RiDEx / supplementary NFT
1 / 1
OCR learning
RiDEx-CFG: rank-two, projected update
1152 for both
1 RiDEx / 4 NFT
CFG credit controls
RiDEx-CFG: full / measurement / paired
576 per arm
1 per arm
Appendix
Table S1: Collection and optimization budgets by task. On HPSv2 and PickScore, DiffusionNFT and RiDEx use the same number of prompts per round and the same number of endpoints per prompt. The supplementary GenEval NFT run is ci5cav0s ; primary H200 results use iv1oo0cy . Epochs count collection rounds; the final column specifies the number of NFT updates within each round. Reward fields and proposal construction are detailed below.
Setting
RiDEx-Grad value
Backbone / resolution
SD3.5M / 512×512
Random-number initialization
42
LoRA rank / α
32 / 64
Collection solver / evaluations
DPM2 / 10
Collection CFG
1
Intervention index / radius
8 (zero-based) / 0.005
Appendix
Table S2: RiDEx-Grad configuration for HPSv2 and PickScore. Shared checkpoints summarize overlapping trajectories; peak scores are selected separately from recorded evaluations. Collection-policy averaging and raw-adapter evaluation serve separate roles.
Task
r=.001
.0025
.005
.01
GenEval*
0.108
0.185
0.271
0.376
OCR*
0.137
0.210
0.276
0.351
HPSv2
0.665
0.700
0.735
0.784
PickScore
0.659
0.697
0.726
0.767
Appendix
Table S3: Relative magnitude error across the derivative-check radius ladder. These continuous-path measurements complement the sign-agreement results. Zero-action replay error is zero. *GenEval/OCR use a CLIP proxy. Each analysis has 32 states, with GenEval summaries clustered over 31 distinct prompts.
Task
Sign (%)
Error ≤5% (%)
cnorm
GenEval*
96.8
19.4
19.31
OCR*
93.8
31.2
8.93
HPSv2
100.0
0.0
161.86
PickScore
100.0
0.0
61.18
Appendix
Table S4: Testing credit magnitude on frozen states. The upper panel reports sign agreement and 5% magnitude accuracy at r=0.005 , with the task/random gradient norm ratio cnorm from a separate check. Asterisks mark CLIP proxies. The lower panel counts correctly ordered absolute reward gaps among 496 state pairs per scorer. Paired predictions use r=0.005 .
Figure S1 : Directional slopes along the continuous scoring path. The left panels show relative magnitude error between autodiff and centered differences across four radii. The right panels show sign agreement. Each task has 32 states, with 31 distinct GenEval prompts. Intervals are aggregated by prompt.
Task
Basis
Eff. rank
Captured
Transfer
32/64
32/64
F/R
GenEval
CLIP
31.2/62.2
0.28/0.34
0.05/0.67
GenEval
DINO
32.0/64.0
0.52/0.68
0.08/0.81
OCR
CLIP
32.0/64.0
0.25/0.28
0.05/0.64
OCR
DINO
32.0/64.0
0.46/0.63
0.06/0.79
HPSv2
CLIP
31.6/62.7
0.24/0.29
0.12/0.64
Appendix
Table S5: Local re-estimation improves held-out readout response. F/R denotes response cosine for frozen/re-estimated bases. Effective rank and captured fraction are reported at sketch widths 32/64. All task/readout pairs show higher response correspondence with state-specific re-estimation.
Figure S2 : Finite readout spaces are state-dependent. Widths 32/64, all retained width-64 spectral positions, and paired frozen-versus-re-estimated basis transfer are shown. Re-estimation raises response cosine by roughly 0.514–0.616 for CLIP and 0.731–0.748 for DINOv2. State-level paired differences are aggregated over 16 prompt clusters per condition. The preference tasks use shared probe states and readouts, so their spectra describe the same local readout geometry.
Figure S3 : Readout sensitivity and preference gain favor different directions. Full, parallel, complement, and random directions have equal radius. Upper panels show reward gains; lower panels show readout-response norms. The two preference tasks and CLIP/DINO bases reveal the response under distinct objectives and readouts.
Task
DiffusionNFT
RiDEx
GenEval
0.9432 (70)
0.9733 (110)
OCR
0.9285 (40)
0.9683 (30)
HPSv2
34.2538 (150)
38.5154 (200)
PickScore
22.6407 (290)
24.0847 (300)
Appendix
Table S6: Peak rewards in the supplementary evaluation trajectories. Entries report each trajectory’s maximum reward, with its collection epoch in parentheses. GenEval here uses reference ci5cav0s ; the main H200 comparison uses iv1oo0cy . Both retain the evaluation-reward field specified in the implementation details.
Figure S4 : RiDEx-Grad preference feedback persists throughout the recorded training runs. All 226 HPSv2 and 323 PickScore events contribute terminal gains, realized credit-sign agreement, and saved-state credit change. The final-30-round sign agreements are 90.73% and 91.77%. Credit-sign agreement and credit change measure the post-update action at the saved state. Terminal gains use display units (HPSv2 ×100 ; PickScore ×26 ). Credit and update projections retain canonical training units.
Figure S5 : Local and global gradients contribute distinct directions. The four tasks contain 134/416/226/323 training events (1,099 total). Top: raw local/global gradient cosine; bottom: raw norm ratio. Gray rugs mark the 177 zero-local-norm OCR events, for which cosine is undefined. CFG-variant statistics precede projection, trust scaling, and norm matching; preference statistics precede gradient addition and Adam. Norm ratios describe these pre-processing gradients; the optimizer applies the task-specific combination rule.
Figure S7 : Optimization scale and collection response across four tasks. Columns show policy loss, reference-prediction MSE, parameter-step norm, and all-endpoint minus clean-reward gap. The GenEval diagnostic comparator uses the reference run specified in the source data. Shared horizons are 133/415/215/295; marked gaps indicate unavailable fields. OCR NFT updates are averaged within epoch, and log/symlog axes preserve its dynamic range. Objective values are interpreted within their respective loss definitions.
Figure S8 : Random paired collection as an endpoint-learning control. Rows show reward, policy loss, and reference-prediction penalty. A0 covers all four tasks through epoch 100; A1 covers GenEval through epoch 70 and OCR through epoch 80.
Reinforcement learning improves diffusion models using rewards on generated images, but repeated denoising and gradient estimation make training expensive. We introduce \method, an efficient method that constructs useful update directions by regressing centered rewards on centered final denoised images. It requires neither reward derivatives nor adjoint backpropagation through the sampling trajectory. A unified first-variation formulation identifies the value-gradient target with or without a trajectory-KL penalty. A population analysis under Gaussian noising explains the regression geometry, while a subspace analysis motivates clean-image coordinates. The implementation adapts rollout bandwidth and query weights to reward continuity: local rollouts suit smooth rewards, whereas broader groups provide informative contrasts for discrete rewards. Trained jointly on five rewards, one SD3.5-Medium model matches or exceeds DiffusionNFT's joint model on seven of eight metrics, reaching GenEval 0.94, OCR 0.92, and PickScore 23.82 at about one fifth of its estimated training compute. In single-reward training, \method reaches matched reward thresholds 2.1--6.7× faster than reproduced DiffusionNFT on identical hardware. Ablations support the clean-image estimator, direct direction transfer, and reward-dependent rollout choices. Code is available at https://github.com/jy-wang-tsinghua/DIFFUSIONRVGE.
Jiayuan Wang, Tongyao Pang
Qiuzhen College, Tsinghua University, Beijing 100084, China · Yau Mathematical Sciences Center, Shuangqing Complex, Tsinghua University, Beijing 100084, China
Aligning diffusion models with human preferences usually relies on a sparse terminal reward evaluated on the final generated samples, presenting a severe temporal credit-assignment challenge across the multi-step denoising process. We propose Latent Reward Registers, a mechanism that estimates terminal preference directly from intermediate noisy latents by prepending learnable, position-free register tokens to the input sequence of a frozen Diffusion Transformer (DiT). This independent readout mechanism extracts latent reward evidence without altering the generator's hidden states or velocity field. The resulting dense, differentiable reward signal throughout the full denoising process facilitates two alignment strategies. For training, Reward-Gradient On-Policy Distillation (RG-OPD) distills reward-guided updates along on-policy trajectories, bypassing the computationally expensive rollouts of standard policy gradients. For inference, Reward-Guided Sampling (RGS) steers trajectories via magnitude-matched reward gradients without parameter updates. Empirically, at high noise levels (u = 0.8), the registers reach the highest pairwise accuracy among the evaluated latent reward models. Furthermore, RG-OPD outperforms online reinforcement learning baselines while reducing GPU hours by up to 33x, and RGS establishes a new state-of-the-art among training-free methods, strictly enhancing both alignment and perceptual metrics. Code and weights are available at https://github.com/Guanys-dar/latent-reward-register
Yuanshen Guan, Zipeng Feng, Chengru Song +2
1Kling Team · University of Science and Technology of China
Reinforcement learning (RL) post-training provides a direct way to align diffusion models with human preferences and task-specific rewards. However, current RL algorithms for diffusion models remain fragmented: reverse-trajectory methods rely on discretized likelihood ratios, whereas forward-matching methods train on reward-labeled noising versions of the rollout samples. This paper shows that these seemingly different losses arise from a single path-space principle. Starting from the regularized diffusion-RL objective, we use importance sampling between sampling SDEs to obtain an explicit policy-gradient estimator on trajectory space. The estimator contains the stochastic Itô integral underlying Flow-GRPO-type updates; we derive an equivalent variance-reduced value-gradient form that recovers the forward-matching structure of AWM and DiffusionNFT. This identifies the empirical gap between these method families as a variance-reduction effect rather than a difference in RL principle. The derivation yields a unified design space organized by value-gradient estimation, weight functions, and sampling choices. Within this space, we propose a multi-sample KDE value-gradient estimator that reuses rollout groups, together with scale-bounded weight families that retain stable existing recipes while excluding singular ones. Experiments on SD3.5-M and Qwen-Image models validate the variance-reduction explanation and show that the resulting recipe improves over prior diffusion-RL baselines.
Yixian Xu, Yuanrui Zhang, Shengjie Luo +3
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Huazhong University of Science and Technology