RiDEx: Reward-Informed Directional Exploration for Diffusion Reinforcement Learning
Authors: Song Yan, Chenfeng Wang, Wei Zhai, Xingchen Wang, Tao Zhang, Yawen Shao, Shi Pan, Min Li, Zheng-Jun Zha
Organizations: University of Science and Technology of China · Li Auto Inc. · Huazhong University of Science and Technology · Tongyi Lab · Xi’an High-tech Research Institute
Reward-based diffusion post-training updates model parameters from task rewards. While existing approaches improve sampling and policy optimization, they still rely largely on isotropic random perturbations in a high-dimensional latent space for exploration. Such variation often spends the perturbation budget on changes with little influence on the target reward, making exploration inefficient. We study where diffusion models should explore and how to learn from that exploration. We introduce Reward-Informed Directional Exploration (RiDEx), which concentrates local exploration in reward-relevant subspaces of the denoising velocity field. Conditional residuals or differentiable reward gradients propose the subspace, and equal-and-opposite perturbations at the same generation state measure its terminal reward response. The reward difference supplies signed local supervision in the explored coordinates, while all generated images contribute to global DiffusionNFT learning. RiDEx thereby couples exploration-space selection with feedback about the changes it produces. On Stable Diffusion 3.5 Medium, RiDEx attains the highest peak rewards among the evaluated methods on four alignment and preference tasks. Its gradient-direction instantiation achieves 15x and 29x target-attainment speedups over DiffusionNFT on HPSv2 and PickScore, measured in H200 GPU hours. Controlled experiments confirm that reward-aligned directions elicit stronger responses at equal perturbation magnitude, and that applying their measured credit improves curve-average reward across all four tasks.
Figures & tables
Figure 1: Reward-informed exploration turns targeted changes into learning signals. Left: At equal perturbation norm, unrelated variations leave the chair blue, whereas a reward-relevant change targets the requested color. Center: RiDEx proposes directions, scores paired endpoints, and supervises their originating state. Right: Preference target attainment in H200 GPU hours. The schematic depicts RiDEx-Grad; Eqs. 3 , 7 , and 11 define the interventions, normalized credit, and CFG update.
Figure 2: Reward-efficient learning in H200 GPU hours. Dashed lines mark the shared target in each panel, with attainment measured at the first checkpoint reaching that score. Table 2 gives the corresponding GPU times. RiDEx denotes RiDEx-CFG on GenEval and OCR and RiDEx-Grad on HPSv2 and PickScore.
Figure 3: Qualitative comparison across alignment and preference benchmarks. Columns span GenEval (counting and color binding), OCR (text legibility in natural scenes), HPSv2, and PickScore prompts, comparing pretrained SD3.5-M, DiffusionNFT, and RiDEx.
Method
GenEval
OCR
HPSv2
PickScore
Flow-GRPO [ 24 ]
0.9500
0.9200
31.6000
23.5000
ReFL [ 43 ]
—
—
35.8073
23.9211
ExploreNet-GRPO [ 21 ]
—
—
30.1798
22.1234
DiffusionNFT [ 44 ]
0.9501
0.9285
34.2538
22.6407
RiDEx
0.9733
0.9683
38.5154
24.0847
Table 1: Reward scores on SD3.5 Medium. Bold marks the highest peak reward for each task. HPSv2 is scaled by 100; PickScore is reported in logit units.
Task
Target
NFT (h)
RiDEx (h)
Speedup
GenEval
0.9501
31.39
12.07
2.6×
OCR
0.9285
16.44
8.22
2.0×
HPSv2
34.2538
114.63
7.64
15.0×
PickScore
22.6407
222.52
7.67
29.0×
Table 2: H200 GPU hours to matched reward targets. Approximate times are read from the target-attaining Figure 2 checkpoints; speedups use unrounded coordinates. The same target is applied to both methods.
Figure 4: The exploration space must match the target reward. Seven directions are compared with same-state CFG at equal radius and query count. Zero marks the CFG response. Points and bars give the prompt-cluster mean and its 95% interval.
Figure 5: Local learning depends on direction, state, and credit application. (a,b) Held-out branch advantage when changing the space or the source of its coefficients (95% bootstrap intervals). (c–f) Learning curves comparing full credit (circles), measurement only (squares), and paired collection (triangles) across training epochs, summarized in Table 3 .
Figure 6: Reward relevance is visible at the endpoint. (a–d) Rotation toward an orthogonal direction at fixed radius eliminates reward gain. (e–h) Rewards of 1,024 branch pairs relative to their clean endpoint R0 . Bars show 95% prompt-cluster intervals.
Arm
GenEval
OCR
HPSv2
PickScore
Paired collection
0.9123
0.7640
36.3102
23.2071
Measurement only
0.9206
0.9100
37.0169
23.4473
Full credit
0.9260
0.9305
37.7607
23.6770
Table 3: Mean AUC for the credit ablation. For each task, the trapezoidal area under the reward curve in Fig. 5 is divided by the length of the evaluation window shared by the three arms.
Target
Direction
Odd oh
Even eh
Win+ (%)
HPSv2
Own
1.5624
-0.0240
100.0
CFG
0.0674
-0.0017
87.5
Random
0.0023
-0.0005
55.1
Shuffled
0.0017
-0.0008
52.0
PickScore
Own
0.3916
-0.0004
100.0
CFG
0.0215
0.0011
81.2
Table 4: Reward-matched positive probes improve every tested preference endpoint. Means over 256 states per target. Win+ is Pr(R+>R0) under the target evaluator. The odd term measures directional contrast, the even term measures the shared clean-relative response, and their sum gives the positive-branch gain.
Figure 7: Reward relevance vs. readout sensitivity. Continuing Fig. 6 (a–h), panels (i,j) compare full ( g ), readout-parallel ( ∥ ), complement ( ⊥ ), and random directions at equal radius for CLIP (circles) and DINOv2 (squares).
Figure 8: Energy and rank do not replace target alignment. Task reward gains across intervention steps (a,d,g,j), radii (b,e,h,k), and retained basis ranks (c,f,i,l) for CLIP (circles) and DINOv2 (squares), with 95% prompt-cluster intervals.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Task / comparison
Exploration / local update
Endpoints per epoch
NFT updates per epoch
HPSv2 / PickScore
RiDEx-Grad: rank-one, direct update
1,152 for both
1 / 1
GenEval learning
RiDEx-CFG: rank-two, projected update
576 RiDEx / supplementary NFT
1 / 1
OCR learning
RiDEx-CFG: rank-two, projected update
1152 for both
1 RiDEx / 4 NFT
CFG credit controls
RiDEx-CFG: full / measurement / paired
576 per arm
1 per arm
Appendix
Table S1: Collection and optimization budgets by task. On HPSv2 and PickScore, DiffusionNFT and RiDEx use the same number of prompts per round and the same number of endpoints per prompt. The supplementary GenEval NFT run is ci5cav0s ; primary H200 results use iv1oo0cy . Epochs count collection rounds; the final column specifies the number of NFT updates within each round. Reward fields and proposal construction are detailed below.
Setting
RiDEx-Grad value
Backbone / resolution
SD3.5M / 512×512
Random-number initialization
42
LoRA rank / α
32 / 64
Collection solver / evaluations
DPM2 / 10
Collection CFG
1
Intervention index / radius
8 (zero-based) / 0.005
Appendix
Table S2: RiDEx-Grad configuration for HPSv2 and PickScore. Shared checkpoints summarize overlapping trajectories; peak scores are selected separately from recorded evaluations. Collection-policy averaging and raw-adapter evaluation serve separate roles.
Task
r=.001
.0025
.005
.01
GenEval*
0.108
0.185
0.271
0.376
OCR*
0.137
0.210
0.276
0.351
HPSv2
0.665
0.700
0.735
0.784
PickScore
0.659
0.697
0.726
0.767
Appendix
Table S3: Relative magnitude error across the derivative-check radius ladder. These continuous-path measurements complement the sign-agreement results. Zero-action replay error is zero. *GenEval/OCR use a CLIP proxy. Each analysis has 32 states, with GenEval summaries clustered over 31 distinct prompts.
Task
Sign (%)
Error ≤5% (%)
cnorm
GenEval*
96.8
19.4
19.31
OCR*
93.8
31.2
8.93
HPSv2
100.0
0.0
161.86
PickScore
100.0
0.0
61.18
Appendix
Table S4: Testing credit magnitude on frozen states. The upper panel reports sign agreement and 5% magnitude accuracy at r=0.005 , with the task/random gradient norm ratio cnorm from a separate check. Asterisks mark CLIP proxies. The lower panel counts correctly ordered absolute reward gaps among 496 state pairs per scorer. Paired predictions use r=0.005 .
Figure S1 : Directional slopes along the continuous scoring path. The left panels show relative magnitude error between autodiff and centered differences across four radii. The right panels show sign agreement. Each task has 32 states, with 31 distinct GenEval prompts. Intervals are aggregated by prompt.
Task
Basis
Eff. rank
Captured
Transfer
32/64
32/64
F/R
GenEval
CLIP
31.2/62.2
0.28/0.34
0.05/0.67
GenEval
DINO
32.0/64.0
0.52/0.68
0.08/0.81
OCR
CLIP
32.0/64.0
0.25/0.28
0.05/0.64
OCR
DINO
32.0/64.0
0.46/0.63
0.06/0.79
HPSv2
CLIP
31.6/62.7
0.24/0.29
0.12/0.64
Appendix
Table S5: Local re-estimation improves held-out readout response. F/R denotes response cosine for frozen/re-estimated bases. Effective rank and captured fraction are reported at sketch widths 32/64. All task/readout pairs show higher response correspondence with state-specific re-estimation.
Figure S2 : Finite readout spaces are state-dependent. Widths 32/64, all retained width-64 spectral positions, and paired frozen-versus-re-estimated basis transfer are shown. Re-estimation raises response cosine by roughly 0.514–0.616 for CLIP and 0.731–0.748 for DINOv2. State-level paired differences are aggregated over 16 prompt clusters per condition. The preference tasks use shared probe states and readouts, so their spectra describe the same local readout geometry.
Figure S3 : Readout sensitivity and preference gain favor different directions. Full, parallel, complement, and random directions have equal radius. Upper panels show reward gains; lower panels show readout-response norms. The two preference tasks and CLIP/DINO bases reveal the response under distinct objectives and readouts.
Task
DiffusionNFT
RiDEx
GenEval
0.9432 (70)
0.9733 (110)
OCR
0.9285 (40)
0.9683 (30)
HPSv2
34.2538 (150)
38.5154 (200)
PickScore
22.6407 (290)
24.0847 (300)
Appendix
Table S6: Peak rewards in the supplementary evaluation trajectories. Entries report each trajectory’s maximum reward, with its collection epoch in parentheses. GenEval here uses reference ci5cav0s ; the main H200 comparison uses iv1oo0cy . Both retain the evaluation-reward field specified in the implementation details.
Figure S4 : RiDEx-Grad preference feedback persists throughout the recorded training runs. All 226 HPSv2 and 323 PickScore events contribute terminal gains, realized credit-sign agreement, and saved-state credit change. The final-30-round sign agreements are 90.73% and 91.77%. Credit-sign agreement and credit change measure the post-update action at the saved state. Terminal gains use display units (HPSv2 ×100 ; PickScore ×26 ). Credit and update projections retain canonical training units.
Figure S5 : Local and global gradients contribute distinct directions. The four tasks contain 134/416/226/323 training events (1,099 total). Top: raw local/global gradient cosine; bottom: raw norm ratio. Gray rugs mark the 177 zero-local-norm OCR events, for which cosine is undefined. CFG-variant statistics precede projection, trust scaling, and norm matching; preference statistics precede gradient addition and Adam. Norm ratios describe these pre-processing gradients; the optimizer applies the task-specific combination rule.
Figure S7 : Optimization scale and collection response across four tasks. Columns show policy loss, reference-prediction MSE, parameter-step norm, and all-endpoint minus clean-reward gap. The GenEval diagnostic comparator uses the reference run specified in the source data. Shared horizons are 133/415/215/295; marked gaps indicate unavailable fields. OCR NFT updates are averaged within epoch, and log/symlog axes preserve its dynamic range. Objective values are interpreted within their respective loss definitions.
Figure S8 : Random paired collection as an endpoint-learning control. Rows show reward, policy loss, and reference-prediction penalty. A0 covers all four tasks through epoch 100; A1 covers GenEval through epoch 70 and OCR through epoch 80.
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Huazhong University of Science and Technology