Reinforcement learning (RL) post-training substantially improves the visual quality, text rendering, and compositional accuracy of diffusion transformers. Because capabilities such as rendering requested text or composing multiple objects are inherently prompt-dependent, a natural hypothesis is that improving them requires learning complex, input-dependent transformations. To isolate how much of these gains an input-independent shift explains, we probe each RL-tuned model with a deliberately simple parameterization: a fixed set of steering vectors added to the block outputs of the frozen base model. On SD3.5-M, these vectors hold 65x fewer parameters than the RL-tuned LoRA (0.29M against 18.8M). We fit them to match the velocity predictions of public RL-tuned checkpoints, with no reward in the objective. Although the vectors are identical for every prompt, token, and sampling seed, they recover 90.0-96.3% of the reward gains of six RL-tuned image models, and 89.8% and 111.9% on two RL-tuned video models. Initialized at zero and trained with online RL, the vectors alone exceed the reward gains of weight-based RL on text rendering (OCR) and human preference (PickScore) on SD3.5-M. A constant shift thus suffices for most of the gain of policy-gradient RL post-training: without new input-dependent transformations, it steers the pretrained generator toward what it can already compute.
Figures & tables
Figure 2: The steering vectors add the same vector to every token of every prompt. Block ℓ on two text-rendering prompts, with a linear layer W , token inputs zi and zj , and block outputs hi and hj . Base + s adds the same s (black) to every token of both prompts; s changes only with the block, guidance pass, step bin and stream. In the RL-tuned model, the weight change gives each input its own shift, ΔWzi=ΔWzj (gray). In block ℓ , s is one vector and ΔW a matrix: on SD3.5-M, all 188 vectors together hold 0.29M parameters, 65 × fewer than the RL-tuned LoRA’s 18.8M.
Figure 3: One steering vector shifts every token of a block output. The drawing shows one stream. The guidance pass p and step bin k select the outlined vector, which is added to every token h .
Prompts
Gain over base
Share (%)
Base
RL method
Fine-tuning
Reward
× seeds
smean
sfit
RL-tuned
smean
sfit
SD3.5-M
Flow-GRPO
LoRA
PickScore
2,048 × 1
+0.952
+1.416
+1.560
61.0
90.8
[89.8, 91.7]
Flow-GRPO, no KL
LoRA
PickScore
2,048 × 1
+1.198
+1.612
+1.693
70.8
95.2
[94.4, 96.1]
Flow-GRPO
LoRA
OCR
1,018 × 1
+0.074
+0.325
+0.344
21.5
94.4
[90.8, 97.9]
Flow-GRPO
LoRA
GenEval
553 × 4
+0.033
+0.260
+0.271
12.2
95.9
[92.0, 99.7]
FLUX.1-dev
Flow-GRPO
LoRA
PickScore
2,048 × 1
+0.985
+1.370
+1.522
64.7
90.0
[88.9, 91.1]
Table 1: Steering vectors recover most of each RL-tuned model’s reward gain. Gains are raw reward points over the base model. Shares are of the RL-tuned model’s gain, with the 95% interval of the sfit share in brackets.
Figure 4: Fitting matches the RL-tuned model’s velocity. The RL-tuned model’s final latent x0 is re-noised to xt , and the loss L trains only s (dashed arrow), with no reward.
Figure 5: Steering vectors recover most of the gain of each RL-tuned image model. Each prompt’s images share one starting noise.
Figure 6: Steering vectors recover most of the gain of both RL-tuned video models. Each prompt’s clips share one starting noise. Wan2.1-1.3B shows four evenly spaced frames per clip, and Wan2.2-5B the last frame, in which the model must circle in red the option that fills the dashed square.
Measure
RL-tuned
Base + sfit
HPSv2.1 gain
+0.
0407
+0.
0393
ImageReward gain
+0.
388
+0.
400
Color energy
550
542
Empty-prompt diversity
850
847
Within-prompt diversity
517
522
Table 2: The steering vectors reproduce effects beyond the training reward. The SD3.5-M PickScore-tuned model with KL on the PickScore test set. Color energy (the summed chroma variance [ 45 ] ) and DINOv3 [ 53 ] diversity are ratios to the base model.
Figure 7: (a) A same-size LoRA recovers detectably more after supervised fine-tuning (DreamBooth) but not after RL, and (b) vectors fitted to parts of SD3.5-M recover most of the RL gain. Bars are 95% intervals, and dashed lines mark the fine-tuned model’s gain.
Share of
Preferred
Reward
Vectors
RL gain (%)
vs Base
vs RL-tuned
PickScore
sfit
90.8
[89.8, 91.7]
64%
51%
sRL
103.6
[102.0, 105.1]
62%
52%
OCR
sfit
94.4
[90.8, 97.9]
60%
50%
sRL
104.6
[102.1, 107.3]
65%
52%
Table 3: RL on the steering vectors alone exceeds the gain of both RL-tuned models. Flow-GRPO trains only the steering vectors sRL of the frozen SD3.5-M, from zero. Shares are of the RL-tuned model’s test-set gain, with 95% intervals, and the sfit rows repeat Tab. 1 . Preferred: the steered image’s win rate in pairwise comparisons (wins plus half the ties) over 50 test prompts and 25 raters ( Sec. G ). All win rates against the base model differ from 50% ( p<0.05 , sign test); none against the RL-tuned models do.
Figure 8: Steering vectors trained by RL alone change the images as the RL-tuned models do. Each prompt’s images share one starting noise.
Figure 9: Steering vectors and RL reach the image through what image tokens read from text. Share of the gain kept when one route reads the base model’s keys and values, with 95% intervals. Image reads image carries the image stream’s own changes and what image tokens read from text in earlier blocks.
SD3.5-M
FLUX.1-dev
Wan2.1-1.3B
Wan2.2-5B
PickScore
PickScore, no KL
OCR
GenEval
PickScore
HPSv2.1
HPSv3
VBVR-Pro
Learning rate
3×10−3
10−2
3×10−3
10−2
3×10−3
3×10−3
3×10−3
3×10−3
LR decay from
3,750
3,750
none
none
1,250
1,250
1,250
1,250
States per update
64
64
64
64
64
32
64
64
Training prompts
10,000
10,000
10,000
10,000
2,048
1,024
512
512
Training states
new
new
new
new
8 fixed
16 fixed
8 fixed
8 fixed
Table S1 : Fitting hyperparameters and cost of the steering vectors for the RL-tuned models of Tab. 1 . The learning rate decays linearly to 10% of its peak from the listed update to the last. GPU-hours count sampling the RL-tuned model, measuring smean and training.
PickScore
OCR
Learning rate
5×10−3
5×10−3
Images per update
576
576
Training updates
1,300
1,500
B200 GPU-hours
123
183
EMA
decay 0.9, updated every 8 updates
Table S2 : Training settings of Flow-GRPO on the steering vectors sRL of Tab. 3 . Other settings follow the PickScore and OCR recipes of Flow-GRPO [ 37 ] .
Base model
Block outputs
Width
Passes
Parameters
SD3.5-M
47
1,536
2
288,768
FLUX.1-dev
76
3,072
1
466,944
Wan2.1-1.3B
30
1,536
2
184,320
Wan2.2-5B
30
3,072
1
184,320
Table S3 : The steering vectors on every base model. Parameters are the block outputs times the width, the passes and two step bins. FLUX.1-dev is guidance-distilled and the VBVR-Pro fine-tune of Wan2.2-5B samples without guidance, so both run one pass.
Base model
RL method
Reward
Kept (%)
SD3.5-M
Flow-GRPO
PickScore
85
Flow-GRPO, no KL
PickScore
94
Flow-GRPO
OCR
83
Flow-GRPO
GenEval
88
FLUX.1-dev
Flow-GRPO
PickScore
94
DanceGRPO
HPSv2.1
86
Table S4 : Two step bins keep most of the energy of each RL-tuned model’s change over the sampling steps. Kept is relative to eight contiguous groups of sampling steps. For unrelated steps it is 25% on SD3.5-M and 25% to 26% on FLUX.1-dev.
Figure S1 : The steering vectors recover most of their RL-tuned model’s gain across training seeds, prompt counts, training states, sampler settings and resolutions. Raw gain over the base model with 95% intervals, dashed lines for the RL-tuned model’s gain and grey bands for the training setting. (a) Training seed j . (b) Training prompts. (c) Training states. (d) Sampling steps N and guidance scale w . (e) Resolution.
Figure S2 : Steering vectors fitted at one sampler setting transfer to other settings and resolutions. Each column shares one prompt and starting noise. † marks the setting of the fit, which also uses 512 px.
Figure S3 : The steering vectors reproduce the loss of color energy and diversity of both PickScore-tuned models. SD3.5-M, as ratios to the base model on the PickScore test set. Bars, mostly hidden by the markers, are 95% intervals where measured.
Figure S4 : On PickScore, the steered base model draws nearly the same images as the RL-tuned model from the same noise. SD3.5-M with the PickScore-tuned model, on one prompt (top) and the empty prompt (bottom). Per block, each column shares a seed.
Share (%)
Base
RL method
Reward
RL-tuned gain
sfit
LoRA
Rank-1 RL LoRA
sfit minus LoRA
SD3.5-M
Flow-GRPO
PickScore
+1.560
90.8
90.8
43.1
0.000 [ − 0.013, +0.013]
Flow-GRPO
OCR
+0.344
94.4
97.0
62.8
− 0.009 [ − 0.022, +0.003]
Flow-GRPO
GenEval
+0.271
95.9
93.4
–
+0.007 [ − 0.004, +0.017]
FLUX.1-dev
Flow-GRPO
PickScore
+1.346
92.1
89.5
–
+0.036 [ − 0.005, +0.079]
DanceGRPO
HPSv2.1
+0.055
99.2
97.4
–
+0.001 [0.000, +0.002]
Table S5 : A same-size LoRA shows no detected advantage over the steering vectors. The same-size LoRA sits on attention output projections, with rank 2 on the image models and rank 1 on the video models, so it has as many parameters as s : 288,768 on SD3.5-M, 466,944 on FLUX.1-dev and 184,320 on Wan2.1 and Wan2.2-5B. The last column is in raw reward points.
Intervention
Parameters
PickScore
OCR
Steering vectors sfit
288,768
+1.43
+0.324
Conditioning offset, fitted
6,144
+0.42
+0.032
Gate scales, fitted
432
+0.26
+0.027
Prompt suffix
0
+0.06
+0.011
CFG scale 6
0
+0.00
+0.032
CFG scale 7.5
0
− 0.05
+0.040
Table S6 : Other input-independent interventions gain less than the steering vectors. The gate scales are the gate scalars of Calibri [ 57 ] , and the conditioning offset is added to the conditioning embedding of the adaptive normalization layers. The prompt suffix appends quality words or “with clear, legible text”.
Figure S5 : The steering vectors draw a new subject’s coarse appearance and miss its details, which the same-size LoRA keeps. FLUX.1-dev models on held-out prompts with the trigger word cttoy, with one starting noise per row. Base + s and the same-size LoRA are fitted to the DreamBooth LoRA.
Figure S6 : The steering vectors can draw a fictional person’s coarse look but another identity, which the same-size LoRA keeps. FLUX.1-dev models of one of the six people on held-out prompts with the trigger word sks, one starting noise per row. Base + s and the same-size LoRA are fitted to the person’s DreamBooth LoRA. Tab. S7 gives all six people.
Person
Base + s
Same-size LoRA
1
68 [65, 70]
82 [80, 84]
2
71 [68, 74]
85 [83, 86]
3
73 [70, 75]
81 [79, 83]
4
79 [77, 81]
89 [85, 92]
5
83 [82, 85]
91 [90, 93]
6
71 [68, 74]
85 [82, 87]
Table S7 : Steering vectors recover part of the identity gain of DreamBooth on six fictional people. Share of the DreamBooth LoRA’s ArcFace similarity gain over FLUX.1-dev in %, with 95% intervals over the 25 held-out prompts.
Figure S7 : The frozen blocks turn the same shift into different velocity changes. A sketch: the same s moves the block outputs A, B and C equally, and their velocities change by different amounts and in different directions.
vs Base
vs RL-tuned
Reward
Vectors
PF
Aes
All
PF
Aes
All
PickScore
sfit
65
67
64
49
53
51
sRL
68
67
62
51
48
52
OCR
sfit
62
67
60
49
47
50
sRL
63
60
65
52
50
52
Table S8 : Pairwise user study on SD3.5-M. Win rate (%) of the steered image, wins plus half the ties, for prompt following (PF), aesthetics (Aes) and overall preference (All). Each cell holds about 500 judgments over 50 test prompts and 25 raters. All win rates against the base model differ from 50% ( p<0.05 , sign test), and none against the RL-tuned models does.
Share of the RL gain (%)
Base
RL-tuned model or score
RL-tuned gain
smean
sfit
SD3.5-M
PickScore
+1.560 [+1.521, +1.599]
61.0 [59.7, 62.3]
90.8 [89.8, 91.7]
PickScore, no KL
+1.693 [+1.648, +1.734]
70.8 [69.4, 72.2]
95.2 [94.4, 96.1]
OCR
+0.344 [+0.319, +0.369]
21.5 [15.4, 26.9]
94.4 [90.8, 97.9]
GenEval
+0.271 [+0.250, +0.293]
12.2 [6.5, 17.5]
95.9 [92.0, 99.7]
FLUX.1-dev
Flow-GRPO, PickScore
+1.522 [+1.481, +1.561]
64.7 [62.8, 66.6]
90.0 [88.9, 91.1]
Table S9 : On every row and score, the steering vectors recover a larger share of the RL gain than smean . Gains are raw reward points on the test sets of Tab. 1 and on the VBVR-Pro out-of-domain split. MQ, TA and VQ are VideoAlign’s motion quality, text alignment and visual quality, and overall is their sum.
Base
RL method
Update
Reward
Steps
Guidance
Resolution
SD3.5-M
Flow-GRPO
LoRA, r=32
PickScore
40
4.5
512
jieliu/SD3.5M-FlowGRPO-PickScore
Flow-GRPO, no KL
LoRA, r=32
PickScore
40
4.5
512
jieliu/SD3.5M-FlowGRPO-PickScore-without-KL
Flow-GRPO
LoRA, r=32
OCR
40
4.5
512
jieliu/SD3.5M-FlowGRPO-Text
Table S10 : The eight RL-tuned models span four base models, four RL methods, and LoRA and full fine-tuning. Full is a fine-tune of all weights, and FLUX.1-dev takes the guidance scale as an input to one pass. Wan2.1 samples with UniPC, and VBVR-Pro with Flow-CPS from a given first frame.
Passes
Streams
Bins
Parameters
Updates
LR
Gain
Mean-difference steering vectors smean
1
Both
8
577,536
–
–
+0.62
2
Both
8
1,155,072
–
–
+0.99
Fitted steering vectors sfit
1
Image
1
36,864
1,800
10−2
+1.11
1
Both
1
72,192
1,800
10−2
+1.30
Table S11 : Separate vectors for the two guidance passes raise the gain more than step bins do. PickScore gain on SD3.5-M. Passes is 2 with separate vectors per guidance pass, Streams is the image stream or both streams, and Bins counts the step bins. The row 2, Both, 2 is the layout of Tab. 1 .
Steering vectors kept
OCR
PickScore
SD3.5-M, test prompts
RL-tuned model
+0.344
+1.560
All steering vectors
+0.325
+1.416
Conditional pass c only
+0.344
+1.500
Unconditional pass ∅ only
− 0.046
− 1.335
Step bin t≥τ only
+0.309
+1.155
Table S12 : The conditional pass, the early step bin and the text stream carry more of the gain than their counterparts. Gain when one part of the full sfit is added alone, in OCR reward and raw PickScore points. SD3.5-M uses the steering vectors and test sets of Tab. 1 , with blocks counted from 0.
Base
RL-tuned model or fit
Reward
Prompts × seeds
Gain over base
Share (%)
RL-tuned models, gain over the base model
Wan2.1-1.3B
GenRL GRPO, RL step 500
VideoAlign TA
64 × 2
+0.058 [ − 0.005, +0.124]
–
GenRL GRPO, RL step 1,000
VideoAlign TA
64 × 2
+0.135 [+0.075, +0.198]
–
GenRL GRPO, RL step 1,500
VideoAlign TA
64 × 2
+0.091 [+0.015, +0.168]
–
SOLACE
Self-confidence
64 × 2
− 1.115 [ − 1.211, − 1.021]
–
RAVEN
CM-GRPO
Sum of five rewards
100 × 2
+0.128 [ − 0.001, +0.262]
–
Table S13 : Two of six video RL-tuned models gain on the reward we evaluate, and the steering vectors recover the gain of both. Top: each video RL-tuned model’s gain on a reward it trained on, or on the rule score for RLVLM. Bottom: sfit fitted to GenRL GRPO at step 1,500 and to VBVR-Pro RLVR, evaluated on 81-frame clips unless a row names 33 frames.
Model
Share (%)
sfit minus model
GenEval test, seeds 0 to 3: 641 fixed, 108 both fail
sfit
91.4 [88.8, 93.9]
–
smean
30.1 [26.0, 34.2]
+61.3 [+56.8, +65.8]
Same-size LoRA
90.6 [87.9, 93.0]
+0.8 [ − 1.7, +3.3]
RL-tuned, next seed
93.3 [91.1, 95.4]
− 1.9 [ − 4.9, +1.0]
Both fail: sfit
33.3 [22.8, 45.5]
–
Table S14 : The steering vectors solve most of the test pairs that the RL-tuned model fixes. Share of the prompt-seed pairs that the base model fails and the RL-tuned model solves that each SD3.5-M model also solves. The last column is in percentage points.
Figure S8 : Replacing what image tokens read from text removes the rendered text. An OCR test prompt. In the last two images, image tokens read the base model’s text keys and values.
Figure S9 : The steered base model mostly draws what the RL-tuned model draws. Each strip shows one SD3.5-M RL-tuned model, and within a strip each column shares one test prompt and starting noise. Both PickScore strips show the same prompts and base images.
Figure S10 : (continued) The OCR-tuned and GenEval-tuned models.
Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.
Diffusion and flow models achieve State-Of-The-Art (SOTA) generative performance, yet many practically important behaviors such as fine-grained prompt fidelity, compositional correctness, and text rendering are weakly specified by score or flow matching pretraining objectives. Reinforcement Learning (RL) fine-tuning with external, black-box rewards is a natural remedy, but diffusion RL is often brittle. Trajectory-based methods incur high memory cost and high-variance gradient estimates; forward-process approaches converge faster but can suffer from distribution drift, and hence reward hacking. In this work, we present Centered Reward Distillation (CRD), a diffusion RL framework derived from KL-regularized reward maximization built on forward-process-based fine-tuning. The key insight is that the intractable normalizing constant cancels under within-prompt centering, yielding a well-posed reward-matching objective. To enable reliable text-to-image fine-tuning, we introduce techniques that explicitly control distribution drift: (i) decoupling the sampler from the moving reference to prevent ratio-signal collapse, (ii) KL anchoring to a CFG-guided pretrained model to control long-run drift and align with the inference-time semantics of the pre-trained model, and (iii) reward-adaptive KL strength to accelerate early learning under large KL regularization while reducing late-stage exploitation of reward-model loopholes. Experiments on text-to-image post-training with GenEval and OCR rewards show that CRD achieves competitive SOTA reward optimization results with fast convergence and reduced reward hacking, as validated on unseen preference metrics.
Reinforcement learning (RL) has become a standard technique for post-training diffusion-based image synthesis models, as it enables learning from reward signals to explicitly improve desirable aspects such as image quality and prompt alignment. In this paper, we propose an online RL variant that reduces the variance in the model updates by sampling paired trajectories and pulling the flow velocity in the direction of the more favorable image. Unlike existing methods that treat each sampling step as a separate policy action, we consider the entire sampling process as a single action. We experiment with both high-quality vision language models and off-the-shelf quality metrics for rewards, and evaluate the outputs using a broad set of metrics. Our method converges faster and yields higher output quality and prompt alignment than previous approaches.
Diffusion and flow-matching models scale because pretraining is supervised regression: a clean sample is noised analytically, and a model regresses against a closed-form target. RL post-training aligns the model with a reward. In image generation, this makes samples compose objects correctly, render text legibly, and match human preferences. Existing methods rely on costly SDE rollouts, reward gradients, or surrogate losses, sacrificing pretraining's regression structure. We show that the structure extends to RL post-training. Under KL-regularized reward maximization, the optimal generative process tilts the clean-endpoint distribution towards samples with higher reward and leaves the noising law unchanged. Combining this with the adjoint-matching optimality condition and a REINFORCE identity, we derive Reinforce Adjoint Matching (RAM): a consistency loss that corrects the pretraining target with the reward. At each step, we draw a clean endpoint from the current model, evaluate its reward, noise it as in pretraining, and regress. No SDE rollouts, backward adjoint sweeps, or reward gradients are required. Like the pretraining objective, RAM is simple and scales. On Stable Diffusion 3.5M, RAM achieves the highest reward on composability, text rendering, and human preference, reaching Flow-GRPO's peak reward in up to 50× fewer training steps.
Andreas Bergmeister, Stefanie Jegelka, Nikolas Nüsken +2
1TU Munich, MCML · 2MIT CSAIL · 3King’s College London +2