RL Post-Training in Diffusion Transformers Is Activation Steering
Organizations: Virginia Tech · fal · Georgia Tech · IBM Research
Abstract
Reinforcement learning (RL) post-training substantially improves the visual quality, text rendering, and compositional accuracy of diffusion transformers. Because capabilities such as rendering requested text or composing multiple objects are inherently prompt-dependent, a natural hypothesis is that improving them requires learning complex, input-dependent transformations. To isolate how much of these gains an input-independent shift explains, we probe each RL-tuned model with a deliberately simple parameterization: a fixed set of steering vectors added to the block outputs of the frozen base model. On SD3.5-M, these vectors hold 65x fewer parameters than the RL-tuned LoRA (0.29M against 18.8M). We fit them to match the velocity predictions of public RL-tuned checkpoints, with no reward in the objective. Although the vectors are identical for every prompt, token, and sampling seed, they recover 90.0-96.3% of the reward gains of six RL-tuned image models, and 89.8% and 111.9% on two RL-tuned video models. Initialized at zero and trained with online RL, the vectors alone exceed the reward gains of weight-based RL on text rendering (OCR) and human preference (PickScore) on SD3.5-M. A constant shift thus suffices for most of the gain of policy-gradient RL post-training: without new input-dependent transformations, it steers the pretrained generator toward what it can already compute.
Figures & tables
| Prompts | Gain over base | Share (%) | ||||||||
| Base | RL method | Fine-tuning | Reward | seeds | RL-tuned | |||||
| SD3.5-M | Flow-GRPO | LoRA | PickScore | 2,048 1 | +0.952 | +1.416 | +1.560 | 61.0 | 90.8 | [89.8, 91.7] |
| Flow-GRPO, no KL | LoRA | PickScore | 2,048 1 | +1.198 | +1.612 | +1.693 | 70.8 | 95.2 | [94.4, 96.1] | |
| Flow-GRPO | LoRA | OCR | 1,018 1 | +0.074 | +0.325 | +0.344 | 21.5 | 94.4 | [90.8, 97.9] | |
| Flow-GRPO | LoRA | GenEval | 553 4 | +0.033 | +0.260 | +0.271 | 12.2 | 95.9 | [92.0, 99.7] | |
| FLUX.1-dev | Flow-GRPO | LoRA | PickScore | 2,048 1 | +0.985 | +1.370 | +1.522 | 64.7 | 90.0 | [88.9, 91.1] |
| Measure | RL-tuned | Base + | ||
|---|---|---|---|---|
| HPSv2.1 gain | +0. | 0407 | +0. | 0393 |
| ImageReward gain | +0. | 388 | +0. | 400 |
| Color energy | 550 | 542 | ||
| Empty-prompt diversity | 850 | 847 | ||
| Within-prompt diversity | 517 | 522 | ||
| Share of | Preferred | ||||
|---|---|---|---|---|---|
| Reward | Vectors | RL gain (%) | vs Base | vs RL-tuned | |
| PickScore | 90.8 | [89.8, 91.7] | 64% | 51% | |
| 103.6 | [102.0, 105.1] | 62% | 52% | ||
| OCR | 94.4 | [90.8, 97.9] | 60% | 50% | |
| 104.6 | [102.1, 107.3] | 65% | 52% | ||
| SD3.5-M | FLUX.1-dev | Wan2.1-1.3B | Wan2.2-5B | |||||
| PickScore | PickScore, no KL | OCR | GenEval | PickScore | HPSv2.1 | HPSv3 | VBVR-Pro | |
| Learning rate | ||||||||
| LR decay from | 3,750 | 3,750 | none | none | 1,250 | 1,250 | 1,250 | 1,250 |
| States per update | 64 | 64 | 64 | 64 | 64 | 32 | 64 | 64 |
| Training prompts | 10,000 | 10,000 | 10,000 | 10,000 | 2,048 | 1,024 | 512 | 512 |
| Training states | new | new | new | new | 8 fixed | 16 fixed | 8 fixed | 8 fixed |
| PickScore | OCR | |
|---|---|---|
| Learning rate | ||
| Images per update | 576 | 576 |
| Training updates | 1,300 | 1,500 |
| B200 GPU-hours | 123 | 183 |
| EMA | decay 0.9, updated every 8 updates | |
| Base model | Block outputs | Width | Passes | Parameters |
|---|---|---|---|---|
| SD3.5-M | 47 | 1,536 | 2 | 288,768 |
| FLUX.1-dev | 76 | 3,072 | 1 | 466,944 |
| Wan2.1-1.3B | 30 | 1,536 | 2 | 184,320 |
| Wan2.2-5B | 30 | 3,072 | 1 | 184,320 |
| Base model | RL method | Reward | Kept (%) |
| SD3.5-M | Flow-GRPO | PickScore | 85 |
| Flow-GRPO, no KL | PickScore | 94 | |
| Flow-GRPO | OCR | 83 | |
| Flow-GRPO | GenEval | 88 | |
| FLUX.1-dev | Flow-GRPO | PickScore | 94 |
| DanceGRPO | HPSv2.1 | 86 |
| Share (%) | |||||||
| Base | RL method | Reward | RL-tuned gain | LoRA | Rank-1 RL LoRA | minus LoRA | |
| SD3.5-M | Flow-GRPO | PickScore | +1.560 | 90.8 | 90.8 | 43.1 | 0.000 [ 0.013, +0.013] |
| Flow-GRPO | OCR | +0.344 | 94.4 | 97.0 | 62.8 | 0.009 [ 0.022, +0.003] | |
| Flow-GRPO | GenEval | +0.271 | 95.9 | 93.4 | – | +0.007 [ 0.004, +0.017] | |
| FLUX.1-dev | Flow-GRPO | PickScore | +1.346 | 92.1 | 89.5 | – | +0.036 [ 0.005, +0.079] |
| DanceGRPO | HPSv2.1 | +0.055 | 99.2 | 97.4 | – | +0.001 [0.000, +0.002] | |
| Intervention | Parameters | PickScore | OCR |
|---|---|---|---|
| Steering vectors | 288,768 | +1.43 | +0.324 |
| Conditioning offset, fitted | 6,144 | +0.42 | +0.032 |
| Gate scales, fitted | 432 | +0.26 | +0.027 |
| Prompt suffix | 0 | +0.06 | +0.011 |
| CFG scale 6 | 0 | +0.00 | +0.032 |
| CFG scale 7.5 | 0 | 0.05 | +0.040 |
| Person | Base + | Same-size LoRA |
|---|---|---|
| 1 | 68 [65, 70] | 82 [80, 84] |
| 2 | 71 [68, 74] | 85 [83, 86] |
| 3 | 73 [70, 75] | 81 [79, 83] |
| 4 | 79 [77, 81] | 89 [85, 92] |
| 5 | 83 [82, 85] | 91 [90, 93] |
| 6 | 71 [68, 74] | 85 [82, 87] |
| vs Base | vs RL-tuned | ||||||
|---|---|---|---|---|---|---|---|
| Reward | Vectors | PF | Aes | All | PF | Aes | All |
| PickScore | 65 | 67 | 64 | 49 | 53 | 51 | |
| 68 | 67 | 62 | 51 | 48 | 52 | ||
| OCR | 62 | 67 | 60 | 49 | 47 | 50 | |
| 63 | 60 | 65 | 52 | 50 | 52 | ||
| Share of the RL gain (%) | ||||
| Base | RL-tuned model or score | RL-tuned gain | ||
| SD3.5-M | PickScore | +1.560 [+1.521, +1.599] | 61.0 [59.7, 62.3] | 90.8 [89.8, 91.7] |
| PickScore, no KL | +1.693 [+1.648, +1.734] | 70.8 [69.4, 72.2] | 95.2 [94.4, 96.1] | |
| OCR | +0.344 [+0.319, +0.369] | 21.5 [15.4, 26.9] | 94.4 [90.8, 97.9] | |
| GenEval | +0.271 [+0.250, +0.293] | 12.2 [6.5, 17.5] | 95.9 [92.0, 99.7] | |
| FLUX.1-dev | Flow-GRPO, PickScore | +1.522 [+1.481, +1.561] | 64.7 [62.8, 66.6] | 90.0 [88.9, 91.1] |
| Base | RL method | Update | Reward | Steps | Guidance | Resolution |
|---|---|---|---|---|---|---|
| SD3.5-M | Flow-GRPO | LoRA, | PickScore | 40 | 4.5 | 512 |
| jieliu/SD3.5M-FlowGRPO-PickScore | ||||||
| Flow-GRPO, no KL | LoRA, | PickScore | 40 | 4.5 | 512 | |
| jieliu/SD3.5M-FlowGRPO-PickScore-without-KL | ||||||
| Flow-GRPO | LoRA, | OCR | 40 | 4.5 | 512 | |
| jieliu/SD3.5M-FlowGRPO-Text | ||||||
| Passes | Streams | Bins | Parameters | Updates | LR | Gain |
| Mean-difference steering vectors | ||||||
| 1 | Both | 8 | 577,536 | – | – | +0.62 |
| 2 | Both | 8 | 1,155,072 | – | – | +0.99 |
| Fitted steering vectors | ||||||
| 1 | Image | 1 | 36,864 | 1,800 | +1.11 | |
| 1 | Both | 1 | 72,192 | 1,800 | +1.30 | |
| Steering vectors kept | OCR | PickScore |
|---|---|---|
| SD3.5-M, test prompts | ||
| RL-tuned model | +0.344 | +1.560 |
| All steering vectors | +0.325 | +1.416 |
| Conditional pass only | +0.344 | +1.500 |
| Unconditional pass only | 0.046 | 1.335 |
| Step bin only | +0.309 | +1.155 |
| Base | RL-tuned model or fit | Reward | Prompts seeds | Gain over base | Share (%) |
| RL-tuned models, gain over the base model | |||||
| Wan2.1-1.3B | GenRL GRPO, RL step 500 | VideoAlign TA | 64 2 | +0.058 [ 0.005, +0.124] | – |
| GenRL GRPO, RL step 1,000 | VideoAlign TA | 64 2 | +0.135 [+0.075, +0.198] | – | |
| GenRL GRPO, RL step 1,500 | VideoAlign TA | 64 2 | +0.091 [+0.015, +0.168] | – | |
| SOLACE | Self-confidence | 64 2 | 1.115 [ 1.211, 1.021] | – | |
| RAVEN | CM-GRPO | Sum of five rewards | 100 2 | +0.128 [ 0.001, +0.262] | – |
| Model | Share (%) | minus model |
|---|---|---|
| GenEval test, seeds 0 to 3: 641 fixed, 108 both fail | ||
| 91.4 [88.8, 93.9] | – | |
| 30.1 [26.0, 34.2] | +61.3 [+56.8, +65.8] | |
| Same-size LoRA | 90.6 [87.9, 93.0] | +0.8 [ 1.7, +3.3] |
| RL-tuned, next seed | 93.3 [91.1, 95.4] | 1.9 [ 4.9, +1.0] |
| Both fail: | 33.3 [22.8, 45.5] | – |
RL Post-Training
Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.
See all 464 papers