Reinforcement-Learning-based post-training has recently emerged as a promising paradigm for aligning text-to-image diffusion models with human preferences. In recent studies, increasing the rollout group size yields pronounced performance improvements, indicating substantial room for further alignment gains. However, scaling rollouts on large-scale foundational diffusion models (e.g., FLUX.1-12B) imposes a heavy computational burden. To alleviate this bottleneck, we explore the integration of FP4 quantization into Diffusion RL rollouts. Yet, we identify that naive quantized pipelines inherently introduce risks of performance degradation. To overcome this dilemma between efficiency and training integrity, we propose Sol-RL (Speed-of-light RL), a novel FP4-empowered Two-stage Reinforcement Learning framework. First, we utilize high-throughput NVFP4 rollouts to generate a massive candidate pool and extract a highly contrastive subset. Second, we regenerate these selected samples in BF16 precision and optimize the policy exclusively on them. By decoupling candidate exploration from policy optimization, Sol-RL integrates the algorithmic mechanisms of rollout scaling with the system-level throughput gains of NVFP4. This synergistic algorithm-hardware design effectively accelerates the rollout phase while reserving high-fidelity samples for optimization. We empirically demonstrate that our framework maintains the training integrity of BF16 precision pipeline while fully exploiting the throughput gains enabled by FP4 arithmetic. Extensive experiments across SANA, FLUX.1, and SD3.5-L substantiate that our approach delivers superior alignment performance across multiple metrics while accelerating training convergence by up to 4.64×, unlocking the power of massive rollout scaling at a fraction of the cost.
Figures & tables
Figure 1: Sol-RL enables efficient and high-fidelity text-to-image alignment. (Left) High-quality images generated by FLUX.1 and SANA fine-tuned with our method, demonstrating superior generation capabilities across diverse styles. (Right) ImageReward training curves. They demonstrate that Sol-RL achieves substantial wall-clock speedups (up to 4.64× ) to reach an equivalent reward level, ultimately converging to a higher alignment ceiling.
Figure 2: Decoupled two-stage reinforcement learning pipeline of Sol-RL. We separate the high-throughput FP4 exploration from the selective BF16 high-contrastive rollout. This framework achieves up to 2.4× acceleration compared to naive scaling while avoiding quantization-induced corruption, introducing merely a 2% computational overhead.
Figure 3: Pitfalls and Potential of NVFP4 rollouts. (a) Time breakdown of high-precision rollout scaling and direct quantized rollout. The x-axis labels follow the format K -in- N ( P ), denoting that K samples are selected for training from N generated rollouts under P precision. (b) Directly integrating FP4 rollout in RL pipeline leads to severe instability and performance degradation compared to the BF16 baseline. (c) Conversely, the dense diagonal distribution of intra-group relative reward rankings validates NVFP4 quantized rollouts as a reliable proxy for reward sorting.
Method
ImageReward
CLIPScore
PickScore
HPSv2
(Base w/o CFG: 0.455)
(Base w/o CFG: 0.2630)
(Base w/o CFG: 0.8096)
(Base w/o CFG: 0.2566)
Score
Δ(↑)
Score
Δ(↑)
Score
Δ(↑)
Score
Δ(↑)
DanceGRPO
1.4937
+1.0387
0.2898
+0.0268
0.8807
+0.0711
0.3552
+0.0986
FlowGRPO
1.5331
+1.0781
0.2884
+0.0254
0.8743
+0.0647
0.3501
+0.0935
AWM
1.6693
+1.2143
0.3039
+0.0409
0.8842
+0.0746
0.3664
+0.1098
DiffusionNFT
1.6707
+1.2157
0.2991
+0.0361
0.8852
+0.0756
0.3613
+0.1047
Table 1: Quantitative comparison of alignment performance. Evaluated on FLUX.1 under an identical GPU-hour budget. Δ indicates the performance improvement over the Base (w/o CFG) model. Bold and darker green background indicate the best results, while lighter green background indicates the second best.
Figure 4: Comparison across diverse foundation models and alignment metrics. Evaluated under identical wall-clock budgets (GPU Hours), Sol-RL (green) consistently outperforms the DiffusionNFT baseline (grey). Across all tested combinations of models and reward functions, our decoupled scaling strategy accelerates convergence to the baseline’s equivalent performance by up to 4.64× , ultimately converging to a remarkably higher final alignment ceiling.
Figure 5: Visual comparison before and after Sol-RL. Compared to the SANA base model without fine-tuning (top row), the counterpart optimized across multiple rewards (HPSv2, PickScore, CLIPScore and OCR) via Sol-RL (bottom row) exhibits substantial improvements in complex detail rendering and semantic alignment across various prompts.
Table 7
Figure 6: Visualization of NVFP4 and BF16 rollouts. Despite minor localized deviations, the NVFP4 quantized rollouts maintain the overall semantic layout and structure.
Table 9
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Hyperparameter
SANA-1.5 1600M
FLUX.1-dev
SD 3.5 Large
Model
Image resolution
1024×1024
512×512
1024×1024
Gradient checkpointing
✗
✓
✗
LoRA target modules
to_{q,k,v,out}
to_{q,k,v,out}
attn.{to,add}_{q,k,v,out}
LoRA
Rank r
32
Alpha α
64
Init mode
Gaussian
Appendix
Table 7: Training hyperparameters. Rows with a single value in the rightmost column are shared across all models.
Reward Metric
Kendall τ ( ↑ )
Spearman ρ ( ↑ )
Top/Btm 4 Match
Top/Btm 8 Match
Top/Btm 12 Match
Top ( ↑ )
Btm ( ↓ )
Top ( ↑ )
Btm ( ↓ )
Top ( ↑ )
Btm ( ↓ )
CLIPScore
0.752
0.900
95.7%
4.5%
93.9%
6.2%
92.2%
8.2%
HPSv2
0.827
0.943
97.6%
3.4%
95.5%
5.3%
93.9%
7.1%
ImageReward
0.807
0.932
97.2%
3.9%
95.1%
5.9%
93.4%
7.6%
PickScore
0.806
0.934
97.1%
3.8%
95.4%
5.6%
93.6%
7.2%
Overall
0.798
0.927
96.9%
3.9%
95.0%
5.7%
93.3%
7.5%
Appendix
Table 8: Group-Relative Ranking Consistency. Global reward correlation ( τ , ρ ) and exact match rates (Top/Bottom k ) between FP4-accelerated samples and high-fidelity BF16 baselines. These results demonstrate that our acceleration preserves intra-group relative rankings, thereby ensuring the effectiveness of FP4-driven exploration.
Figure 7: Qualitative comparison on PickScore-optimized models. We compare images generated by Flux.1-dev base models against their Sol-RL, DiffusionNFT and FlowGRPO fine-tuned variants. Sol-RL produces images with stronger semantic alignment to the prompt, richer fine-grained details, and more coherent artistic style.
Figure 8: Qualitative comparison on HPSv2-optimized models. We compare images generated by Flux.1-dev base models against their Sol-RL, DiffusionNFT and FlowGRPO fine-tuned variants. Sol-RL produces images with stronger semantic alignment to the prompt, richer fine-grained details, and more coherent artistic style.
Figure 9: Qualitative comparison on ImageReward-optimized models. We compare images generated by Flux.1-dev base models against their Sol-RL, DiffusionNFT and FlowGRPO fine-tuned variants. Sol-RL produces images with stronger semantic alignment to the prompt, richer fine-grained details, and more coherent artistic style.
Diffusion models have become a dominant paradigm for high-quality generative modeling, while post-training is essential for adapting them to diverse downstream applications. However, post-training of large diffusion models is still challenging due to the prohibitive memory footprints and slow training speed, which existing parameter-efficient fine-tuning methods only partially address. To overcome these limitations, we propose FourTune, an efficient post-training framework for diffusion models based on an end-to-end W4A4G4 paradigm. FourTune introduces a triple-branch hybrid pipeline that augments the standard LoRA architecture with a frozen numerical stabilizer to isolate quantization-sensitive outliers, enabling stable training under native 4-bit computation. In addition, FourTune employs hardware-efficient block-wise quantization and customized fused kernels to support efficient quantized backpropagation and reduce memory bandwidth overhead. Across customization, reinforcement learning, and distillation tasks, FourTune matches the quality of full-precision fine-tuning. On FLUX.1-dev (12B), FourTune reduces memory overhead by 2.25× and increases end-to-end training throughput by 2.27× compared to BF16 LoRA.
We present, to our knowledge, the first end-to-end FP4 RL post-training, in which both the rollout and training policies, including their forward and backward passes, operate at 4-bit precision. A systematic study reveals that the dominant source of degradation in FP4 RL is not training-side quantization error but rollout activation quantization: outliers stretch the dynamic range so far that a large number of activation values underflow to zero under FP4. Counterintuitively, restoring the training policy to higher precision while keeping the rollout in FP4 makes accuracy worse than full FP4 baseline, exposing rollout-training mismatch as the principal failure mode and ruling out standard pretraining-style fixes. We address this with Rollout Residual Quantization (Rollout-ResQ): a single residual correction term constrained to a hardware-friendly sparsity pattern, added only to the FP4 rollout matmul -- a lightweight correction that recovers most of the precision lost to outlier-driven underflow without inflating the rollout's compute footprint. On Qwen2.5-3B and Qwen2.5-Math-7B, Rollout-ResQ paired with the HiFloat4 (HiF4) format -- whose three-level hierarchical scaling preserves resolution under FP4's tight 4-bit budget -- closes the accuracy gap to BF16 from 4.9% to 1.1%, bringing fully quantized FP4 RL within striking distance of full precision. Applied to the open-standard MXFP4, the same recipe narrows the gap from 13.6% to 5.3%, revealing that FP4 format choice is a key factor that determines the ceiling on recoverable accuracy. Together, these results establish HiF4 as the enabling format for end-to-end FP4 RL post-training, and Rollout-ResQ as the activation-side mechanism that makes the gap to BF16 closable.
Reinforcement learning (RL) has become a standard technique for post-training diffusion-based image synthesis models, as it enables learning from reward signals to explicitly improve desirable aspects such as image quality and prompt alignment. In this paper, we propose an online RL variant that reduces the variance in the model updates by sampling paired trajectories and pulling the flow velocity in the direction of the more favorable image. Unlike existing methods that treat each sampling step as a separate policy action, we consider the entire sampling process as a single action. We experiment with both high-quality vision language models and off-the-shelf quality metrics for rewards, and evaluate the outputs using a broad set of metrics. Our method converges faster and yields higher output quality and prompt alignment than previous approaches.