MAR-GRPO: Stabilized GRPO for AR-diffusion Hybrid Image Generation
Authors: Xiaoxiao Ma, Jiachen Lei, Tianfei Ren, Jie Huang, Siming Fu, Aiming Hao, Jiahong Wu, Xiangxiang Chu, +1 more
Organizations: University of Science and Technology of China, Hefei, China · AMAP, Alibaba Group, Hangzhou, China · Zhejiang University, Hangzhou, China
Reinforcement learning (RL) has been successfully applied to autoregressive (AR) and diffusion models. However, extending RL to hybrid AR-diffusion frameworks remains challenging due to interleaved inference and noisy log-probability estimation. In this work, we study masked autoregressive models (MAR) and show that the diffusion head plays a critical role in training dynamics, often introducing noisy gradients that lead to instability and early performance saturation. To address this issue, we propose a stabilized RL framework for MAR. We introduce multi-trajectory expectation (MTE), which estimates the optimization direction by averaging over multiple diffusion trajectories, thereby reducing diffusion-induced gradient noise. To avoid over-smoothing, we further estimate token-wise uncertainty from multiple trajectories and apply multi-trajectory optimization only to the top-k% uncertain tokens. In addition, we introduce a consistency-aware token selection strategy that filters out AR tokens that are less aligned with the final generated content. Extensive experiments across multiple benchmarks demonstrate that our method consistently improves visual quality, training stability, and spatial structure understanding over baseline GRPO and pre-RL models. Code is available at: https://github.com/AMAP-ML/mar-grpo.
Figures & tables
Figure 1. Stabilizing MAR optimization via improved training dynamics. Left: Compared to GRPO with a fixed decoder, standard optimization suffers from unstable structures, noisy details, and diversity collapse. Middle: Our method achieves stabilized training dynamics, producing more consistent structures, finer details, and improved diversity. Right: Training curves show that ours stabilizes reward dynamics and achieves higher peak performance, avoiding the collapse observed in baseline GRPO.
Figure 2. (a) Gradient comparison between the end-to-end GRPO baseline and the frozen diffusion head counterpart. The End-to-end baseline introduces "noisier" gradients to the AR component, disturbing the training progress. (b) Test set HPS v.s. training iteration. HPS curves show that end-to-end baseline suffers from reward hacking and early saturation. (c) End-to-end training degrades image quality in later stages, which can be addressed by freezing the diffusion head.
Figure 3. The proposed multi-trajectory expectation estimates an uncertainty map by sampling multiple diffusion trajectories during rollout, capturing regions that are most sensitive to diffusion-induced stochasticity. Based on a top- k% uncertainty threshold, our noise-corrected optimization selectively aggregates multi-trajectory signals to calibrate the GRPO loss, effectively reducing stochastic noise in gradient estimation and yielding a more reliable optimization direction.
Figure 4. Visualization of estimated uncertainty and corresponding uncertainty maps (bottom) with generated images (top). We highlight top 30% most uncertain regions for clarity.
Figure 5. Compared to GRPO with fixed diffusion head, incorporating multi-seed trajectory expectation further corrects the optimization direction, leading to better training dynamics (smoother KL loss growth and lower, more stable gradient norms in later stages).
Figure 6. Progressive improvement across multiple generation aspects. From left to right: base model, GRPO, +Fixed Decoder, +MTE, and our full model. GRPO leads to unstable optimization, while fixing the decoder improves stability. MTE further enhances visual quality by reducing diffusion-induced stochasticity. Our full model, with consistency-aware token selection, achieves the best results in detail, structure, appearance, counting, and spatial relations.
Model
Param
Preference
T2I-compbench
HPS
ImgRwd
Pickscore
Aesthetic
Color
Shape
Texture
2d-spat.
3d-spat.
Numeracy
Complex
SD3
2B
30.22
0.95
22.41
5.90
0.8094
0.5864
0.7297
0.3219
0.4044
0.6078
0.3780
FLUX.1-dev
12B
31.35
1.10
22.77
6.27
0.7407
0.5718
0.6922
0.2863
0.3866
0.6185
-
Sana-1.5
1.6B
30.36
1.01
22.60
6.07
0.7625
0.5426
0.6761
0.3814
0.4088
0.6110
0.3727
LlamaGen
0.8b
23.92
-0.38
20.50
5.21
0.2996
0.3212
0.3888
0.1004
0.1530
0.2747
0.2501
Show-o
1.3B
27.98
0.86
22.00
5.90
0.7327
0.5264
0.6815
0.3697
0.3996
0.6209
0.3572
Table 1. Quantitative comparison results on the human preference correlated metrics (HPS, Imagereward, Pickscore and Aesthetic) and T2I-compbench. Our method ("Ours") achieves consistent gains across multiple metrics, significantly outperforming the standard GRPO implementation (“+GRPO”) as well as AR and diffusion baselines in visual quality and structural consistency. “spat." is short for spatial. † : We observe failure cases in the base model, see Appendix for detailed analysis.
Figure 7. Ablations of masking ratio, similarity threshold, and diffusion seeds. (a) Visual results under different masking ratios ( k ) and similarity thresholds ( τ ), showing trade-off between detail and consistency. (b) Larger masking ratios improve stability, with marginal gains when k>0.3 . (c) Masking ratio, similarity threshold, and number of diffusion seeds all affect performance.
HPS
Aes.
Shape
Texture
2d-spat.
NOVA
26.76
5.79
0.5489
0.6712
0.2983
GRPO*
27.57
5.78
0.5392
0.6690
0.3069
+Fix Decoder*
28.65
5.83
0.5495
0.6801
0.3134
+MTE
29.06
5.90
0.5752
0.6945
0.3289
+Sim (Ours)
29.35
6.01
0.5780
0.6980
0.3306
Table 2. Quantitative results on human preference metrics. Compared to vanilla GRPO and its fixed-decoder variant (“+Fix Decoder”), our method, which incorporates multi-trajectory expectation (“+MTE”) and consistency-aware token selection (“+Sim”), achieves improved performance. Due to the instability of the baseline methods, we report peak performance for “GRPO” and “+Fix Decoder”, where “*” indicates the best achieved performance during training.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8. (a) AR features encode clear structural information, while diffusion features primarily capture fine-grained details. (b) Compared to standard diffusion models, the diffusion head in MAR reconstructs most image content at very early timesteps.
Figure 9. Samples generated with different diffusion trajectories is highly deterministic.
Baseline
Ours
Reward func
HPS
Mixed
HPS
Mixed
Learning-rate
1e-6 (2e-6 collapsed)
2e-6
Training Steps
best ( ∼ 250)
best ( ∼ 400)
500
800
Beta
0.01
0.01
Inner loop
1
1
Appendix
Table 3. Training hyperparameters for NOVA.
Baseline
Ours
Reward func
HPS
Mixed
HPS
Mixed
Learning-rate
1e-6
1e-6
Training Steps
best ( ∼ 300)
best ( ∼ 800)
500
800
Beta
0.01
0.01
Inner loop
1
1
Appendix
Table 4. Training hyperparameters for Harmon.
Figure 10. Training dynamics when fixing the AR module and tuning the diffusion head only. Although the reward can improve slightly at the beginning, it drops quickly afterward, indicating that optimizing the diffusion head alone is unstable and does not provide reliable gains for MAR post-training.
Figure 11. Failure cases of the Harmon base model. The model occasionally produces invalid or severely degraded images due to an insufficiently trained diffusion head.
Department of Computer Engineering, Middle East Technical University, Ankara, Türkiye · Department of Mathematics and Computer Science (IMADA), University of Southern Denmark, Odense, Denmark · ROMER Robotics and AI Center, Middle East Technical University, Ankara, Türkiye