Combining Chain-of-Thought (CoT) with Reinforcement Learning (RL) improves text-to-image (T2I) generation, yet the underlying interaction between CoT's exploration and RL's optimization remains unclear. We present a systematic entropy-based analysis that yields three key insights: (1) CoT expands the generative exploration space, while RL contracts it toward high-reward regions; (2) final reward is strongly negatively correlated with both the mean and variance of image-token entropy, highlighting the need to reduce uncertainty and instability; and (3) the entropy of the textual CoT directly governs downstream image quality, with lower-entropy CoTs leading to better generations. Motivated by these findings, we propose Entropy-Guided Group Relative Policy Optimization (EG-GRPO), a fine-tuning strategy that reallocates optimization budget by uncertainty: low-entropy tokens are excluded from reward-driven updates to preserve stability, while high-entropy tokens receive an entropy bonus that encourages structured exploration without collapse. Experiments on standard T2I benchmarks demonstrate that EG-GRPO achieves state-of-the-art performance.
Figures & tables
Figure 1: Comparison of different text-to-image generation methods: (a) autoregressive text-to-image generation, (b) CoT, and (c) with CoT and GRPO optimization.
Figure 2: Entropy–reward distributions of different methods. CoT ( Janus-Pro+CoT ) expands the exploratory space with more diverse outputs, while GRPO fine-tuning ( T2I-R1 ) contracts it toward higher-reward regions, yielding more stabilized, high-quality generations.
Figure 3: Left: Reward vs. CoT entropy (stable cases, Image Entropy Std < 0.011). Higher CoT entropy correlates with lower image reward. Right: Reward distributions across different CoTs for the same prompt. Images from the same CoT cluster together, with certain CoTs consistently yielding lower rewards.
Figure 4: Left: Reward vs. entropy std. Higher instability (larger std) consistently lowers reward. Middle: Relation between entropy std (x-axis) and the negative correlation of reward–entropy mean (y-axis). Greater instability strengthens the negative correlation. Right: Reward vs. entropy mean under high-variance cases (std > 0.03). Large std implies exploratory generation where RL has not converged; in this regime, reducing mean entropy is especially beneficial.
Model
T2I-CompBench
WISE
Color
Shape
Texture
Culture
Spatio-temporal
Science
PixArt- α ( Chen et al., 2023 )
66.90
49.27
64.77
45.00
49.00
46.33
SD-v1.5 ( Rombach et al., 2022 )
37.58
37.13
41.86
34.00
33.50
26.00
FLUX.1-dev ( Labs, 2024 )
74.07
57.18
69.22
48.00
60.00
42.67
Emu3 ( Wang et al., 2024 )
75.44
57.06
71.64
34.00
46.50
37.67
Show-o ( Xie et al., 2024 )
56.00
41.00
46.00
28.00
44.00
35.33
Table 1: Comparison of models across T2I-CompBench and WISE . Spatio-temporal is the average of Time and Space; Science is the average of Biology, Physics, and Chemistry. ∗ Results are obtained by evaluating the officially released model under the same experimental settings for fair comparison.
Model
T2I-CompBench
WISE
Color
Shape
Texture
Culture
Spatio-temporal
Science
EG-GRPO
84.11
60.88
77.38
49.00
56.00
46.33
w/ only sem
81.29
55.68
74.10
47.00
56.00
43.67
w/ only tok
79.25
53.73
72.46
45.00
56.00
43.00
w/o All
82.58
58.67
76.94
48.00
55.50
45.00
Table 2: Ablation results of EG-GRPO. We compare the full method with variants that apply entropy guidance only to textual CoT tokens (w/ only sem), only to image tokens (w/ only tok), or not at all (w/o All).
Step
Diversity (Vendi Score)
100
2.7305
200
2.7233
400
2.7212
600
2.7151
800
2.7159
Table 3: Evolution of diversity during the GRPO training process. As the model converges towards high-reward regions, a slight reduction in diversity is observed, reflecting the exploration-exploitation trade-off.
Metric
EG-GRPO
T2I-R1
Quality Mean
13.86
13.85
Diversity (Vendi Score)
2.593
2.592
Table 4: Comparison of Diversity (Vendi Score) between EG-GRPO (Ours) and T2I-R1 (Baseline) on a subset of samples with similar quality scores. Our method maintains diversity comparable to the baseline.
Figure 5: Entropy distributions of EG-GRPO vs. T2I-R1: left for textual CoT tokens, right for image tokens.
Figure 6: Qualitative case study of our method on diverse prompts. These results are randomly sampled.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
T2I-R1 (Baseline)
EG-GRPO (Ours)
Overhead
Step Time (s)
50.20
50.88
+1.35%
GPU Memory (GB)
30.79
31.17
+0.38 GB
Appendix
Table 5: Computational overhead comparison between T2I-R1 and EG-GRPO. The overhead introduced by our entropy-guided mechanism is marginal in both time and memory.
Step
100
200
400
600
800
Reward (T2I-R1)
2.1105
2.1130
2.1208
2.1339
2.1340
Reward (EG-GRPO)
2.1411
2.1836
2.1968
2.2075
2.2117
Appendix
Table 6: Convergence comparison: Reward scores at different training steps. EG-GRPO achieves higher performance consistently, indicating a superior time-to-convergence ratio.
Figure 7: Comparison of generation diversity for T2I-R1 (left) and EG-GRPO (right).
School of Artificial Intelligence, Beijing University of Posts and Telecommunications1 · Beijing Key Laboratory of Multimodal Data Intelligent Perception and Governance3 · Institute of Artificial Intelligence (TeleAI), China Telecom2