Recent text-to-image generation models have achieved remarkable visual quality, but improving them through post-training remains challenging because no single reward signal captures the full range of human preference. In this work, we develop a simple and effective post-training recipe for open-domain text-to-image generation based on the composition of complementary reward signals. Our reward system consists of two main components: a preference reward, trained on large-scale human preference data using a Bradley-Terry objective to capture overall human aesthetic and perceptual preferences, and rubric-based rewards, which explicitly evaluate prompt faithfulness and other desirable properties while providing safeguards against reward hacking. A key challenge is how to combine these heterogeneous reward signals. We show that a naive weighted average leads to suboptimal optimization behavior, and propose a simple reward composition strategy that more effectively balances preference optimization with rubric satisfaction. In the Arena text-to-image leaderboard (https://arena.ai/), our RL-trained Flux2dev achieves an Elo rating 69 points above the base model, and our post-trained Ideogram-4 surpasses every open-source model on the leaderboard, reaching an Elo of 1223.5. (Claims of state-of-the-art performance are based on the Arena leaderboard snapshot as of September 4, 2026.) Our results suggest that effective rewards for frontier generative-model training require broad coverage of user intent and robustness to exploitation under optimization. To support reproducible research, we release Arena-T2I-Training, a 1K subset of training data that recovers some gains of full-scale training, providing a resource that we hope will facilitate future work on post-training for text-to-image models.
Figures & tables
Deployed model
Arena score
FLUX.2-dev
base model
1132.8±14
Ours
1202±13
Ideogram-4
Base model (best open-source model)
1204±4
Ours
1224±13
Table 1: Arena scores on the Arena text-to-image leaderboard. Public scores are from the September 4, 2026 leaderboard snapshot.
Figure 1: Visualization. Preference-only training drifts toward generic embellishment and ignores requested content; faithfulness-only training washes out detail and drops requested style and scenery; Arena-T2I-Hard recovers the requested elements but keeps unrequested decoration; the ungated constraint flattens detail even where the prompt invites it; the intent-gated Stage 1 keeps the constraint pressure only where the prompt closes the spec, yielding compliant renders without an obvious quality tax.
Figure 2: Two recurring failure modes. Left: the prompt requests no text, yet the RL policy introduces garbled strings on the napkin. Right: the prompt requests Elsa Beskow’s storybook illustration style; the base model preserves the requested style, whereas the RL policy drifts toward photorealism.
Gemini-3.5-flash
GPT-5.4
Id
Arena RM
PickScore
Arena RM
PickScore
Baseline
DiffusionNFT
.290
.290
.396
.396
faithfulness only
.366
.366
.367
.367
Arena-T2I-Hard
.589
.577
.557
.537
preference only
.507
.461
.417
.517
Table 2: All pairwise win rates vs. the frozen FLUX.2 base. Each cell reports the run’s best checkpoint among 30,60,90,120,150 , selected on the 1k Arena validation rewrite prompts. Please refer to Table 7 for 95% percentile-bootstrap confidence intervals for every cell.
Figure 3: RL outcome scales with the preference reward model’s training data. The x-axis shows the number of human preference pairs used to train the reward model, and the y-axis shows the MMRBv2 win rate of the resulting Stage 1 RL model against the frozen base.
Gemini-3.5-flash
GPT-5.4
Mechanism
Variant
Arena RM
PickScore
Arena RM
PickScore
Constrain Gate
ungated
0.6046
0.5200
0.5555
0.4985
intent-gated
0.6416
0.5810
0.5615
0.6105
Hack applicability gate
ungated
0.5591
0.5695
0.5000
0.5515
applicable-gated
0.6346
0.5710
0.5670
0.5845
Hack signal integration
additive
0.5260
0.4194
0.4730
0.4105
Table 3: Ablation study on the ensembling mechanisms. Win rate vs. the base model on the 1k Arena validation prompts; each cell is the best checkpoint among {30,60,90,120,150} , selected under the Gemini judge.
Gemini-3.5-flash
GPT-5.4
Id
Arena RM
PickScore
Arena RM
PickScore
Baseline
Arena-T2I-Hard
.401
.524
.579
0.451
preference only
.266
.334
.285
0.403
Ours
Stage 1 (w/o anti-hack)
.553
.537
.623
.525
Table 4: All pairwise win rates vs. the frozen Ideogram-4 base on the 1k Arena validation prompts, judged by Gemini-3.5-flash and re-judged by GPT-5.4.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
FLUX.2 [dev]
Ideogram 4
Rollouts (per training epoch)
Prompt groups G
48
48
Rollouts per prompt K
24
24
Images per epoch
1,152
1,152
Sampling steps
10
10
Resolution
5122
5122
Appendix
Table 5: Training hyperparameters for the two base models.
Figure 4: The complete per-prompt reward battery for two real evaluation prompts, shown alongside the as-served renders from the FLUX.2 base model and the Stage 1 model. For each prompt, we list every question posed by the reward: the presence-oriented faithfulness DAG, in which a question whose parent fails is automatically scored ‘no” without being asked; the absence-oriented constraint checklist, generated in full here because both prompts are labeled strict ; and, at the bottom, the two armed hack-prevention detector questions shared across prompts.
Id
MMRBv2 prompts
Baseline
faithfulness only
0.3357
Arena-T2I-Hard
0.5563
preference only
0.5272
Ours
Stage 1
0.5632
Appendix
Table 6: Prompt-set transfer: we swap the prompt set to the 791 public MMRBv2 T2I test prompts. The checkpoints are selected on the 1k Arena validation prompts.
Gemini-3.5-flash
GPT-5.4
Id
Arena RM
PickScore
Arena RM
PickScore
Baseline
DiffusionNFT
.290 [.266, .315]
.290 [.266, .315]
.396 [.370, .421]
.396 [.370, .421]
faithfulness only
.366 [.340, .392]
.366 [.340, .392]
.367 [.344, .390]
.367 [.344, .390]
Arena-T2I-Hard
.589 [.561, .615]
.577 [.550, .603]
.557 [.533, .580]
.537 [.513, .560]
preference only
.507 [.480, .533]
.461 [.434, .487]
.417 [.394, .441]
.517 [.492, .541]
Appendix
Table 7: 95% percentile-bootstrap confidence intervals for every cell of Table 2 . Each cell re-states the win rate vs. the frozen FLUX.2 base on the 1k Arena validation rewrite prompts together with its interval. The DiffusionNFT and faithfulness-only baselines train with neither preference reward, so their Arena-RM and PickScore columns repeat the same run. All intervals use 10,000 bootstrap replicates over prompt pairs with a fixed seed.
Gemini-3.5-flash
GPT-5.4
Variant
Arena RM
PickScore
Arena RM
PickScore
Constraint gate
ungated
.605 [.580, .630]
.520 [.493, .546]
.555 [.532, .579]
.498 [.475, .521]
intent-gated
.642 [.617, .666]
.581 [.554, .607]
.561 [.539, .584]
.611 [.587, .634]
Hack applicability gate
ungated
.559 [.533, .586]
.570 [.543, .596]
.500 [.477, .522]
.551 [.529, .574]
Appendix
Table 8: 95% percentile-bootstrap confidence intervals for every cell of Table 3 . The intent-gated variant is the Stage 1 run, and the applicable-gated and one-sided-veto variants are the Stage 2 run, so those rows repeat the corresponding intervals of Table 7 . All intervals use 10,000 bootstrap replicates over prompt pairs with a fixed seed.
Figure 5: Single-preference training vs. base across checkpoints, evaluated on the 1 k arena validation prompts with gemini-3.5-flash rewrites. The faithfulness panel shows both independently trained preference rewards below parity and degrading with training, even while aesthetics holds at or above parity for the arena-RM run: faithfulness is traded away for aesthetics.
Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.
Text-to-image generation powers content creation across design, media, and data augmentation. Post-training of text-to-image generative models is a promising path to improve human preference alignment, factuality, and aesthetics. We introduce SOLACE (Self-Originating LAtent Confidence Estimation), a post-training framework that replaces external reward supervision with an internal self-confidence signal: we re-noise the model's own outputs and measure how accurately it recovers the injected noise, treating low reconstruction error as high self-confidence. SOLACE converts this intrinsic signal into scalar rewards for reinforcement learning, requiring no external reward models, annotators, or preference data. By reinforcing high-confidence generations, SOLACE delivers consistent gains in compositional generation, text rendering, and text-image alignment. Integrating SOLACE with external rewards yields complementary improvements while alleviating reward hacking.
Post-training is crucial for improving text-to-image models, yet existing pipelines typically rely on manually curated training data and fixed training distributions. We introduce APT (Agentic Post-Training), a fully automated framework that adaptively post-trains a base model according to its evolving capability profile. APT coordinates three agents to construct evaluation sets from user requirements, diagnose model-specific strengths and failure modes, and generate reinforcement learning data with an adaptive prompt distribution that emphasizes underperforming capabilities. By repeatedly evaluating, diagnosing, and updating the generator, APT lets the current model's behavior shape its next training tasks, forming a feedback loop inspired by recursive self-improvement. For general-purpose settings without user-specified evaluation scenarios, we further develop an evaluation harness that enables broad and diverse evaluation set generation. Experiments across multiple base models, benchmarks, and generation architectures demonstrate that APT is model-agnostic, benchmark-agnostic, architecture-agnostic, and fully automated.
Sirui Zhang, Fangyun Wei, Dong Chen +5
University of Science and Technology of China · Microsoft Research Asia · University of Sydney +2
Existing RL post-training methods for text-to-image generation usually convert the final-image reward into a single scalar advantage and apply it with the same strength to the entire generative trajectory. However, text-to-image generation naturally has temporal and spatial structure: different denoising steps are responsible for different generation stages, and the content that truly determines text alignment often appears only in part of the image. This granularity mismatch makes it difficult for policy updates to focus on the generative components that actually affect the reward. To address this issue, we propose SpatioTemporal Adaptive Reward (STAR) Allocation for RL post-training of text-to-image diffusion and flow models. STAR uses text-image attention inside the generative model and starts from the core content that the user truly cares about in the prompt. It constructs spatial allocation maps that dynamically vary across denoising steps and rollouts, and allocates the same group-relative advantage to more relevant latent regions with almost no additional computational overhead. STAR then applies stronger policy updates to these regions through a spatially resolved policy objective. We use Stable Diffusion 3.5 Medium as the base model and evaluate on three tasks: GenEval, OCR text rendering, and PickScore. Experimental results show that STAR improves compositional semantic alignment, text rendering, and preference optimization without changing the external reward source, achieving 0.9759, 0.9757, and 23.60 on GenEval, OCR, and PickScore, respectively.