Improving Text-to-Image Generation with Intrinsic Self-Confidence Rewards

Seungwook Kim; Minsu Cho

cs.CVMar 1, 2026

Improving Text-to-Image Generation with Intrinsic Self-Confidence Rewards

Authors: Seungwook Kim, Minsu Cho

Organizations: POSTECH · GenGenAI · RLWRLD

Abstract

Text-to-image generation powers content creation across design, media, and data augmentation. Post-training of text-to-image generative models is a promising path to improve human preference alignment, factuality, and aesthetics. We introduce SOLACE (Self-Originating LAtent Confidence Estimation), a post-training framework that replaces external reward supervision with an internal self-confidence signal: we re-noise the model's own outputs and measure how accurately it recovers the injected noise, treating low reconstruction error as high self-confidence. SOLACE converts this intrinsic signal into scalar rewards for reinforcement learning, requiring no external reward models, annotators, or preference data. By reinforcing high-confidence generations, SOLACE delivers consistent gains in compositional generation, text rendering, and text-image alignment. Integrating SOLACE with external rewards yields complementary improvements while alleviating reward hacking.

Figures & tables

Figure 2: Overview of SOLACE. Given a text prompt $c$ , we generate $G$ different latents. Without decoding, we re-noise the latents using $K$ noise probes across $t\in\mathcal{T}\subset[0,1]$ . For each generated latent $z_{0}^{(i)}$ , we formulate the text-to-image generative model’s self-confidence of the generated latent as the ability to denoise the re-noised latent. We leverage this self-confidence as an internal reward scalar value, which we use to post-train the text-to-image generative model using GRPO [ 64 , 41 ] . We omit the KL term in this figure for better readability.

Figure 3: Qualitative results of SOLACE on SD3.5 [ 16 ] across DrawBench [ 61 ] , GenEval [ 21 ] and OCR [ 14 ] . SOLACE shows consistent improvements over the baseline SD3.5.

Figure 5: Effect of SOLACE post-training SD3.5-M after post-training on PickScore [ 35 ] using FlowGRPO [ 41 ] . SOLACE complements external rewards, showing the best compositional generation and visual appeal on GenEval [ 21 ] . Post-training on external rewards yields high visual appeal, but sacrifices compositionality as shown above (Column 3: Generates yellow motorcycle instead / generates unwanted human).

Number of noise probes $K$
	Task-specific		Image Quality		Human Preference
	GenEval	OCR	ClipScore	Aesthetic	PickScore	HPSv2.1	ImageReward	UnifiedReward
$K=4$	0.71	0.66	0.287	5.37	22.34	0.273	0.81	3.08
$K=8$ (Ours)	0.71	0.67	0.288	5.39	22.41	0.278	0.87	3.11
$K=16$	0.70	0.67	0.288	5.42	22.34	0.278	0.86	3.09
Classifier-Free Guidance for self-confidence calculation

Table 2: Ablation study results of SOLACE. We validate the design choices of SOLACE over number of noise probes

K

, the usage of CFG for self-confidence calculation, and online/offline self-confidence calculation. Our current configurations yield superior results.

Figure 6: Rationale of SOLACE. Distributions of self-confidence under three inference settings. The distribution shifts rightward (higher self-confidence) as visual quality improves, showing that noise recovery accuracy is predictive of sample quality.

	Task-specific		Image Quality		Human Preference
	GenEval	OCR	ClipScore	Aesthetic	PickScore	HPSv2.1	ImageReward	UnifiedReward
SD3.5-M (2.5B)	0.65	0.61	0.282	5.36	22.34	0.279	0.84	3.08
+ SOLACE (Ours)	0.71	0.67	0.288	5.39	22.41	0.278	0.87	3.11
SD3.5-L $\dagger$ (8.1B)	0.71	0.68	0.289	5.50	22.91	0.288	0.96	3.25
(Reproduced)	0.51	0.68	0.284	5.28	21.86	0.264	0.70	2.98
† + SOLACE (Ours)	0.58	0.74	0.288	5.25	21.91	0.253	0.65	2.98

Table 3: Applying SOLACE to SD3.5-L [ 16 ] and FLUX.1-Dev [ 4 ] . We apply SOLACE on additional models of SD3.5-L and FLUX.1-Dev, to verify the effect of SOLACE given (1) a larger base model, and (2) a different architecture from SD3.5-M.

\dagger

denotes results taken from DiffusionNFT [ 96 ] . We base our experiments on our reproduced results based on the official weights of SD3.5-L [ 16 ] and FLUX.1-Dev [ 4 ] . The results show that SOLACE consistently results in improved compositionality, text rendering and text-image alignment, while being competitive at human preference metrics.

	Task-specific		Image Quality		Human Preference
	GenEval	OCR	ClipScore	Aesthetic	PickScore	HPSv2.1	ImageReward	UnifiedReward
SDXL [ 53 ]	0.23	0.127	0.284	5.58	22.34	0.274	0.67	2.92
+ SOLACE (Ours)	0.25	0.144	0.284	5.57	22.33	0.270	0.70	2.94

Table 4: Applying SOLACE to SDXL [ 53 ] . SOLACE yields improvements in compositional generation and text rendering on a UNet-based diffusion model, demonstrating architecture-agnostic applicability.

	Subj.	BG	Aesth.	Motion	Dyn.
	Consist.	Consist.	Qual.	Smooth.	Deg.
Wan2.1-1.3B	0.94	0.96	0.59	0.97	0.47
+ SOLACE	0.95	0.97	0.58	0.97	0.51

Table 5: Applying SOLACE to Wan2.1-1.3B for text-to-video generation. Evaluation on VBench-1.0 subset. SOLACE improves subject consistency, background consistency, and dynamic degree while maintaining competitive motion smoothness.

Figure 7: Qualitative results of SOLACE on Wan2.1-1.3B. SOLACE produces videos with improved visual quality and prompt adherence compared to the base model.

Inference at $512\times 512$
	Task-specific		Image Quality		Human Preference
	GenEval	OCR	ClipScore	Aesthetic	PickScore	HPSv2.1	ImageReward	UnifiedReward
SD3.5-M	0.65	0.61	0.282	5.36	22.34	0.279	0.84	3.08
+ SOLACE 512	0.71	0.67	0.288	5.39	22.41	0.284	0.87	3.10
+ SOLACE 1024	0.68	0.63	0.284	5.39	22.39	0.284	0.87	3.10
Inference at $1024\times 1024$

Table 6: Resolution analysis. SOLACE trained at

512\times 512

transfers effectively to

1024\times 1024

inference, with consistent gains in compositional generation and text rendering across resolutions.

	Task-specific		Image Quality		Human Preference
	GenEval	OCR	ClipScore	Aesthetic	PickScore	HPSv2.1	ImageReward	UnifiedReward
SD3.5-M	0.65	0.61	0.282	5.36	22.34	0.279	0.84	3.08
+ SOLACE (Ours)	0.71	0.67	0.288	5.39	22.41	0.278	0.87	3.11
Gemini 2.5-Flash	0.75	0.72	0.270	5.70	23.02	0.287	0.79	3.45
GPT-image-1.5	0.84	0.81	0.286	5.54	23.24	0.301	1.11	3.57

Table 7: Comparison with closed-source models. While closed-source models achieve higher absolute scores due to larger capacities and proprietary training, SOLACE narrows the gap from the SD3.5-M baseline, particularly in compositional generation and text rendering.

Figure 8: Visualization of training collapse in SOLACE. Self-confidence (y-axis) versus training iteration under different settings. Using $\rho>0.6$ or sampling rollouts without CFG drives a steep, short-horizon increase in self-confidence, followed by degenerate outputs—evidence of reward hacking. SOLACE’s default settings ( $\rho{=}0.6$ and CFG for rollouts) avoid this behavior while preserving steady improvements.

	CLIPScore $\uparrow$	Diversity Score $\uparrow$
	(RareBench)	(DrawBench)
SD3.5-M	0.2752	0.9519
SD3.5-M + SOLACE	0.2746	0.9545

Table 8: Semantic correctness and diversity analysis. CLIPScore on RareBench [ 50 ] (rare compositions) and diversity score on DrawBench [ 61 ] (64 samples per prompt, 50 prompts) show that SOLACE preserves both semantic accuracy on uncommon concepts and sample diversity.

	Task-specific		Image Quality		Human Preference
	GenEval	OCR	ClipScore	Aesthetic	PickScore	HPSv2.1	ImageReward	UnifiedReward
SD3.5-M	0.65	0.61	0.282	5.36	22.34	0.279	0.84	3.08
+ SOLACE	0.71	0.67	0.288	5.39	22.41	0.278	0.87	3.11
+ SOLACE (positive-only)	0.69	0.62	0.285	5.80	21.57	0.281	0.91	3.20

Table 9: Effect of negative advantages. Removing negative advantages (positive-only variant) degrades compositional generation, text rendering, and text-image alignment, demonstrating that the full signed advantage is important for SOLACE’s effectiveness.

(i) Text-rendering (OCR) prompts — default
Characteristics	Dense, explicit textual content (exact strings, font/placement hints), strong conditioning for legibility and alignment.
Examples	“A postage stamp design featuring the motto ”Unity in Diversity”, showcasing a vibrant collage of people from various ethnic backgrounds, each holding hands in a circle, set against a backdrop of colorful, interwoven patterns symbolizing unity and cultural richness.”
	“In a luxurious hotel lobby, an elegant digital display above the elevator reads ”Now Playing”. Soft, ambient elevator music fills the space, enhancing the serene and welcoming atmosphere. A plush, modern sofa and a glass coffee table are seen in the foreground, with polished marble floors reflecting the ambient light.”
	“A sleek, modern corporate lobby featuring a large, minimalist sculpture prominently inscribed with ”Innovate or Perish”, reflecting the company’s commitment to forward-thinking. The sculpture stands against a backdrop of glass and steel, with subtle lighting enhancing its form and the powerful message it conveys.”
(ii) PickScore prompts
Characteristics	Open-ended statements; often adds context with simple concatentation of adjectives; weaker constraints on text content/layout.

Table 10: Prompt sources compared for SOLACE. Denser, text-focused prompts (OCR) provide stronger supervision signals for intrinsic self-confidence, leading to larger gains than more open-ended (PickScore) or simple compositional (GenEval) prompts.

Caption dataset used for SOLACE
	Task-specific		Image Quality		Human Preference
	GenEval	OCR	ClipScore	Aesthetic	PickScore	HPSv2.1	ImageReward	UnifiedReward
-	0.65	0.61	0.282	5.36	22.34	0.279	0.84	3.08
PickScore prompts	0.70	0.62	0.285	5.26	22.13	0.278	0.65	2.96
GenEval prompts	0.71	0.62	0.286	5.32	22.35	0.275	0.80	3.05
OCR prompts (Ours)	0.71	0.67	0.288	5.39	22.41	0.278	0.87	3.11

Table 11: Additional ablation/comparative results. The results show that our current design choices for the (1) Caption dataset used, (2) Group size

G

, and (3) Aggregated self-confidence rewrads yield the best performances.

Category	Hyperparameter	Value (SOLACE, SD3.5-M)
Model	Base model	stabilityai/stable-diffusion-3.5-medium (SD3.5-M)
	Components trained	Transformer (denoiser) only; VAE and all text encoders frozen
LoRA	LoRA usage	use_lora = True
	Rank $r$	$32$
	Scaling factor $\alpha$	$64$
	Init of LoRA weights	Gaussian

Table 12: Hyperparameters and key implementation details for SOLACE training on SD3.5-M.

Figure 9: User study interface used to collect human preferences between pairs of AI-generated images.

Figure 10: Additional qualitative results of SOLACE. We present additional qualitative results of SOLACE when applied to (1) Flow-GRPO [ 41 ] post-trained SD3.5-M [ 16 ] , (2) FLUX.1-Dev [ 4 ] , and (3) SD3.5-L [ 16 ] . Best viewed on electronics.

T2I Diffusion Models

Text-to-image (T2I) diffusion models generate images from prompts by learning to reverse noise under text conditioning.

See all 394 papers

Post-Training Frontier Text-to-Image Models by Composing Preference and Rubric Rewards
Oct 2, 2026Yuanhao Ban, I-Hung Hsu, Anastasios Angelopoulos +3T2I Generation Reward Design for RL
Arena Intelligence Inc · UCLA
APT: Feedback-Driven Agentic Post-Training for Text-to-Image Generation
Oct 13, 2026Sirui Zhang, Fangyun Wei, Dong Chen +5T2I Generation RL for Image Generation
University of Science and Technology of China · Microsoft Research Asia · University of Sydney +2
TextAlign: Preference Alignment for Text Rendering with Hierarchical Rewards
May 19, 2026Mingxuan Cui, Jingpu Yang, Fengxian Ji +7Diffusion Model Alignment Reward Modeling
Mohamed bin Zayed University of Artificial Intelligence · Chinese Academy of Sciences Institute of Automation · Northeastern University +1

Oct 2, 2026J/K move · Enter open · S save

Post-Training Frontier Text-to-Image Models by Composing Preference and Rubric Rewards

Yuanhao Ban, I-Hung Hsu, Anastasios Angelopoulos, Wei-Lin Chiang+2

Arena Intelligence Inc · UCLA

T2I Generation Reward Design for RL RL for Image Generation Reward Modeling T2I Generation Evaluation

Open paper AskPDF arXiv

Recent text-to-image generation models have achieved remarkable visual quality, but improving them through post-training remains challenging because no single reward signal captures the full range of human preference. In this work, we develop a simple and effective post-training recipe for open-domain text-to-image generation based on the composition of complementary reward signals. Our reward system consists of two main components: a preference reward, trained on large-scale human preference data using a Bradley-Terry objective to capture overall human aesthetic and perceptual preferences, and rubric-based rewards, which explicitly evaluate prompt faithfulness and other desirable properties while providing safeguards against reward hacking. A key challenge is how to combine these heterogeneous reward signals. We show that a naive weighted average leads to suboptimal optimization behavior, and propose a simple reward composition strategy that more effectively balances preference optimization with rubric satisfaction. In the Arena text-to-image leaderboard (https://arena.ai/), our RL-trained Flux2dev achieves an Elo rating 69 points above the base model, and our post-trained Ideogram-4 surpasses every open-source model on the leaderboard, reaching an Elo of 1223.5. (Claims of state-of-the-art performance are based on the Arena leaderboard snapshot as of September 4, 2026.) Our results suggest that effective rewards for frontier generative-model training require broad coverage of user intent and robustness to exploitation under optimization. To support reproducible research, we release Arena-T2I-Training, a 1K subset of training data that recovers some gains of full-scale training, providing a resource that we hope will facilitate future work on post-training for text-to-image models.

Abstract

T2I Diffusion Models

Explore similar work

Post-Training Frontier Text-to-Image Models by Composing Preference and Rubric Rewards

APT: Feedback-Driven Agentic Post-Training for Text-to-Image Generation

TextAlign: Preference Alignment for Text Rendering with Hierarchical Rewards