Evolutionary Token-Level Prompt Optimization for Diffusion Models

Domício Pereira Neto; João Correia; Penousal Machado

cs.AIApr 10, 2026

Evolutionary Token-Level Prompt Optimization for Diffusion Models

Authors: Domício Pereira Neto, João Correia, Penousal Machado

Organizations: University of Coimbra Coimbra, Portugal

Abstract

Text-to-image diffusion models exhibit strong generative performance but remain highly sensitive to prompt formulation, often requiring extensive manual trial and error to obtain satisfactory results. This motivates the development of automated, model-agnostic prompt optimization methods that can systematically explore the conditioning space beyond conventional text rewriting. This work investigates the use of a Genetic Algorithm (GA) for prompt optimization by directly evolving the token vectors employed by CLIP-based diffusion models. The GA optimizes a fitness function that combines aesthetic quality, measured by the LAION Aesthetic Predictor V2, with prompt-image alignment, assessed via CLIPScore. Experiments on 36 prompts from the Parti Prompts (P2) dataset show that the proposed approach outperforms the baseline methods, including Promptist and random search, achieving up to a 23.93% improvement in fitness. Overall, the method is adaptable to image generation models with tokenized text encoders and provides a modular framework for future extensions, the limitations and prospects of which are discussed.

Figures & tables

Figure 1: General structure and workflow of the evolutionary token optimization studied in this work. The main components and their respective inputs and outputs are depicted.

Parameter	Value
Inference steps	1
Guidance Scale	0
Image Size	512x512
$(a,b)$	$(0.4,0.6)$
Generations	100
Population size	64

Table 1: Parameters used in the experimentation. The first set of parameters are related to SDXL Turbo, while the second is related to the GA.

Algorithm	LAION Aesthetic V2 [1,10] $\uparrow$				CLIPScore [-1,1] $\uparrow$				Fitness [0,1] $\uparrow$				Wins [0-36] $\uparrow$
Algorithm	Avg.	Std.	Max	$\Delta$ base (%)	Avg.	Std.	Max	$\Delta$ base (%)	Avg.	Std.	Max	$\Delta$ base (%)	# prompts
SDXL Turbo (no optimization)	5.78	0.56	6.81	0.00	0.2672	0.0482	0.3909	0.00	0.5519	0.0617	0.7311	0.00	0
GA Mutated	7.30	0.20	7.87	26.29	0.3266	0.0522	0.4209	22.22	0.6840	0.0596	0.7944	23.93	28
GA Empty	7.45	0.21	7.95	28.94	0.2562	0.0397	0.3752	-4.12	0.6056	0.0444	0.7450	9.73	1
GA Random	7.39	0.23	7.91	27.88	0.2248	0.0381	0.3040	-15.88	0.5654	0.0423	0.6572	2.45	0
Random Search	6.93	0.37	7.43	19.87	0.1946	0.0343	0.2830	-27.18	0.5107	0.0408	0.5947	-7.47	0

Table 2: Results comparison between SDXL Turbo with no optimization and prompt optimization methods under weights

(a,b)=(0.4,0.6)

. Each GA line corresponds to a different type of population initialization. The results are reported for LAION Aesthetic V2, CLIPScore, and Fitness. The columns show the mean, standard deviation, maximum, and percentage change relative to the original baseline. The highest mean and percentage change per metric are highlighted in bold, as well as the algorithm with highest average fitness.

Figure 2: Final outputs from baseline SDXL Turbo, GA Mutated, GA Empty, GA Random, Random Search, and Promptist for 6 prompts of the first 6 P2 categories (see Section 4 ). Rows correspond to prompts and columns to methods, with aesthetic, CLIP, and fitness scores above each image; purple marks the highest-fitness image, while red or blue mark the best aesthetic or CLIPScore when they do not match the fitness optimum.

Figure 3: Final outputs from baseline SDXL Turbo, GA Mutated, GA Empty, GA Random, Random Search, and Promptist for 6 prompts of the last 6 P2 categories (see Section 4 ). Rows correspond to prompts and columns to methods, with aesthetic, CLIP, and fitness scores above each image; purple marks the highest-fitness image, while red or blue mark the best aesthetic or CLIPScore when they do not match the fitness optimum

Prompt Optimization for T2I Generation

Prompt optimization for text-to-image generation improves how natural-language instructions control image content, style, composition, and quality.

See all 32 papers

Alignment-Guided Score Matching for Text-to-Image Alignment in Diffusion Models
May 28, 2026Jaa-Yeon Lee, Yeobin Hong, Taesung Kwon +1Diffusion Model Alignment Text-Image Alignment
Graduate School of AI, KAIST, South Korea.
TPSO: Training-Free Diverse Image Generation via Semantic Prompt Embedding Optimization
Nov 25, 2025Debin Meng, Chen Jin, Zheng Gao +3T2I Generation T2I Diffusion Models
Queen Mary University of London · Centre for AI, AstraZeneca, UK · University of Bedfordshire +1
Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models
May 26, 2026Austin Wang, Jiaqi Han, Stefano Ermon +1Diffusion Model Alignment Preference Optimization
Caltech · Stanford University

May 28, 2026J/K move · Enter open · S save

Alignment-Guided Score Matching for Text-to-Image Alignment in Diffusion Models

Jaa-Yeon Lee, Yeobin Hong, Taesung Kwon, Jong Chul Ye

Graduate School of AI, KAIST, South Korea.

Diffusion Model Alignment Text-Image Alignment Diffusion Guidance Diffusion Model Fine-Tuning

Open paper AskPDF arXiv

Diffusion models generate highly realistic images but often struggle with precise text-image alignment. While recent post-training methods improve alignment using external rewards or human preference signals, their performance heavily depends on reward quality and does not directly address alignment within the diffusion process itself. Recent reward-free approaches such as SoftREPA demonstrate that optimizing soft text tokens via contrastive learning can effectively improve text-image representation alignment, outperforming standard parameter-efficient fine-tuning baselines. However, the contrastive formulation can excessively penalize negative pairs, which manifests as characteristic failure cases such as over-counting and repetition. To address this issue, we propose a lightweight, reward-free post-training method that refines soft tokens by integrating contrastive alignment guidance directly into the score-matching objective of diffusion models. By assigning alignment directions at the score level, our approach mitigates these limitations and yields more coherent and semantically faithful generations. Experiments show that our method matches SoftREPA while substantially improving its failure cases, achieving over 35% improvement in counting accuracy on the GenEval benchmark. Our method is seamlessly applicable to existing diffusion backbones (SD1.5, SDXL, and SD3), and is complementary to existing RL-based diffusion post-training methods. Project page: https://jaayeon.github.io/AGSM

Abstract

Prompt Optimization for T2I Generation

Explore similar work

Alignment-Guided Score Matching for Text-to-Image Alignment in Diffusion Models

TPSO: Training-Free Diverse Image Generation via Semantic Prompt Embedding Optimization

Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models