Text-to-image diffusion models exhibit strong generative performance but remain highly sensitive to prompt formulation, often requiring extensive manual trial and error to obtain satisfactory results. This motivates the development of automated, model-agnostic prompt optimization methods that can systematically explore the conditioning space beyond conventional text rewriting. This work investigates the use of a Genetic Algorithm (GA) for prompt optimization by directly evolving the token vectors employed by CLIP-based diffusion models. The GA optimizes a fitness function that combines aesthetic quality, measured by the LAION Aesthetic Predictor V2, with prompt-image alignment, assessed via CLIPScore. Experiments on 36 prompts from the Parti Prompts (P2) dataset show that the proposed approach outperforms the baseline methods, including Promptist and random search, achieving up to a 23.93% improvement in fitness. Overall, the method is adaptable to image generation models with tokenized text encoders and provides a modular framework for future extensions, the limitations and prospects of which are discussed.
Figures & tables
Figure 1: General structure and workflow of the evolutionary token optimization studied in this work. The main components and their respective inputs and outputs are depicted.
Parameter
Value
Inference steps
1
Guidance Scale
0
Image Size
512x512
(a,b)
(0.4,0.6)
Generations
100
Population size
64
Table 1: Parameters used in the experimentation. The first set of parameters are related to SDXL Turbo, while the second is related to the GA.
Algorithm
LAION Aesthetic V2 [1,10] ↑
CLIPScore [-1,1] ↑
Fitness [0,1] ↑
Wins [0-36] ↑
Avg.
Std.
Max
Δ base (%)
Avg.
Std.
Max
Δ base (%)
Avg.
Std.
Max
Δ base (%)
# prompts
SDXL Turbo (no optimization)
5.78
0.56
6.81
0.00
0.2672
0.0482
0.3909
0.00
0.5519
0.0617
0.7311
0.00
0
GA Mutated
7.30
0.20
7.87
26.29
0.3266
0.0522
0.4209
22.22
0.6840
0.0596
0.7944
23.93
28
GA Empty
7.45
0.21
7.95
28.94
0.2562
0.0397
0.3752
-4.12
0.6056
0.0444
0.7450
9.73
1
GA Random
7.39
0.23
7.91
27.88
0.2248
0.0381
0.3040
-15.88
0.5654
0.0423
0.6572
2.45
0
Random Search
6.93
0.37
7.43
19.87
0.1946
0.0343
0.2830
-27.18
0.5107
0.0408
0.5947
-7.47
0
Table 2: Results comparison between SDXL Turbo with no optimization and prompt optimization methods under weights (a,b)=(0.4,0.6) . Each GA line corresponds to a different type of population initialization. The results are reported for LAION Aesthetic V2, CLIPScore, and Fitness. The columns show the mean, standard deviation, maximum, and percentage change relative to the original baseline. The highest mean and percentage change per metric are highlighted in bold, as well as the algorithm with highest average fitness.
Figure 2: Final outputs from baseline SDXL Turbo, GA Mutated, GA Empty, GA Random, Random Search, and Promptist for 6 prompts of the first 6 P2 categories (see Section 4 ). Rows correspond to prompts and columns to methods, with aesthetic, CLIP, and fitness scores above each image; purple marks the highest-fitness image, while red or blue mark the best aesthetic or CLIPScore when they do not match the fitness optimum.
Figure 3: Final outputs from baseline SDXL Turbo, GA Mutated, GA Empty, GA Random, Random Search, and Promptist for 6 prompts of the last 6 P2 categories (see Section 4 ). Rows correspond to prompts and columns to methods, with aesthetic, CLIP, and fitness scores above each image; purple marks the highest-fitness image, while red or blue mark the best aesthetic or CLIPScore when they do not match the fitness optimum
Diffusion models generate highly realistic images but often struggle with precise text-image alignment. While recent post-training methods improve alignment using external rewards or human preference signals, their performance heavily depends on reward quality and does not directly address alignment within the diffusion process itself. Recent reward-free approaches such as SoftREPA demonstrate that optimizing soft text tokens via contrastive learning can effectively improve text-image representation alignment, outperforming standard parameter-efficient fine-tuning baselines. However, the contrastive formulation can excessively penalize negative pairs, which manifests as characteristic failure cases such as over-counting and repetition. To address this issue, we propose a lightweight, reward-free post-training method that refines soft tokens by integrating contrastive alignment guidance directly into the score-matching objective of diffusion models. By assigning alignment directions at the score level, our approach mitigates these limitations and yields more coherent and semantically faithful generations. Experiments show that our method matches SoftREPA while substantially improving its failure cases, achieving over 35% improvement in counting accuracy on the GenEval benchmark. Our method is seamlessly applicable to existing diffusion backbones (SD1.5, SDXL, and SD3), and is complementary to existing RL-based diffusion post-training methods. Project page: https://jaayeon.github.io/AGSM
Image diversity remains a fundamental challenge for text-to-image diffusion models. Low-diversity generation often leads to repetitive outputs, increasing sampling redundancy and hindering both creative exploration and downstream applications. A key factor is the tendency of diffusion models to collapse toward strong modes in the learned distribution. Existing attempts to improve diversity, such as steering-based guidance, often introduce distortions that degrade image quality. To address this issue, we propose Token-Prompt Embedding Space Optimization (TPSO), a training-free and model-agnostic module. TPSO introduces learnable parameters to explore underrepresented regions of the token embedding space, reducing the tendency to repeatedly sample from strong modes of the distribution. Meanwhile, a prompt-level semantic constraint regulates distribution shifts, preventing quality degradation while preserving semantic fidelity. Extensive experiments on MS-COCO across three representative diffusion backbones demonstrate that TPSO substantially improves diversity, boosting performance from 1.10 to 4.18, while maintaining image quality with only a modest inference-time overhead of 3.6% to 8.9%. Code is available at: https://github.com/Open-Debin/TPSO.
Debin Meng, Chen Jin, Zheng Gao +3
Queen Mary University of London · Centre for AI, AstraZeneca, UK · University of Bedfordshire +1
Preference optimization has emerged as an efficient alternative to online reinforcement learning from human feedback (RLHF) for aligning text-to-image diffusion models. However, existing methods largely reduce supervision to binary pairwise comparisons. This pairwise reduction is limiting when training data naturally contains multiple candidate images for the same prompt, and when continuous reward scores can provide richer information than a single winner-loser label. To address these limitations, we propose Diffusion LAIR, a reward-aware listwise preference optimization method for diffusion models. For each prompt, LAIR converts reward scores across a group of candidate images into centered advantage weights, then optimizes an advantage-weighted regression objective on the implicit reward, defined as the denoising-loss improvement of the current model over a fixed reference model, with a quadratic penalty that regularizes the magnitude of the implicit reward. The resulting objective uses all candidates simultaneously rather than selecting pairs, and remains conservative by explicitly controlling the magnitude of the implicit reward. The LAIR objective admits a bounded closed-form optimum in implicit-reward space, clarifying how the regularization strength controls the magnitude of the preference update. Experiments show that Diffusion LAIR outperforms strong preference optimization baselines on SD1.5 and SDXL across text-to-image generation, compositional generation, and image editing benchmarks.