Denoising Diffusion Probabilistic Models have emerged as powerful generative priors across domains ranging from molecular design to high-resolution image synthesis. However, standard diffusion sampling is inherently biased toward high-likelihood regions, often limiting the diversity of outputs. In many domains, a single, high-likelihood sample is insufficient. Instead, diverse portfolios of high-performing solutions are required, motivating methods that uncover a model's full expressive range. Here we show that integrating Quality-Diversity (QD) optimisation with diffusion models enables the discovery of diverse, high-performing outputs. We first propose Quality-Diversity Diffusion, a framework that adapts QD to diffusion models by directly optimising the model's latent noise and conditioning embeddings and leveraging embedding models to quantify diversity. Second, we introduce a novel h-space Spectral Mutation operator, which guides search along directions of maximal semantic sensitivity by analysing principal singular vectors of the model's input-to-activation Jacobian. We evaluate our combined approach, Spectral-Guided Quality-Diversity Diffusion, on text-to-image generation. Against 12 baselines, our method consistently achieves the lowest FID score, demonstrating an excellent balance of image quality and diversity.
Figures & tables
Figure 1: Flux.1 [dev] ( Labs, 2024 ) outputs under standard sampling versus our Spectral-Guided Quality-Diversity Diffusion. (Left) “a cube made of porcupine”, (Middle) “a street with several cars on it”, and (Right) “a knight holding a long sword”. Random sampling converges to structurally similar modes. Our method exhibits more visual diversity while maintaining prompt alignment.
Figure 2: Visualisation of the reverse diffusion process.
Figure 3: Quality-Diversity Diffusion Framework.
Figure 4: The h -space Spectral Mutation Operator. Initial noise xT is propagated to a timestep t∗ where singular vectors v of the Jacobian mapping the latent xt to the model’s internal activations h are computed. The latent is perturbed along this direction ( x~t=xt+αv ) and the result x~t is deterministically inverted to the timestep to form the offspring genotype x~T .
Table 5
Figure 5: Quality vs Diversity.
Table 7
Figure 6: Quality vs Diversity.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Pie charts representing the Category and Challenge Distribution of the sampled prompts.
Algorithm
Total Fitness
Avg. Pairwise Dist.
Num. Solutions
Spectral-Guided QD-Diffusion (Ours)
20.8 ± 3.0
0.473 ± 0.08
62.7 ± 8.0
Fitness Threshold Ablation
17.56 ± 1.8
0.673 ± 0.07
64.0 ± 0.0
Appendix
Table 5: Comparison between our full method and the fitness threshold ablation across diversity and fitness metrics.
Figure 8: Qualitative comparison for the prompt “a knight holding a long sword”. The ablation (a) exhibits high diversity but often fails to meet semantic constraints, whereas our method (b) maintains high fitness and validity.
Threshold Value
Total Fitness
Avg. Pairwise Dist.
Num. Solutions
0.26
20.090 ± 0.965
0.576 ± 0.069
64.000 ± 0.000
0.28
20.673 ± 1.100
0.542 ± 0.068
64.000 ± 0.000
0.30 (Default)
21.104 ± 1.143
0.500 ± 0.072
64.000 ± 0.000
0.32
17.647 ± 8.035
0.344 ± 0.146
52.000 ± 21.709
0.34
12.454 ± 9.789
0.256 ± 0.137
36.083 ± 24.645
Appendix
Table 6: Robustness analysis of the fitness threshold. Performance varies smoothly, with 0.3 serving as a stable default.
Algorithm
Total Fitness
Diversity
No. of Solutions
Mixed Genotype
20.782 ± 3.013
0.473 ± 0.083
62.667 ± 8.000
Latent Noise
20.393 ± 4.238
0.351 ± 0.093
61.333 ± 11.151
Prompt Embedding
19.988 ± 4.005
0.480 ± 0.096
61.333 ± 11.151
Pooled Embedding
18.792 ± 6.042
0.357 ± 0.150
57.056 ± 16.805
Appendix
Table 7: Performance comparison across algorithms. Bold values indicate the best performer or algorithms not significantly worse than the best (p ≥ 0.05).
Algorithm
Total Fitness
Avg. Pairwise Dist.
Num. Solutions
α=5
19.668 ± 5.020
0.451 ± 0.070
60.000 ± 13.856
α=10 (Default)
21.104 ± 1.143
0.500 ± 0.072
64.000 ± 0.000
α=20
21.087 ± 1.015
0.562 ± 0.059
64.000 ± 0.000
Appendix
Table 8: Ablation of mutation strength α . The method remains stable across varying perturbation magnitudes.
Algorithm
Total Fitness
Avg. Pairwise Dist.
Num. Solutions
k=1
18.617 ± 6.630
0.453 ± 0.083
56.615 ± 18.026
k=3 (Default)
21.104 ± 1.090
0.495 ± 0.071
64.000 ± 0.000
k=5
21.091 ± 1.104
0.480 ± 0.075
64.000 ± 0.000
k=10
21.191 ± 1.158
0.465 ± 0.068
64.000 ± 0.000
Appendix
Table 9: Impact of the number of spectral directions k on algorithm performance.
Target Layer
Total Fitness
Avg. Pairwise Dist.
Num. Solutions
mid_blocks (Default)
21.104 ± 1.143
0.500 ± 0.072
64.000 ± 0.000
down_blocks.2
21.380 ± 1.170
0.499 ± 0.071
64.000 ± 0.000
up_blocks.1
20.021 ± 5.551
0.465 ± 0.093
59.636 ± 14.473
up_blocks.2
20.098 ± 5.596
0.467 ± 0.109
59.636 ± 14.473
Appendix
Table 10: Comparison of spectral mutation performance when applied to different U-Net layers.
Figure 9: SDXL-Turbo Top-16 images with highest fitness for sample prompts.
Figure 10: Flux Top-16 images with highest fitness for sample prompts.
Figure 11: SDXL-Turbo Top-16 images with highest fitness for sample prompts for Variation Operator Experiments.
Figure 12: Flux top-16 images with highest fitness for sample prompts for Variation Operator Experiments.
ID
Category
Challenge Level
Prompt
1
Abstract
Basic
happiness
2
Abstract
Basic
metal
3
Abstract
Basic
element
4
Vehicles
Simple Detail
an airplane taking off of a runway
5
Vehicles
Fine-grained Detail
an antique car by a beach
6
Vehicles
Fine-grained Detail
a drop-top sports car coming around a bend in the road
Appendix
Table 11: List of prompts with categories and challenge levels.
Algorithm
SDXL-Turbo (s)
Flux (s)
Spectral-Guided (Ours)
0.78
14564.48
Sep-CMA-ES
0.002
7.22
CMA-MAE
0.001
6.96
Adam
0.90
-
Random Seeds
0.0003
1.37
Diverse Diffusion
0.002
16.39
Appendix
Table 12: Computational Overhead: Median Time per Image (Seconds)
Diffusion model sampling studies how to generate high-quality samples efficiently by designing reverse processes, solvers, and inference-time controls.
Image diversity remains a fundamental challenge for text-to-image diffusion models. Low-diversity generation often leads to repetitive outputs, increasing sampling redundancy and hindering both creative exploration and downstream applications. A key factor is the tendency of diffusion models to collapse toward strong modes in the learned distribution. Existing attempts to improve diversity, such as steering-based guidance, often introduce distortions that degrade image quality. To address this issue, we propose Token-Prompt Embedding Space Optimization (TPSO), a training-free and model-agnostic module. TPSO introduces learnable parameters to explore underrepresented regions of the token embedding space, reducing the tendency to repeatedly sample from strong modes of the distribution. Meanwhile, a prompt-level semantic constraint regulates distribution shifts, preventing quality degradation while preserving semantic fidelity. Extensive experiments on MS-COCO across three representative diffusion backbones demonstrate that TPSO substantially improves diversity, boosting performance from 1.10 to 4.18, while maintaining image quality with only a modest inference-time overhead of 3.6% to 8.9%. Code is available at: https://github.com/Open-Debin/TPSO.
Debin Meng, Chen Jin, Zheng Gao +3
Queen Mary University of London · Centre for AI, AstraZeneca, UK · University of Bedfordshire +1
Few-step distilled diffusion models generate high-quality images quickly, but often lose per-prompt diversity, producing near-identical samples across random seeds. Optimizing the initial noise at inference time offers an appealing way to recover this diversity, yet existing methods directly update the initial noise in an unconstrained Euclidean space, ignoring both the geometry of the Gaussian prior and the model's sensitivity to noise frequencies. They therefore introduce auxiliary quality-control objectives to maintain generation fidelity, adding compute and weighting hyperparameters while still requiring conservative updates to prevent degradation. In this work, we propose MoNO, a training-free method that performs Manifold-constrained Noise Optimization on a low-dimensional, quality-stabilizing noise manifold. MoNO sequentially optimizes each new initial noise so that its predicted visual feature complements previous generations, while Riemannian updates on an affine low-frequency sphere preserve prior likelihood and fix unstable high-frequency components by construction. This enables large geodesic steps, removes the need for auxiliary quality-control objectives, and converges in far fewer iterations than prior noise-optimization methods. Experiments with multiple distilled text-to-image diffusion models show that MoNO consistently improves per-prompt diversity while maintaining image quality.
We introduce DiverseVAR, a framework that enhances the diversity of text-conditioned visual autoregressive models (VAR) at test time without requiring retraining, fine-tuning, or substantial computational overhead. While VAR models have recently emerged as strong competitors to diffusion and flow models for image generation, they suffer from a critical limitation in diversity, often producing nearly identical images even for simple prompts. This issue has largely gone unnoticed amid the predominant focus on image quality. We address this limitation at test time in two stages. First, inspired by diversity enhancement techniques in diffusion models, we propose injecting noise into the text embedding. This introduces a trade-off between diversity and image quality: as diversity increases, the image quality sharply declines. To preserve quality, we propose scale-travel: a novel latent refinement technique inspired by time-travel strategies in diffusion models. Specifically, we use a multi-scale autoencoder to extract coarse-scale tokens that enable us to resume generation at intermediate stages. Extensive experiments show that combining text-embedding noise injection with our scale-travel refinement significantly enhances diversity while minimizing image-quality degradation, achieving a new Pareto frontier in the diversity-quality trade-off. Project page: https://diverse-var.github.io
Mingue Park, Prin Phunyaphibarn, Phillip Y. Lee +1