Denoising Diffusion Probabilistic Models have emerged as powerful generative priors across domains ranging from molecular design to high-resolution image synthesis. However, standard diffusion sampling is inherently biased toward high-likelihood regions, often limiting the diversity of outputs. In many domains, a single, high-likelihood sample is insufficient. Instead, diverse portfolios of high-performing solutions are required, motivating methods that uncover a model's full expressive range. Here we show that integrating Quality-Diversity (QD) optimisation with diffusion models enables the discovery of diverse, high-performing outputs. We first propose Quality-Diversity Diffusion, a framework that adapts QD to diffusion models by directly optimising the model's latent noise and conditioning embeddings and leveraging embedding models to quantify diversity. Second, we introduce a novel h-space Spectral Mutation operator, which guides search along directions of maximal semantic sensitivity by analysing principal singular vectors of the model's input-to-activation Jacobian. We evaluate our combined approach, Spectral-Guided Quality-Diversity Diffusion, on text-to-image generation. Against 12 baselines, our method consistently achieves the lowest FID score, demonstrating an excellent balance of image quality and diversity.
Figures & tables
Figure 1: Flux.1 [dev] ( Labs, 2024 ) outputs under standard sampling versus our Spectral-Guided Quality-Diversity Diffusion. (Left) “a cube made of porcupine”, (Middle) “a street with several cars on it”, and (Right) “a knight holding a long sword”. Random sampling converges to structurally similar modes. Our method exhibits more visual diversity while maintaining prompt alignment.
Figure 2: Visualisation of the reverse diffusion process.
Figure 3: Quality-Diversity Diffusion Framework.
Figure 4: The h -space Spectral Mutation Operator. Initial noise xT is propagated to a timestep t∗ where singular vectors v of the Jacobian mapping the latent xt to the model’s internal activations h are computed. The latent is perturbed along this direction ( x~t=xt+αv ) and the result x~t is deterministically inverted to the timestep to form the offspring genotype x~T .
Table 5
Figure 5: Quality vs Diversity.
Table 7
Figure 6: Quality vs Diversity.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Pie charts representing the Category and Challenge Distribution of the sampled prompts.
Algorithm
Total Fitness
Avg. Pairwise Dist.
Num. Solutions
Spectral-Guided QD-Diffusion (Ours)
20.8 ± 3.0
0.473 ± 0.08
62.7 ± 8.0
Fitness Threshold Ablation
17.56 ± 1.8
0.673 ± 0.07
64.0 ± 0.0
Appendix
Table 5: Comparison between our full method and the fitness threshold ablation across diversity and fitness metrics.
Figure 8: Qualitative comparison for the prompt “a knight holding a long sword”. The ablation (a) exhibits high diversity but often fails to meet semantic constraints, whereas our method (b) maintains high fitness and validity.
Threshold Value
Total Fitness
Avg. Pairwise Dist.
Num. Solutions
0.26
20.090 ± 0.965
0.576 ± 0.069
64.000 ± 0.000
0.28
20.673 ± 1.100
0.542 ± 0.068
64.000 ± 0.000
0.30 (Default)
21.104 ± 1.143
0.500 ± 0.072
64.000 ± 0.000
0.32
17.647 ± 8.035
0.344 ± 0.146
52.000 ± 21.709
0.34
12.454 ± 9.789
0.256 ± 0.137
36.083 ± 24.645
Appendix
Table 6: Robustness analysis of the fitness threshold. Performance varies smoothly, with 0.3 serving as a stable default.
Algorithm
Total Fitness
Diversity
No. of Solutions
Mixed Genotype
20.782 ± 3.013
0.473 ± 0.083
62.667 ± 8.000
Latent Noise
20.393 ± 4.238
0.351 ± 0.093
61.333 ± 11.151
Prompt Embedding
19.988 ± 4.005
0.480 ± 0.096
61.333 ± 11.151
Pooled Embedding
18.792 ± 6.042
0.357 ± 0.150
57.056 ± 16.805
Appendix
Table 7: Performance comparison across algorithms. Bold values indicate the best performer or algorithms not significantly worse than the best (p ≥ 0.05).
Algorithm
Total Fitness
Avg. Pairwise Dist.
Num. Solutions
α=5
19.668 ± 5.020
0.451 ± 0.070
60.000 ± 13.856
α=10 (Default)
21.104 ± 1.143
0.500 ± 0.072
64.000 ± 0.000
α=20
21.087 ± 1.015
0.562 ± 0.059
64.000 ± 0.000
Appendix
Table 8: Ablation of mutation strength α . The method remains stable across varying perturbation magnitudes.
Algorithm
Total Fitness
Avg. Pairwise Dist.
Num. Solutions
k=1
18.617 ± 6.630
0.453 ± 0.083
56.615 ± 18.026
k=3 (Default)
21.104 ± 1.090
0.495 ± 0.071
64.000 ± 0.000
k=5
21.091 ± 1.104
0.480 ± 0.075
64.000 ± 0.000
k=10
21.191 ± 1.158
0.465 ± 0.068
64.000 ± 0.000
Appendix
Table 9: Impact of the number of spectral directions k on algorithm performance.
Target Layer
Total Fitness
Avg. Pairwise Dist.
Num. Solutions
mid_blocks (Default)
21.104 ± 1.143
0.500 ± 0.072
64.000 ± 0.000
down_blocks.2
21.380 ± 1.170
0.499 ± 0.071
64.000 ± 0.000
up_blocks.1
20.021 ± 5.551
0.465 ± 0.093
59.636 ± 14.473
up_blocks.2
20.098 ± 5.596
0.467 ± 0.109
59.636 ± 14.473
Appendix
Table 10: Comparison of spectral mutation performance when applied to different U-Net layers.
Figure 9: SDXL-Turbo Top-16 images with highest fitness for sample prompts.
Figure 10: Flux Top-16 images with highest fitness for sample prompts.
Figure 11: SDXL-Turbo Top-16 images with highest fitness for sample prompts for Variation Operator Experiments.
Figure 12: Flux top-16 images with highest fitness for sample prompts for Variation Operator Experiments.
ID
Category
Challenge Level
Prompt
1
Abstract
Basic
happiness
2
Abstract
Basic
metal
3
Abstract
Basic
element
4
Vehicles
Simple Detail
an airplane taking off of a runway
5
Vehicles
Fine-grained Detail
an antique car by a beach
6
Vehicles
Fine-grained Detail
a drop-top sports car coming around a bend in the road
Appendix
Table 11: List of prompts with categories and challenge levels.
Algorithm
SDXL-Turbo (s)
Flux (s)
Spectral-Guided (Ours)
0.78
14564.48
Sep-CMA-ES
0.002
7.22
CMA-MAE
0.001
6.96
Adam
0.90
-
Random Seeds
0.0003
1.37
Diverse Diffusion
0.002
16.39
Appendix
Table 12: Computational Overhead: Median Time per Image (Seconds)
Diffusion model sampling studies how to generate high-quality samples efficiently by designing reverse processes, solvers, and inference-time controls.