Strong guidance in text-to-image (T2I) diffusion models concentrates sampling toward high-density modes of the conditional distribution, collapsing independently sampled noise vectors onto nearly identical visual prototypes. This limits utility for tasks that demand distributional breadth, most critically synthetic data augmentation. Diversity, however, is a collective property of the whole batch: a set can preserve every pairwise distance yet still collapse onto a low-dimensional subspace. Existing sum-of-pairwise noise-optimization and repulsion objectives respond only weakly to this higher-order failure. We propose DetNO (Determinant Noise Optimization), which formulates diverse generation as RKHS volume maximization on the initial noise, maximizing the log-determinant of an RBF Gram matrix over one-step Tweedie predictions. This set-level functional equals the squared batch volume; its repulsion is strongest when the batch is most collapsed and self-attenuates as diversity is achieved. An anchor-based fidelity term keeps the optimized noises near plausible regions, and Lagrangian relaxation gives a two-term objective with a single hyperparameter λ governing the diversity-quality trade-off, without training or sampler modification. Across three backbones (SD1.5, SDXL, SD3), DetNO achieves leading intra-batch diversity in VendiScore and Mean Self-Similarity, and a downstream few-shot classification experiment confirms that broader mode coverage brings statistically significant accuracy gains for synthetic data augmentation in the low-shot regime. Fixing the pipeline and varying only the batch objective, both RBF-based aggregators outperform InfoNCE, and the log-determinant attains the best downstream accuracy on both benchmarks. Its diversity advantage over InfoNCE widens as the batch grows. Code is available at https://github.com/mobled37/DetNO.
Figures & tables
Figure 1: Qualitative comparison on SD3. SD3 baseline sampling collapses onto a narrow visual prototype for underspecified prompts, reproducing nearly identical appearance and scene layout across seeds, a direct consequence of CFG concentrating the conditional distribution onto its high-density peak. DetNO recovers the suppressed degrees of freedom in initial noise, producing semantically diverse outputs that vary in viewpoint, identity, clothes, and scene composition while remaining faithful to the prompt.
Figure 2: Conceptual illustration of DetNO. (a) Strong CFG guidance concentrates sampling toward the high-density peak, leaving the degrees of freedom in the initial noise unexploited. DetNO recovers this freedom by optimizing the initial noise batch to reach low-density regions that standard sampling cannot access. (b) The diversity term (green) maximizes the RKHS volume collectively spanned by all Tweedie predictions, while the fidelity term (yellow) regularizes each sample toward its unoptimized reference, expanding distributional coverage while preserving prompt fidelity.
Figure 3: Toy example on a 2D ring mixture with 8 inner modes (80%) and 8 outer modes (20%). (a) Ground truth distribution. (b) Classifier-free guidance collapses onto the majority inner ring, without covering outer modes. (c) CNO recovers the outer modes but severely distorts the inner ring, producing only 4 inner modes whose sample distributions no longer align with the ground truth geometry. (d) DetNO achieves full coverage of all 16 modes across both rings, faithfully preserving the original mode structure.
Reference-Based
Reference-Free
Diversity
Model
Method
Prec ↑
Rec ↑
Den ↑
Cov ↑
CLIP ↑
Pick ↑
IR ↑
MSS ↓
Vendi ↑
SD1.5
DDIM
0.7005
0.6061
0.6112
0.3631
30.9096
21.3413
0.1086
0.1509
4.7352
PG
0.7008
0.6044
0.6120
0.3634
30.9103
21.3424
0.1085
0.1508
4.7357
SPELL
0.6722
0.5645
0.5639
0.3439
30.7538
21.2556
0.0264
0.1409
4.7636
DiversityPrompt
0.6679
0.6442
0.5345
0.3370
30.9380
21.1956
0.1194
0.1455
4.7509
CNO
0.7247
0.6318
0.6647
0.3933
30.7925
21.2984
0.0702
0.1383
4.7679
Table 1: Quantitative comparison across different backbone models. Bold and underline denote the best and second-best results, respectively.
Figure 4: Qualitative comparison on SD1.5. For each prompt, we show one representative sample from DDIM, CNO, and DetNO (ours). DDIM and CNO produce visually near-identical outputs across seeds, differing only in minor details such as slight pose or hair difference. DetNO generates substantially different images, varying in viewpoint, object appearance, scene layout, and human identity, while remaining faithful to the prompt.
Oxford-IIIT Pets (SD3)
CUB-200-2011 (SD1.5)
Method
1-shot
2-shot
4-shot
8-shot
1-shot
2-shot
4-shot
8-shot
Real only
17.6 ± 2.0
41.2 ± 2.9
57.7 ± 2.7
65.4 ± 0.7
8.2 ± 0.5
15.1 ± 0.6
27.9 ± 0.9
47.0 ± 1.3
+ DDIM
52.2 ± 1.8
58.2 ± 0.9
59.1 ± 1.3
71.8 ± 0.7
24.6 ± 0.9
31.1 ± 0.9
38.5 ± 0.9
50.9 ± 0.8
+ CNO
51.7 ± 1.7
57.8 ± 0.6
59.4 ± 1.6
71.4 ± 0.7
24.5 ± 0.7
30.6 ± 0.8
38.4 ± 0.5
50.7 ± 1.1
+ DetNO
54.2 ± 1.1 ‡
59.3 ± 1.3 ‡
61.2 ± 1.7 †
71.8 ± 0.9
26.3 ± 0.5 †‡
32.1 ± 0.7 ‡
39.8 ± 0.6 †‡
51.8 ± 0.9
Table 2: Downstream few-shot classification accuracy (%). Mean ± std over 10 splits. Synthetic images (16/class) generated with SD3 (Oxford-IIIT Pets) and SD1.5 (CUB-200-2011). † and ‡ denote statistically significant improvement over DDIM and CNO respectively (Welch’s t -test, p<0.01 ).
Method
SD1.5 (intra-batch)
1-shot accuracy (%)
Pick ↑
Vendi ↑
Pets (SD3) ↑
CUB (SD1.5) ↑
DDIM (Baseline)
21.34
4.7352
52.2 ± 1.8
24.6 ± 0.9
CNO (InfoNCE)
21.30
4.7679
51.7 ± 1.7
24.5 ± 0.7
∑i<jSij (pairwise-RBF)
21.23
4.8006
53.7 ± 1.1
25.8 ± 0.6
−logdet (DetNO)
21.25
4.8024
54.2 ± 1.1
26.3 ± 0.5
Table 3: Pairwise-RBF ablation with downstream evaluation. SD1.5, B=5 , λ=0.05 ; S , σ , fidelity term, and optimizer are held identical across rows, so only the batch aggregator differs: InfoNCE (CNO), the sum-of-pairwise RBF ∑i<jSij , and the log-determinant (DetNO). Downstream columns follow the protocol of Tab. 2 . Bold and underline denote the best and second-best results.
Figure 5: Quality–diversity Pareto curves on SD3 for different pooling windows w∈{8,16,32,64} . Each point on a curve corresponds to a different fidelity weight λ∈{0.01,0.05,0.1} , evaluated on 200 randomly sampled MS-COCO prompts with 5 images per prompt. All configurations improve over the unoptimized baseline (gray star) on this evaluation. Smaller w yields a wider diversity range at the cost of a steeper quality trade-off, while larger w preserves quality more effectively with moderate diversity gains.
A
B
Prec ↑
Recall ↑
Den ↑
Cov ↑
CLIP ↑
Pick ↑
IR ↑
MSS ↓
Vendi ↑
0.7005
0.6061
0.6112
0.3631
30.9096
21.3413
0.1086
0.1509
4.7352
✓
0.6816
0.6191
0.5686
0.3468
30.7146
21.1857
-0.0489
0.1201
4.8107
✓
0.7014
0.6024
0.5924
0.3531
30.9112
21.3459
0.1187
0.1516
4.7334
✓
✓
0.6744
0.6807
0.5541
0.3570
31.1396
21.2536
0.1088
0.1229
4.8024
Table 4: Ablation study on the loss components of Eq. 11 in the SD1.5. A: diversity term, B: fidelity term.
Figure 6: Qualitative ablation of the fidelity term on SD3. Top: unoptimized baseline. With fidelity (middle), DetNO increases diversity while preserving image quality. Without fidelity (bottom), unconstrained volume maximization pushes noise vectors into out-of-distribution regions, producing artifacts such as cartoon-like renderings, color corruption, and degenerate outputs.
Figure 7: Batch-size scaling on SD1.5. (a) VendiScore gain over the DDIM baseline versus batch size B for CNO and DetNO. (b) DetNO’s VendiScore advantage over CNO, which widens monotonically with B . All other settings follow Tab. S2 .
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Step 0
Step 1
Step 2
Step 3
Step 4
Step 5
Step 6
Step 7
Step 8
Step 9
μmin
0.244
0.349
0.304
0.256
0.230
0.226
0.234
0.248
0.261
0.271
Effective rank
2.856
3.246
3.065
2.867
2.754
2.752
2.802
2.871
2.931
2.963
Rz
1.138∗∗∗
0.891
0.975
1.106∗∗∗
1.197∗∗∗
1.207∗∗∗
1.173∗∗∗
1.125∗∗∗
1.086∗∗
1.063∗
Appendix
Table S1 : Gradient dynamics during DetNO optimization on SD1.5 ( B=5 , 20 prompts × 3 seeds). Rz>1 at 8/10 steps ( ∗∗∗p<0.001 , ∗∗p<0.01 , ∗p<0.05 ).
Backbone
w
η
λ
Nopt
B
NFE
ϵ
SD1.5
16
0.01
0.05
3
5
50
10−5
SDXL
16
0.01
0.05
3
3
50
10−5
SD3
32
0.01
0.01
3
5
28
10−5
Appendix
Table S2 : Hyperparameter configuration per backbone.
Method
t=T (step 0)
t=0.5T (step 25)
t≈0 (step 49)
DDIM
0.548±0.138
0.133±0.078
0.120±0.073
CNO
0.444±0.200
0.115±0.092
0.101±0.083
DetNO
0.403±0.179
0.032±0.057
0.026±0.052
Appendix
Table S3 : Mean pairwise cosine similarity of Tweedie predictions at key timesteps. Averaged over 20 MS-COCO prompts ( B=5 , NFE =50 , SD1.5). Lower indicates more diverse predictions. DetNO maintains the lowest similarity throughout the denoising trajectory.
σ
Prec ↑
Recall ↑
Den ↑
Cov ↑
CLIP ↑
Pick ↑
IR ↑
MSS ↓
Vendi ↑
0.01
0.6698
0.6369
0.5200
0.3314
31.3793
21.5105
0.2444
0.1588
4.7121
0.5
0.6687
0.6389
0.5141
0.3289
31.3807
21.5129
0.2519
0.1579
4.7160
1.0
0.6554
0.6502
0.4945
0.3209
31.2677
21.4233
0.1796
0.1390
4.7669
dynamic median
0.6538
0.6561
0.5003
0.3214
31.2722
21.4133
0.1675
0.1361
4.7733
median (ours)
0.6744
0.6807
0.5541
0.3570
31.1396
21.2536
0.1088
0.1229
4.8024
Appendix
Table S4 : Sigma ablation on SD1.5 ( B =5, Nopt =3, λ =0.05, w =16).
Few-step distilled diffusion models generate high-quality images quickly, but often lose per-prompt diversity, producing near-identical samples across random seeds. Optimizing the initial noise at inference time offers an appealing way to recover this diversity, yet existing methods directly update the initial noise in an unconstrained Euclidean space, ignoring both the geometry of the Gaussian prior and the model's sensitivity to noise frequencies. They therefore introduce auxiliary quality-control objectives to maintain generation fidelity, adding compute and weighting hyperparameters while still requiring conservative updates to prevent degradation. In this work, we propose MoNO, a training-free method that performs Manifold-constrained Noise Optimization on a low-dimensional, quality-stabilizing noise manifold. MoNO sequentially optimizes each new initial noise so that its predicted visual feature complements previous generations, while Riemannian updates on an affine low-frequency sphere preserve prior likelihood and fix unstable high-frequency components by construction. This enables large geodesic steps, removes the need for auxiliary quality-control objectives, and converges in far fewer iterations than prior noise-optimization methods. Experiments with multiple distilled text-to-image diffusion models show that MoNO consistently improves per-prompt diversity while maintaining image quality.
Image diversity remains a fundamental challenge for text-to-image diffusion models. Low-diversity generation often leads to repetitive outputs, increasing sampling redundancy and hindering both creative exploration and downstream applications. A key factor is the tendency of diffusion models to collapse toward strong modes in the learned distribution. Existing attempts to improve diversity, such as steering-based guidance, often introduce distortions that degrade image quality. To address this issue, we propose Token-Prompt Embedding Space Optimization (TPSO), a training-free and model-agnostic module. TPSO introduces learnable parameters to explore underrepresented regions of the token embedding space, reducing the tendency to repeatedly sample from strong modes of the distribution. Meanwhile, a prompt-level semantic constraint regulates distribution shifts, preventing quality degradation while preserving semantic fidelity. Extensive experiments on MS-COCO across three representative diffusion backbones demonstrate that TPSO substantially improves diversity, boosting performance from 1.10 to 4.18, while maintaining image quality with only a modest inference-time overhead of 3.6% to 8.9%. Code is available at: https://github.com/Open-Debin/TPSO.
Debin Meng, Chen Jin, Zheng Gao +3
Queen Mary University of London · Centre for AI, AstraZeneca, UK · University of Bedfordshire +1
Diffusion models typically generate image batches from independent Gaussian initial noises. We argue that this independence assumption is only one choice within a broader class of valid joint noise designs. Instead, one can specify a coupling of the initial noises: each noise remains marginally standard Gaussian, so the pretrained diffusion model receives the same single-sample input distribution, while the dependence across samples is chosen by design. This reframes initial-noise control from selecting or optimizing individual seeds to designing the dependence structure of a multi-sample gallery. This view gives a general framework for initial-noise design, covering several existing methods as special cases and leading naturally to new coupled-noise constructions. Coupled noise can improve generation on its own without adding sampling cost, and it is flexible enough to serve as a structured initialization for optimization-based pipelines when additional computation is available. Empirically, repulsive Gaussian coupling improves gallery diversity on SD1.5, SDXL, and SD3 while largely preserving prompt alignment and image quality. It matches or outperforms recent test-time noise-optimization baselines on several diversity metrics at the same sampling cost as independent generation. Subspace couplings also support fixed-object background generation, producing diverse, natural backgrounds compared with specialized inpainting baselines, with a tunable trade-off in foreground fidelity.
Jing Jia, Liyue Shen, Guanyang Wang
Department of Computer Science Rutgers University Piscataway, NJ 08854 · Department of EECS University of Michigan Ann Arbor, MI 48109 · Department of Statistics Rutgers University Piscataway, NJ 08854