Beyond Pairwise Repulsion: Collective Diversity via Determinant Maximization in Diffusion Sampling

Kwanyoung Lee; SeungJu Cha; Yebin Ahn; Sungho Koh; Dong-Jin Kim

cs.CVOct 15, 2026

Beyond Pairwise Repulsion: Collective Diversity via Determinant Maximization in Diffusion Sampling

Authors: Kwanyoung Lee, SeungJu Cha, Yebin Ahn, Sungho Koh, Dong-Jin Kim

Organizations: Hanyang University, South Korea

Abstract

Strong guidance in text-to-image (T2I) diffusion models concentrates sampling toward high-density modes of the conditional distribution, collapsing independently sampled noise vectors onto nearly identical visual prototypes. This limits utility for tasks that demand distributional breadth, most critically synthetic data augmentation. Diversity, however, is a collective property of the whole batch: a set can preserve every pairwise distance yet still collapse onto a low-dimensional subspace. Existing sum-of-pairwise noise-optimization and repulsion objectives respond only weakly to this higher-order failure. We propose DetNO (Determinant Noise Optimization), which formulates diverse generation as RKHS volume maximization on the initial noise, maximizing the log-determinant of an RBF Gram matrix over one-step Tweedie predictions. This set-level functional equals the squared batch volume; its repulsion is strongest when the batch is most collapsed and self-attenuates as diversity is achieved. An anchor-based fidelity term keeps the optimized noises near plausible regions, and Lagrangian relaxation gives a two-term objective with a single hyperparameter $λ$ governing the diversity-quality trade-off, without training or sampler modification. Across three backbones (SD1.5, SDXL, SD3), DetNO achieves leading intra-batch diversity in VendiScore and Mean Self-Similarity, and a downstream few-shot classification experiment confirms that broader mode coverage brings statistically significant accuracy gains for synthetic data augmentation in the low-shot regime. Fixing the pipeline and varying only the batch objective, both RBF-based aggregators outperform InfoNCE, and the log-determinant attains the best downstream accuracy on both benchmarks. Its diversity advantage over InfoNCE widens as the batch grows. Code is available at https://github.com/mobled37/DetNO.

Figures & tables

Figure 1: Qualitative comparison on SD3. SD3 baseline sampling collapses onto a narrow visual prototype for underspecified prompts, reproducing nearly identical appearance and scene layout across seeds, a direct consequence of CFG concentrating the conditional distribution onto its high-density peak. DetNO recovers the suppressed degrees of freedom in initial noise, producing semantically diverse outputs that vary in viewpoint, identity, clothes, and scene composition while remaining faithful to the prompt.

Figure 2: Conceptual illustration of DetNO. (a) Strong CFG guidance concentrates sampling toward the high-density peak, leaving the degrees of freedom in the initial noise unexploited. DetNO recovers this freedom by optimizing the initial noise batch to reach low-density regions that standard sampling cannot access. (b) The diversity term (green) maximizes the RKHS volume collectively spanned by all Tweedie predictions, while the fidelity term (yellow) regularizes each sample toward its unoptimized reference, expanding distributional coverage while preserving prompt fidelity.

Figure 3: Toy example on a 2D ring mixture with 8 inner modes (80%) and 8 outer modes (20%). (a) Ground truth distribution. (b) Classifier-free guidance collapses onto the majority inner ring, without covering outer modes. (c) CNO recovers the outer modes but severely distorts the inner ring, producing only 4 inner modes whose sample distributions no longer align with the ground truth geometry. (d) DetNO achieves full coverage of all 16 modes across both rings, faithfully preserving the original mode structure.

		Reference-Based				Reference-Free			Diversity
Model	Method	Prec $\uparrow$	Rec $\uparrow$	Den $\uparrow$	Cov $\uparrow$	CLIP $\uparrow$	Pick $\uparrow$	IR $\uparrow$	MSS $\downarrow$	Vendi $\uparrow$
SD1.5	DDIM	0.7005	0.6061	0.6112	0.3631	30.9096	21.3413	0.1086	0.1509	4.7352
	PG	0.7008	0.6044	0.6120	0.3634	30.9103	21.3424	0.1085	0.1508	4.7357
	SPELL	0.6722	0.5645	0.5639	0.3439	30.7538	21.2556	0.0264	0.1409	4.7636
	DiversityPrompt	0.6679	0.6442	0.5345	0.3370	30.9380	21.1956	0.1194	0.1455	4.7509
	CNO	0.7247	0.6318	0.6647	0.3933	30.7925	21.2984	0.0702	0.1383	4.7679

Table 1: Quantitative comparison across different backbone models. Bold and underline denote the best and second-best results, respectively.

Figure 4: Qualitative comparison on SD1.5. For each prompt, we show one representative sample from DDIM, CNO, and DetNO (ours). DDIM and CNO produce visually near-identical outputs across seeds, differing only in minor details such as slight pose or hair difference. DetNO generates substantially different images, varying in viewpoint, object appearance, scene layout, and human identity, while remaining faithful to the prompt.

	Oxford-IIIT Pets (SD3)				CUB-200-2011 (SD1.5)
Method	1-shot	2-shot	4-shot	8-shot	1-shot	2-shot	4-shot	8-shot
Real only	17.6 $\pm$ 2.0	41.2 $\pm$ 2.9	57.7 $\pm$ 2.7	65.4 $\pm$ 0.7	8.2 $\pm$ 0.5	15.1 $\pm$ 0.6	27.9 $\pm$ 0.9	47.0 $\pm$ 1.3
+ DDIM	52.2 $\pm$ 1.8	58.2 $\pm$ 0.9	59.1 $\pm$ 1.3	71.8 $\pm$ 0.7	24.6 $\pm$ 0.9	31.1 $\pm$ 0.9	38.5 $\pm$ 0.9	50.9 $\pm$ 0.8
+ CNO	51.7 $\pm$ 1.7	57.8 $\pm$ 0.6	59.4 $\pm$ 1.6	71.4 $\pm$ 0.7	24.5 $\pm$ 0.7	30.6 $\pm$ 0.8	38.4 $\pm$ 0.5	50.7 $\pm$ 1.1
+ DetNO	54.2 $\pm$ 1.1 ‡	59.3 $\pm$ 1.3 ‡	61.2 $\pm$ 1.7 †	71.8 $\pm$ 0.9	26.3 $\pm$ 0.5 †‡	32.1 $\pm$ 0.7 ‡	39.8 $\pm$ 0.6 †‡	51.8 $\pm$ 0.9

Table 2: Downstream few-shot classification accuracy (%). Mean

\pm

std over 10 splits. Synthetic images (16/class) generated with SD3 (Oxford-IIIT Pets) and SD1.5 (CUB-200-2011).

\dagger

and

\ddagger

denote statistically significant improvement over DDIM and CNO respectively (Welch’s

t

-test,

p<0.01

Method	SD1.5 (intra-batch)		1-shot accuracy (%)
Method	Pick $\uparrow$	Vendi $\uparrow$	Pets (SD3) $\uparrow$	CUB (SD1.5) $\uparrow$
DDIM (Baseline)	21.34	4.7352	52.2 $\pm$ 1.8	24.6 $\pm$ 0.9
CNO (InfoNCE)	21.30	4.7679	51.7 $\pm$ 1.7	24.5 $\pm$ 0.7
$\sum_{i<j}S_{ij}$ (pairwise-RBF)	21.23	4.8006	53.7 $\pm$ 1.1	25.8 $\pm$ 0.6
$-\log\det$ (DetNO)	21.25	4.8024	54.2 $\pm$ 1.1	26.3 $\pm$ 0.5

Table 3: Pairwise-RBF ablation with downstream evaluation. SD1.5,

B\!=\!5

\lambda\!=\!0.05

;

S

\sigma

, fidelity term, and optimizer are held identical across rows, so only the batch aggregator differs: InfoNCE (CNO), the sum-of-pairwise RBF

\sum_{i<j}S_{ij}

, and the log-determinant (DetNO). Downstream columns follow the protocol of Tab. 2 . Bold and underline denote the best and second-best results.

Figure 5: Quality–diversity Pareto curves on SD3 for different pooling windows

w\in\{8,16,32,64\}

. Each point on a curve corresponds to a different fidelity weight

\lambda\in\{0.01,0.05,0.1\}

, evaluated on 200 randomly sampled MS-COCO prompts with 5 images per prompt. All configurations improve over the unoptimized baseline (gray star) on this evaluation. Smaller

w

yields a wider diversity range at the cost of a steeper quality trade-off, while larger

w

preserves quality more effectively with moderate diversity gains.

A	B	Prec $\uparrow$	Recall $\uparrow$	Den $\uparrow$	Cov $\uparrow$	CLIP $\uparrow$	Pick $\uparrow$	IR $\uparrow$	MSS $\downarrow$	Vendi $\uparrow$
		0.7005	0.6061	0.6112	0.3631	30.9096	21.3413	0.1086	0.1509	4.7352
✓		0.6816	0.6191	0.5686	0.3468	30.7146	21.1857	-0.0489	0.1201	4.8107
	✓	0.7014	0.6024	0.5924	0.3531	30.9112	21.3459	0.1187	0.1516	4.7334
✓	✓	0.6744	0.6807	0.5541	0.3570	31.1396	21.2536	0.1088	0.1229	4.8024

Table 4: Ablation study on the loss components of Eq. 11 in the SD1.5. A: diversity term, B: fidelity term.

Figure 6: Qualitative ablation of the fidelity term on SD3. Top: unoptimized baseline. With fidelity (middle), DetNO increases diversity while preserving image quality. Without fidelity (bottom), unconstrained volume maximization pushes noise vectors into out-of-distribution regions, producing artifacts such as cartoon-like renderings, color corruption, and degenerate outputs.

Figure 7: Batch-size scaling on SD1.5. (a) VendiScore gain over the DDIM baseline versus batch size

B

for CNO and DetNO. (b) DetNO’s VendiScore advantage over CNO, which widens monotonically with

B

. All other settings follow Tab. S2 .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Step 0	Step 1	Step 2	Step 3	Step 4	Step 5	Step 6	Step 7	Step 8	Step 9
$\mu_{\min}$	$0.244$	$0.349$	$0.304$	$0.256$	$0.230$	$0.226$	$0.234$	$0.248$	$0.261$	$0.271$
Effective rank	$2.856$	$3.246$	$3.065$	$2.867$	$2.754$	$2.752$	$2.802$	$2.871$	$2.931$	$2.963$
$R_{z}$	$1.138^{***}$	$0.891$	$0.975$	$1.106^{***}$	$1.197^{***}$	$1.207^{***}$	$1.173^{***}$	$1.125^{***}$	$1.086^{**}$	$1.063^{*}$

Appendix

Table S1 : Gradient dynamics during DetNO optimization on SD1.5 (

B{=}5

, 20 prompts

\times

3 seeds).

R_{z}>1

at 8/10 steps (

{}^{***}p{<}0.001

{}^{**}p{<}0.01

{}^{*}p{<}0.05

Backbone	$w$	$\eta$	$\lambda$	$N_{\mathrm{opt}}$	$B$	NFE	$\epsilon$
SD1.5	16	0.01	0.05	3	5	50	$10^{-5}$
SDXL	16	0.01	0.05	3	3	50	$10^{-5}$
SD3	32	0.01	0.01	3	5	28	$10^{-5}$

Appendix

Table S2 : Hyperparameter configuration per backbone.

Method	$t{=}T$ (step 0)	$t{=}0.5T$ (step 25)	$t{\approx}0$ (step 49)
DDIM	$0.548\pm 0.138$	$0.133\pm 0.078$	$0.120\pm 0.073$
CNO	$0.444\pm 0.200$	$0.115\pm 0.092$	$0.101\pm 0.083$
DetNO	$\mathbf{0.403\pm 0.179}$	$\mathbf{0.032\pm 0.057}$	$\mathbf{0.026\pm 0.052}$

Appendix

Table S3 : Mean pairwise cosine similarity of Tweedie predictions at key timesteps. Averaged over 20 MS-COCO prompts (

B{=}5

, NFE

{=}50

, SD1.5). Lower indicates more diverse predictions. DetNO maintains the lowest similarity throughout the denoising trajectory.

$\sigma$	Prec $\uparrow$	Recall $\uparrow$	Den $\uparrow$	Cov $\uparrow$	CLIP $\uparrow$	Pick $\uparrow$	IR $\uparrow$	MSS $\downarrow$	Vendi $\uparrow$
0.01	0.6698	0.6369	0.5200	0.3314	31.3793	21.5105	0.2444	0.1588	4.7121
0.5	0.6687	0.6389	0.5141	0.3289	31.3807	21.5129	0.2519	0.1579	4.7160
1.0	0.6554	0.6502	0.4945	0.3209	31.2677	21.4233	0.1796	0.1390	4.7669
dynamic median	0.6538	0.6561	0.5003	0.3214	31.2722	21.4133	0.1675	0.1361	4.7733
median (ours)	0.6744	0.6807	0.5541	0.3570	31.1396	21.2536	0.1088	0.1229	4.8024

Appendix

Table S4 : Sigma ablation on SD1.5 (

B

=5,

N_{\mathrm{opt}}

=3,

\lambda

=0.05,

w

=16).

Method	Time/batch (s) $\downarrow$	GPU Mem. (GB) $\downarrow$	Extra Fwd. Passes
SD3 (Baseline)	16.3	28.3	0
CNO	17.7	28.3	$N_{\text{opt}}+1=4$
DetNO (Ours)	17.8	28.3	$N_{\text{opt}}+1=4$

Appendix

Table S5 : Computational cost comparison (SD3, NFE=28,

B=5

, 1024

\times

1024).

Method	Prec $\uparrow$	Recall $\uparrow$	Den $\uparrow$	Cov $\uparrow$	CLIP $\uparrow$	Pick $\uparrow$	IR $\uparrow$	MSS $\downarrow$	Vendi $\uparrow$
Baseline	0.6261	0.5223	0.4527	0.2480	31.8805	22.5719	1.0503	0.2944	4.2517
Full Gradient	0.6385	0.5309	0.4520	0.2499	31.8775	22.5541	1.0468	0.2824	4.2964
Stop Gradient	0.6303	0.5184	0.4437	0.2488	31.8400	22.5597	1.0465	0.2858	4.2892

Appendix

Table S6 : Ablation: Full Gradient vs. Stop Gradient (SD3, MS-COCO, seed=42, single run)

$s$	Prec $\uparrow$	Recall $\uparrow$	Density $\uparrow$	Coverage $\uparrow$	CLIP $\uparrow$	Pick $\uparrow$	IR $\uparrow$	MSS $\downarrow$	Vendi $\uparrow$
0.5	0.9963	0.0002	3.6769	0.0149	18.3868	17.7436	-2.2765	0.5097	2.9807
0.8	0.7752	0.0194	0.8544	0.0688	26.2606	19.2687	-1.8094	0.2623	4.3496
1.0 (DDIM)	0.7005	0.6061	0.6112	0.3631	30.9096	21.3413	0.1086	0.1509	4.7352
1.2	0.6151	0.0385	0.2208	0.0099	22.1671	17.3286	-2.0955	0.2787	4.3096
1.5	0.9984	0.0000	0.7066	0.0009	18.9369	16.0270	-2.2767	0.5503	3.1409
2.0	1.0000	0.0000	0.7958	0.0009	19.3291	15.8109	-2.2789	0.6051	2.8754

Appendix

Table S7 : Ablation on noise scale

s

(SD1.5).

\mathbf{z}_{T}\sim\mathcal{N}(\mathbf{0},s^{2}\mathbf{I})

Figure S1 : Qualitative results on SDXL.

Diversity-Aware Sampling

See all 20 papers

Manifold-Constrained Noise Optimization for Diverse Diffusion Sampling
Jul 27, 2026Qitan Shi, Cheng Jin, Ziyuan Liu +1Diffusion Model Sampling Image Generation
Tsinghua University
TPSO: Training-Free Diverse Image Generation via Semantic Prompt Embedding Optimization
Nov 25, 2025Debin Meng, Chen Jin, Zheng Gao +3T2I Generation T2I Diffusion Models
Queen Mary University of London · Centre for AI, AstraZeneca, UK · University of Bedfordshire +1
Couple to Control: Joint Initial Noise Design in Diffusion Models
May 11, 2026Jing Jia, Liyue Shen, Guanyang WangDiffusion Sampling Image Generation
Department of Computer Science Rutgers University Piscataway, NJ 08854 · Department of EECS University of Michigan Ann Arbor, MI 48109 · Department of Statistics Rutgers University Piscataway, NJ 08854

Jul 27, 2026J/K move · Enter open · S save

Manifold-Constrained Noise Optimization for Diverse Diffusion Sampling

Qitan Shi, Cheng Jin, Ziyuan Liu, Yuantao Gu

Tsinghua University

Diffusion Model Sampling Image Generation Few-Step Diffusion Sampling T2I Diffusion Models Diffusion Models

Open paper AskPDF arXiv

Few-step distilled diffusion models generate high-quality images quickly, but often lose per-prompt diversity, producing near-identical samples across random seeds. Optimizing the initial noise at inference time offers an appealing way to recover this diversity, yet existing methods directly update the initial noise in an unconstrained Euclidean space, ignoring both the geometry of the Gaussian prior and the model's sensitivity to noise frequencies. They therefore introduce auxiliary quality-control objectives to maintain generation fidelity, adding compute and weighting hyperparameters while still requiring conservative updates to prevent degradation. In this work, we propose MoNO, a training-free method that performs Manifold-constrained Noise Optimization on a low-dimensional, quality-stabilizing noise manifold. MoNO sequentially optimizes each new initial noise so that its predicted visual feature complements previous generations, while Riemannian updates on an affine low-frequency sphere preserve prior likelihood and fix unstable high-frequency components by construction. This enables large geodesic steps, removes the need for auxiliary quality-control objectives, and converges in far fewer iterations than prior noise-optimization methods. Experiments with multiple distilled text-to-image diffusion models show that MoNO consistently improves per-prompt diversity while maintaining image quality.

Abstract

Diversity-Aware Sampling

Explore similar work

Manifold-Constrained Noise Optimization for Diverse Diffusion Sampling

TPSO: Training-Free Diverse Image Generation via Semantic Prompt Embedding Optimization

Couple to Control: Joint Initial Noise Design in Diffusion Models