On-the-fly Repulsion in the Contextual Space for Rich Diversity in Diffusion Transformers

Omer Dahary; Benaya Koren; Daniel Garibi; Daniel Cohen-Or

cs.CVMar 30, 2026

On-the-fly Repulsion in the Contextual Space for Rich Diversity in Diffusion Transformers

Authors: Omer Dahary, Benaya Koren, Daniel Garibi, Daniel Cohen-Or

Organizations: Tel Aviv University, Israel · Snap Research, Israel

Abstract

Modern Text-to-Image (T2I) diffusion models have achieved remarkable semantic alignment, yet they often suffer from a significant lack of variety, converging on a narrow set of visual solutions for any given prompt. This typicality bias presents a challenge for creative applications that require a wide range of generative outcomes. We identify a fundamental trade-off in current approaches to diversity: modifying model inputs requires costly optimization to incorporate feedback from the generative path. In contrast, acting on spatially-committed intermediate latents tends to disrupt the forming visual structure, leading to artifacts. In this work, we propose to apply repulsion in the Contextual Space as a novel framework for achieving rich diversity in Diffusion Transformers. By intervening in the multimodal attention channels, we apply on-the-fly repulsion during the transformer's forward pass, injecting the intervention between blocks where text conditioning is enriched with emergent image structure. This allows for redirecting the guidance trajectory after it is structurally informed but before the composition is fixed. Our results demonstrate that repulsion in the Contextual Space produces significantly richer diversity without sacrificing visual fidelity or semantic adherence. Furthermore, our method is uniquely efficient, imposing a small computational overhead while remaining effective even in modern "Turbo" and distilled models where traditional trajectory-based interventions typically fail.

Figures & tables

Figure 1. Example results of our Contextual Space repulsion framework using Flux-dev. The base model (top) typically converges on a narrow set of visual solutions. By applying semantic intervention within the internal multi-modal attention channels, our approach (bottom) produces a diverse set of images with minimal computational overhead.

Panel of Figure 2 — Figure 2. Conceptual comparison of diversity strategies in dual-stream DiT architectures. Here $p^{(i)}$ denotes the prompt embedding for sample $i$ , $z_{t}^{(i)}$ denotes the latent at timestep $t$ for sample $i$ , and the red double-arrow icon indicates the point of diversity manipulation. (a) Upstream : Interventions on noise or prompt embeddings lack structural feedback from the emerging image. (b) Downstream : Repulsion in image latents acts on a fixed visual mode and can push samples off the data manifold, causing artifacts. (c) Ours : By applying on-the-fly repulsion within the Contextual Space (text-attention channels), we steer the model’s generative intent. This allows for a semantically driven intervention synchronized with the emergent visual structure.

Panel of Figure 3 — Figure 3. Comparison of interpolation and extrapolation between the internal representations of two images. Intermediate frames are generated by denoising the source image while linearly blending its internal features with those of the target; extrapolation extends this vector beyond the endpoints. While Latent Space interpolation leads to structural blurring and artifacts due to spatial misalignment, the Contextual Space maintains high visual fidelity. This demonstrates that the Contextual Space enables smooth semantic transitions by decoupling generative intent from fixed spatial structures.

Panel of Figure 4 — Figure 4. Qualitative results. For each prompt, we compare the base model results (top) to our results (bottom).

Panel of Figure 5 — Figure 5. Integration with image editing models. We demonstrate that our method can be successfully integrated into Flux-Kontext to generate high-quality diverse results.

Figure 6. Quantitative evaluation. Pareto frontiers comparing our method against baseline methods using Flux-dev. We evaluate the trade-off between semantic diversity (Vendi Score) and three performance axes: (Left) Human Preference [ImageReward $\uparrow$ ], (Middle) Prompt Alignment [VQAScore $\uparrow$ ], and (Right) Distributional Fidelity [KID $\downarrow$ ]. Our method (red) achieves a superior frontier across all metrics.

Method		SD3.5-Large	SD3.5-Turbo	Flux-dev
Base Model		13.83s	4.18s	10.34s
Ours (Contextual)		18.12s	5.52s	12.80s
SGI	8 Candidates	66.79s	13.15s	47.47s
	16 Candidates	76.79s	23.73s	56.32s
	32 Candidates	101.44s	46.15s	75.39s
	64 Candidates	145.14s	91.30s	113.99s

Table 1. Runtime comparison for generating a group of four images. Our method provides a significant speedup over optimization-based diversity methods like SGI while maintaining a low overhead relative to the base model.

Figure 7. Overall user preference comparison. Distribution of user choices comparing our method with five competing approaches. Bars indicate the percentage of cases in which users preferred our results (green), preferred competing methods (red), or rated both equally (gray).

Panel of Figure 8 — Figure 8. Ablation of the repulsion scale $\eta$ . We visualize the impact of the repulsion scale on our results. At $\eta=0$ (top row), the base model exhibits low diversity, producing similar architectural styles and environments across multiple seeds. As $\eta$ increases, our Contextual Space repulsion introduces progressively larger variations, while maintaining high image quality and prompt alignment.

Figure 9. Ablation of Repulsion Space. Pareto frontiers comparing repulsion applied to text attention tokens (Contextual Space, $\hat{f}_{T}$ ) versus image attention tokens ( $\hat{f}_{I}$ ) within the Flux-dev architecture. We evaluate the trade-off between semantic diversity (Vendi Score) and three performance axes: (Left) Human Preference [ImageReward $\uparrow$ ], (Middle) Prompt Alignment [VQAScore $\uparrow$ ], and (Right) Distributional Fidelity [KID $\downarrow$ ]. Our method (red) achieves a superior frontier across all metrics.

Panel of Figure 10 — Figure 10. Qualitative Ablation of Repulsion Space. For each prompt, we compare repulsion applied in the image attention space (Image) versus our Contextual Space (Contextual). While image-space repulsion is limited by spatial rigidity, our method achieves more varied compositions.

Panel of Figure 11 — Figure 11. Qualitative results. For each prompt, we compare the base model results (top) to our results (bottom). All batches were generated using the same random seed initialization. Additional results are provided in Appendix B .

Panel of Figure 12 — Figure 12. Qualitative comparison of our Contextual Repulsion approach against baseline methods. Each quadrant displays four generated samples per method for a given prompt.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 13. Quantitative evaluation on SD3.5-Large.

Figure 14. Quantitative evaluation on SD3.5-Turbo.

Panel of Figure 15 — Figure 15. Qualitative results on SD3.5-Large.

Panel of Figure 16 — Figure 16. Qualitative results on SD3.5-Turbo.

Panel of Figure 17 — Figure 17. Additional qualitative results on Flux-dev. All batches were generated using the same random seed initialization.

Panel of Figure 18 — Figure 18. Additional qualitative results on Flux-dev. All batches were generated using the same random seed initialization.

Panel of Figure 19 — Figure 19. Additional qualitative results on Flux-dev. All batches were generated using the same random seed initialization.

Method		Vendi ( $\uparrow$ )	IR ( $\uparrow$ )	VQA ( $\uparrow$ )	KID $\times 10^{-4}$ ( $\downarrow$ )
Base Model		1.780	1.075	0.883
Ours	${\eta=5\cdot 10^{6}}$	1.810	1.102	0.884	0.066
	${\eta=1\cdot 10^{7}}$	1.831	1.092	0.883	0.103
	${\eta=1\cdot 10^{8}}$	1.869	1.075	0.883	0.157
	${\eta=5\cdot 10^{8}}$	1.898	1.070	0.880	0.172
CADS	$s=10^{-20}$	1.908	0.377	0.719	0.558

Appendix

Table 2. Detailed metrics for the Flux-dev Pareto frontiers in Figure 6 .

Method		Vendi ( $\uparrow$ )	IR ( $\uparrow$ )	VQA ( $\uparrow$ )	KID $\times 10^{-4}$ ( $\downarrow$ )
Base Model		1.819	1.051	0.905
Ours	${\eta=5\cdot 10^{2}}$	1.851	1.018	0.904	0.619
	${\eta=5\cdot 10^{4}}$	1.878	1.012	0.904	0.627
	${\eta=5\cdot 10^{5}}$	1.941	0.988	0.900	0.625
	${\eta=5\cdot 10^{6}}$	1.980	0.940	0.890	0.445
CADS	$s=10^{-12}$	2.004	0.131	0.717	0.941

Appendix

Table 3. Detailed metrics for the SD3.5-Large Pareto frontiers in Figure 13 .

Method		Vendi ( $\uparrow$ )	IR ( $\uparrow$ )	VQA ( $\uparrow$ )	KID $\times 10^{-4}$ ( $\downarrow$ )
Base Model		1.724	0.978	0.891
Ours	${\eta=1\cdot 10^{5}}$	1.819	0.914	0.887	1.796
	${\eta=5\cdot 10^{5}}$	1.879	0.899	0.884	1.786
	${\eta=1\cdot 10^{6}}$	1.914	0.864	0.876	1.897
	${\eta=1\cdot 10^{7}}$	2.079	0.562	0.822	1.914
CADS	$s=0.1$	1.808	0.551	0.772	0.158

Appendix

Table 4. Detailed metrics for the SD3.5-Turbo Pareto frontiers in Figure 14 .

Metric	Choice	Base Model	CADS	SGI	PG	SPARKE	Average
Diversity	Ours	71.6	52.2	56.7	80.0	34.4	61.1
	Comp.	12.9	30.0	11.1	14.4	53.1	22.0
	Tie	15.5	17.8	32.2	5.6	12.5	16.9
Quality	Ours	49.1	67.8	15.6	82.2	85.9	58.0
	Comp.	6.9	11.1	31.1	12.2	3.1	13.1
	Tie	44.0	21.1	53.3	5.6	10.9	28.9

Appendix

Table 5. User study results comparing our method against five competing approaches across four evaluation metrics. Values show the percentage of times users preferred our method (Ours), the competitor (Comp.), or rated both equally (Tie). Results are aggregated from 450 pairwise comparisons per metric.

Batch size	Vendi	Vendi (avg. pair)	ImageReward
4	1.819	1.393	0.914
8	2.295	1.401	0.923
16	2.768	1.404	0.928

Appendix

Table 6. Scalability across batch sizes. Quantitative results on SD3.5-Turbo for varying batch sizes. We report the average Vendi score per pair to normalize for batch size constraints.

Model	Timestep interval	Vendi	ImageReward
SD3.5-Turbo	[0,1/4]	1.764	0.829
	[1/4,2/4]	1.776	0.811
	[2/4,3/4]	1.809	0.745
	[3/4,1]	1.988	0.660
	[0,1]	2.064	0.501
SD3.5-Large	[0,1/7]	1.849	0.942

Appendix

Table 7. Effect of the timestep interval on diversity and human preference. We evaluate different intervention windows during the diffusion trajectory for SD3.5-Large and SD3.5-Turbo.

	SD3.5-Turbo		SD3.5-Large
Block group	Vendi	ImageReward	Vendi	ImageReward
First third	1.878	0.774	1.887	0.895
Middle third	1.947	0.844	1.947	0.902
Last third	1.765	0.913	1.835	0.985
All blocks	1.764	0.829	1.960	0.852

Appendix

Table 8. Performance across different transformer block groups. Results are reported for interventions applied to the first, middle, or last third of the blocks for SD3.5-Large and SD3.5-Turbo.

T2I Generation

Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.

See all 299 papers

Breaking the Lock-in: Diversifying Text-to-Image Generation via Representation Modulation
Jun 5, 2026Dahee Kwon, Haeun Lee, Jaesik ChoiDiffusion Transformer T2I Generation
Korea Advanced Institute of Science and Technology (KAIST) · INEEJI
TPSO: Training-Free Diverse Image Generation via Semantic Prompt Embedding Optimization
Nov 25, 2025Debin Meng, Chen Jin, Zheng Gao +3T2I Generation T2I Diffusion Models
Queen Mary University of London · Centre for AI, AstraZeneca, UK · University of Bedfordshire +1
Learning to Read the Contextual Tokens in Diffusion Transformers
Oct 5, 2026Omer Dahary, Etai Sella, Hadar Averbuch-Elor +2Diffusion Model Alignment Diffusion Transformer
Tel Aviv University · Cornell University

Jun 5, 2026J/K move · Enter open · S save

Breaking the Lock-in: Diversifying Text-to-Image Generation via Representation Modulation

Dahee Kwon, Haeun Lee, Jaesik Choi

Korea Advanced Institute of Science and Technology (KAIST) · INEEJI

Diffusion Transformer T2I Generation

Open paper AskPDF arXiv

Recent text-to-image models built on large-scale Transformer backbones and flow-based objectives deliver strong text-image alignment and high visual quality, yet often produce overly similar samples under a fixed prompt. Existing diversity-enhancement methods alleviate this issue, but typically require expensive sampling or auxiliary optimization, incurring non-trivial overhead. To investigate the root cause of this homogeneity, we examine intermediate Transformer features and observe that the zero-frequency spatial average (DC) component rapidly converges across seeds early in generation, causing early trajectory lock-in that limits downstream variation. Building on this observation, we propose DC Attenuation for diVersity Enhancement (DAVE), a training-free representation-level intervention that selectively attenuates this component in the early regime. DAVE preserves the sampling pipeline with negligible overhead, improving prompt-consistent diversity while maintaining competitive image quality.

Abstract

T2I Generation

Explore similar work

Breaking the Lock-in: Diversifying Text-to-Image Generation via Representation Modulation

TPSO: Training-Free Diverse Image Generation via Semantic Prompt Embedding Optimization

Learning to Read the Contextual Tokens in Diffusion Transformers