TokenRepel: Generating Diverse Image Sets Without Compromising Quality

Qingtao Yu; Zhen Wang; Ning Lai; Boyang Hu; Hongdong Li; Dylan Campbell

cs.CVOct 29, 2026

TokenRepel: Generating Diverse Image Sets Without Compromising Quality

Authors: Qingtao Yu, Zhen Wang, Ning Lai, Boyang Hu, Hongdong Li, Dylan Campbell

Organizations: The Australian National University · Homenet Research · CarryAI

Abstract

We present an inference-time approach that enhances the visual diversity of image sets generated by text-to-image models, without compromising the average image quality. Our method leverages the intrinsic geometry of hidden representations in pretrained models to diversify images generated from the same prompt. First, we introduce a token repulsion mechanism that increases the angle between feature tokens across samples. It takes a negative Riemannian gradient step on a von Mises-Fisher mixture distribution of tokens on the unit hypersphere. Second, to preserve the quality of each output image, we propose a token density preservation mechanism that projects the repulsion-induced prediction change onto a half-space aligned with the model's score direction. This encourages the updates to remain within the high-density regions learned by the generative model. We further extend this mechanism to the output of each intermediate layer, enabling finer-grained iterative guidance. We evaluate our method on three types of pretrained text-to-image models. Experiments show substantial gains in visual diversity under identical prompts, while maintaining a comparable level of image fidelity.

Figures & tables

Panel of Figure 2 — Figure 2: Illustration of our TokenRepel algorithm. 2(a) Left: The tokens of Image 1 are repelled by taking a negative gradient step induced by the distribution formed from the tokens of the other images. 2(a) Right: Update formulation for $\mathbf{h}_{ij}$ , where $\bar{\mu}(\hat{\mathbf{h}}_{ij})$ denotes the weighted mean of the other image tokens with respect to their cosine similarity to $\mathbf{h}_{ij}$ . 2(b) Left: Contours representing the learned data distribution of the generative model, a green dashed arrow representing the original score, and a black solid arrow representing the score after applying token repulsion. 2(b) Right: The refinement process applied to $\nabla\log p^{r}_{t}(x_{t})$ to project it into the higher-density half-space (gray solid arrow). 2(c) Locations of these two operations inside a transformer layer.

Panel of Figure 3 — Figure 3: Diversity–quality trade-offs on GenEval across three models. Each curve varies one method-specific hyperparameter; the yellow star marks the model baseline.

FLUX.1-dev:
Method	Latency over baseline $\downarrow$	Diversity			Image Fidelity
Method	Latency over baseline $\downarrow$	DreamSim $\uparrow$	Vendi $\uparrow$	LPIPS $\uparrow$	HPSv3 $\uparrow$	CLIP $\uparrow$	ImR $\uparrow$	PickS $\uparrow$	GenEval $\uparrow$
Baseline	1.00 $\times$	0.2561	1.8747	0.5821	7.5266	0.3613	0.9579	23.3245	0.6653
CADS	1.00 $\times$	0.2599	1.8694	0.5869	7.5498	0.3608	0.9690	23.3233	0.6551
PG	1.00 $\times$	0.2593	1.8786	0.6019	7.4796	0.3606	0.9798	23.2855	0.6598
NegToMe	1.01 $\times$	0.3151	1.9829	0.6572	7.3467	0.3616	1.0055	23.1479	0.6670

Table 1: Single-setting comparison on GenEval at HPSv3 thresholds of 7.4, 8.8, and 5.5 for FLUX.1-dev, FLUX.2-Klein-9B, and SDXL, respectively.

Panel of Figure 4 — Figure 4: FLUX.1-dev qualitative results on GenEval under the same initial noise.

Panel of Figure 5 — Figure 5: FLUX.2-Klein-9B qualitative results on GenEval under the same initial noise.

Panel of Figure 6 — Figure 6: SDXL qualitative results on GenEval under the same initial noise.

Panel of Figure 7 — Figure 7: Hyperparameter sweeps for TokenRepel without token density preservation on GenEval using FLUX.2-Klein-9B. Left: $\delta\in\{0^{\circ},10^{\circ},20^{\circ},30^{\circ},40^{\circ}\}$ with $\bar{\theta}=60^{\circ}$ . Middle: $\bar{\theta}\in\{0^{\circ},20^{\circ},40^{\circ},60^{\circ},80^{\circ}\}$ with $\delta=30^{\circ}$ . Right: concentration $\kappa\in\{1,10,100,10^{3},\infty\}$ , where $\kappa=\infty$ corresponds to argmax, with $\delta=30^{\circ}$ and $\bar{\theta}=60^{\circ}$ .

Panel of Figure 8 — Figure 8: GenEval diversity–quality trade-offs for FLUX.1-dev, FLUX.2-Klein-9B, and SDXL. Each curve sweeps the repulsion angle $\delta$ ; w/o P uses no density preservation, B applies it to the final block output, and L applies it layerwise within the denoising network. The yellow star marks the model baseline.

Method	DreamSim $\uparrow$	Vendi $\uparrow$	LPIPS $\uparrow$	HPSv3 $\uparrow$	CLIP $\uparrow$	ImageReward $\uparrow$	PickScore $\uparrow$
Baseline	0.098	1.425	0.320	9.384	0.364	0.582	21.777
Ours	0.209	1.699	0.502	9.301	0.357	0.509	21.591

Table 2: Image-editing results on KontextBench using FLUX Kontext.

Panel of Figure 9 — Figure 9: Effect of batch size on DreamSim (left), Vendi Score (middle), and HPSv3 (right) for the baseline and TokenRepel with FLUX.2-Klein-9B on GenEval.

Panel of Figure 10 — Figure 10: Effect of the number of repulsion steps (from 3 to 24) on DreamSim (left) and HPSv3 (middle), and the fraction of tokens eligible for repulsion at each timestep with $\bar{\theta}=45^{\circ}$ (right), measured with FLUX.1-dev on GenEval.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Parameter	Value	DreamSim $\uparrow$	Vendi $\uparrow$	LPIPS $\uparrow$	HPSv3 $\uparrow$	CLIP $\uparrow$	ImageReward $\uparrow$	PickScore $\uparrow$
Baseline	–	–	0.2561	1.8747	0.5821	7.5266	0.3613	0.9579	23.3245
Ours	$\delta$	5	0.2782	1.9204	0.6066	7.5045	0.3613	0.9562	23.3084
		10	0.3002	1.9501	0.6271	7.4389	0.3607	0.9477	23.2911
		20	0.3314	2.0090	0.6519	7.4321	0.3619	0.9603	23.2617
		30	0.3550	2.0420	0.6635	7.2391	0.3606	0.9419	23.1613
NegToMe	$\alpha$	0.5	0.3151	1.9829	0.6572	7.3467	0.3616	1.0055	23.1479

Appendix

Table 3: All GenEval sweep results of Figure 3 for FLUX.1-dev. Bold settings are used in Table 1 .

Method	Parameter	Value	DreamSim $\uparrow$	Vendi $\uparrow$	LPIPS $\uparrow$	HPSv3 $\uparrow$	CLIP $\uparrow$	ImageReward $\uparrow$	PickScore $\uparrow$
Baseline	–	–	0.1964	1.6791	0.5495	9.0595	0.3876	1.3912	23.6764
Ours	$\delta$	20	0.2534	1.8085	0.6402	9.0646	0.3871	1.3800	23.6484
		40	0.2643	1.8280	0.6599	8.9825	0.3860	1.3430	23.6246
		60	0.2799	1.8545	0.6833	8.7870	0.3853	1.3175	23.5176
		80	0.2806	1.8506	0.6876	8.7634	0.3854	1.3077	23.5003
NegToMe	$\alpha$	0.5	0.2280	1.7577	0.6224	8.9915	0.3863	1.3694	23.6605

Appendix

Table 4: All GenEval sweep results of Figure 3 for FLUX.2-Klein-9B. Bold settings are used in Table 1 .

Method	Parameter	Value	DreamSim $\uparrow$	Vendi $\uparrow$	LPIPS $\uparrow$	HPSv3 $\uparrow$	CLIP $\uparrow$	ImageReward $\uparrow$	PickScore $\uparrow$
Baseline	–	–	0.3212	1.9565	0.7243	5.9234	0.3828	0.9481	22.9723
Ours	$\delta$	5	0.3459	2.0288	0.7374	5.8322	0.3829	0.9508	22.9027
		10	0.3700	2.0863	0.7480	5.6233	0.3804	0.8938	22.7996
		15	0.3851	2.1091	0.7545	5.4437	0.3780	0.8587	22.7336
		20	0.3892	2.1318	0.7575	5.4396	0.3796	0.8619	22.7217
NegToMe	$\alpha$	0.07	0.3341	1.9886	0.7360	5.9201	0.3827	0.9410	22.9368

Appendix

Table 5: All GenEval sweep results of Figure 3 for SDXL. Bold settings are used in Table 1 .

Panel of Figure 11 — Figure 11: Effect of correcting the first $N$ transformer layers on DreamSim (left) and HPSv3 (right) for FLUX.2-Klein-9B on DrawBench with $\delta=40^{\circ}$ and $\bar{\theta}=60^{\circ}$ . TokenRepel is applied at the first sampling step; $N=0$ uses no correction.

Panel of Figure 12 — Figure 12: Alignment between the repulsion displacement and the high-density direction before correction for FLUX.2-Klein-9B on DrawBench with $\bar{\theta}=60^{\circ}$ . The plots show the mean token-wise cosine (left) and the fraction of negatively aligned tokens (right) as $\delta$ varies.

Panel of Figure 13 — Figure 13: One example of fidelity preservation on FLUX.2-Klein-9B. Left: generated images of baseline, TokenRepel without preservation (w/o P), and TokenRepel with layerwise preservation (L). Without preservation, TokenRepel increases diversity but introduces artifacts. Right: heatmaps of the top principal component of the features at the 15th layer. Layerwise preservation keeps the feature distribution closer to the baseline.

Panel of Figure 14 — Figure 14: FLUX.1-dev qualitative results on additional prompts with $\delta=50^{\circ}$ , under the same initial noise.

Panel of Figure 15 — Figure 15: FLUX.2-Klein-9B qualitative results on additional prompts with $\delta=80^{\circ}$ , under the same initial noise.

Panel of Figure 16 — Figure 16: SDXL qualitative results on additional prompts with $\delta=25^{\circ}$ , under the same initial noise.

Panel of Figure 17 — Figure 17: More FLUX.1-dev qualitative results on GenEval under the same initial noise.

Panel of Figure 18 — Figure 18: More FLUX.2-Klein-9B qualitative results on GenEval under the same initial noise.

Panel of Figure 19 — Figure 19: More SDXL qualitative results on GenEval under the same initial noise.

Panel of Figure 20 — Figure 20: Qualitative image-editing results on KontextBench using FLUX Kontext under the same initial noise. Each example shows the source image and the instruction (top), followed by four edits from the baseline and TokenRepel.

Panel of Figure 21 — Figure 21: Qualitative image-editing results on KontextBench using FLUX Kontext under the same initial noise. Each example shows the source image and the instruction (top), followed by four edits from the baseline and TokenRepel.

Panel of Figure 22 — Figure 22: FLUX.1-dev results as the token rotation angle $\delta$ increases, under the same initial noise. $\delta=0^{\circ}$ is the baseline. Diversity increases with $\delta$ .

Method	Swept parameter	Swept values	Fixed settings
FLUX.1-dev
Ours	$\delta$	$5,10,\mathbf{20},30$	$\bar{\theta}=45$ ; steps 1–3; constraint in the first 30 layers
NegToMe	$\alpha$	$\mathbf{0.5},0.75,1,1.45$	threshold $0.65$ ; steps 1–7; all layers
PG	coefficient	$\mathbf{0.05},0.1,0.14,0.19$	adaptive median bandwidth; every step
CADS	noise scale	$\mathbf{0.0005},0.001,0.0013,0.0016,0.0022$	$\tau_{1}=0.6$ , $\tau_{2}=0.9$ , $\psi=1$ , rescaling
Contextual	$\eta$	$10^{7},10^{8},10^{9},\mathbf{10^{10}}$	50 iterations; first step

Appendix

Table 6: Hyperparameter settings used in Figure 3 . Each method sweeps one parameter and keeps the others fixed. Bold values are the settings selected for Table 1 . For TokenRepel,

\delta

and

\bar{\theta}

are in degrees.

T2I Generation

Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.

See all 303 papers

Breaking the Lock-in: Diversifying Text-to-Image Generation via Representation Modulation
Jun 5, 2026Dahee Kwon, Haeun Lee, Jaesik ChoiDiffusion Transformer T2I Generation
Korea Advanced Institute of Science and Technology (KAIST) · INEEJI
TPSO: Training-Free Diverse Image Generation via Semantic Prompt Embedding Optimization
Nov 25, 2025Debin Meng, Chen Jin, Zheng Gao +3T2I Generation T2I Diffusion Models
Queen Mary University of London · Centre for AI, AstraZeneca, UK · University of Bedfordshire +1
DiverseVAR: Balancing Diversity and Quality of Next-Scale Visual Autoregressive Models
Nov 26, 2025Mingue Park, Prin Phunyaphibarn, Phillip Y. Lee +1Image Generation Autoregressive Image Generation
KAIST

Jun 5, 2026J/K move · Enter open · S save

Breaking the Lock-in: Diversifying Text-to-Image Generation via Representation Modulation

Dahee Kwon, Haeun Lee, Jaesik Choi

Korea Advanced Institute of Science and Technology (KAIST) · INEEJI

Diffusion Transformer T2I Generation

Open paper AskPDF arXiv

Recent text-to-image models built on large-scale Transformer backbones and flow-based objectives deliver strong text-image alignment and high visual quality, yet often produce overly similar samples under a fixed prompt. Existing diversity-enhancement methods alleviate this issue, but typically require expensive sampling or auxiliary optimization, incurring non-trivial overhead. To investigate the root cause of this homogeneity, we examine intermediate Transformer features and observe that the zero-frequency spatial average (DC) component rapidly converges across seeds early in generation, causing early trajectory lock-in that limits downstream variation. Building on this observation, we propose DC Attenuation for diVersity Enhancement (DAVE), a training-free representation-level intervention that selectively attenuates this component in the early regime. DAVE preserves the sampling pipeline with negligible overhead, improving prompt-consistent diversity while maintaining competitive image quality.

Abstract

T2I Generation

Explore similar work

Breaking the Lock-in: Diversifying Text-to-Image Generation via Representation Modulation

TPSO: Training-Free Diverse Image Generation via Semantic Prompt Embedding Optimization

DiverseVAR: Balancing Diversity and Quality of Next-Scale Visual Autoregressive Models