Let Triggers Control: Frequency-Aware Dropout for Effective Token Control

Junyoung Koh; Hoyeon Moon; Dongha Kim; Seungmin Lee; Sanghyun Park; Min Song

cs.CVMar 28, 2026

Let Triggers Control: Frequency-Aware Dropout for Effective Token Control

Authors: Junyoung Koh, Hoyeon Moon, Dongha Kim, Seungmin Lee, Sanghyun Park, Min Song

Organizations: Yonsei University · Onoma AI

Abstract

Text-to-image models such as Stable Diffusion have achieved unprecedented levels of high-fidelity visual synthesis. As these models advance, personalization of generative models -- commonly facilitated through Low-Rank Adaptation (LoRA) with a dedicated trigger token -- has become a significant area of research. Previous works have naively assumed that fine-tuning with a single trigger token to represent new concepts. However, this often results in poor controllability, where the trigger token alone fails to reliably evoke the intended concept. We attribute this issue to the frequent co-occurrence of the trigger token with the surrounding context during fine-tuning, which entangles their representations and compromises the token's semantic distinctiveness. To disentangle this, we propose Frequency-Aware Dropout (FAD) -- a novel regularization technique that improves prompt controllability without adding new parameters. FAD consists of two key components: co-occurrence analysis and curriculum-inspired scheduling. Qualitative and quantitative analyses across token-based diffusion models (SD~1.5 and SDXL) and natural language--driven backbones (FLUX and Qwen-Image) demonstrate consistent gains in prompt fidelity, stylistic precision, and user-perceived quality. Our method provides a simple yet effective dropout strategy that enhances controllability and personalization in text-to-image generation. Notably, it achieves these improvements without introducing additional parameters or architectural modifications, making it readily applicable to existing models with minimal computational overhead.

Figures & tables

Figure 1: Comparison of images generated using Normal Dropout and Frequency-Aware Dropout with the prompt: pochacco, riding a bike, sunny day, flower petals , where pochacco serves as the trigger token.

Figure 2: Comparison between Normal Dropout (top) and Frequency-Aware Dropout (FAD) (bottom). Top : Normal Dropout lets style cues scatter and the trigger token is ignored. Bottom : FAD raises the dropout of tokens that often co-occur with the trigger, binding the style to the trigger token $t$ .

Panel of Figure 3 — Figure 3: Tag frequency for each dataset. The trigger token is indicated in red .

		FLUX.1-dev	Qwen-Image
Dataset	Method	DINO $\uparrow$	DINO $\uparrow$
hsng	Normal	0.5628	0.7347
	FAD (anchor: japanese man)	0.5725	0.7840
mbst	Normal	0.5316	0.6784
	FAD (anchor: american man)	0.5453	0.6840
faker	Normal	0.5268	0.6373

Table 3: DINO score comparison under anchoring token training for DiT-based backbones. Best values per dataset are in bold.

Panel of Figure 4 — Figure 4: Anchoring results for hsng trained with anchor token japanese man . When the anchor is removed (“except”), FAD/sFAD degenerates to a generic face (often child-like), indicating tighter identity binding. The Normal baseline still shows identity leakage under the except prompt, suggesting weaker disentanglement.

Panel of Figure 5 — Figure 5: Attention map results using trigger token pikachu and faker under three dropout strategies: Normal dropout (left) , FAD (middle) and sFAD (right) . Each image is generated with the following prompts: pikachu — pikachu, animal focus, black eyes, closed mouth, full body, looking at viewer, no humans, simple background, smile, solo, standing, straight-on . faker — faker, 1boy, asian, black eyes, black hair, glasses, grey background, hand on own chin, hand up, looking at viewer, male focus, photorealistic, red jacket, round eyewear, short hair, simple background, solo, upper body .

Hyperparameter	Value
Train batch size	8
Gradient accumulation steps	64
Max train steps	1500
Min adaptive dropout	0.35
Max adaptive dropout	1.0
Step dropout start	0.1

Table 4: Training hyperparameters for SD 1.5 and SDXL.

Hyperparameter	FLUX	Qwen
Base model	FLUX.1-dev	Qwen-Image
LoRA rank	32	32
Learning rate	$1\!\times\!10^{-4}$	$1\!\times\!10^{-4}$
Num epochs	2	2
Dataset repeat	50	50
Grad. accum. steps	1	1

Table 5: Training hyperparameters for FLUX and Qwen-Image.

Panel of Figure 6 — Figure 6: Multi-concept Training: Normal Dropout (left) vs. sFAD (right). The model trained with sFAD successfully reproduces both characters, whereas Normal Dropout results in noticeable deformations.

Model	Method	Pikachu			Pochacco
		o	x	$\Delta$	o	x	$\Delta$
SD 1.5	Normal	0.9987	0.9987	+0.0000	0.9431	0.9350	-0.0081
	Adaptive	0.9987	0.9987	+0.0000	0.9528	0.9487	-0.0041
	Step	0.9987	0.9986	-0.0001	0.9487	0.9428	-0.0059
SDXL	Normal	0.9992	0.9972	-0.0020	0.8996	0.8341	-0.0655
	Adaptive	0.9996	0.9451	-0.0545	0.9091	0.7514	-0.1577

Table 9: CCIP (

\uparrow

) scores with and without trigger token. Bold indicates the largest decrease in

\Delta

(x - o).

Panel of Figure 7 — Figure 7: Comparison of FID ( $\downarrow$ ) scores of three methods across all steps for six datasets, extracted from SD 1.5 .

Char.	Step	Metric	SD 1.5		SDXL
			Linear	Exp_up	Linear	Exp_up
Faker	0.1 $\sim$ 0.8	FID ( $\downarrow$ )	196.119	195.863	191.29	197.76
		DINO ( $\uparrow$ )	0.9100	0.9100	0.9131	0.9107
		InsightFace ( $\downarrow$ )	26.976	26.954	24.51	25.13
	0.8 $\sim$ 0.1	FID ( $\downarrow$ )	195.411	194.655	196.85	194.93
		DINO ( $\uparrow$ )	0.9107	0.9100	0.9087	0.9120

Table 10: Comparison of SD 1.5 and SDXL for Step (0.1~~0.8) vs Step (0.8~~0.1) using Linear / Exp_up methods.

Panel of Figure 8 — Figure 8: Comparison of FID ( $\downarrow$ ) scores of three methods across all steps for six datasets, extracted from SDXL .

Panel of Figure 9 — Figure 9: Comparison of DINO ( $\uparrow$ ) scores of three methods across all steps for six datasets, extracted from SD 1.5 .

Panel of Figure 10 — Figure 10: Comparison of DINO ( $\uparrow$ ) scores of three methods across all steps for six datasets, extracted from SDXL .

Panel of Figure 11 — Figure 11: Comparison of InsightFace ( $\downarrow$ ) scores of three methods across all steps for four human datasets, extracted from SD 1.5 .

Panel of Figure 12 — Figure 12: Comparison of CCIP ( $\uparrow$ ) scores of three methods across all steps for two character datasets, extracted from SD 1.5 .

Panel of Figure 13 — Figure 13: Comparison of InsightFace ( $\downarrow$ ) scores of three methods across all steps for four human datasets, extracted from SD XL .

Panel of Figure 14 — Figure 14: Comparison of CCIP ( $\uparrow$ ) scores of three methods across all steps for two character datasets, extracted from SDXL .

Evaluation Prompt: You will be shown TWO images: - Image 1 : the ground truth character image (reference only) - Image 2 : a generated image that should depict the same character. Your task is to critically evaluate how well Image 2 resembles Image 1 , considering the character identity and overall image quality. Ground Truth Character key features (from Image 1) : Character (mbst): mbst, beard, brown hair, short hair Evaluation Dimensions Character Similarity : • How closely does the generated character (Image 2) resemble the reference character (Image 1)? • Consider race, facial structure, hairstyle, eye color, and other distinctive features. Composition & Image Quality : Assess technical and artistic quality of Image 2, including: • Composition coherence • Deformities (e.g., extra limbs, distorted face) • Texture, lighting, color accuracy, clarity Scoring Criteria Character Similarity (10-point scale) : 10 = highly similar, 1 = completely different • Deduct 2 points for incorrect race or major facial mismatch • Deduct 1 point for missing or incorrect key features (e.g., hair/eye color) Composition & Image Quality (10-point scale) : 10 = excellent, 1 = very poor • Deduct 2 points for major deformities • Deduct 1 point for lighting/color/texture issues or minor artifacts Output Format (strict) Image2 Evaluation: - [Concise bullet points listing observed issues or mismatches] Scores: Character Similarity: [score] / 10 Composition & Image Quality: [score] / 10 Notes • Do not evaluate Image 1. • Be critical and specific. • Start your answer immediately after this instruction.

Table 11: The full version of evaluation prompts for comparative evaluation with GPT-4.1

Image 2 Evaluation: Character Similarity : • Hairstyle is similar (short, brown hair, styled to the side), but appears slightly neater and more stylized than in Image 1 ( -1 points ) . • Beard is present, but the style is somewhat different ( -1 points ) . • General facial structure is fairly close but the face in Image 2 appears slimmer and more angular ( -1 points ) . Composition & Image Quality : • No major issues with race or overall appearance. • Image 2 shows the person holding a microphone, altering focus compared to the original ( -1 points ) . • No major deformities or artifacts detected. • Lighting and texture are clean, but the image feels more artificial and softened, losing some realism compared to the reference.

Scores: Character Similarity: 7 / 10 Composition & Image Quality: 9 / 10

Table 12: The full version of evaluation result for comparative evaluation with GPT-4.1

Controllable Image Generation

Controllable image generation steers images using spatial, visual, semantic, stylistic, or physical constraints beyond ordinary text prompts.

See all 215 papers

Aug 14, 2026

Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation

Text-to-image diffusion models have two major drawbacks that severely limit their practical utility: (1) standard models lack an intrinsic mechanism for continuous, concept-specific guidance (e.g., for precisely controlling how aesthetically pleasing an image looks), and (2) they lack reliability for tasks requiring high local coherence (e.g., generating text or human hands). To tackle these issues, we introduce a novel notion of concept-wise mutual information and find large, concept-dependent differences between individual layers, demonstrating that the generation of specific structures is localized in distinct parts of the network. We exploit this insight by reinforcing the impact of concept-relevant layers in Concept Guidance (CoG), a precise, target-specific guidance method that works for models out-of-the-box without additional training, external models, gradients, or prompt engineering. CoG first quantifies each layer's concept-specific impact and then guides denoising using a weighted combination of predictions generated with concept-relevant layers skipped. We demonstrate performance increases across various targets and popular models like PixArt-alpha, SD3, SD3.5, and FLUX.1-dev. Code is available at https://github.com/CompVis/concept_guidance

Nikolai Röhrich, Isabell Hans, Felix Krause +1

LMU Munich, Germany · Konrad Zuse School of Excellence in Reliable AI (relAI), Germany · Munich Center for Machine Learning (MCML), Germany

Diffusion Model Guidance T2I Generation Controllable Image Generation Diffusion Guidance Diffusion Models

Ask PDF

Nov 25, 2025

TPSO: Training-Free Diverse Image Generation via Semantic Prompt Embedding Optimization

Image diversity remains a fundamental challenge for text-to-image diffusion models. Low-diversity generation often leads to repetitive outputs, increasing sampling redundancy and hindering both creative exploration and downstream applications. A key factor is the tendency of diffusion models to collapse toward strong modes in the learned distribution. Existing attempts to improve diversity, such as steering-based guidance, often introduce distortions that degrade image quality. To address this issue, we propose Token-Prompt Embedding Space Optimization (TPSO), a training-free and model-agnostic module. TPSO introduces learnable parameters to explore underrepresented regions of the token embedding space, reducing the tendency to repeatedly sample from strong modes of the distribution. Meanwhile, a prompt-level semantic constraint regulates distribution shifts, preventing quality degradation while preserving semantic fidelity. Extensive experiments on MS-COCO across three representative diffusion backbones demonstrate that TPSO substantially improves diversity, boosting performance from 1.10 to 4.18, while maintaining image quality with only a modest inference-time overhead of 3.6% to 8.9%. Code is available at: https://github.com/Open-Debin/TPSO.

Debin Meng, Chen Jin, Zheng Gao +3

Queen Mary University of London · Centre for AI, AstraZeneca, UK · University of Bedfordshire +1

T2I Generation T2I Diffusion Models

Ask PDF

Apr 7, 2026

Improving Controllable Generation: Faster Training and Better Performance via $x_0$ -Supervision

Text-to-Image (T2I) diffusion/flow models have recently achieved remarkable progress in visual fidelity and text alignment. However, they remain limited when users need to precisely control image layouts, something that natural language alone cannot reliably express. Controllable generation methods augment the initial T2I model with additional conditions that more easily describe the scene. Prior works straightforwardly train the augmented network with the same loss as the initial network. Although natural at first glance, this can lead to very long training times in some cases before convergence. In this work, we revisit the training objective of controllable diffusion models through a detailed analysis of their denoising dynamics. We show that direct supervision on the clean target image, dubbed

x_0

-supervision, or an equivalent re-weighting of the diffusion loss, yields faster convergence. Experiments on multiple control settings demonstrate that our formulation accelerates convergence by up to 2

\times

according to our novel metric (mean Area Under the Convergence Curve - mAUCC), while also improving both visual quality and conditioning accuracy. Our code is available at https://github.com/CEA-LIST/x0-supervision

Amadou S. Sangare, Adrien Maglo, Mohamed Chaouch +1

Université Paris-Saclay, CEA, List, F-91120, Palaiseau, France

Controllable Image Generation Conditional Diffusion Models T2I Diffusion Models Diffusion Models

Ask PDF

Abstract

Controllable Image Generation

Explore similar work

Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation

TPSO: Training-Free Diverse Image Generation via Semantic Prompt Embedding Optimization

Improving Controllable Generation: Faster Training and Better Performance via x0x_0x0​-Supervision

Improving Controllable Generation: Faster Training and Better Performance via $x_0$ -Supervision