Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation

Nikolai Röhrich; Isabell Hans; Felix Krause; Björn Ommer

cs.CVAug 14, 2026

Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation

Authors: Nikolai Röhrich, Isabell Hans, Felix Krause, Björn Ommer

Organizations: LMU Munich, Germany · Konrad Zuse School of Excellence in Reliable AI (relAI), Germany · Munich Center for Machine Learning (MCML), Germany

Abstract

Text-to-image diffusion models have two major drawbacks that severely limit their practical utility: (1) standard models lack an intrinsic mechanism for continuous, concept-specific guidance (e.g., for precisely controlling how aesthetically pleasing an image looks), and (2) they lack reliability for tasks requiring high local coherence (e.g., generating text or human hands). To tackle these issues, we introduce a novel notion of concept-wise mutual information and find large, concept-dependent differences between individual layers, demonstrating that the generation of specific structures is localized in distinct parts of the network. We exploit this insight by reinforcing the impact of concept-relevant layers in Concept Guidance (CoG), a precise, target-specific guidance method that works for models out-of-the-box without additional training, external models, gradients, or prompt engineering. CoG first quantifies each layer's concept-specific impact and then guides denoising using a weighted combination of predictions generated with concept-relevant layers skipped. We demonstrate performance increases across various targets and popular models like PixArt-alpha, SD3, SD3.5, and FLUX.1-dev. Code is available at https://github.com/CompVis/concept_guidance

Figures & tables

Figure 1: Concept Guidance (CoG) improves arbitrary quantifiable concepts across a variety of T2I models. CoG enables models to recover from classical failure modes while also delivering better generative quality for abstract concepts.

Figure 2: Concept Guidance precisely approximates the target direction by combining per-layer skip predictions using concept-relevant layers (

\alpha,\beta,\gamma

). It computes individual skip-layer noise predictions (

\epsilon_{[\theta\setminus\alpha]},\epsilon_{[\theta\setminus\beta]},\epsilon_{[\theta\setminus\gamma]}

) and extrapolates over them to precisely estimate the target direction.

Panel of Figure 3 — Figure 3: Different concepts concentrate in different layers, enabling concept-specific layer selection for guidance. Bar plots report per-layer mutual information in (a) FLUX.1-dev and (b) PixArt- $\alpha$ , revealing strong, concept-dependent variation.

Figure 4: Concept Guidance Workflow. Left: In a one-time profiling stage, each layer is skipped individually, generations are scored, and top- $k$ layers are selected. Right: At inference, per-layer skipped predictions are aggregated into a performance-weighted negative prediction; CoG then extrapolates from beyond the standard prediction.

Figure 5: Optimizing aesthetics with CoG improves perceptual quality. Side-by-side examples compare CFG vs. CoG.

Figure 6: CoG improves fine-grained local coherence for hard failure cases like visible text and hands. Side-by-side examples compare CFG vs. CoG.

Table 7

Figure 7: Multi-target Concept Guidance simultaneously improves multiple concepts on FLUX.1-dev compared to CFG.

Table 9

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Task	$k$	Skipped Layers ( $\mathcal{K}$ )	Weights ( $\Omega$ )
PixArt- $\alpha$	Human Hands	[4]	[3, 23, 8, 18]	[1.00, 0.88, 0.73, 0.63]
PixArt- $\alpha$	Aesthetics	[3]	[9, 20, 5]	[1.00, 0.28, 0.12]
SD3	Visible Text	[3]	[7, 11, 4]	[1.00, 0.55, 0.45]
	Human Hands	[4]	[9, 6, 5, 10]	[1.00, 0.48, 0.45, 0.21]
	Aesthetics	[3]	[6, 7, 8]	[1.00, 0.71, 0.70]
SD3.5	Visible Text	[2]	[15, 10]	[1.00, 0.49]

Appendix

Table 6: Layer Configurations. Top-

k

skipped layers (

\mathcal{K}

) and their corresponding weights (

\Omega

) for all evaluated models and target concepts.

Figure 8: Comparison of Classifier-Free Guidance [ 16 ] and Concept Guidance for additional targets where layers are profiled using a VLM ( InternVL3-14B ).

Model	Method	Text (↑)	Hands (↑)	Aesthetics (↑)
PixArt- $\alpha$	STG [ 21 ]	-	0.384	6.502
PixArt- $\alpha$	CoG single	-	0.478 $\blacktriangle$ 24.48%	6.606 $\blacktriangle$ 1.6%
SD3	STG [ 21 ]	0.441	0.662	6.359
SD3	CoG single	0.462 $\blacktriangle$ 4.8%	0.664 $\blacktriangle$ 0.3%	6.359 $\blacktriangle$ 0.0%
SD3.5	STG [ 21 ]	0.531	0.540	6.297
SD3.5	CoG single	0.564 $\blacktriangle$ 6.2%	0.603 $\blacktriangle$ 11.7%	6.344 $\blacktriangle$ 0.7%

Appendix

Table 7: Leveraging per-concept layer information yields significant gains over guiding with a fixed layer as is the case in Spatio-Temporal Skip-Guidance.

Table 13

Model	CFG [ 16 ]	CoG single ( k =1)	CoG multi ( k =2)	+Sec / Layer
PixArt- $\alpha$	3.5	3.9	4.3	0.4
Stable Diffusion 3	4.4	6.2	8	1.8
Stable Diffusion 3.5	5.9	7.7	9.5	1.8
FLUX.1-dev	12.9	25.3	37.7	12.4

Appendix

Table 10: Inference time (s) on an A100 GPU.

Model	Metric	Guidance Scale ( $\lambda$ )
Model	Metric	1.25	1.50	1.75	2.00	2.25	2.50	2.75	3.00
PixArt- $\alpha$ †	Text ( $\uparrow$ )	-	-	-	-	-	-	-	-
	Hands ( $\uparrow$ )	0.396	0.373	0.418	0.504	0.503	0.496	0.492	0.468
	Aesthetics ( $\uparrow$ )	6.647	6.678	6.695	6.723	6.716	6.759	6.740	6.734
SD3	Text ( $\uparrow$ )	0.482	0.472	0.486	0.489	0.506	0.481	0.480	0.474
	Hands ( $\uparrow$ )	0.602	0.631	0.660	0.670	0.640	0.662	0.605	0.629

Appendix

Table 11: CoG Guidance Scale Sweep. For most models and tasks, the optimal guidance scale falls in the range between

2.0

and

2.5

. † PixArt-

\alpha

does not generate any visible text.

Figure 11: Prompt Samples for Text Generation.

Figure 12: Prompt Samples for Hand Generation.

Figure 13: Prompt Samples for Aesthetics.

Figure 14: Uncurated Comparison of Classifier-Free Guidance (CFG) [ 16 ] , Naive Skip-Guidance as found in Stable Diffusion 3 [ 42 ] , Spatio-Temporal Skip Guidance [ 21 ] and Concept Guidance for Text Generation (FLUX.1-dev).

Figure 15: Uncurated Comparison of Classifier-Free Guidance (CFG) [ 16 ] , Naive Skip-Guidance as found in Stable Diffusion 3 [ 42 ] , Spatio-Temporal Skip Guidance [ 21 ] and Concept Guidance for Aesthetics (FLUX.1-dev).

Explore similar work

Rectify Then Diffuse: Disentangling Concepts Before Denoising Trajectory Unfolds
Aug 4, 2026Ning Zhu, An Chen, Mengfei Zhao +4T2I Generation Attention Control in Diffusion Models
Glasgow College, University of Electronic Science and Technology of China · School of Mathematical Sciences, University of Electronic Science and Technology of China
Injecting Image Guidance into Text-Conditioned Diffusion Models at Inference
May 24, 2026Agata Żywot, Iason Skylitsis, Thijmen Nijdam +4Diffusion Model Guidance Test-Time Optimization
University of Amsterdam, Netherlands
DICE: Distilling Classifier-Free Guidance into Text Embeddings
Feb 6, 2025Zhenyu Zhou, Defang Chen, Can Wang +2Diffusion Model Alignment Diffusion Model Guidance
Zhejiang University, State Key Laboratory of Blockchain and Data Security · University at Buffalo, State University of New York

Aug 4, 2026, cs.CVJ/K move · Enter open · S save

Rectify Then Diffuse: Disentangling Concepts Before Denoising Trajectory Unfolds

Ning Zhu, An Chen, Mengfei Zhao, Juntao Xu+3

Glasgow College, University of Electronic Science and Technology of China · School of Mathematical Sciences, University of Electronic Science and Technology of China

T2I Generation Attention Control in Diffusion Models Compositional T2I Generation T2I Diffusion Models

Open paper AskPDF arXiv

Text-to-image diffusion models can generate individual concepts well, but they often omit or merge concepts incorrectly with multiple concepts. We trace these failures to an early coordination bottleneck: before denoising begins, prompt-conditioned attention may allocate different concepts to strongly overlapping spatial support, which can keep their attention coupled as denoising proceeds. This observation motivates treating compositional generation as a boundary-condition problem rather than repeatedly controlling the evolving trajectory. To this end, we propose Rectify-then-Diffuse (RTD), a training-free framework that rectifies the initial allocation once before standard denoising. Firstly, we propose Soft-Overlap Disentanglement (SOD), which converts normalized overlap between pilot concept maps into a differentiable and layout-agnostic separation objective. Secondly, we introduce Isotropic Gradient Rectification (IGR), which normalizes the SOD gradient and applies a bounded latent displacement with a consistent scale across prompts and initializations. Extensive experiments show that RTD achieves state-of-the-art compositional fidelity and robust gains. On the AE-Bench object pair subset, RTD improves BLIP-VQA by 45.8% and ImageReward by 19.6% over CO3 while running 2.3

\times

faster. Code will be released at https://github.com/Z-yiwei/rectify-then-diffuse

Abstract

Explore similar work

Rectify Then Diffuse: Disentangling Concepts Before Denoising Trajectory Unfolds

Injecting Image Guidance into Text-Conditioned Diffusion Models at Inference

DICE: Distilling Classifier-Free Guidance into Text Embeddings