Improving Controllable Generation: Faster Training and Better Performance via $x_0$-Supervision

Amadou S. Sangare; Adrien Maglo; Mohamed Chaouch; Bertrand Luvison

cs.CVApr 7, 2026

Improving Controllable Generation: Faster Training and Better Performance via $x_0$ -Supervision

Authors: Amadou S. Sangare, Adrien Maglo, Mohamed Chaouch, Bertrand Luvison

Organizations: Université Paris-Saclay, CEA, List, F-91120, Palaiseau, France

Abstract

Text-to-Image (T2I) diffusion/flow models have recently achieved remarkable progress in visual fidelity and text alignment. However, they remain limited when users need to precisely control image layouts, something that natural language alone cannot reliably express. Controllable generation methods augment the initial T2I model with additional conditions that more easily describe the scene. Prior works straightforwardly train the augmented network with the same loss as the initial network. Although natural at first glance, this can lead to very long training times in some cases before convergence. In this work, we revisit the training objective of controllable diffusion models through a detailed analysis of their denoising dynamics. We show that direct supervision on the clean target image, dubbed $x_0$ -supervision, or an equivalent re-weighting of the diffusion loss, yields faster convergence. Experiments on multiple control settings demonstrate that our formulation accelerates convergence by up to 2 $\times$ according to our novel metric (mean Area Under the Convergence Curve - mAUCC), while also improving both visual quality and conditioning accuracy. Our code is available at https://github.com/CEA-LIST/x0-supervision

Figures & tables

Figure 2: Evolution of the final image during a DDIM [ 32 ] sampling with $200$ steps. The top row is the result for Stable Diffusion $1.5$ and the bottom row is the result for a segmentation ControlNet [ 38 ] based on Stable Diffusion $1.5$ . As we can see, images are generated in a coarse-to-fine fashion. The early steps determine the overall layout of the scene (red arrow), while the next steps add fine-grained details (blue arrow). Since semantic segmentation already reveals the target layout, the coarse phase is much faster and the model have more time to add fine-grained details.

Figure 3: Our approach. For a more efficient controllable generation training, we propose to convert any predictor to an $x_{0}$ -predictor, and supervise with the clean image. This simple trick significantly improves the convergence speed and the final performance.

Panel of Figure 4 — Figure 4: Convergence curves for ControlNet and OminiControl on different tasks. We use an EMA weight of $0.9$ to smooth the curves. We can notice that the convergence is faster with $x_{0}$ -supervision.

Panel of Figure 5 — Figure 5: Convergence curves for GLIGEN on different tasks. We use an EMA weight of $0.9$ to smooth the curves. We can notice that the convergence is faster with $x_{0}$ -supervision.

Panel of Figure 6 — Figure 6: The noise schedule and the evolution of the signal-to-noise ratio in Stable Diffusion. The SNR decrease very quickly, using it as the loss weighting function significantly down-weights the learning signal for low SNRs.

ControlNet (Diffusion)
Supervision	Depth			Semantic Seg			Canny Edge			Pose
Supervision	FID $\downarrow$	RMSE $\downarrow$	mAUCC $\downarrow$	FID $\downarrow$	mIoU $\uparrow$	mAUCC $\uparrow$	FID $\downarrow$	F1 $\uparrow$	mAUCC $\uparrow$	FID $\downarrow$	mAP ${}^{\text{pose}}\uparrow$	mAUCC $\uparrow$
$\epsilon$ -ControlNet	$17.68$	$35.79$	$17.70$	$30.05$	$35.84$	$25.19$	$31.18$	$9.22$	$8.09$	$44.09$	$58.00$	$35.86$
$v$ -ControlNet	$17.82$	$36.53$	$21.89$	$32.17$	$31.11$	$13.48$	$31.61$	$8.92$	$6.93$	$44.32$	$52.34$	$22.72$
$x_{0}$ -ControlNet	17.50	35.42	15.98	29.55	39.54	31.52	30.31	9.56	8.76	44.09	59.18	42.19
T2I-Adapter (Diffusion)

Table 1: Comparison between the different supervision signals on different spatially-aligned control modalities and methods. The curves are smoothed with EMA weight

0.9

before computing the mAUCC .

Supervision	Box+Text			Box+Text+Image
Supervision	FID $\downarrow$	mAP ${}^{\text{box}}\uparrow$	mAUCC $\uparrow$	FID $\downarrow$	mAP ${}^{\text{box}}\uparrow$	mAUCC $\uparrow$
GLIGEN (Diffusion)
$\epsilon$ -GLIGEN	$32.58$	$30.70$	$8.28$	21.40	21.31	$6.15$
$v$ -GLIGEN	$35.99$	$8.39$	$1.54$	$23.56$	$2.18$	$9.56$
$x_{0}$ -GLIGEN	28.38	33.30	18.38	$24.23$	$20.76$	8.07

Table 2: Comparison between the different supervision signals on different non-spatially-aligned control modalities. The curves are smoothed with EMA weight

0.9

before computing the mAUCC .

Supervision	Box+Text (mAUCC)			Box+Text+Image (mAUCC)
Supervision	16	32	64	16	32	64
$\epsilon$ -GLIGEN	$1.41$	$2.58$	$8.28$	$1.77$	$3.26$	$6.15$
$x_{0}$ -GLIGEN	1.71	7.72	18.38	1.82	4.50	8.07

Table 3:

x_{0}

-supervision has higher batch size efficiency. Since non-spatially-aligned controls require high batch sizes to converge, we compare how reducing the batch size affects the convergence speed for

x_{0}

- and

\epsilon

-supervision.

Panel of Figure 7 — Figure 7: Convergence comparison between $x_{0}$ -supervision and $\frac{\sigma_{t}^{2}}{\alpha_{t}^{2}}\epsilon$ -supervision. We can see that they have the same convergence speed, hence validating our insights in Sec. 3.3 .

Controllable Image Generation

Controllable image generation steers images using spatial, visual, semantic, stylistic, or physical constraints beyond ordinary text prompts.

See all 210 papers

Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation
Aug 14, 2026Nikolai Röhrich, Isabell Hans, Felix Krause +1Diffusion Model Guidance T2I Generation
LMU Munich, Germany · Konrad Zuse School of Excellence in Reliable AI (relAI), Germany · Munich Center for Machine Learning (MCML), Germany
FlashControl: One-Step Controllable Generator via Distillation-Friendly Single-Stream Teachers
Oct 15, 2026Ngan Nguyen, Dung Nguyen, Quan Dao +4Controllable Image Generation Conditional Image Generation
Qualcomm AI Research · Rutgers University · Adelaide University +1
MixDiffusion: Mixing Diffusion-based Uni-condition Text-to-Image Generation Models for Multi-condition Image Synthesis
Jul 20, 2026Pengcheng Wan, Liang Han, Lin Xu +2Conditional Image Generation Controllable Image Generation
Harbin Institute of Technology (Shenzhen)

Aug 14, 2026J/K move · Enter open · S save

Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation

Nikolai Röhrich, Isabell Hans, Felix Krause, Björn Ommer

LMU Munich, Germany · Konrad Zuse School of Excellence in Reliable AI (relAI), Germany · Munich Center for Machine Learning (MCML), Germany

Diffusion Model Guidance T2I Generation Controllable Image Generation Diffusion Guidance Diffusion Models

Open paper AskPDF arXiv

Text-to-image diffusion models have two major drawbacks that severely limit their practical utility: (1) standard models lack an intrinsic mechanism for continuous, concept-specific guidance (e.g., for precisely controlling how aesthetically pleasing an image looks), and (2) they lack reliability for tasks requiring high local coherence (e.g., generating text or human hands). To tackle these issues, we introduce a novel notion of concept-wise mutual information and find large, concept-dependent differences between individual layers, demonstrating that the generation of specific structures is localized in distinct parts of the network. We exploit this insight by reinforcing the impact of concept-relevant layers in Concept Guidance (CoG), a precise, target-specific guidance method that works for models out-of-the-box without additional training, external models, gradients, or prompt engineering. CoG first quantifies each layer's concept-specific impact and then guides denoising using a weighted combination of predictions generated with concept-relevant layers skipped. We demonstrate performance increases across various targets and popular models like PixArt-alpha, SD3, SD3.5, and FLUX.1-dev. Code is available at https://github.com/CompVis/concept_guidance

Abstract

Controllable Image Generation

Explore similar work

Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation

FlashControl: One-Step Controllable Generator via Distillation-Friendly Single-Stream Teachers

MixDiffusion: Mixing Diffusion-based Uni-condition Text-to-Image Generation Models for Multi-condition Image Synthesis