Despite significant progress in text-to-image diffusion models, achieving precise spatial control over generated outputs remains challenging. One of the popular approaches for this task is ControlNet, which introduces an auxiliary conditioning module into the architecture. To improve alignment of the generation image and control, ControlNet++ proposes a cycle consistency loss to refine correspondence between controls and outputs, but restricts its application to the final denoising steps, while the main structure is introduced at an early stage of generation. To address this issue, we suggest InnerControl -- a training strategy that enforces spatial consistency across a broad range of diffusion steps, including early high-noise ones. Specifically, we train lightweight control prediction probes -- small convolutional networks -- to reconstruct input control signals (e.g., edges, depth) from intermediate diffusion features across denoising steps. We show empirically that such models can extract signals even from very noisy latents and utilize these models to generate pseudo ground truth controls during training. Suggested approach enables alignment loss that minimizes the difference between predicted and target condition over most of the diffusion trajectory. Our experiments demonstrate that our method improves control alignment, with perceptual fidelity preserved or improved in most settings and small FID trade-offs in others. Our approach can be applied to both UNet- and Transformer-based diffusion architectures and be integrated with established training techniques (e.g., ControlNet++) to achieve high performance across different condition methods such as edge, segmentation and depth conditions.
Figures & tables
Figure 1: Visualizing the trade-off between control consistency (RMSE) and image fidelity (FID) when extending reward losses to early denoising stages. Left: Generated samples illustrate visual artifacts when applying only reward loss ( top ) compared to reward plus alignment loss ( bottom ) during early diffusion steps. Right: Quantitative analysis shows the trade-off between control precision (RMSE ↓) and image fidelity (FID ↓), highlighting their inverse relationship.
Figure 2: Left: RMSE between final-image depth and DPT prediction for single-step predicted images (green) and intermediate-feature depth estimates (blue) in SD1.5. Right: RMSE between control depth and DPT prediction for single-step predicted images (green) and intermediate-feature depth estimates (blue) in ControlNet.
Figure 3: One-step predictions ( top ) at increasing noise levels, with corresponding depth maps from DPT [ 32 ] ( middle ) and intermediate UNet features ( bottom ).
Figure 4: Pipeline overview. Schematic illustration of our InnerControl framework, highlighting the integration of the alignment loss. The key difference from ControlNet++ is the Alignment module , which processes intermediate features extracted from the diffusion denoiser (UNet decoder features and Transformer block features). These features are passed through a lightweight aggregation network to predict spatial control signals (e.g., depth or edge maps), which are then compared with the input control cspat to enforce consistency at each denoising step.
Method
T2I Model
Hed Edge
LineArt Edge
Depth Map
Segmentation
SSIM ↑
FID ↓
SSIM ↑
FID ↓
RMSE ↓
FID ↓
mIoU ↑
FID ↓
Guidance scale = 7.5
ControlNet
SDXL
—
—
—
—
40.00
—
—
—
T2I‑Adapter
SDXL
—
—
0.639
—
39.75
—
—
—
T2I‑Adapter
SD1.5
—
—
—
—
48.40
22.52
—
—
Gligen
SD1.4
0.563
—
—
—
38.83
18.36
—
—
Table 1: Unified comparison on the MultiGen‑20M benchmark (segmentation on ADE20K). Controllability is evaluated using SSIM ( ↑ ) for HED/LineArt, RMSE ( ↓ ) for depth, and mIoU ( ↑ ) for segmentation; fidelity is measured by FID ( ↓ ). Numbers for earlier baselines are taken from prior publications; ControlNet++, Ctrl‑U, and InnerControl are re-trained by us under identical settings (Sec. 5.1 ).
Method
Model
RMSE ↓
FID ↓
Guidance scale = 7.5
ControlNet++
SDXL
30.24
17.00
InnerControl (Ours)
SDXL
28.31
17.10
Guidance scale = 4.5
ControlNet++
SD 3.5
31.07
14.24
InnerControl (Ours)
SD 3.5
30.31
14.34
Table 2: Depth-control results for modern diffusion backbones on MultiGen-20M.
Figure 5: Qualitative Comparison with Baselines: Results for LineArt (top), HED and Depth (middle), and segmentation (bottom), using identical prompts and a guidance scale of 7.5 . Our method better aligns outputs with input controls than competing approaches.
Estimator
Base
R
A
R + A (Ours)
DPT-Large
33.1
29.2
32.1
27.9
DA-V2-Large
33.9
32.4
28.9
31.5
Table 3: Independent depth evaluation on FLUX.1. RMSE ( ↓ ) for the same generations under DPT-Large and Depth Anything V2-Large. A/R denote alignment/reward; R+A is InnerControl.
Method
Steps
Steps Reward
RMSE ↓
FID ↓
CLIP ↑
Guidance scale = 7.5
ControlNet
0
0
33.95
18.61
32.175
ControlNet
920
0
32.80
18.55
32.05
ControlNet
0
920
25.70
22.43
31.43
ControlNet
0
400
27.63
18.59
32.00
ControlNet
0
200
29.66
18.51
32.11
Table 4: Ablation study on the Depth Map control task from the MultiGen-20M benchmark, analyzing the influence of the number of timesteps over which reward and alignment losses are applied.
Model
Metric
Base
R
A
R+A (InnerControl)
SD1.5
RMSE ↓
34.0
27.6
32.8
26.1
FID ↓
18.6
18.6
18.6
18.3
SDXL
RMSE ↓
34.9
30.2
33.8
28.3
FID ↓
15.5
17.0
15.6
17.1
FLUX
RMSE ↓
33.1
29.2
32.1
27.9
FID ↓
16.0
23.6
15.8
22.5
Table 5: Ablation study. R – reward, A – alignment, R+A – proposed InnerControl
Model
Metric
Base
Base + Alignment
ControlNet
RMSE ↓
34.0
32.8
FID ↓
18.6
18.6
ControlNet++
RMSE ↓
27.6
26.1
FID ↓
18.6
18.3
CTRL-U
RMSE ↓
26.5
25.4
FID ↓
18.7
18.5
Table 6: Pipeline agnostic comparison (SD1.5)
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Segmentation Mask
Depth Map
Hed Edge
LineArt Edge
Reward Model (RM)
UperNet-R50
DPT-Hybrid
ControlNet∗
ControlNet∗
RM Performance
ADE20K(mIoU): 42.05
NYU(AbsRel): 8.69
-
-
Evaluation Model (EM)
Mask2Former
DPT-Large
ControlNet∗
ControlNet∗
EM Performance
ADE20K(mIoU): 56.01
NYU(AbsRel): 8.32
-
-
Reward Loss
CrossEntropy Loss
MSE Loss
MSE Loss
MSE Loss
Loss Weight α
0.5
0.5
1.0
10.0
Appendix
Table 7: Details about some training parameters and reward models. ControlNet∗ denotes utilizing the same model to extract signal as ControlNet [ 49 ]
Segmentation Mask
Depth Map
HED Edge
LineArt Edge
Dataset
ADE20K [ 52 , 51 ]
MultiGen20M [ 28 ]
MultiGen20M [ 28 ]
MultiGen20M [ 28 ]
Training Samples
20,210
2,560,000
2,560,000
2,560,000
Evaluation Samples
2,000
5,000
5,000
5,000
Evaluation Metric
mIoU ↑
RMSE ↓
SSIM ↑
SSIM ↑
Appendix
Table 8: Datasets and evaluation details for explored tasks. ↑ denotes higher is better, ↓ – lower is better.
Component
Time per iteration
Peak Memory
Delta per GPU
Reward-only step
0.15 s
-
Alignment-only step
0.01 s
-
Full training iteration (rewarding)
0.47 s
41 Gb
Full training iteration (ours)
0.46 s
44 Gb
Appendix
Table 9: Component Performance Metrics
β
loss
g.s.
RMSE
FID
CLIP
0.5
MSE
7.5
26.09
17.82
31.72
1
MSE
7.5
26.09
18.29
32.00
2
MSE
7.5
26.21
18.24
31.82
5
MSE
7.5
25.98
18.08
31.68
Appendix
Table 10: Ablations on different β values for alignment loss in the depth control task.
β
steps
loss
g.s.
mIoU
FID
CLIP
0.05
[450,980]
CrossEntropy
7.5
40.22
37.7
30.2
0.1
[450,980]
CrossEntropy
7.5
41.68
40.9
29.90
0.1
[400,980]
CrossEntropy
7.5
41.68
43.0
29.91
0.1
[450,920]
CrossEntropy
7.5
40.20
41.1
26.25
Appendix
Table 11: Ablations on different β values and the steps at which alignment loss is applied in the segmentation control task.
Figure 6: Reward-model and alignment-probe MSE across diffusion timesteps for depth control (SD3.5). The reward signal is strongest at low-noise steps, while the alignment probe remains stable over a wider step range, motivating the split schedule used in the experiments.
Question
Depth
Edges
Segment.
Prompt fidelity (Q1)
55%
67%
50%
Control Alignment (Q2)
65%
79%
72%
Aesthetic quality (Q3)
63%
77%
85%
Appendix
Table 13: User study . Results of pairwise comparisons on 100 randomly sampled validation image pairs. 12 participants took part in the study. Values denote the percentage of cases where InnerControl is preferred over Reward.
Figure 7: Quality comparison for attention-based and convolution-based predictions for depth maps extraction task
Backbone
Num. of Parameters
UNet
8.50M
UNet (Unified)
9.49M
SD3.5 (fast)
8.10M
SD3.5 (max)
79.8M
FLUX
9.90M
Appendix
Table 14: Comparison of Alignment Model Parameters across different architectures.
Figure 8: Probe MSE across diffusion timesteps for two SD3.5 probe configurations. The fast configuration, which uses only five selected Transformer layers, remains close to the max configuration that aggregates all valid layers.
Figure 9: Quantitative and Qualitative Comparison of Models. Results comparing the performance of the unified model against individual, single-task models on three diverse visual tasks: Depth estimation, Holistically-Nested Edge Detection (HED), and LineArt extraction.
Figure 10: Comparison of signals estimated from one-step prediction and intermediate features for segmentation, HED, and LineArt control tasks.
Figure 11: Application of small networks for guidance during inference.
Figure 12: Visualization of difference between extracted signal from intermediate features and input control after our training applied ( top ) and for standard ControlNet ( bottom )
Figure 13: Visualization of various numbers of sampling steps for depth prediction. The red rectangles indicate artifacts detected in the depth predictions.
Figure 14: Visualization of one-step prediction, estimated depth of the generated image, corresponding DPT depth estimation, and depth extracted from intermediate features.
Figure 15: Visualization of one-step prediction, estimated HED of the generated image, corresponding HED estimation, and HED extracted from intermediate features.
Figure 16: Visualization of one-step prediction, estimated segmentation mask of the generated image, corresponding segmentation maps, and segmentation maps extracted from intermediate features.
Figure 17: Visualization of one-step prediction, estimated LineArt of the generated image, corresponding LineArt estimation, and LineArt extracted from intermediate features.
Method
RMSE ↓
FID ↓
CLIP ↑
Klein
46.06
26.15
29.16
UniPic2
65.02
28.92
28.41
ControlNet++ (FLUX.1)
29.22
23.61
29.22
InnerControl (FLUX.1)
27.91
22.54
28.88
Appendix
Table 15: Comparison with models supporting native I2I generation on the FLUX depth benchmark. Depth conditions and prompts are identical across methods.
Question
vs. ControlNet++
vs. Klein
vs. UniPic2
Prompt
fidelity (Q1)
94.9%
76.5%
84.1%
Control
alignment (Q2)
88.0%
72.6%
91.5%
Aesthetic
quality (Q3)
97.8%
31.1%
79.1%
Appendix
Table 16: User study on the FLUX depth setting. Values denote the percentage of cases where InnerControl is preferred over the competing model.
Figure 18: More visualizations for InnerControl (ours) method (depth maps)
Figure 19: More visualizations for InnerControl (ours) method (LineArt)
Figure 20: More visualizations for InnerControl (ours) method (HED)
Figure 21: More visualizations for InnerControl (ours) method based on SD3.5 model (depth maps)
Figure 22: More visualizations for InnerControl (ours) method based on FLUX model (depth maps)
Figure 23: FLUX depth-control qualitative comparison (moved from main paper). For each sample: input depth condition, ControlNet++ and InnerControl output.
Controllable text-to-image diffusion models can often follow the global layout of spatial conditions, yet still violate fine-grained structures such as object boundaries, thin contours, and medium/small conditioned regions. This limitation is especially problematic for VAE-based latent diffusion, where spatial compression can weaken high-frequency and low-area condition signals. We propose PixelControl, a pixel-space controllable diffusion framework for fine-grained condition fidelity. Built on a PixelDiT-style backbone, PixelControl avoids the latent bottleneck and introduces two complementary designs. First, Structure-Aware Control Injection derives a condition structure map and uses it to strengthen injected control residuals around spatially sensitive regions. Second, Multi-Scale Pyramid Cycle Loss verifies generated images against condition-derived structures across multiple resolutions, balancing global layout consistency with local boundary and detail accuracy. PixelControl supports depth, segmentation, edge, and their combinations through modality-specific control branches with lightweight gated fusion. Experiments across depth, segmentation, and edge control show that PixelControl improves structural fidelity and visual quality over existing controllable generation methods, with especially strong gains on boundaries and medium/small conditioned regions. The project page can be found at: https://linxin0.github.io/pixelcontrol_homepage/pixelcontrol-site/
Xin Lin, Haodong Li, Zhifei Zhang +5
University of California, San Diego · Adobe Research · Nanyang Technological University
Text-to-image diffusion models can generate visually stunning images, yet, controlling what appears and how it appears, remains surprisingly difficult, especially when operating solely within the constraints of the text-conditioning space. For example, changing a subject or adjusting an attribute often leads to unintended side effects, such as altered backgrounds or distorted details. This is because most existing text-based control methods treat the embedding space as Euclidean and apply simple linear transformations, which do not reflect how semantic concepts are actually organized. In this work, we take a step back and ask: what is the true geometry of these embeddings? We find that text encoder representations lie on a hypersphere, where concepts are not linear directions but structured, anisotropic distributions better captured by Kent distributions. Building on this insight, we propose HEART, a training-free framework that performs Kent-aware geodesic transformations directly on the hypersphere. By respecting the underlying geometry, HEART enables intuitive and precise edits, such as consistent subject replacement and fine-grained attribute control, while preserving the original scene. Importantly, HEART requires no finetuning, inversion, or optimization, and generalizes across diffusion model architectures. Our results show that a simple shift in perspective, from linear to spherical, can unlock fast, and controllable image generation.
Text-to-Image (T2I) diffusion/flow models have recently achieved remarkable progress in visual fidelity and text alignment. However, they remain limited when users need to precisely control image layouts, something that natural language alone cannot reliably express. Controllable generation methods augment the initial T2I model with additional conditions that more easily describe the scene. Prior works straightforwardly train the augmented network with the same loss as the initial network. Although natural at first glance, this can lead to very long training times in some cases before convergence. In this work, we revisit the training objective of controllable diffusion models through a detailed analysis of their denoising dynamics. We show that direct supervision on the clean target image, dubbed x0-supervision, or an equivalent re-weighting of the diffusion loss, yields faster convergence. Experiments on multiple control settings demonstrate that our formulation accelerates convergence by up to 2× according to our novel metric (mean Area Under the Convergence Curve - mAUCC), while also improving both visual quality and conditioning accuracy. Our code is available at https://github.com/CEA-LIST/x0-supervision
Amadou S. Sangare, Adrien Maglo, Mohamed Chaouch +1
Université Paris-Saclay, CEA, List, F-91120, Palaiseau, France