Controllable text-to-image diffusion models can often follow the global layout of spatial conditions, yet still violate fine-grained structures such as object boundaries, thin contours, and medium/small conditioned regions. This limitation is especially problematic for VAE-based latent diffusion, where spatial compression can weaken high-frequency and low-area condition signals. We propose PixelControl, a pixel-space controllable diffusion framework for fine-grained condition fidelity. Built on a PixelDiT-style backbone, PixelControl avoids the latent bottleneck and introduces two complementary designs. First, Structure-Aware Control Injection derives a condition structure map and uses it to strengthen injected control residuals around spatially sensitive regions. Second, Multi-Scale Pyramid Cycle Loss verifies generated images against condition-derived structures across multiple resolutions, balancing global layout consistency with local boundary and detail accuracy. PixelControl supports depth, segmentation, edge, and their combinations through modality-specific control branches with lightweight gated fusion. Experiments across depth, segmentation, and edge control show that PixelControl improves structural fidelity and visual quality over existing controllable generation methods, with especially strong gains on boundaries and medium/small conditioned regions. The project page can be found at: https://linxin0.github.io/pixelcontrol_homepage/pixelcontrol-site/
Figures & tables
Figure 2: Overview of the proposed PixelControl. The condition is injected into the backbone through structure-aware residual modulation, and the model is trained with flow matching and multi-scale cycle supervision to preserve both global layout and fine condition details.
Figure 3: Effect of SACI on injected control features. From left to right, we show the original image, the input condition, the Sobel structure map derived from the condition, the injected feature without SACI, and the structure-modulated feature with SACI. SACI strengthens the injected control response around boundaries and local discontinuities.
Method
SI-RMSE ↓
MAE ↓
AbsRel ↓
δ1↑
δ2↑
δ3↑
Pearson ↑
AnyControl
0.1277
0.0908
0.2715
0.6241
0.8215
0.9112
0.8310
ControlNet
0.2103
0.1680
0.4514
0.3987
0.6241
0.7752
0.5593
Ctrl-Adapter
0.2300
0.1869
0.4842
0.3490
0.5780
0.7482
0.4762
OminiControl
0.1774
0.1317
0.3658
0.5067
0.7086
0.8291
0.6709
PixelPonder
0.0971
0.0625
0.1949
0.7374
0.8956
0.9507
0.9049
UniControlNet
0.1764
0.1340
0.3719
0.4903
0.7011
0.8257
0.6713
Table 1: Depth-control structural fidelity. Best results are bolded.
Figure 4: Single-condition qualitative comparison across depth, edge, and segmentation controls. For each example, we show the input condition, generated images from all methods, and the re-estimated maps.
Method
mIoU ↑
Pixel Acc. ↑
mAcc ↑
Boundary-F1 ↑
AnyControl
0.5220
0.7204
0.6006
0.5061
ControlNet
0.3496
0.6955
0.5065
0.3558
Ctrl-Adapter
0.3544
0.6797
0.4747
0.3488
UniControlNet
0.2922
0.6664
0.4184
0.3070
Ours
0.5943
0.8352
0.6577
0.6973
Table 2: Segmentation-control structural fidelity. Best results are bolded.
Method
Edge F1 ↑
Recall ↑
Chamfer ↓
Soft-IoU ↑
AnyControl
0.4127
0.4195
13.9251
0.0595
ControlNet
0.4927
0.5051
14.1997
0.1256
Ctrl-Adapter
0.3472
0.2633
19.6873
0.0355
OminiControl
0.4850
0.3755
17.8455
0.1734
PixelPonder
0.3211
0.2281
33.3204
0.1025
UniControlNet
0.5517
0.6163
11.5315
0.1341
Table 3: Edge-control structural fidelity. Best results are bolded.
Depth
Segmentation
Edge
Method
FID ↓
CLIP-Img ↑
LPIPS ↓
FID ↓
CLIP-Img ↑
LPIPS ↓
FID ↓
CLIP-Img ↑
LPIPS ↓
AnyControl
45.8426
0.7280
0.6776
47.2984
0.7176
0.7049
48.3428
0.7144
0.6385
ControlNet
62.9528
0.6850
0.7073
46.5417
0.7158
0.6910
46.2963
0.7211
0.6052
Ctrl-Adapter
52.7158
0.6898
0.6920
41.0080
0.7274
0.6493
48.3437
0.7039
0.6147
OminiControl
39.6671
0.7572
0.6283
–
–
–
37.9785
0.7989
0.5321
PixelPonder
34.9906
0.7755
0.5649
–
–
–
54.5853
0.7076
0.6165
Table 4: Image quality and image consistency across depth, segmentation, and edge control. Best results are bolded.
Method
Depth AbsRel ↓
Edge Chamfer ↓
Seg. Boundary-F1 ↑
AnyControl
0.3430
6.315
0.4728
ControlNet
1.4900
6.398
0.2985
Ctrl-Adapter
1.5326
8.741
0.2986
OminiControl
0.4757
7.513
–
PixelPonder
0.2727
12.568
–
UniControlNet
0.4303
4.381
0.2531
Table 5: Non-large conditioned-object fidelity. We evaluate only YOLO-detected non-large objects that are explicitly grounded in the condition. Best results are bolded.
Figure 5: Non-large conditioned-region qualitative comparison. Red boxes highlight non-large objects or local structures specified by the condition. PixelControl better preserves these regions while maintaining the global structure.
Variant
Depth ↓
Seg. ↑
Edge ↓
Non-large Depth ↓
Non-large Seg. ↑
Non-large Edge ↓
Avg. FID ↓
PixelDiT baseline
0.2233
0.4824
12.851
0.3312
0.4218
6.957
35.82
+ SACI only
0.1814
0.6127
12.622
0.2184
0.5655
6.849
33.47
+ MPCL only
0.1666
0.6412
5.663
0.2019
0.5894
2.797
32.70
Ours
0.1434
0.6973
5.456
0.1419
0.6409
2.603
30.21
Table 6: Overall architecture ablation. The PixelDiT baseline uses naive condition injection and only the FM loss. SACI is applied only to depth and segmentation controls, so its gains mainly appear on depth and segmentation fidelity. Edge improvements are primarily brought by the pyramid cycle supervision.
SACI strength
Depth AbsRel ↓
Seg. BF1 ↑
Avg. FID ↓
No SACI ( α=0 )
0.2233
0.4824
35.82
α=0.5
0.2085
0.5564
34.77
α=1.0
0.1974
0.6211
33.92
α=2.0
0.1814
0.6127
33.47
α=4.0
0.1856
0.6041
33.72
Table 7: Ablation on SACI strength. The α controls how strongly structure-aware regions modulate the injected control residual.
MPCL setting
Depth AbsRel ↓
Non-large Edge ↓
Avg. FID ↓
No MPCL
0.1814
6.849
33.47
Low-res only (128, 64)
0.1586
4.724
32.25
High-res only (512)
0.1694
3.138
31.92
Full pyramid (512, 256, 128, 64)
0.1434
2.603
30.21
Table 8: Ablation on MPCL scales. Low-resolution scales emphasize global layout, high-resolution scales emphasize fine details, and the full pyramid provides the best balance between condition fidelity and visual quality.
Method
Depth AbsRel ↓
Edge Chamfer ↓
FID ↓
AnyControl
0.3154
14.82
49.19
OminiControl
0.3821
18.06
42.33
UniControlNet
0.3907
12.48
44.22
Ours
0.1822
6.42
33.27
Table 9: Depth+edge multi-condition control. We evaluate structural fidelity for both conditions and report FID for visual quality.
Figure 6: Depth+edge multi-condition qualitative comparison. Given both depth and edge conditions, PixelControl better preserves global geometry and local contours, producing outputs that satisfy both active controls more faithfully.
Text-to-Image (T2I) diffusion/flow models have recently achieved remarkable progress in visual fidelity and text alignment. However, they remain limited when users need to precisely control image layouts, something that natural language alone cannot reliably express. Controllable generation methods augment the initial T2I model with additional conditions that more easily describe the scene. Prior works straightforwardly train the augmented network with the same loss as the initial network. Although natural at first glance, this can lead to very long training times in some cases before convergence. In this work, we revisit the training objective of controllable diffusion models through a detailed analysis of their denoising dynamics. We show that direct supervision on the clean target image, dubbed x0-supervision, or an equivalent re-weighting of the diffusion loss, yields faster convergence. Experiments on multiple control settings demonstrate that our formulation accelerates convergence by up to 2× according to our novel metric (mean Area Under the Convergence Curve - mAUCC), while also improving both visual quality and conditioning accuracy. Our code is available at https://github.com/CEA-LIST/x0-supervision
Amadou S. Sangare, Adrien Maglo, Mohamed Chaouch +1
Université Paris-Saclay, CEA, List, F-91120, Palaiseau, France
Recent advances in text-to-image (T2I) generation have enabled controllable image synthesis by incorporating conditions beyond text. However, most existing diffusion-based methods are limited to a single type of control condition (e.g., bounding boxes or keypoints), which restricts their flexibility. To address this limitation, we propose MixDiffusion, a training-free diffusion framework for multi-condition T2I generation. MixDiffusion theoretically supports an arbitrary number of control conditions, including bounding boxes, keypoints, sketches, depth maps, reference images, and text, by collaboratively integrating multiple pre-trained uni-condition diffusion models. The key insight of the proposed approach is to derive the predicted noise distribution in each denoising step of the diffusion-based multi-condition image generation model from the predicted noise distributions of multiple diffusion-based uni-condition models with a derived integration formula, which is supported by rigorous theory proof. Owing to its training-free nature, MixDiffusion is easy to deploy and readily extensible to new control modalities.
Diffusion models have become a leading approach in visual generative modeling, particularly effective in text-to-image and controllable generation. Their main drawback is slow inference as it requires many sampling steps. While recent distillation methods enable one-step text-to-image generation, extending these techniques to controllable generation is still challenging. Existing controllable models, such as ControlNet, rely on dual-branch design that make distillation difficult. In this work, we adopt a simpler strategy. Instead of adding extra conditional branches, we finetune a selected subset of layers in the original text-to-image diffusion model to directly enable controllable generation. This results in DeltaControl, a single-flow, multi-step diffusion model capable of supporting spatial-condition generation while matching the performance of dual-branch approaches. Building on this teacher model, we further apply step-distillation to obtain FlashControl, a one-step model capable of spatial controllable generation with a single forward pass.
Ngan Nguyen, Dung Nguyen, Quan Dao +4
Qualcomm AI Research · Rutgers University · Adelaide University +1