Recent advances in diffusion-based controllable visual generation have led to remarkable improvements in image quality. However, these powerful models are typically deployed on cloud servers due to their large computational demands, raising serious concerns about user data privacy. To enable secure and efficient on-device generation, we explore in this paper controllable diffusion models built upon linear attention architectures, which offer superior scalability and efficiency, even on edge devices. Yet, our experiments reveal that existing controllable generation frameworks, such as ControlNet and OminiControl, either lack the flexibility to support multiple heterogeneous condition types or suffer from slow convergence on such linear-attention models. To address these limitations, we propose a novel controllable diffusion framework tailored for linear attention backbones like SANA. The core of our method lies in a unified gated conditioning module working in a dual-path pipeline, which effectively integrates multi-type conditional inputs, such as spatially aligned and non-aligned cues. Extensive experiments on multiple tasks and benchmarks demonstrate that our approach achieves state-of-the-art controllable generation performance based on linear-attention models, surpassing existing methods in terms of fidelity and controllability.
Figures & tables
Figure 2: Convergence behavior. The introduction of gated modulation results in a substantially steeper decline in training loss compared with interaction mechanisms that rely solely on attention, indicating that the model learns conditional information more rapidly and more effectively, ultimately achieving a lower loss. This trend is also reflected in the CLIP-Image scores, where w/ gated method consistently outperforms the baseline from the earliest training stage and maintains a clear lead throughout.
Figure 3: Exploration of different methods and comparison of results. We adopt a shared module to handle the noisy latent and image condition, thus maximally retaining the original information flow of the model. The internal interaction enables flexible and general conditional control. Furthermore, a unified gating mechanism allows both non-spatial and spatial information to be effectively injected, improving performance while greatly accelerating the convergence of spatial tasks.
Methods
Base model
Parameters
Ratio
ControlNet
SD1.5 / 860M
361M
∼ 42%
T2I-Adapter
77M
∼ 9.0%
IP-Adapter
449M
∼ 52.2%
ControlNet
SANA / 1.6B
590M
∼ 36.9%
IP-Adapter
33.7M
∼ 2.11%
Lora interaction + Ours
SANA / 1.6B
18.9M + 0.09M w/ Gate
∼ 1.18% + ∼ 0.006%
Table 1: Additional parameters introduced by different image conditioning methods. For our method, results are reported with LoRA rank set to 16.
Figure 4: Details of the gated control method. Even a minimal token-wise gating mechanism could effectively filter tokens by strength, enabling adaptive token fusion with negligible cost.
Task
Methods / Setting
Base Model
Resolution
Controllability
Image Quality
Alignment
F1 ↑ / MSE ↓
FID ↓
SSIM ↑
MUSIQ ↑
CLIP Text ↑
CLIP Image ↑
Canny
ControlNet
SD1.5
5122
0.35
18.74
0.36
67.81
0.305
0.752
T2I Adapter
0.22
20.06
0.35
67.89
0.305
0.748
OminiControl
SANA
0.23
22.91
0.35
71.99
0.263
0.750
Ours
0.26
21.97
0.37
71.86
0.263
0.762
Depth
ControlNet
SD1.5
5122
923
23.03
0.34
70.73
0.308
0.726
Table 2: Quantitative comparison with baseline methods on five spatially aligned tasks. We evaluate methods based on Controllability (F1-Score for Canny, MSE for others), Image Quality (SSIM, FID and MUSIQ), and Alignment (CLIP Text and CLIP Image). For F1-Score (used in Canny to Image task), higher is better; for MSE, lower is better. Best results are shown in bold . The second-best results are highlighted with underlines.
Figure 5: Visualization results of our model on subject-driven tasks. The conditioning images are sourced from the DreamBooth dataset.
Method
FID ↓
SSIM ↑
CLIP Score ↑
w/o gating
22.6
0.36
0.74
w/o interaction
20.0
0.40
0.76
After-FFN
18.2
0.41
0.77
Elementwise
18.8
0.42
0.77
Input features
20.3
0.39
0.76
Ours
19.0
0.42
0.77
Table 3: Ablation studies on the mechanism of gated control for information integration. Best results are in bold .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 1: Gated control on the original OminiControl. Our approach enables the model to capture control signals much earlier during training, even when using softmax attention.
Figure 2: Robustness to sampling steps and guidance scale. Our model produces better and more stable outputs than OminiControl under both low-step inference and varying guidance scales.
Figure 3: Further visual comparisons on the subject-driven tasks. Our approach preserves object-specific features with greater fidelity, while simultaneously adapting the environment according to the provided editing prompt in a natural and flexible manner.
Method
Identity
Material
Color
Natural
Modification
Average
preservation
quality
fidelity
appearance
accuracy
score
IP-Adapter (SANA)
24.8
30.4
37.2
56.1
44.8
38.7
Ours
52.9
63.5
58.4
72.4
55.6
60.6
Appendix
Table 1: Quantitative evaluation results (in percentage) across different evaluation criteria. Higher values indicate better performance.
Figure 4: Intuitive analysis of the ablation on gated application through visualizations. Without gating, the model’s ability to leverage conditional information is substantially impaired, which may lead to outputs that do not comply with the condition or generate unreasonable artifacts. Moreover, using input features from different sources to generate the gate scores can affect global aspects of the image, such as style and color.
Figure 5: Comparison between our approach and SANA-based ControlNet. Our method more accurately follows the conditions and can generate more natural and realistic colorization.
Figure 6: Examples of image editing and multi-condition control. Our model can acquire basic editing capability within limited training steps. Moreover, it is able to simultaneously incorporate multiple conditions.
Conditional diffusion models generate or infer data by steering iterative denoising with observations, labels, text, physical constraints, or other inputs.
Diffusion models have become a leading approach in visual generative modeling, particularly effective in text-to-image and controllable generation. Their main drawback is slow inference as it requires many sampling steps. While recent distillation methods enable one-step text-to-image generation, extending these techniques to controllable generation is still challenging. Existing controllable models, such as ControlNet, rely on dual-branch design that make distillation difficult. In this work, we adopt a simpler strategy. Instead of adding extra conditional branches, we finetune a selected subset of layers in the original text-to-image diffusion model to directly enable controllable generation. This results in DeltaControl, a single-flow, multi-step diffusion model capable of supporting spatial-condition generation while matching the performance of dual-branch approaches. Building on this teacher model, we further apply step-distillation to obtain FlashControl, a one-step model capable of spatial controllable generation with a single forward pass.
Ngan Nguyen, Dung Nguyen, Quan Dao +4
Qualcomm AI Research · Rutgers University · Adelaide University +1
Controllable text-to-image diffusion models can often follow the global layout of spatial conditions, yet still violate fine-grained structures such as object boundaries, thin contours, and medium/small conditioned regions. This limitation is especially problematic for VAE-based latent diffusion, where spatial compression can weaken high-frequency and low-area condition signals. We propose PixelControl, a pixel-space controllable diffusion framework for fine-grained condition fidelity. Built on a PixelDiT-style backbone, PixelControl avoids the latent bottleneck and introduces two complementary designs. First, Structure-Aware Control Injection derives a condition structure map and uses it to strengthen injected control residuals around spatially sensitive regions. Second, Multi-Scale Pyramid Cycle Loss verifies generated images against condition-derived structures across multiple resolutions, balancing global layout consistency with local boundary and detail accuracy. PixelControl supports depth, segmentation, edge, and their combinations through modality-specific control branches with lightweight gated fusion. Experiments across depth, segmentation, and edge control show that PixelControl improves structural fidelity and visual quality over existing controllable generation methods, with especially strong gains on boundaries and medium/small conditioned regions. The project page can be found at: https://linxin0.github.io/pixelcontrol_homepage/pixelcontrol-site/
Xin Lin, Haodong Li, Zhifei Zhang +5
University of California, San Diego · Adobe Research · Nanyang Technological University
Text-to-Image (T2I) diffusion/flow models have recently achieved remarkable progress in visual fidelity and text alignment. However, they remain limited when users need to precisely control image layouts, something that natural language alone cannot reliably express. Controllable generation methods augment the initial T2I model with additional conditions that more easily describe the scene. Prior works straightforwardly train the augmented network with the same loss as the initial network. Although natural at first glance, this can lead to very long training times in some cases before convergence. In this work, we revisit the training objective of controllable diffusion models through a detailed analysis of their denoising dynamics. We show that direct supervision on the clean target image, dubbed x0-supervision, or an equivalent re-weighting of the diffusion loss, yields faster convergence. Experiments on multiple control settings demonstrate that our formulation accelerates convergence by up to 2× according to our novel metric (mean Area Under the Convergence Curve - mAUCC), while also improving both visual quality and conditioning accuracy. Our code is available at https://github.com/CEA-LIST/x0-supervision
Amadou S. Sangare, Adrien Maglo, Mohamed Chaouch +1
Université Paris-Saclay, CEA, List, F-91120, Palaiseau, France