Gated Condition Injection without Multimodal Attention: Towards Controllable Linear-Attention Transformers

Yuhe Liu; Zhenxiong Tan; Yujia Hu; Songhua Liu; Xinchao Wang

cs.CVMar 29, 2026

Gated Condition Injection without Multimodal Attention: Towards Controllable Linear-Attention Transformers

Authors: Yuhe Liu, Zhenxiong Tan, Yujia Hu, Songhua Liu, Xinchao Wang

Organizations: National University of Singapore · Shanghai Jiao Tong University

Abstract

Recent advances in diffusion-based controllable visual generation have led to remarkable improvements in image quality. However, these powerful models are typically deployed on cloud servers due to their large computational demands, raising serious concerns about user data privacy. To enable secure and efficient on-device generation, we explore in this paper controllable diffusion models built upon linear attention architectures, which offer superior scalability and efficiency, even on edge devices. Yet, our experiments reveal that existing controllable generation frameworks, such as ControlNet and OminiControl, either lack the flexibility to support multiple heterogeneous condition types or suffer from slow convergence on such linear-attention models. To address these limitations, we propose a novel controllable diffusion framework tailored for linear attention backbones like SANA. The core of our method lies in a unified gated conditioning module working in a dual-path pipeline, which effectively integrates multi-type conditional inputs, such as spatially aligned and non-aligned cues. Extensive experiments on multiple tasks and benchmarks demonstrate that our approach achieves state-of-the-art controllable generation performance based on linear-attention models, surpassing existing methods in terms of fidelity and controllability.

Figures & tables

Figure 2: Convergence behavior. The introduction of gated modulation results in a substantially steeper decline in training loss compared with interaction mechanisms that rely solely on attention, indicating that the model learns conditional information more rapidly and more effectively, ultimately achieving a lower loss. This trend is also reflected in the CLIP-Image scores, where w/ gated method consistently outperforms the baseline from the earliest training stage and maintains a clear lead throughout.

Figure 3: Exploration of different methods and comparison of results. We adopt a shared module to handle the noisy latent and image condition, thus maximally retaining the original information flow of the model. The internal interaction enables flexible and general conditional control. Furthermore, a unified gating mechanism allows both non-spatial and spatial information to be effectively injected, improving performance while greatly accelerating the convergence of spatial tasks.

Methods	Base model	Parameters	Ratio
ControlNet	SD1.5 / 860M	361M	$\sim$ 42%
T2I-Adapter		77M	$\sim$ 9.0%
IP-Adapter		449M	$\sim$ 52.2%
ControlNet	SANA / 1.6B	590M	$\sim$ 36.9%
IP-Adapter	SANA / 1.6B	33.7M	$\sim$ 2.11%
Lora interaction + Ours	SANA / 1.6B	18.9M + 0.09M w/ Gate	$\sim$ 1.18% + $\sim$ 0.006%

Table 1: Additional parameters introduced by different image conditioning methods. For our method, results are reported with LoRA rank set to 16.

Figure 4: Details of the gated control method. Even a minimal token-wise gating mechanism could effectively filter tokens by strength, enabling adaptive token fusion with negligible cost.

Task	Methods / Setting	Base Model	Resolution	Controllability	Image Quality			Alignment
Task	Methods / Setting	Base Model	Resolution	F1 $\uparrow$ / MSE $\downarrow$	FID $\downarrow$	SSIM $\uparrow$	MUSIQ $\uparrow$	CLIP Text $\uparrow$	CLIP Image $\uparrow$
Canny	ControlNet	SD1.5	$512^{2}$	0.35	18.74	0.36	67.81	0.305	0.752
	T2I Adapter	SD1.5		0.22	20.06	0.35	67.89	0.305	0.748
	OminiControl	SANA		0.23	22.91	0.35	71.99	0.263	0.750
	Ours	SANA		0.26	21.97	0.37	71.86	0.263	0.762
Depth	ControlNet	SD1.5	$512^{2}$	923	23.03	0.34	70.73	0.308	0.726

Table 2: Quantitative comparison with baseline methods on five spatially aligned tasks. We evaluate methods based on Controllability (F1-Score for Canny, MSE for others), Image Quality (SSIM, FID and MUSIQ), and Alignment (CLIP Text and CLIP Image). For F1-Score (used in Canny to Image task), higher is better; for MSE, lower is better. Best results are shown in bold . The second-best results are highlighted with underlines.

Figure 5: Visualization results of our model on subject-driven tasks. The conditioning images are sourced from the DreamBooth dataset.

Method	FID $\downarrow$	SSIM $\uparrow$	CLIP Score $\uparrow$
w/o gating	22.6	0.36	0.74
w/o interaction	20.0	0.40	0.76
After-FFN	18.2	0.41	0.77
Elementwise	18.8	0.42	0.77
Input features	20.3	0.39	0.76
Ours	19.0	0.42	0.77

Table 3: Ablation studies on the mechanism of gated control for information integration. Best results are in bold .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Gated control on the original OminiControl. Our approach enables the model to capture control signals much earlier during training, even when using softmax attention.

Figure 2: Robustness to sampling steps and guidance scale. Our model produces better and more stable outputs than OminiControl under both low-step inference and varying guidance scales.

Figure 3: Further visual comparisons on the subject-driven tasks. Our approach preserves object-specific features with greater fidelity, while simultaneously adapting the environment according to the provided editing prompt in a natural and flexible manner.

Method	Identity	Material	Color	Natural	Modification	Average
	preservation	quality	fidelity	appearance	accuracy	score
IP-Adapter (SANA)	24.8	30.4	37.2	56.1	44.8	38.7
Ours	52.9	63.5	58.4	72.4	55.6	60.6

Appendix

Table 1: Quantitative evaluation results (in percentage) across different evaluation criteria. Higher values indicate better performance.

Figure 4: Intuitive analysis of the ablation on gated application through visualizations. Without gating, the model’s ability to leverage conditional information is substantially impaired, which may lead to outputs that do not comply with the condition or generate unreasonable artifacts. Moreover, using input features from different sources to generate the gate scores can affect global aspects of the image, such as style and color.

Figure 5: Comparison between our approach and SANA-based ControlNet. Our method more accurately follows the conditions and can generate more natural and realistic colorization.

Figure 6: Examples of image editing and multi-condition control. Our model can acquire basic editing capability within limited training steps. Moreover, it is able to simultaneously incorporate multiple conditions.

Conditional Diffusion Models

Conditional diffusion models generate or infer data by steering iterative denoising with observations, labels, text, physical constraints, or other inputs.

See all 277 papers

FlashControl: One-Step Controllable Generator via Distillation-Friendly Single-Stream Teachers
Oct 15, 2026Ngan Nguyen, Dung Nguyen, Quan Dao +4Controllable Image Generation Conditional Image Generation
Qualcomm AI Research · Rutgers University · Adelaide University +1
PixelControl: Fine-Grained Condition Fidelity in Text-to-Image Diffusion
Aug 16, 2026Xin Lin, Haodong Li, Zhifei Zhang +5Controllable Image Generation T2I Generation
University of California, San Diego · Adobe Research · Nanyang Technological University
Improving Controllable Generation: Faster Training and Better Performance via $x_0$ -Supervision
Apr 7, 2026Amadou S. Sangare, Adrien Maglo, Mohamed Chaouch +1Controllable Image Generation Conditional Diffusion Models
Université Paris-Saclay, CEA, List, F-91120, Palaiseau, France

Oct 15, 2026J/K move · Enter open · S save

FlashControl: One-Step Controllable Generator via Distillation-Friendly Single-Stream Teachers

Ngan Nguyen, Dung Nguyen, Quan Dao, Dimitris N. Metaxas+3

Qualcomm AI Research · Rutgers University · Adelaide University+1

Controllable Image Generation Conditional Image Generation One-Step Generative Modeling Diffusion Model Fine-Tuning Diffusion Model Distillation

Open paper AskPDF arXiv

Diffusion models have become a leading approach in visual generative modeling, particularly effective in text-to-image and controllable generation. Their main drawback is slow inference as it requires many sampling steps. While recent distillation methods enable one-step text-to-image generation, extending these techniques to controllable generation is still challenging. Existing controllable models, such as ControlNet, rely on dual-branch design that make distillation difficult. In this work, we adopt a simpler strategy. Instead of adding extra conditional branches, we finetune a selected subset of layers in the original text-to-image diffusion model to directly enable controllable generation. This results in DeltaControl, a single-flow, multi-step diffusion model capable of supporting spatial-condition generation while matching the performance of dual-branch approaches. Building on this teacher model, we further apply step-distillation to obtain FlashControl, a one-step model capable of spatial controllable generation with a single forward pass.

Abstract

Conditional Diffusion Models

Explore similar work

FlashControl: One-Step Controllable Generator via Distillation-Friendly Single-Stream Teachers

PixelControl: Fine-Grained Condition Fidelity in Text-to-Image Diffusion

Improving Controllable Generation: Faster Training and Better Performance via x0x_0x0​-Supervision

Improving Controllable Generation: Faster Training and Better Performance via $x_0$ -Supervision