Efficient Text-Guided Convolutional Adapter for the Diffusion Model

Aryan Das; Koushik Biswas; Swalpa Kumar Roy; Badri Narayana Patro; Vinay Kumar Verma

cs.CVFeb 16, 2026

Efficient Text-Guided Convolutional Adapter for the Diffusion Model

Authors: Aryan Das, Koushik Biswas, Swalpa Kumar Roy, Badri Narayana Patro, Vinay Kumar Verma

Organizations: VIT Bhopal · IIIT Delhi · Tezpur University Assam · IIT Kanpur

Abstract

We introduce the Nexus Adapters, novel text-guided efficient adapters to the diffusion-based framework for the Structure Preserving Conditional Generation (SPCG). Recently, structure-preserving methods have achieved promising results in conditional image generation by using a base model for prompt conditioning and an adapter for structure input, such as sketches or depth maps. These approaches are highly inefficient and sometimes require equal parameters in the adapter compared to the base architecture. It is not always possible to train the model since the diffusion model is itself costly, and doubling the parameter is highly inefficient. In these approaches, the adapter is not aware of the input prompt; therefore, it is optimal only for the structural input but not for the input prompt. To overcome the above challenges, we proposed two efficient adapters, Nexus Prime and Slim, which are guided by prompts and structural inputs. Each Nexus Block incorporates cross-attention mechanisms to enable rich multimodal conditioning. Therefore, the proposed adapter has a better understanding of the input prompt while preserving the structure. We conducted extensive experiments on the proposed models and demonstrated that the Nexus Prime adapter significantly enhances performance, requiring only 8M additional parameters compared to the baseline, T2I-Adapter. Furthermore, we also introduced a lightweight Nexus Slim adapter with 18M fewer parameters than the T2I-Adapter, which still achieved state-of-the-art results. Code: https://github.com/arya-domain/Nexus-Adapters

Figures & tables

Figure 1: Architecture of our method. The framework consists of two parts: (1) A frozen Stable Diffusion model. (2) A lightweight Nexus Adapter trained to inject external condition signals. A condition image is processed through the adapter to produce multiscale features, while a text prompt is encoded by a frozen CLIP Text Encoder. Both visual and textual features are injected into the UNet via additive fusion and cross-attention. Detailed designs of Prime and Slim Nexus Blocks are shown on the right.

Figure 2: Qualitative comparison across four conditioning types showing that Nexus Prime consistently yields the most accurate and semantically aligned outputs, while Nexus Slim offers competitive performance with reduced complexity.

Model	Venue	Flops (G)	Params. (M)
ControlNet [ 2 ]	ICCV 2023	116.61	361.28
T2I-Adapter [ 3 ]	AAAI 2024	29.97	77.37
ControlNet++ [ 4 ]	ECCV 2024	116.61	361.28
CtrLoRA [ 5 ]	ICLR 2025	116.61 $\xrightarrow{}$ 135.15	361.28 $\xrightarrow{}$ 37
UniCon [ 39 ]	ICLR 2025	111.62	150
Nexus Slim Adapter	–	23.77	59.29

Table 1: Model complexity (↓) measured by Flops and number of trainable parameters. Lower values indicate higher efficiency. The best (lowest) values are in bold , the second-best are underlined , and the third-best are wavy underlined .

Model	Canny	Depth	Sketch	Segmentation
ControlNet [ 2 ]	27.11	27.21	27.08	26.74
T2I-Adapter [ 3 ]	26.42	26.16	26.91	26.69
ControlNet++ [ 4 ]	26.79	27.26	27.24	27.19
CtrLoRA [ 5 ]	26.86	26.00	25.16	25.32
UniCon [ 39 ]	26.62	27.09	26.80	26.71
Nexus Slim Adapter	26.39	26.71	26.86	25.89

Table 2: Comparison of CLIP Scores (↑) across conditioning tasks. The highest values are shown in bold , the second-highest are underlined , and the third-highest are wavy underlined .

Model	Canny	Depth	Sketch	Segmentation
ControlNet [ 2 ]	23.14	25.98	25.23	27.36
T2I-Adapter [ 3 ]	24.94	26.33	26.81	27.96
ControlNet++ [ 4 ]	23.89	25.49	24.86	25.63
CtrLoRA [ 5 ]	22.89	25.95	26.32	26.03
UniCon [ 39 ]	23.16	25.60	24.91	26.72
Nexus Slim Adapter	23.67	25.30	26.40	27.01

Table 3: Comparison of FID scores (↓) across different conditioning tasks. The lowest values are shown in bold , the second-lowest are underlined , and the third-lowest are wavy underlined .

Figure 3: Qualitative comparison of image outputs with and without prompts, using only Sketch and Depth maps as conditional inputs.

Groups	FLOPs (G)	Params (M)	FID ↓	CLIP ↑
4	17.36	42.23	27.14	25.11
8	14.29	33.71	30.49	22.59

Table 4: Ablation on grouped convolution configuration in Nexus Slim Adapter Architecture (Canny condition).

Method	Model	Sketch		Depth
	Model	FID	CLIP	FID	CLIP
w/o P2A	ControlNet	33.04	19.13	32.32	19.39
	ControlNet++	32.56	19.24	31.98	19.31
	CtrLoRA	30.41	20.18	30.17	20.22
	UniCon	31.18	19.78	31.40	19.51
	Nexus Slim Adapter	29.34	20.71	28.33	20.58

Table 5: Ablation study comparing models under no-text-prompt settings using Sketch and Depth conditioning. FID (

\downarrow

) and CLIP (

\uparrow

) scores are reported. Two settings are evaluated: Without Prompt to Adapter (w/o P2A) and Without Prompt to Stable Diffusion and Adapter (w/o P2SD+A) .

Method	Model	Sketch		Depth
	Model	FID	CLIP	FID	CLIP
w/o FT	ControlNet [ 2 ]	28.18	24.03	28.81	24.68
	T2I-Adapter [ 3 ]	27.84	23.52	29.37	24.03
	ControlNet++ [ 4 ]	29.54	24.35	28.36	24.37
	CtrLoRA [ 5 ]	28.83	23.03	28.80	22.76
	UniCon [ 39 ]	27.73	24.41	28.79	24.13

Table 6: Evaluation of models on the CUB-200 dataset for fine-grained image generation under Sketch and Depth conditions. FID (

\downarrow

) and CLIP (

\uparrow

) scores are reported under two configurations: Without Finetuning (w/o FT) and With Finetuning (w FT) .

Model	Train Time (hrs)	Infer Time (ms/img)
ControlNet	$\approx$ 129	$\approx$ 38
T2I-Adapter	$\approx$ 33	$\approx$ 9
ControlNet++	$\approx$ 129	$\approx$ 38
CtrLoRA	$\approx$ 150	$\approx$ 45
UniCon	$\approx$ 124	$\approx$ 37
Nexus Slim Adapter	$\approx$ 26	$\approx$ 7

Table 7: Estimated training and inference time across models. Training time is computed for

200k

steps with batch size 2.

Blocks	Model	Depth		Sketch
		FID↓	CLIP↑	FID↓	CLIP↑
2	Nexus Slim Adapter	33.43	17.69	32.88	18.35
	Nexus Prime Adapter	28.52	21.98	30.50	20.49
3	Nexus Slim Adapter	30.38	20.75	29.95	20.86
	Nexus Prime Adapter	26.11	22.30	27.42	21.53
4	Nexus Slim Adapter	25.30	26.71	26.40	26.86

Table 8: Ablation on the number of adapter blocks. FID (↓) and CLIP Score (↑) for Depth and Sketch conditioning tasks. The 4-block configuration represents the full model.

Figure 4: Qualitative ablation on conflicting prompts and conditional images. The actual object in the image-conditioning input is highlighted with a blue background, and a conflicting prompt is provided as test input. We can observe that the model preserve the image-conditioned structural input while generating the object based on the text prompt.

Diffusion Model Adaptation

See all 25 papers

UNITY: Attention Flow Networks for Adaptive Conditioning in Diffusion
Jun 18, 2026Aryan Das, Koushik Biswas, Moloud Abdar +1Compositional Image Generation Conditional Diffusion Models
VIT Bhopal, India · IIIT Delhi, India · The University of Queensland, Australia +1
Nexus: Structured Synergy for Efficient Text-to-Image Generation using Rectified Flow Model
Aug 17, 2026Yizhao WangT2I Generation Diffusion Model Acceleration
School of Computer Science, Henan Institute of Science and Technology
RefAdapt-DiT: Adaptive Joint Attention for Reference-Conditioned Diffusion Transformers
Sep 26, 2026Jian Tang, Jiawei Fan, Qiannan Zhou +3Geometry-Conditioned Image Generation Attention Control in Diffusion Models
Platform and Content Group, Tencent

Jun 18, 2026J/K move · Enter open · S save

UNITY: Attention Flow Networks for Adaptive Conditioning in Diffusion

Aryan Das, Koushik Biswas, Moloud Abdar, Vinay Kumar Verma

VIT Bhopal, India · IIIT Delhi, India · The University of Queensland, Australia+1

Compositional Image Generation Conditional Diffusion Models Diffusion Model Inference Acceleration Diffusion Model Adaptation

Open paper AskPDF arXiv

We introduce UNITY, a Universal-to-Specialized adapter for efficient and scalable composite conditioning in diffusion based image generation. Unlike prior methods that train separate adapters for each conditioning modality, UNITY jointly learns shared semantics across multiple conditioning types and subsequently specializes without modifying the underlying architecture. The proposed two stage training paradigm consists of a Universal Stage that captures cross modal representations across all conditioning modalities using half of the total training steps, followed by a Specialization Stage that refines modality specific features using the remaining training budget. At the core of UNITY are the Morphable Attention Flow (MAF) Network and Morph Wrapper modules, which enable channel aware and spatially adaptive feature alignment through learnable flow fields and attention based fusion. This constant complexity formulation supports flexible operation under both single and composite conditioning settings while significantly reducing inference latency and memory consumption. Extensive experiments across multiple datasets demonstrate that UNITY achieves state of the art image fidelity while maintaining superior memory efficiency. Code: https://github.com/arya-domain/UNITY

Abstract

Diffusion Model Adaptation

Explore similar work

UNITY: Attention Flow Networks for Adaptive Conditioning in Diffusion

Nexus: Structured Synergy for Efficient Text-to-Image Generation using Rectified Flow Model

RefAdapt-DiT: Adaptive Joint Attention for Reference-Conditioned Diffusion Transformers