We introduce the Nexus Adapters, novel text-guided efficient adapters to the diffusion-based framework for the Structure Preserving Conditional Generation (SPCG). Recently, structure-preserving methods have achieved promising results in conditional image generation by using a base model for prompt conditioning and an adapter for structure input, such as sketches or depth maps. These approaches are highly inefficient and sometimes require equal parameters in the adapter compared to the base architecture. It is not always possible to train the model since the diffusion model is itself costly, and doubling the parameter is highly inefficient. In these approaches, the adapter is not aware of the input prompt; therefore, it is optimal only for the structural input but not for the input prompt. To overcome the above challenges, we proposed two efficient adapters, Nexus Prime and Slim, which are guided by prompts and structural inputs. Each Nexus Block incorporates cross-attention mechanisms to enable rich multimodal conditioning. Therefore, the proposed adapter has a better understanding of the input prompt while preserving the structure. We conducted extensive experiments on the proposed models and demonstrated that the Nexus Prime adapter significantly enhances performance, requiring only 8M additional parameters compared to the baseline, T2I-Adapter. Furthermore, we also introduced a lightweight Nexus Slim adapter with 18M fewer parameters than the T2I-Adapter, which still achieved state-of-the-art results. Code: https://github.com/arya-domain/Nexus-Adapters
Figures & tables
Figure 1: Architecture of our method. The framework consists of two parts: (1) A frozen Stable Diffusion model. (2) A lightweight Nexus Adapter trained to inject external condition signals. A condition image is processed through the adapter to produce multiscale features, while a text prompt is encoded by a frozen CLIP Text Encoder. Both visual and textual features are injected into the UNet via additive fusion and cross-attention. Detailed designs of Prime and Slim Nexus Blocks are shown on the right.
Figure 2: Qualitative comparison across four conditioning types showing that Nexus Prime consistently yields the most accurate and semantically aligned outputs, while Nexus Slim offers competitive performance with reduced complexity.
Model
Venue
Flops (G)
Params. (M)
ControlNet [ 2 ]
ICCV 2023
116.61
361.28
T2I-Adapter [ 3 ]
AAAI 2024
29.97
77.37
ControlNet++ [ 4 ]
ECCV 2024
116.61
361.28
CtrLoRA [ 5 ]
ICLR 2025
116.61 135.15
361.28 37
UniCon [ 39 ]
ICLR 2025
111.62
150
Nexus Slim Adapter
–
23.77
59.29
Table 1: Model complexity (↓) measured by Flops and number of trainable parameters. Lower values indicate higher efficiency. The best (lowest) values are in bold , the second-best are underlined , and the third-best are wavy underlined .
Model
Canny
Depth
Sketch
Segmentation
ControlNet [ 2 ]
27.11
27.21
27.08
26.74
T2I-Adapter [ 3 ]
26.42
26.16
26.91
26.69
ControlNet++ [ 4 ]
26.79
27.26
27.24
27.19
CtrLoRA [ 5 ]
26.86
26.00
25.16
25.32
UniCon [ 39 ]
26.62
27.09
26.80
26.71
Nexus Slim Adapter
26.39
26.71
26.86
25.89
Table 2: Comparison of CLIP Scores (↑) across conditioning tasks. The highest values are shown in bold , the second-highest are underlined , and the third-highest are wavy underlined .
Model
Canny
Depth
Sketch
Segmentation
ControlNet [ 2 ]
23.14
25.98
25.23
27.36
T2I-Adapter [ 3 ]
24.94
26.33
26.81
27.96
ControlNet++ [ 4 ]
23.89
25.49
24.86
25.63
CtrLoRA [ 5 ]
22.89
25.95
26.32
26.03
UniCon [ 39 ]
23.16
25.60
24.91
26.72
Nexus Slim Adapter
23.67
25.30
26.40
27.01
Table 3: Comparison of FID scores (↓) across different conditioning tasks. The lowest values are shown in bold , the second-lowest are underlined , and the third-lowest are wavy underlined .
Figure 3: Qualitative comparison of image outputs with and without prompts, using only Sketch and Depth maps as conditional inputs.
Groups
FLOPs (G)
Params (M)
FID ↓
CLIP ↑
4
17.36
42.23
27.14
25.11
8
14.29
33.71
30.49
22.59
Table 4: Ablation on grouped convolution configuration in Nexus Slim Adapter Architecture (Canny condition).
Method
Model
Sketch
Depth
FID
CLIP
FID
CLIP
w/o P2A
ControlNet
33.04
19.13
32.32
19.39
ControlNet++
32.56
19.24
31.98
19.31
CtrLoRA
30.41
20.18
30.17
20.22
UniCon
31.18
19.78
31.40
19.51
Nexus Slim Adapter
29.34
20.71
28.33
20.58
Table 5: Ablation study comparing models under no-text-prompt settings using Sketch and Depth conditioning. FID ( ↓ ) and CLIP ( ↑ ) scores are reported. Two settings are evaluated: Without Prompt to Adapter (w/o P2A) and Without Prompt to Stable Diffusion and Adapter (w/o P2SD+A) .
Method
Model
Sketch
Depth
FID
CLIP
FID
CLIP
w/o FT
ControlNet [ 2 ]
28.18
24.03
28.81
24.68
T2I-Adapter [ 3 ]
27.84
23.52
29.37
24.03
ControlNet++ [ 4 ]
29.54
24.35
28.36
24.37
CtrLoRA [ 5 ]
28.83
23.03
28.80
22.76
UniCon [ 39 ]
27.73
24.41
28.79
24.13
Table 6: Evaluation of models on the CUB-200 dataset for fine-grained image generation under Sketch and Depth conditions. FID ( ↓ ) and CLIP ( ↑ ) scores are reported under two configurations: Without Finetuning (w/o FT) and With Finetuning (w FT) .
Model
Train Time (hrs)
Infer Time (ms/img)
ControlNet
≈ 129
≈ 38
T2I-Adapter
≈ 33
≈ 9
ControlNet++
≈ 129
≈ 38
CtrLoRA
≈ 150
≈ 45
UniCon
≈ 124
≈ 37
Nexus Slim Adapter
≈ 26
≈ 7
Table 7: Estimated training and inference time across models. Training time is computed for 200k steps with batch size 2.
Blocks
Model
Depth
Sketch
FID↓
CLIP↑
FID↓
CLIP↑
2
Nexus Slim Adapter
33.43
17.69
32.88
18.35
Nexus Prime Adapter
28.52
21.98
30.50
20.49
3
Nexus Slim Adapter
30.38
20.75
29.95
20.86
Nexus Prime Adapter
26.11
22.30
27.42
21.53
4
Nexus Slim Adapter
25.30
26.71
26.40
26.86
Table 8: Ablation on the number of adapter blocks. FID (↓) and CLIP Score (↑) for Depth and Sketch conditioning tasks. The 4-block configuration represents the full model.
Figure 4: Qualitative ablation on conflicting prompts and conditional images. The actual object in the image-conditioning input is highlighted with a blue background, and a conflicting prompt is provided as test input. We can observe that the model preserve the image-conditioned structural input while generating the object based on the text prompt.
We introduce UNITY, a Universal-to-Specialized adapter for efficient and scalable composite conditioning in diffusion based image generation. Unlike prior methods that train separate adapters for each conditioning modality, UNITY jointly learns shared semantics across multiple conditioning types and subsequently specializes without modifying the underlying architecture. The proposed two stage training paradigm consists of a Universal Stage that captures cross modal representations across all conditioning modalities using half of the total training steps, followed by a Specialization Stage that refines modality specific features using the remaining training budget. At the core of UNITY are the Morphable Attention Flow (MAF) Network and Morph Wrapper modules, which enable channel aware and spatially adaptive feature alignment through learnable flow fields and attention based fusion. This constant complexity formulation supports flexible operation under both single and composite conditioning settings while significantly reducing inference latency and memory consumption. Extensive experiments across multiple datasets demonstrate that UNITY achieves state of the art image fidelity while maintaining superior memory efficiency. Code: https://github.com/arya-domain/UNITY
Aryan Das, Koushik Biswas, Moloud Abdar +1
VIT Bhopal, India · IIIT Delhi, India · The University of Queensland, Australia +1
Diffusion and flow matching models have made significant progress in text-to-image generation, yet high computation, quadratic complexity, and large memory footprint hinder high-resolution synthesis and edge deployment. We propose Nexus, which integrates sparse architecture, linear complexity, and low-bit quantization. It combines MoE feed-forward layers, gated DeltaNet attention, and per-expert low-bit training to reduce computation and memory. Their joint optimization allows Nexus to achieve generation quality comparable to mainstream models such as SDXL and SD3 while delivering markedly higher inference efficiency. Experiments on COCO and LAION validate its effectiveness.
Yizhao Wang
School of Computer Science, Henan Institute of Science and Technology
Diffusion Transformers (DiTs) have become the standard backbone for high-quality generative modeling, yet deploying them in conditional generation tasks remains computationally prohibitive because bidirectional joint attention repeatedly processes large reference streams. While existing optimization schemes mitigate generic temporal redundancy, they typically rely on coarse-grained static reuse and overlook the distinct dynamics of references and targets. Specifically, we observe that reference representations often evolve slowly along the generation trajectory, while the target often assigns little attention mass to them; reference drift and this target-to-reference exposure jointly shape how strongly stale reference states affect the target. To exploit these patterns, we introduce \RefAdapt, a training-free framework for adaptive control of joint attention between references and targets. Instead of rigid static strategies, \RefAdapt combines consecutive target-Q change with previously observed target-to-reference attention mass to control reference computation adaptively at block granularity. Under ultra-few-step settings, \RefAdapt enables speedups of up to 2.097× on 4-step MiniMax H3 and 3.54× on 8-step Qwen Image Edit, while maintaining comparable visual quality.