Organizations: University of California, Irvine, CA 92697, USA · University of Alabama at Birmingham, Birmingham, AL 35294, USA · Oak Ridge National Laboratory, Oak Ridge, TN 37831, USA
Existing image generation models face critical challenges regarding the trade-off between computation and fidelity. Specifically, models relying on a pretrained Variational Autoencoder (VAE) suffer from information loss, limited detail, and the inability to support end-to-end training. In contrast, models operating directly in the pixel space incur prohibitive computational cost. Although cascade models can mitigate computational cost, stage-wise separation prevents effective end-to-end optimization, hampers knowledge sharing, and often results in inaccurate distribution learning within each stage. To address these challenges, we introduce a unified multi-stage generative framework formulated under the stochastic interpolant formalism with our Conditional Dependent Coupling strategy as a Flow-Matching variant. It decomposes the generative process into interpolant trajectories at multiple stages, ensuring accurate distribution learning while enabling end-to-end optimization. Importantly, the entire process is modeled as a single unified Diffusion Transformer, eliminating the need for disjoint modules and also enabling knowledge sharing. Under explicitly stated assumptions, we provably reduce both transport cost and asymptotic inference time; we empirically validate the underlying NFE-transport-cost and per-evaluation-cost relations. Extensive experiments demonstrate that our method achieves competitive fidelity and substantially lower wall-clock cost in the low-NFE regime across multiple resolutions.
Figures & tables
Figure 1: Data Distribution Learning and Transition.
Method
Params
FID ↓
IS ↑
Prec. ↑
Rec. ↑
NFE
Speed
GAN
BigGAN [ 39 ]
112M
6.95
224.5
0.89
0.38
1
1.46
GigaGAN [ 40 ]
569M
3.45
225.5
0.84
0.61
1
1.32
StyleGAN [ 41 ]
166M
2.30
265.1
0.78
0.53
1
0.96
Diffusion / Flow
ADM [ 42 ]
554M
10.94
101.0
0.69
0.63
250×2
9.46
U - ViT [ 9 ]
287M
3.40
219.9
0.83
0.52
-
-
LDM - 4 - G [ 4 ]
400M
3.60
247.7
-
-
250×2
4.18
Table 1: Comprehensive comparison of generative models on ImageNet-1k/256. Missing values are denoted “-”. In the NFE field, “×2" indicates that CFG incurs an NFE of 2 per sampling step.
Figure 3Figure 4
Figure 6: FID under varying γ and σ .
Architecture Comparison
Fusion Ablation
Method
FID
Params
Time
Variant
FID
Unified
1.81
677M
76h
w/ Attn
1.81
Stage-spec.
2.68
130×3+458 M
78h
w/o Attn
1.84
Table 2: Ablation studies on Unified Architecture and Fusion Mechanism.
Interpolant Variant
Hierarchy Setting
Type
Definition
FID
Stages
Res. Path
FID
Linear
αt=1−τ;βt=τ
1.81
4
32…256
1.81
Trig.
αt=cos2πτ;βt=sin2πτ
2.11
2
64, 256
1.97
VP-SDE
αt=1−τ2;βt=τ
2.03
-
-
-
Table 3: Ablation studies on Interpolants (Left) and Hierarchy Stages (Right).
Figure 8: Visualization of each stage in CDC-FM, including the output at every stage, the details added at each stage, and the corresponding hierarchical structure of those details.
Figure 9: The per-stage FID under varying Scfg .
Resolution dk
322
642
1282
Time / function-eval (ms)
5.6
22.0
87.4
Normalized to d1
1.0×
≈4.0×
≈16.0×
Appendix
Table 4: Per-function-evaluation cost across resolutions (single forward pass of the unified DiT, in milliseconds, averaged over 200 calls on a single H100 GPU at batch size 1 ). The cost scales approximately linearly with the generation dimension dk , which empirically supports Assumption G.5 (G.4) used in the proof of Theorem M.1 .
(t1,t2)
(0.0,1.0)
(0.0,0.75)
(0.0,0.5)
(0.0,0.25)
(0.25,0.5)
Transport cost C
1.00
0.563
0.250
0.063
0.063
Required NFE
64
36
16
4
4
NFE / C
64
63.9
64.0
63.5
63.5
Appendix
Table 5: Empirical relationship between transportation cost C(t1,t2)=E[∣x1−x0∣2] and the NFE required to reach a fixed target FID ( 1.85 ) using SiT as the generative model on ImageNet-1k/256. For each (t1,t2) pair we draw 1,000 pairs of intermediate states from a linear interpolant and sweep the NFE upward until the target FID is hit. Lower transport cost consistently leads to proportionally fewer NFEs, and the trend is consistent across different intermediate states; this empirically supports Assumption G.6 (G.5).
Method
FID ↓
IS ↑
NFE
Speed (s/img) ↓
Single-stage Pixel ( K=1 )
1.82
305.8
250×2
8.73
CDC-FM (Ours)
1.81
304.3
16×4×2
5.18
Appendix
Table 6: Single-stage pixel baseline ( K=1 ) vs. CDC-FM under an identical backbone, training schedule, optimizer, and ODE solver. Removing only the conditional dependent coupling and re-collapsing the model to one stage costs ∼1.7× wall-clock at matched FID, isolating the efficiency gain attributable to the proposed coupling. Speed is reported in seconds per image at batch size 1 on a single H100 GPU.
Stages
Resolution Path
FID ↓
2
32→256
2.06
2
64→256
1.97
3
32→128→256
1.83
4
32→64→128→256
1.81
Appendix
Table 7: Impact of the stage count K and the resolution path on ImageNet-1k/256. Excessively large per-stage resolution ratios degrade FID (rows 1–2). With moderate ratios, fewer stages remain viable (rows 3–4): the gap between K=3 and K=4 is small, while K=4 obtains the best FID. Choices below the 322 floor yield insufficient structural/semantic content and were excluded.
N1
N2
N3
N4
Total NFE
FID ↓
16
16
16
16
64
1.81
8
16
16
16
56
1.83
4
16
16
16
52
1.85
16
8
16
16
56
1.85
16
4
16
16
52
1.87
16
16
8
16
56
1.88
Appendix
Table 8: Per-stage NFE allocation ablation on ImageNet-1k/256 with the 32→64→128→256 hierarchy. We hold three of the per-stage NFEs at 16 and reduce the fourth to {8,4} . Reducing the NFE of any single stage degrades FID, but the degradation is substantially larger at later, higher-resolution stages, indicating that solver-step budget should be biased toward high-resolution stages.
Model/Method
Params
FID ↓
IS ↑
BigGAN [ 39 ]
-
8.43
177.9
ADM [ 42 ]
554M
7.72
172.7
VDM++ [ 11 ]
2B
2.65
278.1
DiT - XL/2 [ 5 ]
675M
3.04
240.8
U - ViT [ 9 ]
501M
4.05
263.8
SiD2 [ 10 ]
2.0B
2.19
-
Appendix
Table 9: Comparison on ImageNet-1k/512 conditional generation.
Optimal-transport couplings have been shown to reduce training variance in unconditional flow models, but their role in conditional generation remains unclear. A natural approach constructs separate couplings for each condition, but this is impractical for large or continuous conditioning spaces found in modern image foundation models. We introduce Global Transport (GT), a global class-agnostic optimal-transport coupling, computed without class labels. GT can associate different conditions with different regions of the source noise, and consequently worsens performance without guidance. However, when combined with classifier-free guidance (CFG), GT consistently improves generation across domains, model scales, and sampling budgets. This reversal suggests that couplings for conditional flows should be evaluated both empirically and theoretically under the guided flow used at inference, rather than on unguided generation. We evaluate GT over both discrete class and continuous text conditioned image generation across model scales, and investigate how coupling choice alters guided trajectories. These results identify coupling design in the guided flow setting as a simple training time axis to improve performance without modifying existing architectures, samplers, or guidance mechanisms.
Katarina Petrović, Zander W. Blasingame, Danyal Rehman +5
AITHYRA · University of Oxford · Mila - Québec AI Institute +7
We introduce UNITY, a Universal-to-Specialized adapter for efficient and scalable composite conditioning in diffusion based image generation. Unlike prior methods that train separate adapters for each conditioning modality, UNITY jointly learns shared semantics across multiple conditioning types and subsequently specializes without modifying the underlying architecture. The proposed two stage training paradigm consists of a Universal Stage that captures cross modal representations across all conditioning modalities using half of the total training steps, followed by a Specialization Stage that refines modality specific features using the remaining training budget. At the core of UNITY are the Morphable Attention Flow (MAF) Network and Morph Wrapper modules, which enable channel aware and spatially adaptive feature alignment through learnable flow fields and attention based fusion. This constant complexity formulation supports flexible operation under both single and composite conditioning settings while significantly reducing inference latency and memory consumption. Extensive experiments across multiple datasets demonstrate that UNITY achieves state of the art image fidelity while maintaining superior memory efficiency. Code: https://github.com/arya-domain/UNITY
Aryan Das, Koushik Biswas, Moloud Abdar +1
VIT Bhopal, India · IIIT Delhi, India · The University of Queensland, Australia +1
Diffusion/flow models have emerged as the de facto paradigms for image generation, giving rise to a diverse family of modern generators with increasingly heterogeneous architectures and representations. Despite their remarkable generation capabilities, the cross-model alignment of latent representations and the consistency of their underlying denoising dynamics remain underexplored. In this work, we uncover an intriguing cross-model compatibility: independently trained generators exhibit similar latent spatial organization, while directly handing off intermediate states between unaligned models can still preserve coarse visual content. Motivated by these observations, we propose InstantFusion, a lightweight shared latent interface that connects the denoising trajectories of heterogeneous image generators. Specifically, to enable flexible cross-model state translation, model-specific Latent AutoEncoders (LAEs) are introduced to project intermediate states from different generators into a shared latent space and decode them into the latent space of a target generator. This shared space is constrained by two complementary objectives: aligning intermediate states derived from the same image at matched noise levels and synchronizing their corresponding local denoising velocities. Once trained, InstantFusion allows denoising to seamlessly transition across frozen generators without requiring a dedicated mapping for every model pair. The versatility of InstantFusion is demonstrated across acceleration, multi-preference composition, cross-architecture on-policy distillation, and cross-model composition, achieving competitive performance across these diverse settings.
Xuanlang Dai, Jiazi Bu, Yujie Zhou +5
Fudan University · Shanghai AI Laboratory · Shanghai Jiao Tong University +1