InstantFusion: A Shared Latent Interface for Heterogeneous Image Generators

Xuanlang Dai; Jiazi Bu; Yujie Zhou; Yuhong Liu; Beichen Zhang; Penghui Yang; Ziyu Liu; Yuhang Zang

cs.CVOct 9, 2026

InstantFusion: A Shared Latent Interface for Heterogeneous Image Generators

Authors: Xuanlang Dai, Jiazi Bu, Yujie Zhou, Yuhong Liu, Beichen Zhang, Penghui Yang, Ziyu Liu, Yuhang Zang

Organizations: Fudan University · Shanghai AI Laboratory · Shanghai Jiao Tong University · The Chinese University of Hong Kong

Abstract

Diffusion/flow models have emerged as the de facto paradigms for image generation, giving rise to a diverse family of modern generators with increasingly heterogeneous architectures and representations. Despite their remarkable generation capabilities, the cross-model alignment of latent representations and the consistency of their underlying denoising dynamics remain underexplored. In this work, we uncover an intriguing cross-model compatibility: independently trained generators exhibit similar latent spatial organization, while directly handing off intermediate states between unaligned models can still preserve coarse visual content. Motivated by these observations, we propose InstantFusion, a lightweight shared latent interface that connects the denoising trajectories of heterogeneous image generators. Specifically, to enable flexible cross-model state translation, model-specific Latent AutoEncoders (LAEs) are introduced to project intermediate states from different generators into a shared latent space and decode them into the latent space of a target generator. This shared space is constrained by two complementary objectives: aligning intermediate states derived from the same image at matched noise levels and synchronizing their corresponding local denoising velocities. Once trained, InstantFusion allows denoising to seamlessly transition across frozen generators without requiring a dedicated mapping for every model pair. The versatility of InstantFusion is demonstrated across acceleration, multi-preference composition, cross-architecture on-policy distillation, and cross-model composition, achieving competitive performance across these diverse settings.

Figures & tables

Figure 1: Overview of InstantFusion. During training, model-specific LAEs map the intermediate states of frozen generators into a shared representation space, where both latent states and denoising dynamics are aligned. During inference, a sender state is translated through the shared space into the receiver’s latent space, and an adaptive criterion determines when the receiver takes over denoising.

Figure 2: Cross-model compatibility. (a) Intermediate states from Qwen-Image, SD3, and FLUX.1 exhibit similar spatial self-similarity patterns. (b) Direct handoff between related models preserves generation quality, while cross-family handoff retains the main subject and coarse layout. (c) These observations motivate a lightweight interface for connecting heterogeneous generators.

Figure 3: Architecture of the LAE. (a) $\sigma$ denotes the scalar noise level associated with denoising step $t_{k}$ . It is spatially broadcast and concatenated with the intermediate state $\mathbf{x}_{t_{k}}^{A}$ . Convolutional layers and time-conditioned residual blocks then map the input to its encoded representation $\mathbf{z}_{t_{k}}^{A}$ , with a skip connection preserving the original state. (b) Details of the time-conditioned resblock: Each residual block embeds $\sigma$ and predicts scale and shift parameters that modulate group-normalized features, conditioning the transformation on the current denoising stage.

Figure 4: Effect of denoising-dynamics alignment. Training without $\mathcal{L}_{\mathrm{dyn}}$ establishes correspondence between intermediate states $\mathbf{x}$ , but leaves their associated denoising velocities $\mathbf{v}$ poorly aligned. Adding $\mathcal{L}_{\mathrm{dyn}}$ improves velocity alignment. Each line in the UMAP visualization connects the $\mathbf{v}$ feature points produced by different models when conditioned on the same prompt.

Figure 5: Training dynamics of the model-specific LAEs. The total loss and the three component losses ( $\mathcal{L}_{\mathrm{rec}}$ , $\mathcal{L}_{\mathrm{align}}$ , and $\mathcal{L}_{\mathrm{dyn}}$ ) decrease steadily and converge during training.

Figure 6: Qualitative comparison of Qwen-Image, FLUX.1, and SD3 outputs after 1, 5, and 10 repeated latent round trips. The original outputs before any round trip serve as baselines.

Figure 7: Qualitative results of cross-model latent transfer among heterogeneous generators.

Table 1: Evaluation of the shared latent interface. (a) Reconstruction fidelity across repeated latent round-trips. (b) Quantitative generation performance under cross-architecture on-policy distillation. (c) Reconstruction fidelity degradation upon removing the reconstruction loss

\mathcal{L}_{\mathrm{rec}}

$\blacktriangledown$ Stronger Model
Models	New Model	GenEval-v2	CVTG-2K	Clip	OneIG-Bench
Models	New Model	GenEval-v2	CVTG-2K	Clip	Alignment	Text	Diversity	Style	Reasoning	Avg.
Qwen-Image [ 34 ]	–	41.73	0.6699	25.46	0.8304	0.9138	0.2091	0.2230	0.2486	0.4849
$\blacktriangledown$ Generation Baselines
SD3 [ 9 ]	–	22.14	0.3682	28.65	0.7934	0.4161	0.2327	0.3596	0.2770	0.4157
FLUX.1 [ 4 ]	–	22.86	0.5180	27.75	0.7894	0.5736	0.2136	0.3663	0.2653	0.4416

Table 2: Comparison of base models and cross-model combinations, demonstrating that guidance from a small set of high-quality generators can improve the generation quality of weaker models.

$\blacktriangledown$ Generation Baselines
Models	New model	Time $\downarrow$	GenEval -v2 $\uparrow$	Clip $\uparrow$	Aesthetic $\uparrow$	OneIG-Bench $\uparrow$
Models	New model	Time $\downarrow$	GenEval -v2 $\uparrow$	Clip $\uparrow$	Aesthetic $\uparrow$	Alignment	Text	Diversity	Style	Reasoning	Avg.
Qwen-Image [ 34 ]	–	18.84	41.73	25.46	5.9797	0.8304	0.9138	0.2091	0.2230	0.2486	0.4849
Qwen-Few-Step	–	0.893	32.94	27.48	6.1430	0.8804	0.9752	0.0885	0.3234	0.3069	0.5148
FLUX.2-Klein [ 4 ]	–	0.516	25.43	27.71	5.9017	0.5718	0.1863	0.3930	0.4059	0.2415	0.3597
Z-Image-Turbo [ 6 ]	–	3.521	39.21	25.02	4.9892	0.8658	0.9788	0.2193	0.4573	0.3137	0.5669

Table 3: Quantitative comparison of generation quality and inference efficiency. Performance benchmarks across baselines, existing acceleration techniques, and InstantFusion. Furthermore, InstantFusion exhibits orthogonal compatibility when integrated with training-free acceleration methods like DiCache, yielding cumulative speedups without compromising generation fidelity.

$\blacktriangledown$ Reward-Specialized GRPO Baselines
Models	New Model	GenEval-v2	HPSv2					HPSv3
Models	New Model	GenEval-v2	Anime	Concept	Paint.	Photo	Avg.	Anime	Concept	Paint.	Photo	Avg.
GRPO baseline (HPSv2)	–	28.83	32.21	31.67	31.91	28.95	31.18	12.14	12.88	13.30	9.94	12.07
GRPO baseline (HPSv3)	–	26.53	31.42	30.57	30.93	27.63	30.14	12.31	13.06	13.48	9.73	12.15
GRPO baseline (GenEval-v2)	–	35.89	28.55	27.28	27.47	25.47	27.19	10.98	11.30	11.71	7.69	10.42
$\blacktriangledown$ Direct V Fusion

Table 4: Evaluation results of reward-specialized GRPO models and model combination strategies.

Table 12

Figure 8: Evolution of generated images over training steps during cross-model OPD.

Figure 9: Comparison between the predicted clean images at adaptively selected handoff steps and the corresponding fully denoised outputs, demonstrating preservation of global structure.

Rollout Schedule	GenEval-v2	HPSv2
Rollout Schedule	GenEval-v2	Anime	Concept	Paint.	Photo	Avg.
SD3 $\rightarrow$ FLUX.1 $\rightarrow$ Qwen	19.99	27.04	25.41	25.46	24.69	25.65
SD3 $\rightarrow$ Qwen $\rightarrow$ FLUX.1	24.93	24.35	22.80	22.72	21.91	22.95
FLUX.1 $\rightarrow$ SD3 $\rightarrow$ Qwen	25.22	27.86	25.41	25.30	24.87	25.86
FLUX.1 $\rightarrow$ Qwen $\rightarrow$ SD3	27.96	25.68	22.85	22.96	22.68	23.54
Qwen $\rightarrow$ SD3 $\rightarrow$ FLUX.1	30.76	26.60	24.14	24.23	24.06	24.76

Table 7: Robustness of the shared latent interface across multi-stage rollout schedules. We evaluate different execution orders across three heterogeneous generators (SD3, FLUX.1, and Qwen-Image). The consistent performance across arbitrary transition permutations confirms the stability and invariance of the joint shared latent space under cascading cross-model transfers.

Metric	Individual Generators			Hybrid Generation
Metric	SD3	FLUX.1	Qwen	FLUX.1 $\rightarrow$ SD3	SD3 $\rightarrow$ Qwen	FLUX.1 $\rightarrow$ Qwen	FLUX.1 $\rightarrow$ SD3 $\rightarrow$ Qwen
Latency (s/image)	$3.61$	$10.70$	$20.82$	$7.16$	$12.21$	$15.7$	$11.53$
Theoretical NFE	$50$	$50$	$50$	$25+25$	$25+25$	$25+25$	$17+17+16$

Table 8: End-to-end inference latency and theoretical NFEs. Latency is measured at

1024\times 1024

resolution, averaged over 20 runs, and reported in seconds per image. For hybrid configurations, the NFE entries show how the 50 denoising evaluations are allocated across models.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Table 17

Shared Space	Reconstructed Model	Switch	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
TaylorSeer ( $\mathcal{N}=6$ )	FLUX.1	–	28.945	0.6556	0.4020
SD3–Qwen	Qwen	1	32.418	0.9433	0.03811
		5	24.653	0.8719	0.10096
		10	22.255	0.8253	0.13709
	SD3	1	30.331	0.9401	0.04244
		5	21.301	0.8143	0.15164

Appendix

Table 11: Reconstruction fidelity under different shared-space configurations.

Figure 10: Cross-model state handoffs at different denoising timesteps (Qwen-Image, Flux.1).

Figure 11: Cross-model state handoffs at different denoising timesteps (Qwen-Image, SD3).

Figure 12: Qualitative comparison of shared latent reconstruction (SD3).

Figure 13: Qualitative comparison of shared latent reconstruction (FLUX.1).

Figure 14: Qualitative comparison of shared latent reconstruction (Qwen-Image).

Figure 15: Cross-architecture state handoffs initiated from Qwen-Image to SD3.

Figure 16: Cross-architecture state handoffs initiated from Qwen-Image to FLUX.1.

Figure 17: Qualitative results under generation acceleration.

Figure 18: Multi-preference composition cases combining GenEval-v2 and HPSv2.

Figure 19: Multi-preference composition cases combining HPSv2 and HPSv3.

Explore similar work

CrossFlow: One-Step Generation Across Latent and Pixel Spaces
Jun 18, 2026Xiyuan Wang, Xiao Zhang, Yang Li +4Flow Matching Latent Diffusion Models
Institute for Artificial Intelligence, Peking University · Tencent · Fudan University
Poly-OPD: Heterogeneous Multi-Teacher On-Policy Distillation for Capability-Selectable Flow Models
Aug 5, 2026Siming Fu, Haojun Xu, Ruizhe He +9Diffusion Model Distillation Multi-Teacher Knowledge Distillation
1Joy Future Academy · 2Zhejiang University · 3Beihang University
UNITY: Attention Flow Networks for Adaptive Conditioning in Diffusion
Jun 18, 2026Aryan Das, Koushik Biswas, Moloud Abdar +1Compositional Image Generation Conditional Diffusion Models
VIT Bhopal, India · IIIT Delhi, India · The University of Queensland, Australia +1

Jun 18, 2026J/K move · Enter open · S save

CrossFlow: One-Step Generation Across Latent and Pixel Spaces

Xiyuan Wang, Xiao Zhang, Yang Li, Ruoxi Jiang+3

Institute for Artificial Intelligence, Peking University · Tencent · Fudan University

Flow Matching Latent Diffusion Models One-Step Generative Modeling Flow-Based Generative Modeling

Open paper AskPDF arXiv

Most diffusion and flow-matching generators define the prior, probability path, and prediction target in the same representation space. Latent diffusion improves efficiency by moving this path into an autoencoder latent space, but the final sample is still produced by a separately trained decoder. This separation creates a mismatch: the generator is optimized for latent-space prediction, while final quality depends on how the decoder handles generated latents that may differ from clean encoder outputs. We introduce CrossFlow, a cross-space flow formulation that maps noisy latent inputs directly to pixel-space images. The key technical step is a velocity-free one-step objective: the latent trajectory defines the training path, but the supervised prediction is an image rather than a latent displacement. This lets one model act both as a one-step latent-to-pixel generator and as a decoder replacement for latent diffusion pipelines. On class-conditional ImageNet-1k at

256\times256

, CrossFlow-XL achieves 1.62 FID with one function evaluation. Ablations show that the latent encoder and pixel-space perceptual and adversarial losses are important for fidelity. These results indicate that cross-space flow objectives can combine the efficiency of latent representations with direct pixel-space supervision, without requiring a separate decoder at inference.

Abstract

Explore similar work

CrossFlow: One-Step Generation Across Latent and Pixel Spaces

Poly-OPD: Heterogeneous Multi-Teacher On-Policy Distillation for Capability-Selectable Flow Models

UNITY: Attention Flow Networks for Adaptive Conditioning in Diffusion