UltraDiff: Transferring High-Fidelity Priors to Compressed Latent Spaces for High-Resolution Image Generation

Jingjing Ren; Haitian Zheng; Haoyu Chen; Connelly Barnes; Zhifei Zhang; Yuqian Zhou; Zhe Lin; Lei Zhu

cs.CVOct 15, 2026

UltraDiff: Transferring High-Fidelity Priors to Compressed Latent Spaces for High-Resolution Image Generation

Authors: Jingjing Ren, Haitian Zheng, Haoyu Chen, Connelly Barnes, Zhifei Zhang, Yuqian Zhou, Zhe Lin, Lei Zhu

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Adobe Research

Abstract

High-resolution synthesis with Latent Diffusion Models (LDMs) faces a fundamental trade-off between computational cost and generative fidelity. This trade off is largely governed by the latent compression factor f (e.g. f8 denotes f=8 and f32 f=32). While aggressive compression (f32) significantly improves efficiency, it suffers from severe information loss and ambiguity compared to standard f8 models. We propose UltraDiff, a framework that reformulates this challenge as joint latent diffusion learning followed by marginal score distillation. We introduce MixTeacher, a dual branch DiT that models the joint f8-f32 distribution, utilizing shared attention to resolve the ambiguity of the compressed space with high fidelity f8 priors. Crucially, we propose vector field distillation in the native f32 space. Theoretically, this compels the student to learn the conditional expectation of the teacher trajectory, effectively filtering out compression artifacts. Across the evaluated 1K to 4K settings, UltraDiff approaches f8 level quality while preserving f32 efficiency. With 20 sampling steps on a single A100 GPU, it generates a 2K image in 6.2 seconds over 40 times faster than our f8 baseline.

Figures & tables

Figure 1: Our UltraDiff generates high-quality images across a variety of scenes, showcasing rich details and diverse content. Please zoom in to view the details. Additional images are provided in the supplementary materials.

Figure 2: Effectiveness of Cross-Space Knowledge Transfer. GenEval scores during training. UltraDiff bridges the gap to the f8 teacher.

Figure 3: Overview of UltraDiff. (Left) The MixTeacher models the joint distribution of f8 and f32 latents via a dual-branch DiT with shared attention. It acts as an omniscient supervisor that sees both high-fidelity (f8) and high-efficiency (f32) views of the data. (Right) The Student is a standard f32 DiT trained via Marginal Score Distillation . It learns to approximate the teacher’s optimal denoising direction projected onto the f32 subspace with a flow-matching objective.

Method	CLIP $\uparrow$	IR $\uparrow$	GenEval $\uparrow$	Token
f32 from scratch	27.21	0.70	0.54	$\frac{H}{32}\times\frac{W}{32}$
f32 after f8 initialization	28.91	0.91	0.61	$\frac{H}{32}\times\frac{W}{32}$
UltraDiff f32 student	29.27	0.98	0.72	$\frac{H}{32}\times\frac{W}{32}$
f8 anchor	29.25	1.08	0.73	$\frac{H}{8}\times\frac{W}{8}$

Table 1: Controlled comparison under the same framework. All f32 variants use the same f32 VAE and DiT backbone. Initialization alone transfers part of the f8 prior, whereas marginal score distillation recovers most of the left gap.

Stage	Steps	Compute	Output
f32 transfer	120K	66.7 GPU-days	f32 initialization
MixTeacher training	5K	29.6 GPU-days	Joint f8–f32 teacher
Student distillation and scaling	50K + 10K	366.7 GPU-days	Standalone f32 student
UltraDiff adaptation total	185K	463.0 GPU-days	Final UltraDiff model

Table 2: Stage-wise training resources. Compute is aggregated over 64 NVIDIA A100 GPUs.

Panel of Figure 5 — Figure 5: Visual comparison of (a) the f8 base model, (b) f32 model distilled from an f8 teacher ( Ren et al., 2025a ) , (c) f32 model distilled from DINOv2 features via REPA ( Yu et al., 2024 ) , and (d) f32 model distilled from our MixTeacher.

Model	DPG $\uparrow$	GenEval $\uparrow$	HPSv3 $\uparrow$	FID $\downarrow$	Aesthetic $\uparrow$	CLIP $\uparrow$
No distillation + high-res	74.02	0.602	7.68	149.59	6.16	29.31
UltraDiff	85.40	0.680	9.39	146.60	6.34	30.14

Table 6: Controlled attribution of the high-resolution gains. Both models use the same f8-initialized f32 backbone and the same mixed-resolution scaling stage. DPG, GenEval, and HPSv3 are evaluated at 1K; FID, aesthetic score, and CLIP similarity are evaluated at 4K.

Figure 6: High-resolution image synthesis comparison at 2K and 4K. UltraDiff produces sharper details, cleaner textures, and stronger text–image alignment than competing high-resolution models.

Resolution	Method	Params(B)	Latent( $C\times H\times W$ )	Latency(s) / Step	FID $\downarrow$	pFID $\downarrow$	Aesthetic $\uparrow$	CLIP $\uparrow$
$2048^{2}$	Ultrapixel ( Ren et al., 2024 )	5.1	$16\times\frac{H}{32}\times\frac{W}{32}$	12 / 20	46.60	43.03	6.24	30.21
	Diffusion-4k ( Zhang et al., 2025 )	12	$16\times\frac{H}{42}\times\frac{W}{42}$	23 / 50	37.79	42.43	6.28	29.21
	Sana-2K ( Xie et al., 2024 )	1.6	$32\times\frac{H}{32}\times\frac{W}{32}$	5.8 / 20	45.99	42.99	6.18	30.13
	Hunyuan-2.1 ( Team, 2025 )	17	$64\times\frac{H}{32}\times\frac{W}{32}$	82 / 50	43.45	48.72	6.44	30.17
	Ours	2.0	$64\times\frac{H}{32}\times\frac{W}{32}$	6.2 / 20	39.52	42.17	6.40	30.53
$4096^{2}$	Ultrapixel ( Ren et al., 2024 )	5.1	$16\times\frac{H}{42}\times\frac{W}{42}$	73 / 20	155.17	57.14	5.91	27.39

Table 7: Comparison with other T2I models on higher resolution of

2048\times 2048

and

4096\times 4096

Method	Token	Params(B)	DPG $\uparrow$	GenEval $\uparrow$	HPSv3 $\uparrow$
SDXL ( Podell et al., 2023 )	$\frac{H}{8}\times\frac{W}{8}$	2.6	74.6	0.55	7.87
PlayGroundv2.5 ( Li et al., 2024 )	$\frac{H}{8}\times\frac{W}{8}$	2.6	75.5	0.56	9.90
PixArt $\Sigma$ ( Chen et al., 2024 )	$\frac{H}{16}\times\frac{W}{16}$	0.6	80.5	0.54	9.20
SD3-medium ( Esser et al., 2024 )	$\frac{H}{16}\times\frac{W}{16}$	2.0	84.1	0.62	9.08
FLUX-dev ( Black Forest Labs, 2024 )	$\frac{H}{16}\times\frac{W}{16}$	12	83.8	0.67	10.15
FLUX-schnell ( Black Forest Labs, 2024 )	$\frac{H}{16}\times\frac{W}{16}$	12	84.8	0.71	9.37

Table 8: Comparison with other T2I models on

1024^{2}

Resolution	f8 anchor	UltraDiff	Speedup
$1024^{2}$	27.3s	3.2s	$8.5\times$
$2048^{2}$	249.8s	6.2s	$40.3\times$
$4096^{2}$	OOM	28.3s	–

Table 9: Controlled inference efficiency. Time per image with 20 sampling steps on one NVIDIA A100 GPU.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Model Depth ( $N$ )	28 layers
Hidden Dimension ( $D$ )	2048
Attention Heads	16
KV Heads	16
Head Dimension	$2048/16=128$
FFN Multiplier	1.3 (SwiGLU variant)

Appendix

Table 10: UltraDiff DiT Architecture Specifications. The model follows a standard Transformer backbone with Rotary Positional Embeddings (RoPE) and QK-Norm.

VAE	Scale	Channels	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	rFID $\downarrow$	PatchFID $\downarrow$
Ours (f8)	f8	16	30.9588	0.8719	0.0343	2.1645	3.7048
Ours (f32)	f32	64	26.6258	0.7416	0.0681	6.7395	3.2330
SANA 1.5 DC-AE [ Xie et al., 2024 ]	f32	32	25.3647	0.6888	0.0930	7.8557	4.2969
FLUX.2 autoencoder [ Black Forest Labs, 2025 ]	f8	32	33.8557	0.9242	0.0144	1.1360	0.9926
HunyuanImage-2.1 VAE [ Team, 2025 ]	f32	64	29.4606	0.8217	0.1111	9.7390	10.1938
Stable Diffusion 3.5 VAE [ Esser et al., 2024 ]	f8	16	30.6493	0.8609	0.0298	2.4047	1.3692

Appendix

Table 11: VAE reconstruction quality. Scale and channels describe the native VAE latent output. All models are evaluated under the same reconstruction protocol.

Configuration	f8 VAE (Teacher Anchor)	f32 VAE (Target)
Type	2D KL-VAE	2D KL-VAE
Latent Space	Continuous (Gaussian)	Continuous (Gaussian)
Downsampling Factor	$8\times$	$32\times$
Latent Channels ( $C$ )	16	64
Encoder
Channel Base	128	128

Appendix

Table 12: Detailed Hyperparameters for f8 and f32 VAEs. Both models utilize 2D KL-regularized architectures with continuous latent spaces. The f32 model features a deeper network and larger latent channels to handle the

32\times

compression rate.

Parameter	Value
Optimizer	AdamW
Adam $\beta_{1},\beta_{2}$	0.9, 0.95
Learning Rate (LR)	$1\times 10^{-4}$
Weight Decay	0.01
LR Schedule	Linear Warmup + Cosine Decay
Warmup Steps	500

Appendix

Table 13: Training hyperparameters. Common optimizer settings and stage-specific training lengths used in the UltraDiff pipeline.

Latent Diffusion Models

Latent diffusion models denoise compact representations to generate or transform images, video, audio, and other complex data efficiently.

See all 197 papers

UltraImageGen: Efficient Ultra-High-Resolution Image Generation with Hierarchical Local Attention
Oct 18, 2025Yuyao Zhang, Yu-Wing TaiImage Generation Diffusion Model Inference Acceleration
Dartmouth College, USA
Spatial Gram Alignment for Ultra-High-Resolution Image Synthesis
May 20, 2026Jinjin Zhang, Xiefan Guo, Di HuangDiffusion Model Alignment Latent Diffusion Models
Beihang University
Unified Latents (UL): How to train your latents
Feb 19, 2026Jonathan Heek, Emiel Hoogeboom, Thomas Mensink +1Diffusion Autoencoder Latent Diffusion Models
Google DeepMind Amsterdam

Oct 18, 2025J/K move · Enter open · S save

UltraImageGen: Efficient Ultra-High-Resolution Image Generation with Hierarchical Local Attention

Yuyao Zhang, Yu-Wing Tai

Dartmouth College, USA

Image Generation Diffusion Model Inference Acceleration T2I Diffusion Models

Open paper AskPDF arXiv

Ultra-high-resolution text-to-image generation is increasingly vital for applications requiring fine-grained textures and global structural fidelity, yet state-of-the-art text-to-image diffusion models such as FLUX and SD3 remain confined to sub 2MP (<

1K\times2K

) resolutions due to the quadratic complexity of attention mechanisms and the scarcity of high-quality high-resolution training data. We present UltraImageGen, a novel framework that introduces hierarchical local attention with low-resolution global guidance, enabling efficient, scalable, and semantically coherent image synthesis at ultra-high resolutions. Specifically, high-resolution latents are divided into hardware aligned fixed-size local windows to reduce attention complexity from quadratic to near-linear, while a low-resolution latent equipped with scaled positional embeddings injects global semantics as an anchor. A lightweight LoRA adaptation bridges global and local pathways during denoising, ensuring consistency across structure and detail. To maximize inference efficiency and achieve scalable ultra-high-resolution generation, we repermute token sequence in window-first order, so that the GPU-friendly dense local blocks in attention calculation equals to the fixed-size local window in 2D regardless of resolution. Together ourwork reliably scales the pretrained model to resolutions higher than

8K

with more than

10\times

speed up and significantly lower memory usage. Extensive experiments demonstrate that ourwork achieves superior quality while maintaining computational efficiency, establishing a practical paradigm for advancing ultra-high-resolution image generation.

Abstract

Latent Diffusion Models

Explore similar work

UltraImageGen: Efficient Ultra-High-Resolution Image Generation with Hierarchical Local Attention

Spatial Gram Alignment for Ultra-High-Resolution Image Synthesis

Unified Latents (UL): How to train your latents