Efficient Coarse-to-Fine Diffusion Models with Time Step Sequence Redistribution

Yu-Shan Tai; An-Yeu; Wu

cs.CVMar 22, 2026

Efficient Coarse-to-Fine Diffusion Models with Time Step Sequence Redistribution

Authors: Yu-Shan Tai, An-Yeu, Wu

Organizations: Andy · Graduate Institute of Electrical Engineering, National Taiwan University, Taipei, Taiwan

Abstract

Recently, diffusion models (DMs) have made significant strides in high-quality image generation. However, the multi-step denoising process often results in considerable computational overhead, impeding deployment on resource-constrained edge devices. Existing methods mitigate this issue by compressing models and adjusting the time step sequence. However, they overlook input redundancy and require lengthy search times. In this paper, we propose Coarse-to-Fine Diffusion Models with Time Step Sequence Redistribution. Recognizing indistinguishable early-stage generated images, we introduce Coarse-to-Fine Denoising (C2F) to reduce computation during coarse feature generation. Furthermore, we design Time Step Sequence Redistribution (TRD) for efficient sampling trajectory adjustment, requiring less than 10 minutes for search. Experimental results demonstrate that the proposed methods achieve near-lossless performance with an 80% to 90% reduction in computation on CIFAR10 and LSUN-Church.

Figures & tables

Figure 1: (a) Original model, (b) C2F, and (c) C2F-DM w/ TRD.

Figure 2: Prediction of the final images $\hat{x}_{t_{0}}^{i}$ and intermediate inputs $x_{t_{i}}$ during denoising, which are noisy in the early stage.

Figure 3: Overview of multi-resolution fine-tuning .

Figure 4: (a) Time Step Sequence Redistribution (TRD). (b) Combined flow (C2F-DM w/ TRD).

Figure 5: (a) FID (

\downarrow

) under different

t_{T^{f}}

and the cut index of Eigen variance 0.99 on

\hat{x}^{i}_{t_{0}}

of (b) C2F-DM 16→32 and (c) C2F-DM 8→32 . DM 32 denotes the original DM. For DMs employing C2F, with the low resolution set at 16 and 8, we denote them as C2F-DM 16→32 and C2F-DM 8→32 .

Figure 6: MACs (

\downarrow

) and FID (

\downarrow

) on (a) CIFAR10 and (b) LSUN-Church.

Figure 7: Generated images $x_{t_{0}}$ under $T^{f}=5$ for LSUN-Church.

Method	Original	2DMs	1DM-xy	1DM-2T	1DM-label
FID $\downarrow$	4.46	4.70	5.48	6.03	4.91

Table I: FID of different multi-resolution fine-tuning strategies

Dataset	CIFAR10	LSUN-Church
Search time (minutes)	1.4	8.8

Table II: Search time under different datasets (

T^{f}=5

)

Diffusion Model Sampling

Diffusion model sampling studies how to generate high-quality samples efficiently by designing reverse processes, solvers, and inference-time controls.

See all 79 papers

Depth as Time in One-Step Generative Models
Oct 2, 2026Arnold Caleb Asiimwe, William Yang, Sanghyuk Chun +2Diffusion Models One-Step Generative Modeling
Princeton University
Timestep-Aware Block Masking for Efficient Diffusion Model Inference
Mar 20, 2026Haodong He, Yuan Gao, Weizhong Zhang +1Diffusion Model Acceleration Diffusion Model Inference Acceleration
School of Computer Science, Wuhan University · School of Artificial Intelligence, Wuhan University · School of Data Science, Fudan University
DRiffusion: Draft-and-Refine Process Parallelizes Diffusion Models with Ease
Mar 26, 2026Runsheng Bai, Chengyu Zhang, Yangdong DengDiffusion Model Sampling Diffusion Model Inference Acceleration
CSAIL, MIT · IST, Nanjing University · School of Software, Tsinghua University; FuturististAI Lab, Shanghai Tsinghua International Innovation Center

Oct 2, 2026J/K move · Enter open · S save

Depth as Time in One-Step Generative Models

Arnold Caleb Asiimwe, William Yang, Sanghyuk Chun, Esin Tureci+1

Princeton University

Diffusion Models One-Step Generative Modeling Flow-Based Generative Modeling Model Compression

Open paper AskPDF arXiv

The recent wave of one-step generative models, which compress the multi-step trajectory of diffusion via either distillation or learned flow maps, has reached an inflection point where they can generate high-quality images. Here, we ask a natural question that follows from these advances: what happens to the denoising trajectory of multi-step diffusion when generation is compressed into a single forward pass? We offer an empirical observation we call depth as time: the denoising computation that multi-step diffusion performs across sampling steps appears to unfold across the depth of a single forward pass, and can be recovered by decoding intermediate layers with the model's own output head. Most interestingly, we show that this depthwise computation depends on the transport task a flow map is trained to solve. The most surprising case is MeanFlow, where probing shorter transport intervals reveals both denoising and renoising within a single network evaluation. In contrast, generators trained without a time-indexed transport task, such as drifting models, do not exhibit the same depthwise denoising. Consequently, we show that models that exhibit the depthwise denoising phenomenon are more compressible across the layerwise computation: a MeanFlow SiT-L/2 model can be compressed by

16.6\times

in parameters into a single time-conditioned block. We offer an explanation for this denoise-then-renoise behavior and show that, when we treat the layerwise computation explicitly as a flow, a single time-conditioned block can be trained to denoise across layers, compressing a MeanFlow SiT-L/2 model by

16.6\times

in parameters. Together, these results suggest that the temporal computation of diffusion is not eliminated by one-step generation, but reorganized across network depth.

Abstract

Diffusion Model Sampling

Explore similar work

Depth as Time in One-Step Generative Models

Timestep-Aware Block Masking for Efficient Diffusion Model Inference

DRiffusion: Draft-and-Refine Process Parallelizes Diffusion Models with Ease