While diffusion models excel at image generation, their denoising trajectories pass through corrupted intermediate states far from the image manifold. We ask whether generation can instead proceed through recognizable images throughout the entire trajectory. We introduce a generative framework built around a variation-reducing degradation process that contracts diverse images toward a shared prototype while preserving recognizable intermediate states. Because such manifold traversal lacks a simple analytical form, we construct the trajectories using generative-prior-based image morphing. We then distill the reverse process into a diversification operator that transforms the prototype into increasingly varied samples. To resolve the one-to-many ambiguity of the reverse process, we associate each trajectory with its initial noise latent and apply perturbations to improve robustness against accumulated errors. At inference, every step yields a recognizable image, allowing early termination and enabling a controllable speed--variation trade-off.
Figures & tables
Figure 1: (a) Standard diffusion generates images by denoising Gaussian noise toward the data manifold, producing corrupted intermediate states for most of the trajectory. (b) Our method instead starts from a shared prototype already on the data manifold and reverses our variation-reducing degradation process, moving the point within the data distribution to gradually diversify it, enabling a controllable speed–variation trade-off.
Figure 2: Overview. (a) Generate images with a pretrained LDM, select a prototype, and construct variation-reducing trajectories. (b) We train the diversification operator using noise and variation perturbations to improve robustness during iterative generation, conditioned on the associated noise latent η . (c) The trained model generates clean, natural trajectories with a controllable speed-variation trade-off.
Figure 3
Figure 3: Qualitative comparison of generated sequence strips across FFHQ, CelebA, Church, and Bedroom datasets. For each dataset grid, the top section shows xt , the middle section shows x^0 , and the bottom section shows Ours.
Figure 4: Image quality versus diversity across inference steps. Image quality is measured using ImageReward, while diversity is measured using the Vendi Score. Arrows indicate the direction of generation, while circles ( ∙ , ∙ , ∙ , ∙ ) and squares mark ( ■ , ■ , ■ , ■ ) the initial and final steps, respectively. We compare our trajectory with the diffusion baseline’s intermediate states xt and predicted clean images x^0 . The dashed curve represents the variation-reducing trajectories in the curated training dataset. Our method maintains high image quality as diversity progressively increases.
Figure 5: Comparison of our method with and without noise association η . Noise association η enables greater variation throughout the trajectory, whereas omitting it results in trajectories with limited variation and minimal changes.
Figure 6: Comparison of our method with ( smax=5 ) and without ( smax=1 ) variation perturbation smax . Using variation perturbation produces trajectories that more faithfully match the curated ground-truth trajectories.
Figure 7: Effect of noise perturbation τ on image quality. Both insufficient ( τ=0 ) and excessive ( τ=500 and τ=800 ) noise perturbation degrade image quality, resulting in corrupted, distorted, or blurry images. In contrast, a moderate noise perturbation ( τ=300 ) provides the best balance and produces the highest-quality results with lowest FID as shown in Table 2 .
Table 9
Figure 8: Examples of low-quality trajectories from the LSUN-Church dataset.
Diffusion-based models decompose sampling into many small Gaussian denoising steps -- an assumption that breaks down when generation is compressed to a few coarse transitions. Existing few-step methods address this through distillation, consistency training, or adversarial objectives, but sacrifice the likelihood framework in the process. We introduce Normalizing Trajectory Models (NTM), which models each reverse step as an expressive conditional normalizing flow with exact likelihood training. Architecturally, NTM combines shallow invertible blocks within each step with a deep parallel predictor across the trajectory, forming an end-to-end network trainable from scratch or initializable from pretrained flow-matching models. Its exact trajectory likelihood further enables self-distillation: a lightweight denoiser trained on the model's own score produces high-quality samples in four steps. On text-to-image benchmarks, NTM matches or outperforms strong image generation baselines in just four sampling steps while uniquely retaining exact likelihood over the generative trajectory.
Distilling diffusion models into few-step student models substantially reduces sampling latency, but induces severe diversity collapse: samples generated from different random seeds show minimal variation. In this work, we investigate the root cause of this failure by probing the internal representations of student models relative to their teachers. We find that distilled students maintain total activation variance comparable to their teachers, but concentrate it into substantially fewer directions. This directional collapse is localized to an identifiable intermediate block at the initial denoising step. Consequently, diversity collapse stems from a misallocation of activation variance rather than an inherent loss of generative capacity. Guided by this finding, we propose ReDiverse to reinstate the missing variance directions. At training time, ReDiverse-FT fine-tunes a lightweight adapter that regresses student activations onto the teacher's, reviving the teacher's feature variance characteristic. At inference time, ReDiverse-TF introduces a training-free mechanism that steers activations along precomputed teacher principal directions. Across image and video distillation benchmarks, ReDiverse consistently restores sample diversity while preserving generation quality, further supported by the user study.
Furkan Horoz, Ozgur Kara, Kaan Oktay +4
University of Illinois at Urbana-Champaign · fal · Johns Hopkins University
Recent diffusion-based image generation backbones have grown substantially in scale, making the network inference cost increase rapidly. While diffusion distillation techniques can reduce the number of inference steps, high-quality image generation within a single full-backbone-forward compute budget remains challenging. Existing one-step methods typically allocate this budget to a single evaluation of a monolithic student. However, approximating the heterogeneous coarse-to-fine transport with a single monolithic mapping is difficult and often leads to over-smoothed outputs. To address this issue, we propose Phase-wise Velocity Distillation (PVD), which partitions the generation timeline into a coarse and a fine phase, and models the transition within each phase via the average velocity. A dedicated half-sized expert is assigned to each phase, decoupling structural composition from detail refinement while keeping the cumulative computation equivalent to one full-backbone forward pass. We show that the use of two half-sized phase-specific experts outperforms a single full-size monolithic student. On class-conditional image generation, PVD achieves an FID of 1.48 on ImageNet 256 x 256. On more complex text-to-image (T2I) tasks, PVD-distilled models (Stable Diffusion 3.5-Medium, FLUX.1-dev, Qwen-Image) produce results competitive with their multi-step teachers, significantly outperforming prior distillation methods. Moreover, across the evaluated T2I backbones, PVD reduces active parameters by 49.10-50.89% and peak VRAM by 45.76-48.36% compared to the corresponding teachers. Source code and distilled models are available at https://github.com/PolyU-VCLab/PVD.
Zhen Guo, Rongyuan Wu, Qiaosi Yi +3
Department of Computing, The Hong Kong Polytechnic University · OPPO Research Institute