cs.CVNov 17, 2026

Hierarchical Diffusion Transformer for End-to-End Image Generation

Authors: Xiaoyu Wu, Yifei Wang, Weihang Guo, Chen Wei, Lydia E Kavraki, Niloofar Mireshghallah, Steven Wu

Organizations: Carnegie Mellon University · Rice University

Abstract

Pixel-space diffusion learns visual representations and image generation jointly in a single end-to-end training stage, without a separately pretrained visual encoder. However, plain pixel-space Transformers still trail leading two-stage latent diffusion models in generation quality and training efficiency. We hypothesize that part of this gap arises from retaining one token per patch throughout the backbone, which does not explicitly encourage a visual hierarchy that summarizes related content at progressively coarser levels. To address this, we introduce Hierarchical Diffusion Transformers (HiT), which progressively group related tokens and represent each group with a single token. The main Transformer bottleneck processes these compact group tokens, and ungrouping restores features to their original image positions. A compression budget regulates the average compression ratio across the batch, allowing group memberships and counts to vary across images and diffusion timesteps during both training and generation. The model exhibits emergent semantic structure, learning to group coherent objects and object parts without segmentation labels. In text-to-image generation, HiT further learns to group caption words with image patches depicting the same concepts, without region annotations. HiT achieves 1.48 FID on ImageNet 256×256256\times256 and reaches comparable generation quality to JiT using approximately 40% of its cumulative training FLOPs. For text-to-image generation, HiT exceeds UniLiP's average score across five benchmarks with roughly 6×6\times fewer denoiser parameters and 3×3\times fewer forward FLOPs. These results suggest that learning a visual hierarchy within the generator breaks down the barrier to stronger image generation without external visual pretraining.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Pixel-Space Diffusion Transformers

    Jul 20, 2026Renye Yan, Jikang Cheng, You Wu +8Diffusion TransformerPixel-Space Diffusion Models

  2. Just Image Transformers with Late-Stage Pixel Refinement

    Nov 9, 2026Denis Lukovnikov, Andreas Müller, Asja FischerDiffusion-Based Image GenerationDiffusion Transformer

  3. Learning Representations to Guide Pixel Diffusion Transformers

    Oct 9, 2026Zihao Pan, Mengting Chen, Cheng Shi +6Diffusion TransformerPixel-Space Diffusion Models