LoopPrune: Recurrent Pruning for Looped Diffusion Transformers
Authors: Weizhong Huang, Huaijia Kou, Yan Zhang, Fei Chao, Xiawu Zheng, Liujuan Cao
Organizations: Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China.
Looped diffusion Transformers reuse shared blocks to deepen computation and match larger models with fewer parameters, reducing storage and deployment burdens. Yet repeated attention and MLP computations remain costly. Pruning reduces storage and computation by removing redundant parameters or units. Training-free methods struggle to resolve recurrent contributions, while training-based methods lack explicit constraints on position-specific errors affecting later inputs and final outputs. We propose LoopPrune, a pruning and recovery framework addressing these requirements. Recurrent importance estimation combines shared-removal effects and position variation to select a shared structure; shared reconstruction pools recurrent input statistics to compensate weights. Loop-position recovery compares complete denoising predictions at identical student states and matches text guidance, updating all student parameters against a fixed dense teacher. Extensive experiments across four open-source models, six benchmarks, and three pruning paradigms demonstrate effectiveness. At 30% structured parameter pruning, Looped-DiT B/16 retains 93.14% of dense six-benchmark average accuracy, exceeding the best baseline by 9.49%. Across PyTorch, vLLM-Omni, and SGLang Diffusion, pruned SDXSv3 reduces peak allocated GPU memory by 34.36-44.64% and accelerates denoising 1.46-1.74x over dense models.
Figures & tables
Figure 1: Performance and efficiency of LoopPrune . B/16: retention of six-benchmark mean accuracy relative to Dense. SDXSv3: GPU memory savings and denoising speedups. Teal marks each setting’s strongest baseline.
Figure 2: Why should pruning consider all loops? (a) All-loop channel selection better agrees with later positions. (b) All-loop reconstruction reduces output deviations across positions. (c) Identical inputs enter pruned and dense modules at each position: compare outputs at (1) and final predictions after identical remaining computation at (2). Output deviation falls while prediction deviation rises, motivating all-loop prediction matching. Looped-DiT B/16, 30% structured pruning; definitions and full analysis: Section 5.5 and Appendix A.1.
Figure 3: LoopPrune pruning–reconstruction–recovery pipeline. (a) Score all positions and select a shared mask, obtaining Wp . (b) Pool recurrent inputs to compensate retained weights without training, obtaining Wp . (c) Recover student W⋆ by matching complete predictions and text guidance to a fixed teacher. Arrows connect stage outputs; matched trials select budget-preserving mask exchanges. Images share prompt and noise: recovery brings the vase, leaves, and layout closer to Dense.
Figure 4: Why importance uses two terms. Arrows show aligned, canceling, or weak recurrent gradients. Canceling gradients can deviate strongly from their mean (large vj ), whereas weak gradients have small deviations and vj .
Figure 5: Shared reconstruction. Pooled recurrent inputs estimate H . Under the fixed mask, correction ΔxK moves the pruned projection output toward the teacher output.
Figure 6: Why compare complete predictions? Black/rose/teal: teacher/student/recovered student. Left: a small local gap can grow after identical teacher continuation. Right: recovery targets the complete current-step prediction gap; the backward arrow updates the student through the fixed teacher.
Table 1: Representative results across models and pruning formats. Average is the unweighted mean of six benchmarks; bold marks the best pruning score within a configuration.
Figure 7: Additional pruning budgets on B/16. (a,b) 10%/20% structured parameter pruning; (c,d) 40%/60% unstructured sparsity. Rose denotes LoopPrune ; gray dashed lines mark dense. Wanda* uses a symbolic short bar labeled with its measured score; other bar tops follow the panel scale.
Figure 8: Results on Looped-DiT B/32. (a) 2:4 and (b) 50% unstructured sparsity. Rose denotes LoopPrune ; dashed lines mark dense. Wanda* has symbolic height with its measured score labeled; other bar tops follow the panel scale.
Figure 11: Recurrent pruning analysis. (a) Overlap with each position’s least-important 25% channels; dashed: random. (b) Relative projection MSE, identical retained channels. (c) Positions 1/4 propagated errors (top, shared scale); teacher guidance difference/student guided-prediction error (bottom). (d) Median local/propagated errors, normalized by each input’s four-position mean; stars: (c). Local error compares module outputs, whereas propagated error compares complete predictions after the same remaining teacher computation. (e) Median relative-error ratio: guidance difference/individual predictions. COCO [ 30 ] /GenEval [ 11 ] protocols: Appendices A.1, A.2.
Table 2: Component ablations on B/16 at 30% structured pruning. Each variant removes one component from Full. Avg. (3) averages GenEval, DPG, and PRISM; bold marks column maxima.
Figure 10: Hyperparameter sensitivity: B/16, 30% structured pruning. GenEval/DPG/PRISM mean versus one coefficient per panel; λ is relative to HKK ’s diagonal mean. Teal circles: other settings; rose diamonds: defaults.
Diffusion Transformers (DiTs) deliver remarkable image and video generation quality but incur high computational cost, limiting scalability and on-device deployment. We introduce CoReDiT, a structured token pruning framework for DiTs across vision tasks. CoReDiT uses a linear-time spatial coherence score to estimate local redundancy in the latent token lattice and skips high coherence (redundant) tokens in self-attention. To maintain a dense representation and avoid visual discontinuities, we reconstruct skipped attention outputs via coherence-guided aggregation of spatially neighboring retained tokens. We further introduce a progressive, block-adaptive pruning schedule that increases pruning gradually and allocates larger budgets to blocks and denoising steps with higher redundancy. Across state-of-the-art diffusion backbones including PixArt-α and MagicDrive-V2, CoReDiT achieves up to 55% self-attention FLOPs reduction and inference speedups of 1.33x on cloud GPUs and 1.72x on mobile NPUs, while maintaining high visual quality. Notably, CoReDiT also increases on-device memory head-room, enabling higher-resolution generation.
Diffusion Transformers (DiTs) have demonstrated impressive performance in image generation but suffer from substantial computational overhead and resource consumption. Post-training pruning offers a promising solution; however, due to DiTs' unique architectural design and parameter distribution, traditional pruning methods are inapplicable, leading to significant performance degradation. Specifically, prior methods developed for LLMs, which derive metrics through a series of approximations, amplify the relative contribution of weights in the saliency metric. In addition, weights in DiTs exhibit significantly larger magnitudes than those in LLMs. Moreover, existing pruning granularity overlooks variations in model structures. In this paper, we propose DiT-Pruning, which improves pruning performance by introducing customized saliency criteria and pruning granularity. We design a novel metric that balances the contributions of weights and activations from an energy-based perspective, enabling more effective identification of important elements. Furthermore, we observe distinct clustering patterns in the two-dimensional weight space. Accordingly, we adopt a clustering-aware pruning granularity, enabling effective sparse allocation. Extensive evaluations on various DiTs show that our method consistently preserves image quality, especially under high sparsity. For FLUX.1-dev at 512x512 resolution on MJHQ, DiT-Pruning achieves only a 0.001 loss in CLIP score at 50% sparsity, dramatically outperforming recent pruning methods.
Chengzhi Hu, Xuewen Liu, Jing Zhang +3
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
Uniform diffusion language models (UDLMs) generate text by iteratively denoising sequences of ordinary vocabulary tokens, any of which may be corrupted and subsequently revised. During training, UDLMs must denoise from a context that may itself be unreliable, whereas masked diffusion language models (MDLMs) always condition on a clean context. This challenging training objective makes UDLMs more FLOP-intensive to train than MDLMs, but it yields stronger downstream reasoning performance. Recently, selective looping, which recursively applies a few early Transformer layers within each denoising step, has been shown to improve training efficiency in MDLMs under an iso-FLOP budget. We investigate looping in UDLMs while keeping the training objective and sampler unchanged. We find that looping significantly improves training efficiency: looped UDLMs reach the validation NLL of a non-looped UDLM trained on about 0.5 trillion tokens with up to 2.8x fewer training FLOPs on OpenWebText and 3.3x fewer on FineWeb-Edu. We also find that, at a matched NFE budget, looped UDLMs reach up to 9.8% lower generation perplexity than the non-looped UDLM, with most gains achieved at moderate loop counts. Finally, we study different loop-count distributions and find that sampling smaller loop counts more frequently during training improves inference efficiency: on downstream tasks, this strategy uses 12% fewer loops than uniform loop-count sampling without degrading accuracy.