Skipping the Zeros in Diffusion Models for Sparse Data Generation

Phil Sidney Ostheimer; Mayank Nagda; Andriy Balinskyy; Gabriel Vicente Rodrigues; Jean Radig; Carl Herrmann; Stephan Mandt; Marius Kloft

cs.LGMay 3, 2026

Skipping the Zeros in Diffusion Models for Sparse Data Generation

Authors: Phil Sidney Ostheimer, Mayank Nagda, Andriy Balinskyy, Gabriel Vicente Rodrigues, Jean Radig, Carl Herrmann, Stephan Mandt, Marius Kloft, +1 more

Organizations: 1RPTU University Kaiserslautern-Landau · 2Heidelberg Univer · University of California, Irvine.

Abstract

Diffusion models (DMs) excel on dense continuous data, but are not designed for sparse continuous data. They do not model exact zeros that represent the deliberate absence of a signal. As a result, they erase sparsity patterns and perform unnecessary computation on mostly zero entries. With Sparsity-Exploiting Diffusion (SED), we model only non-zero values, preserving sparsity. SED delivers computational savings while maintaining or improving generation quality by skipping zeros during training and inference. Across physics and biology benchmarks, SED matches or surpasses conventional DMs and domain-specific baselines, while vision experiments provide intuitive insights into the limitations of dense DMs and the benefits of SED.

Explore similar work

RT-Lynx: Putting the GEMM Sparsity In a Right Way for Diffusion Models
May 26, 2026Xing Cong, Hanlin Tang, Kan Liu +3Multimodal Diffusion Transformers Sparse Expert Activation
Training-free sparse attention based on cumulative energy filtering
Jun 15, 2026Chunlu Li, Yixuan Pan, Bai Du +5Dynamic Sparse Attention Spatial Attention
Forward-Learned Discrete Diffusion: Learning how to noise to denoise faster
May 18, 2026Grigory Bartosh, Teodora Pandeva, Sushrut Karmalkar +1Generative Models Factor

May 26, 2026 · cs.LGJ/K move · Enter open · S save

RT-Lynx: Putting the GEMM Sparsity In a Right Way for Diffusion Models

Xing Cong, Hanlin Tang, Kan Liu, Lan Tao+2

1Alibaba Group · 2Independent Researcher

Diffusion Transformers (DiT) achieve strong performance in image generation but incur substantial inference costs. While prior work has reduced this cost via quantization and distillation, semi-structured sparsity, which can nearly halve FLOPs, remains underexplored. A key reason is that most existing approaches focus on weight sparsification, and pruning 50% of the weights can remove critical model capacity and degrade generation quality. Our study, however, shows that DiT activations are intrinsically sparse and significantly more robust to N:M semi-structured sparsification than weights. Motivated by this observation, we advocate a paradigm shift from weight sparsification to activation sparsification. We propose RT-Lynx, which applies N:M sparsification to activations and incorporates error-compensation techniques to mitigate accuracy loss. We further implement highly optimized CUDA kernels tailored to this setting, achieving up to a 1.55x speedup on average in linear layers. Extensive experiments across multiple diffusion models demonstrate that our method preserves the generation quality of the original models while substantially accelerating inference.

Multimodal Diffusion Transformers Sparse Expert Activation Discrete Diffusion Model Matrix Multiplication Graphics Processing Unit Kernels Image Generation Wrong

Open paper Ask about it PDF

Abstract

Explore similar work

RT-Lynx: Putting the GEMM Sparsity In a Right Way for Diffusion Models

Training-free sparse attention based on cumulative energy filtering

Forward-Learned Discrete Diffusion: Learning how to noise to denoise faster