Diffusion Model Inference Acceleration

Ask about this topic

Overview

Diffusion models repeatedly denoise a sample, so inference acceleration studies how to reduce the cost of those steps for image, video, language, and action generation. Methods reduce the number of evaluations with faster solvers, timestep schedules, and few-step distillation, or reduce work within each step using caching, sparse attention, token pruning, compression, and lower-precision computation. Systems work also targets memory movement, GPU parallelism, and serving, while adaptive methods allocate computation according to uncertainty, content, or task difficulty. The central challenge is preserving fidelity, diversity, and task accuracy as skipped or approximated computations can accumulate errors; gains therefore depend on the model, workload, hardware, and end-to-end evaluation.

Latest papers

CardsList

509 papers73 papers in the last four weeks, up 66% on the four weeks before. 0.4% of all new papers.

Feb 5, 2026

FlashBlock: Attention Caching for Efficient Long-Context Block Diffusion

Generating long-form content, such as minute-long videos and extended texts, is increasingly important for modern generative models. Block diffusion improves inference efficiency via KV caching and block-wise causal inference and has been widely adopted in diffusion language models and video generation. However, in long-context settings, block diffusion still incurs substantial overhead from repeatedly computing attention over a growing KV cache. We identify an underexplored property of block diffusion: cross-step redundancy of attention within a block. Our analysis shows that attention outputs from tokens outside the current block remain largely stable across diffusion steps, while block-internal attention varies significantly. Based on this observation, we propose FlashBlock, a cached block-external attention mechanism that reuses stable attention output, reducing attention computation and KV cache access without modifying the diffusion process. Moreover, FlashBlock is orthogonal to sparse attention and can be combined as a complementary residual reuse strategy, substantially improving model accuracy under aggressive sparsification. Experiments on diffusion language models and video generation demonstrate up to 1.44×\times higher token throughput and up to 1.6×\times reduction in attention time, with negligible impact on generation quality. Project page: https://caesarhhh.github.io/FlashBlock/.

Zhuokun Chen, Jianfei Cai, Bohan Zhuang

1Monash University · 2Zhejiang University

Video Diffusion ModelsSelf-AttentionBlockwise Diffusion Language ModelsEfficient AttentionDiffusion Model CachingAttention Mechanisms+2
AskPDF
Feb 1, 2026

PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers

Diffusion Transformers are fundamental for video and image generation, but their efficiency is bottlenecked by the quadratic complexity of attention. While block sparse attention accelerates computation by attending only critical key-value blocks, it suffers from degradation at high sparsity by discarding context. In this work, we discover that attention scores of non-critical blocks exhibit distributional stability, allowing them to be approximated accurately and efficiently rather than discarded, which is essentially important for sparse attention design. Motivated by this key insight, we propose PISA, a training-free Piecewise Sparse Attention that covers the full attention span with sub-quadratic complexity. Unlike the conventional keep-or-drop paradigm that directly drop the non-critical block information, PISA introduces a novel exact-or-approximate strategy: it maintains exact computation for critical blocks while efficiently approximating the remainder through block-wise Taylor expansion. This design allows PISA to serve as a faithful proxy to full attention, effectively bridging the gap between speed and quality. Experimental results demonstrate that PISA achieves 1.91 times and 2.57 times speedups on Wan2.1-14B and Hunyuan-Video, respectively, while consistently maintaining the highest quality among sparse attention methods. Notably, even for image generation on FLUX, PISA achieves a 1.2 times acceleration without compromising visual quality. Code is available at: https://github.com/xie-lab-ml/piecewise-sparse-attention.

Haopeng Li, Shitong Shao, Wenliang Zhong +4

The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology

Diffusion Model Inference AccelerationAttention ApproximationBlock-Sparse AttentionDiffusion TransformerDiffusion-Based Image GenerationVideo Diffusion Models
AskPDF
Jan 28, 2026

Window-Diffusion: Accelerating Diffusion Language Model Inference with Windowed Token Pruning and Caching

Diffusion language models (DLMs) generate text through iterative denoising, but inference requires full-sequence attention at every iteration, resulting in substantial redundant computation on masked tokens. Block-wise diffusion can reduce this cost, yet it typically relies on retraining and constrained update orders, limiting its direct applicability to pretrained DLMs. Our token-level analysis reveals pronounced structural locality in DLM inference. Decoding is driven by a small set of prefix-localized active tokens; the influence of distant undecoded context diminishes rapidly, and decoded tokens exhibit stage-wise temporal stability, enabling reuse of intermediate representations except for a brief post-decode transient. Motivated by these observations, we propose \placeholder\footnote{The source code is available at https://github.com/vhicrgit/Window-Diffusion.}, a window-based token pruning and caching method for inference. We maintain a local computation window that slides rightward as denoising progresses, and partition undecoded tokens into: (i) active tokens that are computed online, (ii) buffer tokens whose KV states are cached and periodically refreshed, and (iii) far-field tokens that are pruned outside the window. Computation is restricted to active and buffer tokens within the window, while far-field tokens are omitted at each stage. Experiments on LLaDA and Dream show that, under matched compute budgets, our method achieves up to 99×99\times inference speedup while largely preserving generation performance.

Fengrui Zuo, Zhiwei Ke, Yiming Liu +3

School of Computer Science and Technology, University of Science and Technology of China · Suzhou Institute of Advanced Research, University of Science and Technology of China

Diffusion Language Model InferenceToken PruningDiffusion Model CachingAttention MechanismsDiffusion Model Inference Acceleration
AskPDF
Jan 26, 2026

ART for Diffusion Sampling: A Reinforcement Learning Approach to Timestep Schedule

We consider time discretization for score-based diffusion models to generate samples from a learned reverse-time dynamic on a finite grid. Uniform and hand-crafted grids can be suboptimal given a budget on the number of time steps. We introduce Adaptive Reparameterized Time (ART), which controls the clock speed of a reparameterized time variable to redistribute computation along the sampling trajectory while preserving the terminal time, with the objective of minimizing the aggregate Euler discretization error. We derive a randomized companion ART-RL that recasts ART as a continuous-time reinforcement learning problem with Gaussian policies, and prove a two-directional bridge between the two: the deterministic ART optimum lifts to an optimal Gaussian policy, and conversely any optimal Gaussian policy must recover the ART control through its mean. This bridge turns continuous-time actor--critic learning into a principled, rather than heuristic, route to the deterministic timestep optimum. Within the official EDM pipeline, ART-RL improves FID on CIFAR--10 across a wide range of budgets; after one-time offline training, the distilled deterministic schedule transfers without retraining to AFHQv2, FFHQ, and ImageNet at no extra inference cost.

Yilie Huang, Wenpin Tang, Xunyu Zhou

Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027, USA. · Department of Industrial Engineering and Operations Research & Data Science Institute, Columbia University, New York, NY 10027, USA.

Diffusion SamplingAdaptive SamplingRL for Diffusion ModelsDiffusion Timestep SchedulingDiffusion Model Inference Acceleration
AskPDF
Jan 25, 2026

Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decoding

Diffusion Large Language Models (dLLMs) offer a compelling paradigm for natural language generation, leveraging parallel decoding and bidirectional attention to achieve superior global coherence compared to autoregressive models. While recent works have accelerated inference via KV cache reuse or heuristic decoding, they overlook the intrinsic inefficiencies within the block-wise diffusion process. Specifically, they suffer from spatial redundancy by modeling informative-sparse suffix regions uniformly and temporal inefficiency by applying fixed denoising schedules across all the decoding process. To address this, we propose Streaming-dLLM, a training-free framework that streamlines inference across both spatial and temporal dimensions. Spatially, we introduce attenuation guided suffix modeling to approximate the full context by pruning redundant mask tokens. Temporally, we employ a dynamic confidence aware strategy with an early exit mechanism, allowing the model to skip unnecessary iterations for converged tokens. Extensive experiments show that Streaming-dLLM achieves up to 68.2X speedup while maintaining generation quality, highlighting its effectiveness in diffusion decoding. The code is available at https://github.com/xiaoshideta/Streaming-dLLM.

Zhongyu Xiao, Zhiwei Hao, Jianyuan Guo +4

School of information and Electronics, Beijing Institute of Technology, Beijing · Department of Computer Science, City University of Hong Kong, Hong Kong · School of Computer Science, Wuhan University, Wuhan +2

Adaptive InferenceBlockwise Diffusion Language ModelsDiffusion Language Model InferenceToken PruningDiffusion Language Model DecodingDiffusion Model Inference Acceleration
AskPDF
Jan 24, 2026

Turbo4DGen: Ultra-Fast Acceleration for 4D Generation

4D generation, or dynamic 3D content generation, integrates spatial, temporal, and view dimensions to model realistic dynamic scenes, playing a foundational role in advancing world models and physical AI. However, maintaining long-chain consistency across both frames and viewpoints through the unique spatio-camera-motion (SCM) attention mechanism introduces substantial computational and memory overhead, often leading to out-of-memory (OOM) failures and prohibitive generation times. To address these challenges, we propose Turbo4DGen, an ultra-fast acceleration framework for diffusion-based multi-view 4D content generation. Turbo4DGen introduces a spatiotemporal cache mechanism that persistently reuses intermediate attention across denoising steps, combined with dynamically semantic-aware attention pruning and an adaptive SCM chain bypass scheduler, to drastically reduce redundant SCM attention computation. Our experimental results show that Turbo4DGen achieves an average 9.7×\times speedup without quality degradation on the ObjaverseDy and Consistent4D datasets. To the best of our knowledge, Turbo4DGen is the first dedicated acceleration framework for 4D generation.

Yuanbin Man, Ying Huang, Zhile Ren +1

Department of Computer Science Engineering, University of Texas at Arlington, TX, USA · Independent Researcher, CA, USA

4D Scene GenerationDiffusion Model Inference AccelerationEfficient Attention
AskPDF
Jan 20, 2026

A Gaussian Perspective for Distributional Discrepancy in Generative Diffusion Models

This paper introduces an analytical approach to quantifying and optimizing the distributional discrepancy in generative diffusion models. For a multivariate Gaussian source, we explicitly derive the closed-form evolution trajectory and the resulting Kullback-Leibler (KL) divergence between the distributions of the source data and the reversely sampled data. Asymptotic analysis via the Euler-Maclaurin expansion characterizes the convergence behavior of this KL divergence, extracting its dominant term as an explicit functional of the noise schedule. Minimizing this dominant term via the calculus of variations yields a noise schedule described by a tangent law, inherently determined by the source covariance spectrum. We further prove that the Gaussian source exhibits an extremal property for the KL divergence among general source distributions with a given covariance. We also utilize the analytical KL divergence as a principled metric to identify efficient time discretization strategies for pretrained diffusion models, and demonstrate via experiments over diverse datasets that the identified strategies consistently outperform established baselines, particularly under constrained function evaluation budgets.

Qiang Sun, H. Vincent Poor, Wenyi Zhang

Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China · Department of Electrical Engineering, Princeton University, Princeton, NJ, USA

Diffusion SamplingKullback-Leibler DivergenceDiffusion Model Inference Acceleration
AskPDF
Jan 14, 2026

Mixture of Distributions Matters: Dynamic Sparse Attention for Efficient Video Diffusion Transformers

While Diffusion Transformers (DiTs) have achieved notable progress in video generation, this long-sequence generation task remains constrained by the quadratic complexity inherent to self-attention mechanisms, creating significant barriers to practical deployment. Although sparse attention methods attempt to address this challenge, existing approaches either rely on oversimplified static patterns or require computationally expensive sampling operations to achieve dynamic sparsity, resulting in inaccurate pattern predictions and degraded generation quality. To overcome these limitations, we propose a Mixture-Of-Distribution DiT (MOD-DiT), a novel sampling-free dynamic attention framework that accurately models evolving attention patterns through a two-stage process. First, MOD-DiT leverages prior information from early denoising steps and adopts a {distributed mixing approach} to model an efficient linear approximation model, which is then used to predict mask patterns for a specific denoising interval. Second, an online block masking strategy dynamically applies these predicted masks while maintaining historical sparsity information, eliminating the need for repetitive sampling operations. Extensive evaluations demonstrate consistent acceleration and quality improvements across multiple benchmarks and model architectures, validating MOD-DiT's effectiveness for efficient, high-quality video generation while overcoming the computational limitations of traditional sparse attention approaches.

Yuxi Liu, Yipeng Hu, Zekun Zhang +2

Peking University · University of Electronic Science and Technology of China · University of Science and Technology of China

Video Diffusion ModelsDiffusion TransformerSparse AttentionDiffusion Model Inference Acceleration
AskPDF
Jan 13, 2026

SnapGen++: Unleashing Diffusion Transformers for Efficient High-Fidelity Image Generation on Edge Devices

Recent advances in diffusion transformers (DiTs) have set new standards in image generation, yet remain impractical for on-device deployment due to their high computational and memory costs. In this work, we present an efficient DiT framework tailored for mobile and edge devices that achieves transformer-level generation quality under strict resource constraints. Our design combines three key components. First, we propose a compact DiT architecture with an adaptive global-local sparse attention mechanism that balances global context modeling and local detail preservation. Second, we propose an elastic training framework that jointly optimizes sub-DiTs of varying capacities within a unified supernetwork, allowing a single model to dynamically adjust for efficient inference across different hardware. Finally, we develop Knowledge-Guided Distribution Matching Distillation, a step-distillation pipeline that integrates the DMD objective with knowledge transfer from few-step teacher models, producing high-fidelity and low-latency generation (e.g., 4-step) suitable for real-time on-device use. Together, these contributions enable scalable, efficient, and high-quality diffusion models for deployment on diverse hardware.

Dongting Hu, Aarush Gupta, Magzhan Gabidolla +12

Snap Inc. · University of Melbourne · MBZUAI

Diffusion TransformerDiffusion Model DistillationFew-Step Diffusion SamplingDiffusion Model Inference AccelerationOn-Device Inference
AskPDF
Dec 29, 2025

Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusion

Diffusion-based video super-resolution (VSR) methods deliver strong perceptual quality but are often unsuitable for latency-sensitive scenarios due to reliance on future frames and expensive multi-step denoising. We propose Stream-DiffVSR, a causally conditioned diffusion framework for efficient online VSR. Operating strictly on past frames, Stream-DiffVSR integrates a four-step distilled denoiser for fast inference, an Auto-regressive Temporal Guidance (ARTG) module that injects motion-aligned cues during latent denoising, and a lightweight temporal-aware decoder with a Temporal Processor Module (TPM) to enhance detail and temporal coherence. Unlike chunk-wise streaming inference, our strictly frame-by-frame causal design avoids sequence-level waiting, substantially reducing time-to-first-frame and end-to-end latency. Stream-DiffVSR processes 720p frames in 0.328 seconds on an RTX 4090 and consistently outperforms prior diffusion-based baselines. Compared with the online state-of-the-art TMP, it improves perceptual quality (LPIPS +0.095). Compared with prior diffusion-based VSR methods such as MGLD-VSR, it reduces per-frame runtime by over 130x. Moreover, Stream-DiffVSR substantially lowers time-to-first-frame for diffusion-based VSR, reducing initial delay from over 4600 seconds to 0.328 seconds, making diffusion-based VSR markedly more practical for low-latency online and streaming deployment. Project page: https://jamichss.github.io/stream-diffvsr-project-page/

Hau-Shiang Shiu, Chin-Yang Lin, Zhixiang Wang +4

National Yang Ming Chiao Tung University · Shanda AI Research Tokyo · MediaTek Inc.

Video Super-ResolutionDiffusion Model Inference Acceleration
AskPDF
Dec 28, 2025

ReDiF: Resource-Efficient Few-Step Diffusion Distillation via Reinforcement Learning

Step distillation accelerates diffusion sampling by training a few-step student to imitate a many-step teacher, but distillation itself remains expensive. Typically, this requires thousands of GPU-hours and a large pre-generated trajectory dataset. We introduce ReDiF, which casts step distillation as terminal-reward policy optimization rather than step-wise regression. The student is optimized against a reward computed on the terminal sample, measuring alignment with the teacher's output, instead of matching the teacher's intermediate trajectory under a reconstruction or consistency loss. Because the reward need not be differentiable or trajectory-aligned, ReDiF admits non-differentiable objectives, multi-objective combinations, and preferences the teacher does not express, while exploration lets the student find sampling paths matched to its own step schedule. ReDiF converges in 400 policy updates with 3200 rollouts on a single A100 GPU with 1,000 noise-class pairs and no paired dataset: about 4 GPU-hours, against roughly 336 A100-hours reported for DMD2 on the same EDM teacher. At 8 steps on ImageNet-64, it achieves an FID 3.64 points better than the strongest retrained distillation baseline in the low-training regime under the same single-GPU budget. The formulation is also orthogonal to existing distillation objectives: added to the DMD2 loss, it further improves DMD2's FID by 4.4.

Amirhossein Tighkhorshid, Zahra Dehghanian, Hamid R. Rabiee

Computer Engineering Sharif University of Technology

Diffusion DistillationRL for Diffusion ModelsTeacher-Student LearningDiffusion Model Inference Acceleration
AskPDF
Dec 16, 2025

Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Models

Masked Discrete Diffusion Models (MDMs) have achieved strong performance across a wide range of multimodal tasks, including image understanding, generation, and editing. However, their inference speed remains suboptimal due to the need to repeatedly process redundant masked tokens at every sampling step. In this work, we propose Sparse-LaViDa, a novel modeling framework that dynamically truncates unnecessary masked tokens at each inference step to accelerate MDM sampling. To preserve generation quality, we introduce specialized register tokens that serve as compact representations for the truncated tokens. Furthermore, to ensure consistency between training and inference, we design a specialized attention mask that faithfully matches the truncated sampling procedure during training. Built upon the state-of-the-art unified MDM LaViDa-O, Sparse-LaViDa achieves up to a 2x speedup across diverse tasks including text-to-image generation, image editing, and mathematical reasoning, while maintaining generation quality.

Shufan Li, Jiuxiang Gu, Kangning Liu +4

Adobe · UCLA

Masked Diffusion ModelsDiffusion Language Model InferenceToken PruningDiffusion Model Inference AccelerationDiffusion ModelsMultimodal Diffusion Models
AskPDF
Dec 9, 2025

Early Estimation of Language to Latent Alignment in Diffusion Models

Conditional diffusion models frequently suffer from language-image misalignments. Due to the ambiguity of intermediate noise corrupted latents, assessing prompt adherence currently requires completing the entire sampling trajectory. This late-stage evaluation incurs even higher computational costs during test-time scaling strategies, such as Best-of-N (BoN) sampling, as all misaligned trajectories must finish generation before being discarded. To tackle this, we propose NoisyCLIP, a noise-aware twin-tower model that enables early language-to-latent alignment estimation. By learning a vision encoder on noise-corrupted latents, we allow the model to "see" through the ambiguity of intermediate diffusion steps. To facilitate this training, we investigate noise-data augmentation sampling strategies and introduce two new benchmark datasets: Noisy-Conceptual-Captions and Noisy-GenAI-Bench. When applied as an early-stopping criterion for BoN, NoisyCLIP at half cost matches or beats frozen CLIP at full cost. Ultimately, this transforms alignment assessment from an expensive final check into a continuous monitoring tool, drastically reducing compute costs without sacrificing semantic fidelity.

Vasco Ramos, Regev Cohen, Idan Szpektor +1

NOVALINCS, NOVA University of Lisbon, Portugal · Google Research

Diffusion Model AlignmentVision-Language AlignmentConditional Diffusion ModelsDiffusion Model Inference Acceleration
AskPDF
Nov 27, 2025

Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer

The landscape of high-performance image generation models is currently dominated by proprietary systems, such as Nano Banana Pro and Seedream 4.0. Leading open-source alternatives, including Qwen-Image, Hunyuan-Image-3.0 and FLUX.2, are characterized by massive parameter counts (20B to 80B), making them impractical for inference, and fine-tuning on consumer-grade hardware. To address this gap, we propose Z-Image, an efficient 6B-parameter foundation generative model built upon a Scalable Single-Stream Diffusion Transformer (S3-DiT) architecture that challenges the "scale-at-all-costs" paradigm. By systematically optimizing the entire model lifecycle -- from a curated data infrastructure to a streamlined training curriculum -- we complete the full training workflow in just 314K H800 GPU hours (approx. $630K). Our few-step distillation scheme with reward post-training further yields Z-Image-Turbo, offering both sub-second inference latency on an enterprise-grade H800 GPU and compatibility with consumer-grade hardware (<16GB VRAM). Additionally, our omni-pre-training paradigm also enables efficient training of Z-Image-Edit, an editing model with impressive instruction-following capabilities. Both qualitative and quantitative experiments demonstrate that our model achieves performance comparable to or surpassing that of leading competitors across various dimensions. Most notably, Z-Image exhibits exceptional capabilities in photorealistic image generation and bilingual text rendering, delivering results that rival top-tier commercial models, thereby demonstrating that state-of-the-art results are achievable with significantly reduced computational overhead. We publicly release our code, weights, and online demo to foster the development of accessible, budget-friendly, yet state-of-the-art generative models.

Z-Image Team, Huanqia Cai, Sihan Cao +21

Z-Image Team, Alibaba Group

Diffusion TransformerDiffusion Model Inference AccelerationT2I Diffusion Models
AskPDF
Nov 24, 2025

Sphinx: Efficiently Serving Novel View Synthesis using Regression-Guided Selective Refinement

Novel View Synthesis (NVS) generates novel views of a scene from a sparse set of input images and is a key component in applications such as immersive rendering and 3D content generation. Existing NVS approaches broadly exhibit an efficiency--quality tradeoff: regression-based models generate novel views with relatively low latency but can struggle with perceptual quality, particularly in challenging regions or regions not visible in the input views (disoccluded regions), while diffusion-based models produce higher-quality and temporally consistent views at substantially greater computational cost. This motivates a hybrid approach that leverages the complementary strengths of regression and diffusion to bridge this efficiency--quality gap. We present Sphinx, a training-free hybrid inference and serving system for NVS that combines regression-level efficiency with diffusion-level quality. Sphinx first uses fast regression-based NVS to generate coarse novel views, which provide a strong initialization for diffusion and avoid expensive generation from pure noise. It then selectively applies diffusion refinement where additional computation is needed, reducing the denoising workload while preserving the quality benefits of diffusion-based NVS. To efficiently serve the resulting heterogeneous requests, Sphinx further introduces an SLO-aware serving framework that dynamically allocates GPU resources based on each request's remaining computation and urgency. Evaluated across diverse NVS models, datasets, and serving configurations, Sphinx achieves an average 2.3 times inference speedup (up to 2.7 times) with less than 10% perceptual degradation, while its SLO-aware scheduler achieves up to 10 times higher SLO attainment than FCFS scheduling across hardware platforms.

Yuchen Xia, Souvik Kundu, Mosharaf Chowdhury +1

University of Illinois Urbana-Champaign, Illinois, USA · Intel AI Group, USA · University of Michigan, Michigan, USA

Novel View SynthesisDiffusion Model ServingDiffusion Model Inference Acceleration
AskPDF
Nov 21, 2025

DAPS++: Rethinking Diffusion Inverse Problems with Decoupled Posterior Annealing

From a Bayesian perspective, score-based diffusion solves inverse problems through joint inference, embedding the likelihood with the prior to guide the sampling process. However, this formulation fails to explain its practical behavior: the prior offers limited guidance, while reconstruction is largely driven by the measurement-consistency term, leading to an inference process that is effectively decoupled from the diffusion dynamics. We show that the diffusion prior in these solvers functions primarily as a warm initializer that places estimates near the data manifold, while reconstruction is driven almost entirely by measurement consistency. Based on this observation, we introduce DAPS++, which fully decouples diffusion-based initialization from likelihood-driven refinement, allowing the likelihood term to guide inference more directly while maintaining numerical stability and providing insight into why unified diffusion trajectories remain effective in practice. By requiring fewer function evaluations (NFEs) and measurement-optimization steps, DAPS++ achieves high computational efficiency and robust reconstruction performance across diverse image restoration tasks.

Hao Chen, Renzheng Zhang, Scott S. Howard

University of Notre Dame, Notre Dame, IN 46556, USA

Diffusion Models for Image RestorationBayesian Inverse ProblemsDiffusion-Based Inverse ProblemsDiffusion Posterior SamplingDiffusion Model Inference AccelerationImage Inverse Problems
AskPDF
Nov 18, 2025

InstantViR: Real-Time Video Inverse Problem Solver with Distilled Diffusion Prior

Video inverse problems are fundamental to streaming, telepresence, and AR/VR, where high perceptual quality must coexist with tight latency constraints. Diffusion-based priors currently deliver state-of-the-art reconstructions, but existing approaches either adapt image diffusion models with ad hoc temporal regularizers - leading to temporal artifacts - or rely on native video diffusion models whose iterative posterior sampling is far too slow for real-time use. We introduce InstantViR, an amortized inference framework for ultra-fast video reconstruction powered by a pre-trained video diffusion prior. We distill a powerful bidirectional video diffusion model (teacher) into a causal autoregressive student that maps a degraded video directly to its restored version in a single forward pass, inheriting the teacher's strong temporal modeling while completely removing iterative test-time optimization. The distillation is prior-driven: it only requires the teacher diffusion model and known degradation operators, and does not rely on externally paired clean/noisy video data. To further boost throughput, we replace the video-diffusion backbone VAE with a high-efficiency LeanVAE via an innovative teacher-space regularized distillation scheme, enabling low-latency latent-space processing. Across streaming random inpainting, Gaussian deblurring and super-resolution, InstantViR matches or surpasses the reconstruction quality of diffusion-based baselines while running at over 35 FPS on NVIDIA A100 GPUs, achieving up to 100 times speedups over iterative video diffusion solvers. These results show that diffusion-based video reconstruction is compatible with real-time, interactive, editable, streaming scenarios, turning high-quality video restoration into a practical component of modern vision systems.

Weimin Bai, Suzhe Xu, Yiwei Ren +4

Peking University · Huaqiao University · Kuaishou Technology

Diffusion-Based Inverse ProblemsVideo Diffusion ModelsDiffusion Model Inference AccelerationDiffusion Model DistillationVideo Super-Resolution
AskPDF
Oct 18, 2025

UltraImageGen: Efficient Ultra-High-Resolution Image Generation with Hierarchical Local Attention

Ultra-high-resolution text-to-image generation is increasingly vital for applications requiring fine-grained textures and global structural fidelity, yet state-of-the-art text-to-image diffusion models such as FLUX and SD3 remain confined to sub 2MP (< 1K×2K1K\times2K) resolutions due to the quadratic complexity of attention mechanisms and the scarcity of high-quality high-resolution training data. We present UltraImageGen, a novel framework that introduces hierarchical local attention with low-resolution global guidance, enabling efficient, scalable, and semantically coherent image synthesis at ultra-high resolutions. Specifically, high-resolution latents are divided into hardware aligned fixed-size local windows to reduce attention complexity from quadratic to near-linear, while a low-resolution latent equipped with scaled positional embeddings injects global semantics as an anchor. A lightweight LoRA adaptation bridges global and local pathways during denoising, ensuring consistency across structure and detail. To maximize inference efficiency and achieve scalable ultra-high-resolution generation, we repermute token sequence in window-first order, so that the GPU-friendly dense local blocks in attention calculation equals to the fixed-size local window in 2D regardless of resolution. Together ourwork reliably scales the pretrained model to resolutions higher than 8K8K with more than 10×10\times speed up and significantly lower memory usage. Extensive experiments demonstrate that ourwork achieves superior quality while maintaining computational efficiency, establishing a practical paradigm for advancing ultra-high-resolution image generation.

Yuyao Zhang, Yu-Wing Tai

Dartmouth College, USA

Image GenerationDiffusion Model Inference AccelerationT2I Diffusion Models
AskPDF
Sep 29, 2025

DC-Gen: Post-Training Diffusion Acceleration with Deeply Compressed Latent Space

Existing text-to-image diffusion models excel at generating high-quality images, but face significant efficiency challenges when scaled to high resolutions, like 4K image generation. While previous research accelerates diffusion models in various aspects, it seldom handles the inherent redundancy within the latent space. To bridge this gap, this paper introduces DC-Gen, a general framework that accelerates text-to-image diffusion models by leveraging a deeply compressed latent space. Rather than a costly training-from-scratch approach, DC-Gen uses an efficient post-training pipeline to preserve the quality of the base model. A key challenge in this paradigm is the representation gap between the base model's latent space and a deeply compressed latent space, which can lead to instability during direct fine-tuning. To overcome this, DC-Gen first bridges the representation gap with a lightweight embedding alignment training. Once the latent embeddings are aligned, only a small amount of LoRA fine-tuning is needed to unlock the base model's inherent generation quality. We verify DC-Gen's effectiveness on SANA and FLUX.1-Krea. The resulting DC-Gen-SANA and DC-Gen-FLUX models achieve quality comparable to their base models but with a significant speedup. Specifically, DC-Gen-FLUX reduces the latency of 4K image generation by 53x on the NVIDIA H100 GPU. When combined with NVFP4 SVDQuant, DC-Gen-FLUX generates a 4K image in just 3.5 seconds on a single NVIDIA 5090 GPU, achieving a total latency reduction of 138x compared to the base FLUX.1-Krea model. Code: https://github.com/dc-ai-projects/DC-Gen.

Wenkun He, Yuchao Gu, Junyu Chen +11

Diffusion Model Fine-TuningDiffusion Model Inference AccelerationT2I Diffusion Models
AskPDF
Sep 25, 2025

Enabling Approximate Joint Sampling in Diffusion LMs

In autoregressive language models, each token is sampled by conditioning on all the past tokens; the overall string has thus been sampled from the correct underlying joint distribution represented by the model. In contrast, masked diffusion language models generate text by unmasking tokens out of order and potentially in parallel. Generating an overall string sampled from the correct underlying joint distribution would (again) require exactly one token unmasking in every full-model forward pass. The more tokens unmasked in parallel, the further away the string is from the true joint; this can be seen in the resulting drop in accuracy (but, increase in speed). In this paper we devise a way to {\em approximately} sample multiple tokens from the joint distribution in a single full-model forward pass; we do so by developing a new lightweight single-layer ``sampler" on top of an existing large diffusion LM. One forward pass of the full model can now be followed by multiple forward passes of only this sampler layer, to yield multiple unmasked tokens. Our sampler is trained to mimic exact joint sampling from the (frozen) full model. We show the effectiveness of our approximate joint sampling for both pretrained-only (Dream-7B-Base, Llada-7B-Base) and instruction-tuned (Dream-7B-Instruct, Dream-7B-Coder) models on language modeling and math & coding tasks. When four tokens are unmasked for each full-model denoising step, our sampling algorithm achieves a MAUVE score of 0.87 (vs marginal baseline of 0.31) with respect to the true joint distribution.

Parikshit Bansal, Sujay Sanghavi

UT Austin

Masked Diffusion ModelsDiffusion Language Model DecodingDiffusion Language ModelsDiffusion Model Inference Acceleration
AskPDF
Aug 24, 2025

TinySR: Pruning Diffusion for Real-World Image Super-Resolution

Real-world image super-resolution (Real-ISR) focuses on recovering high-quality images from low-resolution inputs that suffer from complex degradations like noise, blur, and compression. Recently, diffusion models (DMs) have shown great potential in this area by leveraging strong generative priors to restore fine details. However, their iterative denoising process incurs high computational overhead, posing challenges for real-time applications. Although one-step distillation methods, such as OSEDiff and TSD-SR, offer faster inference, they remain fundamentally constrained by their large, over-parameterized model architectures. In this work, we present TinySR, a compact yet effective diffusion model specifically designed for Real-ISR that achieves real-time performance while maintaining perceptual quality. We introduce a Dynamic Inter-block Activation and an Expansion-Corrosion Strategy to facilitate more effective decision-making in depth pruning. We achieve VAE compression through channel pruning, attention removal and lightweight SepConv. We eliminate time- and prompt-related modules and perform pre-caching techniques to further speed up the model. TinySR significantly reduces computational cost and model size, achieving up to 5.68x speedup and 83% parameter reduction compared to its teacher TSD-SR, while still providing high quality results.

Linwei Dong, Qingnan Fan, Yuhang Yu +4

Zhejiang University · Vivo Mobile Communication Co. Ltd · Zhejiang Lab

Diffusion Models for Image RestorationDiffusion Model PruningDiffusion Model Inference AccelerationImage Super-Resolution
AskPDF
Aug 23, 2025

Sig-DEG for Distillation: Making Diffusion Models Faster and Lighter

Diffusion models achieve state-of-the-art generative performance but remain computationally expensive at inference, often requiring many discretization steps. We propose Sig-DEG (Signature-based Differential Equation Generator), a generator for distilling pre-trained diffusion models. Inspired by high-order stochastic Taylor expansions, Sig-DEG combines partial signatures that summarize Brownian motion over subintervals with a recurrent structure to approximate stochastic differential equation solutions to achieve universality. We formulate distillation as supervised learning, matching fine-resolution teacher trajectories on a coarse time grid. At inference, Sig-DEG enables fast generation, as the partial signatures can be simulated exactly from their known distributions without requiring fine-grained Brownian paths. Experiments on image and time-series generation demonstrate competitive sample quality with inference speedups of up to several hundred times over the teacher. A Schrödinger bridge example further demonstrates Sig-DEG's applicability beyond diffusion distillation.

Wen Ge, Lei Jiang, Lingyi Yang +7

University College London · University of Oxford · JPMorganChase +1

Path SignaturesDiffusion Model DistillationDiffusion Model Inference AccelerationDiffusion-Based Image GenerationStochastic Differential Equations
AskPDF
Jun 29, 2025

Double-Diffusion: Balancing Speed, Accuracy, and Uncertainty in Probabilistic Forecasting for Urban Sensor Networks

Urban sensor networks need forecasts that are accurate, carry useful uncertainty, and refresh fast enough to act on as new readings arrive. These goals conflict: deterministic models give no distribution, while diffusion forecasters model uncertainty but denoise from pure noise over many steps. We present Double-Diffusion, which integrates a closed-form graph-heat prior into a denoising diffusion model. The prior is a parameter-free low-pass forecast over the sensor graph, and it serves two roles: it is the residual target the model generates, and it conditions the denoiser. The reverse process therefore starts near the prior and denoises a short warm-started chain instead of synthesizing from pure noise; the name denotes this composition, a graph diffusion feeding a denoising diffusion. A compact denoiser, DD-Net, is trained as a Denoising Diffusion Probabilistic Model (DDPM) in the Resfusion warm-start formulation, so generation refines the prior over a short truncated chain rather than synthesizing from pure noise; a graph-spectral read-out of the prior residual sets its switchable spatial filter per domain from training data alone. On four real-world air quality and traffic networks, Double-Diffusion attains the best CRPS of all probabilistic methods on every dataset and stays competitive in point accuracy with the strongest baselines, at a fraction of the sampling cost of from-noise diffusion. The code is available at: https://github.com/teddyicare/Double-Diffusion

Hanlin Dong, Arian Prabowo, Hao Xue +4

University of New South Wales Sydney, Australia · University of Maryland College Park, United States · The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China

Spatiotemporal ForecastingDiffusion Model Inference AccelerationProbabilistic Time Series ForecastingDiffusion ModelsGraph Diffusion
AskPDF
Jun 25, 2025

Stochastic and Non-local Closure Modeling for Nonlinear Dynamical Systems via Latent Score-based Generative Models

We propose a latent score-based generative AI framework for learning stochastic, non-local closure models and constitutive laws in nonlinear dynamical systems of computational mechanics. This work addresses a key challenge of modeling complex multiscale dynamical systems without a clear scale separation, for which numerically resolving all scales is prohibitively expensive, e.g., for engineering turbulent flows. While classical closure modeling methods leverage domain knowledge to approximate subgrid-scale phenomena, their deterministic and local assumptions can be too restrictive in regimes lacking a clear scale separation. Recent developments of diffusion-based stochastic models have shown promise in the context of closure modeling, but their prohibitive computational inference cost limits practical applications in many real-world settings. This work addresses this limitation by jointly training convolutional autoencoders with conditional diffusion models in latent space, significantly reducing the dimensionality of the sampling process while preserving essential physical characteristics. Numerical results demonstrate that the joint training approach helps discover a proper latent space that not only guarantees small reconstruction errors but also ensures good performance of the diffusion model in the latent space. When integrated into numerical simulations, the proposed stochastic modeling framework via latent conditional diffusion models achieves significant computational acceleration while maintaining comparable predictive accuracy to standard diffusion models in physical space.

Xinghao Dong, Huchen Yang, Jin-Long Wu

Department of Mechanical Engineering, University of Wisconsin-Madison, Madison, WI, 53706, USA

Dynamical SystemsLatent Diffusion ModelsConditional Diffusion ModelsDiffusion Model Inference Acceleration
AskPDF
Jun 25, 2025

Ctrl-Z Sampling: Scaling Diffusion Sampling with Controlled Random Zigzag Explorations

Diffusion models generate conditional samples by progressively denoising Gaussian noise, yet the denoising trajectory can stall at visually plausible but low-quality outcomes with conditional misalignment or structural artifacts. We interpret this behavior as local optima in a surrogate quality landscape: Once early denoising commits to a suboptimal global structure, later steps mainly sharpen details and seldom correct the underlying mistake. While existing inference-time approaches explore alternative diffusion states via re-noising with fixed strength or direction, they exhibit limited capacity to escape steep quality plateaus. We propose Controlled Random Zigzag Sampling (Ctrl-Z Sampling),a scalable sampling strategy that detects plateaus in quality landscape via a surrogate score, and allocates exploration only when a plateau is detected. Upon detection, Ctrl-Z Sampling rolls back to noisier states, samples a set of alternative continuations, and updates the trajectory when a candidate improves the score, otherwise escalating the exploration depth to escape the current plateau. The proposed method is model-agnostic and broadly compatible with existing diffusion frameworks. Experiments show that Ctrl-Z Sampling consistently improves generation quality over other inference-time scaling samplers across different NFE budgets, offering a scalable compute-quality trade-off. Code available at: https://github.com/ShunqiM/Ctrl-Z-Sampling.

Shunqi Mao, Wei Guo, Chaoyi Zhang +3

School of Computer Science, The University of Sydney, Australia

Diffusion SamplingConditional Diffusion ModelsDiffusion Model Inference AccelerationDiffusion-Based Denoising
AskPDF
Feb 12, 2025

Disentangling Total-Variance and Signal-to-Noise-Ratio Improves Diffusion Models

Diffusion models have been widely adopted as a powerful generative framework across numerous scientific domains, yet their inherently iterative sampling process remains a significant computational bottleneck. While this is typically mitigated by reducing the number of reverse diffusion steps, sample quality becomes highly sensitive to the chosen noise schedule, i.e. the manner in which noise is introduced and the signal is reduced at each step. Although prior work has improved on the original variance-preserving and variance-exploding schedules, these approaches passively adjust the total variance, without direct independent control over it. In this work, we propose a novel total-variance/signal-to-noise-ratio disentangled (TV/SNR) framework, where TV and SNR can be controlled independently. Our approach reveals that schedules with exponentially exploding TV can be consistently improved by adopting a constant TV schedule while preserving the underlying SNR schedule. Furthermore, we demonstrate that generalizing the SNR schedule of optimal transport flow matching significantly enhances generation efficiency. Crucially, these gains remain consistent across various reverse diffusion solvers and reduce the search space to two independent and interpretable parameters, enabling fast and robust sampling for complex scientific applications, such as molecular structure generation.

Khaled Kahouli, Winfried Ripken, Stefan Gugler +3

BIFOLD – Berlin Institute for the Foundations of Learning and Data · Machine Learning Group, Technische Universität Berlin · Google DeepMind +3

Diffusion Model SamplingDiffusion Timestep SchedulingDiffusion Model Inference Acceleration
AskPDF
Feb 6, 2025

DICE: Distilling Classifier-Free Guidance into Text Embeddings

Text-to-image diffusion models are capable of generating high-quality images, but suboptimal pre-trained text representations often result in these images failing to align closely with the given text prompts. Classifier-free guidance (CFG) is a popular and effective technique for improving text-image alignment in the generative process. However, CFG introduces significant computational overhead. In this paper, we present DIstilling CFG by sharpening text Embeddings (DICE) that replaces CFG in the sampling process with half the computational complexity while maintaining similar generation quality. DICE distills a CFG-based text-to-image diffusion model into a CFG-free version by refining text embeddings to replicate CFG-based directions. In this way, we avoid the computational drawbacks of CFG, enabling high-quality, well-aligned image generation at a fast sampling speed. Furthermore, examining the enhancement pattern, we identify the underlying mechanism of DICE that sharpens specific components of text embeddings to preserve semantic information while enhancing fine-grained details. Extensive experiments on multiple Stable Diffusion v1.5 variants, SDXL, and PixArt-αα demonstrate the effectiveness of our method. Code is available at https://github.com/zju-pi/dice.

Zhenyu Zhou, Defang Chen, Can Wang +2

Zhejiang University, State Key Laboratory of Blockchain and Data Security · University at Buffalo, State University of New York

Diffusion Model AlignmentDiffusion Model GuidanceDiffusion Model DistillationDiffusion Model Inference AccelerationT2I Diffusion Models
AskPDF
Jun 11, 2024

RDM: Recurrent Diffusion Model for Human Motion Generation

Human motion generation is a challenging task due to its high dimensionality and the difficulty of generating fine-grained motions. Diffusion methods have been proposed due to their high sample quality and expressiveness. Early approaches treat the entire sequence as a whole, which is computationally expensive and restricts sequence length. In contrast, autoregressive diffusion models generate longer sequences. However, their reliance on fully denoising previous frames complicates training and inference. Consequently, we propose RDM, a new recurrent diffusion formulation inspired by Recurrent Neural Networks (RNNs). RDM explicitly conditions the diffusion process on preceding noisy frames, avoiding the cost of full denoising. To overcome the challenge of maintaining its probabilistic nature, we employ Normalizing Flows to model recurrent connections. Our evaluations demonstrate RDM's effectiveness: it achieves comparable performance to autoregressive baselines and generates long sequences that remain aligned with the text. RDM also skips diffusion steps during inference, reducing computational cost by up to 18x over autoregressive baselines. Project page: https://www.mirgahney.com/rfdm.io/.

Mirgahney Mohamed, Harry Jake Cunningham, Riccardo Barbano +2

Department of Computer Science, University College London

Diffusion Models for Motion GenerationDiffusion ModelsText-to-Motion GenerationDiffusion Model Inference Acceleration
AskPDF
Mar 10, 2024

Decoupled Data Consistency with Diffusion Purification for Image Restoration

Diffusion models have recently gained traction as a powerful class of deep generative priors, excelling in a wide range of image restoration tasks due to their exceptional ability to model data distributions. To solve image restoration problems, many existing techniques achieve data consistency by incorporating additional likelihood gradient steps into the reverse sampling process of diffusion models. However, the additional gradient steps pose a challenge for real-world practical applications as they incur a large computational overhead, thereby increasing inference time. They also present additional difficulties when using accelerated diffusion model samplers, as the number of data consistency steps is limited by the number of reverse sampling steps. In this work, we propose a novel diffusion-based image restoration solver that addresses these issues by decoupling the reverse process from the data consistency steps. Our method involves alternating between a reconstruction phase to maintain data consistency and a refinement phase that enforces the prior via diffusion purification. Our approach demonstrates versatility, making it highly adaptable for efficient problem-solving in latent space. Additionally, it reduces the necessity for numerous sampling steps through the integration of consistency models. The efficacy of our approach is validated through comprehensive experiments across various image restoration tasks, including image denoising, deblurring, inpainting, and super-resolution.

Xiang Li, Soo Min Kwon, Shijun Liang +3

Department of Electrical Engineering and Computer Science, University of Michigan · Departments of Computational Mathematics, Science and Engineering, and Biomedical Engineering, Michigan State University

Diffusion Models for Image RestorationDiffusion-Based Inverse ProblemsDiffusion Model Inference Acceleration
AskPDF