Diffusion Models

Ask about this topic

Overview

Diffusion models learn a data distribution by gradually corrupting examples with noise and training a denoiser to reverse that process. Major approaches include DDPMs and score-based SDEs, latent diffusion and diffusion Transformers, flow and consistency methods, and discrete masked or continuous-time Markov diffusion; conditioning and guidance adapt them to specific tasks. They are used for images, video, audio, language, molecules, and scientific forecasting, with evaluations ranging from ImageNet FID to language perplexity and domain-specific measures of accuracy and uncertainty. A central tension is how to make sampling faster and generation more controllable without sacrificing quality, calibration, or diversity, while theory and ongoing audits address generalization, memorization, and privacy.

Latest papers

CardsList

1,192 papers152 papers in the last four weeks, up 77% on the four weeks before. 0.9% of all new papers.

Apr 2, 2026

Generative models on phase space

Deep generative models such as diffusion and flow matching are powerful machine learning tools capable of learning and sampling from high-dimensional distributions. They are particularly useful when the training data appears to be concentrated on a submanifold of the data embedding space. For high-energy physics data, consisting of collections of relativistic energy-momentum 4-vectors, this submanifold can enforce extremely strong physically-motivated priors, such as energy and momentum conservation. If these constraints are learned only approximately, rather than exactly, this can inhibit the interpretability and reliability of such generative models. To remedy this deficiency, we introduce generative models which are, by construction, confined at every step of their sampling trajectory to the manifold of massless N-particle Lorentz-invariant phase space in the center-of-momentum frame. In the case of diffusion models, the "pure noise" forward process endpoint corresponds to the uniform distribution on phase space, which provides a clear starting point from which to identify how correlations among the particles emerge during the reverse (de-noising) process. We demonstrate that our models are able to learn both few-particle and many-particle distributions with various singularity structures, paving the way for future interpretability studies using generative models trained on simulated jet data.

Zachary Bogorad, Ibrahim Elsharkawy, Yonatan Kahn +2

Theory Division, Fermi National Accelerator Laboratory, Batavia, IL, USA · Department of Physics, University of Toronto and Vector Institute, Toronto, ON, Canada and NERSC, Lawrence Berkeley National Laboratory, Berkeley, California, USA · Department of Physics, University of Toronto and Vector Institute, Toronto, ON, Canada +2

Generative ModelingConstrained Generative ModelingPhysics-Informed Generative ModelingDiffusion ModelsHigh-Energy Physics
AskPDF
Apr 2, 2026

Why Gaussian Diffusion Models Fail on Discrete Data and How to Prevent It?

Diffusion models have become a standard approach for generative modeling in continuous domains, yet their application to discrete data remains challenging. We investigate why Gaussian diffusion models with the DDPM solver struggle to sample from discrete distributions that are represented as a mixture of delta-distributions in the continuous space. Using a toy Random Hierarchy Model, we identify a critical sampling interval in which the density of noisified data becomes multimodal. In this regime, DDPM occasionally enters low-density regions between modes producing out-of-distribution inputs for the model and degrading sample quality. We show that existing heuristics, including self-conditioning and a solver we term q-sampling, help alleviate this issue. Furthermore, we demonstrate that combining self-conditioning with switching from DDPM to q-sampling within the critical interval improves generation quality on real data. We validate these findings across conditional and unconditional tasks in multiple domains, including text, programming code, and proteins.

Alexander Shabalin, Simon Elistratov, Viacheslav Meshchaninov +2

Constructor University, Bremen, Germany · Lomonosov Moscow State University, Moscow, Russia

Diffusion ModelsDiffusion Sampling
AskPDF
Apr 2, 2026

Bias mitigation in graph diffusion models

Most existing graph diffusion models have significant bias problems. We observe that the forward diffusion's maximum perturbation distribution in most models deviates from the standard Gaussian distribution, while reverse sampling consistently starts from a standard Gaussian distribution, which results in a reverse-starting bias. Together with the inherent exposure bias of diffusion models, this results in degraded generation quality. This paper proposes a comprehensive approach to mitigate both biases. To mitigate reverse-starting bias, we employ a newly designed Langevin sampling algorithm to align with the forward maximum perturbation distribution, establishing a new reverse-starting point. To address the exposure bias, we introduce a score correction mechanism based on a newly defined score difference. Our approach, which requires no network modifications, is validated across multiple models, datasets, and tasks, achieving state-of-the-art results.Code is at https://github.com/kunzhan/spp

Meng Yu, Kun Zhan

School of Information Science & Engineering, Lanzhou University

Diffusion ModelsGraph DiffusionDiffusion Model SamplingLangevin Dynamics
AskPDF
Apr 2, 2026

Why Instruction-Based Unlearning Fails in Diffusion Models?

Instruction-based unlearning has proven effective for modifying the behavior of large language models at inference time, but whether this paradigm extends to other generative models remains unclear. In this work, we investigate instruction-based unlearning in diffusion-based image generation models and show, through controlled experiments across multiple concepts and prompt variants, that diffusion models systematically fail to suppress targeted concepts when guided solely by natural-language unlearning instructions. By analyzing both the CLIP text encoder and cross-attention dynamics during the denoising process, we find that unlearning instructions do not induce sustained reductions in attention to the targeted concept tokens, causing the targeted concept representations to persist throughout generation. These results reveal a fundamental limitation of prompt-level instruction in diffusion models and suggest that effective unlearning requires interventions beyond inference-time language control.

Zeliang Zhang, Rui Sun, Jiani Liu +2

University of Rochester · UCLA · UCSB

Diffusion Model UnlearningMachine UnlearningDiffusion ModelsDiffusion-Based Image Generation
AskPDF
Mar 31, 2026

Data-Driven Reachability Analysis via Diffusion Models with PAC Guarantees

We present a data-driven framework for reachability analysis of nonlinear dynamical systems that requires no explicit model. A denoising diffusion probabilistic model learns the time-evolving state distribution of a dynamical system from trajectory data alone. The predicted reachable set takes the form of a sublevel set of a nonconformity score derived from the reconstruction error, with the threshold calibrated via the Learn Then Test procedure so that the probability of excluding a reachable state is bounded with high probability. Experiments on three nonlinear systems, a forced Duffing oscillator, a planar quadrotor, and a high-dimensional reaction-diffusion system, confirm that the empirical miss rate remains below the Probably Approximately Correct (PAC) bound while scaling to state dimensions beyond the reach of classical grid-based and polynomial methods.

Yanliang Huang, Peng Xie, Wenyuan Wu +2

School of Computation, Information and Technology, Technical University of Munich, Munich, Germany. · School of Engineering, Hainan Bielefeld University of Applied Sciences, Hainan, China.

Reachability AnalysisDiffusion Models
AskPDF
Mar 31, 2026

Video Models Reason Early: Exploiting Plan Commitment for Maze Solving

Video diffusion models exhibit emergent reasoning capabilities like solving mazes and puzzles, yet little is understood about how they reason during generation. We take a first step towards understanding this and study the internal planning dynamics of video models using 2D maze solving as a controlled testbed. Our investigations reveal two findings. Our first finding is early plan commitment: video diffusion models commit to a high-level motion plan within the first few denoising steps, after which further denoising alters visual details but not the underlying trajectory. Our second finding is that path length, not obstacle density, is the dominant predictor of maze difficulty, with a sharp failure threshold at 12 steps. This means video models can only reason over long mazes by chaining together multiple sequential generations. To demonstrate the practical benefits of our findings, we introduce Chaining with Early Planning, or ChEaP, which only spends compute on seeds with promising early plans and chains them together to tackle complex mazes. This improves accuracy from 7% to 67% on long-horizon mazes and by 2.5x overall on hard tasks in Frozen Lake and VR-Bench across Wan2.2-14B and HunyuanVideo-1.5. Our analysis reveals that current video models possess deeper reasoning capabilities than previously recognized, which can be elicited more reliably with better inference-time scaling.

Kaleb Newman, Tyler Zhu, Olga Russakovsky

Princeton University

Video Diffusion ModelsVideo ReasoningDiffusion ModelsMotion PlanningDiffusion-Based PlanningInference-Time Scaling
AskPDF
Mar 31, 2026

Diffusion-Based Feature Denoising with NNMF for Robust handwritten digit multi-class classification

This work presents a robust multi-class classification framework for handwritten digits that combines diffusion-driven feature denoising with a hybrid feature representation. Inspired by our previous work on brain tumor classification, the proposed approach operates in a feature space to improve the robustness to noise and adversarial attacks. This manuscript is submitted as an extended abstract rather than a full-length press-ready paper. First, the input images are converted into tight, interpretable exemplification using Non-negative Matrix Factorization (NNMF). In parallel, special deep features are extracted using a computational neural network (CNN). These integral features are combined into a united hybrid representation. The main objective of this work is to extend our previously validated two-class framework to a multi-class handwritten digit classification scenario. To improve robustness, a step diffusion operation is used in the feature space by gradually adding Gaussian noise. A feature denoiser network is trained to reverse this operation and rebuild clean representations from tilted inputs. The courteous features are then applied for multi-class classification. The suggested method is evaluated in both baseline and adversarial settings using AutoAttack. The experimental outcome present that the diffusion-based hybrid model is both effective and robust, the CNN baseline models outperforming while maintain powerful classification performance. These results explain the activity of feature-level diffusion defense for reliable multi-class handwritten digit classification.

Hiba Adil Al-kharsan, Róbert Rajkó

Doctoral School of Computer Science, University of Szeged, Árpád tér 2, H-6720 Szeged, Hungary · University Research and Innovation Center (EKIK), Óbuda University, Bécsi út 96/b, H-1034 Budapest, Hungary

Diffusion-Based DenoisingHandwritten Digit RecognitionMulticlass ClassificationAdversarial RobustnessOptical Character RecognitionDiffusion Models
AskPDF
Mar 31, 2026

SHIFT: Stochastic Hidden-Trajectory Deflection for Removing Diffusion-based Watermark

Diffusion-based watermarking methods embed verifiable marks by manipulating the initial noise or the reverse diffusion trajectory. However, these methods share a critical assumption: verification can succeed only if the diffusion trajectory can be faithfully reconstructed. This reliance on trajectory recovery constitutes a fundamental and exploitable vulnerability. We propose S‾\underline{\mathbf{S}}tochastic Hi‾\underline{\mathbf{Hi}}dden-Trajectory Def‾\underline{\mathbf{f}}lect‾\underline{\mathbf{t}}ion (SHIFT\mathbf{SHIFT}), a training-free attack that exploits this common weakness across diverse watermarking paradigms. SHIFT leverages stochastic diffusion resampling to deflect the generative trajectory in latent space, making the reconstructed image statistically decoupled from the original watermark-embedded trajectory while preserving strong visual quality and semantic consistency. Extensive experiments on nine representative watermarking methods spanning noise-space, frequency-domain, and optimization-based paradigms show that SHIFT achieves 95%--100% attack success rates with nearly no loss in semantic quality, without requiring any watermark-specific knowledge or model retraining.

Rui Bao, Zheng Gao, Xiaoyu Li +3

University of New South Wales · Griffith University

Watermark RemovalDiffusion Model WatermarkingDiffusion Models
AskPDF
Mar 31, 2026

Emotion Diffusion Classifier with Adaptive Margin Discrepancy Training for Facial Expression Recognition

Facial Expression Recognition (FER) is essential for human-machine interaction, as it enables machines to interpret human emotions and internal states from facial affective behaviors. Although deep learning has significantly advanced FER performance, most existing deep-learning-based FER methods rely heavily on discriminative classifiers for fast predictions. These models tend to learn shortcuts and are vulnerable to even minor distribution shifts. To address this issue, we adopt a conditional generative diffusion model and introduce the Emotion Diffusion Classifier (EmoDC) for FER, which demonstrates enhanced adversarial robustness. However, retraining EmoDC using standard strategies fails to penalize incorrect categorical descriptions, leading to suboptimal recognition performance. To improve EmoDC, we propose margin-based discrepancy training, which encourages accurate predictions when conditioned on correct categorical descriptions and penalizes predictions conditioned on mismatched ones. This method enforces a minimum margin between noise-prediction errors for correct and incorrect categories, thereby enhancing the model's discriminative capability. Nevertheless, using a fixed margin fails to account for the varying difficulty of noise prediction across different images, limiting its effectiveness. To overcome this limitation, we propose Adaptive Margin Discrepancy Training (AMDiT), which dynamically adjusts the margin for each sample. Extensive experiments show that AMDiT significantly improves the accuracy of EmoDC over the baseline model with standard denoising diffusion training under 100-step evaluations. Additionally, AMDiT-enhanced EmoDC has better generalization and robustness than state-of-the-art discriminative classifiers.

Rongkang Dong, Cuixin Yang, Cong Zhang +2

Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Kowloon, Hong Kong · School of Control Science and Engineering, Shandong University

Adversarial RobustnessFacial Expression RecognitionConditional Diffusion ModelsDiffusion Models
AskPDF
Mar 31, 2026

iPoster: Content-Aware Layout Generation for Interactive Poster Design via Graph-Enhanced Diffusion Models

We present iPoster, an interactive layout generation framework that empowers users to guide content-aware poster layout design by specifying flexible constraints. iPoster enables users to specify partial intentions within the intention module, such as element categories, sizes, positions, or coarse initial drafts. Then, the generation module instantly generates refined, context-sensitive layouts that faithfully respect these constraints. iPoster employs a unified graph-enhanced diffusion architecture that supports various design tasks under user-specified constraints. These constraints are enforced through masking strategies that precisely preserve user input at every denoising step. A cross content-aware attention module aligns generated elements with salient regions of the canvas, ensuring visual coherence. Extensive experiments show that iPoster not only achieves state-of-the-art layout quality, but offers a responsive and controllable framework for poster layout design with constraints.

Xudong Zhou, Jinyuan Liang, Qiuyi Guo +1

Beijing Institute of Technology, Beijing, China

Constrained Generative ModelingDiffusion Models
AskPDF
Mar 31, 2026

Diffusion Mental Averages

Can a diffusion model produce its own "mental average" of a concept-one that is as sharp and realistic as a typical sample? We introduce Diffusion Mental Averages (DMA), a model-centric answer to this question. While prior methods aim to average image collections, they produce blurry results when applied to diffusion samples from the same prompt. These data-centric techniques operate outside the model, ignoring the generative process. In contrast, DMA averages within the diffusion model's semantic space, as discovered by recent studies. Since this space evolves across timesteps and lacks a direct decoder, we cast averaging as trajectory alignment: optimize multiple noise latents so their denoising trajectories progressively converge toward shared coarse-to-fine semantics, yielding a single sharp prototype. We extend our approach to multimodal concepts (e.g., dogs with many breeds) by clustering samples in semantically-rich spaces such as CLIP and applying Textual Inversion or LoRA to bridge CLIP clusters into diffusion space. This is, to our knowledge, the first approach that delivers consistent, realistic averages, even for abstract concepts, serving as a concrete visual summary and a lens into model biases and concept representation.

Phonphrm Thawatdamrongkit, Sukit Seripanitkarn, Supasorn Suwajanakorn

VISTEC, Thailand

Diffusion ModelsT2I Diffusion Models
AskPDF
Mar 30, 2026

Unrestrained Simplex Denoising for Discrete Data. A Non-Markovian Approach Applied to Graph Generation

Denoising models such as Diffusion or Flow Matching have recently advanced generative modeling for discrete structures, yet most approaches operate directly in the discrete state space, leading to abrupt state changes. We introduce simplex denoising, a simple yet effective generative framework that operates on the probability simplex. The key idea is a non-Markovian noising scheme in which, for a given clean data point, noisy representations at different times are conditionally independent. While preserving the theoretical guarantees of denoising-based generative models, our method removes unnecessary constraints, thereby improving performance and simplifying the formulation. Empirically, unrestrained simplex denoising surpasses strong discrete diffusion and flow-matching baselines across synthetic and real-world graph benchmarks. These results highlight the probability simplex as an effective framework for discrete generative modeling.

Yoann Boget, Alexandros Kalousis

Department of Computer Science, University of Geneva, Switzerland · DMML Group, Geneva School for Business administration HES-SO, Switzerland

Generative ModelingDiffusion ModelsGraph Generation
AskPDF
Mar 30, 2026

Downscaling weather forecasts from Low- to High-Resolution with Diffusion Models

We introduce a probabilistic diffusion-based method for global atmospheric downscaling implemented within the Anemoi framework. The approach transforms low-resolution ensemble forecasts into high-resolution ensembles by learning the conditional distribution of finer-scale residuals, defined as the difference between the high-resolution fields and the interpolated low-resolution inputs. The system is trained on reforecast pairs from ECMWF IFS, using coarse fields at 100 km to reconstruct fine-scale variability at 30 km resolution. The bulk of the training focuses on recovering small-scale structures, while fine-tuning in high-noise regimes enables the generation of extremes. Evaluation against the medium-range IFS ensemble target shows that the model increases probabilistic skill (FCRPS) for surface variables, reproduces target power spectra at small scales, captures physically consistent multivariate relationships such as wind-pressure coupling, and generates extreme values consistent with those of the target ensemble in tropical cyclones.

Joffrey Dumont Le Brazidec, Simon Lang, Martin Leutbecher +9

European Centre for Medium-Range Weather Forecasts (ECMWF)

Weather ForecastingProbabilistic ForecastingDiffusion Models
AskPDF
Mar 30, 2026

ColorFLUX: A Structure-Color Decoupling Framework for Old Photo Colorization

Old photos preserve invaluable historical memories, making their restoration and colorization highly desirable. While existing restoration models can address some degradation issues like denoising and scratch removal, they often struggle with accurate colorization. This limitation arises from the unique degradation inherent in old photos, such as faded brightness and altered color hues, which are different from modern photo distributions, creating a substantial domain gap during colorization. In this paper, we propose a novel old photo colorization framework based on the generative diffusion model FLUX. Our approach introduces a structure-color decoupling strategy that separates structure preservation from color restoration, enabling accurate colorization of old photos while maintaining structural consistency. We further enhance the model with a progressive Direct Preference Optimization (Pro-DPO) strategy, which allows the model to learn subtle color preferences through coarse-to-fine transitions in color augmentation. Additionally, we address the limitations of text-based prompts by introducing visual semantic prompts, which extract fine-grained semantic information directly from old photos, helping to eliminate the color bias inherent in old photos. Experimental results on both synthetic and real datasets demonstrate that our approach outperforms existing state-of-the-art colorization methods, including closed-source commercial models, producing high-quality and vivid colorization.

Bingchen Li, Zhixin Wang, Fan Li +5

University of Science and Technology of China · Huawei Noah’s Ark Lab

Image RestorationImage-to-Image TranslationDiffusion Models
AskPDF
Mar 30, 2026

From Independent to Correlated Diffusion: Generalized Generative Modeling with Probabilistic Computers

Diffusion models have emerged as a powerful framework for generative tasks in deep learning. They decompose generative modeling into two computational primitives: deterministic neural-network evaluation and stochastic sampling. Current implementations usually place most computation in the neural network, but diffusion as a framework allows a broader range of choices for the stochastic transition kernel. Here, we generalize the stochastic sampling component by replacing independent noise injection with Markov chain Monte Carlo (MCMC) dynamics that incorporate known interaction structure. Standard independent diffusion is recovered as a special case when couplings are set to zero. By explicitly incorporating Ising couplings into the diffusion dynamics, the noising and denoising processes exploit spatial correlations representative of the target system. The resulting framework maps naturally onto probabilistic computers (p-computers) built from probabilistic bits (p-bits), which provide orders-of-magnitude advantages in sampling throughput and energy efficiency over GPUs. We demonstrate the approach on equilibrium states of the 2D ferromagnetic Ising model and the 3D Edwards-Anderson spin glass, showing that correlated diffusion produces samples in closer agreement with MCMC reference distributions than independent diffusion. More broadly, the framework shows that p-computers can enable new classes of diffusion algorithms that exploit structured probabilistic sampling for generative modeling.

Nihal Sanjay Singh, Mazdak Mohseni-Rajaee, Shaila Niazi +1

Department of Electrical and Computer Engineering, University of California, Santa Barbara, Santa Barbara, CA, 93106, USA

Diffusion ModelsPhysics-Informed Generative ModelingMarkov Chain Monte Carlo
AskPDF
Mar 30, 2026

Beyond Dataset Distillation: Lossless Dataset Concentration via Diffusion-Assisted Distribution Alignment

The high cost and accessibility problem associated with large datasets hinder the development of large-scale visual recognition systems. Dataset Distillation addresses these problems by synthesizing compact surrogate datasets for efficient training, storage, transfer, and privacy preservation. The existing state-of-the-art diffusion-based dataset distillation methods face three issues: lack of theoretical justification, poor efficiency in scaling to high data volumes, and failure in data-free scenarios. To address these issues, we establish a theoretical framework that justifies the use of diffusion models by proving the equivalence between dataset distillation and distribution matching, and reveals an inherent efficiency limit in the dataset distillation paradigm. We then propose a Dataset Concentration (DsCo) framework that uses a diffusion-based Noise-Optimization (NOpt) method to synthesize a small yet representative set of samples, and optionally augments the synthetic data via "Doping", which mixes selected samples from the original dataset with the synthetic samples to overcome the efficiency limit of dataset distillation. DsCo is applicable in both data-accessible and data-free scenarios, achieving SOTA performances for low data volumes, and it extends well to high data volumes, where it nearly reduces the dataset size by half with no performance degradation.

Tongfei Liu, Yufan Liu, Bing Li +1

State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chines Academy of Sciences, 95 Zhongguancun East Road, Beijing, 100190, Beijing, China. · School of Artificial Intelligence, University of Chinese Academy of Sciences, 1 Yanqihu East Road, Beijing, 101408, Beijing, China. · PeopleAI, Inc., 68 Zhichun Road, Beijing, 100098, Beijing, China. +1

Dataset DistillationDataset CondensationDiffusion-Based Dataset DistillationDistribution MatchingDiffusion ModelsSynthetic Data Augmentation
AskPDF
Mar 29, 2026

You Only Erase Once: Erasing Anything without Bringing Unexpected Content

We present YOEO, an approach for object erasure. Unlike recent diffusion-based methods which struggle to erase target objects without generating unexpected content within the masked regions due to lack of sufficient paired training data and explicit constraint on content generation, our method allows to produce high-quality object erasure results free of unwanted objects or artifacts while faithfully preserving the overall context coherence to the surrounding content. We achieve this goal by training an object erasure diffusion model on unpaired data containing only large-scale real-world images, under the supervision of a sundries detector and a context coherence loss that are built upon an entity segmentation model. To enable more efficient training and inference, a diffusion distillation strategy is employed to train for a few-step erasure diffusion model. Extensive experiments show that our method outperforms the state-of-the-art object erasure methods. Code will be available at https://zyxunh.github.io/YOEO-ProjectPage/.

Yixing Zhu, Qing Zhang, Wenju Xu +1

School of Computer Science and Engineering, Sun Yat-sen University, China · Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China · Amazon

Object RemovalDiffusion-Based Image EditingDiffusion Models
AskPDF
Mar 27, 2026

DRUM: Diffusion-based Raydrop-aware Unpaired Mapping for Sim2Real LiDAR Segmentation

LiDAR-based semantic segmentation is a key component for autonomous mobile robots, yet large-scale annotation of LiDAR point clouds is prohibitively expensive and time-consuming. Although simulators can provide labeled synthetic data, models trained on synthetic data often underperform on real-world data due to a data-level domain gap. To address this issue, we propose DRUM, a novel Sim2Real translation framework. We leverage a diffusion model pre-trained on unlabeled real-world data as a generative prior and translate synthetic data by reproducing two key measurement characteristics: reflectance intensity and raydrop noise. To improve sample fidelity, we introduce a raydrop-aware masked guidance mechanism that selectively enforces consistency with the input synthetic data while preserving realistic raydrop noise induced by the diffusion prior. Experimental results demonstrate that DRUM consistently improves Sim2Real performance across multiple representations of LiDAR data. The project page is available at https://miya-tomoya.github.io/drum.

Tomoya Miyawaki, Kazuto Nakashima, Yumi Iwashita +1

Graduate School of Information Science and Electrical Engineering, Kyushu University, Japan. · Faculty of Information Science and Electrical Engineering, Kyushu University, Japan. · Jet Propulsion Laboratory, California Institute of Technology, USA.

Synthetic-to-Real Domain Adaptation3D Semantic SegmentationSynthetic Data Generation for RoboticsDiffusion Models
AskPDF
Mar 27, 2026

InstaVSR: Taming Diffusion for Efficient and Temporally Consistent Video Super-Resolution

Video super-resolution (VSR) seeks to reconstruct high-resolution frames from low-resolution inputs. While diffusion-based methods have substantially improved perceptual quality, extending them to video remains challenging for two reasons: strong generative priors can introduce temporal instability, and multi-frame diffusion pipelines are often too expensive for practical deployment. To address both challenges simultaneously, we propose InstaVSR, a lightweight diffusion framework for efficient video super-resolution. InstaVSR combines three ingredients: (1) a pruned one-step diffusion backbone that removes several costly components from conventional diffusion-based VSR pipelines, (2) recurrent training with flow-guided temporal regularization to improve frame-to-frame stability, and (3) dual-space adversarial learning in latent and pixel spaces to preserve perceptual quality after backbone simplification. On an NVIDIA RTX 4090, InstaVSR processes a 30-frame video at 2K×\times2K resolution in under one minute with only 7 GB of memory usage, substantially reducing the computational cost compared to existing diffusion-based methods while maintaining favorable perceptual quality with significantly smoother temporal transitions.

Jintong Hu, Bin Chen, Zhenyu Hu +3

Insta360 Research · Peking University · Wuhan University

Video Super-ResolutionDiffusion ModelsDiffusion Model Inference Acceleration
AskPDF
Mar 26, 2026

Lingshu-Cell: A generative cellular world model for transcriptome modeling toward virtual cells

Modeling cellular states and predicting their responses to perturbations are central challenges in computational biology and the development of virtual cells. Existing foundation models for single-cell transcriptomics provide powerful static representations, but they do not explicitly model the distribution of cellular states for generative simulation. Here, we introduce Lingshu-Cell, a masked discrete diffusion model that learns transcriptomic state distributions and supports conditional simulation under perturbation. By operating directly in a discrete token space that is compatible with the sparse, non-sequential nature of single-cell transcriptomic data, Lingshu-Cell captures complex transcriptome-wide expression dependencies across approximately 18,000 genes without relying on prior gene selection, such as filtering by high variability or ranking by expression level. Across diverse tissues and species, Lingshu-Cell accurately reproduces transcriptomic distributions, marker-gene expression patterns and cell-subtype proportions, demonstrating its ability to capture complex cellular heterogeneity. Moreover, by jointly embedding cell type or donor identity with perturbation, Lingshu-Cell can predict whole-transcriptome expression changes for novel combinations of identity and perturbation. It achieves leading performance on the Virtual Cell Challenge H1 genetic perturbation benchmark and in predicting cytokine-induced responses in human PBMCs. Together, these results establish Lingshu-Cell as a flexible cellular world model for in silico simulation of cell states and perturbation responses, laying the foundation for a new paradigm in biological discovery and perturbation screening.

Han Zhang, Guo-Hua Yuan, Chaohao Yuan +6

DAMO Academy, Alibaba Group

Single-Cell Perturbation ModelingSingle-Cell TranscriptomicsWorld ModelsDiffusion Models
AskPDF
Mar 25, 2026

Calibri: Enhancing Diffusion Transformers via Parameter-Efficient Calibration

In this paper, we uncover the hidden potential of Diffusion Transformers (DiTs) to significantly enhance generative tasks. Through an in-depth analysis of the denoising process, we demonstrate that introducing a single learned scaling parameter can significantly improve the performance of DiT blocks. Building on this insight, we propose Calibri, a parameter-efficient approach that optimally calibrates DiT components to elevate generative quality. Calibri frames DiT calibration as a black-box reward optimization problem, which is efficiently solved using an evolutionary algorithm and modifies just ~100 parameters. Experimental results reveal that despite its lightweight design, Calibri consistently improves performance across various text-to-image models. Notably, Calibri also reduces the inference steps required for image generation, all while maintaining high-quality outputs.

Danil Tokhchukov, Aysel Mirzoeva, Andrey Kuznetsov +1

MSU · FusionBrain Lab, AXXX

Diffusion TransformerT2I Diffusion ModelsDiffusion ModelsDiffusion Model Inference Acceleration
AskPDF
Mar 25, 2026

A-SelecT: Automatic Timestep Selection for Diffusion Transformer Representation Learning

Diffusion models have significantly reshaped the field of generative artificial intelligence and are now increasingly explored for their capacity in discriminative representation learning. Diffusion Transformer (DiT) has recently gained attention as a promising alternative to conventional U-Net-based diffusion models, demonstrating a promising avenue for downstream discriminative tasks via generative pre-training. However, its current training efficiency and representational capacity remain largely constrained due to the inadequate timestep searching and insufficient exploitation of DiT-specific feature representations. In light of this view, we introduce Automatically Selected Timestep (A-SelecT) that dynamically pinpoints DiT's most information-rich timestep from the selected transformer feature in a single run, eliminating the need for both computationally intensive exhaustive timestep searching and suboptimal discriminative feature selection. Extensive experiments on classification and segmentation benchmarks demonstrate that DiT, empowered by A-SelecT, surpasses all prior diffusion-based attempts efficiently and effectively.

Changyu Liu, James Chenhao Liang, Wenhao Yang +6

University of Missouri–Kansas City · U. S. Naval Research Laboratory · Lamar University +5

Visual Representation LearningDiffusion ModelsDiffusion TransformerDiffusion Timestep Scheduling
AskPDF
Mar 25, 2026

Polynomial Speedup in Diffusion Models with the Multilevel Euler-Maruyama Method

We introduce the Multilevel Euler-Maruyama (ML-EM) method compute solutions of SDEs and ODEs using a range of approximators f1,…,fkf^1,\dots,f^k to the drift ff with increasing accuracy and computational cost, only requiring a few evaluations of the most accurate fkf^k and many evaluations of the less costly f1,…,fk−1f^1,\dots,f^{k-1}. If the drift lies in the so-called Harder than Monte Carlo (HTMC) regime, i.e. it requires ε−γε^{-γ} compute to be εε-approximated for some γ>2γ>2, then ML-EM εε-approximates the solution of the SDE with ε−γε^{-γ} compute, improving over the traditional EM rate of ε−γ−1ε^{-γ-1}. In other terms it allows us to solve the SDE at the same cost as a single evaluation of the drift. In the context of diffusion models, the different levels f1,…,fkf^{1},\dots,f^{k} are obtained by training UNets of increasing sizes, and ML-EM allows us to perform sampling with the equivalent of a single evaluation of the largest UNet. Our numerical experiments confirm our theory: we obtain up to fourfold speedups for image generation on the CelebA dataset downscaled to 64x64, where we measure a γ≈2.5γ\approx2.5. Given that this is a polynomial speedup, we expect even stronger speedups in practical applications which involve orders of magnitude larger networks.

Arthur Jacot

Diffusion Model Inference AccelerationDiffusion SamplingStochastic Differential EquationsDiffusion Models
AskPDF
Mar 25, 2026

Realiz3D: 3D Generation Made Photorealistic via Domain-Aware Learning

We often aim to generate images that are both photorealistic and 3D-consistent, adhering to precise geometry, material, and viewpoint controls. Typically, this is achieved by fine-tuning an image generator, pre-trained on billions of real images, using renders of synthetic 3D assets, where annotations for control signals are available. While this approach can learn the desired controls, it often compromises the realism of the images due to domain gap between photographs and renders. We observe that this issue largely arises from the model learning an unintended association between the presence of control signals and the synthetic appearance of the images. To address this, we introduce Realiz3D, a lightweight framework for training diffusion models, that decouples controls and visual domain. The key idea is to explicitly learn visual domain, real or synthetic, separately from other control signals by introducing a co-variate that, fed into small residual adapters, shifts the domain. Then, the generator can be trained to gain controllability, without fitting to specific visual domain. In this way, the model can be guided to produce realistic images even when controls are applied. We enhance control transferability to the real domain by leveraging insights about roles of different layers and denoising steps in diffusion-based generators, informing new training and inference strategies that further mitigate the gap. We demonstrate the advantages of Realiz3D in tasks as text-to-multiview generation and texturing from 3D inputs, producing outputs that are 3D-consistent and photorealistic.

Ido Sobol, Kihyuk Sohn, Yoav Blum +4

Technion · Meta AI

Multi-View Image SynthesisImage GenerationControllable Image GenerationDiffusion ModelsSynthetic-to-Real Domain AdaptationDomain Adaptation
AskPDF
Mar 25, 2026

LGTM: Training-Free Light-Guided Text-to-Image Diffusion Model via Initial Noise Manipulation

Diffusion models have demonstrated high-quality performance in conditional text-to-image generation, particularly with structural cues such as edges, layouts, and depth. However, lighting conditions have received limited attention and remain difficult to control within the generative process. Existing methods handle lighting through a two-stage pipeline that relights images after generation, which is inefficient. Moreover, they rely on fine-tuning with large datasets and heavy computation, limiting their adaptability to new models and tasks. To address this, we propose a novel Training-Free Light-Guided Text-to-Image Diffusion Model via Initial Noise Manipulation (LGTM), which manipulates the initial latent noise of the diffusion process to guide image generation with text prompts and user-specified light directions. Through a channel-wise analysis of the latent space, we find that selectively manipulating latent channels enables fine-grained lighting control without fine-tuning or modifying the pre-trained model. Extensive experiments show that our method surpasses prompt-based baselines in lighting consistency, while preserving image quality and text alignment. This approach introduces new possibilities for dynamic, user-guided light control. Furthermore, it integrates seamlessly with models like ControlNet, demonstrating adaptability across diverse scenarios.

Ryugo Morita, Stanislav Frolov, Brian Bernhard Moser +3

RPTU Kaiserslautern-Landau & DFKI GmbH, Kaiserslautern, Germany · Faculty of Science and Engineering, Hosei University, Tokyo, Japan

Controllable Image GenerationT2I Diffusion ModelsDiffusion ModelsDiffusion Model Guidance
AskPDF
Mar 25, 2026

HAM: A Training-Free Style Transfer Approach via Heterogeneous Attention Modulation for Diffusion Models

Diffusion models have demonstrated remarkable performance in image generation, particularly within the domain of style transfer. Prevailing style transfer approaches typically leverage pre-trained diffusion models' robust feature extraction capabilities alongside external modular control pathways to explicitly impose style guidance signals. However, these methods often fail to capture complex style reference or retain the identity of user-provided content images, thus falling into the trap of style-content balance. Thus, we propose a training-free style transfer approach via h\textbf{h}eterogeneous a\textbf{a}ttention m\textbf{m}odulation (HAM\textbf{HAM}) to protect identity information during image/text-guided style reference transfer, thereby addressing the style-content trade-off challenge. Specifically, we first introduces style noise initialization to initialize latent noise for diffusion. Then, during the diffusion process, it innovatively employs HAM for different attention mechanisms, including Global Attention Regulation (GAR) and Local Attention Transplantation (LAT), which better preserving the details of the content image while capturing complex style references. Our approach is validated through a series of qualitative and quantitative experiments, achieving state-of-the-art performance on multiple quantitative metrics.

Yeqi He, Liang Li, Zhiwen Yang +3

Hangzhou Dianzi University · Institute of Computing Technology, Chinese Academy of Sciences · Macao Polytechnic University

Image Style TransferAttention Control in Diffusion ModelsDiffusion ModelsDiffusion-Based Image Editing
AskPDF
Mar 25, 2026

Can We Change "Stroke Size" to Simplify Flow Matching?

Deep-learning-based image generation models are useful but difficult to train. Across the different time steps along a generative path, high-noise states present distinct optimization and prediction challenges. Inspired by the varying stroke sizes used in human painting, we hypothesize that applying rough strokes early and fine strokes late can simplify flow-matching image generation. We develop MultiStroke, which progressively varies the spatial granularity of the clean-signal component in both the evolving state and velocity regression target while leaving the Gaussian component uncoarsened. A lightweight corrector aligns adjacent roughness levels during sampling. On ImageNet-1K, MultiStroke reduces FID by 22.5%22.5\% relative to a matched flow-matching baseline, despite the limited corrector capacity. Code is available at https://github.com/DaChengRuoQue/MultiStroke-Code. A quick video explanation is available at https://dachengruoque.github.io/MultiStroke-Web/.

Yunwei Bai, Ying Kiat Tan, Tsuhan Chen

National University of Singapore

Diffusion Models
AskPDF
Mar 24, 2026

Manifold Generalization Provably Proceeds Memorization in Diffusion Models

Diffusion models often generate novel samples even when the learned score is only coarse -- a phenomenon not accounted for by the standard view of diffusion training as density estimation. In this paper, we show that, under the manifold hypothesis, this behavior can instead be explained by coarse scores capturing the geometry of the data while discarding the fine-scale distributional structure of the population measure~μdataμ_{\scriptscriptstyle\mathrm{data}}. Concretely, whereas estimating the full data distribution μdataμ_{\scriptscriptstyle\mathrm{data}} supported on a kk-dimensional manifold is known to require the classical minimax rate O~(N−1/k)\tilde{\mathcal{O}}(N^{-1/k}), we prove that diffusion models trained with coarse scores can exploit the regularity of the manifold support and attain a near-parametric rate toward a different target distribution. This target distribution has density uniformly comparable to that of~μdataμ_{\scriptscriptstyle\mathrm{data}} throughout any O~(N−β/(4k))\tilde{\mathcal{O}}\bigl(N^{-β/(4k)}\bigr)-neighborhood of the manifold, where ββ denotes the manifold regularity. Our guarantees therefore depend only on the smoothness of the underlying support, and are especially favorable when the data density itself is irregular, for instance non-differentiable. In particular, when the manifold is sufficiently smooth, we obtain that generalization -- formalized as the ability to generate novel, high-fidelity samples -- occurs at a statistical rate strictly faster than that required to estimate the full population distribution~μdataμ_{\scriptscriptstyle\mathrm{data}}.

Zebang Shen, Ya-Ping Hsieh, Niao He

ETH Zurich

Diffusion ModelsMemorization in Generative Models
AskPDF
Mar 24, 2026

CDMT-EHR: A Continuous-Time Diffusion Framework for Generating Mixed-Type Time-Series Electronic Health Records

Electronic health records (EHRs) are invaluable for clinical research, yet privacy concerns severely restrict data sharing. Synthetic data generation offers a promising solution, but EHRs present unique challenges: they contain both numerical and categorical features that evolve over time. While diffusion models have demonstrated strong performance in EHR synthesis, existing approaches predominantly rely on discrete-time formulations, which suffer from finite-step approximation errors and coupled training-sampling step counts. We propose a continuous-time diffusion framework for generating mixed-type time-series EHRs with three contributions: (1) continuous-time diffusion with a bidirectional gated recurrent unit backbone for capturing temporal dependencies, (2) unified Gaussian diffusion via learnable continuous embeddings for categorical variables, enabling joint cross-feature modeling, and (3) a factorized learnable noise schedule that adapts to per-feature-per-timestep learning difficulties. Experiments on two large-scale intensive care unit datasets demonstrate that our method outperforms existing approaches in downstream task performance, distribution fidelity, and discriminability, while requiring only 50 sampling steps compared to 1,000 for baseline methods. Classifier-free guidance further enables effective conditional generation for class-imbalanced clinical scenarios.

Shaonan Liu, Yuichiro Iwashita, Soichiro Nakako +2

Department of Core Informatics, Graduate School of Informatics, Osaka Metropolitan University, Sakai, Osaka 599-8531, Japan · Smart Data & Knowledge Services Department, German Research Center for Artificial Intelligence (DFKI GmbH), 67663 Kaiserslautern, Germany · Department of Computer Science, RPTU University Kaiserslautern-Landau, 67663 Kaiserslautern, Germany +1

Synthetic Data GenerationDiffusion ModelsElectronic Health RecordsLongitudinal EHR ModelingDiffusion Timestep Scheduling
AskPDF
Mar 24, 2026

FG-Portrait: 3D Flow Guided Editable Portrait Animation

Motion transfer from the driving to the source portrait remains a key challenge in the portrait animation. Current diffusion-based approaches condition only on the driving motion, which fails to capture source-to-driving correspondences and consequently yields suboptimal motion transfer. Although flow estimation provides an alternative, predicting dense correspondences from 2D input is ill-posed and often yields inaccurate animation. We address this problem by introducing 3D flows, a learning-free and geometry-driven motion correspondence directly computed from parametric 3D head models. To integrate this 3D prior into diffusion model, we introduce 3D flow encoding to query potential 3D flows for each target pixel to indicate its displacement back to the source location. To obtain 3D flows aligned with 2D motion changes, we further propose depth-guided sampling to accurately locate the corresponding 3D points for each pixel. Beyond high-fidelity portrait animation, our model further supports user-specified editing of facial expression and head pose. Extensive experiments demonstrate the superiority of our method on consistent driving motion transfer as well as faithful source identity preservation.

Yating Xu, Yunqi Miao, Evangelos Ververas +2

National University of Singapore · University of Warwick · Imperial College London +1

Video GenerationDiffusion Models
AskPDF