Diffusion Models

Deep learning-based antimicrobial peptide (AMP) discovery faces critical challenges such as limited controllability, lack of representations that efficiently model antimicrobial properties, and low experimental hit rates. To address these challenges, we introduce OmegAMP, a framework designed for reliable AMP generation with increased controllability. Its diffusion-based generative model leverages a novel conditioning mechanism to achieve fine-grained control over desired physicochemical properties and to direct generation towards specific activity profiles, including species-specific effectiveness. This is further enhanced by a biologically informed encoding space that significantly improves overall generative performance. Complementing these generative capabilities, OmegAMP leverages a novel synthetic data augmentation strategy to train classifiers for AMP filtering, drastically reducing false positive rates and thereby increasing the likelihood of experimental success. Our in silico experiments demonstrate that OmegAMP delivers state-of-the-art performance across key stages of the AMP discovery pipeline, enabling us to achieve an unprecedented success rate in wet lab experiments. We tested 25 candidate peptides, 24 of them (96%) demonstrated antimicrobial activity, proving effective even against multi-drug resistant strains. Our findings underscore OmegAMP's potential to significantly advance computational frameworks in the fight against antimicrobial resistance.

Supervised deep learning achieves strong performance in biomedical image segmentation but relies on costly pixel-wise annotations, motivating semi-supervised approaches that exploit unlabeled data. We introduce a diffusion-based teacher--student framework in which segmentation predictions are used to condition image denoising, encouraging the production of more informative pseudo-labels. The teacher is first pretrained through an unsupervised reconstruction task using diffusion-style corruption, timestep conditioning, and denoising. Starting from a corrupted empty mask, the model predicts an intermediate segmentation that conditions image denoising, encouraging the predicted mask to capture structural information useful for recovering the original image. The resulting teacher is then co-trained with a student using supervised segmentation on labeled samples and cross pseudo-supervision on unlabeled data. We further introduce a multi-round extension during co-training, in which the teacher generates multiple stochastic image reconstructions and corresponding segmentation predictions, providing additional reconstruction and alignment signals to improve its pseudo-labels. We evaluate the proposed framework on three public 2D biomedical segmentation datasets and a 3D left atrial segmentation benchmark. Across several labeling regimes, our method achieves competitive or superior performance compared with state-of-the-art semi-supervised approaches, with the largest gains observed under severe label scarcity.

Image generation algorithms are increasingly integral to diverse aspects of human society, driven by their practical applications. However, insufficient oversight in artificial Intelligence generated content (AIGC) can facilitate the spread of malicious content and increase the risk of copyright infringement. Among the diverse range of image generation models, the Latent Diffusion Model (LDM) is currently the most widely used, dominating the majority of the Text-to-Image model market. Currently, most attribution methods for LDMs rely on directly embedding watermarks into the generated images or their intermediate noise, a practice that compromises both the quality and the robustness of the generated content. To address these limitations, we introduce TraceMark-LDM, an novel algorithm that integrates watermarking to attribute generated images while guaranteeing non-destructive performance. Unlike current methods, TraceMark-LDM leverages watermarks as guidance to rearrange random variables sampled from a Gaussian distribution. To mitigate potential deviations caused by inversion errors, the small absolute elements are grouped and rearranged. Additionally, we fine-tune the LDM encoder to enhance the robustness of the watermark. Experimental results show that images synthesized using TraceMark-LDM exhibit superior quality and attribution accuracy compared to state-of-the-art (SOTA) techniques. Notably, TraceMark-LDM demonstrates exceptional robustness against various common attack methods, consistently outperforming SOTA methods.

Offline high-definition maps provide essential geometric and topological priors for autonomous driving systems. Pure-vision solutions have become the predominant paradigm for offline mapping due to their cost-effectiveness and scalability. However, local-to-global mapping under visual conditions confronts two fundamental challenges: single-shot local observations are susceptible to viewpoint variation and environmental interference, leading to geometric deviations, while multi-source local information exhibits heterogeneous confidence, rendering globally consistent aggregation difficult. To address these, this paper proposes L2G-Map, a framework comprising hierarchical prior diffusion refinement and elliptical space Bayesian fusion. The former jointly embeds temporal context and centerline priors to guide structure completion and topology recovery during denoising, alleviating the information incompleteness inherent in pure-vision settings. The latter incorporates an adaptive weighting strategy driven by elliptical distance propagation, enabling probabilistically optimal aggregation of multi-source information under the Bayesian posterior update paradigm. Extensive experiments on nuScenes and Argoverse benchmark datasets verify the effectiveness of L2G-Map. The proposed refinement component yields consistent local map accuracy improvements across different datasets. Under sensor-degraded conditions, a 3.27% mIoU gain is achieved. Furthermore, the adaptive fusion component significantly enhances the accuracy of global maps. The fused global map can be flexibly embedded into different online map models, yielding an 18.26% mIoU improvement in semantic map construction and a 20.00% enhancement in vectorized map construction, demonstrating the overall advantages of the proposed closed-loop pipeline. Source code will be available at https://github.com/lynn-yu/L2G-Map.

Diffusion models (DMs) have advanced text-to-image (T2I) synthesis, yet their personalization capabilities raise serious privacy and copyright concerns. Malicious actors can misuse these models to generate unauthorized portraits or artistic style replicas. Existing proactive defenses primarily rely on applying adversarial perturbations to reference images to disrupt training. However, these approaches face limitations: they assume all training images are pre-perturbed and are prone to failure when datasets contain unperturbed images or undergo minor data transformations. In this paper, we introduce PersGuard, a novel backdoor-based framework designed to prevent unauthorized personalization of pre-trained T2I diffusion models. Unlike perturbation-based methods, we assume protectors can embed protective backdoors into the models before their release. This mechanism ensures that if a downstream user fine-tunes the model on protected images, the model retains the backdoor and generates predefined protective outputs; conversely, for unprotected images, the backdoor is effectively removed during fine-tuning to ensure normal model utility. We formulate the backdoor injection as a unified optimization problem incorporating three objectives: a backdoor behavior loss to activate protection, a prior preservation loss to maintain standard generation capabilities, and a novel backdoor retention loss. The retention loss is specifically designed to mirror personalization loss, ensuring the backdoor remains robust during downstream fine-tuning. Extensive experiments across gray-box and black-box settings, multi-object protection, and facial identity protection demonstrate that PersGuard provides superior privacy protection compared to existing perturbation-based methods.

Score-based distillation methods train one-step diffusion models in two stages: they first train a teacher score model, then distill it into a one-step student model. However, this distillation process can introduce bias from two sources: errors in the teacher score model and student score estimate. We propose DiffRatio, a new approach for training one-step diffusion models without teacher supervision based on a density-ratio formulation that connects score-based distribution matching and GAN-style adversarial training. Instead of using a teacher score model to provide training targets, DiffRatio directly learns a log density ratio between the student and data distributions across diffusion time steps. As a pre-trained score model is only used to initialize the one-step generator and does not supervise training, this design simplifies the training pipeline, mitigates gradient estimation bias, and reduces the size of auxiliary networks. The learned density ratio can be used as a verifier, enabling a principled inference-time parallel scaling method that further improves sample quality without external rewards or extra sequential computation. DiffRatio achieves strong one-step generation results on CIFAR-10 and ImageNet (64x64 and 512x512), outperforming most teacher-supervised distillation methods.

This paper investigates how diffusion generative models leverage (unknown) low-dimensional structure to accelerate sampling. Focusing on two mainstream samplers -- the denoising diffusion implicit model (DDIM) and the denoising diffusion probabilistic model (DDPM), we prove that their iteration complexities under exact score functions are at most the order of $k/\varepsilon$ (up to log factor), where $\varepsilon$ is the precision in total variation distance and $k$ is some intrinsic dimension of the target distribution. We further extend these convergence guarantees to the setting in which the score functions are learned from data rather than known exactly, showing that the convergence performance degrades gracefully under suitable score estimation assumptions. We then show that these assumptions are attainable via kernel-based score estimators with finite-sample guarantees that also adapt to the low-dimensional structure. Our results apply to a broad family of target distributions without requiring smoothness or log-concavity. Our findings provide the first rigorous evidence for the adaptivity of the DDIM-type samplers to unknown low-dimensional structure, and improve over the state-of-the-art DDPM theory regarding total variation convergence.

Video anomaly detection (VAD) is a vital yet complex open-set task in computer vision, commonly tackled through reconstruction-based methods. However, these methods struggle with two key limitations: (1) insufficient robustness in open-set scenarios, where unseen normal motions are frequently misclassified as anomalies, and (2) an overemphasis on, but restricted capacity for, local motion reconstruction, which are inherently difficult to capture accurately due to their diversity. To overcome these challenges, we introduce a novel frequency-guided diffusion model with perturbation training. First, we enhance robustness by training a generator to produce perturbed samples, which are similar to normal samples and target the weakness of the reconstruction model. This training paradigm expands the reconstruction domain of the model, improving its generalization to unseen normal motions. Second, to address the overemphasis on motion details, we employ the 2D Discrete Cosine Transform (DCT) to separate high-frequency (local) and low-frequency (global) motion components. By guiding the diffusion model with observed high-frequency information, we prioritize the reconstruction of low-frequency components, enabling more accurate and robust anomaly detection. Extensive experiments on five widely used VAD datasets demonstrate that our approach surpasses state-of-the-art methods, underscoring its effectiveness in open-set scenarios and diverse motion contexts. Our project website is https://xiaofeng-tan.github.io/projects/FG-Diff.

We consider the problem of reconstructing a HxWx31 hyperspectral image from a $H\times W$ grayscale snapshot measurement that is captured using only a single diffractive lens and a filterless panchromatic photosensor. This problem is severely ill-posed, but we present a model that produces high-quality results in simulation and experiment. We make efficient use of limited training data by creating a conditional denoising diffusion model that operates on small patches in a shift-invariant manner. During inference, we synchronize per-patch hyperspectral predictions using guidance by physical consistency with the system's optical point spread function. Our experiments reveal that the patch size can be as small as the point spread function, with local optical cues being the main source of information about complete spectra. Also, by drawing multiple samples, our model provides per-pixel uncertainty estimates that strongly correlate with reconstruction error.

Detailed and photorealistic 3D human modeling is essential for various applications and has seen tremendous progress. However, full-body reconstruction from a monocular RGB image remains challenging due to the ill-posed nature of the problem and sophisticated clothing topology with self-occlusions. In this paper, we propose PSHuman, a novel framework that explicitly reconstructs human meshes utilizing priors from the multiview diffusion model. It is found that directly applying multiview diffusion on single-view human images leads to severe geometric distortions, especially on generated faces. To address it, we propose a cross-scale diffusion that models the joint probability distribution of global full-body shape and local facial characteristics, enabling detailed and identity-preserved novel-view generation without any geometric distortion. Moreover, to enhance cross-view body shape consistency of varied human poses, we condition the generative model on parametric models like SMPL-X, which provide body priors and prevent unnatural views inconsistent with human anatomy. Leveraging the generated multi-view normal and color images, we present SMPLX-initialized explicit human carving to recover realistic textured human meshes efficiently. Extensive experimental results and quantitative evaluations on CAPE and THuman2.1 datasets demonstrate PSHumans superiority in geometry details, texture fidelity, and generalization capability.

Out-of-Distribution (OoD) detection aims to justify whether a given sample is from the training distribution of the classifier-under-protection, i.e., In-Distribution (InD), or from an unknown out distribution. Recent researches have leveraged Diffusion Models (DMs) for OoD detection due to their powerful distribution modeling capability. Given an input image, an InD-pretrained DM produces a corresponding InD-aligned counterpart, which serves as a generative reference for comparison. However, existing DM-based methods typically assess this underlying discrepancy through visual-level distances in the raw image space, which may be misaligned with the distributional discrepancy relevant to OoD detection. In this work, we investigate the fundamentals of discrepancy assessment in DM-based OoD detection, asking how the discrepancy between an input and its DM-generated counterpart should be formulated, and in which representation spaces and with which metrics it should be measured. To this end, we propose to assess the discrepancy in a classifier-relative manner by exploiting the representation spaces of the classifier-under-protection, whose training on InD data encodes rich task-relevant InD knowledge. In particular, we quantify two types of discrepancy: feature-level covariate discrepancy in deep feature representations and logit-level concept discrepancy in output logits, enabling effective differentiation between InD and OoD samples. Moreover, a subspace-based strategy is devised to refine representations of the DM generation to promote discrepancy assessment. Together, these designs form our novel detection framework, namely DDR. Extensive experiments on the challenging large-scale ImageNet-1K dataset demonstrate the superior detection performance of DDR over both DM-based and non-DM-based methods.

Despite their empirical success across a wide range of generative tasks, the fundamental principles underlying the ability of diffusion models to learn data distributions are poorly understood. In this work, we develop a new mathematical framework that explains how diffusion models can effectively learn low-dimensional distributions from a finite number of training samples without suffering from the curse of dimensionality. Specifically, motivated by the intrinsic low-dimensional structure of image data, we theoretically analyze a setting in which the data distribution is modeled as a mixture of low-rank Gaussians. Under suitable network parameterization, we show that optimizing the training objective of diffusion models is equivalent to solving the canonical subspace clustering problem over the training samples, where each subspace basis corresponds to the low-rank covariance of a Gaussian component. This equivalence allows us to show that the sample complexity for learning the underlying distribution scales linearly with the intrinsic dimension of the data, rather than exponentially with the ambient dimension. Our theoretical findings are further supported by empirical evidence that demonstrates phase transition phenomena in generalization on both synthetic and real-world image datasets. Moreover, we establish a correspondence between the learned subspace bases and semantic attributes of image data, providing a principled foundation for controllable image generation.

We introduce Seed-to-Seed Translation (StS), a novel approach that combines GANs and diffusion models (DMs) for unpaired Image-to-Image Translation. Our approach is aimed at global translations of complex automotive scenes, where close adherence to the structure and semantics of the source image is essential. We demonstrate that the semantic information encoded in the space of inverted latents (seeds) of a pretrained DM, dubbed as the seed-space, can be used for discriminative tasks, and leverage this information to perform image-to-image translation. Our method involves training an sts-GAN, an unpaired seed-to-seed translation model, based on CycleGAN. The translated seeds are used as the starting point for the DM's sampling process, while structure preservation is ensured using a ControlNet. We demonstrate the effectiveness of our approach for structure-preserving translation of complex automotive scenes, showcasing superior performance compared to existing GAN-based and diffusion-based methods. In addition to advancing the SoTA in automotive scene translations, our approach offers a fresh perspective on leveraging the semantic information encoded within the seed-space of pretrained DMs for effective image editing and manipulation.

Visual diffusion models have revolutionized the field of creative AI, producing high-quality and diverse content. However, they inevitably memorize training images or videos, subsequently replicating their concepts, content, or styles during inference. This phenomenon raises significant concerns about privacy, security, and copyright within generated outputs. In this survey, we provide the first comprehensive review of replication in visual diffusion models, marking a novel contribution to the field by systematically categorizing the existing studies into unveiling, understanding, and mitigating this phenomenon. Specifically, unveiling mainly refers to the methods used to detect replication instances. Understanding involves analyzing the underlying mechanisms and factors that contribute to this phenomenon. Mitigation focuses on developing strategies to reduce or eliminate replication. Beyond these aspects, we also review papers focusing on its real-world influence. For instance, in the context of healthcare, replication is critically worrying due to privacy concerns related to patient data. Finally, the paper concludes with a discussion of the ongoing challenges, such as the difficulty in detecting and benchmarking replication, and outlines future directions including the development of more robust mitigation techniques. By synthesizing insights from diverse studies, this paper aims to equip researchers and practitioners with a deeper understanding at the intersection between AI technology and social good. We release this project at https://github.com/WangWenhao0716/Awesome-Diffusion-Replication.

We discuss a connection between a generative model, called the diffusion model, and nonequilibrium thermodynamics for the Fokker-Planck equation, called stochastic thermodynamics. Using techniques from stochastic thermodynamics, we derive the speed-accuracy relations for diffusion models, which are inequalities that relate the accuracy of data generation to the entropy production rate. This relation can be interpreted as the speed of the diffusion dynamics in the absence of the non-conservative force. From a stochastic thermodynamic perspective, our results provide quantitative insight into how best to generate data in diffusion models. The optimal learning protocol is introduced by the geodesic of space of the 2-Wasserstein distance in optimal transport theory. We numerically illustrate the validity of the speed-accuracy relations for diffusion models with different noise schedules and different data. We numerically discuss our results for optimal and suboptimal learning protocols. We also demonstrate the applicability of our results to data generation from the real-world image datasets.

Diffusion models excel in solving imaging inverse problems due to their ability to model complex image priors. However, their reliance on large, clean datasets for training limits their practical use where clean data is scarce. In this paper, we propose EMDiffusion, an expectation-maximization (EM) approach to train diffusion models from corrupted observations. Our method alternates between reconstructing clean images from corrupted data using a known diffusion model (E-step) and refining diffusion model weights based on these reconstructions (M-step). This iterative process leads the learned diffusion model to gradually converge to the true clean data distribution. We validate our method through extensive experiments on diverse computational imaging tasks, including random inpainting, denoising, and deblurring, achieving new state-of-the-art performance.

Human motion generation is a challenging task due to its high dimensionality and the difficulty of generating fine-grained motions. Diffusion methods have been proposed due to their high sample quality and expressiveness. Early approaches treat the entire sequence as a whole, which is computationally expensive and restricts sequence length. In contrast, autoregressive diffusion models generate longer sequences. However, their reliance on fully denoising previous frames complicates training and inference. Consequently, we propose RDM, a new recurrent diffusion formulation inspired by Recurrent Neural Networks (RNNs). RDM explicitly conditions the diffusion process on preceding noisy frames, avoiding the cost of full denoising. To overcome the challenge of maintaining its probabilistic nature, we employ Normalizing Flows to model recurrent connections. Our evaluations demonstrate RDM's effectiveness: it achieves comparable performance to autoregressive baselines and generates long sequences that remain aligned with the text. RDM also skips diffusion steps during inference, reducing computational cost by up to 18x over autoregressive baselines. Project page: https://www.mirgahney.com/rfdm.io/.

The rapid advancement of diffusion models has enabled the generation of high-fidelity images from textual prompts, yet achieving precise, disentangled control over specific attributes remains a significant challenge. A fundamental limitation arises because visual differences between images are often far more descriptive and nuanced than what can be captured through human-crafted text descriptions, which frequently fail to convey fine-grained semantic details. To address this, we introduce ViDiT (Visual Direction Transfer for Diffusion), a framework that expands the editing vocabulary by capturing latent semantics directly from image-edit pairs. ViDiT learns the underlying transformation by optimizing a single, global, and continuous editing direction from a small set of before-and-after'' examples. This optimization process transfers visual changes into the diffusion model's conditioning space, allowing for detailed edits that text alone cannot easily describe. ViDiT operates on a Learn Once'' principle, which completely eliminates the need for model fine-tuning or expensive per-image optimization during inference. Once learned, these continuous directions enable ``Edit Anywhere'' capabilities, allowing users to apply highly disentangled manipulations, such as changes in facial features, animal attributes, or artistic styles, to any image in a zero-shot manner with granular control over the edit intensity. Quantitative and qualitative evaluations demonstrate that ViDiT outperforms existing text-based editing methods in maintaining input faithfulness while achieving precise, scalable attribute control.

We propose denoising diffusion variational inference (DDVI), a black-box variational inference algorithm for latent variable models which relies on diffusion models as flexible approximate posteriors. Specifically, our method introduces an expressive class of diffusion-based variational posteriors that perform iterative refinement in latent space; we train these posteriors with a novel regularized evidence lower bound (ELBO) on the marginal likelihood inspired by the wake-sleep algorithm. Our method is easy to implement (it fits a regularized extension of the ELBO), is compatible with black-box variational inference, and generally outperforms alternative classes of approximate posteriors based on normalizing flows or adversarial networks. We find that DDVI improves inference and learning in deep latent variable models across common benchmarks as well as on a motivating task in biology---inferring latent ancestry from human genomes---where it outperforms strong baselines on the 1000 Genomes dataset.

Diffusion models have shown promising results for a wide range of generative tasks with continuous data, such as image and audio synthesis. However, little progress has been made on using diffusion models to generate symbolic music since this class of generative models struggles with discrete data while its iterative sampling process is computationally expensive. In this work, we propose a diffusion model integrated with a Generative Adversarial Network, aiming to mitigate the slow sampling drawback of diffusion models applied to symbolic music generation. We enable the use of diffusion model by representing symbolic music as image-like pianorolls with velocity values instead of discrete event tokens and then used a Variational Autoencoder to obtain their latent representations to train the proposed diffusion model. Our results demonstrate the successful control of our diffusion model to generate symbolic music with a desired emotion. Our model uses merely 15 denoising diffusion steps to produce high-quality music, while most diffusion models for symbolic music generation require hundreds or thousands of steps.

While diffusion models excel at generating high-quality samples, their latent variables typically lack semantic meaning and are not suitable for representation learning. Here, we propose InfoDiffusion, an algorithm that augments diffusion models with low-dimensional latent variables that capture high-level factors of variation in the data. InfoDiffusion relies on a learning objective regularized with the mutual information between observed and hidden variables, which is designed to improve latent space quality and to prevent the latents from being ignored by expressive diffusion-based decoders. Empirically, we find that InfoDiffusion learns disentangled and human-interpretable latent representations that are competitive with state-of-the-art generative and contrastive methods, while retaining the high sample quality of diffusion models. Our method enables manipulating the attributes of generated images and has the potential to assist tasks that require exploring a learned latent space to generate quality samples, e.g., generative design.

Diffusion models have proven to be a flexible and effective framework for modelling probability distributions on finite-dimensional spaces. However, many physical modelling problems such as time series are naturally described over function spaces. In this work we apply diffusion models to such stochastic processes. To do so we consider a spectral representation of the data, obtained using a kernel, thereby dissociating the stochastic part of the processes from their space-time structure. As a result, the stochasticity of the processes is entirely encoded in the spectral coefficients, which we truncate and model using standard finite-dimensional diffusion models. By truncating the representation in the spectral domain we ensure our resulting model defines valid stochastic processes, thereby naturally satisfying consistency and exchangeability criteria. Projecting our spectral diffusion models back to the original input space, we show that for any given marginals our approach corresponds to a diffusion model with correlated noise, with explicit covariance matrix given by the kernel. We demonstrate our method's effectiveness for modelling various multimodal datasets as well as conditional sampling by amortising our models with respect to a context set.