Overview

Diffusion models learn a data distribution by gradually corrupting examples with noise and training a denoiser to reverse that process. Major approaches include DDPMs and score-based SDEs, latent diffusion and diffusion Transformers, flow and consistency methods, and discrete masked or continuous-time Markov diffusion; conditioning and guidance adapt them to specific tasks. They are used for images, video, audio, language, molecules, and scientific forecasting, with evaluations ranging from ImageNet FID to language perplexity and domain-specific measures of accuracy and uncertainty. A central tension is how to make sampling faster and generation more controllable without sacrificing quality, calibration, or diversity, while theory and ongoing audits address generalization, memorization, and privacy.

Latest papers

1,278 papers90 papers in the last four weeks, down 44% on the four weeks before. 0.7% of all new papers.

Jan 21, 2026

HyperNet-Adaptation for Diffusion-Based Test Case Generation

The increasing deployment of deep learning systems requires systematic evaluation of their reliability in real-world scenarios. Traditional gradient-based adversarial attacks introduce small perturbations that rarely correspond to realistic failures and mainly assess robustness rather than functional behavior. Generative test generation methods offer an alternative but are often limited to simple datasets or constrained input domains. Although diffusion models enable high-fidelity image synthesis, their computational cost and limited controllability restrict their applicability to large-scale testing. We present HyNeA, a generative testing method that enables direct and efficient control over diffusion-based generation. HyNeA provides dataset-free controllability through hypernetworks, allowing targeted manipulation of the generative process without relying on architecture-specific conditioning mechanisms or dataset-driven adaptations such as fine-tuning. HyNeA employs a distinct training strategy that supports instance-level tuning to identify failure-inducing test cases without requiring datasets that explicitly contain examples of similar failures. This approach enables the targeted generation of realistic failure cases at substantially lower computational cost than search-based methods. Experimental results show that HyNeA improves controllability and test diversity compared to existing generative test generators and generalizes to domains where failure-labeled training data is unavailable.
Jan 20, 2026

Diff-MN: Diffusion Parameterized MoE-NCDE for Continuous Time Series Generation with Irregular Observations

Time series generation (TSG) is widely used across domains, yet most existing methods assume regular sampling and fixed output resolutions. These assumptions are often violated in practice, where observations are irregular and sparse, while downstream applications require continuous and high-resolution TS. Although Neural Controlled Differential Equation (NCDE) is promising for modeling irregular TS, it is constrained by a single dynamics function, tightly coupled optimization, and limited ability to adapt learned dynamics to newly generated samples from the generative model. We propose Diff-MN, a continuous TSG framework that enhances NCDE with a Mixture-of-Experts (MoE) dynamics function and a decoupled architectural design for dynamics-focused training. To further enable NCDE to generalize to newly generated samples, Diff-MN employs a diffusion model to parameterize the NCDE temporal dynamics parameters (MoE weights), i.e., jointly learn the distribution of TS data and MoE weights. This design allows sample-specific NCDE parameters to be generated for continuous TS generation. Experiments on ten public and synthetic datasets demonstrate that Diff-MN consistently outperforms strong baselines on both irregular-to-regular and irregular-to-continuous TSG tasks. The code is available at the link https://github.com/microsoft/TimeCraft/tree/main/Diff-MN.
Jan 16, 2026

GenDA: Generative Data Assimilation on Complex Urban Areas via Classifier-Free Diffusion Guidance

Urban wind flow reconstruction is essential for assessing air quality, heat dispersion, and pedestrian comfort, yet remains challenging when only sparse sensor data are available. We propose GenDA, a generative data assimilation framework that reconstructs high-resolution wind fields on unstructured meshes from limited observations. The model employs a multiscale graph-based diffusion architecture trained on computational fluid dynamics (CFD) simulations and interprets classifier-free guidance as a learned posterior reconstruction mechanism: the unconditional branch learns a geometry-aware flow prior, while the sensor-conditioned branch injects observational constraints during sampling. This formulation enables obstacle-aware reconstruction and generalization to held-out mesh geometries, wind directions, and sensor configurations within the studied urban-flow setting, without retraining. We consider both sparse fixed sensors and trajectory-based observations using the same reconstruction procedure. When evaluated against supervised graph neural network (GNN) baselines and classical reduced-order data assimilation methods, GenDA reduces the relative root-mean-square error (RRMSE) by 25-57% and increases the structural similarity index (SSIM) by 23-33% across the tested meshes. Experiments are conducted on Reynolds-averaged Navier-Stokes (RANS) simulations of a real urban neighborhood in Bristol, United Kingdom, at a characteristic Reynolds number of Re≈2×107\mathrm{Re}\approx2\times10^{7}, featuring complex building geometry and irregular terrain. The proposed framework provides a scalable path toward generative, geometry-aware data assimilation for environmental monitoring in complex domains.
Jan 6, 2026

Systematic Hazard Sampling: Minimal-Variance Inference for Discrete Diffusion and Flow Models

Uniform-noise discrete diffusion and flow models generate sequences non-autoregressively through iterative, context-dependent token replacements. However, these models are typically formulated as time-inhomogeneous continuous- or discrete-time Markov chains (CTMC/DTMC), sampled using independent Bernoulli change decisions per discretization step. This induces Poisson-binomial variance in per-position jump counts that grows with the number of required edits, leading to the common under-editing (residual noise) and over-editing (cascading substitutions) failure modes that degrade sample quality. We identify this sampler-induced variance as an orthogonal source of degradation, distinct from model-side errors and addressable purely at inference time. We propose Systematic Hazard Sampling (SHS), a training-free, drop-in, and hyperparameter-free inference principle for any sampler that admits a stay-vs.-replace decomposition. SHS models per-token edits as events driven by cumulative hazard (CTMC) or jump mass (DTMC) and triggers an edit whenever this quantity exceeds unit-spaced thresholds with a single random phase per position. For any fixed cumulative mass, this preserves the expected jump count while achieving the minimum conditional variance possible among unbiased integer estimators (at most 1/4), without altering per-jump destination sampling. Experiments on four uniform-noise discrete diffusion and flow language models spanning ~110M to ~3B parameters show that SHS consistently improves sample quality across numbers of function evaluations (NFE). Code is available at https://github.com/Jang-seunghwan/Systematic-Hazard-Sampling.
Dec 22, 2025

Diffusion Models in Simulation-Based Inference: A Tutorial Review

Diffusion models have recently emerged as powerful learners for simulation-based inference (SBI), enabling fast and accurate estimation of latent parameters from simulated and real data. Their score-based formulation offers a flexible way to learn conditional or joint distributions over parameters and observations, thereby providing a versatile solution to various modeling problems. In this tutorial review, we synthesize recent developments on diffusion models for SBI, covering design choices for training, inference, and evaluation. We highlight opportunities created by various concepts such as guidance, score composition, flow matching, consistency models, and joint modeling. Furthermore, we discuss how efficiency and statistical accuracy are affected by noise schedules, parameterizations, and samplers. Finally, we illustrate these concepts with case studies across parameter dimensionalities, simulation budgets, and model types, and outline open questions for future research.
Dec 22, 2025

MixFlow Training: Alleviating Exposure Bias with Slowed Interpolation Mixture

This paper studies the training-testing discrepancy (a.k.a. exposure bias) problem for improving the diffusion models. During training, the input of a prediction network at one training timestep is the corresponding ground-truth noisy data that is an interpolation of the noise and the data, and during testing, the input is the generated noisy data. We present a novel training approach, named MixFlow, for improving the performance. Our approach is motivated by the Slow Flow phenomenon: the ground-truth interpolation that is the nearest to the generated noisy data at a given sampling timestep is observed to correspond to a higher-noise timestep (termed slowed timestep), i.e., the corresponding ground-truth timestep is slower than the sampling timestep. MixFlow leverages the interpolations at the slowed timesteps, named slowed interpolation mixture, for post-training the prediction network for each training timestep. Experiments over class-conditional image generation (including SiT, REPA, and RAE) and text-to-image generation validate the effectiveness of our approach. Our approach MixFlow over the RAE models achieve strong generation results on ImageNet: 1.43 FID (without guidance) and 1.10 (with guidance) at 256 x 256, and 1.55 FID (without guidance) and 1.10 (with guidance) at 512 x 512.
Dec 22, 2025

OmniEgoCap: Device-Agnostic Sequence-Level Egocentric Motion Reconstruction

Commercial egocentric devices capture human behavior in everyday settings, yet full-body motion reconstruction must generalize across diverse cameras and mountings. This is challenging because head trajectories under-determine body motion, while hand observations are intermittent and camera-dependent: a hand may disappear simply because it has left the camera's field of view. Existing methods treat such absences as missing data, relying on fixed visibility regimes or post-hoc optimization, and fail to generalize across devices. We present OmniEgoCap, a sequence-level diffusion framework for device-agnostic egocentric full-body motion reconstruction. Our key insight is that egocentric videos contain sequence-level evidence about both the wearer and the camera-induced visibility structure, enabling OmniEgoCap to infer consistent body shape and coherent motion from sparse head and hand cues. To prevent overfitting to a single camera setup, we further introduce geometry-aware visibility augmentation, which synthesizes visibility patterns from realistic variations in camera geometry. We also present OmniEgoDB, the first mocap benchmark that pairs ground-truth motion with recordings from multiple consumer egocentric devices. Experiments on synthetic, real-device, and in-the-wild settings demonstrate superior motion reconstruction, robust cross-device generalization, and coherent in-the-wild results.
Dec 17, 2025

FADTI: Fourier and Attention Driven Diffusion for Multivariate Time Series Imputation

Multivariate time series imputation is fundamental in applications such as healthcare, traffic forecasting, and biological modeling, where sensor failures and irregular sampling lead to pervasive missing values. Existing Transformer- and diffusion-based imputers achieve strong performance, but they often rely mainly on time-domain modeling and lack adaptive spectral bias for recovering structured temporal gaps. We propose FADTI, a Fourier- and attention-driven diffusion framework for multivariate time series imputation. FADTI introduces a Fourier Bias Projection (FBP) module that injects learnable frequency-aware bias into intermediate hidden states during denoising. It projects intermediate hidden states onto Fourier bases, avoiding direct spectral estimation from masked or zero-filled inputs. With DFT, STFT, and FSST instantiations, FBP captures global periodicity, localized time--frequency variations, and non-stationary oscillatory patterns. By coupling FBP with self-attention and gated convolution, FADTI integrates frequency-domain guidance, temporal dependency modeling, and probabilistic denoising in a unified framework. Experiments on multiple benchmarks, including a new biological imputation benchmark, show that FADTI improves accuracy, uncertainty estimation, and sampling efficiency, especially under high missing rates and structured missing patterns. Code is available at https://github.com/RazeenLI/FADTI
Dec 16, 2025

4D-RaDiff: Latent Point Diffusion for 4D Radar Point Cloud Generation

Automotive radar has shown promising developments in environment perception due to its cost-effectiveness and robustness in adverse weather conditions. However, the limited availability of annotated radar data poses a significant challenge for advancing radar-based perception systems. To address this limitation, we propose a novel framework to generate 4D radar point clouds for training and evaluating object detectors. Unlike image-based diffusion, our method is designed to consider the sparsity and unique characteristics of radar point clouds by applying diffusion to a latent point cloud representation. Within this latent space, generation is controlled via conditioning at either the object or scene level. The proposed 4D-RaDiff converts unlabeled bounding boxes into high-quality radar annotations and transforms existing LiDAR point cloud data into realistic radar scenes. Experiments demonstrate that incorporating synthetic radar data of 4D-RaDiff as data augmentation method during training of object detection models consistently improves performance compared to training on real data only. In addition, pre-training on our synthetic radar data achieves competitive detection performance, providing a promising, scalable approach to reduce dependence on costly manual annotations.
Dec 16, 2025

Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Models

Masked Discrete Diffusion Models (MDMs) have achieved strong performance across a wide range of multimodal tasks, including image understanding, generation, and editing. However, their inference speed remains suboptimal due to the need to repeatedly process redundant masked tokens at every sampling step. In this work, we propose Sparse-LaViDa, a novel modeling framework that dynamically truncates unnecessary masked tokens at each inference step to accelerate MDM sampling. To preserve generation quality, we introduce specialized register tokens that serve as compact representations for the truncated tokens. Furthermore, to ensure consistency between training and inference, we design a specialized attention mask that faithfully matches the truncated sampling procedure during training. Built upon the state-of-the-art unified MDM LaViDa-O, Sparse-LaViDa achieves up to a 2x speedup across diverse tasks including text-to-image generation, image editing, and mathematical reasoning, while maintaining generation quality.
Dec 7, 2025

Latent Nonlinear Denoising Score Matching for Enhanced Learning of Structured Distributions

Score-based generative models commonly employ linear forward diffusions with Gaussian reference distributions, limiting their ability to directly incorporate structural information into the noising dynamics. We introduce latent nonlinear denoising score matching (LNDSM), which combines nonlinear, structure-informed latent diffusions with the variational autoencoder (VAE) training framework. In our construction, the forward diffusion is designed to admit a prescribed reference distribution, such as a Gaussian mixture model, that reflects qualitative features of the latent data distribution. We derive a computable approximation to a cross-entropy upper bound in the VAE objective using Euler-Maruyama discretization and the resulting Gaussian conditional transitions. To stabilize training, we remove two zero-mean control variates whose variances become large for small time steps. The resulting objective enables joint training of the encoder, decoder, and latent score network. Numerical experiments on benchmark image datasets demonstrate the effectiveness of LNDSM in learning structured distributions, outperforming conventional latent score-based generative modeling in sample quality and mode balance across several settings, while achieving improved training efficiency. These results highlight the potential of nonlinear, structure-informed latent diffusions for generative modeling.
Dec 2, 2025

Locality-Aware Continual Unlearning for Diffusion Models

Real-world deployment of text-to-image diffusion models requires continual concept removal as new privacy, copyright, or safety obligations arise over time. Existing unlearning methods, however, are designed for single-step deletion and collapse after only 3-5 sequential applications. We trace this instability to two compounding factors: (i) coarse mapping targets that cause degradation to accumulate unnecessarily across steps, and (ii) the absence of local protection for semantically neighboring concepts, whose shared internal representations make them the first to suffer collateral damage. Because this damage is strongest in the local semantic neighborhood of the forget concept, global replay alone cannot prevent it. Building on this analysis, we propose Locality-Aware Continual Unlearning (LACU), a framework with two complementary mechanisms. Locality-Aware Target Selection chooses, for each forget prompt, the context-preserving mapping prompt that the diffusion model itself treats as most similar to the original prompt, measured by score-prediction distance (how differently the model denoises the same noisy image under two text conditions), ensuring each update is as small and targeted as possible. Locality-Aware Replay uses the same metric to identify the retain concepts closest to the forget concept in the model's own representation and replays them as a local functional regularizer, directly shielding the most vulnerable neighborhood. Combined with teacher-student distillation and lightweight ℓ2\ell_2 parameter regularization, LACU maintains stable unlearning over 10 sequential steps, preserving significantly higher related retention (RRaccRR_{\text{acc}}) and general retention (GRaccGR_{\text{acc}}) than recent baselines.
Nov 26, 2025

E0: Expressive Fine-Grained Discrete Action Prediction for Vision-Language-Action Models via Tweedie Discrete Diffusion

Vision-Language-Action (VLA) models offer a unified framework for robotic manipulation by integrating visual perception, language understanding, and control generation. However, existing VLA systems still struggle to generalize across diverse tasks, scenes, and camera viewpoints, and often produce coarse or unstable actions. We argue that these limitations are closely tied to the structural properties of actions in VLA settings, including the inherent multi-peaked nature of action distributions, the token-based symbolic reasoning of pretrained VLM/VLA backbones, and the effective finite resolution imposed by real-world robotic control. Motivated by these properties, we introduce E0, a tweedie discrete diffusion framework that formulates action generation as iterative denoising over quantized action tokens. By operating in a discrete action space with a principled diffusion process, E0 naturally aligns with token-based reasoning, supports fine-grained yet executable action control, and avoids the distributional mismatch of masking-based discrete diffusion. Experiments on LIBERO, VLABench, ManiSkill, and a real-world Franka arm demonstrate that E0 achieves state-of-the-art performance across diverse environments, outperforming strong baselines by 8.1% on average.
Nov 25, 2025

Diversifying Long Prompt Image Generation through Structured Prompt Embedding Space Sampling

Modern text-to-image models produce impressive visual results from richly specified prompts, yet their behavior under long prompts remains insufficiently understood. In this paper, we study a practical failure mode in which accumulated semantic constraints progressively suppress output variation, causing diversity to collapse even when many visual factors remain unspecified. We show that this phenomenon appears consistently across recent generation models as prompt length increases, and provide a theoretical motivation that connects long-prompt conditioning with reduced sampling entropy in the prompt embedding space. Based on this observation, we introduce PromptMoG, a training-free approach that samples prompt embeddings from a Mixture-of-Gaussians distribution to restore generative flexibility while maintaining semantic fidelity. To support systematic evaluation, we further present LPD-Bench, a structured benchmark of long and semantically dense prompts for measuring both fidelity and diversity under compositional text conditioning. Extensive experiments on four large-scale diffusion models, including SD3.5-Large, Flux.1-Krea-Dev, CogView4, and Qwen-Image, show that PromptMoG consistently improves diversity for long-prompt image generation. The code is publicly available at https://github.com/basiclab/PromptMoG.
Nov 25, 2025

SONIC: Spectral Optimization of Noise for Inpainting with Consistency

We propose a novel training-free method for inpainting with off-the-shelf text-to-image models. While guidance-based methods in theory allow generic models to be used for inverse problems such as inpainting -- in practice their effectiveness is limited, leading to the necessity of specialized inpainting-specific models. In this work, we argue the missing ingredient for training-free generic model usage is proper optimization of the initial noise sample. We optimize the initial noise to approximately reproduce the unmasked image, in as few as tens of optimization steps, then use it with a conventional training-free inpainting method. Critically, we propose two core ideas that make this possible: (i) we perform linear approximation that avoids the costly and often impractical unrolling required to relate the initial noise sample to model output -- which potentially is why this relationship was previously overlooked; and (ii) perform spectral preconditioning by optimizing the initial noise sample in the spectral domain with Adam, which stabilizes the optimization. We demonstrate our method on various inpainting tasks, outperforming the state of the art. Project website: https://ubc-vision.github.io/sonic/
Nov 24, 2025

Do View-Consistent Representations Matter for Camera-Controlled Video Generation?

Camera-controlled video generation has attracted increasing interest for its applications in simulation, gaming, and world modeling. However, current models often deform rigid objects and scene structure as the viewpoint changes. We investigate whether this failure is related to a measurable internal property: the view consistency of diffusion representations. Across a diverse collection of camera-controlled image-to-video diffusion models, we find that models exhibiting more view-consistent representations generate videos with superior 3D consistency. We complement this correlation-based evidence with a within-model controlled intervention, ViCoDR, which directly supervises the view consistency of internal diffusion representations, during training. Across camera-controlled image-to-video, camera-controlled text-to-video, and multi-view generation tasks, ViCoDR consistently improves output 3D consistency. Finally, we evaluate the loss used for these interventions as an auxiliary training objective by retraining a camera-controlled I2V model using either ViCoDR or existing methods for supervising diffusion representations under the same training setup. We show that the ViCoDR loss improves output 3D consistency more reliably than the alternatives and remains effective across design variants, while exhibiting camera-control trade-offs and requiring additional training computation. Together, these results establish view consistency of internal diffusion representations as an important property for analyzing and improving camera-controlled video generation. Project page: https://danier97.github.io/ViCoDR.
Nov 21, 2025

DeltaDeno: Zero-Shot Anomaly Generation via Delta-Denoising Attribution

Anomaly generation is often framed as few-shot fine-tuning with anomalous samples, which contradicts the scarcity that motivates generation and tends to overfit category priors. We tackle the setting where no real anomaly samples or training are available. We propose Delta-Denoising (DeltaDeno), a training-free zero-shot anomaly generation method that localizes and edits defects by contrasting two diffusion branches driven by a minimal prompt pair under a shared schedule. By accumulating per-step denoising deltas into an image-specific localization map, we obtain a mask to guide the latent inpainting during later diffusion steps and preserve the surrounding context while generating realistic local defects. To improve stability and control, DeltaDeno performs token-level prompt refinement that aligns shared content and strengthens anomaly tokens, and applies a spatial attention bias restricted to anomaly tokens in the predicted region. Experiments on public datasets show that DeltaDeno achieves great generation, realism and consistent gains in downstream detection performance. Code will be made publicly available at https://github.com/CROVO1026/DeltaDeno.
Nov 11, 2025

VEDA: Generation of 3D Molecules via Variance-Exploding Diffusion with Annealing

Diffusion models show promise for 3D molecular generation, but face a fundamental trade-off between sampling efficiency and conformational accuracy. While flow-based models are fast, they often produce geometrically inaccurate structures, as they have difficulty capturing the multimodal distributions of molecular conformations. In contrast, denoising diffusion models are more accurate but suffer from slow sampling, a limitation attributed to sub-optimal integration between diffusion dynamics and SE(3)-equivariant architectures. To address this, we propose VEDA, a unified SE(3)-equivariant framework that combines variance-exploding diffusion with annealing to efficiently generate conformationally accurate 3D molecular structures. Specifically, our key technical contributions include: (1) a VE schedule that enables noise injection functionally analogous to simulated annealing, improving 3D accuracy and reducing relaxation energy; (2) a novel preconditioning scheme that reconciles the coordinate-predicting nature of SE(3)-equivariant networks with a residual-based diffusion objective, and (3) a new arcsin-based scheduler that concentrates sampling in critical intervals of the logarithmic signal-to-noise ratio. On the QM9 and GEOM-DRUGS datasets, VEDA matches the sampling efficiency of flow-based models, achieving state-of-the-art valency stability and validity with only 100 sampling steps. More importantly, VEDA's generated structures are remarkably stable, as measured by their relaxation energy during GFN2-xTB optimization. The median energy change is only 1.72 kcal/mol, significantly lower than the 32.3 kcal/mol from its architectural baseline, SemlaFlow. Our framework demonstrates that principled integration of VE diffusion with SE(3)-equivariant architectures can achieve both high chemical accuracy and computational efficiency.
Nov 8, 2025

A PDE Perspective on Generative Diffusion Models

Score-based diffusion models have emerged as a powerful class of generative methods, with successful applications across diverse domains. Despite their empirical success, their mathematical foundations remain only partially understood, particularly regarding the stability and consistency of the stochastic and partial differential equations underlying their dynamics. In this work, we develop a partial differential equation (PDE) framework for score-based diffusion processes associated with the heat flow of a compactly supported data measure. After establishing weak well-posedness on positive-time intervals, we derive a sharp one-sided divergence bound for the score and use it to obtain sharp uniform LpL^p-stability estimates for the generative Fokker--Planck equation. This bound is the Euclidean form of the Li--Yau differential inequality. Through entropy stability methods, we prove that the generated laws converge in the 22-Wasserstein distance at rate t\sqrt t as t→0+t\to0^+ to a probability measure supported on the data support. This holds for both deterministic and stochastic generation, for positive terminal densities with finite absolute entropy and second moment. In the deterministic case, almost every trajectory converges to a point of the data support; for empirical data, we further obtain a trajectory-wise square-root rate. These results clarify how exact-score dynamics concentrate on the data support without necessarily recovering the data distribution. An explicitly solvable case identifies the limiting weights and their dependence on Gaussian initialization, while a relative-entropy estimate quantifies the effect of score approximation. Together, these findings connect the stability and concentration mechanisms with score learning and early stopping, providing a mathematical perspective on the balance between imitation fidelity and generative diversity.
Nov 8, 2025

AD-DAE: Alzheimer's Disease Progression Modeling with Unpaired Longitudinal MRI using Diffusion Auto-Encoders

Generative modeling frameworks have emerged as an effective approach to capture high-dimensional image distributions from large datasets without requiring domain-specific knowledge, a capability essential for disease progression modeling. Recent generative approaches have attempted to capture progression by mapping images to a latent space and guiding representations to generate follow-up images from previous time points. However, these methods impose constraints on distribution learning, resulting in latent spaces with limited controllability for generating follow-up images without paired subject-specific longitudinal guidance. In order to enable controlled movements in the latent representational space and generate progression images from a previous time-point image without subject-specific guidance, we introduce a conditionable Diffusion Auto-encoder framework that forms a compact latent space capturing high-level semantics and providing means to control generation. Our approach leverages this latent space to condition and apply controlled shifts to the representations of previous time-point images by isolating progression and subject identity information for generating follow-up images. The shifts are implicitly guided by correlating with progression attributes and constraining to Alzheimer's disease specific regions, without paired longitudinal guidance. We validate the generations through image quality metrics, volumetric progression analysis, and downstream tasks in Alzheimer's disease datasets from different sources. This demonstrates the effectiveness of our approach for Alzheimer's progression modeling and longitudinal image generation.
Nov 5, 2025

Inference-Time Optimization of Prompt Embeddings in Diffusion Models: A Comparison of sep-CMA-ES and Adam

Deep diffusion models have revolutionized image generation by producing high-quality outputs. However, achieving specific objectives with these models often requires costly adaptations such as fine-tuning, which can be resource-intensive and time-consuming. An alternative approach is inference-time control, which involves optimizing the prompt embeddings to guide the generation process without altering the model weights. We explore prompt-embedding search optimization for the Stable Diffusion XL Turbo model, comparing a gradient-free evolutionary approach, the Separable Covariance Matrix Adaptation Evolution Strategy (sep-CMA-ES), against the widely used gradient-based optimizer Adaptive Moment Estimation (Adam). Candidate images are evaluated by a weighted objective that combines LAION Aesthetic Predictor V2 and CLIPScore, enabling explicit trade-offs between aesthetic quality and prompt-image alignment. On 36 prompts sampled from Parti Prompts (P2) under three weight settings (aesthetics-only, balanced, alignment-only), sep-CMA-ES consistently achieves higher objective values than Adam. We additionally analyze divergence from the unoptimized baseline using cosine similarity and SSIM and report the compute and memory footprints. These results suggest that sep-CMA-ES is an effective inference-time optimizer for prompt-embedding search, improving aesthetics-alignment trade-offs and resource usage without model fine-tuning.
Oct 31, 2025

Rethinking Robust Adversarial Concept Erasure in Diffusion Models

Concept erasure methods aim to remove specific unsafe target concepts in diffusion models while preserving image generation utility. To address the vulnerability that erased concepts can be easily recovered under adversarial attacks, adversarial concept erasure methods integrate adversarial optimization into the concept erasure process. However, existing adversarial concept erasure methods face a trade-off between robustness and computational cost. We attribute this to adversarial optimization techniques that use random samples to approximate the adversarial objective function. Adversarial optimization that uses a small number of samples fails to produce adversarial embeddings that accurately capture the target concept space. To mitigate this limitation, we propose Semantic-Guided Adversarial Optimization, which uses a single sample to produce adversarial embeddings that better capture the target concept space. We also propose Semantic-Guided Concept Erasure, which automatically maps the target concept to a semantically similar surrogate. Extensive experiments on not-safe-for-work content, artistic styles, and object-related concepts demonstrate that our method, S-GRACE (Semantic-Guided Robust Adversarial Concept Erasure) achieves state-of-the-art erasure robustness and superior image generation utility, with significantly lower computational cost than existing methods. Our code is available at https://github.com/Qhong-522/S-GRACE.
Oct 29, 2025

MedForj: An open, large-scale foundational generative prior for high-resolution 3D brain MRI

This work introduces MedForj, a suite of 3D foundational generative priors based on diffusion models. The MedForj models were trained on 72,65972{,}659 1~mm isotropic 3D T1T_1-weighted MRI human brain image volumes from 38,17438{,}174 subjects, drawn from a curated corpus of 80,67580{,}675 volumes from 42,50642{,}506 subjects spanning 3838 publicly available datasets. These training images were manually inspected to exclude those with poor quality and excessive pathology, and otherwise were minimally processed. The models include six different diffusion training strategies: rectified flow, latent diffusion rectified flow, flow matching, velocity prediction, clean prediction, and noise prediction. Image samples produced by each of these models were compared to each other and against real, ground truth data under downstream segmentation distributions, FID, five inverse problems, and blind human inspection in an observer study. Flow matching was the strongest strategy overall, achieving the best inverse problem solving results at 28.8028.80~dB PSNR and 0.8740.874 SSIM averaged over the five forward problems, the highest rate of reconstructions judged real by blind human raters at 72.6%72.6\%, and the closest per-structure match to real segmented anatomy in a permutation test. It was not best everywhere: rectified flow produced the most convincing unconditional samples in the observer study and the best FID, and the latent rectified-flow model achieved the smallest joint distributional distance to real anatomy. No other strategy, however, performed consistently well across all four evaluations. We therefore recommend flow matching as the default MedForj prior, while releasing every strategy so that the choice can be revisited per application. All model weights and corresponding code are publicly available at https://github.com/piksl-research/medforj.
Oct 27, 2025

Multi-Agent Conditional Diffusion Planning for Decentralized Wireless Resource Allocation

In wireless communication systems, efficient and adaptive resource allocation is essential for enhancing overall Quality of Service (QoS). Compared with conventional Model-Free Reinforcement Learning (MFRL), Model-Based RL (MBRL) first learns a generative world model for subsequent planning, enabling more stable training through the reuse of historical experience. However, deploying MBRL in large-scale wireless networks remains challenging due to high-dimensional stochastic dynamics, strong inter-agent cooperation, and stringent communication constraints. To overcome these challenges, we propose a Multi-Agent Conditional Diffusion Model Planner (MA-CDMP) for decentralized communication resource management. Following the Decentralized Training with Decentralized Execution (DTDE) paradigm, MA-CDMP models each communication node as an autonomous agent and employs Diffusion Models (DMs) to capture environment dynamics. An inverse dynamics model is further incorporated to guide action generation, improving sample efficiency and policy scalability. Moreover, to efficiently approximate large-scale agent interactions, we introduce a Mean Field (MF) mechanism to assist the classifier in conditional DMs. This mitigates inter-agent non-stationarity and enhances cooperation with minimal communication overhead. We further establish an upper bound on the distributional approximation error induced by MF-based diffusion generation, guaranteeing convergence stability and reliable modeling of multi-agent stochastic dynamics. Extensive experiments demonstrate that MA-CDMP consistently outperforms existing MARL baselines in terms of average reward and QoS metrics, showcasing its scalability and practicality for real-world wireless network optimization.
Oct 26, 2025

Beyond the Semicircle: Free Diffusion Models with Prescribed Equilibria

A growing class of machine-learning objects -- covariance and Gram matrices, kernel and attention matrices, MIMO channel matrices, density operators -- are naturally spectra rather than coordinate vectors. Building a denoising diffusion model for such data by corrupting eigenvalues coordinatewise is not merely elegant: it converges to the wrong limit, because eigenvalues repel rather than move independently. Free probability theory supplies the correct forward process, with free convolution replacing classical convolution and Voiculescu's conjugate variable replacing the score, but constant-coefficient free diffusions have a hidden limitation of their own: their only possible equilibrium is the semicircular law, whatever the target distribution looks like. We show that state-dependent free volatility removes this restriction. For any sufficiently regular compactly supported target law, we explicitly construct, through a closed-form Hilbert-transform drift, a free Fokker--Planck flow having that law as a stationary spectral distribution. Under an additional convex-potential condition, the construction admits a globally relaxing diffusion interpretation, reverse-time dynamics, and a trainable score. The generalization is genuine rather than cosmetic: no change of spectral variable reduces it to the constant-coefficient case, and it is a Wasserstein gradient flow only for an explicitly characterized family of coefficients that contains no bounded non-constant member. We instantiate the construction on the Marchenko--Pastur law, the limiting spectrum of isotropic sample covariance matrices, and validate it numerically: the free-versus-coordinatewise gap, an exact dimension-independent score driving reverse-time matrix dynamics at several matrix sizes, the designed Marchenko--Pastur equilibrium, and end-to-end generation with a learned score.
Oct 20, 2025

Not Every Time and Frequency Need to Be Forgotten in Diffusion Unlearning

Data unlearning aims to remove the influence of specific training samples from a trained model. In fine-tuning methods, data unlearning relies primarily on loss maximization over forget samples, which often leads to quality degradation or incomplete forgetting. Existing methods perform unlearning uniformly across diffusion stages, ignoring diffusion dynamics from noise to data. Our systematic study of diffusion phases shows that forgetting in diffusion models is uneven across time and frequency, with theoretical justification of distributive distortion and forgetting-utility trade-off. By selectively forgetting time and frequency in diffusion models, we achieve both higher unlearning success rates and improved generation quality across diverse settings, including both conditional and unconditional scenarios. We also introduce an improved SSCD metric that measures dissimilarity using a normalized perturbation distance. Together, we provide practical insights for understanding and improving data unlearning in diffusion models.
Oct 14, 2025

A Connection Between Score Matching and Local Intrinsic Dimension

The local intrinsic dimension (LID) of data is a fundamental quantity in signal processing and learning theory, but quantifying the LID of high-dimensional, complex data has been a historically challenging task. Recent works have discovered that diffusion models capture the LID of data through the spectra of their score estimates and through the rate of change of their density estimates under various noise perturbations. While these methods can accurately quantify LID, they require either many forward passes of the diffusion model or use of gradient computation, limiting their applicability in compute- and memory-constrained scenarios. We show that the LID is a lower bound on the denoising score matching loss, motivating use of the denoising score matching loss as a LID estimator. Moreover, we show that the equivalent implicit score matching loss also approximates LID via the normal dimension and is closely related to a recent LID estimator, FLIPD. Our experiments on a manifold benchmark and with Stable Diffusion 3.5 indicate that the denoising score matching loss is a highly competitive and scalable LID estimator, achieving superior accuracy and memory footprint under increasing problem size and quantization level.
Oct 3, 2025

Multi-task Neural Diffusion Processes

Neural diffusion processes provide a scalable, non-Gaussian approach to modelling distributions over functions, but existing formulations are limited to single-task inference and do not capture dependencies across related tasks. We propose multi-task neural diffusion processes (MT-NDPs), an extension that incorporates a task encoder to enable task-conditioned probabilistic regression and few-shot adaptation across related functions. The task encoder extracts a low-dimensional representation from context observations and conditions the diffusion model on this representation, allowing information sharing across tasks while accommodating variable-sized context sets and preserving permutation invariance with respect to the ordering of context observations. We evaluate MT-NDPs through synthetic experiments and two real-world applications involving wind turbine data and school examination outcomes. The results demonstrate improved predictive accuracy relative to single-task diffusion baselines when context observations are available, alongside competitive performance against neural-process and Gaussian-process alternatives. In the wind-energy application, MT-NDPs achieve the lowest average coverage error among the evaluated models at larger context sizes, demonstrating improved predictive uncertainty calibration. Overall, our findings highlight the potential of task-conditioned diffusion models for uncertainty-aware multi-task regression and few-shot adaptation to previously unseen tasks.
Oct 3, 2025

DELTA: Robustly Training Label-Conditional Diffusion Models with Weak Annotations

While label-conditional diffusion models have recently exhibited remarkable generative capabilities, their success heavily relies on massive, cleanly labeled datasets. In practice, categorical supervision is rarely perfect: it is often corrupted by noise, clouded by ambiguity, or partially missing. Training directly on such weak annotations severely degrades generation quality, and existing robust methods offer only fragmented solutions that depend on scarce auxiliary priors. To overcome these limitations, we propose DELTA, a unified framework that robustly trains Diffusion modELs across all three weak annoTAtion types. By treating the unknown true label as a latent variable, DELTA optimizes a principled variational objective that jointly recovers the clean data distribution and infers the true label posteriors. To make this joint training computationally tractable, we further introduce a median-centered timestep sampling strategy that efficiently concentrates estimation where the diffusion process is most informative. Extensive experiments on synthetic and real-world benchmarks demonstrate that DELTA produces high-fidelity, class-consistent samples across diverse weak supervision scenarios, outperforming specialized baselines while demanding strictly less prior knowledge.
Oct 1, 2025

Affordance-Guided Diffusion Prior for 3D Hand Reconstruction

How can we reconstruct 3D hand poses when large portions of the hand are heavily occluded by itself or by objects? Humans often resolve such ambiguities by leveraging contextual knowledge -- such as affordances, where an object's shape and function suggest how the object is typically grasped. Inspired by this observation, we propose a generative prior for hand pose refinement guided by affordance-aware textual descriptions of hand-object interactions (HOI). Our method employs a diffusion-based generative model that learns the distribution of plausible hand poses conditioned on affordance descriptions, which are inferred from a large vision-language model (VLM). This enables the refinement of occluded regions into more accurate and functionally coherent hand poses. Extensive experiments on HOGraspNet, a 3D hand-affordance dataset with severe occlusions, demonstrate that our affordance-guided refinement significantly improves hand pose estimation over both recent regression methods and diffusion-based refinement lacking contextual reasoning.