Overview

Diffusion model guidance studies how to steer reverse denoising so generated samples satisfy goals such as matching text or images, honoring observations, meeting physical or safety constraints, and avoiding unwanted content. Common approaches include classifier-free and classifier guidance, gradients from reward or measurement models, attention and representation edits, and particle-based methods such as sequential Monte Carlo and Doob h-transforms. These techniques are used across text-to-image and video generation, image restoration, molecular design, and robot planning, with evaluation ranging from compositional-alignment benchmarks to inverse-problem fidelity and task success. A central tension is that stronger or more elaborate guidance can improve control but also damage realism, diversity, or fidelity to the model’s learned distribution, while adding computation; no alternative to standard classifier-free guidance is consistently best across settings.

Latest papers

397 papers33 papers in the last four weeks, down 38% on the four weeks before. 0.3% of all new papers.

May 27, 2025

ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generation

Recent open-weight text-to-image (T2I) diffusion models still struggle with multi-instance prompts, often omitting or merging instances and mixing semantics among similar objects. We trace these failures to early denoising steps, before instance boundaries are reliably stabilized. Existing training-free guidance is largely driven by cross-attention or other token-conditioned semantic signals. Such guidance can separate concepts at the token level, but largely assumes that distinct instance regions have already emerged. In early denoising steps, it cannot reliably carve out these regions, so count failures and semantic mixing persist. By contrast, self-attention exposes class-agnostic instance layouts during early denoising. To exploit this asymmetry, we propose ISAC\textbf{ISAC} (I\textbf{I}nstance-to-S\textbf{S}emantic A\textbf{A}ttention C\textbf{C}ontrol), a training-free, model-agnostic objective that first stabilizes self-attention layouts and then binds cross-attention semantics within them, without fine-tuning or external vision models. Across T2I-CompBench, HRS-Bench, and our newly curated IntraCompBench, ISAC consistently outperforms prior training-free methods. Furthermore, ISAC enhances layout-to-image controllers by refining coarse, overlapping bounding boxes into dense instance masks. Code and IntraCompBench are available at https://shjo-april.github.io/ISAC.
Feb 6, 2025

DICE: Distilling Classifier-Free Guidance into Text Embeddings

Text-to-image diffusion models are capable of generating high-quality images, but suboptimal pre-trained text representations often result in these images failing to align closely with the given text prompts. Classifier-free guidance (CFG) is a popular and effective technique for improving text-image alignment in the generative process. However, CFG introduces significant computational overhead. In this paper, we present DIstilling CFG by sharpening text Embeddings (DICE) that replaces CFG in the sampling process with half the computational complexity while maintaining similar generation quality. DICE distills a CFG-based text-to-image diffusion model into a CFG-free version by refining text embeddings to replicate CFG-based directions. In this way, we avoid the computational drawbacks of CFG, enabling high-quality, well-aligned image generation at a fast sampling speed. Furthermore, examining the enhancement pattern, we identify the underlying mechanism of DICE that sharpens specific components of text embeddings to preserve semantic information while enhancing fine-grained details. Extensive experiments on multiple Stable Diffusion v1.5 variants, SDXL, and PixArt-αα demonstrate the effectiveness of our method. Code is available at https://github.com/zju-pi/dice.
Feb 4, 2025

From Uncertain to Safe: Conformal Adaptation of Diffusion Models for Safe PDE Control

The application of deep learning for partial differential equation (PDE)-constrained control is gaining increasing attention. However, existing methods rarely consider safety requirements crucial in real-world applications. To address this limitation, we propose Safe Diffusion Models for PDE Control (SafeDiffCon), which introduce the uncertainty quantile as model uncertainty quantification to achieve optimal control under safety constraints through both post-training and inference phases. Firstly, our approach post-trains a pre-trained diffusion model to generate control sequences that better satisfy safety constraints while achieving improved control objectives via a reweighted diffusion loss, which incorporates the uncertainty quantile estimated using conformal prediction. Secondly, during inference, the diffusion model dynamically adjusts both its generation process and parameters through iterative guidance and fine-tuning, conditioned on control targets while simultaneously integrating the estimated uncertainty quantile. We evaluate SafeDiffCon on three control tasks: 1D Burgers' equation, 2D incompressible fluid, and controlled nuclear fusion problem. Results demonstrate that SafeDiffCon is the only method that satisfies all safety constraints, whereas other classical and deep learning baselines fail. Furthermore, while adhering to safety constraints, SafeDiffCon achieves the best control performance. The code can be found at https://github.com/AI4Science-WestlakeU/safediffcon.
Dec 10, 2024

Rethinking Negative Prompting for Concept Suppression: Unconditional Anchoring and Localized Guidance

Text-to-image diffusion models have rapidly transitioned from research prototypes to widely deployed tools. Along with their impressive capabilities, these models can also generate harmful or undesirable content (e.g., NSFW imagery), motivating practical mechanisms for removing such concepts. Negative prompting, which uses the negative-prompt prediction as the guidance anchor, is among the most widely used mechanisms for steering sampling trajectories away from unwanted concepts. However, for jailbreak prompts, it can fail in a regime we refer to as guidance collapse, where the model's denoising predictions for the prompt and negative prompt become similar, so that the guidance difference driving negative prompting vanishes and the guided prediction degenerates to the negative-prompt prediction, undermining suppression. To fix this issue, we propose TraSCE (Trajectory Steering for ConcEpt Suppression), a training-free inference-time method that makes negative prompting reliable under guidance collapse. TraSCE combines two complementary components: (i) unconditional anchoring, which makes the fallback under guidance collapse a neutral unconditional prediction rather than the negative-prompt prediction, and (ii) an alignment-localized steering step, which deliberately induces this collapse for adversarial prompts and steers the sample to the unconditional fallback, while remaining minimally intrusive on benign prompts. We evaluate TraSCE on adversarial prompt benchmarks targeting nudity and violence, and further demonstrate its applicability to suppressing artistic styles and objects. Across these settings, TraSCE substantially improves the robustness of negative prompting to jailbreak prompts without any training, weight updates, or pre-collected concept-specific data (prompts or images), while keeping the impact on general image quality small.
Dec 3, 2024

Filterless Snapshot Hyperspectral Imaging using Guided Patch Diffusion

We consider the problem of reconstructing a HxWx31 hyperspectral image from a H×WH\times W grayscale snapshot measurement that is captured using only a single diffractive lens and a filterless panchromatic photosensor. This problem is severely ill-posed, but we present a model that produces high-quality results in simulation and experiment. We make efficient use of limited training data by creating a conditional denoising diffusion model that operates on small patches in a shift-invariant manner. During inference, we synchronize per-patch hyperspectral predictions using guidance by physical consistency with the system's optical point spread function. Our experiments reveal that the patch size can be as small as the point spread function, with local optical cues being the main source of information about complete spectra. Also, by drawing multiple samples, our model provides per-pixel uncertainty estimates that strongly correlate with reconstruction error.
Nov 26, 2024

ContrastiveCFG: Guiding Diffusion Sampling by Contrasting Positive and Negative Concepts

As Classifier-Free Guidance (CFG) has proven effective in conditional diffusion model sampling for improved condition alignment, many applications use a negated CFG term as a Negative Prompting (NP) to filter out unwanted features from samples. However, simply negating CFG guidance creates an inverted probability distribution, often distorting samples away from the marginal distribution. Inspired by recent advances in conditional diffusion models for inverse problems, here we present a novel method to achieve guidance toward the given condition using contrastive loss. Specifically, our guidance term aligns or repels the denoising direction based on the given condition through contrastive loss, achieving a similar guiding effect to traditional CFG for positive conditions while overcoming the limitations of existing negative guidance methods. Experimental results demonstrate that our approach effectively injects or removes the given concepts while maintaining sample quality across diverse scenarios, from simple class conditions to complex and overlapping text prompts.
Aug 5, 2024

ProCreate, Don't Reproduce! Propulsive Energy Diffusion for Creative Generation

In this paper, we propose ProCreate, a simple and easy-to-implement method to improve sample diversity and creativity of diffusion-based image generative models and to prevent training data reproduction. ProCreate operates on a set of reference images and actively propels the generated image embedding away from the reference embeddings during the generation process. We propose FSCG-8 (Few-Shot Creative Generation 8), a few-shot creative generation dataset on eight different categories -- encompassing different concepts, styles, and settings -- in which ProCreate achieves the highest sample diversity and fidelity. Furthermore, we show that ProCreate is effective at preventing replicating training data in a large-scale evaluation using training text prompts. Code and FSCG-8 are available at https://github.com/Agentic-Learning-AI-Lab/procreate-diffusion-public. The project page is available at https://procreate-diffusion.github.io.