cs.LGJun 11, 2026

Towards More General Control of Diffusion Models Using Jeffrey Guidance

Authors: Raphaël Razafindralambo, Rémy Sun, Frédéric Precioso, Jes Frellsen, Pierre-Alexandre Mattei

Abstract

A key strength of diffusion models lies in their flexibility, since their outputs can be controlled at sampling time through guidance. However, beyond simple cases such as conditional sampling, the target distribution is often left implicit, defined only through a sampling rule or a heuristic energy function. To address this, we propose Jeffrey guidance, a principled framework that extends diffusion-model control to applications beyond what standard guidance can express. It leverages Jeffrey's rule of conditioning to update marginal distributions towards a prescribed target, preserving the conditional structure and minimally perturbing the joint distribution. We first demonstrate Jeffrey guidance by targeting a prescribed embedding distribution. With Inception embeddings as the target, this leads to substantial reductions in FID on both CIFAR-10 and FFHQ. We further apply Jeffrey guidance to fairness on CelebA-HQ, updating an unconditional diffusion model to enforce independence between attributes.

Explore similar work

CardsList
  1. Spectral Guidance for Flexible and Efficient Control of Diffusion Models

    May 27, 2026Gabriel Moreira, Manuel Marques, João Paulo Costeira +1Generative Processes

  2. General and Efficient Steering of Diffusion Models

    Feb 11, 2026Qingsong Wang, Mikhail Belkin, Yusu WangActivation Steering