Controllable Diverse Sampling through Pullback Directions

Sylvain Dehayem Kenfouo; Aaron Zweig; Mingxuan Zhang; Elham Azizi

cs.AIOct 9, 2026

Controllable Diverse Sampling through Pullback Directions

Authors: Sylvain Dehayem Kenfouo, Aaron Zweig, Mingxuan Zhang, Elham Azizi

Organizations: École Polytechnique, Institut Polytechnique de Paris · New York Genome Center · Irving Institute for Cancer Dynamics, Columbia University · Department of Systems Biology, Columbia University ·

Abstract

Conditional diffusion models can generate high-quality images that align closely with a given prompt. However, samples generated from different random seeds may remain visually similar, limiting creative exploration. Existing approaches can increase variation, but often require additional optimization or sampling overhead, while stronger perturbations may introduce distortions that degrade image quality or prompt alignment. We propose a training-free method that leverages the local pullback geometry induced by the mapping from the conditioning-embedding space to the image space to identify directions of maximal output sensitivity. By steering different sampling trajectories along these directions, we encourage distinct generations while maintaining consistency with the conditioning signal. Experiments on MS-COCO and BrushNet show the efficacy of the method in improving diversity on prompt-conditioned image generation and inpainting.

Figures & tables

Figure 1: Qualitative comparison on a BrushBench example with BrushNet-SDXL. Each row contains masked samples (see Figure 2 in the Appendix). Clean DDIM, CADS, and TPSO introduce local changes, whereas adaptive pullback produces more pronounced variations.

Stable Diffusion 1.5
	Alignment	Diversity
Method	CLIP $\uparrow$	MSS $\downarrow$	Vendi $\uparrow$	DINO Mean $\downarrow$
Clean DDIM	$\mathbf{31.3957}\pm 2.4730$	$0.3359\pm 0.0519$	$4.6831\pm 0.1971$	$0.5945\pm 0.1638$
CADS	$31.2537\pm 2.4253$	$0.3066\pm 0.0433$	$4.7833\pm 0.1426$	$0.5469\pm 0.1628$
TPSO	$30.9592\pm 2.2941$	$\mathbf{0.3048}\pm 0.0430$	$\mathbf{4.7838}\pm 0.1464$	$\mathbf{0.5143}\pm 0.1729$
Pullback (ours)	$31.3153\pm 2.4671$	$0.3194\pm 0.0501$	$4.7377\pm 0.1833$	$0.5663\pm 0.1661$

Table 1: Generation results on MSCOCO 2017 validation captions with five generated images per caption. Stable Diffusion 1.5 and XL results use all 500 captions. Entries report mean

\pm

standard deviation across captions. The best mean within each backbone is shown in bold.

Stable Diffusion 1.5
	Alignment	Diversity
Method	CLIP $\uparrow$	DINO $\downarrow$	MSS $\downarrow$	Vendi $\uparrow$
Clean DDIM	$\mathbf{32.0003}\pm 2.7383$	$0.8930\pm 0.1094$	$0.9643\pm 0.0323$	$1.1746\pm 0.1441$
CADS	$31.9284\pm 2.7278$	$0.8736\pm 0.1215$	$0.9594\pm 0.0367$	$1.1973\pm 0.1642$
TPSO	$31.9825\pm 2.7077$	$0.8807\pm 0.1174$	$0.9621\pm 0.0336$	$1.1844\pm 0.1490$
Pullback (ours)	$31.8796\pm 2.8055$	$\mathbf{0.8504}\pm 0.1336$	$\mathbf{0.9498}\pm 0.0414$	$\mathbf{1.2380}\pm 0.1817$

Table 2: BrushBench inpainting results for Stable Diffusion 1.5 and Stable Diffusion XL, with five images per example. Stable Diffusion 1.5 and XL results use all 500 examples. Entries report mean

\pm

standard deviation across examples. The best result within each backbone is shown in bold.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Illustration of the text-guided inpainting task. Given a text prompt, the model regenerates the masked region while preserving the unmasked region.

Figure 3: Token-dependent cross-attention at DDIM timestep $t=499$ . Each visualization averages 28 U-Net cross-attention maps over attention heads and layers. Warmer regions receive greater relative attention for the corresponding token.

Figure 4: Effect of the perturbation strength along latent-feature pullback directions. Rows correspond to the four dominant directions $v_{1},\ldots,v_{4}$ of the pullback metric $G_{x}=J_{x}^{\top}J_{x}$ , where $J_{x}$ is the Jacobian of the selected U-Net feature representation with respect to the noisy latent $x_{t}$ . Columns show the clean sample followed by increasing strengths. Moderate strengths produce predominantly localized, whereas larger strengths introduce visible distortions.

Figure 5: Effect of the perturbation strength along conditioning-space pullback directions. Rows correspond to the four dominant directions of the condition pullback metric $G_{C}=J_{C}^{\top}J_{C}$ , where $J_{C}=\partial\epsilon_{\theta}(x_{t},t,C)/\partial C$ . Columns show the clean sample followed by increasing strengths. The different directions produce distinct global semantic transformations, while large strengths can reduce image quality and fidelity.

Figure 6: Qualitative comparison between latent-feature and conditioning-space pullback directions computed from the same anchor. The top row shows the clean sample and the four dominant latent-feature directions at $s=0.025$ , using an $x$ -space guidance scale of $\gamma=5$ . The bottom row shows the clean sample and the four dominant conditioning directions at $s=1$ . In this example, the latent-feature directions primarily affect localized appearance, pose, and nearby image structure, whereas the conditioning-space directions induce broader semantic changes. All results use the same prompt, initial noise, and reverse-process parameters.

Image Generation

Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.

See all 469 papers

DICT: Data Injection and Contrastive Trajectory Refinement for Conditional Image Generation with Diffusion Models
Jul 4, 2026Chunnan Shang, Xin Zhang, Zhizhong Wang +1Diffusion Models for Image Restoration Diffusion Model Guidance
Zhejiang University, Hangzhou, China
TPSO: Training-Free Diverse Image Generation via Semantic Prompt Embedding Optimization
Nov 25, 2025Debin Meng, Chen Jin, Zheng Gao +3T2I Generation T2I Diffusion Models
Queen Mary University of London · Centre for AI, AstraZeneca, UK · University of Bedfordshire +1
Don't Settle at the Mode! Mitigating Diversity Collapse in Pretrained Flow Models via Feature Self-Guidance
Jun 25, 2026Pradhaan S Bhat, Rishubh Parihar, Abhijnya Bhat +1Flow Matching T2I Generation
Indian Institute of Science · Stanford University

Jul 4, 2026J/K move · Enter open · S save

DICT: Data Injection and Contrastive Trajectory Refinement for Conditional Image Generation with Diffusion Models

Chunnan Shang, Xin Zhang, Zhizhong Wang, Hongwei Wang

Zhejiang University, Hangzhou, China

Diffusion Models for Image Restoration Diffusion Model Guidance Conditional Image Generation

Open paper AskPDF arXiv

Diffusion models have become a dominant paradigm for conditional image generation, yet existing approaches generally follow two directions: task-specific designs that can improve performance but limit generalization, and training-free loss guidance that compresses rich conditions into scalar objectives and applies stepwise guidance, leading to information bottlenecks and error accumulation along the sampling trajectory. Given the urgent need for an effective unified framework across diverse conditional image generation tasks, we propose Data Injection and Contrastive Trajectory Refinement (DICT), a training-free inference method that enhances conditional image generation without introducing task-dependent architectures. DICT introduces Data Injection, where noise-perturbed conditional signals are integrated into early denoising stages; by performing guided denoising on these injected signals, DICT adaptively selects and distills task-salient information from the raw condition, effectively preserving spatial richness and ensuring precise condition-to-generation alignment. Furthermore, DICT applies Contrastive Trajectory Refinement across adjacent denoising states, enabling pairwise comparisons that progressively improve sample quality. These designs keep inference simple while improving cross-task transfer under a unified diffusion formulation. Extensive experiments on conditional image generation tasks (e.g., style transfer, image super-resolution, and image deblurring) show consistent gains in fidelity and perceptual quality over representative task-specific and loss-guided baselines.

Abstract

Image Generation

Explore similar work

DICT: Data Injection and Contrastive Trajectory Refinement for Conditional Image Generation with Diffusion Models

TPSO: Training-Free Diverse Image Generation via Semantic Prompt Embedding Optimization

Don't Settle at the Mode! Mitigating Diversity Collapse in Pretrained Flow Models via Feature Self-Guidance