DOHF: Online Diffusion Fine-tuning with Doob's $h$-transform Guidance

Zhengyi Guo; Jiayuan Sheng; Wenpin Tang; David D. Yao

cs.LGSep 25, 2026

DOHF: Online Diffusion Fine-tuning with Doob's $h$ -transform Guidance

Authors: Zhengyi Guo, Jiayuan Sheng, Wenpin Tang, David D. Yao

Abstract

Reward-based diffusion fine-tuning faces practical challenges when desirable outcomes are rare or conditioning corrections are costly to estimate. In this work, we propose Diffusion Online $h$ -guidance Fine-tuning (DOHF), which turns Doob's $h$ -transform into a practical online training algorithm. DOHF assigns optimality weights to generated samples, estimates the normalized local correction $\nabla\log h$ under the current rollout policy, and distills it directly into the generative model. Theoretically, we characterize the population-optimal DiffusionNFT update as well as the various classfier free guidance methods through a unified $h$ -transform perspective. Methodologically, our framework accommodates black-box and non-differentiable rewards without additional network evaluations. We further show improved alignments under three empirical scenarios. Our work demonstrates how adapting probabilistic conditioning through inexpensive estimation and iterative distillation can improve generative learning across statistical sampling and visual generation.

Figures & tables

Figure 1 : Visualization of on/off-policy h-transform guidance.

Method	Iterations	Minority-mode fraction	Mean $\pm$ std
True data	–	00 0.96%	$(0.98,0.98)\pm(0.44,0.44)$
Base pretrained	–	00 1.12%	$(0.99,0.98)\pm(0.42,0.41)$
Off-policy $h$ -guidance	2000	00 14.88%	$(0.29,0.25)\pm(0.73,0.71)$
DOHF(ours)	0 200	100.00%	$(-0.82,-0.88)\pm(0.38,0.38)$

Table 1: Fraction of samples falling in the minority cluster after fine-tuning.

Method	Updates	Avg. Margin Error $\downarrow$	Max Margin Error $\downarrow$	$W_{2}$ vs. DG sampler Diaconis and Gangolli (1995) $\downarrow$
Uniform	–	0.802	1.975	13.48
Off-policy	800	0.379	1.005	11.00
DOHF	300	0.210	0.531	9.84

Table 2 : Comparison of margin matching and distributional similarity. The margin errors are standardized by the corresponding generated marginal standard deviations.

Panel of Figure 2 — Figure 2 : Qualitative comparison across different guidance and fine-tuning methods. The prompts are taken from Pick-a-Pic Kirstain et al. (2023) .

Model	#Iter	GenEval	PickScore	ClipScore	HPSv2.1	Aesthetic	ImgRwd	LPIPS
SD3.5-M w/o CFG	–	4.49%	0.179	0.178	0.150	4.513	-1.809	0.771
SD3.5-M w/ CFG	–	41.35%	0.203	0.266	0.248	5.428	0.250	0.674
SD3.5-L w/ CFG	–	25.00%	0.193	0.243	0.208	4.919	-0.452	0.733
DOHF w/o CFG	2k	44.27%	0.216	0.264	0.327	6.327	1.115	0.538
DOHF w/ CFG	2k	62.52%	0.218	0.277	0.336	6.240	1.329	0.521
Off-policy Guidance	2k	19.52%	0.184	0.201	0.177	4.739	-1.320	0.746

Table 3 : Evaluation Results under resolution

256\times 256

. Bold : best; Underline : second best.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Birth $\backslash$ Death	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec	Total
Jan	1	0	0	0	1	2	0	0	1	0	1	0	6
Feb	1	0	0	1	0	0	0	0	0	1	0	2	5
Mar	1	0	0	0	2	1	0	0	0	0	0	1	5
Apr	3	0	2	0	0	0	1	0	1	3	1	1	12
May	2	1	1	1	1	1	1	1	1	1	1	0	12
Jun	2	0	0	0	1	0	0	0	0	0	0	0	3

Appendix

Table 4: Birthday and death month of 82 descendants of Queen Victoria.

Panel of Figure 3 — Figure 3 : Qualitative comparison across different methods and prompts. The prompts are: (Prompt 1) “Pixel art of a space station launching with two suns and a comet, featuring a retro space helmet, minigun, and a bright explosion in a dream-like atmosphere with CGA shading.”; (Prompt 2) “The devil using a heat press.”; (Prompt 3) “Ink drawing of a Shaolin monk in fighting pose.”; (Prompt 4) “A side view of a fox, with a flat design, Artwork of t-shirt graphic.”

Panel of Figure 4 — Figure 4 : Qualitative comparison across different methods and prompts. The prompts are: (Prompt 1, color) “A young woman with red hairs and green eyes”; (Prompt 2, spatial) “A penguin astronaut, in a penguin spacesuit, standing next to the apollo landing”; (Prompt 3, text) “A pikachu holding a sign that says ”I’m cute!” ”; (Prompt 4, text) “Gandalf holding a sign with a text ”you shall not pass” ”.

Explore similar work

WEIRDO: WEak resIdual Regularized DOob's h-transform diffusion alignment
Sep 30, 2026Denis SuchkovDiffusion Alignment Wasserstein Gradient Flows
HSE University, Russian Federation
Are we really tilting? The mechanics of reward guidance in flow and diffusion models
Jun 1, 2026Sanjit Dandapanthula, Nicholas M. BoffiReward Gradients Reward Functions
Carnegie Mellon University
Explicit Critic Guidance for Aligning Diffusion Models
May 26, 2026Zhengyang Liang, Qihang Zhang, Ceyuan YangDiffusion Alignment Diffusion-Based Reinforcement Learning Methods
University of Toronto · Vector Institute · The Chinese University of Hong Kong

Sep 30, 2026, math.STJ/K move · Enter open · S save

WEIRDO: WEak resIdual Regularized DOob's h-transform diffusion alignment

Denis Suchkov

HSE University, Russian Federation

Diffusion Alignment Wasserstein Gradient Flows Regularization Guidance Stochastic Optimal Control Generative Models

Open paper AskPDF arXiv

We study the problem of estimating the guidance that steers the distribution learned by a diffusion generative model toward a tilted target

q_0 \propto w\,p_0

at inference time. Relying on the stochastic optimal control approach, we observe that the exact drift correction is the gradient of the logarithm of Doob's

h

-function, and we study the problem of estimating it from a sample. In the present paper, we assume that the score of the pretrained model is available, that the tilting weight is bounded and positive, and that the reference distribution has a bounded support, no smoothness of the weight is required. Introducing a penalized least-squares risk in which the penalty is the residual of the space-time harmonicity equation satisfied by the

h

-function, measured in a dual Sobolev norm, we derive high-probability bounds on the squared error of the resulting guidance estimate. Since the penalty vanishes at the target, the estimator is free of regularization bias, and in favourable scenarios its rate of convergence is faster than the minimax rate of estimating first-order derivatives of a smooth regression function. Assuming that

w

is bounded and positive with

\mathbb{E}_{p_0}[w^{-\mathrm{s}}] < \infty

for some

\mathrm{s} \in (0,\infty]

, and that the reference data are compactly supported, we prove that the guidance is estimable in squared

L^2

at rate

\varepsilon_n^{\mathrm{s}/(\mathrm{s}+4)}

, where

\varepsilon_n = n^{-2(β-1)/(2(β-1)+d)}.

We also transfer the obtained bounds to the total variation distance between the marginals of the estimated and the exactly guided samplers, and illustrate the performance of the suggested approach with numerical experiments.

Birth $\backslash$ Death	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec	Total
Jan	1	0	0	0	1	2	0	0	1	0	1	0	6
Feb	1	0	0	1	0	0	0	0	0	1	0	2	5
Mar	1	0	0	0	2	1	0	0	0	0	0	1	5
Apr	3	0	2	0	0	0	1	0	1	3	1	1	12
May	2	1	1	1	1	1	1	1	1	1	1	0	12
Jun	2	0	0	0	1	0	0	0	0	0	0	0	3

Birth $\backslash$ Death	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec	Total
Jan	1	0	0	0	1	2	0	0	1	0	1	0	6
Feb	1	0	0	1	0	0	0	0	0	1	0	2	5
Mar	1	0	0	0	2	1	0	0	0	0	0	1	5
Apr	3	0	2	0	0	0	1	0	1	3	1	1	12
May	2	1	1	1	1	1	1	1	1	1	1	0	12
Jun	2	0	0	0	1	0	0	0	0	0	0	0	3

Abstract

Explore similar work

WEIRDO: WEak resIdual Regularized DOob's h-transform diffusion alignment

Are we really tilting? The mechanics of reward guidance in flow and diffusion models

Explicit Critic Guidance for Aligning Diffusion Models

Birth $\backslash$ Death	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec	Total
Jan	1	0	0	0	1	2	0	0	1	0	1	0	6
Feb	1	0	0	1	0	0	0	0	0	1	0	2	5
Mar	1	0	0	0	2	1	0	0	0	0	0	1	5
Apr	3	0	2	0	0	0	1	0	1	3	1	1	12
May	2	1	1	1	1	1	1	1	1	1	1	0	12
Jun	2	0	0	0	1	0	0	0	0	0	0	0	3