WEIRDO: WEak resIdual Regularized DOob's h-transform diffusion alignment

Denis Suchkov

math.STSep 30, 2026

WEIRDO: WEak resIdual Regularized DOob's h-transform diffusion alignment

Authors: Denis Suchkov

Organizations: HSE University, Russian Federation

Abstract

We study the problem of estimating the guidance that steers the distribution learned by a diffusion generative model toward a tilted target $q_0 \propto w\,p_0$ at inference time. Relying on the stochastic optimal control approach, we observe that the exact drift correction is the gradient of the logarithm of Doob's $h$ -function, and we study the problem of estimating it from a sample. In the present paper, we assume that the score of the pretrained model is available, that the tilting weight is bounded and positive, and that the reference distribution has a bounded support, no smoothness of the weight is required. Introducing a penalized least-squares risk in which the penalty is the residual of the space-time harmonicity equation satisfied by the $h$ -function, measured in a dual Sobolev norm, we derive high-probability bounds on the squared error of the resulting guidance estimate. Since the penalty vanishes at the target, the estimator is free of regularization bias, and in favourable scenarios its rate of convergence is faster than the minimax rate of estimating first-order derivatives of a smooth regression function. Assuming that $w$ is bounded and positive with $\mathbb{E}_{p_0}[w^{-\mathrm{s}}] < \infty$ for some $\mathrm{s} \in (0,\infty]$ , and that the reference data are compactly supported, we prove that the guidance is estimable in squared $L^2$ at rate $\varepsilon_n^{\mathrm{s}/(\mathrm{s}+4)}$ , where $\varepsilon_n = n^{-2(β-1)/(2(β-1)+d)}.$ We also transfer the obtained bounds to the total variation distance between the marginals of the estimated and the exactly guided samplers, and illustrate the performance of the suggested approach with numerical experiments.

Figures & tables

Penalty $\mathcal{P}(h)$	order in $h$	value at $h^{\ast}$	approximation floor
Sobolev, $\int_{I}\\|\nabla h\\|^{2}_{\rho_{t}}\nu(dt)$	1	$\\|\nabla h^{\ast}\\|^{2}\neq 0$	— (nonzero gap for $\lambda>0$ )
strong residual, $\\|\mathcal{R}[h]\\|^{2}_{L^{2}(\nu\otimes\rho)}$	2	$0$	$N^{-2(\beta-2)}$ , non-informative at $\beta=2$
weak residual, $\\|\mathcal{R}[h]\\|^{2}_{\mathcal{V}^{\ast}}$	1	$0$	$N^{-2(\beta-1)}$

Table 1: Why the residual is measured weakly. The floor is

\inf_{h\in\mathcal{H}_{N}}\mathcal{P}(h)

at resolution

N,

only the last is both gap-free and informative at

\beta=2

Panel of Figure 1 — Figure 1 : (a) Guidance error at $d=1$ , $\pm 1$ s.d. over $8$ seeds, against slopes $0.857$ and $0.667$ . (b) Held-out reward shift, as a share of the exact tilt’s, against the FID it costs over that dataset’s base ( $24.5$ , $7.4$ ), so grey is both base models and lower is cheaper. Colour is the method, line style the dataset, $c\in[0.25,20]$ , bars $\pm 1$ s.d. over $3$ seeds, DOIT is off the panel at $4.8$ and $21$ times its base, absolute values in Figure 3 . (c) The five largest rises in held-out probability among $64$ samples, base above and ours at $c=20$ below on the same initial noise, all $64$ in Figure 4 .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

	checkpoint	parameters	share	SIR-32	FID
CIFAR-10	google/ddpm-cifar10-32	$35.7$ M	$0.653$	$0.854$	$24.5$
MNIST	dvgodoy/ddpm-cifar10-32-mnist	$35.7$ M	$0.523$	$0.630$	$7.4$

Appendix

Table 2 : Base models. “Share” is the fraction of base samples the held-out head assigns to the target superclass, “SIR-32” the same after

32

-fold importance resampling with weights

w

, and FID is to the model’s own reference sample (Appendix J.3 ).

penalty	error at $n=10^{3}$	error at $n=10^{6}$	predicted exponent
value LS ( $\mu=0$ )	$6.46\times 10^{-2}$	$9.71\times 10^{-5}$	$0.667$
Sobolev	$4.41\times 10^{-2}$	$1.40\times 10^{-4}$	—
strong residual	$4.46\times 10^{-3}$	$2.34\times 10^{-5}$	—
weak residual (ours)	$6.51\times 10^{-3}$	$1.64\times 10^{-5}$	$\mathbf{0.857}$

Appendix

Table 3 : Penalty ablation at

d=1

in the cubic class (

\beta=4

8

seeds, each penalty at its own oracle

(N,\mu)

: relative guidance error at the smallest and the largest sample size, against the exponent the theory predicts for it. A rate is predicted only for the weak residual and, from the classical derivative-recovery bound, for least squares, the Sobolev penalty carries a regularization bias and the strong residual an approximation floor (Table 1 ), and neither is analysed here. “Value LS” is the

\mu=0

baseline of Figure 2 .

Figure 2 : Guidance error against

n

d=1

in the tensor cubic B-spline class (

\beta=4

), for the weak-residual estimator and for least squares on the values followed by differentiation, against the predicted slopes

0.857

and

0.667

, intercepts fitted only. Bars are

\pm 1

standard deviation over

8

seeds. The weak residual’s fitted exponent,

0.842\pm 0.026

, matches its prediction, least squares runs above its own,

0.909\pm 0.038

against

0.667

, so over this range it is measured far from its asymptote and its steeper slope is a climb out of a worse level rather than a faster rate.

	weak residual (ours)		differentiate		ratio
$d$	$n=10^{3}$	largest $n$	$n=10^{3}$	largest $n$	smallest	largest
$1$	$6.5\times 10^{-3}$	$1.6\times 10^{-5}$	$6.5\times 10^{-2}$	$9.7\times 10^{-5}$	$5.9\times$	$13.5\times$
$2$	$2.3\times 10^{-2}$	$6.9\times 10^{-5}$	$3.5\times 10^{-1}$	$4.2\times 10^{-4}$	$5.4\times$	$15.3\times$
$3$	$8.3\times 10^{-2}$	$3.7\times 10^{-4}$	$2.5$	$3.6\times 10^{-3}$	$9.7\times$	$30.2\times$

Appendix

Table 4 : The two estimators at equal budget,

8

seeds, each at its own oracle

(N,\mu)

on the same grids. “Differentiate” is least squares on the values, then differentiated. Entries are relative squared guidance error, the largest

n

10^{6}

, and

3.2\cdot 10^{5}

d=3

	ours		DEFT		DOIT
$c$	%	FID	%	FID	%	FID
$0.01^{\dagger}$	—	—	—	—	$-43.8\pm 5.1$	$43.1\pm 0.7$
$0.05^{\dagger}$	—	—	—	—	$38.7\pm 2.0$	$87.0\pm 1.2$
$0.1^{\dagger}$	—	—	—	—	$116.0\pm 2.5$	$221.6\pm 1.0$
$0.25$	$3.2\pm 0.3$	$24.5\pm 0.2$	—	—	—	—
$0.5$	$6.1\pm 0.4$	$24.6\pm 0.2$	—	—	—	—

Appendix

Table 5 : CIFAR-10, animals, along the guidance scale (

\gamma

for DOIT): held-out reward shift as % of the exact tilt’s, and FID to the model’s own reference sample. Base FID

24.5\pm 0.2

, share

0.653

. Bold marks, at each scale, the largest shift and the smallest FID, a row holds

c

fixed, which leaves the three methods at different FIDs, so the operative reading is the frontier of Figure 3 and not the row. DOIT is entered at the three

\gamma

of the shared sweep, where its cap binds on every guided step so that the three agree to

0.2

points, and at the small

\gamma

its authors’ own setting calls for. Daggered rows are that setting —

\eta=0.4

and no cap, read against the base of the same sampler — and are not comparable with the rest of the row they sit in, the wider grid over

\gamma

\tau

and the sampler is in Appendix J.8 . Here and in Table 6 ,

\pm

is one standard deviation over the three seeds, which dominates the standard error over the

6000

paired samples (at most

2.3

for the shift).

Figure 3 : What the alignment costs. Horizontal is the held-out reward shift as a percentage of the exact tilt’s, vertical the FID to the model’s own reference sample, so the grey point is the base model and a lower curve is cheaper, bars are

\pm 1

s.d. over three seeds on both axes. Left: CIFAR-10,

c\in[0.25,20]

. Right: MNIST,

c\in\{1,5,20\}

. DOIT is off both panels, at FID

117

and

153

against bases of

24.5

and

7.4,

drawing it would compress the region the figure exists to show, and its numbers are in Tables 5 and 6 .

Figure 4 : CIFAR-10, animals, the first $64$ samples of seed $0$ , uncurated. Base and ours at $c=20$ above, DEFT and SIR-32 below, each panel labelled with the share of animals it carries and with its FID. DEFT is drawn at $c=2$ , the scale of its sweep whose FID is closest to the one our own panel is drawn at, because a matched departure from the base distribution is the reading under which the two frontiers are compared, at a matched $c$ it would instead sit at FID $98$ . The guided panels share the initial noise with the base panel, so a sample can be compared with its counterpart in the same position, SIR-32 does not, by construction.

	ours		DEFT		DOIT
$c$	%	FID	%	FID	%	FID
$0.01^{\dagger}$	—	—	—	—	$88.5\pm 3.7$	$14.1\pm 0.8$
$0.05^{\dagger}$	—	—	—	—	$280.0\pm 7.9$	$99.4\pm 2.2$
$0.1^{\dagger}$	—	—	—	—	$283.4\pm 7.9$	$108.0\pm 1.4$
$1$	$13.9\pm 1.0$	$7.4\pm 0.1$	$59.8\pm 3.5$	$9.3\pm 0.3$	$296.4\pm 4.1$	$153.3\pm 3.6$
$5$	$62.8\pm 4.7$	$7.7\pm 0.1$	$145.6\pm 4.6$	$25.2\pm 0.2$	$296.4\pm 4.0$	$153.1\pm 3.5$

Appendix

Table 6 : MNIST, odd digits, tilted under the recipe of Appendix J.1 : held-out reward shift as % of the exact tilt’s and FID to the model’s own reference sample. Base share

0.523

, FID

7.4\pm 0.1,

SIR-32 share

0.630

. Bold marks, at each scale, the largest shift and the smallest FID, and as in Table 5 the row, which holds

c

fixed, is not the operative comparison. Daggered rows are DOIT at the small

\gamma

and in the setting its authors use,

\eta=0.4

and no cap, read against the base of that sampler,

\gamma=0.01

is the one point at which DOIT reaches a usable frontier at all.

\pm

is one standard deviation over the three seeds.

Figure 5 : MNIST, odd digits, the first $64$ samples of seed $0$ , uncurated. Base and ours at $c=20$ above, DEFT at $c=1$ and SIR-32 below, each panel labelled with the share of odd digits and with its FID, as in Figure 4 , DEFT is drawn at the scale of its sweep whose FID is closest to ours. The guided panels share the initial noise with the base panel. The base model draws some digits mirrored.

sampler	method	$\overline{\|\delta\|}$	binds	capped $\to$ uncapped
$\eta=0$	ours	$0.031$	$5.3\%$	$0.831/32.9\to 0.739/43.4$
	DEFT	$0.158$	$77.9\%$	$0.941/97.7\to$ —
	DOIT	$2.4$	$100\%$	$0.689/116.7\to 0.995/463.2$
$\eta=0.4$	ours	$0.031$	$5.0\%$	$0.854/34.6\to 0.805/46.2$
	DOIT, $\gamma=1$	$2.3$	$100\%$	$0.464/70.2\to 0.996/437.1$
	DOIT, $\gamma=0.1$	$0.23$	$95.2\%$	$0.462/70.0\to 0.889/221.6$

Appendix

Table 7 : The cap at

c=20

on CIFAR-10 (

\gamma=1

for DOIT, and

\gamma=0.1

, the small-

\gamma

setting of its authors), where it binds at all: the mean uncapped correction against the cap of

0.1

, the share of guided steps it binds on, and what removing it does. At

c=5

and below it binds on at most

1\%

of the steps of either trained method. The uncapped point was not sampled for the reported DEFT head, nor was that head run under the stochastic sampler, so those cells are empty.

$\beta$ (order)	weak floor	strong floor	strong / weak at $N=24$
$4$ ( $3$ )	$N^{-1.44}$	$N^{-1.89}$	$7.4$
$3$ ( $2$ )	$N^{-1.81}$	$N^{-1.00}$	$311$
$2$ ( $1$ )	$N^{-2.46}$	$\mathbf{N^{+0.29}}$	$8500$

Appendix

Table 8 : Approximation floors against resolution,

d=1

4

replicates, slopes over

N\in[2,24]

, predictions

-2(\beta-1)

and

-2(\beta-2)

$\min w$	$n=10^{3}$	$n=10^{6}$	measured	envelope	clip
$0.5$ (base)	$3.0\times 10^{-3}$	$1.0\times 10^{-5}$	$0.787$	$0.779$	$0.00$
$1$	$1.2\times 10^{-2}$	$2.2\times 10^{-5}$	$0.919$	$0.779$	$0.00$
$0.3$	$2.6\times 10^{-2}$	$2.0\times 10^{-5}$	$1.014$	$0.779$	$0.00$
$0.1$	$4.3\times 10^{-2}$	$2.2\times 10^{-5}$	$1.076$	$0.337$	$0.00$
$0.03$	$1.5\times 10^{-1}$	$2.9\times 10^{-3}$	$0.547$	$0.004$	$0.40$
$3\times 10^{-3}$	$2.6\times 10^{-1}$	$\mathbf{2.1\times 10^{-1}}$	$\mathbf{0.028}$	$0.002$	$0.44$

Appendix

Table 9 : The finite-

\mathrm{s}

ladder,

d=1

4

replicates, each cell at its own oracle

(N,\mu),

slopes over the seven sample sizes, “clip” the mass on which

h^{\ast}<\mathrm{b}

n=10^{6}

Explore similar work

Sep 25, 2026cs.LG

DOHF: Online Diffusion Fine-tuning with Doob's $h$ -transform Guidance

Reward-based diffusion fine-tuning faces practical challenges when desirable outcomes are rare or conditioning corrections are costly to estimate. In this work, we propose Diffusion Online

h

-guidance Fine-tuning (DOHF), which turns Doob's

h

-transform into a practical online training algorithm. DOHF assigns optimality weights to generated samples, estimates the normalized local correction

\nabla\log h

under the current rollout policy, and distills it directly into the generative model. Theoretically, we characterize the population-optimal DiffusionNFT update as well as the various classfier free guidance methods through a unified

h

-transform perspective. Methodologically, our framework accommodates black-box and non-differentiable rewards without additional network evaluations. We further show improved alignments under three empirical scenarios. Our work demonstrates how adapting probabilistic conditioning through inexpensive estimation and iterative distillation can improve generative learning across statistical sampling and visual generation.

Zhengyi Guo, Jiayuan Sheng, Wenpin Tang +1

Diffusion Models Generative Models

Ask PDF

Sep 30, 2026stat.ML

Steepest Guidance: A Practical and Principled Approach to Inference-Time Alignment of Flow and Diffusion-based Models

Inference-time alignment of flow and diffusion-based models is critical for achieving flexible generative modeling. Theoretically, Doob's

h

-transform provides an elegant solution to this problem, and most existing methods are based on this principle. However, in practice, estimating the optimal guidance derived from Doob's

h

-transform at inference time is challenging. To deal with this issue, we regard inference-time alignment as a sequential optimization problem in the space of probability measures and propose a novel framework called Steepest Guidance, based on the principle of maximizing local improvement in the objective. We provide a theoretical analysis of the proposed method and demonstrate its effectiveness through extensive experiments.

Shokichi Takakura, Akifumi Wachi, Rei Higuchi +2

LY Corporation · The University of Tokyo · RIKEN AIP

Diffusion Alignment Diffusion Models Generative Models Guidance Robust Optimization Principles

Ask PDF

Jun 1, 2026cs.LG

Are we really tilting? The mechanics of reward guidance in flow and diffusion models

Reward guidance algorithms steer a learned generative process toward the reward-tilted measure at inference time. While empirically powerful, these methods are prone to reward hacking: the guided model over-optimizes the reward at the cost of fidelity to the learned distribution. Prior work has attributed this to the complexity of neural reward functions or implicit biases in diffusion training, but its fundamental origins remain poorly understood. We show that reward hacking arises from an approximation made in most practical implementations of reward-guided diffusion -- finite-particle plug-in estimation of the Doob h-function -- even in the simplest non-trivial settings of Gaussian and Gaussian mixture targets with quadratic rewards. In closed form, we isolate two distinct failure modes of the plug-in estimator: it leads to reward hacking within each mode and it cannot select high-reward modes. We propose a closed-form reward damping schedule that corrects the within-mode bias with no additional compute, and clarify the role of best-of-n sampling in compensating for the mode selection failure. Experiments on Gaussian mixture targets, a 2D checkerboard, and FLUX.1 text-to-image generation confirm that our theoretical insights carry over to practical settings.

Sanjit Dandapanthula, Nicholas M. Boffi

Carnegie Mellon University

Reward Gradients Reward Functions Diffusion Models Gaussian Mixture Models Mechanics

Ask PDF

Abstract

Explore similar work

DOHF: Online Diffusion Fine-tuning with Doob's hhh-transform Guidance

Steepest Guidance: A Practical and Principled Approach to Inference-Time Alignment of Flow and Diffusion-based Models

Are we really tilting? The mechanics of reward guidance in flow and diffusion models

DOHF: Online Diffusion Fine-tuning with Doob's $h$ -transform Guidance