Risk-Averse Inference-Time Diffusion Model Alignment with Finite-Sample Guarantees

Yuchen Liang; Xian Yu

stat.MLOct 13, 2026

Risk-Averse Inference-Time Diffusion Model Alignment with Finite-Sample Guarantees

Authors: Yuchen Liang, Xian Yu

Organizations: Indiana University Indianapolis · The Ohio State University

Abstract

Inference-time alignment steers diffusion models toward downstream objectives without costly fine-tuning, but existing methods typically optimize expected cost (or reward) without explicitly controlling tail risk. In this work, we propose a risk-averse framework for inference-time alignment of diffusion models that balances expected cost and Conditional Value-at-Risk (CVaR) under trajectory-level Kullback-Leibler regularization. We characterize the optimal path distribution and reverse transitions as functions of the CVaR threshold, reducing the alignment problem to a one-dimensional optimization problem over the threshold. We show that this one-dimensional optimization problem can be nonconvex in general, establish sufficient conditions for convexity, and introduce quadratic regularization to enable efficient derivative-free ternary search. Under suitable assumptions, we provide finite-sample guarantees for threshold optimization and value estimation, and combine them into a terminal total-variation bound relative to the regularized target under exact resampling. Experiments on synthetic data and image generation using Stable Diffusion demonstrate improved tail performance over risk-neutral baselines while maintaining competitive average cost.

Figures & tables

Method	Mean Cost $\downarrow$	Empirical CVaR $\downarrow$	$\hat{\mathbb{P}}(C>\tau_{80})$ (%) $\downarrow$
Base	$2.373\pm 0.109$	$8.379\pm 0.526$	$19.37\pm 0.62$
SVDD [ 7 ]	$0.706\pm 0.003$	$1.343\pm 0.011$	$1.95\pm 0.20$
CVaR-SVDD	$\mathbf{0.587\pm 0.002}$	$\mathbf{1.017\pm 0.006}$	0 observed

Table 1: Sampling performance on synthetic Cauchy data. Entries are mean

\pm

standard error across ten seeds. All metrics are costs , so smaller values are better. “0 observed” indicates no sample exceeded

\tau_{80}

Panel of Figure 1 — Figure 1: Synthetic mixed-Cauchy results. Vertical lines mark the $80$ -th, $90$ -th, and $95$ -th cost percentiles estimated from $50000$ base-model samples. Shading indicates pointwise $95\%$ confidence intervals.

Figure 2: Absolute threshold error $|\widehat{\eta}-\eta_{\kappa}^{\star}|$ versus the number of oracle evaluations. Gradient descent results are averaged over five randomly sampled learning rates.

Method	Mean Reward $\uparrow$	Avg Tail Reward $\uparrow$
Compressibility
Base	$-95.986\pm 0.224$	$-123.377\pm 1.042$
SVDD [ 7 ]	$-73.034\pm 0.121$	$-89.671\pm 0.292$
CVaR-SVDD	$\mathbf{-67.136\pm 0.131}$	$\mathbf{-78.405\pm 0.135}$
Aesthetic score
Base	$5.8845\pm 0.0076$	$5.4600\pm 0.0105$

Table 2: Image generation for the cat prompt. Entries are mean

\pm

standard error across five seeds. Both metrics are rewards , so larger values are better.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

\bar{w}_{t,m}^{(i)}=\frac{\exp\bigl(-\widehat{V}_{t-1}(z_{t,m}^{(i)})/\alpha\bigr)}{\sum_{j=1}^{M}\exp\bigl(-\widehat{V}_{t-1}(z_{t,j}^{(i)})/\alpha\bigr)},\quad m=1,\ldots,M.

Appendix

Algorithm 3

\mathtt{SVDD}

Sampling [ 7 ]

Figure 3: One realization of the terminal samples from the frozen base sampler, vanilla risk-neutral SVDD ( $\lambda=0$ ), and CVaR-SVDD ( $\lambda=1$ ) on the synthetic mixed-Cauchy example, using one representative random seed. The star denotes the target $x^{\star}=(3,3)$ , the point colors denote the bounded quadratic cost, and the dashed circles denote the common cost thresholds $\tau_{80}$ , $\tau_{90}$ , and $\tau_{95}$ .

Method	Mean cost	Median cost	Empirical CVaR	$\hat{\mathbb{P}}(C>\tau_{80})$ (%)	$\hat{\mathbb{P}}(C>\tau_{90})$ (%)	$\hat{\mathbb{P}}(C>\tau_{95})$ (%)
Base	$2.373$ $[2.126,\,2.621]$	$1.004$ $[0.9917,\,1.016]$	$8.379$ $[7.190,\,9.569]$	$19.37$ $[17.97,\,20.77]$	$9.760$ $[8.791,\,10.73]$	$4.800$ $[4.208,\,5.392]$
SVDD [ 7 ]	$0.7057$ $[0.7000,\,0.7114]$	$0.6703$ $[0.6601,\,0.6806]$	$1.343$ $[1.319,\,1.367]$	$1.950$ $[1.507,\,2.393]$	$0.1400$ $[0.07968,\,0.2003]$	0 observed
CVaR-SVDD	$\mathbf{0.5865}$ $[0.5814,\,0.5917]$	$\mathbf{0.6008}$ $[0.5937,\,0.6078]$	$\mathbf{1.017}$ $[1.003,\,1.031]$	0 observed	0 observed	0 observed

Appendix

Table 3: Complete sampling performance of synthetic Cauchy data. Here

\tau_{90}

and

\tau_{95}

represent empirical VaR under the base model, calculated from

50000

independent samples. Lower values are better throughout.

Panel of Figure 4 — Figure 4: Cost survival curves for the synthetic mixed-Cauchy data using the SMC sampler in Algorithm 4 . Left: empirical survival probabilities $\widehat{\mathbb{P}}\{c(x_{0})>\tau\}$ for vanilla SMC ( $\lambda=0$ ) and CVaR-SMC ( $\lambda=1$ ). Right: paired difference $\widehat{\mathbb{P}}_{\mathrm{CVaR\text{-}SMC}}\{c(x_{0})>\tau\}-\widehat{\mathbb{P}}_{\mathrm{Vanilla\text{-}SMC}}\{c(x_{0})>\tau\}$ . Negative values indicate an improvement from CVaR guidance. The dashed vertical lines indicate the common thresholds $\tau_{80}$ , $\tau_{90}$ , and $\tau_{95}$ .

Panel of Figure 5 — Figure 5: Ablation on the temperature $\alpha$ for vanilla SVDD ( $\lambda=0$ ) and CVaR-SVDD ( $\lambda=1$ ). Left: Empirical bad-sample probability $\widehat{\mathbb{P}}\{c(x_{0})>\tau_{80}\}$ as a function of $\alpha$ . Both axes are logarithmic. Note that the CVaR point at the smallest temperature has zero observed exceedances and therefore cannot be displayed on the logarithmic vertical axis. Right: Base log-likelihood as a function of tail probability. Each point is annotated by its value of $\alpha$ . Higher values on the vertical axis indicate that the generated samples lie, on average, in higher-density regions of the pretrained base distribution.

$\lambda$	$\widehat{\eta}$	Empirical CVaR $\downarrow$	$\widehat{\mathbb{P}}\{c(x_{0})>\tau_{80}\}$ $\downarrow$
$0$	—	$1.346\;[1.327,1.365]$	$0.01796$
$0.25$	$1.075862$	$1.192\;[1.181,1.203]$	$0.00454$
$0.50$	$1.022672$	$1.116\;[1.106,1.125]$	$0.00088$
$0.75$	$0.982972$	$1.065\;[1.055,1.076]$	$0.00022$
$1$	$0.955166$	$1.031\;[1.023,1.039]$	$0.00004$

Appendix

Table 4: Effect of the risk weight

\lambda

on synthetic sampling performance. Empirical CVaR is reported as the mean [95% confidence interval] across five inference seeds. Exceedance probabilities are seed averages. The threshold

\eta

has no effect when

\lambda=0

and it is thus omitted.

Figure 6: Effect of the regularization strength $\kappa$ on the empirical exceedance probability $\widehat{\mathbb{P}}\{c(x_{0})>\tau_{80}\}$ for CVaR-SVDD. Results are averaged over five random seeds, with shaded regions indicating $95\%$ Student- $t$ confidence intervals. The dotted horizontal line marks the mean at $\kappa=0$ .

$\kappa$	$\widehat{\eta}_{\kappa}$	Mean cost	Empirical CVaR
$0$	$0.950100$	$0.6250\;[0.6189,\,0.6311]$	$1.036\;[1.028,\,1.045]$
$0.0002$	$0.950100$	$0.6250\;[0.6189,\,0.6311]$	$1.036\;[1.028,\,1.045]$
$0.002$	$0.950492$	$0.6251\;[0.6190,\,0.6312]$	$1.037\;[1.028,\,1.045]$
$0.02$	$0.955166$	$0.6261\;[0.6200,\,0.6323]$	$1.038\;[1.030,\,1.046]$
$0.06$	$0.967847$	$0.6289\;[0.6227,\,0.6351]$	$1.042\;[1.034,\,1.050]$
$0.2$	$1.022609$	$0.6405\;[0.6345,\,0.6465]$	$1.061\;[1.054,\,1.069]$

Appendix

Table 5: Effect of the regularization strength

\kappa

on CVaR-SVDD. Performance metrics are reported as means [

95\%

confidence intervals] across five random seeds.

Figure 7: Value-estimator ablation on the synthetic example. We compare conditional-mean/hinge and normalized-exponential targets using neural networks (NN) and clipped linear regressors with fixed bounded features (Linear). Curves show empirical cost survival probabilities averaged over five inference seeds. Dashed vertical lines mark the base-model cost thresholds $\tau_{80}$ , $\tau_{90}$ , and $\tau_{95}$ . Lower curves indicate fewer high-cost outcomes.

Panel of Figure 8 — Figure 8: Full ablation on the numerical optimization of the regularized CVaR threshold $\eta$ . The empirical global minimizer $\eta_{\kappa}^{\star}$ is used as reference. Left: absolute eta error $|\widehat{\eta}-\eta_{\kappa}^{\star}|$ versus the number of oracle evaluations. Right: empirical objective gap $\widehat{G}_{\lambda,\kappa}(\widehat{\eta})-\widehat{G}_{\lambda,\kappa}(\eta_{\kappa}^{\star})$ versus the same oracle-evaluation budget. The uniform-grid baseline uses the same number of grid evaluations as the corresponding ternary-search budget of Algorithm 2 . For gradient descent, one joint evaluation of the objective and its analytic derivative is counted as one oracle evaluation, and the reported curve is averaged over five randomly sampled learning rates.

Reward	Prompt	Method	Mean reward $\uparrow$	Lower-tail mean reward $\uparrow$
Compressibility	cat	Base	$-95.986\pm 0.224$	$-123.377\pm 1.042$
		Vanilla SVDD	$-73.034\pm 0.121$	$-89.671\pm 0.292$
		CVaR-SVDD	$\bm{-67.136\pm 0.131}$	$\bm{-78.405\pm 0.135}$
		Plug-in CVaR-SVDD	$-71.401\pm 0.131$	$-83.434\pm 0.279$
Compressibility	dog	Base	$-100.721\pm 0.722$	$-132.286\pm 1.366$
		Vanilla SVDD	$-71.308\pm 0.113$	$-90.612\pm 0.209$

Appendix

Table 6: Complete image results in reward (i.e., negative cost). Each entry is the mean

\pm

standard error of seed-level metrics. All values are rewards and larger is better. The “lower-tail mean reward” evaluates the average rewards at or below that run’s empirical

20

th-percentile cutoff (including ties). The best result for each metric within each block is boldfaced.

Panel of Figure 9 — Figure 9: Compressibility reward distributions for the cat prompt. Vertical lines in the CDF panels mark the $20$ th-percentile reward of samples for threshold optimization, held fixed across methods. Panel (c) contains a magnified view of (b) near the reference cutoff.

Panel of Figure 10 — Figure 10: Aesthetic reward distributions for the cat prompt. Vertical lines in the CDF panels mark the $20$ th-percentile reward of samples for threshold optimization, held fixed across methods. Panel (c) contains a magnified view of (b) near the reference cutoff.

Panel of Figure 11 — Figure 11: Prompt ablation for compressibility. Vertical lines in the CDF panels mark the $20$ th-percentile reward of samples for threshold optimization, held fixed across methods.

Panel of Figure 12 — Figure 12: Prompt ablation for aesthetics. Vertical lines in the CDF panels mark the $20$ th-percentile reward of samples for threshold optimization, held fixed across methods. Full CDFs include the base sampler; enlarged panels show the three guided methods.

Panel of Figure 13 — Figure 13: SMC ablation for compressibility with cat . Vertical lines in the CDF panels mark the $20$ th-percentile reward of samples for threshold optimization, held fixed across methods.

Figure 14: Random examples from each method’s own worst- $20\%$ compressibility-reward tail for cat . Rows are base, vanilla SVDD, learned CVaR-SVDD, and plug-in CVaR-SVDD. Each row uses its own pooled reward quantile, shown at left. Columns are not paired comparisons.

Figure 15: Random examples from each method’s own worst- $20\%$ aesthetic-reward tail for cat . Rows are base, vanilla SVDD, learned CVaR-SVDD, and plug-in CVaR-SVDD. Each row uses its own pooled reward quantile, shown at left. Columns are not paired comparisons.

Figure 16: Compressibility tail examples for the SMC ablation with cat . Rows are base, vanilla SMC, learned CVaR-SMC, and plug-in CVaR-SMC. Each row uses its own pooled reward quantile, shown at left. Columns are not paired comparisons.

Panel of Figure 17 — Figure 17: Tail examples for dog and rabbit under compressibility and aesthetic rewards. In each panel, rows correspond to base, vanilla SVDD, learned CVaR-SVDD, and plug-in CVaR-SVDD, with four random images from each method’s own pooled lower tail. Columns are not paired comparisons.

Diffusion Model Alignment

Diffusion model alignment steers generated content and internal representations toward prompts, human preferences, safety goals, and task constraints.

See all 185 papers

May 24, 2026

Inference-Time Alignment of Diffusion Models via Trust-Region Iterative Twisted Sequential Monte Carlo

We study inference-time alignment for diffusion-based generative models, aiming to steer a base model toward high-reward outputs without updating its weights. Recent Sequential Monte Carlo (SMC)-based steering methods approximate reward-tilted target distributions in a principled way, but their proposals remain largely tied to the base sampler. Since reward information is mainly used after propagation through particle reweighting and resampling, these methods can require large particle budgets and suffer from weight degeneracy and high-variance estimates. One way to reduce variance and improve particle efficiency is to iteratively learn twisting functions that provide look-ahead guidance, as in twisted SMC. However, existing learnable twisting methods are developed mainly for classical sequential inference and can be unstable when applied to diffusion-based alignment with high-dimensional state spaces and terminal, noisy, or black-box rewards. We propose Trust-Region Iterative Twisted Sequential Monte Carlo (TRI-TSMC), a trust-region framework for learning twisting functions in SMC-based inference-time alignment. Each iteration computes an exact KL-constrained update in path space, which admits a closed-form solution by tempered importance reweighting, and projects this target back to the parameterized twisted family by weighted maximum likelihood. Theoretically, we formalize the value-function interpretation of the optimal twisting function and show that it yields a zero-variance sampler. We prove that the trust-region update follows an escort path toward the target distribution, that the weighted maximum-likelihood update is a forward-KL projection, and that the path reduces residual importance-weight variance. Empirically, TRI-TSMC improves primary alignment objectives on discrete diffusion text generation and text-to-image generation under matched inference-time budgets.

Weixin Wang, Yu Yang, Wei Deng +1

Duke University · Morgan Stanley

Diffusion Model Alignment Inference-Time Optimization Diffusion Language Models

Ask PDF

Sep 30, 2026

Steepest Guidance: A Practical and Principled Approach to Inference-Time Alignment of Flow and Diffusion-based Models

Inference-time alignment of flow and diffusion-based models is critical for achieving flexible generative modeling. Theoretically, Doob's

h

-transform provides an elegant solution to this problem, and most existing methods are based on this principle. However, in practice, estimating the optimal guidance derived from Doob's

h

-transform at inference time is challenging. To deal with this issue, we regard inference-time alignment as a sequential optimization problem in the space of probability measures and propose a novel framework called Steepest Guidance, based on the principle of maximizing local improvement in the objective. We provide a theoretical analysis of the proposed method and demonstrate its effectiveness through extensive experiments.

Shokichi Takakura, Akifumi Wachi, Rei Higuchi +2

LY Corporation · The University of Tokyo · RIKEN AIP

Diffusion Model Alignment Inference-Time Optimization Flow-Based Generative Modeling Inference-Time Guidance

Ask PDF

Jul 20, 2026

DiFA: Inference-Time Forward-Process Alignment for Diffusion Models

The prevailing inference framework for diffusion models formulates generation fundamentally as a problem of numerical integration. This perspective casts the model as an exact estimator, neglecting the inherent statistical uncertainty of the denoising process. In this work, we propose Forward-Process Aligned Diffusion prediction (DiFA), a training-free framework that reframes inference-time data prediction refinement as a sequential state estimation problem. Rather than reusing past outputs solely for numerical integration, DiFA treats iterative data predictions along the reverse trajectory as correlated observations to build a forward-aligned temporal consensus. Inspired by Kalman filtering, this consensus aggregates historical predictions according to structural consistency and noise-level compatibility. To counteract the over-smoothing tendency of temporal consensus, we introduce a deviation guidance mechanism to adaptively preserve residual details. Empirically, DiFA yields significant improvements on CIFAR-10 and ImageNet across the evaluated metrics, including FID, IS, and FD-DINOv2, demonstrating that aligning inference with the forward statistical structure substantially improves generative fidelity.

Shigui Li, Delu Zeng

School of Mathematics, South China University of Technology, Guangzhou, China · School of Electronic and Information Engineering, South China University of Technology, Guangzhou, China

Diffusion Model Alignment Diffusion Sampling Diffusion-Based Image Generation Diffusion Models

Ask PDF

Abstract

Diffusion Model Alignment

Explore similar work

Inference-Time Alignment of Diffusion Models via Trust-Region Iterative Twisted Sequential Monte Carlo

Steepest Guidance: A Practical and Principled Approach to Inference-Time Alignment of Flow and Diffusion-based Models

DiFA: Inference-Time Forward-Process Alignment for Diffusion Models