Step-level Denoising-time Diffusion Alignment with Multiple Objectives

Qi Zhang; Dawei Wang; Shaofeng Zou

cs.LGApr 15, 2026

Step-level Denoising-time Diffusion Alignment with Multiple Objectives

Authors: Qi Zhang, Dawei Wang, Shaofeng Zou

Organizations: School of Electrical, Computer and Energy Engineering, Arizona State University, Tempe, AZ 85281 USA

Abstract

Reinforcement learning (RL) has emerged as a powerful tool for aligning diffusion models with human preferences, typically by optimizing a single reward function under a KL regularization constraint. In practice, however, human preferences are inherently pluralistic, and aligned models must balance multiple downstream objectives, such as aesthetic quality and text-image consistency. Existing multi-objective approaches either rely on costly multi-objective RL fine-tuning or on fusing separately aligned models at denoising time, but they generally require access to reward values (or their gradients) and/or introduce approximation error in the resulting denoising objectives. In this paper, we revisit the problem of RL fine-tuning for diffusion models and address the intractability of identifying the optimal policy by introducing a step-level RL formulation. Building on this, we further propose Multi-objective Step-level Denoising-time Diffusion Alignment (MSDDA), a retraining-free framework for aligning diffusion models with multiple objectives, obtaining the optimal reverse denoising distribution in closed form, with mean and variance expressed directly in terms of single-objective base models. We prove that this denoising-time objective is exactly equivalent to the step-level RL fine-tuning, introducing no approximation error. Moreover, we provide numerical results, which indicate our method outperforms existing denoising-time approaches.

Figures & tables

Figure 1: Overview of our proposed Multi-objective Step-level Denoising-time Diffusion Alignment (MSDDA) algorithm.

Method	Reward Gradient	Reward Function	Approximation Error
MUDM Han et al. (2023)	$\surd$	$\surd$	NA
TFG Ye et al. (2024)	$\surd$	$\surd$	NA
DAS Kim et al. (2025)	$\times$	$\surd$	NA
CoDE Singh et al. (2025)	$\times$	$\surd$	NA
RS Rafailov et al. (2023)	$\times$	$\times$	NA
DB-MPA Cheng et al. (2025)	$\times$	$\times$	$\surd$

Table 1: Comparison of existing diffusion denoising-time alignment methods, all of which are training-free.

\surd

indicates the term is required and

\times

indicates the term is not required.

		SD		MSDDA		RS		CoDe		RGG
		$r_{1}$	$r_{2}$	$r_{1}$	$r_{2}$	$r_{1}$	$r_{2}$	$r_{1}$	$r_{2}$	$r_{1}$	$r_{2}$
Reward (↑)	$w=0.2$	0.22	-0.15	0.40	0.52	0.24	0.37	0.40	0.05	0.21	0.42
	$w=0.5$	0.22	-0.15	0.61	0.26	0.38	0.04	0.60	0.01	0.23	0.23
	$w=0.8$	0.22	-0.15	0.65	0.02	0.65	-0.12	0.66	-0.07	0.27	0.00
Inference Time (↓ sec/img)		5.08		10.14		5.08		185.26		121.58

Figure 2: Quantitative comparison of our proposed MSDDA method and baseline methods. The results for CoDe and RGG are obtained from Cheng et al. (2025) .

Figure 3: Pareto front of our proposed method and the baselines with different preference weight $w$ . The results for CoDe and RGG are obtained from Cheng et al. (2025) .

Panel of Figure 4 — Figure 4: Image samples generated by our proposed method and RS under different preference weights $w$ The left image is generated by Stable Diffusion using the prompt shown in the lower-left corner. The top row shows images generated by our proposed MSDDA, while the bottom row shows images generated by RS.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Training Prompts

Test Prompts

A red colored car. A black colored car. A pink colored car. A black colored dog. A red colored dog. A blue colored dog. A green colored banana. A red colored banana. A black colored banana. A white colored sandwich. A black colored sandwich. An orange colored sandwich. A pink colored giraffe. A yellow colored giraffe. A brown colored giraffe. A red car and a white sheep. A blue bird and a brown bear. A green apple and a black backpack. A green cup and a blue cell phone. A yellow book and a red vase. A white car and a red sheep. A brown bird and a blue bear. A black apple and a green backpack. A blue cup and a green cell phone. A red book and a yellow vase.

A white colored dog. A purple colored dog. A yellow colored dog. A green colored apple. A black colored apple. A blue colored apple. A purple colored apple. A pink colored banana. A pink colored cup. A purple colored sandwich. A green colored giraffe. A blue colored backpack. A blue car and a pink sheep. A red apple and a purple backpack. A pink car and a yellow sheep. A black cup and a yellow cell phone. A blue car and a red giraffe. A yellow bird and a purple sheep. A pink car and a green bear. A purple hat and a black clock. A black chair and a red table. A red car and a blue bird. A green car and a yellow banana. A pink vase and a red apple. A blue book and an orange colored sandwich.

Appendix

Table 2: Training and test prompt datasets used in the experiments.

Panel of Figure 5 — Figure 5: Image samples of our proposed method and RS.

Panel of Figure 6 — Figure 6: Image samples of our proposed method and RS.

Panel of Figure 7 — Figure 7: Image samples of our proposed method and RS.

Panel of Figure 8 — Figure 8: Image samples of our proposed method and RS.

Diffusion Model Alignment

Diffusion model alignment steers generated content and internal representations toward prompts, human preferences, safety goals, and task constraints.

See all 185 papers

Explicit Critic Guidance for Aligning Diffusion Models
May 26, 2026Zhengyang Liang, Qihang Zhang, Ceyuan YangDiffusion Model Alignment Diffusion Model Guidance
University of Toronto · Vector Institute · The Chinese University of Hong Kong
Latent Reward Registers for Diffusion Preference Alignment
Aug 4, 2026Yuanshen Guan, Zipeng Feng, Chengru Song +2Diffusion Model Alignment Diffusion Model Guidance
1Kling Team · University of Science and Technology of China
Explore or Converge? Stage-Guided Per-Step Optimization for Diffusion Models
Aug 7, 2026Renye Yan, Jikang Cheng, You Wu +4Diffusion Model Alignment Temporal Credit Assignment
Peking University Beijing, China · Stanford University Stanford, California, USA

May 26, 2026J/K move · Enter open · S save

Explicit Critic Guidance for Aligning Diffusion Models

Zhengyang Liang, Qihang Zhang, Ceyuan Yang

University of Toronto · Vector Institute · The Chinese University of Hong Kong

Diffusion Model Alignment Diffusion Model Guidance RL for Diffusion Models

Open paper AskPDF arXiv

Online reinforcement learning is becoming increasingly important for aligning diffusion models with non-differentiable objectives. However, existing methods still face limitations in assigning fine-grained credit along denoising trajectories and in realizing stable value-based optimization. We propose a state-aligned latent actor-critic framework for diffusion post-training, in which the diffusion model serves as its own timestep-conditioned value function and predicts values directly on noisy latent states. This enables trajectory-level PPO training, supports stable actor-critic optimization with simple conditioning and value pretraining strategies, and naturally allows the learned critic to be reused for inference-time steering. We further extend the framework to multi-reward optimization, where joint training with complementary rewards helps alleviate reward hacking. Across both UNet- and DiT-based backbones, our method consistently outperforms prior group-relative RL and actor-critic baselines on single-reward and multi-reward benchmarks, while test-time steering provides additional gains in generation quality.

Abstract

Diffusion Model Alignment

Explore similar work

Explicit Critic Guidance for Aligning Diffusion Models

Latent Reward Registers for Diffusion Preference Alignment

Explore or Converge? Stage-Guided Per-Step Optimization for Diffusion Models