Preference alignment has become a standard practice for text-to-image diffusion models. Direct Preference Optimization (DPO) simplifies this process by eliminating explicit reward modeling. Its diffusion variant, Diffusion-DPO, has become a widely adopted baseline. Diffusion-DPO essentially encourages the likelihood of preferred samples while suppressing dispreferred ones. In this paper, we revisit DPO-style alignment methods for diffusion models from the perspective of the manifold hypothesis. Under this view, natural images concentrate near a low-dimensional manifold embedded in the high-dimensional ambient space, whereas DPO directly optimizes preference distributions in the full space without accounting for this geometric structure. This creates a mismatch in the optimization dynamics: it suppresses geometry-preserving tangential updates, while insufficiently restricting hazardous normal-direction updates. This mismatch gradually degrades image quality and diversity. To address this issue, we propose Anisotropic Geometry-Aware Preference Optimization (APO), which replaces the uniform Euclidean treatment of prediction errors with a geometry-aware anisotropic metric derived from the reference model. Concretely, APO adaptively strengthens regularization in directions where the reference denoising function is highly sensitive, while relaxing constraints in directions that permit safe semantic adjustment. This recalibrates preference optimization according to the local manifold geometry, and maintains the original manifold structure. Experiments show that APO achieves strong performance and an average win rate exceeding 60% against various existing alignment methods across diverse benchmarks. It requires significantly fewer training steps than prior methods, and preserves generation diversity throughout training.
Figures & tables
Figure 1: Toy preference optimization on a manifold under different methods. A DDPM with a 4-layer MLP noise predictor is pretrained on a unit circle embedded in R32 . Preference pairs are constructed by randomly sampling one point from each semicircle, with the right semicircle as the winner. Columns denote training progress and rows denote methods. Red points indicate off-manifold samples. DPO, KTO, and InPO exhibit substantial off-manifold drift during alignment, whereas APO remains tightly concentrated around the underlying manifold while converging toward the winner region.
Figure 2: Schematic overview of our method . Standard Diffusion-DPO measures the noise-prediction residual δ=ϵ−ϵθ(xtk,t) under an isotropic Euclidean metric, treating tangent and normal directions equally and allowing updates to drift away from the local manifold Mt . APO instead employs a Jacobian-induced anisotropic metric R(δ;xtk) from the reference model, which strengthens regularization along geometry-sensitive directions while allowing more flexible movement along manifold-aligned directions
Figure 3: Spectrum analysis of the reference Jacobian on the Pick-a-Pic v2 dataset. The plot shows the singular value distribution of Jref , illustrating the anisotropic structure captured by our approach.
Absolute Scores ↑
Win Rate (%) vs. Baseline ↑
Dataset
Method
Aes.
Pick.
CLIP
IR.
Aes.
Pick.
CLIP
IR.
Avg.
HPSv2
Base (SD 1.5)
5.2473
21.037
33.258
0.1726
85.24
87.45
57.83
85.71
79.06
SFT
5.7971
21.738
33.569
0.5162
53.61
68.98
57.06
63.75
60.85
Diffusion-DPO
5.4192
21.551
35.412
0.3430
65.42
70.13
51.20
66.99
63.44
SDPO
5.4365
21.715
34.532
0.3936
58.27
63.83
56.96
61.45
60.13
Diffusion-KTO
5.6084
21.672
34.107
0.6295
60.08
71.62
54.18
59.30
61.30
Table 1: Main results and Win Rate comparison on SD1.5. Left: Absolute scores on four benchmarks. Right: Win Rate (%) of APO (Ours) vs. each baseline. The best results are bolded , second best are underlined . APO achieves the best or second-best performance on nearly all metrics across all three evaluation sets, and the win-rate results further corroborate these gains.
Figure 4: Human evaluation results. We compare Base-SDXL, DPO-SDXL, and APO-SDXL across four dimensions: Text–to-Content Similarity, Image Realism, Aesthetic Quality, and Overall Preference. APO-SDXL outperforms both baselines in all categories.
Figure 5: Qualitative comparison of various preference optimization methods based on SDXL. Prompt: 1) A long library corridor with arched ceiling, symmetrical wooden bookshelves, warm pendant lighting, photorealistic. 2) A smiling beautiful sorceress wearing a high necked blue suit surrounded by swirling rainbow aurora, hyper-realistic, cinematic, post-production. 3) Cowboy on horseback, sunset small town background, medium close-up, detailed leather chaps and revolver holster.
SD1.5
SDXL
Method
Time Hours(h) ↓
Peak Mem. (GB) ↓
Time Hours (h) ↓
Peak Mem. (GB) ↓
Diffusion-DPO
205.6
14.6
198.4
15.8
APO (Ours)
112.7
15.4
105.5
16.5
Table 2: Training Efficiency on HPSv2. We report Total Training Hours (h) and peak GPU memory (GB) on SD1.5 and SDXL. APO reduces the total training time to roughly half of standard Diffusion-DPO on both models, primarily due to faster convergence and fewer optimization steps. Despite introducing additional forward evaluations for Jacobian estimation, APO incurs only a modest increase in peak GPU memory of around 5%. Detailed analysis is provided in Section 5.2 .
Figure 6: Training Dynamics of Reward Scores. We plot the (a) PickScore, (b) Aesthetic Score, and (c) ImageReward of SD1.5 models throughout the training process. APO achieves competitive performance within significantly fewer training steps.
Figure 7: FID-50K throughout preference alignment on COCO-2017. Standard preference optimization methods exhibit an overall increase in FID as training progresses, whereas APO consistently maintains a lower FID and improves distributional fidelity throughout alignment. On SD1.5 and SDXL, APO reaches FID values of approximately 16.65 and 16.23, respectively, demonstrating improved distributional stability during preference optimization.
Figure 8: Ablation study on geometric information. We compare DPO, APO, and time-threshold variants where APO is applied only for t≥tτ with tτ∈{200,400,600,800} . Performance declines monotonically as tτ increases, with the sharpest drop between 400 and 600 steps, indicating that intermediate timesteps benefit most from anisotropic regularization.
Figure 9: Diversity and semantic variation during training. The left panel reports LPIPS Mean Pairwise Distance (MPD) [ 51 ] , while the right panel reports normalized DINO feature variance [ 52 ] . APO consistently maintains higher perceptual and semantic diversity than standard DPO throughout training, with the gap becoming more pronounced as training progresses. This suggests that its anisotropic regularization preserves greater flexibility along manifold-tangent directions while suppressing unstable updates along geometry-sensitive directions. Detailed analysis is provided in Section 5.3.2 .
Table 3: Hyperparameter sensitivity analysis of APO on the geometry strength λ , DPO temperature β , finite-difference perturbation radius r , and number of probe directions nv . The best results are bolded , second-best results are underlined . The results indicate that APO is robust to moderate changes in its hyperparameters and that the default configuration provides a favorable trade-off between alignment performance and computational cost.
Win Rate (%) vs. Baseline ↑
Dataset
Method
Aes.
Pick.
CLIP
IR.
Avg.
HPSv2
vs. Base (SDXL)
89.28
90.12
58.12
87.13
81.16
vs. SFT
56.85
60.17
57.32
58.29
58.16
vs. Diffusion-DPO
60.16
65.37
52.61
63.23
60.34
vs. SDPO
59.14
62.33
54.10
57.58
58.29
vs. InPO
54.06
59.37
47.80
56.02
54.66
Table 4: Win Rate comparison on SDXL. Results greater than 50% indicate that APO is preferred over the corresponding baseline and are bolded . APO consistently achieves above-50% win rates against most baselines, demonstrating its robust preference alignment performance across different benchmarks and evaluation criteria.
Figure 10: Qualitative comparisons among baselines on SD1.5 and SDXL. APO achieves higher prompt alignment, higher aesthetic quality, and richer details; most importantly, the realism of the images is greatly improved, indicating that the manifold constraint is taking effect.
Diffusion models have achieved impressive results in generative tasks such as text-to-image synthesis, yet they often struggle to fully align outputs with nuanced user intent and maintain consistent aesthetic quality. Existing preference-based training methods such as Diffusion Direct Preference Optimization help address these issues, but obtain their supervision targets from the forward process q(xt−1∣xt,x0) derived from terminal samples, which is not directly aligned with the model's actual backward denoising transitions at each step. In this work, we introduce Direct Diffusion Score Preference Optimization (DDSPO), which defines stepwise preference supervision directly over backward denoising transitions through a contrastive policy pair, rather than relying on forward-process approximations from terminal samples. We propose two practical instantiations of the contrastive policy pair: training separate winning and losing models on preference data, and inducing a contrastive policy pair without additional training by using a pretrained reference model conditioned on an original prompt and a semantically degraded variant, requiring neither reward modeling nor manual annotations. Empirical results show that contrastive-policy-pair supervision is more effective than forward-process-based supervision across text-image alignment and aesthetic-quality tasks. Our implementation is available at: https://dohyun-as.github.io/DDSPO
Reinforcement learning from human feedback (RLHF) has been popular for aligning text-to-image (T2I) diffusion models with human preferences. As a mainstream branch of RLHF, Direct Preference Optimization (DPO) offers a computationally efficient alternative that avoids explicit reward modeling and has been widely adopted in diffusion alignment. However, existing preference-based methods for diffusion alignment still rely on reward-induced preference signals and typically assume that human preferences can be adequately modeled by the Bradley--Terry (BT) model, which may fail to capture the full complexity of human preferences. In this paper, we formulate diffusion alignment from a game-theoretic perspective. We propose Diffusion Nash Preference Optimization (Diff.-NPO), an intuitive general preference framework for diffusion alignment. Diff.-NPO encourages the current policy to play against itself to achieve self improvement and lead to a better alignment. Empirically, we demonstrate the effectiveness of Diff.-NPO on the text-to-image generation task via various metrics. Diff.-NPO consistently outperforms existing preference-based diffusion alignment methods.
Jiaming Hu, Jiamu Bai, Haoyu Wang +2
Boston University · Penn State University · University at Albany
Preference optimization has emerged as an efficient alternative to online reinforcement learning from human feedback (RLHF) for aligning text-to-image diffusion models. However, existing methods largely reduce supervision to binary pairwise comparisons. This pairwise reduction is limiting when training data naturally contains multiple candidate images for the same prompt, and when continuous reward scores can provide richer information than a single winner-loser label. To address these limitations, we propose Diffusion LAIR, a reward-aware listwise preference optimization method for diffusion models. For each prompt, LAIR converts reward scores across a group of candidate images into centered advantage weights, then optimizes an advantage-weighted regression objective on the implicit reward, defined as the denoising-loss improvement of the current model over a fixed reference model, with a quadratic penalty that regularizes the magnitude of the implicit reward. The resulting objective uses all candidates simultaneously rather than selecting pairs, and remains conservative by explicitly controlling the magnitude of the implicit reward. The LAIR objective admits a bounded closed-form optimum in implicit-reward space, clarifying how the regularization strength controls the magnitude of the preference update. Experiments show that Diffusion LAIR outperforms strong preference optimization baselines on SD1.5 and SDXL across text-to-image generation, compositional generation, and image editing benchmarks.