Human preference alignment presents a critical yet underexplored challenge for diffusion models in text-to-3D generation. Existing solutions typically require task-specific fine-tuning, posing significant hurdles in data-scarce 3D domains. To address this, we propose Preference Score Distillation (PSD), an optimization-based framework that leverages pretrained 2D reward models for human-aligned text-to-3D synthesis without 3D training data. Our key insight stems from the incompatibility of pixel-level gradients: due to the absence of noisy samples during reward model training, direct application of 2D reward gradients disturbs the denoising process. Noticing that similar issue occurs in the naive classifier guidance in conditioned diffusion models, we fundamentally rethink preference alignment as a classifier-free guidance (CFG)-style mechanism through our implicit reward model. Furthermore, recognizing that frozen pretrained diffusion models constrain performance, we introduce an adaptive strategy to co-optimize preference scores and negative text embeddings. By incorporating CFG during optimization, online refinement of negative text embeddings dynamically enhances alignment. To our knowledge, we are the first to bridge human preference alignment with CFG theory under score distillation framework. Experiments demonstrate the superiority of PSD in aesthetic metrics, seamless integration with diverse pipelines, and strong extensibility.
Figures & tables
Figure 2: Overall illustration of Preference Score Distillation. a) Win (red) and lose (blue) samples are constructed on-the-fly to calculate win and lose scores, then preference score guidance (purple) pushes the denoising trajectory towards high-reward regions and finally improve alignment with reward. b) In each step, two noise is added to the rendering images gθ(c) and reward model determines win/lose based on one-step prediction of pretrained diffusion models ϵϕ . 3D representation θ and negative embedding n are updated by our objective LPSD(θ) and reward score respectively.
Figure 3: Effect of negative embedding optimization strategy on single prompt. Employ negative embedding optimization can significantly improve aesthetic score, but overlarge learning rate will harm visual quality.
Figure 4: Qualitative comparison of single-stage distillation of MVDream [ 52 ] ( 256×256 ). Our PSD significantly improve text alignment (red) and visual quality against comparing method DreamReward [ 80 ] and DreamDPO [ 89 ] .
Rewards
Methods
I.R. ↑
Pick. ↑
Aes. ↑
MPS ↑
T.A. ↑
HPSV2.1
MVDream
-0.22
20.55
5.79
9.30
53.14
DreamDPO
-0.28
20.48
5.80
9.00
75.68
Ours
0.12
20.99
5.92
10.26
75.70
Reward3D
DreamReward
1.78
21.40
6.15
10.19
74.37
Ours
1.80
21.49
6.25
10.40
90.63
Table 1: Quantitative Comparison of single-stage distillation of MVDream [ 52 ] across 200 prompts in Eval3d [ 11 ] with different rewards. Higher values are better ( ↑ ). The best performance is in bold.
Figure 5: Qualitative comparison of 2-stage NeRF [ 41 ] ( 512×512 ) and 3-Stage DMTet [ 50 ] ( 1024×1024 ) generation. PSD improves alignment with the prompts in red.
Pipeline
Methods
I.R. ↑
Pick. ↑
Aes. ↑
MPS ↑
T.A. ↑
2-Stage NeRF
CFD
-0.09
20.16
5.63
10.03
71.84
DreamReward
OOM
DreamDPO
-0.06
20.18
5.40
10.04
76.18
Ours
0.01
20.27
5.95
10.22
81.34
3-Stage DMTet
CFD
-0.44
19.76
5.30
9.45
71.84
RichDreamer*
0.02
19.68
5.92
8.24
76.45
Table 2: Quantitative Comparison of 2-stage NeRF [ 41 ] and 3-Stage DMTet [ 50 ] generation across 40-prompt subset. Higher values are better ( ↑ ). The best performance is in bold. Comparison method DreamReward [ 80 ] suffer from out-of-memory (OOM). * indicates method uses stronger diffusion priors.
Figure 6: User preference study of comparing PSD with MVDream, DreamDPO, and DreamReward.
Figure 7: Preference score guidance on image generation.
Methods
Target
Unseen Rewards
HPSv2.1 ↑
I.R. ↑
Pick. ↑
MPS ↑
DDIM
0.25
0.22
21.30
9.77
w/o δpref
0.25
0.04
21.15
9.51
Ours (Eq. 12 )
0.26
0.22
21.23
10.30
Table 3: Ablations of preference score on 2D image generation. Since our final results and experiments without δpref are performed using optimization, standard DDIM [ 53 ] are marked in gray.
Figure 8: Ablation of choosing different reward models.
Experiments
Target
Unseen Rewards
Generation Speed
HPSv2.1 ↑
I.R. ↑
Pick. ↑
MPS ↑
MVDream
0.20
-0.63
19.22
8.39
2.92 it/s
DreamDPO
0.19
-0.70
19.15
8.58
1.45 it/s
w/o n
0.20
-0.60
19.28
8.51
1.45 it/s
lr=4e−4
0.24
-0.49
19.85
9.98
1.19 it/s
1/5it−1
0.21
-0.38
19.45
9.35
2.45 it/s
Table 4: Ablations on negative embedding optimization strategy. w/o n represents experiments without negative embedding optimization. lr=4e−4 represents experiments with higher learning rate of learnable embedding. 1/5it−1 and 1/2it−1 represent performing preference guidance and negative embedding optimization with a interval of 2 and 5 steps respectively.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Notation
Description
xt
State variable at timestep t
αt,σt
Time-dependent coefficients
ϵ,ϵ′
Noise sampled from Gaussian distribution
ϵϕ(⋅)
Diffusion models parameterized by ϕ
y
Conditioning text prompt embedding
n
Negative prompt embedding
Appendix
Table 5: List of notations and their descriptions.
Figure 9: Comaprsion with CSD [ 86 ] and DreamDPO [ 89 ] . Artifacts are marked with red circles.
Algorithm 2 2D Image Generation using Score Distillation
Figure 11: Visual examples of failure cases.
Figure 12: Visualization of learned negative embeddings on diffusion priors.
Experiments
MVDream
Stable Diffusion 2.1
Pick. ↑
I.R. ↑
Pick. ↑
I.R. ↑
w/o n
20.23
-0.31
21.07
0.30
w/ n
20.31
-0.11
21.12
0.41
Appendix
Table 6: Impact of learned negative embeddings on diffusion priors. Metrics are evaluated across our 40-prompt subset on the average over 5 random seeds.
Figure 13: Visual examples of semantic consistency. DINO feature variance is clipped with threshold 0.20 (red).
Figure 14: Visual example of geometric consistency. Angular difference is clipped with threshold 23∘ (red).
Column 1
Column 2
1. A DSLR photo of a plate of fried chicken and waffles with maple syrup on them
21. A tiger playing the violin
2. A beautiful dress made out of garbage bags, on a mannequin. Studio lighting, high quality, high resolution
22. A wide angle DSLR photo of a colorful rooster
3. A zoomed out DSLR photo of an astronaut chopping vegetables in a sunlit kitchen
23. A lone, ancient tree stands tall in the middle of a quiet field
4. A wide angle zoomed out DSLR photo of A red dragon dressed in a tuxedo and playing chess. The chess pieces are fashioned after robots
24. A squirrel dressed like Henry VIII king of England
5. A zoomed out DSLR photo of a pita bread full of hummus and falafel and vegetables
25. A zoomed out DSLR photo of A punk rock squirrel in a studded leather jacket shouting into a microphone while standing on a stump and holding a beer
6. Several large, solid, cube-shaped parcels, wrapped in brown paper and tied with string, each labeled with a different destination, awaiting dispatch in a post office
26. A dragon-cat hybrid
Appendix
Table 8: 40 prompts for evaluation in 2-stage NeRF and 3-stage DMTet.
Figure 15: More results of single-stage distillation of MVDream
Figure 16: More results of single-stage distillation of MVDream
Figure 17: More results of 2-stage NeRF generation.
Figure 18: More results of 3-stage DMTet generation.
Figure 19: Extended comparison of 2-stage NeRF generation.
Aligning large-scale text-to-image diffusion models with nuanced human preferences remains challenging. While direct preference optimization (DPO) is simple and effective, large-scale finetuning often shows a generalization gap. We take inspiration from test-time guidance and cast preference alignment as classifier-free guidance (CFG): a finetuned preference model acts as an external control signal during sampling. Building on this view, we propose a simple method that improves alignment without retraining the base model. To further enhance generalization, we decouple preference learning into two modules trained on positive and negative data, respectively, and form a contrastive guidance vector at inference by subtracting their predictions (positive minus negative), scaled by a user-chosen strength and added to the base prediction at each step. This yields a sharper and controllable alignment signal. We evaluate on Stable Diffusion 1.5 and Stable Diffusion XL with Pick-a-Pic v2 and HPDv3, showing consistent quantitative and qualitative gains.
Zhou Jiang, Yandong Wen, Zhen Liu
Westlake University · The Chinese University of Hong Kong, Shenzhen
Reinforcement learning from human feedback (RLHF) has been popular for aligning text-to-image (T2I) diffusion models with human preferences. As a mainstream branch of RLHF, Direct Preference Optimization (DPO) offers a computationally efficient alternative that avoids explicit reward modeling and has been widely adopted in diffusion alignment. However, existing preference-based methods for diffusion alignment still rely on reward-induced preference signals and typically assume that human preferences can be adequately modeled by the Bradley--Terry (BT) model, which may fail to capture the full complexity of human preferences. In this paper, we formulate diffusion alignment from a game-theoretic perspective. We propose Diffusion Nash Preference Optimization (Diff.-NPO), an intuitive general preference framework for diffusion alignment. Diff.-NPO encourages the current policy to play against itself to achieve self improvement and lead to a better alignment. Empirically, we demonstrate the effectiveness of Diff.-NPO on the text-to-image generation task via various metrics. Diff.-NPO consistently outperforms existing preference-based diffusion alignment methods.
Jiaming Hu, Jiamu Bai, Haoyu Wang +2
Boston University · Penn State University · University at Albany
Diffusion models have achieved impressive results in generative tasks such as text-to-image synthesis, yet they often struggle to fully align outputs with nuanced user intent and maintain consistent aesthetic quality. Existing preference-based training methods such as Diffusion Direct Preference Optimization help address these issues, but obtain their supervision targets from the forward process q(xt−1∣xt,x0) derived from terminal samples, which is not directly aligned with the model's actual backward denoising transitions at each step. In this work, we introduce Direct Diffusion Score Preference Optimization (DDSPO), which defines stepwise preference supervision directly over backward denoising transitions through a contrastive policy pair, rather than relying on forward-process approximations from terminal samples. We propose two practical instantiations of the contrastive policy pair: training separate winning and losing models on preference data, and inducing a contrastive policy pair without additional training by using a pretrained reference model conditioned on an original prompt and a semantically degraded variant, requiring neither reward modeling nor manual annotations. Empirical results show that contrastive-policy-pair supervision is more effective than forward-process-based supervision across text-image alignment and aesthetic-quality tasks. Our implementation is available at: https://dohyun-as.github.io/DDSPO