Preference Score Distillation: Leveraging 2D Rewards to Align Text-to-3D Generation with Human Preference

Jiaqi Leng; Shuyuan Tu; Haidong Cao; Sicheng Xie; Daoguo Dong; Zuxuan Wu; Yu-Gang Jiang

cs.CVMar 2, 2026

Preference Score Distillation: Leveraging 2D Rewards to Align Text-to-3D Generation with Human Preference

Authors: Jiaqi Leng, Shuyuan Tu, Haidong Cao, Sicheng Xie, Daoguo Dong, Zuxuan Wu, Yu-Gang Jiang

Organizations: Fudan University

Abstract

Human preference alignment presents a critical yet underexplored challenge for diffusion models in text-to-3D generation. Existing solutions typically require task-specific fine-tuning, posing significant hurdles in data-scarce 3D domains. To address this, we propose Preference Score Distillation (PSD), an optimization-based framework that leverages pretrained 2D reward models for human-aligned text-to-3D synthesis without 3D training data. Our key insight stems from the incompatibility of pixel-level gradients: due to the absence of noisy samples during reward model training, direct application of 2D reward gradients disturbs the denoising process. Noticing that similar issue occurs in the naive classifier guidance in conditioned diffusion models, we fundamentally rethink preference alignment as a classifier-free guidance (CFG)-style mechanism through our implicit reward model. Furthermore, recognizing that frozen pretrained diffusion models constrain performance, we introduce an adaptive strategy to co-optimize preference scores and negative text embeddings. By incorporating CFG during optimization, online refinement of negative text embeddings dynamically enhances alignment. To our knowledge, we are the first to bridge human preference alignment with CFG theory under score distillation framework. Experiments demonstrate the superiority of PSD in aesthetic metrics, seamless integration with diverse pipelines, and strong extensibility.

Figures & tables

Figure 2: Overall illustration of Preference Score Distillation. a) Win (red) and lose (blue) samples are constructed on-the-fly to calculate win and lose scores, then preference score guidance (purple) pushes the denoising trajectory towards high-reward regions and finally improve alignment with reward. b) In each step, two noise is added to the rendering images $g_{\theta}(\bm{c})$ and reward model determines win/lose based on one-step prediction of pretrained diffusion models $\bm{\epsilon}_{\phi}$ . 3D representation $\theta$ and negative embedding $n$ are updated by our objective $\mathcal{L}_{\text{PSD}}(\theta)$ and reward score respectively.

Figure 3: Effect of negative embedding optimization strategy on single prompt. Employ negative embedding optimization can significantly improve aesthetic score, but overlarge learning rate will harm visual quality.

Figure 4: Qualitative comparison of single-stage distillation of MVDream [ 52 ] ( $256\times 256$ ). Our PSD significantly improve text alignment (red) and visual quality against comparing method DreamReward [ 80 ] and DreamDPO [ 89 ] .

Rewards	Methods	I.R. $\uparrow$	Pick. $\uparrow$	Aes. $\uparrow$	MPS $\uparrow$	T.A. $\uparrow$
HPSV2.1	MVDream	-0.22	20.55	5.79	9.30	53.14
	DreamDPO	-0.28	20.48	5.80	9.00	75.68
	Ours	0.12	20.99	5.92	10.26	75.70
Reward3D	DreamReward	1.78	21.40	6.15	10.19	74.37
Reward3D	Ours	1.80	21.49	6.25	10.40	90.63

Table 1: Quantitative Comparison of single-stage distillation of MVDream [ 52 ] across 200 prompts in Eval3d [ 11 ] with different rewards. Higher values are better (

\uparrow

). The best performance is in bold.

Figure 5: Qualitative comparison of 2-stage NeRF [ 41 ] ( $512\times 512$ ) and 3-Stage DMTet [ 50 ] ( $1024\times 1024$ ) generation. PSD improves alignment with the prompts in red.

Pipeline	Methods	I.R. $\uparrow$	Pick. $\uparrow$	Aes. $\uparrow$	MPS $\uparrow$	T.A. $\uparrow$
2-Stage NeRF	CFD	-0.09	20.16	5.63	10.03	71.84
	DreamReward	OOM
	DreamDPO	-0.06	20.18	5.40	10.04	76.18
	Ours	0.01	20.27	5.95	10.22	81.34
3-Stage DMTet	CFD	-0.44	19.76	5.30	9.45	71.84
	RichDreamer*	0.02	19.68	5.92	8.24	76.45

Table 2: Quantitative Comparison of 2-stage NeRF [ 41 ] and 3-Stage DMTet [ 50 ] generation across 40-prompt subset. Higher values are better (

\uparrow

). The best performance is in bold. Comparison method DreamReward [ 80 ] suffer from out-of-memory (OOM). * indicates method uses stronger diffusion priors.

Figure 6: User preference study of comparing PSD with MVDream, DreamDPO, and DreamReward.

Figure 7: Preference score guidance on image generation.

Methods	Target	Unseen Rewards
Methods	HPSv2.1 $\uparrow$	I.R. $\uparrow$	Pick. $\uparrow$	MPS $\uparrow$
DDIM	0.25	0.22	21.30	9.77
w/o $\delta_{pref}$	0.25	0.04	21.15	9.51
Ours (Eq. 12 )	0.26	0.22	21.23	10.30

Table 3: Ablations of preference score on 2D image generation. Since our final results and experiments without

\delta_{pref}

are performed using optimization, standard DDIM [ 53 ] are marked in gray.

Figure 8: Ablation of choosing different reward models.

Experiments	Target	Unseen Rewards			Generation Speed
Experiments	HPSv2.1 $\uparrow$	I.R. $\uparrow$	Pick. $\uparrow$	MPS $\uparrow$	Generation Speed
MVDream	0.20	-0.63	19.22	8.39	2.92 it/s
DreamDPO	0.19	-0.70	19.15	8.58	1.45 it/s
w/o $n$	0.20	-0.60	19.28	8.51	1.45 it/s
$lr=4e^{-4}$	0.24	-0.49	19.85	9.98	1.19 it/s
$1/5it^{-1}$	0.21	-0.38	19.45	9.35	2.45 it/s

Table 4: Ablations on negative embedding optimization strategy. w/o n represents experiments without negative embedding optimization.

lr=4e^{-4}

represents experiments with higher learning rate of learnable embedding.

1/5it^{-1}

and

1/2it^{-1}

represent performing preference guidance and negative embedding optimization with a interval of 2 and 5 steps respectively.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
$\bm{x}_{t}$	State variable at timestep $t$
$\alpha_{t},\sigma_{t}$	Time-dependent coefficients
$\epsilon,\epsilon^{\prime}$	Noise sampled from Gaussian distribution
$\bm{\epsilon}_{\phi}(\cdot)$	Diffusion models parameterized by $\phi$
$y$	Conditioning text prompt embedding
$n$	Negative prompt embedding

Appendix

Table 5: List of notations and their descriptions.

Figure 9: Comaprsion with CSD [ 86 ] and DreamDPO [ 89 ] . Artifacts are marked with red circles.

Figure 10: Screen shot of our survey web.

\beta_{r}\leftarrow\gamma\frac{\|\delta_{cls}\|_{2}}{\|\delta_{pref}\|_{2}}\cdot\sigma\!\left(r(y,\hat{\bm{x}}_{0}^{l})-r(y,\hat{\bm{x}}_{0}^{w})\right)

Appendix

Algorithm 1 Preference Score Distillation

\delta_{gen}\leftarrow\bm{\epsilon}_{\phi}(\bm{x}_{t_{i}+\tau}^{w},t_{i}+\tau)-\bm{\epsilon}

Appendix

Algorithm 2 2D Image Generation using Score Distillation

Figure 11: Visual examples of failure cases.

Figure 12: Visualization of learned negative embeddings on diffusion priors.

Experiments	MVDream		Stable Diffusion 2.1
Experiments	Pick. $\uparrow$	I.R. $\uparrow$	Pick. $\uparrow$	I.R. $\uparrow$
w/o $n$	20.23	-0.31	21.07	0.30
w/ $n$	20.31	-0.11	21.12	0.41

Appendix

Table 6: Impact of learned negative embeddings on diffusion priors. Metrics are evaluated across our 40-prompt subset on the average over 5 random seeds.

Figure 13: Visual examples of semantic consistency. DINO feature variance is clipped with threshold 0.20 (red).

Figure 14: Visual example of geometric consistency. Angular difference is clipped with threshold $23^{\circ}$ (red).

Column 1	Column 2
1. A DSLR photo of a plate of fried chicken and waffles with maple syrup on them	21. A tiger playing the violin
2. A beautiful dress made out of garbage bags, on a mannequin. Studio lighting, high quality, high resolution	22. A wide angle DSLR photo of a colorful rooster
3. A zoomed out DSLR photo of an astronaut chopping vegetables in a sunlit kitchen	23. A lone, ancient tree stands tall in the middle of a quiet field
4. A wide angle zoomed out DSLR photo of A red dragon dressed in a tuxedo and playing chess. The chess pieces are fashioned after robots	24. A squirrel dressed like Henry VIII king of England
5. A zoomed out DSLR photo of a pita bread full of hummus and falafel and vegetables	25. A zoomed out DSLR photo of A punk rock squirrel in a studded leather jacket shouting into a microphone while standing on a stump and holding a beer
6. Several large, solid, cube-shaped parcels, wrapped in brown paper and tied with string, each labeled with a different destination, awaiting dispatch in a post office	26. A dragon-cat hybrid

Appendix

Table 8: 40 prompts for evaluation in 2-stage NeRF and 3-stage DMTet.

Figure 15: More results of single-stage distillation of MVDream

Figure 16: More results of single-stage distillation of MVDream

Figure 17: More results of 2-stage NeRF generation.

Figure 18: More results of 3-stage DMTet generation.

Figure 19: Extended comparison of 2-stage NeRF generation.

Text-to-3D Generation

Text-to-3D generation creates 3D objects and scenes from language while balancing semantic alignment, geometry, appearance, and spatial coherence.

See all 64 papers

Rethinking Preference Alignment for Diffusion Models with Classifier-Free Guidance
Feb 21, 2026Zhou Jiang, Yandong Wen, Zhen LiuDiffusion Model Alignment Preference Alignment
Westlake University · The Chinese University of Hong Kong, Shenzhen
Towards General Preference Alignment: Diffusion Models at Nash Equilibrium
May 6, 2026Jiaming Hu, Jiamu Bai, Haoyu Wang +2Diffusion Model Alignment Preference Optimization
Boston University · Penn State University · University at Albany
Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervision
Dec 29, 2025Dohyun Kim, Seungwoo Lyu, Seung Wook Kim +1Diffusion Model Alignment Diffusion Model Fine-Tuning
Dept. of CSE, Korea University · 2NVIDIA

Feb 21, 2026J/K move · Enter open · S save

Rethinking Preference Alignment for Diffusion Models with Classifier-Free Guidance

Zhou Jiang, Yandong Wen, Zhen Liu

Westlake University · The Chinese University of Hong Kong, Shenzhen

Diffusion Model Alignment Preference Alignment Preference Optimization for Diffusion Models T2I Diffusion Models Classifier-Free Guidance

Open paper AskPDF arXiv

Aligning large-scale text-to-image diffusion models with nuanced human preferences remains challenging. While direct preference optimization (DPO) is simple and effective, large-scale finetuning often shows a generalization gap. We take inspiration from test-time guidance and cast preference alignment as classifier-free guidance (CFG): a finetuned preference model acts as an external control signal during sampling. Building on this view, we propose a simple method that improves alignment without retraining the base model. To further enhance generalization, we decouple preference learning into two modules trained on positive and negative data, respectively, and form a contrastive guidance vector at inference by subtracting their predictions (positive minus negative), scaled by a user-chosen strength and added to the base prediction at each step. This yields a sharper and controllable alignment signal. We evaluate on Stable Diffusion 1.5 and Stable Diffusion XL with Pick-a-Pic v2 and HPDv3, showing consistent quantitative and qualitative gains.

Abstract

Text-to-3D Generation

Explore similar work

Rethinking Preference Alignment for Diffusion Models with Classifier-Free Guidance

Towards General Preference Alignment: Diffusion Models at Nash Equilibrium

Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervision