cs.CVFeb 21, 2026
SaveRethinking Preference Alignment for Diffusion Models with Classifier-Free Guidance
Organizations: Westlake University · The Chinese University of Hong Kong, Shenzhen
Abstract
Aligning large-scale text-to-image diffusion models with nuanced human preferences remains challenging. While direct preference optimization (DPO) is simple and effective, large-scale finetuning often shows a generalization gap. We take inspiration from test-time guidance and cast preference alignment as classifier-free guidance (CFG): a finetuned preference model acts as an external control signal during sampling. Building on this view, we propose a simple method that improves alignment without retraining the base model. To further enhance generalization, we decouple preference learning into two modules trained on positive and negative data, respectively, and form a contrastive guidance vector at inference by subtracting their predictions (positive minus negative), scaled by a user-chosen strength and added to the base prediction at each step. This yields a sharper and controllable alignment signal. We evaluate on Stable Diffusion 1.5 and Stable Diffusion XL with Pick-a-Pic v2 and HPDv3, showing consistent quantitative and qualitative gains.
Figures & tables
Figure 1: Toy 2D experiment on DPO (top) and our proposed PGD (below) to demonstrate the overfitting issue in DPO training. Black circles indicate positive sample clusters and red crosses indicate negative sample clusters. is the guidance weight of PGD and is the DPO scale parameter.
Figure 4: Comparison of preference-optimization methods on SDXL. Columns show outputs from the base model (SDXL), DPO, MaPO, NPO, PGD, and cPGD. PGD and cPGD achieves the highest rewards and is the most effective in aligning with human preference implied in the Pick-a-Pic v2 dataset.
| Pick-a-Pic v2 test (424 prompts) | Parti-Prompts (1632 prompts) | |||||||||||||
| Base Model | Inference Strategy | PS | HPSv2 | HPSv3 | Aes | CLIP | IR | PS | HPSv2 | HPSv3 | Aes | CLIP | IR | Avg. |
| – | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | |
| NPO | 58.7 | 59.2 | 69.1 | 52.1 | 37.5 | 53.5 | 55.0 | 56.4 | 62.1 | 55.1 | 39.0 | 51.5 | 54.1 | |
| PGD | 78.8 | 79.0 | 73.6 | 51.9 | 63.7 | 69.3 | 78.7 | 78.4 | 78.2 | 68.8 | 54.0 | 69.1 | 7 0.3 | |
| SDXL | cPGD | 80.0 | 80.2 | 77.1 | 50.9 | 64.9 | 69.8 | 75.8 | 80.3 | 77.8 | 57.2 | 62.0 | 74.0 | 70.8 |
| – | 71.7 | 77.6 | 67.9 | 53.3 | 61.6 | 65.8 | 64.0 | 70.3 | 64.0 | 57.5 | 58.5 | 69.7 | 65.2 | |
Table 1: Win rates of preference optimization methods against the SDXL model on the Pick-a-Pic v2 test set and the Parti-Prompts benchmark. Model checkpoints for other methods are provided by their respective authors. The 1st-best results are bolded and the 2nd-best results are underlined .
| Pick-a-Pic v2 test (424 prompts) | Parti-Prompts (1632 prompts) | |||||||||||||
| Base Model | Inference Strategy | PS | HPSv2 | HPSv3 | Aes | CLIP | IR | PS | HPSv2 | HPSv3 | Aes | CLIP | IR | Avg. |
| – | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | |
| PGD | 78.3 | 71.2 | 67.9 | 62.3 | 58.5 | 63.7 | 68.0 | 65.0 | 59.6 | 58.1 | 55.0 | 56.9 | 6 3.7 | |
| SD1.5 | cPGD | 76.9 | 71.7 | 71.9 | 63.2 | 59.9 | 72.2 | 66.4 | 76.9 | 68.9 | 68.1 | 58.4 | 71.0 | 68.8 |
| – | 76.4 | 67.7 | 66.3 | 65.1 | 55.9 | 60.6 | 67.3 | 64.8 | 64.5 | 62.2 | 53.6 | 61.0 | 63.8 | |
| PGD | 79.2 | 70.3 | 66.3 | 66.3 | 59.4 | 63.2 | 74.2 | 67.4 | 65.0 | 63.1 | 55.5 | 62.9 | 6 6.1 | |
Table 2: Win rates of preference optimization methods against the SD1.5 model on the Pick-a-Pic v2 test set and the Parti-Prompts benchmark. Model checkpoints for other methods are provided by their respective authors. The 1st-best results are bolded and the 2nd-best results are underlined .
Figure 5: Overall comparison on SDXL. Radar axes report mean scores (higher is better): PickScore (PS), HPSv2, HPSv3, CLIP, and ImageReward (IR). Polygons closer to the outer rim indicate better aggregate performance across metrics.
Figure 8: Qualitative effect of increasing guidance weight (left right). Rows show text fidelity, semantic binding, and aesthetic style. Stronger improves alignment and legibility up to a mid range, after which overshooting/rigidity appears.
Figure 9: Effect of guidance weight on automatic metrics (SDXL). Left: PickScore; Middle: CLIP score; Right: HPSv3. “Step” denotes the training steps of the guidance module. Curves rise quickly for small ,
| Set | Method | PS | HPSv2 | HPSv3 | Aes | CLIP | IR |
| Subset | SDXL | 0.222 | 0.277 | 7.079 | 6.052 | 0.363 | 0.658 |
| DPO | 0.225 | 0.282 | 8.032 | 6.112 | 0.361 | 0.823 | |
| PGD | 0.226 | 0.285 | 9.459 | 6.203 | 0.364 | 0.874 | |
| cPGD | 0.228 | 0.289 | 10.045 | 6.267 | 0.365 | 1.031 | |
| Fullset | SDXL | 0.222 | 0.277 | 7.079 | 6.052 | 0.363 | 0.658 |
| DPO | 0.226 | 0.284 | 8.261 | 6.165 | 0.366 | 0.918 |
Table 3: Impact of preference data variance on alignment performance. “Subset” refers to training on a high-quality curated subset (low variance), while “Fullset” uses the full HPDv3 dataset (high variance). The 1st-best results are in bold and the 2nd-best are underlined . All methods are applied to the base SDXL model.
Figure 10: Win rate (%) of different finetuned models on different reward models. For each prompt (with same random seeds), we score images from all three models and report the proportion on which each model achieves the highest score. Results show that cPGD-distill is superior.
Figure 11: Negative pair examples of HPDv3
Figure 12: Positive pair examples of HPDv3
Figure 13: The real 2-D data and the pretrain distribution on the Toy 2D experiment.
Figure 14: Toy 2D experiment on DPO with different values of .
Figure 15: The positive and negative SFT components of the DPO loss on the toy 2D experiment.
Figure 16: Toy 2D experiment on SFT. The top row shows SFT on positive samples only (SFT+); the bottom row shows SFT on the negative samples (SFT-).
Figure 17: Toy 2D experiment on PGD for different guidance weights .
| Method | PS | HPSv2 | HPSv3 | Aes | CLIP | IR | Avg. |
| SDXL | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 |
| NPO | 58.7 | 59.2 | 69.1 | 52.1 | 37.5 | 53.5 | 55.0 |
| MaPO | 55.9 | 65.3 | 61.8 | 68.2 | 50.2 | 68.2 | 61.6 |
| DPO | 71.7 | 77.6 | 67.9 | 53.3 | 61.6 | 65.8 | 66.3 |
| PGD-merge | 87.0 | 88.9 | 84.4 | 59.9 | 63.7 | 77.6 | 76.9 |
| cPGD-merge | 76.2 | 84.2 | 72.4 | 45.0 | 68.9 | 71.7 | 69.7 |
Table 4: Win rates(%) of preference optimization methods against the SDXL model on the Pick-a-Pic v2 test set. Model checkpoints for other methods are provided by their respective authors. The 1st-best results are bolded.
Figure 18: Plug-and-play transfer to KOALA using an SDXL PGD/cPGD score module. Pairwise preference rates (%) of PGD-KOALA (top) and cPGD-KOALA (bottom) against the original KOALA baseline under four evaluation metrics (PS, IR, CLIP, HPSv2). The dashed vertical line denotes parity at 50%; values above 50% indicate that the guided model is preferred. All gains are obtained without any training or fine-tuning of KOALA , highlighting the plug-and-play nature of PGD/cPGD under shared latent-space (VAE) alignment.
| Method | Votes | Vote share (%) |
| Raw | 208 | 18.9 |
| DPO | 324 | 29.5 |
| MaPO | 197 | 17.9 |
| NPO | 256 | 23.3 |
| PGD | 500 | 45.5 |
| cPGD | 363 | 33.0 |
Table 5: Summary of human preference votes. Participants could select 1–3 images per question, hence the total number of votes exceeds the number of questions.
Figure 19: Comparison of SFT training with and without weighting. Blue bars indicate naive SFT, and orange bars indicate weighted SFT. All results are obtained under the cPGD inference strategy with the same guidance weight.
| Guided steps | PickScore win rate (%) | Time ratio ( vs. SDXL) |
| 10 | 70.2 | 1.4 |
| 20 | 72.4 | 1.8 |
| 30 | 74.8 | 2.2 |
| 40 | 76.9 | 2.6 |
| 50 | 79.1 | 3.0 |
Table 6: Guiding only the first of 50 steps on SDXL (cPGD).
Figure 20: Effect of guidance weight on automatic metrics (SDXL). Left: ImageReward; Right: HPSv2. “Step” denotes the training steps of the guidance module. Curves rise quickly for small ,
| Base Model | Inference Strategy | PS | HPSv2 | HPSv3 | Aes | CLIP | IR | Average |
| – | 50.00% | 50.00% | 50.00% | 50.00% | 50.00% | 50.00% | 50.00% | |
| cPGD | 79.95% | 80.19% | 77.12% | 50.94% | 64.86% | 69.81% | 70.48% | |
| PGD | 78.77% | 79.01% | 73.58% | 51.89% | 63.68% | 69.34% | 69.38% | |
| SDXL | NPO | 58.73% | 59.20% | 69.10% | 52.12% | 37.50% | 53.54% | 55.03% |
| – | 71.70% | 77.59% | 67.92% | 53.30% | 61.56% | 65.80% | 66.31% | |
| cPGD | 80.90% | 77.59% | 84.67% | 63.92% | 58.73% | 64.86% | 71.78% |
Table 7: Win rates of preference optimization methods against the SDXL model on the Pick-a-Pic v2 test set benchmark. Model checkpoints for other methods are provided by their respective authors.
| Base Model | Inference Strategy | PS | HPSv2 | HPSv3 | Aes | CLIP | IR | Average |
| – | 50.00% | 50.00% | 50.00% | 50.00% | 50.00% | 50.00% | 50.00% | |
| cPGD | 76.50% | 75.50% | 71.25% | 52.25% | 59.00% | 69.00% | 67.25% | |
| PGD | 85.50% | 73.50% | 78.75% | 69.25% | 46.75% | 73.00% | 71.13% | |
| SDXL | NPO | 63.75% | 65.00% | 78.75% | 60.50% | 42.75% | 55.50% | 61.04% |
| – | 66.75% | 73.75% | 66.00% | 59.75% | 55.25% | 73.00% | 65.75% | |
| cPGD | 75.25% | 79.50% | 71.25% | 57.75% | 61.00% | 73.00% | 69.63% |
Table 8: Win rates of preference optimization methods against the SDXL model on the HPD v2 test set benchmark. Model checkpoints for other methods are provided by their respective authors.
| Base Model | Inference Strategy | PS | HPSv2 | HPSv3 | Aes | CLIP | IR | Average |
| – | 50.00% | 50.00% | 50.00% | 50.00% | 50.00% | 50.00% | 50.00% | |
| cPGD | 75.80% | 80.27% | 77.82% | 57.23% | 62.01% | 74.02% | 71.19% | |
| PGD | 78.74% | 78.37% | 78.19% | 68.75% | 53.98% | 69.06% | 71.18% | |
| SDXL | NPO | 55.02% | 56.37% | 62.07% | 55.09% | 38.97% | 51.53% | 53.18% |
| – | 63.97% | 70.28% | 64.03% | 57.54% | 58.46% | 69.73% | 64.00% | |
| cPGD | 73.90% | 79.23% | 72.79% | 59.38% | 64.09% | 74.45% | 70.64% |
Table 9: Win rates of preference optimization methods against the SDXL model on the Parti-Prompts set benchmark. Model checkpoints for other methods are provided by their respective authors.
| Base Model | Inference Strategy | PS | HPSv2 | HPSv3 | Aes | CLIP | IR | Average |
| – | 50.00% | 50.00% | 50.00% | 50.00% | 50.00% | 50.00% | 50.00% | |
| cPGD | 76.89% | 71.70% | 71.93% | 63.21% | 59.91% | 72.17% | 69.30% | |
| SD1.5 | PGD | 78.30% | 71.23% | 67.92% | 62.26% | 58.49% | 63.68% | 66.98% |
| – | 76.42% | 67.69% | 66.27% | 65.09% | 55.90% | 60.61% | 65.33% | |
| cPGD | 79.01% | 81.84% | 75.47% | 71.70% | 62.26% | 76.18% | 74.41% | |
| DPO-SD1.5 | PGD | 79.25% | 70.28% | 66.27% | 66.27% | 59.43% | 63.21% | 67.45% |
Table 10: Win rates of preference optimization methods against the SD1.5 model on the Pick-a-Pic v2 test set benchmark. Model checkpoints for other methods are provided by their respective authors.
| Base Model | Inference Strategy | PS | HPSv2 | HPSv3 | Aes | CLIP | IR | Average |
| – | 50.00% | 50.00% | 50.00% | 50.00% | 50.00% | 50.00% | 50.00% | |
| cPGD | 80.50% | 88.00% | 84.00% | 73.00% | 61.25% | 79.50% | 77.71% | |
| SD1.5 | PGD | 75.25% | 68.00% | 67.00% | 67.25% | 52.25% | 65.50% | 65.88% |
| – | 76.25% | 69.75% | 68.00% | 66.50% | 57.50% | 65.25% | 67.21% | |
| cPGD | 80.25% | 81.00% | 79.00% | 70.25% | 60.75% | 75.75% | 74.50% | |
| DPO-SD1.5 | PGD | 79.00% | 73.75% | 71.75% | 66.00% | 54.75% | 68.50% | 68.96% |
Table 11: Win rates of preference optimization methods against the SD1.5 model on the HPD v2 test set benchmark. Model checkpoints for other methods are provided by their respective authors.
| Base Model | Inference Strategy | PS | HPSv2 | HPSv3 | Aes | CLIP | IR | Average |
| – | 50.00% | 50.00% | 50.00% | 50.00% | 50.00% | 50.00% | 50.00% | |
| cPGD | 66.36% | 76.90% | 68.87% | 68.14% | 58.39% | 71.02% | 68.28% | |
| SD1.5 | PGD | 68.01% | 65.01% | 59.56% | 58.15% | 54.96% | 56.86% | 60.43% |
| – | 67.34% | 64.83% | 64.46% | 62.19% | 53.55% | 61.03% | 62.23% | |
| cPGD | 73.84% | 74.08% | 69.61% | 69.00% | 59.01% | 67.34% | 68.81% | |
| DPO-SD1.5 | PGD | 74.20% | 67.40% | 65.01% | 63.05% | 55.51% | 62.87% | 64.68% |
Table 12: Win rates of preference optimization methods against the SD1.5 model on the Parti-Prompts set benchmark. Model checkpoints for other methods are provided by their respective authors.
Figure 21: General Results with SDXL, SD1.5.
Figure 22: Comparison of preference-optimization methods on SDXL. Columns show outputs from the base model (SDXL), DPO, MaPO, NPO and PGD.
Figure 23: Comparison of preference-optimization methods on SDXL. Columns show outputs from the base model (SDXL), DPO, MaPO, NPO and PGD.
Figure 24: Comparison of preference-optimization methods on SDXL. Columns show outputs from the base model (SDXL), DPO, MaPO, NPO and PGD.
Figure 25: Comparison of preference-optimization methods on SDXL. Columns show outputs from the base model (SDXL), DPO, MaPO, NPO and cPGD.
Figure 26: Comparison of preference-optimization methods on SDXL. Columns show outputs from the base model (SDXL), DPO, MaPO, NPO and cPGD.
Figure 27: Comparison of preference-optimization methods on SDXL. Columns show outputs from the base model (SDXL), DPO, MaPO, NPO and cPGD.