CRAFT: Aligning Diffusion Models with Fine-Tuning Is Easier Than You Think

Zening Sun; Zhengpeng Xie; Lichen Bai; Shitong Shao; Shuo Yang; Zeke Xie

cs.CVMar 19, 2026

CRAFT: Aligning Diffusion Models with Fine-Tuning Is Easier Than You Think

Authors: Zening Sun, Zhengpeng Xie, Lichen Bai, Shitong Shao, Shuo Yang, Zeke Xie

Organizations: HKUST (GZ) · HIT (SZ)

Abstract

Aligning Diffusion models has achieved remarkable breakthroughs in generating high-quality, human preference-aligned images. Existing techniques, such as supervised fine-tuning (SFT) and DPO-style preference optimization, have become principled tools for fine-tuning diffusion models. However, SFT relies on high-quality images that are costly to obtain, while DPO-style methods depend on large-scale preference datasets, which are often inconsistent in quality. Beyond data dependency, these methods are further constrained by computational inefficiency. To address these two challenges, we propose Composite Reward Assisted Fine-Tuning (CRAFT), a lightweight yet powerful fine-tuning paradigm that requires significantly reduced training data while maintaining computational efficiency. It first leverages a Composite Reward Filtering (CRF) technique to construct a high-quality and consistent training dataset and then perform an enhanced variant of SFT. We also theoretically prove that CRAFT actually optimizes the lower bound of group-based reinforcement learning, establishing a principled connection between SFT with selected data and reinforcement learning. Our extensive empirical results demonstrate that CRAFT with only 100 samples can easily outperform recent SOTA preference optimization methods with thousands of preference-paired samples. Moreover, CRAFT can even achieve 11-220 $\times$ faster convergences than the baseline preference optimization methods, highlighting its extremely high efficiency.

Figures & tables

Figure 1: Training Efficiency Comparison. Compared to SPO and SmPO, CRAFT reaches the same HPSv2.1 performance with 19.7× and 60.1× faster training time respectively, and further achieves superior final performance, demonstrating a dual advantage in both training speed and final generation quality.

Figure 2: Qualitative Improvements over Vanilla SDXL. We present CRAFT, an efficient and effective fine-tuning method designed to enhance diffusion models’ alignment with human preference. The top row displays images generated by the base Vanilla-SDXL model, while the bottom row shows results generated by our CRAFT-finetuned model (CRAFT-SDXL). The comparison clearly shows that CRAFT-SDXL excels at complex instruction following and compositional reasoning, demonstrating significant superiority in adhering to diverse stylistic concepts (e.g., “Cyberpunk Style”), accurately generating specified objects and their attributes (e.g., the position of the cat “beside a rocket” and the teddy bear “wearing a helmet and cape”), and precisely rendering on-image text (e.g., “Deep Learning”).

Figure 3: The overall pipeline of our method mainly consists of three stages: (i) data construction, (ii) composite reward filtering, and (iii) weighted SFT fine-tuning.

\hat{A}^{(i,j)}=\frac{r_{\mathrm{total}}^{(i,j)}-\mathrm{mean}\{r_{\mathrm{total}}^{(i,1)},\dots,r_{\mathrm{total}}^{(i,N)}\}}{\mathrm{std}\{r_{\mathrm{total}}^{(i,1)},\dots,r_{\mathrm{total}}^{(i,N)}\}+\epsilon}

Algorithm 1 Composite Reward Assisted Fine-Tuning

Figure 4: Qualitative comparison. We provide a qualitative comparison of CRAFT-Diffusion and other different preference optimization methods (Vanilla, Diff-DPO, SPO) for SDXL. CRAFT-SDXL generates images of superior quality and prompt fidelity, showcasing significant improvements in detail, composition, and text rendering.

SDXL				SD1.5
Method	Data	GPU Hours	Cost Ratio	Method	Data	GPU Hours	Cost Ratio
Diff-DPO	851,293	$\sim$ 638.1	159.5 $\times$	Diff-DPO	851,293	$\sim$ 80.6	42.4 $\times$
MaPO	928,068	$\sim$ 545.6	136.4 $\times$	Diff-KTO	851,293	$\sim$ 415.6	218.7 $\times$
SmPO	851,293	$\sim$ 120.1	30.0 $\times$	SmPO	851,293	$\sim$ 27.6	14.5 $\times$
SPO	4,000	$\sim$ 63.0	15.8 $\times$	SPO	4,000	$\sim$ 22.0	11.6 $\times$
CRAFT (Ours)	100	$\sim$ 4.0	1.0 $\times$	CRAFT (Ours)	100	$\sim$ 1.9	1.0 $\times$

Table 1: Computational Cost Comparison. We report the total NVIDIA H100 GPU hours for the fine-tuning stage. Our CRAFT framework demonstrates a dramatic reduction in computational cost, achieving significant speedups over all baselines.

Model	HPDv2				Parti-Prompt				Pick-a-Pic
Model	HPSv2.1 $\uparrow$	AES $\uparrow$	ImgReward $\uparrow$	MPS $\uparrow$	HPSv2.1 $\uparrow$	AES $\uparrow$	ImgReward $\uparrow$	MPS $\uparrow$	HPSv2.1 $\uparrow$	AES $\uparrow$	ImgReward $\uparrow$	MPS $\uparrow$
SDXL	27.93	5.803	0.819	14.35	27.32	5.683	0.747	11.38	27.95	5.844	0.649	11.36
Diff-DPO	29.76	5.851	1.037	14.70	28.74	5.789	1.032	11.84	29.59	5.908	0.924	11.68
MaPO	29.05	5.925	0.925	14.46	28.03	5.851	0.864	11.46	28.88	6.001	0.794	11.33
SmPO ∗	31.40	5.950	1.081	14.89	30.03	5.890	1.100	11.96	31.03	6.000	0.984	11.73
SPO ∗	32.32	6.015	1.103	15.36	30.54	5.940	1.058	12.16	31.69	6.068	1.023	12.13

Table 2: Quantitative Results of our CRAFT and other baselines for SDXL. We report the average reward scores of images generated on HPDv2, Parti-Prompt, and Pick-a-Pic datasets. The highest value is shown in bold , the second highest is underlined , and models marked with ∗ are reproduced strictly following the official code and datasets.

Figure 5: Winning Rate Comparison on Parti-Prompt. CRAFT outperforms all baseline methods in average winning rate against the base SDXL model.

Model	HPDv2				Parti-Prompt				Pick-a-Pic
Model	HPS v2.1 $\uparrow$	AES $\uparrow$	ImgReward $\uparrow$	MPS $\uparrow$	HPS v2.1 $\uparrow$	AES $\uparrow$	ImgReward $\uparrow$	MPS $\uparrow$	HPS v2.1 $\uparrow$	AES $\uparrow$	ImgReward $\uparrow$	MPS $\uparrow$
SD1.5	24.57	5.500	0.234	12.24	25.27	5.470	0.246	9.79	24.83	5.488	0.107	9.12
Diff-DPO	25.81	5.602	0.428	12.77	26.07	5.540	0.404	10.18	25.88	5.558	0.269	9.56
Diff-KTO	28.56	5.681	0.740	13.12	28.10	5.667	0.641	10.28	28.10	5.662	0.624	9.55
SmPO ∗	26.41	5.497	0.432	12.65	26.51	5.492	0.425	10.03	26.68	5.534	0.320	9.55
SPO ∗	28.17	5.758	0.578	13.01	27.78	5.718	0.531	10.42	27.78	5.794	0.399	9.80

Table 3: Quantitative Results of our CRAFT and other baselines for SD1.5. We report the average reward scores of images generated on HPDv2, Parti-Prompt, and Pick-a-Pic datasets. The highest value is shown in bold , the second highest is underlined , and models marked with ∗ are reproduced strictly following the official code and datasets.

	Single	Two				Attribute
Method	Object	Object	Counting	Colors	Position	Binding	Overall
SD1.5	95.62	39.14	36.56	74.47	3.50	5.25	42.42
Diff-DPO	98.12	38.38	35.00	77.39	4.75	6.00	43.28
SmPO ∗	96.25	41.92	32.19	75.53	4.00	7.25	42.86
SPO ∗	95.94	49.75	33.44	72.61	6.25	6.25	44.04
CRAFT	96.88	47.22	38.12	77.66	6.75	8.25	45.82

Table 4: GenEval results of our CRAFT and baseline methods for SD1.5. The highest value is shown in bold, the second highest is underlined, and models marked with ∗ are reproduced strictly following the official code and datasets.

Strategy	Setup		Scores
Strategy	Data	Steps	HPS $\uparrow$	AES $\uparrow$	IR $\uparrow$
All	2,312	1,000	32.20	6.08	0.987
Low	500	500	31.86	5.94	1.040
Rand	500	500	33.10	6.07	1.110
Top	100	200	33.23	6.13	1.138

Table 5: Data selection strategy comparison. This table validates our data filtering approach, showing that a small but high-quality dataset is superior. The Top-100 strategy consistently achieves the best performance across all metrics, surpassing strategies that use significantly more, but lower-quality data.

Reward Models			Performance Metrics
HPS	AES	Pic	HPS $\uparrow$	AES $\uparrow$	IR $\uparrow$
✓			30.25	5.94	0.791
		✓	31.71	6.04	0.973
✓	✓		30.93	6.00	0.834
✓	✓	✓	32.20	6.08	0.987

Table 6: Reward model combination ablation. We evaluate the impact of different Composite Reward functions on fine-tuning SDXL. All images are generated with pick-a-pic test sets.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: ControlNet Visualization: Superior Control Fidelity and Data Efficiency of CRAFT. Qualitative comparison using diverse ControlNet conditions ( Canny and Depth ) against existing fine-tuning methods (SDXL, Diff-DPO, MaPO, SmPO). CRAFT consistently demonstrates strong geometric control fidelity and generates visually superior images across all modalities. This high robustness and aesthetic quality are achieved despite utilizing significantly fewer training samples, highlighting CRAFT’s superior data efficiency and generalization capability.

SDXL			SD1.5
Method	Hours	Ratio	Method	Hours	Ratio
Diff-DPO	$\sim$ 638.1	26.5 $\times$	Diff-DPO	$\sim$ 80.6	9.2 $\times$
MaPO	$\sim$ 545.6	22.6 $\times$	Diff-KTO	$\sim$ 415.6	47.2 $\times$
CRAFT	$\sim$ (4.0 + 20.1)	1.0 $\times$	CRAFT	$\sim$ (1.9 + 6.9)	1.0 $\times$

Appendix

Table 7: End-to-end cost in H100 hours. For CRAFT, we report both preprocessing and fine-tuning cost. Even with candidate generation and filtering included, CRAFT remains substantially more efficient than prior methods.

Table 15

Table 16

Figure 7: User study results. Human evaluation corroborates the automatic reward-based metrics and favors CRAFT over the compared baselines.

Method	Object	Object	Counting	Colors	Position	Binding	Overall
	Single	Two				Attribute
SDXL	97.50	70.96	42.81	88.30	11.00	21.00	55.05
Diff-DPO	98.75	80.56	45.62	86.70	11.00	20.75	57.23
SmPO ∗	98.75	79.55	44.69	86.70	10.50	27.00	57.86
SPO ∗	97.81	80.05	38.44	84.04	11.75	20.50	55.43
CRAFT (Ours)	99.06	86.36	36.88	87.23	14.50	23.75	57.97

Appendix

Table 12: GenEval results of our CRAFT and baseline methods for SDXL. The highest value is shown in bold, the second highest is underlined, and models marked with ∗ are reproduced strictly following the official code and datasets.

Figure 8: Qualitative results of CRAFT-SDXL. After CRAFT fine-tuning, the model demonstrates the capability to generate visually superior images with exceptional aesthetic quality.

Diffusion Model Alignment

Diffusion model alignment steers generated content and internal representations toward prompts, human preferences, safety goals, and task constraints.

See all 181 papers

Arena as Offline Reward: Efficient Fine-Grained Preference Optimization for Diffusion Models
May 7, 2026Zhikai Li, Yue Zhao, Edward Zhongwei Zhang +4Diffusion Model Alignment Preference Optimization
Institute of Automation, Chinese Academy of Sciences · University of California, Berkeley · University of California, Santa Barbara
Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models
May 26, 2026Austin Wang, Jiaqi Han, Stefano Ermon +1Diffusion Model Alignment Preference Optimization
Caltech · Stanford University
Rethinking Preference Alignment for Diffusion Models with Classifier-Free Guidance
Feb 21, 2026Zhou Jiang, Yandong Wen, Zhen LiuDiffusion Model Alignment Preference Alignment
Westlake University · The Chinese University of Hong Kong, Shenzhen

May 7, 2026J/K move · Enter open · S save

Arena as Offline Reward: Efficient Fine-Grained Preference Optimization for Diffusion Models

Zhikai Li, Yue Zhao, Edward Zhongwei Zhang, Xuewen Liu+3

Institute of Automation, Chinese Academy of Sciences · University of California, Berkeley · University of California, Santa Barbara

Diffusion Model Alignment Preference Optimization Offline Preference Optimization T2I Diffusion Models Preference Optimization for Diffusion Models

Open paper AskPDF arXiv

Reinforcement learning from human feedback (RLHF) effectively promotes preference alignment of text-to-image (T2I) diffusion models. To improve computational efficiency, direct preference optimization (DPO), which avoids explicit reward modeling, has been widely studied. However, its reliance on binary feedback limits it to coarse-grained modeling on chosen-rejected pairs, resulting in suboptimal optimization. In this paper, we propose ArenaPO, which leverages Arena scores as offline rewards to provide refined feedback, thus achieving efficient and fine-grained optimization without a reward model. This enables ArenaPO to benefit from both the rich rewards of traditional RLHF and the efficiency of DPO. Specifically, we first construct a model Arena in which each model's capability is represented as a Gaussian distribution, and infer these capabilities by traversing the annotated pairwise preferences. Each output image is treated as a sample from the corresponding capability distribution. Then, for a image pair, conditioned on the two capability distributions and the observed pairwise preference, the absolute quality gap is estimated using latent-variable inference based on truncated normal distribution, which serves as fine-grained feedback during training. It does not require a reward model and can be computed offline, thus introducing no additional training overhead. We conduct ArenaPO training on Pick-a-Pic v2 and HPD v3 datasets, showing that ArenaPO consistently outperforms existing baselines.

Abstract

Diffusion Model Alignment

Explore similar work

Arena as Offline Reward: Efficient Fine-Grained Preference Optimization for Diffusion Models

Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models

Rethinking Preference Alignment for Diffusion Models with Classifier-Free Guidance