Alignment of text-to-image diffusion models with reward functions is a key challenge in controllable generation. Existing approaches typically rely on optimizing model inputs or fine-tuning denoising networks, which requires computationally expensive backpropagation through large denoising models. In this work, we propose Direct Trajectory Optimization (DTO), a training-free framework that performs alignment by directly optimizing the sampling trajectory of diffusion models. Specifically, DTO treats intermediate denoising scores and sampled noise as the optimization variables, eliminating the need for gradients through the denoising network. To mitigate the out-of-distribution problem, we introduce Reward-Optimized Score Guidance, which interpolates between the optimized scores and the pretrained model predictions to stabilize sampling, while providing a provable theoretical guarantee of reward improvement under constrained distributions. Empirically, DTO improves alignment performance across three diffusion models on two benchmarks, outperforming existing inference-time alignment methods while maintaining high efficiency. These results suggest that trajectory-level optimization is a promising direction for efficient and effective diffusion model alignment.
Figures & tables
Figure 1: Generation results of different alignment methods. The prompt is “Ralsei and Asriel from Deltarune eating pizza.”
Method
Optimization Formulation
DNO ( Tang et al., 2024 )
maxxT,{zt}R(M(xT,c,n,{zt}),c)
DATE ( Na et al., 2025 )
maxc~R(M(xT,c~,n,{zt}),c)
ReNeg ( Li et al., 2025 )
maxnR(M(xT,c,n,{zt}),c)
Table 1: Comparison of optimization formulations of existing methods.
Base Model
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
SD3
Base
29.71
22.33
29.46
83.94
1.105
BoN
32.07
22.75
29.72
84.96
1.269
DTO(Ours)
43.78
24.50
30.14
85.36
1.289
SDXL
Base
28.19
22.58
30.78
82.60
0.874
BoN
30.90
23.10
31.06
84.33
1.155
DNO
38.23
24.02
31.09
83.77
1.145
Table 2: Results on HPDv2 test set. HPSv2 is used as reward model.
Base Model
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
SD3
Base
29.12
22.50
28.96
75.61
1.065
BoN
31.19
22.89
29.19
77.03
1.260
DTO(Ours)
41.95
24.58
29.46
76.69
1.267
SDXL
Base
27.54
22.44
28.79
71.60
0.730
BoN
30.24
22.99
29.31
73.98
1.066
DNO
32.91
23.28
29.17
72.52
0.942
Table 3: Results on GenAIBench. HPSv2 is used as reward model.
Figure 2: Analysis on efficiency and scaling beahviour of DTO.
Figure 3: Ablation study and human evaluation results.
Figure 4: Generation results of different alignment methods using SDXL. The prompt is “Inside the camp, a fire crackles, casting shadows on the tent walls, with backpacks and maps spread out.”
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Base Model
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
SD3
Base
0.000
0.000
0.000
0.000
0.000
BoN
0.000
0.000
0.001
0.191
0.174
SDXL
Base
0.000
0.000
0.000
0.000
0.000
BoN
0.000
0.000
0.572
0.298
0.006
DNO
0.000
0.000
0.672
0.108
0.008
SD1.5
Base
0.000
0.000
0.000
0.000
0.000
Appendix
Table 4: p-value on HPDv2 test set evaluating whether DTO is significantly better regarding the given metric. A value less than 0.05 is labeled in bold to indicate significance (DTO is statistically better). HPSv2 is used as reward model.
Base Model
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
SD3
Base
29.71
22.33
29.46
83.94
1.105
BoN
30.05
22.49
32.04
87.35
1.216
DTO(Ours)
29.66
22.32
45.62
88.60
1.128
SDXL
Base
28.19
22.58
30.78
82.60
0.874
BoN
28.31
22.66
33.29
85.40
0.944
DNO
28.68
22.62
32.66
83.90
0.945
Appendix
Table 5: Results on HPDv2 test set. CLIPScore is used as reward model.
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
Base
29.71
22.33
29.46
83.94
1.105
BoN
31.10
22.99
29.96
85.30
1.257
DTO(Ours)
35.29
28.53
30.43
85.58
1.223
Appendix
Table 6: Results on HPDv2 test set. PickScore is used as the reward model. SD3 is used as the base model.
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
Base
30.22
22.87
29.33
83.09
1.097
BoN
32.34
23.25
29.73
84.31
1.293
DTO(Ours)
45.86
25.31
30.06
84.74
1.372
Appendix
Table 7: Results on HPDv2 test set using Stable-Diffusion-3.5-Large-Turbo with 4-step sampling. HPSv2 is used as the reward model.
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
Base
30.98
22.73
28.39
80.03
1.145
BoN
32.75
23.06
28.86
81.77
1.252
DTO(Ours)
41.02
24.16
28.99
83.09
1.279
Appendix
Table 8: Results on HPDv2 test set using FLUX.1-dev. HPSv2 is used as the reward model.
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
Base
28.19
22.58
30.78
82.60
0.874
DTO(ODE)
43.01
24.87
31.04
84.63
1.220
DTO(SDE)
43.03
24.89
31.09
84.84
1.204
Appendix
Table 9: Results on HPDv2 test set. HPSv2 is used as reward model.
Figure 5: Criteria of human annotation.
Method
Inference Latency (Seconds/Output) ( ↓ )
GPU Memory ( ↓ )
Base
1.34
5G
BoN
6.30*
20G*
DNO
427.38
18G
DATE
16.63
12G
DTO(Ours)
16.70
11G
Appendix
Table 10: Computation cost comparison of different methods. * denotes that the computation cost of BoN is calculated using batchified inference( N candidates are generated simultanously), resulting in less inference time yet higher GPU memory cost. SD1.5 is used as the base model.
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
BoN ( N=10 )
27.48
21.35
30.21
76.92
0.565
BoN ( N=20 )
28.14
21.45
30.23
78.01
0.643
DTO(Ours)
40.75
23.41
30.51
79.20
0.702
Appendix
Table 11: Wall-clock-matched comparison of BoN and DTO on the HPDv2 test set. HPSv2 is used as the reward model. SD1.5 is used as the base model. N=20 brings the runtime of BoN close to DTO.
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
Base
23.48
20.72
28.98
70.36
0.015
DTO(w/o TO)
34.06
22.24
29.91
74.20
0.255
DTO(w/o ROSG)
44.49
23.42
26.98
71.42
-0.038
DTO(Ours)
40.75
23.41
30.51
79.20
0.702
Appendix
Table 12: Ablation results on HPDv2 test set. HPSv2 is used as reward model.
Figure 6: DTO performance under different learning rate selection.
Method
Mean drift
Max drift
FID
Base
–
–
135.43
DTO (w/o ROSG)
0.43
1.30
235.56
DTO (Full, ours)
0.12
0.36
130.87
Appendix
Table 13: Score drift and FID on SD1.5. Drift is ∥ϵt′∥∥ϵt−ϵt′∥ , summarized by the mean and maximum over timesteps. FID is computed against MSCOCO-2014 training images. HPSv2 is used as the reward model.
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
DTO (Full Inter)
34.10
22.34
30.22
75.57
0.454
DTO (Const w=0.5 )
31.43
21.94
29.94
74.35
0.336
DTO (Ours)
40.75
23.41
30.51
79.20
0.702
Appendix
Table 14: Ablation of the ROSG weight schedule on the HPDv2 test set. HPSv2 is used as the reward model. SD1.5 is used as the base model.
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
DTO ( γt schedule)
37.59
22.88
30.38
76.76
0.529
DTO (Ours)
40.75
23.41
30.51
79.20
0.702
Appendix
Table 15: Comparison of a score-deviation threshold schedule and the default wt on the HPDv2 test set. HPSv2 is used as the reward model. SD1.5 is used as the base model.
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
DTO ( xT only)
37.49
22.82
30.33
76.90
0.508
DTO (First Half)
36.65
22.68
30.26
76.72
0.473
DTO (Second Half)
24.25
20.81
29.35
71.74
0.077
DTO (Full, Ours)
40.75
23.41
30.51
79.20
0.702
Appendix
Table 16: Ablation of the optimized trajectory variables on the HPDv2 test set. HPSv2 is used as the reward model. SD1.5 is used as the base model. ROSG is applied in every variant.
Base
BoN
ReNeg
DATE
DNO
DTO(Ours)
3.00
3.31
3.35
3.36
3.33
3.60
Appendix
Table 17: Results of human evaluation.
Base
BoN
DTO(Ours)
4.04
4.14
4.26
Appendix
Table 18: Human evaluation on 50 SD3 samples from the HPDv2 test set, using the same protocol as Table 17 .
Figure 7: Generation results of DTO under different setups based on SD1.5. The prompt is “A spoon dressed up with eyes and a smile.”
Figure 8: A failure case generated by DTO using SD1.5 as the base model. The prompt is “Keqing from Genshin Impact.”