Alignment of text-to-image diffusion models with reward functions is a key challenge in controllable generation. Existing approaches typically rely on optimizing model inputs or fine-tuning denoising networks, which requires computationally expensive backpropagation through large denoising models. In this work, we propose Direct Trajectory Optimization (DTO), a training-free framework that performs alignment by directly optimizing the sampling trajectory of diffusion models. Specifically, DTO treats intermediate denoising scores and sampled noise as the optimization variables, eliminating the need for gradients through the denoising network. To mitigate the out-of-distribution problem, we introduce Reward-Optimized Score Guidance, which interpolates between the optimized scores and the pretrained model predictions to stabilize sampling, while providing a provable theoretical guarantee of reward improvement under constrained distributions. Empirically, DTO improves alignment performance across three diffusion models on two benchmarks, outperforming existing inference-time alignment methods while maintaining high efficiency. These results suggest that trajectory-level optimization is a promising direction for efficient and effective diffusion model alignment.
Figures & tables
Figure 1: Generation results of different alignment methods. The prompt is “Ralsei and Asriel from Deltarune eating pizza.”
Method
Optimization Formulation
DNO ( Tang et al., 2024 )
maxxT,{zt}R(M(xT,c,n,{zt}),c)
DATE ( Na et al., 2025 )
maxc~R(M(xT,c~,n,{zt}),c)
ReNeg ( Li et al., 2025 )
maxnR(M(xT,c,n,{zt}),c)
Table 1: Comparison of optimization formulations of existing methods.
Base Model
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
SD3
Base
29.71
22.33
29.46
83.94
1.105
BoN
32.07
22.75
29.72
84.96
1.269
DTO(Ours)
43.78
24.50
30.14
85.36
1.289
SDXL
Base
28.19
22.58
30.78
82.60
0.874
BoN
30.90
23.10
31.06
84.33
1.155
DNO
38.23
24.02
31.09
83.77
1.145
Table 2: Results on HPDv2 test set. HPSv2 is used as reward model.
Base Model
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
SD3
Base
29.12
22.50
28.96
75.61
1.065
BoN
31.19
22.89
29.19
77.03
1.260
DTO(Ours)
41.95
24.58
29.46
76.69
1.267
SDXL
Base
27.54
22.44
28.79
71.60
0.730
BoN
30.24
22.99
29.31
73.98
1.066
DNO
32.91
23.28
29.17
72.52
0.942
Table 3: Results on GenAIBench. HPSv2 is used as reward model.
Figure 2: Analysis on efficiency and scaling beahviour of DTO.
Figure 3: Ablation study and human evaluation results.
Figure 4: Generation results of different alignment methods using SDXL. The prompt is “Inside the camp, a fire crackles, casting shadows on the tent walls, with backpacks and maps spread out.”
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Base Model
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
SD3
Base
0.000
0.000
0.000
0.000
0.000
BoN
0.000
0.000
0.001
0.191
0.174
SDXL
Base
0.000
0.000
0.000
0.000
0.000
BoN
0.000
0.000
0.572
0.298
0.006
DNO
0.000
0.000
0.672
0.108
0.008
SD1.5
Base
0.000
0.000
0.000
0.000
0.000
Appendix
Table 4: p-value on HPDv2 test set evaluating whether DTO is significantly better regarding the given metric. A value less than 0.05 is labeled in bold to indicate significance (DTO is statistically better). HPSv2 is used as reward model.
Base Model
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
SD3
Base
29.71
22.33
29.46
83.94
1.105
BoN
30.05
22.49
32.04
87.35
1.216
DTO(Ours)
29.66
22.32
45.62
88.60
1.128
SDXL
Base
28.19
22.58
30.78
82.60
0.874
BoN
28.31
22.66
33.29
85.40
0.944
DNO
28.68
22.62
32.66
83.90
0.945
Appendix
Table 5: Results on HPDv2 test set. CLIPScore is used as reward model.
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
Base
29.71
22.33
29.46
83.94
1.105
BoN
31.10
22.99
29.96
85.30
1.257
DTO(Ours)
35.29
28.53
30.43
85.58
1.223
Appendix
Table 6: Results on HPDv2 test set. PickScore is used as the reward model. SD3 is used as the base model.
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
Base
30.22
22.87
29.33
83.09
1.097
BoN
32.34
23.25
29.73
84.31
1.293
DTO(Ours)
45.86
25.31
30.06
84.74
1.372
Appendix
Table 7: Results on HPDv2 test set using Stable-Diffusion-3.5-Large-Turbo with 4-step sampling. HPSv2 is used as the reward model.
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
Base
30.98
22.73
28.39
80.03
1.145
BoN
32.75
23.06
28.86
81.77
1.252
DTO(Ours)
41.02
24.16
28.99
83.09
1.279
Appendix
Table 8: Results on HPDv2 test set using FLUX.1-dev. HPSv2 is used as the reward model.
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
Base
28.19
22.58
30.78
82.60
0.874
DTO(ODE)
43.01
24.87
31.04
84.63
1.220
DTO(SDE)
43.03
24.89
31.09
84.84
1.204
Appendix
Table 9: Results on HPDv2 test set. HPSv2 is used as reward model.
Figure 5: Criteria of human annotation.
Method
Inference Latency (Seconds/Output) ( ↓ )
GPU Memory ( ↓ )
Base
1.34
5G
BoN
6.30*
20G*
DNO
427.38
18G
DATE
16.63
12G
DTO(Ours)
16.70
11G
Appendix
Table 10: Computation cost comparison of different methods. * denotes that the computation cost of BoN is calculated using batchified inference( N candidates are generated simultanously), resulting in less inference time yet higher GPU memory cost. SD1.5 is used as the base model.
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
BoN ( N=10 )
27.48
21.35
30.21
76.92
0.565
BoN ( N=20 )
28.14
21.45
30.23
78.01
0.643
DTO(Ours)
40.75
23.41
30.51
79.20
0.702
Appendix
Table 11: Wall-clock-matched comparison of BoN and DTO on the HPDv2 test set. HPSv2 is used as the reward model. SD1.5 is used as the base model. N=20 brings the runtime of BoN close to DTO.
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
Base
23.48
20.72
28.98
70.36
0.015
DTO(w/o TO)
34.06
22.24
29.91
74.20
0.255
DTO(w/o ROSG)
44.49
23.42
26.98
71.42
-0.038
DTO(Ours)
40.75
23.41
30.51
79.20
0.702
Appendix
Table 12: Ablation results on HPDv2 test set. HPSv2 is used as reward model.
Figure 6: DTO performance under different learning rate selection.
Method
Mean drift
Max drift
FID
Base
–
–
135.43
DTO (w/o ROSG)
0.43
1.30
235.56
DTO (Full, ours)
0.12
0.36
130.87
Appendix
Table 13: Score drift and FID on SD1.5. Drift is ∥ϵt′∥∥ϵt−ϵt′∥ , summarized by the mean and maximum over timesteps. FID is computed against MSCOCO-2014 training images. HPSv2 is used as the reward model.
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
DTO (Full Inter)
34.10
22.34
30.22
75.57
0.454
DTO (Const w=0.5 )
31.43
21.94
29.94
74.35
0.336
DTO (Ours)
40.75
23.41
30.51
79.20
0.702
Appendix
Table 14: Ablation of the ROSG weight schedule on the HPDv2 test set. HPSv2 is used as the reward model. SD1.5 is used as the base model.
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
DTO ( γt schedule)
37.59
22.88
30.38
76.76
0.529
DTO (Ours)
40.75
23.41
30.51
79.20
0.702
Appendix
Table 15: Comparison of a score-deviation threshold schedule and the default wt on the HPDv2 test set. HPSv2 is used as the reward model. SD1.5 is used as the base model.
Method
HPSv2
PickScore
CLIPScore
VQAScore
ImageReward
DTO ( xT only)
37.49
22.82
30.33
76.90
0.508
DTO (First Half)
36.65
22.68
30.26
76.72
0.473
DTO (Second Half)
24.25
20.81
29.35
71.74
0.077
DTO (Full, Ours)
40.75
23.41
30.51
79.20
0.702
Appendix
Table 16: Ablation of the optimized trajectory variables on the HPDv2 test set. HPSv2 is used as the reward model. SD1.5 is used as the base model. ROSG is applied in every variant.
Base
BoN
ReNeg
DATE
DNO
DTO(Ours)
3.00
3.31
3.35
3.36
3.33
3.60
Appendix
Table 17: Results of human evaluation.
Base
BoN
DTO(Ours)
4.04
4.14
4.26
Appendix
Table 18: Human evaluation on 50 SD3 samples from the HPDv2 test set, using the same protocol as Table 17 .
Figure 7: Generation results of DTO under different setups based on SD1.5. The prompt is “A spoon dressed up with eyes and a smile.”
Figure 8: A failure case generated by DTO using SD1.5 as the base model. The prompt is “Keqing from Genshin Impact.”
Diffusion models generate highly realistic images but often struggle with precise text-image alignment. While recent post-training methods improve alignment using external rewards or human preference signals, their performance heavily depends on reward quality and does not directly address alignment within the diffusion process itself. Recent reward-free approaches such as SoftREPA demonstrate that optimizing soft text tokens via contrastive learning can effectively improve text-image representation alignment, outperforming standard parameter-efficient fine-tuning baselines. However, the contrastive formulation can excessively penalize negative pairs, which manifests as characteristic failure cases such as over-counting and repetition. To address this issue, we propose a lightweight, reward-free post-training method that refines soft tokens by integrating contrastive alignment guidance directly into the score-matching objective of diffusion models. By assigning alignment directions at the score level, our approach mitigates these limitations and yields more coherent and semantically faithful generations. Experiments show that our method matches SoftREPA while substantially improving its failure cases, achieving over 35% improvement in counting accuracy on the GenEval benchmark. Our method is seamlessly applicable to existing diffusion backbones (SD1.5, SDXL, and SD3), and is complementary to existing RL-based diffusion post-training methods. Project page: https://jaayeon.github.io/AGSM
While text-to-image diffusion models achieve impressive visual quality, they frequently struggle to maintain precise alignment with complex compositional prompts. An effective strategy is to improve the inference process of diffusion models, thereby better leveraging their pretrained priors to address misalignment. Existing training-free methods can be divided into two categories. The first category focuses on improving the randomly sampled initial noise, either performing costly search over noise pools or manipulating sampled noise without ensuring reliable semantic injection. The second category focuses on improving the denoising trajectory, lacking explicit mechanisms to timely diagnose and correct semantic errors. we propose AnchorSteer, a training-free framework that exerts fine-grained control over both initialization and the denoising trajectory. AnchorSteer consists of two synergistic components: Semantic Anchoring replaces uninformative Gaussian noise with text-aligned initializations via CLIP-based prior extraction and a novel Latent-Prior Score Distillation Sampling (LP-SDS) objective. Specifically, LP-SDS distills CLIP visual priors into the knowledge distribution of diffusion models, mitigating the domain gap between CLIP-based priors and diffusion-based priors. Reflective Steering transforms passive denoising with an active Think--Erase--Retouch loop that enables mid-generation self-correction. It leverages VLM-based diagnosis to detect semantic deviations and performs targeted latent refinement to suppress erroneous content and recover missing attributes. Extensive experiments on GenEval and T2I-CompBench++ demonstrate that AnchorSteer consistently outperforms existing baselines in text--image alignment while preserving high visual quality.
Test-time alignment (TTA) aims to adapt models to specific rewards during inference. However, existing methods tend to either under-optimise or over-optimise (reward hack) the target reward function. We propose Null-Text Test-Time Alignment (Null-TTA), which aligns diffusion models by optimising the unconditional embedding in classifier-free guidance, rather than manipulating latent or noise variables. Due to the structured semantic nature of the text embedding space, this ensures alignment occurs on a semantically coherent manifold and prevents reward hacking (exploiting non-semantic noise patterns to improve the reward). Since the unconditional embedding in classifier-free guidance serves as the anchor for the model's generative distribution, Null-TTA directly steers model's generative distribution towards the target reward rather than just adjusting the samples, even without updating model parameters. Thanks to these desirable properties, we show that Null-TTA achieves state-of-the-art target test-time alignment while maintaining strong cross-reward generalisation. This establishes semantic-space optimisation as an effective and principled novel paradigm for TTA.
Taehoon Kim, Henry Gouk, Timothy Hospedales
School of Informatics, University of Edinburgh · 2Samsung AI Center, Cambridge