Test-time scaling of diffusions with flow maps
Organizations: NVIDIA · University of Toronto · Vector Institute · Harvard University ·
Abstract
A common recipe to improve diffusion models at test-time so that samples score highly against a user-specified reward is to introduce the gradient of the reward into the dynamics of the diffusion itself. This procedure is often ill posed, as user-specified rewards are usually only well defined on the data distribution at the end of generation. While common workarounds to this problem are to use a denoiser to estimate what a sample would have been at the end of generation, we propose a simple solution to this problem by working directly with a flow map. By exploiting a relationship between the flow map and velocity field governing the instantaneous transport, we construct an algorithm, Flow Map Trajectory Tilting (FMTT), which provably performs better ascent on the reward than standard test-time methods involving the gradient of the reward. The approach can be used to either perform exact sampling via importance weighting or principled search that identifies local maximizers of the reward-tilted distribution. We demonstrate the efficacy of our approach against other look-ahead techniques, and show how the flow map enables engagement with complicated reward functions that make possible new forms of image editing, e.g. by interfacing with vision language models.
Figures & tables
| Method | Mean | Style | World Knowledge | Attribute | Action | Relationship | Compound | Grammar | Logical Reasoning | Entity Layout | Text Generation | NFE |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| FLUX.1 [dev] | 61.78 | 85.20 | 86.23 | 64.53 | 60.46 | 63.07 | 41.24 | 60.96 | 24.08 | 67.16 | 30.17 | 180 |
| Flow Map | 58.16 | 79.70 | 80.22 | 59.51 | 57.22 | 62.44 | 42.01 | 57.62 | 25.92 | 70.90 | 3.74 | 16 |
| FLUX.1 [dev] + Best-of- | 73.75 | 91.10 | 92.88 | 76.39 | 71.58 | 75.51 | 61.86 | 69.52 | 45.18 | 80.97 | 44.83 | 11520 |
| FMTT - 1-step denoiser look-ahead | 72.90 | 91.00 | 90.51 | 73.93 | 70.72 | 78.30 | 60.31 | 69.65 | 42.89 | 81.53 | 39.94 | 11520 |
| FMTT - 1-step flow map look-ahead | 74.86 | 91.90 | 92.72 | 77.78 | 71.86 | 77.41 | 62.50 | 72.59 | 46.33 | 81.72 | 46.55 | 3840 |
| FMTT - 2-step flow map look-ahead | 75.55 | 91.90 | 92.88 | 78.21 | 72.34 | 79.19 | 65.08 | 72.73 | 46.79 | 81.53 | 47.70 | 6720 |
| Method | Mean | Single Obj. | Two Obj. | Counting | Colors | Position | Attr. Binding | NFE |
|---|---|---|---|---|---|---|---|---|
| Diffusions + Flow Maps | ||||||||
| FLUX.1 [dev] | 0.65 | 0.99 | 0.78 | 0.70 | 0.78 | 0.18 | 0.45 | 180 |
| Flow Map | 0.62 | 0.99 | 0.72 | 0.63 | 0.80 | 0.19 | 0.39 | 16 |
| Gradient-Free Search | ||||||||
| FLUX.1 [dev] + Best-of- | 0.75 | 0.99 | 0.94 | 0.83 | 0.86 | 0.26 | 0.57 | 1440 |
| Flow Map + Best-of- | 0.73 | 1.00 | 0.88 | 0.82 | 0.85 | 0.25 | 0.59 | 128 |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Steps | Method | MMD | SW2 | Mean Reward | of Resamples |
|---|---|---|---|---|---|
| 50 | No lookahead | 0.996 | 0.578 | 0.911 | 46 |
| Denoiser | 3.973 | 0.988 | 0.917 | 47 | |
| Flow map | 0.043 | 0.060 | 0.621 | 5 | |
| 100 | No lookahead | 0.460 | 0.443 | 0.958 | 64 |
| Denoiser | 4.131 | 0.979 | 0.949 | 81 | |
| Flow map | 0.065 | 0.083 | 0.925 | 10 |
| Method | Mean | Style | World Knowledge | Attribute | Action | Relationship | Compound | Grammar | Logical Reasoning | Entity Layout | Text Generation | NFE |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Diffusions | ||||||||||||
| FLUX.1 [dev] | 61.78 3.58 | 85.20 2.30 | 86.23 2.62 | 64.53 2.82 | 60.46 3.80 | 63.07 5.46 | 41.24 5.33 | 60.96 2.95 | 24.08 3.63 | 67.16 4.32 | 30.17 5.77 | 180 |
| Flow Map | 58.16 0.66 | 79.70 2.42 | 80.22 1.04 | 59.51 1.26 | 57.22 1.44 | 62.44 0.95 | 42.01 2.20 | 57.62 1.33 | 25.92 2.37 | 70.90 1.75 | 3.74 0.50 | 16 |
| Gradient-Free Search | ||||||||||||
| FLUX.1 [dev] + Best-of- | ||||||||||||
| - | 65.40 2.56 | 87.40 2.09 | 89.72 1.75 | 68.27 2.50 | 62.26 3.64 | 68.02 4.38 | 47.29 1.80 | 63.37 2.16 | 29.36 3.24 | 71.08 71.08 | 35.06 5.66 | 360 |
Diffusion Model Guidance
Diffusion model guidance steers denoising toward desired conditions, rewards, constraints, or away from unwanted concepts.
See all 392 papers