Thompson Sampling using Prior-fitted Diffusion Transformers
Organizations: Cornell University
Abstract
Thompson sampling is natural randomized strategy for optimizing unknown functions in a black-box manner. In continuous global optimization, Thompson sampling has largely been restricted to Gaussian process priors, as these are essentially the only class for which computing the posterior distribution has been tractable in practice. In this work, we introduce prior-fitted diffusion transformers, which are diffusion models that perform Thompson sampling in-context at inference time, after being pre-trained using the prior-fitted network paradigm. To apply these models, we develop a set of practical non-Gaussian priors for general black-box functions---including those that have characteristics that cannot be achieved under Gaussianity, such as priors over unimodal functions. We also quantify the tradeoff between model size and the number of pre-training iterations, and show that these models follow scaling laws akin to those observed in transformers defined over other modalities. Across a set of global optimization benchmarks, we show that prior-fitted diffusion transformers with non-Gaussian priors can achieve improved performance compared to traditional pipelines built on Gaussian processes.
Figures & tables
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Value |
|---|---|
| Context window size | 200 |
| Hidden dimension size | 256 |
| Self-attn depth size | 4 |
| Cross-attn depth size | 4 |
| Feedforward network width | 1024 |
| Timestep embeeding size | 256 |
| 8 | 0.271 | 0.136 | 0.0154 | 0.0856 |
|---|---|---|---|---|
| 12 | 0.299 | 0.703 | 0.00625 | 0.0195 |
| 16 | 0.336 | 0.313 | 0.0367 | 0.0690 |
| 25 | 0.289 | 0.748 | 0.00865 | 0.0142 |
| 8 | 0.587 | 0.543 | 0.0387 | 0.0510 | 0.00767 | 0.00767 | 5.29 | 5.29 | 0.184 | 0.184 | 1 | 1 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 12 | 0.605 | 0.286 | 0.00794 | 0.0713 | 0.0198 | 0.206 | 2.32 | 5.91 | 0.0585 | 0.511 | 0.0289 | 0.965 |
| 16 | 0.655 | 0.429 | 0.00638 | 0.141 | 0.0465 | 0.202 | 10.1 | 45.9 | 0.0862 | 0.271 | 0.0550 | 0.285 |
| 25 | 0.577 | 0.447 | 0.00851 | 0.0398 | 0.0183 | 0.0596 | 5.77 | 8.03 | 0.133 | 0.297 | 0.0153 | 0.0171 |
| 8 | 0.450 | 0.175 | 0.0312 | 0.0584 | 0.00984 | 0.125 | 11.1 | 11.1 | 0.0995 | 0.0995 | 1 | 1 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 12 | 0.477 | 0.090 | 0.00665 | 0.0748 | 0.0203 | 0.100 | 11.3 | 37.2 | 0.139 | 1.30 | 0.327 | 0.391 |
| 16 | 0.290 | 0.293 | 0.0129 | 0.116 | 0.00955 | 0.125 | 11.7 | 21.2 | 0.0580 | 0.233 | 0.0507 | 0.501 |
| 25 | 0.507 | 0.404 | 0.00628 | 0.0670 | 0.0248 | 0.172 | 10.1 | 33.4 | 0.0363 | 0.340 | 0.0215 | 0.419 |
| Method | Time / iter. (ms) | Total, 20 iter. (s) | Wall clock (s) | Relative |
|---|---|---|---|---|
| PFN (ours) | ||||
| TS | ||||
| EI | ||||
| DDOM | ||||
| Diff-BBO | ||||
| DiBO |
| Parameter description | Symbol |
|---|---|
| Input dimension size | |
| Output dimension size | |
| Context window size | |
| Hidden dimension size | |
| Self-attn depth size | |
| Cross-attn depth size |
| Parameters | FLOPs per context window | |
|---|---|---|
| Input Embedding and Fuse | ||
| Self-attn: QKV | ||
| Self-attn: Mask+Attention | — | |
| Self-attn: Projection | ||
| Feedforward after Self-attn | ||
| Time embedding |
| Parameters | ||
|---|---|---|
| 766 | 2 | 1 |
| 1844 | 4 | 1 |
| 4936 | 8 | 1 |
| 14864 | 16 | 1 |
| 80416 | 32 | 2 |
| 302144 | 64 | 2 |
| Budget (FLOPs) | range | |||||
|---|---|---|---|---|---|---|
| 766 – 49.7M | ||||||
| 1.84k – 261M | ||||||
| 1.84k – 261M | ||||||
| 1.84k – 261M | ||||||
| 4.93k – 261M |
| Fit | Form | Coefficients |
|---|---|---|
| Power law | , | |
| Capped | , , |
Thompson Sampling
Thompson sampling selects actions by drawing from posterior beliefs about rewards, balancing exploration and exploitation in sequential decisions.
See all 57 papers