Beyond Foundation Models: PFN Priors Improve Tabular Hyperparameter Optimization
Organizations: HSE University · Yandex
Abstract
Synthetic priors have so far served one purpose: pretraining tabular foundation models. We ask whether the same priors can improve conventional tabular models, which are trained from scratch on each dataset. We propose PriorTune, a method that meta-tunes hyperparameter configurations on synthetic tasks alone. Offline, for a given model, we fit a pool of configurations on datasets sampled from a published PFN prior and greedily select the subset with the highest winrate over the random baseline. Online, a new dataset fits only the selected configurations, so the gain comes at no extra cost. Empirically, we observe that scaling both the number of configurations and synthetic datasets has an equal scaling effect on the performance of the final subset of configurations, which we also explain theoretically. On the full BeyondArena benchmark, LightGBM with TabPrep features with our configurations ranks above TabPFN-3 and RealMLP, while being much more efficient at inference. Overall, our results establish existing PFN priors as a scalable source of meta-training data for hyperparameter selection for non-foundational tabular models.
Figures & tables
| , held out | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Configuration | val | test | val | test | val | test | val | test | |
| A | 105 | 103 | 0.28 | 0.36 | 0.60 | 0.45 | 0.38 | 0.47 | |
| B | 96 | 96 | 0.30 | 0.31 | 0.53 | 0.53 | 0.41 | 0.41 | |
| C | 95 | 97 | 0.34 | 0.34 | 0.55 | 0.55 | 0.40 | 0.40 | |
| D | 60 | 60 | 0.29 | 0.40 | 0.52 | 0.60 | 0.39 | 0.50 | |
| Random reference | – | 100 | – | 0.32 | – | 0.50 | – | 0.42 | |
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| TabArena | BeyondArena | |
|---|---|---|
| Datasets | 51 | 142 |
| Binary / multiclass / regression | 30 / 8 / 13 | 73 / 25 / 44 |
| Splits: i.i.d. / grouped / temporal | 51 / 0 / 0 | 103 / 18 / 21 |
| Rows | 748 / 6,497 / 150,000 | 155 / 5,000 / 1,296,316 |
| Features | 4 / 17 / 1,776 | 3 / 23 / 22,215 |
| PriorTune | PFN | |
|---|---|---|
| Once per model | Grid: 400 (MLP) | Pretraining: 1,180 (TabICLv2) |
| Grid: 300 (LightGBM-TabPrep) | Pretraining: 1,440 (TabICL) | |
| Selection: (T), 10 (T+E, ) | Pretraining: 460 (TabPFNv2) | |
| Pretraining: not reported (TabPFN-3) | ||
| Per dataset | Fit : 0.1 (0.04), LightGBM proxy | Fit and predict: 0.48 (0.007), TabICLv2 |
| Dataset | Task | Split | Rows | Features |
|---|---|---|---|---|
| credit-g | binary | i.i.d. | 666 | 20 |
| mice-protein | multiclass | grouped | 720 | 77 |
| coffee-rating | regression | temporal | 2,137 | 12 |
| predict-students | multiclass | i.i.d. | 2,949 | 36 |
| churn | binary | i.i.d. | 3,333 | 19 |
| emscad | binary | i.i.d. | 11,640 | 17 |
| Model | Recompute | KV cache on GPU | KV cache on CPU |
|---|---|---|---|
| TabPFN-3 | fit_mode= | fit_mode="fit_with_cache" | fit_mode="fit_with_cache" |
| "fit_preprocessors" a | keep_cache_on_device=True | keep_cache_on_device=False | |
| TabICLv2 | kv_cache=False a | kv_cache=True | kv_cache=True b |
| Kumo-Tabular | cache_context=False | cache_context=True c | cache_context=True a |
| Parameter | Distribution |
| Learning rate | LogUniform[5e-3, 1e-1] |
| Feature fraction | Uniform[0.4, 1.0] |
| Bagging fraction | Uniform[0.7, 1.0] |
| Bagging frequency | 1 |
| # leaves | LogUniformInt[2, 200] |
| Min. data in leaf | LogUniformInt[1, 64] |
| Parameter | Distribution |
|---|---|
| TabPrep enabled | Bernoulli(0.25) |
| OpenFE max. features | {2000, 1000} |
| OpenFE generation seed | {42, 84, 168, 336, 672} |
| Cat. combination max. features | {100, 500} |
| Cat. combination generation seed | {42, 84, 168, 336, 672} |
| Random-subset feature combination # subsets | {50, 1, 100} |
| Parameter | Distribution |
|---|---|
| # blocks | UniformInt[1, 4] |
| Block width | 384 |
| Dropout rate | {0.0, Uniform[0.0, 0.5]} |
| Embedding dim | UniformInt[8, 32], step=4 |
| Embedding init. scale | LogUniform[1e-2, 1e1] |
| Learning rate (AdamW) | LogUniform[1e-4, 5e-3] |
| Setting | TabICLv2 | TabICLv1 | TabPFNv1 |
|---|---|---|---|
| Generator | graph_scm | mix_scm | prior_bag |
| Datasets | 4,000 | 2,000 | 2,000 |
| Task mix (reg/bin/multi) | 2,000 / 1,400 / 600 | 0.4 / 0.3 / 0.3 | 0.4 / 0.3 / 0.3 |
| Multiclass classes | |||
| Rows | LogU | LogU | LogU |
| Features | , cap | LogU | LogU |
| Draws | Resampling kept | TabRepo | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | 16 | 8 | 4 | 2 | 1 | Val. | Ref. | Neither | Norm. | Count | |
| LightGBM | 25 | 0.636 | 0.634 | 0.632 | 0.627 | 0.620 | 0.630 | 0.630 | 0.621 | 0.617 | 0.621 |
| 64 | 0.621 | 0.618 | 0.613 | 0.607 | 0.599 | 0.614 | 0.609 | 0.600 | 0.588 | 0.602 | |
| 200 | 0.599 | 0.596 | 0.592 | 0.585 | 0.576 | 0.593 | 0.586 | 0.574 | 0.561 | 0.575 | |
| LightGBM-TabPrep | 25 | 0.636 | 0.636 | 0.633 | 0.629 | 0.622 | 0.632 | 0.631 | 0.623 | 0.618 | 0.622 |
| 64 | 0.621 | 0.619 | 0.615 | 0.610 | 0.602 | 0.616 | 0.609 | 0.601 | 0.597 | 0.607 | |
Hyperparameter Optimization
Hyperparameter optimization selects model and training settings to improve predictive performance, efficiency, or reliability under limited evaluation budgets.
See all 178 papers