Synthetic priors have so far served one purpose: pretraining tabular foundation models. We ask whether the same priors can improve conventional tabular models, which are trained from scratch on each dataset. We propose PriorTune, a method that meta-tunes hyperparameter configurations on synthetic tasks alone. Offline, for a given model, we fit a pool of configurations on datasets sampled from a published PFN prior and greedily select the subset with the highest winrate over the random baseline. Online, a new dataset fits only the selected configurations, so the gain comes at no extra cost. Empirically, we observe that scaling both the number of configurations and synthetic datasets has an equal scaling effect on the performance of the final subset of configurations, which we also explain theoretically. On the full BeyondArena benchmark, LightGBM with TabPrep features with our configurations ranks above TabPFN-3 and RealMLP, while being much more efficient at inference. Overall, our results establish existing PFN priors as a scalable source of meta-training data for hyperparameter selection for non-foundational tabular models.
Figures & tables
Figure 1: Full BeyondArena. Left: Elo under default, tuned, and tuned-and-ensembled deployment. Our families are coloured; FT (half-colored) and FTE (full-colored) are the subsets meta-tuned under T and T+E. Right: Elo against inference latency at batch size 1. GBDTs run on 16 CPU threads, and DNNs and PFNs on one GPU. For PFNs, we run with recompute (light grey), KV cache on CPU (medium grey), on GPU (dark grey). Protocol in Appendix A.1.4 .
Figure 2: Overview. Offline, we sample hyperparameter configurations from a pool C , fit them on N synthetic tasks, and greedily select a subset of K by win rate over random subsets under the given deployment rule. Online, we fit the K selected configurations once per task.
D1
D2
D3
D4 , held out
Configuration
val
test
val
test
val
test
val
test
A
105
103
0.28
0.36
0.60
0.45
0.38
0.47
B
96
96
0.30
0.31
0.53
0.53
0.41
0.41
C
95
97
0.34
0.34
0.55
0.55
0.40
0.40
D
60
60
0.29
0.40
0.52
0.60
0.39
0.50
Random reference
–
100
–
0.32
–
0.50
–
0.42
Table 1: A toy example under the tuned rule. Four configurations, three synthetic datasets D1 – D3 used for selection ( D1 scored by RMSE, the rest by log-loss) and one held-out dataset D4 (log-loss). Cells are validation / test losses (lower is better); the last row is the test loss of the deployed member of a random reference subset. We select a subset of K=2 from the empty set.
Figure 3: Elo on the full BeyondArena by dataset group. Top: tuned (T), our models with their FT subsets. Bottom: tuned and ensembled (T+E), our models with their FTE subsets. Under each panel, the bars show the Elo gain of each meta-tuned subset over the same model with its random subset.
Figure 4: Selection rules, synthetic out-of-fold, T deployment. Each of 200 repetitions selects on 3,000 random synthetic tasks and deploys on the other 1,000. Points are win rates against random- K baselines that no selection saw, ±1.96 standard errors over repetitions; the shaded row is our rule. Values in Table 10 .
Figure 5: TabArena, T deployment, win rate against a random subset of the same size. Synthetic versus TabRepo meta-training data at K=200 . Dots are (C,N) cells; lines are the best cell within each factor-of-two budget bin. For TabRepo (182 datasets), we evaluated 16,000 configurations, so its curve stops at B=16,000×182 .
Figure 6: TabArena, T deployment, K=200 . Random subsets of growing size K′ against the random- K baseline. Solid curve and band: mean and 95% CI over 200 random subsets of size K′ . Horizontal lines are the K configurations meta-tuned on all 4,000 configurations, with 4,000 synthetic tasks or the 182 TabRepo datasets, averaged against the same baselines.
Figure 7: TabArena, T deployment, win rate against a random subset of the same size, K=200 . Three priors, each with 2,000 configurations on 2,000 synthetic tasks, for LightGBM, LightGBM-TabPrep, and the MLP. Top: as C grows at N=2,000 ; bottom: as N grows at C=2,000 . Win rates are averaged over 2,000 repetitions.
Figure 8: TabArena, T deployment, win rate against a random subset of the same size over the (C,N) grid, K=200 . Win rates are averaged over 2,000 repetitions.
Figure 9: Selected hyperparameters against their sampling distribution: K=200 , T regime, selected from the full pool of 4000 configurations on 4000 synthetic datasets. Lines: density of the 200 selected configurations divided by the sampling density (grey, flat at 1); ticks: selected values; bars: shares (grey: sampling). All hyperparameters, also of LightGBM-TabPrep, are in Figures 19 – 21 .
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
TabArena
BeyondArena
Datasets
51
142
Binary / multiclass / regression
30 / 8 / 13
73 / 25 / 44
Splits: i.i.d. / grouped / temporal
51 / 0 / 0
103 / 18 / 21
Rows
748 / 6,497 / 150,000
155 / 5,000 / 1,296,316
Features
4 / 17 / 1,776
3 / 23 / 22,215
Appendix
Table 2: Benchmark statistics. Row and feature counts are per dataset (minimum / median / maximum).
PriorTune
PFN
Once per model
Grid: 400 (MLP)
Pretraining: 1,180 (TabICLv2)
Grid: 300 (LightGBM-TabPrep)
Pretraining: 1,440 (TabICL)
Selection: <0.01 (T), 10 (T+E, K=200 )
Pretraining: 460 (TabPFNv2)
Pretraining: not reported (TabPFN-3)
Per dataset
Fit K=25 : 0.1 (0.04), LightGBM proxy
Fit and predict: 0.48 (0.007), TabICLv2
Appendix
Table 3: Compute in A100 hours; CPU work converted at 61.5 vCPU-hours per A100 hour. Per-dataset costs are means over the 142 BeyondArena-Lite datasets, medians in parentheses.
Dataset
Task
Split
Rows
Features
credit-g
binary
i.i.d.
666
20
mice-protein
multiclass
grouped
720
77
coffee-rating
regression
temporal
2,137
12
predict-students
multiclass
i.i.d.
2,949
36
churn
binary
i.i.d.
3,333
19
emscad
binary
i.i.d.
11,640
17
Appendix
Table 4: Datasets of the latency measurement. Rows are those of the outer training split.
Model
Recompute
KV cache on GPU
KV cache on CPU
TabPFN-3
fit_mode=
fit_mode="fit_with_cache"
fit_mode="fit_with_cache"
"fit_preprocessors" a
keep_cache_on_device=True
keep_cache_on_device=False
TabICLv2
kv_cache=False a
kv_cache=True
kv_cache=True b
Kumo-Tabular
cache_context=False
cache_context=True c
cache_context=True a
Appendix
Table 5: Options of the in-context models for the three timed modes. TabPFN-3 and TabICLv2 take them in the constructor ( tabpfn 9.0.0, tabicl 2.2.0). Kumo-Tabular takes them in the TabArena wrapper.
Parameter
Distribution
Learning rate
LogUniform[5e-3, 1e-1]
Feature fraction
Uniform[0.4, 1.0]
Bagging fraction
Uniform[0.7, 1.0]
Bagging frequency
1
# leaves
LogUniformInt[2, 200]
Min. data in leaf
LogUniformInt[1, 64]
Appendix
Table 6: The hyperparameter tuning space for LightGBM.
Parameter
Distribution
TabPrep enabled
Bernoulli(0.25)
OpenFE max. features
{2000, 1000}
OpenFE generation seed
{42, 84, 168, 336, 672}
Cat. combination max. features
{100, 500}
Cat. combination generation seed
{42, 84, 168, 336, 672}
Random-subset feature combination # subsets
{50, 1, 100}
Appendix
Table 7: The TabPrep axes drawn for the one-in-four configurations with TabPrep enabled.
Parameter
Distribution
# blocks
UniformInt[1, 4]
Block width
384
Dropout rate
{0.0, Uniform[0.0, 0.5]}
Embedding dim
UniformInt[8, 32], step=4
Embedding init. scale
LogUniform[1e-2, 1e1]
Learning rate (AdamW)
LogUniform[1e-4, 5e-3]
Appendix
Table 8: The hyperparameter tuning space for the MLP with cosine embeddings.
Setting
TabICLv2
TabICLv1
TabPFNv1
Generator
graph_scm
mix_scm
prior_bag
Datasets
4,000
2,000
2,000
Task mix (reg/bin/multi)
2,000 / 1,400 / 600
0.4 / 0.3 / 0.3
0.4 / 0.3 / 0.3
Multiclass classes
U{3,…,10}
U{3,…,10}
U{3,…,10}
Rows
LogU [400,60,000]
LogU [1,024,32,768]
LogU [1,024,32,768]
Features
U[1,cap(n)] , cap 100→20
LogU [4,64]
LogU [4,64]
Appendix
Table 9: Sampling settings of the three synthetic priors. LogU: log-uniform. Splits are train/validation/test and stratified for classification.
Figure 10: Elo of all 58 methods on the full BeyondArena under default, tuned, and tuned-and-ensembled deployment. Our families are coloured as in Figure 1 .
Figure 11: Median per-dataset improvement over the tuned MLP with a random subset on the full BeyondArena, by dataset group. Each dataset’s improvement is averaged over its splits. Top: tuned (T), our models with their FT subsets. Bottom: tuned and ensembled (T+E), our models with their FTE subsets. Under each panel, the bars show the gain of each meta-tuned subset over the same model with its random subset, as the difference of the two medians.
Draws R
Resampling kept
TabRepo
Model
K
16
8
4
2
1
Val.
Ref.
Neither
Norm.
Count
LightGBM
25
0.636
0.634
0.632
0.627
0.620
0.630
0.630
0.621
0.617
0.621
64
0.621
0.618
0.613
0.607
0.599
0.614
0.609
0.600
0.588
0.602
200
0.599
0.596
0.592
0.585
0.576
0.593
0.586
0.574
0.561
0.575
LightGBM-TabPrep
25
0.636
0.636
0.633
0.629
0.622
0.632
0.631
0.623
0.618
0.622
64
0.621
0.619
0.615
0.610
0.602
0.616
0.609
0.601
0.597
0.607
Appendix
Table 10: Win rate against random- K of the selection rules of Section 5.1 , synthetic out-of-fold (1,000 held-out tasks), T deployment, against random- K baselines that no selection saw; mean over 200 repetitions ( ±1.96 standard errors of at most 0.0015). Draws R : our rule with R validation resamples paired with R references. Resampling kept: 16 validation resamples with one reference (Val.), 16 references on the original split (Ref.), or one of each (Neither). Bold: the best rule in the row.
Figure 12: Synthetic versus TabRepo meta-training data at equal budget, as in Figure 5 , for K=25 , 64 , and 200 .
Figure 13: Random subsets of growing size K′ against the random- K baseline, as in Figure 6 , for K=25 , 64 , and 200 .
Figure 14: Three priors with 2,000 configurations on 2,000 synthetic tasks, as in the top row of Figure 7 : win rate against the number of configurations C at N=2,000 , for K=25 , 64 , and 200 .
Figure 15: As Figure 14 , against the number of synthetic tasks N at C=2,000 (bottom row of Figure 7 ).
Figure 16: Win rate over the (C,N) grid, as in Figure 8 , for K=25 , 64 , and 200 .
Figure 17: Win rate against the number of synthetic datasets N , one curve per pool size C , with the fit a−b/N at C=4,000 , for K=25 , 64 , and 200 .
Figure 18: Win rate against the number of configurations C , one curve per number of synthetic datasets N , with the fit a+blog2C at N=4,000 , for K=25 , 64 , and 200 .
Figure 19: All hyperparameters of LightGBM: the 200 selected configurations against the sampling distribution, as in Figure 9 .
Figure 20: All hyperparameters of LightGBM-TabPrep, as in Figure 9 . The TabPrep settings count only the 37 selected configurations with TabPrep enabled.
Figure 21: All hyperparameters of the MLP, as in Figure 9 . The dropout rate is 0 with probability 0.5 and uniform otherwise; the label gives the share at 0 (sampled / selected), and the curve is the continuous part.
Hyperparameter optimization selects model and training settings to improve predictive performance, efficiency, or reliability under limited evaluation budgets.
Data-generating priors are a central component of tabular foundation models because they define the task distribution used during pretraining. However, priors are rarely evaluated as independent components, making it difficult to understand how much they affect downstream model behavior. This raises a methodological question: how can priors from different tabular foundation models be compared independently of the architectures and training protocols they were introduced with? To study this question, we implement a unified interface for publicly available priors from recent tabular foundation models and priors constructed from real datasets. We generate training tasks from each prior, train the same model architecture under a fixed training protocol, and evaluate the resulting models on shared downstream classification tasks. We compare priors through both generated-task statistics and downstream predictive performance. Our results show that different priors favor different downstream behaviors, with some achieving stronger absolute performance and others exhibiting more consistent relative rankings across datasets. We further find that data-level similarity only partially explains downstream behavior. Our code is available at https://github.com/automl/TFM-Playground/tree/prior-dev.
Zeynep Türkmen, Kürşat Kaya, Alexander Pfefferle +1
Department of Computer Science, University of Freiburg, Freiburg, Germany · Zuse School ELIZA, Hochschulstr. 10, 64289 Darmstadt · Prior Labs, Freiburg, Germany +1
What determines the quality of a tabular foundation model? Unlike language or vision, tabular foundation models acquire their inductive biases almost entirely from synthetic pretraining distributions, yet the design of these distributions remains poorly understood. Standard synthetic priors are too well-behaved: they omit the irregularities and failure modes that determine deployment robustness. We introduce O'Prior, a compositional realism prior built around four coupled components: a hierarchical SCM meta-generator spanning diverse functional families; a modular realism engine covering heterogeneous marginals, missingness, and target transforms; an explicit stress module injecting confounding and support-query mismatch; and a curriculum-governed, leakage-safe generation protocol. To isolate prior design as the scientific variable, we hold architecture, optimizer, and compute budget fixed and vary only the synthetic task distribution. O'Prior yields consistent and substantial improvements in downstream accuracy and robustness across real tabular benchmarks, with gains concentrated in regimes characterized by distributional irregularities. Ablations confirm that mechanism diversity, realism composition, and shift-aware stress each contribute independently, their effects are not interchangeable. These results establish synthetic prior construction as a first-order and largely overlooked determinant of tabular foundation model quality.
Tabular foundation models (TFMs) are commonly pretrained on large collections of procedurally generated synthetic tasks, yet it remains unclear how well these synthetic pretraining priors support the downstream tasks on which the models are evaluated. We study this question from a distribution-level attribution perspective. We recover or reconstruct the synthetic data generators of four TFMs and compare their generated tasks with datasets from two widely used tabular benchmarks. Each dataset is represented by a common set of structural descriptors capturing schema, feature distributions, dependence structure, response properties, and feature--response relationships. In this space, we measure how broadly and repeatedly each synthetic prior reaches benchmark tasks using structural coverage and normalized density, and examine whether stronger local support is associated with better predictive performance. We find substantial differences across synthetic pretraining priors: some generators provide consistently broader and denser support for benchmark tasks than others. Moreover, stronger synthetic-to-benchmark support is generally associated with better relative model performance. These results suggest that structural coverage provides a useful diagnostic for characterizing synthetic pretraining priors and relating their data-generating assumptions to downstream model behavior.