Tabular foundation models (TFMs) enable in-context prediction on unseen tabular datasets without task-specific training. Mechanistic work has shown that increasing the depth of the ICL Transformer yields weak gains, leaving the main headroom for gains in the synthetic pretraining prior and column-row encoder upstream of the ICL backbone. We introduce TabAbax, a TFM built on TabICLv2 that focuses all modifications on these two upstream components. On the prior side, we further improve the quality and complexity of the synthetic pretraining prior. On the encoder side, a two-pass column-row stack strengthens cross-feature interactions before row compression, using learned Fourier scalar encoding from TabFM and Causilo. Evaluations on BeyondArena and TabArena show strong regression-centric gains: TabAbax reaches 1,238 Elo on the 142-dataset BeyondArena suite (10th of 41 evaluated methods), 1,676 Elo on TabArena's overall 51-dataset suite (11th of 99 methods, +115 over TabICLv2), and 2,024 Elo on TabArena's regression benchmark (7th of 99 methods, +336 over TabICLv2, +21 over TabFM). A prior-only ablation confirms the attribution: the improved prior alone yields +228 Elo for regression, demonstrating that prior quality is the dominant knob for improving regression performance.
Figures & tables
Figure 1: BeyondArena overall Elo across the full 142-dataset suite (98 classification, 44 regression; top-15 leaderboard). TabAbax (dark orange bar) reaches an Elo rating of 1,238, ranking 10th of the 41 evaluated methods, + 68 over the TabICLv2 baseline (1,170). Methods marked ‡ are partially imputed.
Figure 2: TabArena overall Elo across all 51 datasets (top-15 grouped leaderboard). TabAbax (dark orange bar) reaches an Elo rating of 1,676, ranking 11th of all 99 evaluated methods, + 115 over the TabICLv2 official default baseline (1,561) and + 50 over TabPFN-3 (1,626). The full 99-method table is in Table 9 .
Figure 3: Overall architecture of TabAbax. Left: the main pipeline. The input matrix X∈RN×M with targets yi∈Y is encoded in two passes (Steps 2–3) and processed by a 12-layer in-context transformer (Step 4). Right (dashed zoom-ins, top to bottom): Fourier scalar encoding with K learned frequencies; repeated feature grouping with circular shifts (j,j+1,j+3)modm and inducing-point set attention ( TFcol ); [CLS] -based row folding ( TFrow ); and query-adaptive sample selection (QASSMax) inside TFicl , where the query row s∗ attends only to the top q~⋅n context rows.
Figure 4: Schematic overview of the improved synthetic data generation prior. The dataset-level pipeline samples dataset properties, generates a graph, and constructs the tabular data; graphs whose targets are unlearnable are rejected and resampled, so only learnable problems enter the pretraining corpus.
Figure 5: The improved synthetic prior. Top: the dataset-level pipeline; datasets failing the signal check are resampled. Middle: the node-level computation of Step 3. Bottom: the three generation-time changes—richer converter function families ( f and θ drawn per node), strict one-hot width matching (declared, emitted, and one-hot widths agree exactly), and class frequencies from multiple distributions, chiefly π∼Dirichlet(α) .
Benchmark
# Datasets
Cls.
Reg.
< 10K
10K–100K
> 100K
BeyondArena
142
98
44
80
35
27
TabArena
51
38
13
29
20
2
Table 1: Statistics of the benchmark suites considered in our evaluation. Sizes are counted in number of instances per dataset; the size columns report the number of datasets in each range.
Figure 6: BeyondArena regression Elo across the 44 regression datasets (top-15 leaderboard). TabAbax-regression (dark orange bar) reaches an Elo rating of 1,287, ranking 8th of the 41 evaluated methods, + 127 over the TabICLv2 baseline (1,160), while the Kumo family, TabPFN-3.5, Causilo, and TabPFN-3.5-Fast retain their lead. Methods marked ‡ are partially imputed.
Figure 7: BeyondArena classification Elo across the 98 classification datasets (binary + multiclass; top-15 leaderboard). TabAbax-classification (dark orange bar) reaches an Elo rating of 1,219, ranking 11th of the 41 evaluated methods, + 43 over the TabICLv2 baseline (1,176), while the Kumo family, TabPFN-3.5, TabFM, EXAONE-Tabular, and TabPFN-3 retain their lead. Methods marked ‡ are partially imputed.
Rank
Method
Elo ↑
Elo +
Elo −
Avg. rank ↓
Improv. (%) ↓
Winrate ↑
5
Causilo
1,328.7
30.7
26.8
11.4
12.03
0.740
8
TabFM
1,274.4
39.4
36.9
13.7
10.82
0.682
10
TabAbax (ours)
1,237.7
29.0
29.1
15.4
14.47
0.640
13
TabPFN-2.6
1,175.9
26.4
24.0
18.4
16.56
0.565
16
TabICLv2 (baseline)
1,170.0
27.4
28.0
18.7
15.99
0.558
19
LightGBM (tuned + ens.)
1,150.0
27.3
23.2
19.7
17.36
0.533
Table 2: BeyondArena leaderboard, selected methods (142 datasets; the method set follows Table 3 for comparability; Rank is the Elo ordinal among all 41 evaluated methods). Elo + / Elo − are error-bar extents; Avg. rank is the mean per-dataset rank; Improv. is the oracle improvability (lower is better); Winrate is the per-dataset win rate. The full 41-method table is in Table 6 .
Rank
Method
Elo ↑
Elo +
Elo −
Avg. rank ↓
Improv. (%) ↓
Winrate ↑
6
TabFM
1,781.0
101.5
98.4
12.3
6.57
0.885
7
Causilo
1,780.3
81.1
49.3
12.3
9.07
0.884
11
TabAbax (ours)
1,676.0
74.0
58.3
17.7
11.14
0.830
14
TabPFN-2.6
1,577.2
53.9
36.9
24.2
12.98
0.763
16
TabICLv2 (baseline)
1,561.0
55.8
52.0
25.4
12.47
0.751
18
TabDPT-1.3
1,517.2
57.4
41.1
28.9
14.21
0.715
Table 3: Overall TabArena leaderboard, selected methods (51 datasets; Elo anchored at default Random Forest =1000 ; Rank is among all 99 evaluated methods; metric definitions as in Table 2 ). The full 99-method table is in Table 9 .
Figure 8: TabArena regression Elo across all 13 datasets of the benchmark’s full regression suite (top-15 grouped leaderboard). TabAbax (dark orange bar; arena entry TabAbax , regression checkpoint) reaches an Elo rating of 2,024, ranking 7th of all 99 evaluated methods, + 336 over the TabICLv2 official default baseline (1,688), + 217 over TabPFN-3 (1,807), and + 21 over TabFM (2,003). The full 99-method table is in Table 10 .
Figure 9: TabArena classification Elo across all 38 classification datasets (top-15 grouped leaderboard). TabAbax-classification (dark orange bar) reaches an Elo rating of 1,647, ranking 12th of all 99 evaluated methods, + 76 over the TabICLv2 official default baseline (1,571) and + 21 over TabPFN-3 (1,626); the full 99-method table is in Table 11 .
Figure 10: Pairwise win-rate matrix among the top methods on the overall TabArena suite (all 51 datasets). How to read it: locate TabAbax’s row and scan across the columns—each cell (row, column) is the fraction of datasets on which the row method beats the column method (darker green favors the row method, darker purple favors the column method); reading down a column gives the mirror image, i.e., how often that method loses to each row method. The TabAbax row shows wins against the TabICLv2 baseline on 78% of datasets, against TabPFN-3 on 70%, and against Xiaomi-TabLDM, TabPFN-v2.6, and the RealTabPFN-v2.5 variants on 62–73%, while the frontier group (Kumo family, TabPFN-3.5, TabFM, Causilo, Mitra-v2, EXAONE-Tabular) remains ahead (13–42%).
Classification (38 datasets)
Regression (13 datasets)
Method
Elo ↑
Rank ↓
Improv. ↓
Win ↑
Elo ↑
Rank ↓
Improv. ↓
Win ↑
TabAbax (ours)
1,646.7
20.2
13.83
0.804
2,023.7
10.1
3.30
0.907
TabAbax-prior_only
1,582.6
24.7
14.74
0.758
1,915.8
13.6
4.11
0.872
TabICLv2 (baseline)
1,570.6
25.7
14.56
0.748
1,688.1
24.4
6.35
0.760
Table 4: Classification vs. regression split under the official TabArena metrics. The prior-only ablation rows (the unmodified TabICLv2 architecture trained with the improved prior; Section 4.2 ) isolate the prior contribution to the full model’s + 76 classification and + 336 regression Elo gains over the TabICLv2 baseline. Rank is the mean per-dataset rank, Improv. the oracle improvability (%, lower is better), and Win the per-dataset win rate. Full per-suite tables: Tables 11 and 10 .
Figure 11: Isolating the prior changes on the TabArena regression suite (top-15 grouped leaderboard). TabAbax-regression-prior_only (dark orange bar)—the unmodified TabICLv2 architecture trained with the improved prior—reaches an Elo rating of 1,916, + 228 over the TabICLv2 official default baseline (1,688), while the full TabAbax regression checkpoint (2,024; Figure 8 ) adds + 108 from the architectural changes.
Figure 12: Isolating the prior changes on the TabArena classification suite (top-15 grouped leaderboard). TabAbax-classification-prior_only (dark orange bar)—the unmodified TabICLv2 architecture trained with the improved prior—reaches an Elo rating of 1,583, + 12 over the TabICLv2 official default baseline (1,571), while the full TabAbax classification checkpoint (1,647; Figure 9 ) adds + 64 from the architectural changes.
Model
Type
Total params
File size
TabAbax-classifier.ckpt
Classification ( max_classes =10)
28.85M
231.15 MB
TabAbax-regressor.ckpt
Regression ( quantiles =999)
29.84M
239.03 MB
Table 5: Parameter counts and checkpoint sizes of the released TabAbax checkpoints.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Elo
Elo +
Elo −
Avg. rank
Harm. rank
Score
Improv. (%)
Winrate
Kumo-Tabular (default)
1,429.4
42.5
43.4
7.8
2.7
0.747
7.77
0.830
Kumo-Tabular-Medium (default)
1,378.4
40.0
39.7
9.5
4.3
0.669
9.26
0.788
TabPFN-3.5 (default)
1,377.1
39.6
35.5
9.5
4.2
0.654
11.04
0.786
Causilo (default)
1,328.7
30.7
26.8
11.4
6.6
0.558
12.03
0.740
Kumo-Tabular-Small (default)
1,323.0
28.6
30.2
11.6
6.8
0.546
11.81
0.734
TabPFN-3.5-Fast (default)
1,320.6
34.8
25.3
11.7
6.1
0.560
12.14
0.732
Appendix
Table 6: Full BeyondArena leaderboard: all 142 datasets (98 classification and 44 regression), 41 evaluated methods, sorted by Elo.
Method
Elo
Elo +
Elo −
Avg. rank
Harm. rank
Score
Improv. (%)
Winrate
Kumo-Tabular (default)
1,497.5
111.0
80.3
6.2
2.3
0.802
3.52
0.870
Kumo-Tabular-Medium (default)
1,436.9
89.6
58.1
7.9
3.9
0.721
3.81
0.828
TabPFN-3.5 (default)
1,383.0
75.1
61.0
9.7
4.3
0.611
5.92
0.783
Kumo-Tabular-Small (default)
1,376.1
80.0
47.6
9.9
5.9
0.589
5.14
0.777
Causilo (default)
1,341.2
82.8
56.9
11.2
6.2
0.529
6.33
0.744
LimiX-2 (default)
1,293.5
109.1
91.4
13.2
2.7
0.583
6.08
0.695
Appendix
Table 7: Full BeyondArena regression leaderboard: all 44 regression datasets, 41 evaluated methods, sorted by Elo.
Method
Elo
Elo +
Elo −
Avg. rank
Harm. rank
Score
Improv. (%)
Winrate
Kumo-Tabular (default)
1,405.1
60.0
54.3
8.5
2.9
0.722
9.68
0.813
TabPFN-3.5 (default)
1,376.8
44.5
39.9
9.5
4.1
0.673
13.34
0.788
Kumo-Tabular-Medium (default)
1,357.0
54.6
40.2
10.2
4.5
0.645
11.70
0.770
TabPFN-3.5-Fast (default)
1,337.3
37.2
35.6
11.0
5.8
0.610
14.30
0.750
Causilo (default)
1,325.2
30.2
33.0
11.5
6.8
0.571
14.59
0.738
Kumo-Tabular-Small (default)
1,303.2
35.6
35.6
12.4
7.3
0.526
14.81
0.715
Appendix
Table 8: Full BeyondArena classification leaderboard: all 98 classification datasets, 41 evaluated methods, sorted by Elo.
Method
Elo
Elo +
Elo −
Avg. rank
Harm. rank
Score
Improv. (%)
Winrate
Kumo-Tabular (default)
1,957.2
160.7
99.5
6.2
2.4
0.885
4.39
0.946
LimiX-2 (default)
1,948.0
128.3
82.8
6.5
2.2
0.903
3.32
0.944
Kumo-Tabular-Medium (default)
1,898.1
105.0
77.4
7.9
4.2
0.827
5.88
0.929
TabPFN-3.5 (default)
1,859.1
102.0
65.2
9.2
4.2
0.784
7.56
0.916
Kumo-Tabular-Small (default)
1,797.4
79.5
56.1
11.6
7.5
0.721
8.90
0.892
TabFM (default)
1,781.0
101.5
98.4
12.3
3.7
0.738
6.57
0.885
Appendix
Table 9: Full TabArena overall leaderboard: all 51 datasets (38 classification and 13 regression), 99 evaluated methods, sorted by Elo.
Method
Elo
Elo +
Elo −
Avg. rank
Harm. rank
Score
Improv. (%)
Winrate
Kumo-Tabular (default)
2,394.7
264.2
183.1
2.9
1.7
0.969
0.70
0.980
LimiX-2 (default)
2,269.0
329.2
173.5
4.6
1.9
0.931
0.95
0.963
Kumo-Tabular-Medium (default)
2,249.8
208.3
136.4
4.9
3.6
0.896
1.72
0.960
TabPFN-3.5 (default)
2,144.3
220.0
117.0
7.0
4.2
0.826
2.75
0.939
Kumo-Tabular-Small (default)
2,117.9
150.7
84.3
7.6
5.5
0.815
2.71
0.933
Causilo (default)
2,038.6
176.3
109.7
9.7
6.1
0.768
3.27
0.911
Appendix
Table 10: Full TabArena regression leaderboard: all 13 regression datasets, 99 evaluated methods, sorted by Elo.
Method
Elo
Elo +
Elo −
Avg. rank
Harm. rank
Score
Improv. (%)
Winrate
LimiX-2 (default)
1,929.0
135.9
77.4
7.1
2.3
0.893
4.13
0.937
Kumo-Tabular (default)
1,920.4
158.7
110.1
7.4
2.7
0.856
5.65
0.935
Kumo-Tabular-Medium (default)
1,873.0
133.6
94.2
8.9
4.5
0.803
7.31
0.919
TabPFN-3.5 (default)
1,845.4
106.7
69.6
10.0
4.2
0.770
9.20
0.909
TabPFN-3.5-Fast (default)
1,779.8
95.1
69.4
12.8
6.9
0.701
10.45
0.880
TabFM (default)
1,778.4
125.3
117.3
12.8
3.1
0.740
7.42
0.879
Appendix
Table 11: Full TabArena classification leaderboard: all 38 classification datasets, 99 evaluated methods, sorted by Elo.
Tabular foundation models, such as TabPFNv2 and TabICL, have recently dethroned gradient-boosted trees at the top of predictive benchmarks, demonstrating the value of in-context learning for tabular data. We introduce TabICLv2, a new state-of-the-art foundation model for regression and classification built on three pillars: (1) a novel synthetic data generation engine designed for high pretraining diversity; (2) various architectural innovations, including a new scalable softmax in attention improving generalization to larger datasets without prohibitive long-sequence pretraining; and (3) optimized pretraining protocols, notably replacing AdamW with the Muon optimizer. On the TabArena and TALENT benchmarks, TabICLv2 without any tuning surpasses the performance of the current state of the art, RealTabPFN-2.5 (hyperparameter-tuned, ensembled, and fine-tuned on real data). With only moderate pretraining compute, TabICLv2 generalizes effectively to million-scale datasets under 50 GB GPU memory while being markedly faster than RealTabPFN-2.5. We provide extensive ablation studies to quantify these contributions and foster open research by releasing code for inference, pretraining, and synthetic data generation at https://github.com/soda-inria/tabicl.
Jingang Qu, David Holzmüller, Gaël Varoquaux +1
SODA Team, INRIA Saclay, Palaiseau, France · Probabl, France
EXAONE Tabular is a compact tabular foundation model family for classification and regression via in-context learning, producing predictions without dataset-specific gradient updates. Pretrained exclusively on a synthetic structural-causal-model (SCM) prior, its central contribution is an architecture-centered redesign of tabular in-context learning. Rather than compressing features into a fixed row embedding before a separate row-level learner, EXAONE Tabular interleaves feature-axis attention within each item with support-conditioned item-axis attention within each feature at every Transformer layer, mediated by item-summary and feature-summary tokens. Across four public benchmarks, EXAONE Tabular combines strong predictive performance with high efficiency. On TabArena, its 20.81M-parameter classification model ranks first overall, surpassing tuned ensembles and 4-hour AutoML pipelines, while regression reaches the performance regime of the 1.64B-parameter TabFM at roughly 1/11 the inference cost. On BCCO and TALENT, EXAONE Tabular ranks second in classification and first in regression. On ScoringBench, it achieves the best mean rank for both point-estimation and predictive-distribution quality, leading the R2, RMSE, and CRPS evaluations. Together, these results establish EXAONE Tabular as a state-of-the-art compact tabular foundation model family, combining strong predictive performance across classification, point regression, and probabilistic regression with an efficient model design.
Tabular foundation models (TFMs) are becoming a powerful alternative to conventional tabular learning, adapting to new datasets in context without task-specific training. Current architectures, however, face a trade-off: models that preserve cell-level representations are expensive because they repeatedly attend across rows and columns, while faster models collapse each row into a fixed embedding and lose this fine-grained state. We introduce Factorized Tabular Foundation Models (FTFM), which preserve cell representations while factorizing their computation through one contextual latent per row and one per column. Cells no longer attend to one another directly; instead, each cell is updated through the interaction of its row and column latents, reducing repeated attention while retaining feature-specific information. FTFM achieves competitive predictive performance in tabular classification on major benchmarks such as TabArena and BeyondArena.