Tabular foundation models based on in-context learning have recently emerged as strong alternatives to task-specific model fitting. However, the current performance frontier remains dominated by attention-heavy architectures, where attention is used throughout the modeling pipeline. This raises a natural question: is attention necessary at every stage of tabular in-context learning? We introduce SOMTab, a Set-Order Mamba architecture for efficient tabular in-context learning. SOMTab separates representation construction from query-conditioned retrieval. For row and column representations, it maps unordered table tokens into stable latent slots and applies Mamba-based state-space mixing to construct compact representations. For final prediction, it retains attention-based in-context learning to preserve query-conditioned retrieval from labeled context examples. We further introduce DCH-TailMix, a synthetic prior that combines degree-corrected graph heterogeneity with mixed heavy-tailed regimes to diversify synthetic dependency structures. Across tabular benchmarks, SOMTab approaches the performance of strong Transformer-based tabular foundation models while achieving faster inference and lower GPU memory usage, yielding a favorable efficiency--accuracy trade-off.
Figures & tables
Figure 1: Predictive quality–efficiency trade-off on all TALENT classification datasets. The horizontal axis reports the median fit-plus-predict time per 1K labeled context samples, and the vertical axis reports the mean min–max normalized log-loss across datasets. Lower values are better on both axes, and the runtime axis is shown on a logarithmic scale. All methods are rerun on the same H200 server under a common evaluation pipeline; baselines use their official default configurations. The dashed line denotes the empirical Pareto frontier.
Figure 2: Architecture overview of SOMTab. The top illustrates the end-to-end pipeline, where variable- and sample-level Set-to-Order Mamba modules construct compact row representations, followed by an attention-based ICL reader for query prediction. The bottom details the corresponding representation modules and context-only attention mechanism.
Method
Norm. LL ↓
Accuracy ↑
Macro F1 ↑
Time/1K ↓
SOMTab
0.054
0.849
0.785
0.019
TabICLv2
0.029
0.854
0.790
0.463
TabICLv2(1 est.)
0.046
0.853
0.789
0.151
TabICL
0.105
0.846
0.776
0.453
TabPFNv2
0.100
0.849
0.779
0.675
RealMLP
0.498
0.830
0.757
4.379
Table 1: Aggregate performance over all TALENT classification datasets. Normalized log-loss, accuracy, and Macro F1 are averaged across datasets, while runtime is the median fit-plus-predict time per 1K labeled context samples. TabICLv2 (default) uses the standard ensemble setting with eight estimators, whereas TabICLv2 (1 estimator) uses a single estimator.
Figure 3: Runtime and GPU-memory scaling with context length. We fix ntest=1024 , m=32 features, and C=10 classes, and vary the number of context rows from 512 to 28,672. All foundation models use one estimator to isolate architectural scaling from prediction ensembling. Results are measured after one warmup run and report the median of three repeats on identical synthetic data.
Figure 4: Controlled early-training ablation. All variants are trained from scratch for 20K steps under the same configuration. Checkpoints at 5K, 10K, 15K and 20K steps are evaluated on the 15 TALENT datasets where SOMTab has the largest accuracy gaps relative to TabICLv2 in the main benchmark. The vertical axis reports the equally weighted mean accuracy across datasets.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Shape
Description
E(0)
n×m×d
initial cell tokens
Zjvar
Kv×d
latent memory for feature j
Evar
n×m×d
context-conditioned cell tokens
Zisample
4×d
ordered slots for row i
hi
4d
row representation
H
n×4d
representations of all rows
Appendix
Table 2: Principal tensors in the SOMTab forward pass. Batch dimensions are omitted. The main model uses d=128 , Kv=64 , and Kr=Ks=4 .
Graph family
Probability
Base sparsity τ
Structural terms
Parent cap
controlled heavy-tail
0.40
U(−4.0,−0.5)
degree corrections; λ=0
at most 8
modular heavy-tail
0.25
U(−4.0,−0.5)
module-pair and low-rank biases
at most 8
low-rank correlated connectivity
0.15
U(−4.0,−0.5)
correlated low-rank edge logits
at most 8
dense weak-dependency
0.20
U(−2.4,−0.8)
denser modular and low-rank effects
at most 12
Appendix
Table 3: Graph-family mixture in DCH-TailMix. For dense weak-dependency graphs, the parent cap for node j is at least min(8,j−1) under one-based topological indexing. When the initial Bernoulli sample for the final node is too sparse, additional high-probability candidate edges are inserted before applying this cap.
Figure 5: Representative realizations from the four DCH graph families. Nodes are arranged in topological order from left to right; colors identify sampled modules when modular structure is active. Each panel shows one sampled graph.
Figure 6: Illustrative tail behavior of the three mixture components before clipping. Curves show representative draws of the regime parameters.
Family
Construction
Sampled ranges / constraints
linear
random matrix projection
Gaussian, low-rank, SVD-shaped, kernel, or activated matrix; input dimension ≤128
2 – 255 centers; ℓp distance with p∈[0.5,4] log-uniform
GP-like
finite random Fourier-feature map
256 features; axis-aligned probability 0.5 ; spectral tail a∈[2,20] log-uniform
quadratic
symmetric quadratic form with a bias coordinate
input dimension ≤20
Appendix
Table 4: Random function library for latent propagation.
Figure 7: Representative one-dimensional slices of the random function families. Each panel shows the response of one function family along a fixed low-dimensional parent trajectory, illustrating smooth, thresholded, piecewise, and interaction-based mechanisms.
Setting
Stage 1
Stage 2
Initialization
random
Stage-1 checkpoint
Training steps
300,000
100,000
Sequence length
400 – 8192
400 – 8192
Length sampling
log-uniform
log-uniform
Context fraction
U(0.1,0.9)
U(0.5,0.9)
Initial learning rate
8×10−4
5×10−5
Appendix
Table 5: Two-stage pretraining curriculum. Sequence length is the total number of context and query rows.
Variant
Representation
Prior / reader change
SOMTab
Set-Order Mamba
DCH-TailMix + attention reader
Cauchy
Set-Order Mamba
random Cauchy graph prior
SCM–Tree
Set-Order Mamba
70% MLP-SCM + 30% tree-SCM
Tab-Full
Set-Order Mamba
DCH-TailMix + Mamba reader
Appendix
Table 6: Variants in the controlled ablation. All unspecified architecture and optimization settings are shared.
Figure 8: Early-training accuracy under a matched 20K-step budget. Values are equally weighted means over the fixed 15-dataset subset.
Dataset
SOMTab
TabICLv2
TabICL
TabPFNv2
RealMLP
CatBoost
XGBoost
ExtraTrees
RandomForest
mice_protein_expression
1.000
1.000
1.000
1.000
1.000
0.972
0.954
1.000
0.986
shuttle
1.000
0.999
0.999
0.999
1.000
1.000
1.000
0.999
1.000
JapaneseVowels
0.997
0.999
0.999
0.999
0.990
0.960
0.979
0.985
0.974
gas-drift
0.997
0.997
0.997
0.996
0.996
0.991
0.995
0.996
0.994
pendigits
0.996
0.997
0.997
0.996
0.998
0.986
0.989
0.991
0.991
thyroid
0.994
0.994
0.993
0.997
0.994
0.995
0.997
0.983
0.997
Appendix
Table 7: Per-dataset accuracy results on selected TALENT datasets. We report accuracy on 20 TALENT classification datasets. The final 15 rows form the fixed subset used in the controlled early-training ablation.
Figure 9: Performance–efficiency comparison on TabArena. The x-axis shows inference time per 1000 samples on a logarithmic scale, and the y-axis reports the average performance gap to the best method on each dataset (lower is better). SOMTab-H200 and SOMTab-A100 denote measurements obtained on H200 and A100 GPUs, respectively, while other results are taken from the official TabArena evaluation.
Figure 10: Pairwise win-rate comparison on TabArena. Each entry denotes the fraction of datasets where the row model achieves higher predictive performance than the column model.
Tabular foundation models, driven by in-context learning, have rapidly grown in quality and popularity. However, recent approaches with either cell-based architectures or retrieval have sacrificed efficiency for raw performance, restricting their utility in situations where compute is limited or inference speed is crucial. We adopt an alternate approach, sticking with row-based attention while incorporating long context pre-training to eliminate the need for retrieval. By combining this with architectural improvements and SSL pre-training on a newly-sourced, larger corpus of real data results, we present TabDPT-Turbo, a model that provides comparable default performance to TabDPT v1.1 on TabArena-Lite, CC18, and CTR23, at orders of magnitude faster. In our experiments, TabDPT-Turbo is the fastest model overall among leading foundation models. We have released the new model as TabDPT v1.2 at https://github.com/layer6ai-labs/TabDPT-inference.
Tabular foundation models, exemplified by TabPFN, perform prediction via in-context learning, inferring test labels directly from labeled training examples. They have demonstrated competitive performance, particularly on small-to-medium datasets. However, recent tabular foundation models often improve accuracy with increasingly complex architectures, incurring higher inference cost and limiting practical deployment. In this work, we revisit the original TabPFN design and show that a lightweight row-wise attention-only backbone can remain highly competitive with two simple enhancements: a gated attention stabilization mechanism and a small set of learnable register tokens that provide global context and improve pretraining quality. The resulting model, TabSwift, supports both classification and regression, and is competitive with stronger tabular foundation models (e.g., TabPFN v2 and TabICL) while being more efficient at inference. For latency-sensitive serving, we further introduce an adaptive layer-wise early-exit mechanism that dynamically adjusts inference depth per sample. Overall, TabSwift enables efficient and anytime tabular in-context learning for practical deployments.
Si-Yang Liu, Han-Jia Ye
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China.
Transformer-based tabular foundation models such as TabPFN achieve strong predictive performance but incur quadratic computational cost with context length. On the other hand, subquadratic SSM-based alternatives such as Hydra trade away accuracy for efficiency. To balance both, we introduce Tydra, a hybrid Transformer-State Space Model (SSM) architecture for tabular in-context learning that interleaves attention and SSM layers. Across 30 OpenML datasets, Tydra reduces inference time by 30% relative to TabPFN while retaining much of its predictive performance. Tydra also outperforms an approximately ten-times-larger Hydra model while providing faster inference. The results indicate that hybrid architectures are a promising direction for tabular foundation models.
Mieszko Komisarczyk, Saurabh Mathur, Maurice Kraus +2
Department of Computer Science, Technical University of Darmstadt, Germany · Department of Computer Science, The University of Texas at Dallas, USA · Hessian Center for Artificial Intelligence (hessian.ai), Darmstadt, Germany +1