SOMTab: Set-Order Mamba for Efficient Tabular In-Context Learning

Hao Wang; Siyu Zhang; Wei Ma

cs.LGAug 28, 2026

SOMTab: Set-Order Mamba for Efficient Tabular In-Context Learning

Authors: Hao Wang, Siyu Zhang, Wei Ma

Organizations: Institute of Statistics and Big Data, Renmin University of China, Beijing, China

Abstract

Tabular foundation models based on in-context learning have recently emerged as strong alternatives to task-specific model fitting. However, the current performance frontier remains dominated by attention-heavy architectures, where attention is used throughout the modeling pipeline. This raises a natural question: is attention necessary at every stage of tabular in-context learning? We introduce SOMTab, a Set-Order Mamba architecture for efficient tabular in-context learning. SOMTab separates representation construction from query-conditioned retrieval. For row and column representations, it maps unordered table tokens into stable latent slots and applies Mamba-based state-space mixing to construct compact representations. For final prediction, it retains attention-based in-context learning to preserve query-conditioned retrieval from labeled context examples. We further introduce DCH-TailMix, a synthetic prior that combines degree-corrected graph heterogeneity with mixed heavy-tailed regimes to diversify synthetic dependency structures. Across tabular benchmarks, SOMTab approaches the performance of strong Transformer-based tabular foundation models while achieving faster inference and lower GPU memory usage, yielding a favorable efficiency--accuracy trade-off.

Figures & tables

Figure 1: Predictive quality–efficiency trade-off on all TALENT classification datasets. The horizontal axis reports the median fit-plus-predict time per 1K labeled context samples, and the vertical axis reports the mean min–max normalized log-loss across datasets. Lower values are better on both axes, and the runtime axis is shown on a logarithmic scale. All methods are rerun on the same H200 server under a common evaluation pipeline; baselines use their official default configurations. The dashed line denotes the empirical Pareto frontier.

Figure 2: Architecture overview of SOMTab. The top illustrates the end-to-end pipeline, where variable- and sample-level Set-to-Order Mamba modules construct compact row representations, followed by an attention-based ICL reader for query prediction. The bottom details the corresponding representation modules and context-only attention mechanism.

Method	Norm. LL $\downarrow$	Accuracy $\uparrow$	Macro F1 $\uparrow$	Time/1K $\downarrow$
SOMTab	0.054	0.849	0.785	0.019
TabICLv2	0.029	0.854	0.790	0.463
TabICLv2(1 est.)	0.046	0.853	0.789	0.151
TabICL	0.105	0.846	0.776	0.453
TabPFNv2	0.100	0.849	0.779	0.675
RealMLP	0.498	0.830	0.757	4.379

Table 1: Aggregate performance over all TALENT classification datasets. Normalized log-loss, accuracy, and Macro F1 are averaged across datasets, while runtime is the median fit-plus-predict time per 1K labeled context samples. TabICLv2 (default) uses the standard ensemble setting with eight estimators, whereas TabICLv2 (1 estimator) uses a single estimator.

Figure 3: Runtime and GPU-memory scaling with context length. We fix

n_{\mathrm{test}}=1024

m=32

features, and

C=10

classes, and vary the number of context rows from 512 to 28,672. All foundation models use one estimator to isolate architectural scaling from prediction ensembling. Results are measured after one warmup run and report the median of three repeats on identical synthetic data.

Figure 4: Controlled early-training ablation. All variants are trained from scratch for 20K steps under the same configuration. Checkpoints at 5K, 10K, 15K and 20K steps are evaluated on the 15 TALENT datasets where SOMTab has the largest accuracy gaps relative to TabICLv2 in the main benchmark. The vertical axis reports the equally weighted mean accuracy across datasets.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Shape	Description
$E^{(0)}$	$n\times m\times d$	initial cell tokens
$Z_{j}^{\mathrm{var}}$	$K_{v}\times d$	latent memory for feature $j$
$E^{\mathrm{var}}$	$n\times m\times d$	context-conditioned cell tokens
$Z_{i}^{\mathrm{sample}}$	$4\times d$	ordered slots for row $i$
$h_{i}$	$4d$	row representation
$H$	$n\times 4d$	representations of all rows

Appendix

Table 2: Principal tensors in the SOMTab forward pass. Batch dimensions are omitted. The main model uses

d=128

K_{v}=64

, and

K_{r}=K_{s}=4

Graph family	Probability	Base sparsity $\tau$	Structural terms	Parent cap
controlled heavy-tail	$0.40$	$\mathcal{U}(-4.0,-0.5)$	degree corrections; $\lambda=0$	at most $8$
modular heavy-tail	$0.25$	$\mathcal{U}(-4.0,-0.5)$	module-pair and low-rank biases	at most $8$
low-rank correlated connectivity	$0.15$	$\mathcal{U}(-4.0,-0.5)$	correlated low-rank edge logits	at most $8$
dense weak-dependency	$0.20$	$\mathcal{U}(-2.4,-0.8)$	denser modular and low-rank effects	at most $12$

Appendix

Table 3: Graph-family mixture in DCH-TailMix. For dense weak-dependency graphs, the parent cap for node

j

is at least

\min(8,j-1)

under one-based topological indexing. When the initial Bernoulli sample for the final node is too sparse, additional high-probability candidate edges are inserted before applying this cap.

Figure 5: Representative realizations from the four DCH graph families. Nodes are arranged in topological order from left to right; colors identify sampled modules when modular structure is active. Each panel shows one sampled graph.

Figure 6: Illustrative tail behavior of the three mixture components before clipping. Curves show representative draws of the regime parameters.

Family	Construction	Sampled ranges / constraints
linear	random matrix projection	Gaussian, low-rank, SVD-shaped, kernel, or activated matrix; input dimension $\leq 128$
neural	shallow random MLP	depth $1$ – $3$ ; width $1$ – $127$ ; sampled input/output activations
tree	ensemble of random split trees	$1$ – $128$ trees; depth $1$ – $7$ ; input dimension $\leq 32$
discretization	nearest-center partition followed by projection	$2$ – $255$ centers; $\ell_{p}$ distance with $p\in[0.5,4]$ log-uniform
GP-like	finite random Fourier-feature map	$256$ features; axis-aligned probability $0.5$ ; spectral tail $a\in[2,20]$ log-uniform
quadratic	symmetric quadratic form with a bias coordinate	input dimension $\leq 20$

Appendix

Table 4: Random function library for latent propagation.

Figure 7: Representative one-dimensional slices of the random function families. Each panel shows the response of one function family along a fixed low-dimensional parent trajectory, illustrating smooth, thresholded, piecewise, and interaction-based mechanisms.

Setting	Stage 1	Stage 2
Initialization	random	Stage-1 checkpoint
Training steps	$300{,}000$	$100{,}000$
Sequence length	$400$ – $8192$	$400$ – $8192$
Length sampling	log-uniform	log-uniform
Context fraction	$\mathcal{U}(0.1,0.9)$	$\mathcal{U}(0.5,0.9)$
Initial learning rate	$8\times 10^{-4}$	$5\times 10^{-5}$

Appendix

Table 5: Two-stage pretraining curriculum. Sequence length is the total number of context and query rows.

Variant	Representation	Prior / reader change
SOMTab	Set-Order Mamba	DCH-TailMix + attention reader
Cauchy	Set-Order Mamba	random Cauchy graph prior
SCM–Tree	Set-Order Mamba	$70\%$ MLP-SCM + $30\%$ tree-SCM
Tab-Full	Set-Order Mamba	DCH-TailMix + Mamba reader

Appendix

Table 6: Variants in the controlled ablation. All unspecified architecture and optimization settings are shared.

Figure 8: Early-training accuracy under a matched 20K-step budget. Values are equally weighted means over the fixed 15-dataset subset.

Dataset	SOMTab	TabICLv2	TabICL	TabPFNv2	RealMLP	CatBoost	XGBoost	ExtraTrees	RandomForest
mice_protein_expression	1.000	1.000	1.000	1.000	1.000	0.972	0.954	1.000	0.986
shuttle	1.000	0.999	0.999	0.999	1.000	1.000	1.000	0.999	1.000
JapaneseVowels	0.997	0.999	0.999	0.999	0.990	0.960	0.979	0.985	0.974
gas-drift	0.997	0.997	0.997	0.996	0.996	0.991	0.995	0.996	0.994
pendigits	0.996	0.997	0.997	0.996	0.998	0.986	0.989	0.991	0.991
thyroid	0.994	0.994	0.993	0.997	0.994	0.995	0.997	0.983	0.997

Appendix

Table 7: Per-dataset accuracy results on selected TALENT datasets. We report accuracy on 20 TALENT classification datasets. The final 15 rows form the fixed subset used in the controlled early-training ablation.

Figure 9: Performance–efficiency comparison on TabArena. The x-axis shows inference time per 1000 samples on a logarithmic scale, and the y-axis reports the average performance gap to the best method on each dataset (lower is better). SOMTab-H200 and SOMTab-A100 denote measurements obtained on H200 and A100 GPUs, respectively, while other results are taken from the official TabArena evaluation.

Figure 10: Pairwise win-rate comparison on TabArena. Each entry denotes the fraction of datasets where the row model achieves higher predictive performance than the column model.

Tabular ICL

Tabular in-context learning (ICL) uses examples in a table to adapt pretrained models to prediction tasks without task-specific weight updates.

See all 108 papers

TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction
Aug 2, 2026Rasa Hosseinzadeh, Alex Labach, Zexin Xue +3Tabular Foundation Models Tabular ICL
Layer 6 AI, Toronto, Canada · Cohere, Toronto, Canada
TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attention
Jun 5, 2026Si-Yang Liu, Han-Jia YeTabular Foundation Models Efficient Inference
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China.
Tydra: An Efficient Hybrid Model for Tabular Data
Aug 21, 2026Mieszko Komisarczyk, Saurabh Mathur, Maurice Kraus +2Hybrid Transformer-State Space Model Architectures Tabular Foundation Models
Department of Computer Science, Technical University of Darmstadt, Germany · Department of Computer Science, The University of Texas at Dallas, USA · Hessian Center for Artificial Intelligence (hessian.ai), Darmstadt, Germany +1

Aug 2, 2026J/K move · Enter open · S save

TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction

Rasa Hosseinzadeh, Alex Labach, Zexin Xue, Shuyi Han+2

Layer 6 AI, Toronto, Canada · Cohere, Toronto, Canada

Tabular Foundation Models Tabular ICL Tabular Prediction

Open paper AskPDF arXiv

Tabular foundation models, driven by in-context learning, have rapidly grown in quality and popularity. However, recent approaches with either cell-based architectures or retrieval have sacrificed efficiency for raw performance, restricting their utility in situations where compute is limited or inference speed is crucial. We adopt an alternate approach, sticking with row-based attention while incorporating long context pre-training to eliminate the need for retrieval. By combining this with architectural improvements and SSL pre-training on a newly-sourced, larger corpus of real data results, we present TabDPT-Turbo, a model that provides comparable default performance to TabDPT v1.1 on TabArena-Lite, CC18, and CTR23, at orders of magnitude faster. In our experiments, TabDPT-Turbo is the fastest model overall among leading foundation models. We have released the new model as TabDPT v1.2 at https://github.com/layer6ai-labs/TabDPT-inference.

Abstract

Tabular ICL

Explore similar work

TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction

TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attention

Tydra: An Efficient Hybrid Model for Tabular Data