OrionMoE-v1.0: A Mixture-of-Experts Foundation Model for Tabular In-Context Learning

Mohamed Bouadi; Nassim Bouarour; Aditya Tanna; Vinay Kumar Sankarapu

cs.LGOct 14, 2026

OrionMoE-v1.0: A Mixture-of-Experts Foundation Model for Tabular In-Context Learning

Authors: Mohamed Bouadi, Nassim Bouarour, Aditya Tanna, Vinay Kumar Sankarapu

Organizations: Lexsi Labs

Abstract

Tabular foundation models enable in-context learning by predicting unseen observations from a labeled support table without task-specific parameter updates. Despite recent advances, most existing approaches rely on dense architectures that apply the same transformations across features and datasets. We introduce OrionMoE-Small-v1.0, a compact tabular foundation model that combines distribution-aware representations with dataset-conditioned sparse mixture-of-experts computation. Built on a dual-axis transformer architecture, OrionMoE-Small selectively activates experts with routing decisions informed by both individual cell representations and the broader dataset context. This design enables adaptive allocation of computation across heterogeneous tabular problems while retaining a shared computational pathway across tasks. The model is pretrained entirely on synthetic classification tasks generated from a structured causal prior and performs downstream prediction without gradient-based adaptation. Experiments across multiple tabular benchmarks demonstrate competitive performance against substantially larger foundation models, with particularly strong results on small-context and high-dimensional datasets. Further analysis reveals favorable relative performance under class imbalance and highlights the importance of feature composition.

Figures & tables

Model	Params (M)	TabArena	BeyondArena	OpenML-CC18	TALENT
Small models ( $<50$ M)
OrionMoE-Small-v1.0	37.7	1320	1149	1118	1024
Causilo	36.1	1530	1268	1170	1125
TabICLv2	27.6	1277	1089	1121	1054
Kumo-Tabular-Small	28	1558	–	1241	1150
Medium models ( $50$ – $150$ M)

Table 1: Model capacity and predictive performance of recent tabular foundation models. Models are grouped into three capacity regimes: small (

<50

M parameters), medium (

50

–

150

M), and large (

>150

M). We report Elo ratings on TabArena (

N=38

), BeyondArena (

N=98

), OpenML-CC18 (

N=48

), and TALENT (

N=148

). Higher Elo indicates better predictive performance. Ratings are computed independently within each benchmark and are not directly comparable across benchmarks. A dash indicates unavailable results.

Figure 1: Elo against total runtime per 1K samples on 38 TabArena classification tasks. Total time is the fit time per 1,000 training rows plus the prediction time per 1,000 test rows, median over the 38 classification datasets.

	Overall	By Task		By Dataset Size		By Target
Model	Elo	Binary	Multiclass	Small	Medium	Balanced	Imbalanced	Extreme
	( $N=38$ )	( $N=30$ )	( $N=8$ )	( $N=28$ )	( $N=10$ )	( $N=10$ )	( $N=16$ )	( $N=12$ )
Kumo-Tabular	1795	1777	1946	1776	2030	1743	1848	1905
LimiX-2	1769	1761	1862	1764	1945	1794	1855	1747
Kumo-Tabular-Medium	1726	1719	1814	1731	1867	1699	1827	1729
TabPFN-3.5	1677	1692	1672	1637	2009	1674	1750	1685

Table 2: TabArena Elo scores across task and dataset regimes. Elo is recomputed independently within each subset. Higher is better. The number of datasets used to compute each score is shown in parentheses.

Figure 2: Pareto frontier of Elo rating versus total runtime per 1K samples on BeyondArena classification tasks. Higher Elo indicates stronger predictive performance and lower runtime indicates greater efficiency. The frontier summarizes the trade-off between predictive quality and computational cost under more challenging IID, grouped, temporal, large-scale, and high-dimensional evaluation settings.

Figure 3: Elo ratings on BeyondArena across evaluation regime, dataset scale, and dimensionality. Results are reported separately for IID, grouped, and temporal splits, as well as for different dataset-size and feature-dimensionality regimes. The decomposition highlights how the relative ranking of tabular models changes across distribution shifts and dataset characteristics.

	Overall	By Task		By Regime			By Dataset Size			By Dimensionality
Model	Elo	Binary	Multiclass	IID	Grouped	Temporal	Small	Medium	Large	Narrow	Medium	Wide
	( $N=98$ )	( $N=73$ )	( $N=25$ )	( $N=77$ )	( $N=11$ )	( $N=10$ )	( $N=38$ )	( $N=47$ )	( $N=13$ )	( $N=13$ )	( $N=58$ )	( $N=27$ )
TabPFN-3.5	1348	1350	1349	1444	1095	1319	1326	1482	1294	1304	1404	1303
TabPFN-3.5-Fast	1287	1272	1343	1354	1083	1285	1275	1396	1219	1274	1333	1240
Causilo	1268	1275	1255	1342	1126	1123	1304	1356	1106	1241	1300	1248
TabFM	1222	1202	1295	1318	1076	924	1258	1412	727	1286	1302	1088

Table 3: BeyondArena Elo scores across task, split-regime, dataset-size, and dimensionality regimes. Elo is recomputed independently within each subset using datasets evaluated by all participating methods. Higher is better.

N

is the number of datasets used to compute each score. T+E means Tuned + Ensemble.

Figure 4: Elo ratings on BeyondArena after restricting the analysis to datasets with fewer than 100K samples. Results are decomposed by IID, grouped, and temporal evaluation regimes and by feature dimensionality. Removing large-scale datasets isolates the effect of distribution shift and dimensionality from the additional difficulty introduced by dataset scale.

Figure 5: Elo ratings on the subset of OpenML-CC18 tasks completed by all compared models. Restricting the evaluation to common tasks ensures that pairwise comparisons and model rankings are computed over the same benchmark instances.

Panel of Figure 6 — Figure 6: Elo ratings on the common-task subsets of TALENT and BCCO. (a) TALENT and (b) BCCO. For each benchmark, only tasks completed by all compared models are retained, ensuring that the reported Elo rankings are based on a consistent set of evaluation tasks.

Panel of Figure 7 — Figure 7: Elo ratings for models that completed the full benchmark suite. (a) OpenML-CC18 and (b) TALENT. By restricting the comparison to models with complete benchmark coverage, the resulting rankings avoid differences caused by missing evaluations or partial task completion.

Figure 8: Elo ratings on BCCO for models that completed the full benchmark suite (106 datasets). Restricting the comparison to fully evaluated models provides a direct ranking without relying on partially observed benchmark results.

Model	Params (M)	TabArena	BeyondArena	OpenML-CC18	TALENT
Causilo	36.1	1530	1268	1170	1125
OrionMoE-Small-v1.0	37.7	1320	1149	1118	1024
TabICLv2	27.6	1277	1089	1121	1054
TabICLv1	25	1072	–	768	753
Limix-16M	16	1056	–	859	872
EXAONE-Tabular	20.8	1515	1176	1066	1040

Table 4: Comparison of small tabular foundation models (

<50

M parameters) across four classification benchmarks. Elo ratings are computed independently for each benchmark; higher is better. The strongest result within each benchmark is highlighted in bold.

Model	Params (M)	TabArena	BeyondArena	OpenML-CC18	TALENT
OrionMoE-Small-v1.0	37.7	1320	1149	1118	1024
Medium models ( $50$ – $150$ M)
TabPFN-3	53	1351	1166	1015	1021
TabDPT-1.3	63	1215	1041	1061	974
TabDPT-Turbo	63	1090	–	952	–
Mitra-v1	63	1019	–	638	616

Table 5: Comparison of OrionMoE-Small with medium-sized (

50

–

150

M parameters) and large (

>150

M parameters) tabular foundation models. Elo ratings are reported on TabArena (

N=38

), BeyondArena (

N=98

), OpenML-CC18 (

N=48

), and TALENT (

N=148

). Higher is better. Ratings are computed independently within each benchmark and are not directly comparable across benchmarks. A dash indicates unavailable results.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Complementary performance analysis on TabArena classifiCation tasks. (a) Improvability as a function of total runtime per 1K samples, illustrating the trade-off between remaining performance headroom and computational cost. (b) Pairwise win-rate matrix showing how frequently each method outperforms the others across the evaluated tasks.

Model	Elo	#Wins	Avg. Rank $\downarrow$	Improvability $\downarrow$	Train / 1K [s] $\downarrow$	Inference / 1K [s] $\downarrow$	Win Rate [%]
Kumo-Tabular-Large	1795	7	4.95	3.58	2.79	0.27	18.4
LimiX-2	1769	10	5.47	2.38	36.96	10.08	26.3
Kumo-Tabular-Medium	1726	2	6.42	5.43	1.41	0.15	5.3
TabPFN-3.5	1677	3	7.68	7.21	1.85	0.52	7.9
TabFM	1583	6	10.58	5.76	18.45	6.81	15.8
Kumo-Tabular-Small	1558	0	11.45	8.99	0.83	0.09	0.0

Appendix

Table 6: TabArena classifiCation results. Higher is better for Elo, number of wins, and win rate; lower is better for average rank, improvability, and runtime. Training and inference times are normalized per 1,000 samples.

Model	Elo	#Wins	Avg. Rank $\downarrow$	Improvability [%] $\downarrow$	Train / 1K [s] $\downarrow$	Inference / 1K [s] $\downarrow$	Win Rate [%]
TabPFN-3.5	1348	19	6.23	8.66	5.31	1.31	19.4
TabPFN-3.5-Fast	1287	6	7.93	9.66	2.49	0.49	6.1
Causilo	1268	4	8.52	10.27	3.20	0.24	4.1
TabFM	1222	22.5	10.09	8.64	41.89	9.50	23.0
LimiX-2	1188	18.5	11.35	8.75	62.33	18.46	18.9
EXAONE-Tabular	1176	2	11.79	12.81	4.14	0.72	2.0

Appendix

Table 7: BeyondArena classifiCation results over 98 datasets shared by all evaluated methods. Higher is better for Elo, number of wins, and win rate; lower is better for average rank, improvability, and runtime. Training and inference times are normalized per 1,000 samples.

Panel of Figure 10 — Figure 10: Complementary performance analysis on BeyondArena classifiCation tasks. (a) Improvability versus total runtime per 1K samples, capturing the trade-off between additional attainable performance and computational cost. (b) Pairwise win-rate matrix summarizing how consistently each method outperforms competing models across BeyondArena tasks and split regimes.

Figure 11: Mean accuracy on the common-task subset of OpenML-CC18. Each bar reports the average accuracy of a model across the shared set of tasks, with the corresponding overall rank shown inside the bar.

Panel of Figure 12 — Figure 12: Mean accuracy on the common-task subsets of TALENT and BCCO. (a) TALENT and (b) BCCO. Bars report the average accuracy over the shared task set, with the corresponding model rank shown inside each bar.

Figure 13: Mean accuracy on the full BCCO benchmark for models that completed all tasks. Each bar reports the average accuracy over the complete benchmark, with the corresponding model rank shown inside the bar.

Panel of Figure 14 — Figure 14: Mean accuracy for models that completed the full benchmark suite. (a) OpenML-CC18 and (b) TALENT. Bars report the average accuracy across all benchmark tasks completed by every included model, with the corresponding rank shown inside each bar.

Figure 15: Performance across datasets grouped by class imbalance ratio, after merging OpenML-CC18, TALENT, and BCCO. The top panel reports mean accuracy for each model within three imbalance regimes: near-balanced, moderate imbalance, and severe imbalance (

\geq 10\times

). The bottom panel reports the accuracy difference between OrionMoE-Small-v1.0 and the strongest competing baseline within each regime, where values closer to zero indiCate stronger relative performance. Overall, OrionMoE-Small-v1.0 remains competitive across all imbalance levels, with the smallest gap observed in the most severely imbalanced regime.

Dataset	Task ID	Context Size	Query Size	# Features	# Classes
kr-vs-kp	3	2876.4	319.6	36	2
balance-scale	11	562.5	62.5	4	3
mfeat-factors	12	1800.0	200.0	216	10
mfeat-fourier	14	1800.0	200.0	76	10
mfeat-karhunen	16	1800.0	200.0	64	10
mfeat-morphological	18	1800.0	200.0	6	10

Appendix

Table 8: OpenML-CC18 datasets used in the common 48-task evaluation. For each dataset, we report the OpenML task ID, context size, query size, number of input features, and number of classes. Context and query sizes are reported as averages across evaluation folds and can therefore be non-integer

Dataset	Context Size	Query Size	# Features	# Classes
Amazon_employee_access	26215	6554	7	2
BLE_RSSI_dataset_for_Indoor_localization	7987	1997	3	3
BNG(breast-w)	31492	7874	9	2
BNG(cmc)	44236	11060	9	3
BNG(tic-tac-toe)	31492	7874	9	2
Bank_Customer_Churn_Dataset	8000	2000	10	2

Appendix

Table 9: TALENT datasets used in the common 148-dataset evaluation. Context size denotes the number of training examples available to the model at inference time, while query size corresponds to the held-out test set size.

Dataset	Context Size	Query Size	# Features	# Classes
BNG(cmc)	44236	11060	9	3
CPMP-2015-runtime-classification	368	159	22	4
CostaMadre1	207	89	37	2
Fitness_Club_c	1200	300	6	2
GAMETES_Epistasis_2-Way_20atts_0.1H_EDM-1_1	1120	480	20	2
Gender_Gap_in_Spanish_WP	3796	950	13	3

Appendix

Table 10: BCCO datasets used in the common 86-dataset evaluation. For each dataset, we report the effective context size, query size, number of input features, and number of classes.

Tabular Foundation Models

Tabular foundation models learn reusable representations or in-context prediction skills for structured data such as spreadsheets and databases.

See all 188 papers

EXAONE Tabular 1.0 : Technical Report
Aug 26, 2026Moonjung Eo, Min-Kook Suh, Hye-Seung Cho +4Tabular Foundation Models Tabular ICL
LG AI Research
SOMTab: Set-Order Mamba for Efficient Tabular In-Context Learning
Aug 28, 2026Hao Wang, Siyu Zhang, Wei MaTabular ICL Tabular Foundation Models
Institute of Statistics and Big Data, Renmin University of China, Beijing, China
Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models
May 7, 2026Amir Rezaei Balef, Mykhailo Koshil, Katharina EggenspergerTabular Foundation Models Transformer Interpretability
TU Dortmund University, Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Dortmund, Germany · University of Tübingen, Tübingen, Germany

Aug 26, 2026J/K move · Enter open · S save

EXAONE Tabular 1.0 : Technical Report

Moonjung Eo, Min-Kook Suh, Hye-Seung Cho, Jiwon Kim+3

LG AI Research

Tabular Foundation Models Tabular ICL Tabular Classification Tabular Regression

Open paper AskPDF arXiv

EXAONE Tabular is a compact tabular foundation model family for classification and regression via in-context learning, producing predictions without dataset-specific gradient updates. Pretrained exclusively on a synthetic structural-causal-model (SCM) prior, its central contribution is an architecture-centered redesign of tabular in-context learning. Rather than compressing features into a fixed row embedding before a separate row-level learner, EXAONE Tabular interleaves feature-axis attention within each item with support-conditioned item-axis attention within each feature at every Transformer layer, mediated by item-summary and feature-summary tokens. Across four public benchmarks, EXAONE Tabular combines strong predictive performance with high efficiency. On TabArena, its 20.81M-parameter classification model ranks first overall, surpassing tuned ensembles and 4-hour AutoML pipelines, while regression reaches the performance regime of the 1.64B-parameter TabFM at roughly 1/11 the inference cost. On BCCO and TALENT, EXAONE Tabular ranks second in classification and first in regression. On ScoringBench, it achieves the best mean rank for both point-estimation and predictive-distribution quality, leading the

R^2

, RMSE, and CRPS evaluations. Together, these results establish EXAONE Tabular as a state-of-the-art compact tabular foundation model family, combining strong predictive performance across classification, point regression, and probabilistic regression with an efficient model design.

Abstract

Tabular Foundation Models

Explore similar work

EXAONE Tabular 1.0 : Technical Report

SOMTab: Set-Order Mamba for Efficient Tabular In-Context Learning

Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models