EXAONE Tabular is a compact tabular foundation model family for classification and regression via in-context learning, producing predictions without dataset-specific gradient updates. Pretrained exclusively on a synthetic structural-causal-model (SCM) prior, its central contribution is an architecture-centered redesign of tabular in-context learning. Rather than compressing features into a fixed row embedding before a separate row-level learner, EXAONE Tabular interleaves feature-axis attention within each item with support-conditioned item-axis attention within each feature at every Transformer layer, mediated by item-summary and feature-summary tokens. Across four public benchmarks, EXAONE Tabular combines strong predictive performance with high efficiency. On TabArena, its 20.81M-parameter classification model ranks first overall, surpassing tuned ensembles and 4-hour AutoML pipelines, while regression reaches the performance regime of the 1.64B-parameter TabFM at roughly 1/11 the inference cost. On BCCO and TALENT, EXAONE Tabular ranks second in classification and first in regression. On ScoringBench, it achieves the best mean rank for both point-estimation and predictive-distribution quality, leading the R2, RMSE, and CRPS evaluations. Together, these results establish EXAONE Tabular as a state-of-the-art compact tabular foundation model family, combining strong predictive performance across classification, point regression, and probabilistic regression with an efficient model design.
Figures & tables
Figure 1: Performance–size Pareto front on the full TabArena benchmark (classification and regression). EXAONE Tabular lies on the Pareto front in its default setting and is the strongest compact tabular foundation model ( ≤ 100M parameters), outperforming every model shown except TabFM — with which it is statistically tied at ∼ 1.3% of the parameters — by 113 to 488 Elo. Sizes count the weights stored in the evaluated checkpoints.
Component
Role
Cross-Axis Summary Transformer (CAST)
Interleaves feature-axis processing within items with support-conditioned item-axis processing within features; item-summary tokens join feature-axis sequences, while feature-summary tokens join support item-axis sequences.
Synthetic SCM prior
Generates causal, categorical, nonlinear, noisy, and missing-data prediction tasks without using real pretraining tables.
Task-specific prediction heads
Uses separate classification and regression heads for the independently trained task-specific checkpoints.
Training and inference pipeline
Applies task-specific optimization and inference procedures, including preprocessing, test-time ensembling, and chunked support-conditioned inference.
Table 1: Core components of EXAONE Tabular.
Figure 2: Overall framework of EXAONE Tabular. The model combines an SCM-based synthetic prior, a Cross-Axis Summary Transformer with interleaved feature-axis and item-axis attention, task-specific heads for classification and regression, and in-context inference on support and query sets. Item-summary tokens are concatenated with cell states along the feature axis, while feature-summary tokens are concatenated with support-item states along the item axis.
Configuration
Value
Embedding dimension
192
Attention heads
6
Transformer layers
12
Feed-forward expansion
4×
Feature-attention operations per layer
2
Item-summary tokens per item
3
Table 2: Core model configuration.
Figure 3: Accuracy–latency Pareto fronts on TabArena for (a) classification and (b) regression tasks. Elo is plotted against median inference time per 1,000 samples (log scale); dashed staircases trace the Pareto fronts, and horizontal lines mark the 4-hour AutoGluon reference pipelines [ 7 ] . Using its default single-forward-pass inference at 0.605 s per 1,000 samples, EXAONE Tabular lies on the Pareto front of both task types. On classification it attains the highest Elo on the entire leaderboard, exceeding TabPFN-3 by ∼ 125 Elo at comparable latency. On regression it reaches the accuracy regime of TabFM — within overlapping 95% confidence intervals — at roughly 1/11 the inference cost, and exceeds the regression-specialized Nori models [ 32 ] by ∼ 140 Elo.
Figure 4: Default vs. tuned and post-hoc ensembled TabArena Elo for (a) classification and (b) regression. Bars show each method’s default configuration, with additional segments marking the gains from per-task tuning and post-hoc ensembling where those variants exist. Conventional GBDT and neural baselines depend heavily on both, worth ∼ 230 Elo for LightGBM [ 17 ] and RealMLP [ 16 ] on classification and over 420 Elo for RealMLP on regression alone. EXAONE Tabular (navy), like most tabular foundation models, is evaluated in a single untuned default configuration, and ranks first on classification and, on regression, second only to TabFM — a model ∼ 78 × its size. Dashed lines mark the 4-hour AutoGluon 1.4/1.5 pipelines, both of which it clears on both task types.
Figure 5: Pairwise win rates on the full TabArena benchmark (classification and regression). Each entry (A,B) reports the percentage of evaluation splits on which method A attains a lower test error than method B ; cells are shaded from purple (row loses) through white (parity) to green (row wins). Method labels are colored by family — purple: tabular foundation models; blue: neural networks; green: GBDTs. With a single fixed default configuration, EXAONE Tabular wins the majority of splits against the strongest variant of every competing method except TabFM: 65–92% against the other tabular foundation models, and at least 89% against every tuned-and-ensembled GBDT and neural-network baseline. The advantage is therefore consistent at the split level rather than being driven by a small subset of datasets.
Figure 6: Performance comparison on the BCCO benchmark. EXAONE Tabular achieves the second-highest mean accuracy on classification and the highest mean R2 on regression, demonstrating competitive performance across both tasks. Error bars represent 95% bootstrap confidence intervals.
Figure 7: Performance comparison on the TALENT benchmark. EXAONE Tabular achieves the second-highest mean accuracy on classification and the highest mean R2 on regression, demonstrating consistently competitive performance across both tasks. Error bars represent 95% bootstrap confidence intervals.
Figure 8: Joint point-estimation and predictive-distribution rankings on ScoringBench. Each point represents a model with valid results for both metrics and is positioned by its official Autorank mean ranks over 102 regression datasets; lower ranks are better on both axes. Fine-tuned and HPO variants are omitted for readability, leaving 28 models. The dashed diagonal indicates identical rank under R2 and CRPS, while the highlighted Pareto-optimal point identifies EXAONE Tabular, which achieves the best mean rank for both metrics. Name tags are restricted to selected representative models.
Tabular foundation models enable in-context learning by predicting unseen observations from a labeled support table without task-specific parameter updates. Despite recent advances, most existing approaches rely on dense architectures that apply the same transformations across features and datasets. We introduce OrionMoE-Small-v1.0, a compact tabular foundation model that combines distribution-aware representations with dataset-conditioned sparse mixture-of-experts computation. Built on a dual-axis transformer architecture, OrionMoE-Small selectively activates experts with routing decisions informed by both individual cell representations and the broader dataset context. This design enables adaptive allocation of computation across heterogeneous tabular problems while retaining a shared computational pathway across tasks. The model is pretrained entirely on synthetic classification tasks generated from a structured causal prior and performs downstream prediction without gradient-based adaptation. Experiments across multiple tabular benchmarks demonstrate competitive performance against substantially larger foundation models, with particularly strong results on small-context and high-dimensional datasets. Further analysis reveals favorable relative performance under class imbalance and highlights the importance of feature composition.
We introduce Xiaomi-TabLDM, a tabular large data foundation model for classification and regression via in-context learning, which delivers superior prediction accuracy without requiring task-specific fine-tuning. Pretrained exclusively on synthetic data generated from structural causal models (SCMs), our model enables more flexible context utilization and more efficient capacity scaling. i) A new performance standard. Strong regression performance across benchmarks: Xiaomi-TabLDM ranks 1st on OpenML-CTR23 and 2nd on regression across TALENT, TabArena, and BCCO, demonstrating consistently strong regression performance across four complementary benchmark suites. Favorable performance--efficiency trade-off: Xiaomi-TabLDM combines strong predictive performance with substantially lower computational cost. For example, on TabArena regression, it achieves the second-highest Elo while using 82% less training time and 68% less prediction time than the top-ranked TabFM. ii) Large-scale synthetic pretraining. Xiaomi-TabLDM expands the coverage and diversity of synthetic tabular data used for pretraining. We also adopt a three-stage training strategy together with dual-stream feature grouping, lightweight Attention Residual, and sparse Mixture-of-Experts, enabling Xiaomi-TabLDM to learn richer feature interactions and expert specialization across diverse tabular tasks. iii) Test-time scaling. Xiaomi-TabLDM further extends tabular prediction through test-time compute scaling, where allocating additional computation at inference time consistently improves predictive performance over the base model.
Tabular foundation models, such as TabPFNv2 and TabICL, have recently dethroned gradient-boosted trees at the top of predictive benchmarks, demonstrating the value of in-context learning for tabular data. We introduce TabICLv2, a new state-of-the-art foundation model for regression and classification built on three pillars: (1) a novel synthetic data generation engine designed for high pretraining diversity; (2) various architectural innovations, including a new scalable softmax in attention improving generalization to larger datasets without prohibitive long-sequence pretraining; and (3) optimized pretraining protocols, notably replacing AdamW with the Muon optimizer. On the TabArena and TALENT benchmarks, TabICLv2 without any tuning surpasses the performance of the current state of the art, RealTabPFN-2.5 (hyperparameter-tuned, ensembled, and fine-tuned on real data). With only moderate pretraining compute, TabICLv2 generalizes effectively to million-scale datasets under 50 GB GPU memory while being markedly faster than RealTabPFN-2.5. We provide extensive ablation studies to quantify these contributions and foster open research by releasing code for inference, pretraining, and synthetic data generation at https://github.com/soda-inria/tabicl.
Jingang Qu, David Holzmüller, Gaël Varoquaux +1
SODA Team, INRIA Saclay, Palaiseau, France · Probabl, France