Matrix completion underlies problems from tabular imputation to causal inference, yet existing tabular foundation models treat it as entry-by-entry prediction, repeating context for every target and discarding the matrix's two-dimensional structure. We introduce MatrixFormer, a pre-trained matrix-native transformer that predicts a full distribution for every missing entry in a single forward pass. MatrixFormer is trained entirely on synthetic low-rank and latent-factor matrices under diverse missingness patterns. Applied zero-shot and with the same model weights, MatrixFormer achieves competitive performance on causal inference panel-data tasks, language-model benchmark-score completion, tabular imputation, and recommendation systems matrix completion. These results position MatrixFormer as a general-purpose foundation model for matrix completion.
Figures & tables
Figure 1: MatrixFormer forward pass. Each matrix cell is represented by a token with an explicit missingness embedding. Missing values use a learned mask token. Learned row- and column-CLS tokens are added around the matrix, and each transformer block alternates feature attention across columns within a row and datapoint attention across rows within a column. After L blocks, the CLS border is removed and a decoder predicts the masked entries.
Figure 2: Fixed-column inference runtime for legacy TabImpute and MatrixFormer, with logarithmic row and runtime axes.
Figure 3: Normalized causal-panel scores (lower is better; 0 = best and 1 = worst of the seven methods shown in a draw; dashed line at 1). The blue bars show the MatrixFormer ensemble. Top: RMSE on the hidden treated block. Bottom: absolute error of the average treatment effect on the treated. Bars are means over draws with designs weighted equally; whiskers are 95% intervals over draws.
Figure 4: MissBench: Overall, MCAR, MAR, and MNAR. Columnwise NRMSE across 42 datasets at the nominal 40% setting; lower is better. MatrixFormer is blue; baselines are gray. Methods match the UCI figure, with TabImpute (legacy) additionally shown here. The overall panel includes all eleven mechanisms. Rows are sorted by overall NRMSE, with that order shared by the four panels. Whiskers are 95% dataset-bootstrap confidence intervals. † denotes incomplete coverage.
Figure 5: UCI: Overall, MCAR, MAR, and MNAR. The suite contains twelve datasets, five missingness rates, and ten repeats. Each mechanism averages available rates and repeats within datasets, then weights the available datasets equally; lower is better. The overall panel includes all three mechanisms. Whiskers are 95% dataset-bootstrap confidence intervals. Rows follow UCI overall NRMSE. † denotes incomplete coverage.
MovieLens 100K
Netflix small
Method
RMSE
MAE
RMSE
MAE
MatrixFormer
0.8874
0.7279
0.8617
0.7330
SVD
0.8998
0.7430
0.8621
0.7358
PMF
0.9138
0.7534
0.8835
0.7594
UserKNN
0.9382
0.7681
0.9296
0.7920
ItemKNN
0.9944
0.8242
0.9502
0.8160
Table 1: Rating prediction in original rating units; lower is better. RMSE and MAE are macro-averaged over test users. MatrixFormer-Impute uses global observed-training normalization; MatrixFormer retains each expert’s recipe as specified in the text. All six Cornac baselines are retained.
Dataset
Models
Tasks
Observed
MTEB
263
56
77.3%
MMLU
5,452
57
100.0%
Merged
118
114
31.1%
BenchPress
83
49
33.8%
Table 2: Released benchmark-score matrices used for evaluation. Observed entries are the available scores before holding out model rows.
Dataset
k
MatrixFormer
Gaussian
MTEB
5
0.7651
0.7498
10
0.8261
0.8125
15
0.8316
0.8259
MMLU
5
0.8992
0.9064
10
0.8993
0.9118
15
0.9155
0.9266
Table 3: Benchmark-score completion with greedy MI selection. Entries are mean R2 over ten folds at each task budget k , rounded to four decimals. All methods share context rows, selected tasks, target masks, and scoring rules. Numerical differences are descriptive.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Setting
Value embedding fval
MLP 1→768→768 with GELU
Missing entries
learned mask vector mmask ; learned two-entry missingness embedding emiss added to every cell
CLS border
Kc=12 row-CLS tokens tjrow and Kr=12 column-CLS tokens ticol ; corner entry (i,j) is ticol+tjrow
Transformer-based tabular foundation models (TFMs) dominate small to medium tabular predictive benchmark tasks, yet their inference mechanisms remain largely unexplored. We present the first large-scale mechanistic study of layerwise dynamics in 6 state-of-the-art tabular in-context learning models. We explore how predictions emerge across depth, identify distinct stages of inference and reveal latent-space dynamics that differ from those of language models. Our findings indicate substantial depthwise redundancy across multiple models, suggesting iterative refinement with overlapping computations during inference stages. Guided by these insights, we design a proof-of-concept, looped single-layer model that uses only 20% of the original model's parameters while achieving comparable performance. The code is available at https://github.com/amirbalef/is_one_layer_enough.
Amir Rezaei Balef, Mykhailo Koshil, Katharina Eggensperger
TU Dortmund University, Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Dortmund, Germany · University of Tübingen, Tübingen, Germany
Every major data modality now has a foundation model that understands it natively: text has language models, images have vision models, audio has audio models. Tabular data, the modality on which many consequential real-world AI decisions are made, does not. Every approach to tabular AI today, from gradient-boosted trees to the latest tabular foundation models, requires a preprocessing pipeline before any model can consume the data. None of them understand tabular data as a modality. We introduce the Data Language Model (DLM), the missing foundation model for tabular data. A DLM understands tables the way a language model understands sentences: natively, without serialization or preprocessing, directly from raw cell values. It is the tabular data layer on which AI models, agents, and vertical AI applications can be built, eliminating the preprocessing pipelines that currently stand between raw data and every AI system that consumes it. We present Schema-1, the first DLM: a 140M parameter model trained on more than 2.3M synthetic and real-world tabular datasets. Schema-1 outperforms gradient-boosted ensembles, AutoML stacks, and the tabular foundation models we evaluate on established row-level prediction benchmarks. On missing value reconstruction it achieves lower reconstruction error than all classical statistical methods and frontier large language models on mean performance across conditions, establishing that structural understanding of a dataset's own distributional geometry is more useful for imputation than world knowledge encoded in language. It identifies the industry sector of any unseen dataset from raw cell values alone, reliably across any domain, a task no prior tabular model can perform. It is the native tabular understanding layer that has been missing from the AI stack.
Predicting missing cell values in tabular data is a fundamental problem in data cleaning. While state-of-the-art reasoning models show great promise in predicting missing values in tables, by reasoning holistically across rows and columns, they are costly to deploy at scale and tend to be overconfident, often generating hallucinated or false-positive predictions. In this paper, we observe that achieving high-precision missing-value prediction in tables requires a distinct combination of three capabilities: (1) world knowledge, (2) text-based reasoning, and (3) code-based reasoning. We systematically explore design choices for combining these capabilities, and propose an Auto-Fill approach that post-trains three specialist small language models (SLMs), each optimized for one capability. We develop a calibrated ensemble mechanism that either dynamically selects the most confident specialist or abstains, ensuring high accuracy. Extensive experiments on 11 benchmarks with 2200 real tables drawn from diverse domains show that Auto-Fill achieves superior accuracy compared to state-of-the-art reasoning models (e.g., o3-pro, Gemini 3 Pro, and DeepSeek R1), while operating at a fraction (less than 1%) of the cost of these frontier models. Our results highlight the effectiveness of specialization and calibrated abstention in the important domain of tabular data. Auto-Fill is publicly available at https://github.com/lyrain2001/auto-fill.