TILE: Bridging LLMs and Table Foundation Models for Tabular Learning with Strings

Félix Lefebvre; David Holzmüller; Gioia Blayer; Myung Jun Kim; Marine Le Morvan; Gaël Varoquaux

cs.LGOct 12, 2026

TILE: Bridging LLMs and Table Foundation Models for Tabular Learning with Strings

Authors: Félix Lefebvre, David Holzmüller, Gioia Blayer, Myung Jun Kim, Marine Le Morvan, Gaël Varoquaux

Organizations: SODA Team, Inria Saclay · SAP SE · Probabl

Abstract

Real-world tables are often heterogeneous, combining numerical features with free-form string columns. Table foundation models (TFMs) excel on numerical tables but require strings to be encoded numerically, while large language models (LLMs) hold semantic knowledge but are not designed for tabular predictions. Most existing bridges use LLMs as plug-in encoders for cell strings, yet how to extract representations effectively for tabular learning remains underexplored. We show that the interface between LLMs and tables matter as much as the LLM itself. Applied cell-wise, larger LLMs do not give better features. With our interface, the same LLMs improve steadily with size. TILE (Table-Interfaced LLM Embeddings) combines four ingredients: row serialization to provide cross-column context, intermediate-layer extraction to avoid layers specialized for next-token prediction, columnar pooling to remove mean pooling's length bias, and by-pass features to route numbers and categories past the LLM. Paired with TFMs, TILE sets a new state of the art for tables with strings. Across 100 datasets, it beats the strongest baseline on 89% of splits, with gains across 7 LLMs and 10 tabular learners, and ranks first on the 108-dataset STRABLE benchmark.

Figures & tables

Figure 1: TILE outperforms 30 baselines, with every table foundation model. Elo ratings ( Appendix B ) over 100 regression and classification tasks, at 1,024 train size. TILE uses a Llama-3.1-8B backbone. LLM encoder applies the same LLM to each cell in isolation, then a PCA. TabLLM, an LLM fine-tuned to generate tabular predictions, is rated on the 34 datasets where it ran. CASE uses Gemma-3-4B, as its table-tuned backbone is not public ( Appendix E ).

Figure 2: The TILE architecture. Each row is serialized into a sentence and fed to an LLM. Intermediate-layer token embeddings are pooled within string column spans; then standardized and reduced by one PCA per column, both fitted on training rows. The resulting contextualized representation is concatenated with low-cardinality categorical, numerical, and target-encoded categorical features. A TFM makes the final prediction in a single forward pass over the training and query rows.

	Regression (normalized $R^{2}$ )			Classification (normalized AUC)
Method	1,024 (65)	10k (38)	50k (7)	1,024 (35)	10k (10)	50k (2)
TILE (TabPFN-2.5)	90.21	93.19	92.58	88.00	87.62	79.64
TILE (TabICLv2)	88.70	92.32	92.00	88.94	85.93	80.17
TILE (TabPFN-3)	88.66	92.32	92.14	87.72	87.67	80.68
TILE (TabPFN-3.5)	90.67	94.11	94.66	90.35	88.31	84.58
TILE (Causilo)	91.63	95.20	96.05	90.38	88.02	83.67

Table 1: TILE leads at every train size, on both tasks. Normalized scores

\times 100

, aggregated over the datasets large enough for each size; counts in parentheses. Because the set of datasets varies, compare scores within columns only. LLM enc. is Llama-3.1-8B applied to each cell independently; CASE uses Gemma-3-4B ( Appendix E ). Best result in bold, second best underlined. Table H.2 adds RealMLP, XGBoost, CatBoost and Ridge. Table H.1 gives raw scores.

Figure 3: TILE ranks first on STRABLE. Mean normalized score over STRABLE’s 108 datasets, under its evaluation protocol. Scores are normalized per dataset over 51 STRABLE baselines, TILE excluded, so a score of 1 amounts to picking the best baseline on each dataset. Colors indicate the string handling; transparent bars mark learners that are not table foundation models, trained from scratch. The four TILE variants take the first four places, and TILE with Causilo scores above 1.

Figure 4: Read cell by cell, larger LLMs do not help; with row-level context, they do. TILE (brown), an ablation masking inter-column attention (pink), and the cell-wise LLM encoder (blue), across four Llama-3 and three Qwen3 backbones. The LLM encoder processes isolated cells, mean-pools last-layer representations, and applies per-column PCA. TILE extracts intermediate representations and thus loads fewer parameters. Table H.3 gives raw scores.

	TabPFN-2.5		TabICLv2		TabPFN-3
Model	clf	reg	clf	reg	clf	reg
$-$ numerical by-pass	-11.16	-14.19	-11.67	-13.58	-11.65	-14.59
$-$ inter-column attention	-2.99	-6.52	-3.13	-8.92	-7.17	-9.05
$-$ intermediate layer (last layer)	-0.52	-3.55	-1.33	-3.43	-0.63	-3.07
$-$ numbers in the serialization	-1.43	-1.23	-1.70	-1.63	-1.52	-1.76
$-$ feature-capacity window	-0.22	-0.72	-0.10	-0.22	+0.12	-0.41

Table 2: Ablation study. Normalized score (

\times 100

) of the default TILE architecture, with Llama-3.1-8B and each of the three default learners, and the difference obtained when removing one of its components or adding a variation to it. The bottom rows give the cell-wise LLM encoder, alone, with the target-encoding and categorical by-passes, or with TILE’s feature-capacity window. Normalization puts the regression and classification on a comparable scale; Table H.4 gives raw scores. Cell shading grows with the size of the effect (red: drop, blue: gain). The largest drops result from removing the numerical by-pass or inter-column attention. Columnar pooling cannot be removed without changing the shape of the representation; Section F.4 compares pooling strategies.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Source	b-clf	m-clf	reg	Total
TextTabBench	5	2	10	17
CARTE	11	-	40	51
WikiDBs	1	16	15	32
Total	17	18	65	100

Appendix

Table A.1: Task distribution across sources.

Dataset	Task	# rows	# columns		# classes
			cat.	num.
Diabetes	b-clf	17,000	4	12	2
Job Frauds	b-clf	1,732	11	3	2
Kickstarter	b-clf	18,720	9	8	2
Lending Club	b-clf	11,254	12	15	2
Osha Accidents	b-clf	3,598	15	3	2

Appendix

Table A.2: Overview of TextTabBench datasets used in our benchmark. Table statistics after preprocessing.

Dataset	Task	# rows	# columns		# classes
			cat.	num.
Chocolate Bar Ratings	b-clf	2,218	6	5	2
Coffee Ratings	b-clf	1,670	8	-	2
Michelin	b-clf	6,774	5	2	2
NBA Draft	b-clf	1,550	4	5	2
Ramen Ratings	b-clf	3,726	4	-	2

Appendix

Table A.3: Overview of CARTE datasets used in our benchmark. Table statistics after preprocessing.

Dataset	Task	# rows	# columns		# classes
			cat.	num.
CC Authors	b-clf	16,224	7	1	2
Defenders	m-clf	18,610	9	-	10
Philosophers	m-clf	4,230	7	-	10
US Music Albums	m-clf	3,270	10	1	10
Artist Copyrights	m-clf	2,000	9	1	10

Appendix

Table A.4: Overview of WikiDBs datasets used in our benchmark. Table statistics after preprocessing.

	TextTabBench	CARTE	WikiDBs
# columns	15.7	6.8	6.7
cardinality	286.4	371.4	463.7
string length	975.3	298.8	203.6
string similarity	0.16	0.10	0.08

Appendix

Table A.5: Aggregated features of tabular datasets across sources. The cardinality is computed over 1,024 rows. “String similarity" represents cosine similarity of TF-IDF across rows.

	OkCupid	Benchmark	Both
Method	(1)	(100)	(101)
TILE (Causilo)	83.2	87.6	87.6
TILE (TabPFN-3.5)	82.6	86.5	86.5
TILE (TabPFN-2.5)	62.5	85.0	84.8
TILE (TabICLv2)	82.3	84.2	84.2
TILE (TabPFN-3)	68.7	83.6	83.5

Appendix

Table A.6: Adding the excluded dataset. Normalized scores over the excluded table alone, over the benchmark, and over their union. Normalization is over the 15 methods that ran on all 101 tables, a smaller pool than in the rest of the paper, so the ranking is what should be read.

Hyperparameter	Distribution	Range
n_estimators	Integer	[50, 1000]
max_depth	Integer	[2, 6]
min_child_weight	Log-uniform	[1, 100]
subsample	Uniform	[0.5, 1.0]
learning_rate	Log-uniform	[ $10^{-5}$ , 1]
colsample_bylevel	Uniform	[0.5, 1.0]

Appendix

Table E.1: Search space of the tuned XGBoost baseline. Sampled by a 100-iteration random search, scored by

5

-fold cross-validation repeated

5

times on the training rows.

Figure F.1: Each learner handles a broad range of feature counts. Averaged score as a function of the number of components of a single PCA over the concatenated cell embeddings, which here sets the number of features the learner receives. Every estimator benefits from the projection, and each stays close to its best over a wide plateau, from which we read the feature-capacity windows used in Section 3.1 . Thick lines represent aggregated performance across the 100 datasets, while thin lines represent individual datasets. Error intervals display the standard error of the mean across datasets.

Figure F.2: The categorical cardinality threshold has little effect. Averaged normalized score of TILE as a function of the cardinality threshold below which a string column is also passed directly to the tabular learner, for TabPFN-2.5, TabICLv2 and TabPFN-3. The curves are mostly flat. We use 50 in all other experiments.

Figure F.3: Columnar pooling gives the best representations. Performance of three pooling strategies across the layers of Llama-3.1-8B, scored by Ridge on the embeddings alone (no by-pass features), on the 27 probing datasets of Section A.5 . Ridge is used without PCA, so columnar pooling gives it a higher-dimensional input than the other two strategies. Scores are normalized over the strategies shown here, so they are not comparable with those of the other figures.

Figure G.1: TILE holds the accurate end of the Pareto front. Improvability against median time per dataset at

n_{\mathrm{train}}=1{,}024

, split into regression (left) and classification (right). The dashed staircase is the Pareto front; Pareto-optimal pipelines are labeled in bold, and the other TILE variants in light type. Colors indicate the string encoding and labels the learner; squares mark tuned baselines. CASE ran on faster GPUs than TILE and the LLM encoder ( Section G.2 ). TabLLM supports classification only, and is evaluated on the 34 datasets it completes.

Figure G.2: TILE nearly halves the memory of the same LLM applied cell-wise. Improvability ( Section F.1 , lower is better) against the peak GPU memory a method needs on

1{,}024

train rows of the chocolate_bar_ratings dataset of CARTE, taken as the larger of its two stages: building the representation, then fitting the learner and predicting. As in Figure G.1 , both axes are costs and the dashed staircase is the Pareto front. We report it for TILE, for TF-IDF, the LLM encoder and CASE combined with the same learner (TabPFN-2.5), and for the three end-to-end models, which read the raw table with a single network. TabLLM supports classification only, and runs out of memory on one of the 35 classification datasets, hence the open “partially evaluated” marker: its improvability is computed on the 34 datasets it completes.

	Regression ( $R^{2}$ )			Classification (AUC)
Method	1k (65)	10k (38)	50k (7)	1k (35)	10k (10)	50k (2)
TILE (TabPFN-2.5)	70.8	76.7	82.6	89.4	90.5	82.6
TILE (TabICLv2)	70.1	76.4	82.4	89.6	90.3	82.6
TILE (TabPFN-3)	70.2	76.4	82.3	89.4	90.5	82.7
TILE (TabPFN-3.5)	70.9	77.1	83.4	89.7	90.6	83.1
TILE (Causilo)	71.3	77.8	84.1	89.7	90.6	83.0

Appendix

Table H.1: Raw scores at every training-set size. Unnormalized counterpart of Table 1 :

R^{2}

for regression and ROC-AUC for classification, multiplied by 100 and aggregated over the datasets large enough for each training size; their number in parentheses. Because the set of datasets changes with the training size, compare scores within a column rather than across. LLM enc. is Llama-3.1-8B applied to each cell content in isolation, the same LLM as TILE; TF-IDF is followed by an SVD; CASE uses Gemma-3-4B ( Appendix E ). Best result in bold, second best underlined.

	Regression (normalized $R^{2}$ )			Classification (normalized AUC)
Method	1,024 (65)	10k (38)	50k (7)	1,024 (35)	10k (10)	50k (2)
TILE (TabPFN-2.5)	90.21	93.19	92.58	88.00	87.62	79.64
TILE (TabICLv2)	88.70	92.32	92.00	88.94	85.93	80.17
TILE (TabPFN-3)	88.66	92.32	92.14	87.72	87.67	80.68
TILE (TabPFN-3.5)	90.67	94.11	94.66	90.35	88.31	84.58
TILE (Causilo)	91.63	95.20	96.05	90.38	88.02	83.67

Appendix

Table H.2: Normalized scores at every training-set size, all learners. Full version of Table 1 . Normalized scores

\times 100

, aggregated over the datasets large enough for each size; counts in parentheses. Because the set of datasets varies, compare scores within columns only. LLM enc. is Llama-3.1-8B applied to each cell independently; TF-IDF is followed by SVD; CASE uses Gemma-3-4B ( Appendix E ). Best result in bold, second best underlined. Table H.1 gives raw scores.

Method	Base LLM	Regression ( $R^{2}$ )	Classification (AUC)
TILE	Llama-3.1-70B	72.16	89.82
	Llama-3.1-8B	70.76	89.44
	Llama-3.2-3B	69.27	89.03
	Llama-3.2-1B	68.33	89.02
	Qwen3-8B-Base	69.35	89.02
	Qwen3-4B-Base	68.51	88.80

Appendix

Table H.3: Serialization ablation results aggregated over 100 datasets. Scores are raw

R^{2}

/AUC multiplied by 100; best is in bold and second-best is underlined. These results correspond to the ablations presented in Figure 4 . The cell-wise LLM encoder could not be run with Llama-3.1-70B: the full-depth model weights exceed the memory of our GPUs. TILE runs with that backbone, as it only loads the layers up to the extraction point.

	TabPFN-2.5		TabICLv2		TabPFN-3
Model	AUC	$R^{2}$	AUC	$R^{2}$	AUC	$R^{2}$
$-$ numerical by-pass	-0.47	-4.41	-0.50	-4.02	-0.52	-4.39
$-$ inter-column attention	-0.37	-2.70	-0.34	-3.65	-0.83	-3.87
$-$ intermediate layer (last layer)	-0.01	-1.51	-0.09	-1.50	-0.01	-1.38
$-$ numbers in the serialization	-0.14	-0.49	-0.16	-0.65	-0.15	-0.68
$-$ feature-capacity window	-0.02	-0.74	-0.01	-0.18	+0.01	-0.47

Appendix

Table H.4: Ablation study, raw scores. The ablations of Table 2 , with the raw

R^{2}

/AUC (

\times 100

) of the default architecture for each learner, and the raw difference obtained for each variation. Differences on classification are compressed here because ROC-AUC saturates near the top of its range; the normalized scores of Table 2 place the two problem types on a comparable scale.

Pipeline	Classification AUC (32)	Regression $R^{2}$ (76)
TILE + Causilo	86.4 $\pm$ 0.1	73.0 $\pm$ 0.1
TILE + TabPFN-2.5	86.0 $\pm$ 0.1	72.1 $\pm$ 0.1
TILE + TabICLv2	85.9 $\pm$ 0.1	71.4 $\pm$ 0.1
TILE + TabPFN-3	85.7 $\pm$ 0.1	71.3 $\pm$ 0.1
TF-IDF + Causilo	86.4 $\pm$ 0.1	70.7 $\pm$ 0.2
TF-IDF + TabPFN-2.5	85.9 $\pm$ 0.1	69.7 $\pm$ 0.2

Appendix

Table H.5: Raw scores on STRABLE. Unnormalized counterpart of Figure 3 , multiplied by 100: the ROC-AUC averaged over the 32 classification datasets, and the

R^{2}

(clipped at 0) averaged over the 76 regression datasets, each dataset scored by its average over 3 cross-validation folds. The

\pm

is the standard error of that mean due to the cross-validation splits. Labels in parentheses are STRABLE’s preprocessing variants of the same pipeline. Best result in bold.

Method	TabPFN-2.5				TabICLv2				TabPFN-3
String encoding	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF
Airbnb	73.78	72.38	55.48	73.77	71.20	69.86	52.16	71.66	72.17	70.80	64.30	71.24
Beer	66.99	57.30	62.73	58.71	66.33	58.02	62.68	58.78	66.19	59.08	62.91	58.45
COVID Trials	22.49	23.22	18.62	25.07	23.20	22.50	14.19	24.82	22.55	22.30	18.64	23.90
California Houses	69.63	65.18	67.11	67.24	68.08	66.22	66.64	66.53	68.26	63.67	66.49	66.60
IT Salary	48.22	46.77	39.18	46.93	45.57	44.35	0.84	33.50	47.85	38.81	42.39	44.74

Appendix

Table H.6: Regression results for TextTabBench datasets (part 1 of 3). Raw

R^{2}

scores (clipped at 0 if negative) multiplied by 100, averaged over 5 random seeds for a train size of 1,024 samples, for TabPFN-2.5, TabICLv2 and TabPFN-3, each fed with TILE, CASE, the cell-wise LLM encoder (Llama-3.1-8B) or TF-IDF. Best in bold, second best underlined, both computed over all 35 methods.

Method	TabPFN-3.5				EXAONE-Tabular				Causilo
String encoding	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF
Airbnb	72.27	70.12	66.51	73.15	74.52	73.57	64.80	74.27	74.29	73.30	64.22	74.21
Beer	66.80	64.28	64.45	61.84	67.50	59.20	64.30	60.66	67.24	59.75	64.02	59.32
COVID Trials	25.00	26.13	23.90	26.32	23.03	24.81	22.51	25.58	25.85	26.21	20.01	26.76
California Houses	68.93	66.28	66.92	68.01	69.99	68.51	68.13	68.51	70.74	68.07	68.10	68.14
IT Salary	47.98	46.43	38.28	45.50	47.63	46.54	41.60	45.44	46.68	44.38	20.59	29.52

Appendix

Table H.7: Regression results for TextTabBench datasets (part 2 of 3). Raw

R^{2}

scores (clipped at 0 if negative) multiplied by 100, averaged over 5 random seeds for a train size of 1,024 samples, for TabPFN-3.5, EXAONE-Tabular and Causilo, each fed with TILE, CASE, the cell-wise LLM encoder (Llama-3.1-8B) or TF-IDF. Best in bold, second best underlined, both computed over all 35 methods.

Method	XGBoost (tuned)		CatBoost			RealMLP (tuned)		Ridge		ContextTab	TabSTAR
String encoding	LLM encoder	TF-IDF	LLM encoder	TF-IDF	internal	LLM encoder	TF-IDF	LLM encoder	TF-IDF	internal	internal
Airbnb	60.09	71.36	48.67	72.27	71.90	65.84	71.47	62.41	68.56	72.74	69.71
Beer	59.74	56.62	60.18	56.38	56.55	61.33	56.68	53.68	48.06	62.22	49.16
COVID Trials	15.25	21.50	13.18	22.67	18.39	13.15	16.48	4.07	14.99	15.77	14.98
California Houses	61.95	62.50	59.23	60.85	62.03	62.03	62.00	53.55	58.26	65.82	64.72
IT Salary	28.56	36.56	24.09	33.31	43.51	33.82	37.86	25.08	28.13	45.77	41.59

Appendix

Table H.8: Regression results for TextTabBench datasets (part 3 of 3). Raw

R^{2}

scores (clipped at 0 if negative) multiplied by 100, averaged over 5 random seeds for a train size of 1,024 samples, for the remaining learners, fed with the cell-wise LLM encoder (Llama-3.1-8B) or TF-IDF, and the methods with internal string handling. Best in bold, second best underlined, both computed over all 35 methods.

Method	TabPFN-2.5				TabICLv2				TabPFN-3
String encoding	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF
Anime Planet	64.08	46.31	54.11	54.43	63.69	39.48	56.66	54.29	63.21	45.92	57.03	54.52
Babies R Us	81.78	76.26	53.45	79.38	82.64	71.75	68.73	79.39	82.46	77.76	70.49	79.84
Beer Ratings	61.27	51.78	57.67	53.38	60.23	52.03	57.78	54.36	59.89	52.80	56.61	52.32
BikeDekho	80.86	78.12	63.47	78.81	80.01	77.75	71.36	78.66	80.22	78.85	74.62	78.73
BikeWale	80.77	72.46	76.74	75.04	80.26	72.70	77.14	75.32	80.50	75.36	78.82	74.96

Appendix

Table H.9: Regression results for CARTE datasets (part 1 of 3). Raw

R^{2}

Method	TabPFN-3.5				EXAONE-Tabular				Causilo
String encoding	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF
Anime Planet	64.06	50.64	59.31	55.22	64.54	48.39	58.93	55.93	64.55	48.04	59.41	55.42
Babies R Us	82.90	80.86	73.11	81.67	82.48	78.95	73.90	79.73	83.73	80.35	70.64	81.19
Beer Ratings	60.54	56.35	58.86	56.52	61.62	53.08	59.23	55.36	61.52	53.37	58.87	54.57
BikeDekho	79.56	79.95	74.81	78.85	79.84	78.16	73.78	77.90	79.71	79.28	70.28	79.11
BikeWale	80.20	76.53	79.00	76.44	80.96	78.82	78.25	75.90	80.84	79.61	77.80	75.73

Appendix

Table H.10: Regression results for CARTE datasets (part 2 of 3). Raw

R^{2}

Method	XGBoost (tuned)		CatBoost			RealMLP (tuned)		Ridge		ContextTab	TabSTAR
String encoding	LLM encoder	TF-IDF	LLM encoder	TF-IDF	internal	LLM encoder	TF-IDF	LLM encoder	TF-IDF	internal	internal
Anime Planet	55.91	50.86	50.86	50.23	45.05	54.56	50.39	56.03	47.90	53.09	55.79
Babies R Us	55.41	74.83	51.23	74.65	54.22	58.82	77.73	58.43	62.45	74.28	85.33
Beer Ratings	53.63	50.72	53.82	50.51	50.76	54.48	49.88	49.15	40.39	56.77	36.91
BikeDekho	66.46	75.47	64.53	76.17	65.78	68.16	75.24	69.76	38.79	77.25	76.17
BikeWale	73.11	70.53	69.96	70.60	55.43	70.22	72.23	70.15	47.20	71.47	78.68

Appendix

Table H.11: Regression results for CARTE datasets (part 3 of 3). Raw

R^{2}

Method	TabPFN-2.5				TabICLv2				TabPFN-3
String encoding	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF
Artworks Inventory	32.92	27.13	25.01	28.71	33.64	28.18	26.49	29.11	33.16	27.76	28.13	30.01
Business Locations	86.13	77.36	78.46	72.02	82.96	66.91	71.78	61.73	85.88	78.16	77.07	70.58
Dissolved Municipalities	62.71	37.54	56.63	54.56	62.04	46.09	56.10	53.42	62.17	41.34	56.60	53.92
Drawings Catalog	80.50	80.23	80.49	80.58	80.70	79.93	80.44	80.24	80.78	80.60	79.95	80.53
Eclipsing Binary Stars	64.01	27.11	62.06	60.07	63.28	0.00	60.14	58.61	63.49	13.68	61.97	59.87

Appendix

Table H.12: Regression results for WikiDBs datasets (part 1 of 3). Raw

R^{2}

Method	TabPFN-3.5				EXAONE-Tabular				Causilo
String encoding	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF
Artworks Inventory	33.75	29.08	26.60	29.91	33.20	29.10	31.10	30.19	33.12	30.11	29.95	30.17
Business Locations	86.70	79.49	78.79	72.92	85.97	74.44	78.30	71.16	85.76	70.23	77.58	69.15
Dissolved Municipalities	63.24	46.76	58.34	55.88	63.03	54.18	58.66	55.69	64.00	53.60	57.85	55.37
Drawings Catalog	81.25	81.68	81.26	81.23	81.16	81.22	80.85	81.50	81.34	81.30	80.84	81.02
Eclipsing Binary Stars	63.61	57.36	61.82	59.37	64.13	61.45	61.96	60.47	64.14	60.42	62.34	60.62

Appendix

Table H.13: Regression results for WikiDBs datasets (part 2 of 3). Raw

R^{2}

Method	XGBoost (tuned)		CatBoost			RealMLP (tuned)		Ridge		ContextTab	TabSTAR
String encoding	LLM encoder	TF-IDF	LLM encoder	TF-IDF	internal	LLM encoder	TF-IDF	LLM encoder	TF-IDF	internal	internal
Artworks Inventory	28.24	26.32	15.49	21.32	24.82	18.26	19.18	19.36	10.86	28.66	28.23
Business Locations	74.57	70.45	73.63	69.25	69.78	75.22	65.22	72.31	51.89	72.30	66.10
Dissolved Municipalities	50.72	52.56	52.25	53.27	46.94	53.50	49.12	45.22	43.26	56.92	50.97
Drawings Catalog	78.93	78.89	67.25	78.05	78.29	73.80	76.87	69.62	65.16	78.87	75.74
Eclipsing Binary Stars	59.42	58.34	57.96	57.42	48.29	57.29	57.43	60.11	50.47	60.81	55.52

Appendix

Table H.14: Regression results for WikiDBs datasets (part 3 of 3). Raw

R^{2}

Method	TabPFN-2.5				TabICLv2				TabPFN-3
String encoding	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF
Customer Complaints	89.05	88.61	89.23	89.41	89.22	88.57	88.90	89.19	89.12	88.60	88.33	88.89
Diabetes	97.72	97.70	97.69	97.72	97.61	97.63	97.70	97.77	97.89	97.79	97.89	97.87
Job Frauds	99.12	97.82	98.69	98.73	99.10	97.32	98.99	98.76	99.04	97.43	98.59	98.65
Kickstarter	86.70	78.65	85.05	84.67	86.77	85.33	85.18	84.91	87.10	79.64	84.62	84.29
Lending Club	69.94	69.48	69.20	69.16	70.22	69.64	69.01	69.26	70.36	69.48	68.98	68.52

Appendix

Table H.15: Classification results for TextTabBench datasets (part 1 of 3). Raw ROC AUC scores multiplied by 100, averaged over 5 random seeds for a train size of 1,024 samples, for TabPFN-2.5, TabICLv2 and TabPFN-3, each fed with TILE, CASE, the cell-wise LLM encoder (Llama-3.1-8B) or TF-IDF. Best in bold, second best underlined, both computed over all 36 methods.

Method	TabPFN-3.5				EXAONE-Tabular				Causilo
String encoding	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF
Customer Complaints	89.29	89.06	88.80	89.48	89.12	89.17	89.65	89.86	89.28	88.92	89.08	89.45
Diabetes	97.90	97.90	97.89	98.01	97.93	97.82	97.87	97.92	97.92	97.85	97.87	97.90
Job Frauds	99.16	99.04	98.85	98.92	99.15	98.78	99.04	98.79	99.22	98.71	98.70	98.81
Kickstarter	87.46	86.07	86.15	85.93	86.61	86.03	86.23	85.48	87.32	85.33	85.62	85.03
Lending Club	70.65	69.68	69.96	69.31	69.76	69.71	69.43	69.43	69.89	69.74	68.99	69.27

Appendix

Table H.16: Classification results for TextTabBench datasets (part 2 of 3). Raw ROC AUC scores multiplied by 100, averaged over 5 random seeds for a train size of 1,024 samples, for TabPFN-3.5, EXAONE-Tabular and Causilo, each fed with TILE, CASE, the cell-wise LLM encoder (Llama-3.1-8B) or TF-IDF. Best in bold, second best underlined, both computed over all 36 methods.

Method	XGBoost (tuned)		CatBoost			RealMLP (tuned)		Ridge		ContextTab	TabSTAR	TabLLM
String encoding	LLM encoder	TF-IDF	LLM encoder	TF-IDF	internal	LLM encoder	TF-IDF	LLM encoder	TF-IDF	internal	internal	internal
Customer Complaints	89.16	89.37	87.83	88.08	88.43	86.99	88.00	81.59	85.20	88.46	87.67	87.40
Diabetes	97.27	97.30	97.14	97.21	97.67	96.34	96.71	95.57	96.12	97.87	97.48	97.46
Job Frauds	97.63	97.56	98.21	98.10	98.27	98.51	98.18	96.49	96.92	98.92	97.13	–
Kickstarter	82.90	82.00	82.44	81.76	84.62	80.41	79.63	76.32	73.65	83.38	73.24	80.49
Lending Club	68.47	67.74	65.81	65.91	68.63	66.43	66.33	65.49	69.31	68.38	66.74	–

Appendix

Table H.17: Classification results for TextTabBench datasets (part 3 of 3). Raw ROC AUC scores multiplied by 100, averaged over 5 random seeds for a train size of 1,024 samples, for the remaining learners, fed with the cell-wise LLM encoder (Llama-3.1-8B) or TF-IDF, and the methods with internal string handling; TabLLM has no result on two datasets (–). Best in bold, second best underlined, both computed over all 36 methods.

Method	TabPFN-2.5				TabICLv2				TabPFN-3
String encoding	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF
Chocolate Bar Ratings	82.44	78.28	80.17	80.62	82.25	78.80	80.71	80.29	81.61	78.50	78.47	79.06
Coffee Ratings	93.21	84.68	91.81	93.48	92.72	85.27	92.45	93.44	92.31	84.67	90.85	92.51
Michelin	90.89	84.69	89.07	88.20	90.86	86.42	89.20	88.01	90.73	86.05	88.45	87.70
NBA Draft	90.59	89.38	89.25	89.05	90.66	89.32	88.99	88.98	90.87	89.37	88.90	88.89
Ramen Ratings	69.63	68.15	69.57	69.05	70.40	69.45	70.33	69.15	70.12	69.03	68.26	67.76

Appendix

Table H.18: Classification results for CARTE datasets (part 1 of 3). Raw ROC AUC scores multiplied by 100, averaged over 5 random seeds for a train size of 1,024 samples, for TabPFN-2.5, TabICLv2 and TabPFN-3, each fed with TILE, CASE, the cell-wise LLM encoder (Llama-3.1-8B) or TF-IDF. Best in bold, second best underlined, both computed over all 36 methods.

Method	TabPFN-3.5				EXAONE-Tabular				Causilo
String encoding	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF
Chocolate Bar Ratings	82.85	80.17	80.98	81.58	82.62	78.44	81.06	81.26	82.73	79.06	80.67	80.60
Coffee Ratings	93.00	87.04	92.57	93.58	93.23	85.97	91.91	93.73	93.26	85.51	92.39	93.37
Michelin	90.96	87.19	88.66	88.31	90.57	87.32	89.45	88.23	91.01	87.19	88.94	88.12
NBA Draft	90.97	89.25	89.21	88.99	90.68	89.08	89.17	89.06	90.97	89.37	89.32	89.06
Ramen Ratings	70.50	70.05	70.34	70.13	69.80	69.28	70.72	69.21	70.41	69.55	70.46	69.31

Appendix

Table H.19: Classification results for CARTE datasets (part 2 of 3). Raw ROC AUC scores multiplied by 100, averaged over 5 random seeds for a train size of 1,024 samples, for TabPFN-3.5, EXAONE-Tabular and Causilo, each fed with TILE, CASE, the cell-wise LLM encoder (Llama-3.1-8B) or TF-IDF. Best in bold, second best underlined, both computed over all 36 methods.

Method	XGBoost (tuned)		CatBoost			RealMLP (tuned)		Ridge		ContextTab	TabSTAR	TabLLM
String encoding	LLM encoder	TF-IDF	LLM encoder	TF-IDF	internal	LLM encoder	TF-IDF	LLM encoder	TF-IDF	internal	internal	internal
Chocolate Bar Ratings	77.68	78.97	77.69	78.55	70.29	77.11	78.27	77.89	77.53	75.65	80.92	82.30
Coffee Ratings	90.30	91.88	90.42	92.34	74.94	90.90	92.25	89.29	92.38	82.71	93.75	95.45
Michelin	88.31	87.50	88.25	87.13	84.77	88.09	86.55	88.25	86.25	86.71	86.58	87.10
NBA Draft	88.13	88.25	86.71	85.89	88.17	87.87	87.81	86.39	88.04	88.27	88.19	85.90
Ramen Ratings	68.95	68.10	68.26	68.08	67.74	68.85	67.71	69.07	68.88	68.68	79.11	67.37

Appendix

Table H.20: Classification results for CARTE datasets (part 3 of 3). Raw ROC AUC scores multiplied by 100, averaged over 5 random seeds for a train size of 1,024 samples, for the remaining learners, fed with the cell-wise LLM encoder (Llama-3.1-8B) or TF-IDF, and the methods with internal string handling; TabLLM has no result on two datasets (–). Best in bold, second best underlined, both computed over all 36 methods.

Method	TabPFN-2.5				TabICLv2				TabPFN-3
String encoding	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF
Artists Copyright	80.00	74.83	78.61	78.53	80.45	75.09	78.60	77.12	79.64	75.16	76.22	77.36
Artworks Catalog	97.53	97.23	97.21	97.31	97.59	97.38	97.37	97.09	97.30	97.29	97.00	96.99
CC Authors	99.75	99.44	99.31	91.77	99.78	99.28	99.33	92.86	99.74	99.43	99.14	92.29
Defenders	99.48	97.80	99.01	95.23	99.47	97.74	99.04	95.75	99.42	97.76	99.02	94.83
Forward Players	89.23	87.13	88.96	88.80	89.43	87.57	89.08	88.55	89.01	87.10	88.25	88.06

Appendix

Table H.21: Classification results for WikiDBs datasets (part 1 of 3). Raw ROC AUC scores multiplied by 100, averaged over 5 random seeds for a train size of 1,024 samples, for TabPFN-2.5, TabICLv2 and TabPFN-3, each fed with TILE, CASE, the cell-wise LLM encoder (Llama-3.1-8B) or TF-IDF. Best in bold, second best underlined, both computed over all 36 methods.

Method	TabPFN-3.5				EXAONE-Tabular				Causilo
String encoding	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF	TILE	CASE	LLM encoder	TF-IDF
Artists Copyright	80.38	76.89	77.06	78.61	80.31	76.81	79.13	78.56	80.74	77.36	78.43	78.48
Artworks Catalog	97.66	97.72	97.57	97.68	97.73	97.67	97.56	97.63	97.75	97.66	97.48	97.45
CC Authors	99.78	99.54	99.38	93.28	99.72	99.48	99.28	93.04	99.77	99.46	99.31	93.02
Defenders	99.44	98.56	99.12	96.06	99.40	98.23	99.13	95.91	99.51	98.14	99.08	95.55
Forward Players	89.63	89.25	88.96	89.02	89.58	88.53	89.46	89.07	89.71	88.35	89.26	88.95

Appendix

Table H.22: Classification results for WikiDBs datasets (part 2 of 3). Raw ROC AUC scores multiplied by 100, averaged over 5 random seeds for a train size of 1,024 samples, for TabPFN-3.5, EXAONE-Tabular and Causilo, each fed with TILE, CASE, the cell-wise LLM encoder (Llama-3.1-8B) or TF-IDF. Best in bold, second best underlined, both computed over all 36 methods.

Method	XGBoost (tuned)		CatBoost			RealMLP (tuned)		Ridge		ContextTab	TabSTAR	TabLLM
String encoding	LLM encoder	TF-IDF	LLM encoder	TF-IDF	internal	LLM encoder	TF-IDF	LLM encoder	TF-IDF	internal	internal	internal
Artists Copyright	75.88	75.59	76.87	75.43	74.86	75.21	75.94	74.56	73.73	76.92	73.27	75.68
Artworks Catalog	96.38	95.91	96.94	96.57	93.54	95.90	96.35	93.31	93.92	96.82	93.45	96.25
CC Authors	98.48	87.91	97.96	87.95	83.69	98.96	85.61	98.99	80.54	97.86	98.85	99.29
Defenders	98.66	93.47	98.91	94.11	88.96	98.64	93.62	98.59	90.19	97.30	96.61	98.58
Forward Players	88.68	87.67	88.87	87.64	85.98	88.05	87.24	87.22	86.18	88.73	84.08	88.61

Appendix

Table H.23: Classification results for WikiDBs datasets (part 3 of 3). Raw ROC AUC scores multiplied by 100, averaged over 5 random seeds for a train size of 1,024 samples, for the remaining learners, fed with the cell-wise LLM encoder (Llama-3.1-8B) or TF-IDF, and the methods with internal string handling; TabLLM has no result on two datasets (–). Best in bold, second best underlined, both computed over all 36 methods.

Tabular ML

Tabular machine learning develops models and tools for learning from structured data arranged in rows and columns.

See all 113 papers

May 12, 2026

STRABLE: Benchmarking Tabular Machine Learning with Strings

Benchmarking tabular learning has revealed the benefit of dedicated architectures, pushing the state of the art. But real-world tables often contain string entries, beyond numbers, and these settings have been understudied due to a lack of a solid benchmarking suite. They lead to new research questions: Are dedicated learners needed, with end-to-end modeling of strings and numbers? Or does it suffice to encode strings as numbers, as with a categorical encoding? And if so, do the resulting tables resemble numerical tabular data, calling for the same learners? To enable these studies, we contribute STRABLE, a benchmarking corpus of 108 tables, all real-world learning problems with strings and numbers across diverse application fields. We run the first large-scale empirical study of tabular learning with strings, evaluating 445 pipelines. These pipelines span end-to-end architectures and modular pipelines, where strings are first encoded, then post-processed, and finally passed to a tabular learner. We find that, because most tables in the wild are categorical-dominant, advanced tabular learners paired with simple string embeddings achieve good predictions at low computational cost. On free-text-dominant tables, large LLM encoders become competitive. Their performance also appears sensitive to post-processing, with differences across LLM families. Finally, we show that STRABLE is a good set of tables to study "string tabular" learning as it leads to generalizable pipeline rankings that are close to the oracle rankings. We thus establish STRABLE as a foundation for research on tabular learning with strings, an important yet understudied area.

Gioia Blayer, Myung Jun Kim, Félix Lefebvre +8

SODA Team, INRIA Saclay, Palaiseau, France · Prior Labs · University of Freiburg +3

Tabular ML Tabular Prediction

Ask PDF

May 7, 2026

Data Language Models: A New Foundation Model Class for Tabular Data

Every major data modality now has a foundation model that understands it natively: text has language models, images have vision models, audio has audio models. Tabular data, the modality on which many consequential real-world AI decisions are made, does not. Every approach to tabular AI today, from gradient-boosted trees to the latest tabular foundation models, requires a preprocessing pipeline before any model can consume the data. None of them understand tabular data as a modality. We introduce the Data Language Model (DLM), the missing foundation model for tabular data. A DLM understands tables the way a language model understands sentences: natively, without serialization or preprocessing, directly from raw cell values. It is the tabular data layer on which AI models, agents, and vertical AI applications can be built, eliminating the preprocessing pipelines that currently stand between raw data and every AI system that consumes it. We present Schema-1, the first DLM: a 140M parameter model trained on more than 2.3M synthetic and real-world tabular datasets. Schema-1 outperforms gradient-boosted ensembles, AutoML stacks, and the tabular foundation models we evaluate on established row-level prediction benchmarks. On missing value reconstruction it achieves lower reconstruction error than all classical statistical methods and frontier large language models on mean performance across conditions, establishing that structural understanding of a dataset's own distributional geometry is more useful for imputation than world knowledge encoded in language. It identifies the industry sector of any unseen dataset from raw cell values alone, reliably across any domain, a task no prior tabular model can perform. It is the native tabular understanding layer that has been missing from the AI stack.

Eda Erol, Giuliano Pezzoli, Ozer Cem Kelahmet

SchemaLabs

Tabular Foundation Models Incomplete Data Imputation Tabular Prediction

Ask PDF

Apr 23, 2026

Towards Universal Tabular Embeddings: A Benchmark Across Data Tasks

Tabular foundation models aim to learn universal representations of tabular data that transfer across tasks and domains, enabling applications such as table retrieval, semantic search and table-based prediction. Despite the growing number of such models, it remains unclear which approach works best in practice, as existing methods are often evaluated under task-specific settings that make direct comparison difficult. To address this, we introduce TEmBed, the Tabular Embedding Test Bed, a unified benchmark for systematically evaluating tabular embeddings across four representation levels: cell, row, column, and table. Evaluating a diverse set of tabular representation learning models, we show that which model to use depends on the task and representation level. Our results offer practical guidance for selecting tabular embeddings in real-world applications and lay the groundwork for developing more general-purpose tabular representation models.

Liane Vogel, Kavitha Srinivas, Niharika D'Souza +3

Technical University of Darmstadt · IBM Research, USA

Tabular Foundation Models Representation Learning Tabular Representation Learning

Ask PDF

Abstract

Tabular ML

Explore similar work

STRABLE: Benchmarking Tabular Machine Learning with Strings

Data Language Models: A New Foundation Model Class for Tabular Data

Towards Universal Tabular Embeddings: A Benchmark Across Data Tasks