Embeddings are a powerful way to enrich data-driven machine learning models with the world knowledge of large language models (LLMs). Yet, there is limited evidence on how to design effective LLM-based embedding pipelines for tabular prediction. In this work, we systematically benchmark 256 pipeline configurations, covering 8 preprocessing strategies, 16 embedding models, and 2 downstream models. Our results show that it strongly depends on the specific pipeline design whether incorporating the prior knowledge of LLMs improves the predictive performance. In general, concatenating embeddings tends to outperform replacing the original columns with embeddings. Larger embedding models tend to yield better results, while public leaderboard rankings and model popularity are poor performance indicators. Finally, gradient boosting decision trees tend to be strong downstream models. Our findings provide researchers and practitioners with guidance for building more effective embedding pipelines for tabular prediction tasks.
Figures & tables
Abbildung 1 : Overview of our work. We serialize tabular rows into text and use embedding models to convert them into numerical representations. These embeddings are preprocessed and serve as inputs for downstream models. We systematically evaluate 256 pipeline configurations, i.e. , 16 embedding models, 8 preprocessing strategies, and 2 downstream models.
Embedding model
Number of para-meters
Embed-ding di-mension
Down-loads August 2025
Release date
all-MiniLM-L6-v2
22M
384
89.63M
2021
bge-base-en-v1.5
109M
768
3.32M
2023
bge-large-en-v1.5
335M
1024
3.63M
2023
bge-small-en-v1.5
33M
384
4.35M
2023
e5-base-v2
109M
768
0.88M
2022
e5-large-v2
335M
1024
0.91M
2022
Tabelle 1 : Embedding model characteristics.
Abbildung 2 : Top-20 mean embedding pipeline performance. Baselines are shown in light blue.
Abbildung 3 : Correlation matrix heatmap between the predictive performance and embedding model attributes, based on the Spearman rank correlation.
Embedding pipeline
AUC
F1
BA
no PCA, GBDT, conc 2, bge-base-en-v1.5
0.77
0.76
0.75
GBDT, conc 1, RTE
0.72
0.71
0.71
PCA, GBDT, conc 3, gte-base
0.72
0.71
0.71
GBDT
0.72
0.70
0.70
PCA, GBDT, conc 3, gte-small
0.72
0.70
0.70
PCA, GBDT, conc 3, e5-small-v2
0.71
0.70
0.70
Tabelle 2 : Top-20 mean embedding pipeline performance.
Abbildung 4 : Mean AUC gains by concatenating text embeddings instead of replacing the original columns with text embeddings.
Abbildung 5 : Mean AUC gains by applying dimensionality reduction via PCA to text embeddings.
Abbildung 6 : Mean embedding model performance for LR without PCA.
Abbildung 7 : Mean embedding model performance for GBDTs without PCA.
Abbildung 8 : Mean embedding model performance for LR with PCA.
Abbildung 9 : Mean embedding model performance for GBDTs with PCA.
Foundation models have established unified representations for natural language processing, yet this paradigm remains largely unexplored for tabular data. Existing methods face fundamental limitations: LLM-based approaches lack retrieval-compatible vector outputs, whereas text embedding models often fail to capture tabular structure and numerical semantics. To bridge this gap, we first introduce the Tabular Embedding Benchmark (TabBench), a comprehensive suite designed to evaluate the tabular understanding capability of embedding models. We then propose TabEmbed, the first generalist embedding model that unifies tabular classification and retrieval within a shared embedding space. By reformulating diverse tabular tasks as semantic matching problems, TabEmbed leverages large-scale contrastive learning with positive-aware hard negative mining to discern fine-grained structural and numerical nuances. Experimental results on TabBench demonstrate that TabEmbed significantly outperforms state-of-the-art text embedding models, establishing a new baseline for universal tabular representation learning. Code and datasets are publicly available at https://github.com/qiangminjie27/TabEmbed and https://huggingface.co/datasets/qiangminjie27/TabBench.
Minjie Qiang, Mingming Zhang, Xiaoyi Bao +5
Natural Language Processing Lab, Soochow University, Suzhou, China · Ant Group, Hangzhou, China · The Hong Kong Polytechnic University, Hong Kong, China
Tabular foundation models aim to learn universal representations of tabular data that transfer across tasks and domains, enabling applications such as table retrieval, semantic search and table-based prediction. Despite the growing number of such models, it remains unclear which approach works best in practice, as existing methods are often evaluated under task-specific settings that make direct comparison difficult. To address this, we introduce TEmBed, the Tabular Embedding Test Bed, a unified benchmark for systematically evaluating tabular embeddings across four representation levels: cell, row, column, and table. Evaluating a diverse set of tabular representation learning models, we show that which model to use depends on the task and representation level. Our results offer practical guidance for selecting tabular embeddings in real-world applications and lay the groundwork for developing more general-purpose tabular representation models.
Tabular data is the dominant structured-data modality, and learning table representations has become a core research direction. Table-level embeddings in particular underpin a wide range of applications, including table retrieval, data lake discovery, and table classification. Despite their importance, there is still limited understanding of how different embedding approaches behave across tasks, making systematic evaluation and analysis essential. In this work, we introduce a systematic evaluation of table-level embeddings that captures several complementary properties required for downstream effectiveness. We realize this evaluation by extending TEmBed, a recently proposed testbed for tabular embeddings, whose table-level coverage is currently limited to a single retrieval task. An empirical study over the TEmBed model pool confirms that no single model excels across all tasks, demonstrating that table-level embedding quality cannot be reduced to retrieval alone.
Ayeen Poostforoushan, Liane Vogel, Carsten Binnig
Independent Researcher · Technical University of Darmstadt · Technical University of Darmstadt & DFKI & hessian.AI