LightTAR: Light and Fast Target-Aware Representations for Multimodal Tabular Learning
Organizations: Technion – Israel Institute of Technology
Abstract
Multimodal tabular learning involves classification and regression on tables combining structured numerical and categorical features with unstructured text and images. These unstructured fields are commonly represented by pretrained encoder embeddings reduced with principal component analysis (PCA). This reduction prioritizes input variance but may overlook information relevant to predicting the target. We introduce LightTAR, a simple, inexpensive method that adapts dimensionality reduction to the prediction task. LightTAR augments the standard PCA representation with a few target-aware features extracted jointly from the final and an intermediate encoder layer using regularized partial least squares (PLS). We evaluate LightTAR with 21 tabular models, including tree ensembles, neural networks, and tabular foundation models. LightTAR improves average predictive performance for all models and, averaged across models, improves prediction on almost every MulTaBench dataset. These gains come with a small increase in total runtime relative to PCA representations. LightTAR integrates directly into existing encoding pipelines, improving tabular learners' predictive performance without modifying the encoder or the learner.
Figures & tables
| Regression | Classification | ||||
|---|---|---|---|---|---|
| Benchmark | Modality | Gain [95% CI] | W/T/L | Gain [95% CI] | W/T/L |
| MulTaBench | Text | ||||
| MulTaBench | Image | ||||
| STRABLE (ungated) | Text | ||||
| STRABLE (with gate) | Text | ||||
| Representation | Time (s) | LightTAR |
|---|---|---|
| Frozen (encoder + PCA) | ||
| LightTAR | ||
| Fine-Tuning with LoRA |
| MulTaBench Text | MulTaBench Image | |||
|---|---|---|---|---|
| Method | Classification | Regression | Classification | Regression |
| LightTAR: shuffled projection labels, two layers | ||||
| Joint PCA: two layers, 35 components | ||||
| LightTAR: final-layer directions only | ||||
| LightTAR without PCA anchor (5 components) | ||||
| PCA + PLS, last layer only | ||||
| Classification (AUC) | Regression ( ) | |||||
|---|---|---|---|---|---|---|
| Encoder | Frozen | LightTAR | Gain | Frozen | LightTAR | Gain |
| e5-small-v2 | ||||||
| e5-large-v2 | ||||||
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Supervised features | ||||
|---|---|---|---|---|
| 0.0060 (19) | 0.0072 (19) | 0.0056 (18) | 0.0072 (17) | |
| 0.0059 (19) | 0.0089 (18) | 0.0094 (18) | 0.0100 (18) | |
| 0.0080 (20) | 0.0081 (20) | 0.0094 (20) | 0.0073 (20) |
| Family | Learners |
|---|---|
| Tree ensembles | CatBoost, LightGBM, Random Forest, XGBoost |
| Neural networks | RealMLP, TabM |
| Foundation models | Causilo, EXAONE, Kumo Small, Medium and Large, TabDPT 1.1 and 1.3, TabFM, TabICLv2, TabLDM, TabPFN v2, 2.5, v3, 3.5 and 3.5 Fast |
| Learner | Fake Job Posting (AUC) | Jigsaw Toxicity (AUC) | Kickstarter Funding (AUC) | |||
|---|---|---|---|---|---|---|
| Frozen | LightTAR | Frozen | LightTAR | Frozen | LightTAR | |
| CatBoost | ||||||
| LightGBM | ||||||
| XGBoost | ||||||
| Random Forest | ||||||
| TabFM | ||||||
| Learner | Spotify Genres (AUC) | US Accidents (AUC) | Wine Review (AUC) | |||
|---|---|---|---|---|---|---|
| Frozen | LightTAR | Frozen | LightTAR | Frozen | LightTAR | |
| CatBoost | ||||||
| LightGBM | ||||||
| XGBoost | ||||||
| Random Forest | ||||||
| TabFM | – | – | – | – | ||
| Learner | Book Readability ( ) | Mercari Marketplace ( ) | Montgomery Salaries ( ) | |||
|---|---|---|---|---|---|---|
| Frozen | LightTAR | Frozen | LightTAR | Frozen | LightTAR | |
| CatBoost | ||||||
| LightGBM | ||||||
| XGBoost | ||||||
| Random Forest | ||||||
| TabFM | ||||||
| Learner | Video Games Sales ( ) | Zomato Restaurants ( ) | ||
|---|---|---|---|---|
| Frozen | LightTAR | Frozen | LightTAR | |
| CatBoost | ||||
| LightGBM | ||||
| XGBoost | ||||
| Random Forest | ||||
| TabFM | ||||
| Learner | CBIS-DDSM (AUC) | CheXpert (AUC) | CSGO Skin Price (AUC) | |||
|---|---|---|---|---|---|---|
| Frozen | LightTAR | Frozen | LightTAR | Frozen | LightTAR | |
| CatBoost | ||||||
| LightGBM | ||||||
| XGBoost | ||||||
| Random Forest | ||||||
| TabFM | ||||||
| Learner | Justin Instagram (AUC) | PetFinder (AUC) | ZooScan Zooplankton (AUC) | |||
|---|---|---|---|---|---|---|
| Frozen | LightTAR | Frozen | LightTAR | Frozen | LightTAR | |
| CatBoost | ||||||
| LightGBM | ||||||
| XGBoost | ||||||
| Random Forest | ||||||
| TabFM | ||||||
| Learner | Khaadi Clothes ( ) | Letterboxd Movies ( ) | Mango Mass ( ) | |||
|---|---|---|---|---|---|---|
| Frozen | LightTAR | Frozen | LightTAR | Frozen | LightTAR | |
| CatBoost | ||||||
| LightGBM | ||||||
| XGBoost | ||||||
| Random Forest | ||||||
| TabFM | ||||||
Tabular Representation Learning
Tabular representation learning builds useful embeddings of structured rows, columns, and tables for prediction, retrieval, clustering, and generation.
See all 59 papers