Generalized Context in Cross Attention for Transfer Learning of Disjoint Tabular Data

Kazi F. Akhter; Ibna Kowsar; Manar D. Samad

cs.LGAug 28, 2026

Generalized Context in Cross Attention for Transfer Learning of Disjoint Tabular Data

Authors: Kazi F. Akhter, Ibna Kowsar, Manar D. Samad

Organizations: Department of Computer Science Tennessee State University Nashville, TN, USA · Department of Computer Science North Carolina Agricultural and Technical State University Greensboro, NC, USA

Abstract

Unlike images and text, applying transfer learning to tabular data is challenging due to heterogeneity in feature types, structures, and semantics across disparate domains. Existing methods assume shared features across data tables to enable knowledge transfer between domains, which is unrealistic in practice. \mds{This paper introduces generalized context learning to remove the requirement of shared features across domains. The generalized context captured by transformer projection weights for $key$ , $value$ , and $query$ provides rule-based generalization rather than the domain-specific context conventionally learned from transformer activations. Projection weights for $key$ from the source domain interact with the weight for $query$ in the target domain to achieve Cross-domain Attention Transfer Learning (CATTLE) in a data-agnostic manner. Our experiments on ten pairs of disjoint source-target data sets show that CATTLE can learn generalized context from a single source data set and is rank-wise and statistically superior to nine state-of-the-art baselines, including machine learning, deep learning, and transfer learning methods using large-scale pre-trained models. CATTLE achieves the best average rank (2.9) and delivers a 3.7% average AUROC gain over the baseline methods.} The CATTLE source code is available at https://tinyurl.com/pr5s8ywn.

Figures & tables

Figure 1: Overview of the CATTLE framework. Attention-related weights from a pretrained Gated Feature Tokenizer Transformer (gFTT) trained on source data are transferred into a newly initialized gFTT, which is then finetuned using target data. Emb. = Embedding, Cat. = Categorical, and Num. = Numerical features.

Type	OpenML ID	Data set	Domain	Samples	Features	Numeric	Categorical	Classes
Source Data sets	14	mfeat-fourier (MF)	Shape Measurements	2000	76	76	0	10
	31	credit-g (CG)	Banking	1000	20	7	13	2
	38	sick (SK)	Thyroid Disease	3772	29	7	22	2
	980	optdigits (DG)	Optical Digits	5620	64	64	0	2
	1504	steel-plates-fault (SP)	Manufacturing	1941	33	33	0	2
	40664	car-evaluation (CE)	Car Pricing	1728	21	0	21	4

Table 1: Summary of source and target data sets for transfer learning.

Panel of Figure 2 — Figure 2: Transfer learning loss curves: (a) self‐supervised pretraining, (b) supervised pretraining, (c) finetuning a self-supervised pretrained model using a target data set, and (d) finetuning a supervised pretrained model using a target data set.

Parameter	Distribution
XGBoost [ 13 ]
max_depth	UniformInt[1, 10]
learning_rate	LogUniform[ $\exp(-7)$ , 1]
n_estimators	UniformInt[100, 4000]
subsample	UniformFloat[0.2, 1.0]
colsample_bytree	UniformFloat[0.2, 1.0]

Table 2: Hyperparameter search space for all baselines and proposed methods.

	Logistic Regression	XGBoost	MLP	ResNet	FT- Transformer	TabNet	XTab		CM2		TransTab		CATTLE Supervised	CATTLE Self-Supervised
Target							Source		Source		Source
DB	0.822 (0.04)	0.810 (0.04)	0.798 (0.03)	0.818 (0.03)	0.788 (0.04)	0.643 (0.03)	52 AutoML Data sets	0.834 (0.02)	Open Tabs 2000 Data sets	0.803 (0.03)	CD	0.799 (0.04)	0.803 (0.04)	0.807 (0.04)
											CG	0.818 (0.04)	0.808 (0.04)	0.819 (0.04)
VH	0.935 (0.01)	0.925 (0.01)	0.917 (0.01)	0.855 (0.01)	0.914 (0.01)	0.794 (0.10)		0.935 (0.01)		0.893 (0.01)	MF	0.928 (0.01)	0.940 (0.01)	0.933 (0.01)
											DG	0.935 (0.01)	0.932 (0.01)	0.942 (0.01)
CM	0.703 (0.02)	0.727 (0.02)	0.704 (0.02)	0.695 (0.03)	0.724 (0.03)	0.681 (0.08)		0.721 (0.02)		0.729 (0.02)	CH	0.726 (0.01)	0.721 (0.03)	0.729 (0.02)

Table 3: Average area under the ROC curve (AUROC) and corresponding performance rank. Two-letter acronyms of the source and target data sets are taken from Table 1 . Traditional deep and machine learning methods for tabular data are trained and tested directly on target data sets without pretraining on the source.

	Logistic Regression	XGBoost	MLP	ResNet	FT- Transformer	TabNet	XTab		CM2		TransTab		CATTLE Supervised	CATTLE Self-Supervised
Target							Source		Source		Source
DB	0.747 (0.04)	0.729 (0.04)	0.753 (0.03)	0.772 (0.03)	0.754 (0.04)	0.625 (0.05)	52 AutoML Data sets	0.764 (0.02)	Open Tabs 2000 Data sets	0.734 (0.02)	CD	0.734 (0.04)	0.735 (0.02)	0.740 (0.04)
											CG	0.735 (0.04)	0.738 (0.05)	0.746 (0.05)
VH	0.736 (0.03)	0.753 (0.03)	0.754 (0.03)	0.721 (0.04)	0.767 (0.06)	0.574 (0.01)		0.753 (0.02)		0.664 (0.02)	MF	0.776 (0.03)	0.774 (0.02)	0.786 (0.02)
											DG	0.766 (0.03)	0.758 (0.02)	0.773 (0.02)
CM	0.512 (0.04)	0.551 (0.03)	0.526 (0.02)	0.536 (0.04)	0.583 (0.06)	0.766 (0.01)		0.653 (0.03)		0.552 (0.03)	CH	0.510 (0.02)	0.534 (0.04)	0.640 (0.03)

Table 4: Comparison of CATTLE with other baseline methods based on average classification accuracy. Two-letter acronyms of the source and target data sets are taken from Table 1 . Traditional deep and machine learning methods for tabular data are trained and tested directly on target data sets without pretraining on the source.

Figure 3: Win matrix based on Wilcoxon signed-rank statistical tests. The cell value X/Y presents X as the number of times the row method statistically outperforms the column method. Y is the number of source–target data set pairs with statistically significant differences between a pair of methods.

	Bottom Layers $\rightarrow$ Bottom Layers			Top Layers $\rightarrow$ Bottom Layers
	gFTT s ( $W^{0}$ ) $\rightarrow$ gFTT t ( $W^{0}$ )	gFTT s ( $W^{0,1}$ ) $\rightarrow$ gFTT t ( $W^{0,1}$ )	gFTT s ( $W^{0}$ ) $\rightarrow$ gFTT t ( $W^{0,1}$ )	gFTT s ( $W^{L}$ ) $\rightarrow$ gFTT t ( $W^{0}$ )	gFTT s ( $W^{L},W^{L-1}$ ) $\rightarrow$ gFTT t ( $W^{0,1}$ )	gFTT s ( $W^{L}$ ) $\rightarrow$ gFTT t ( $W^{0,1}$ ) [Proposed]
Cardiovascular $\rightarrow$ diabetes	0.818 (0.06)	0.816 (0.06)	0.813 (0.06)	0.810 (0.05)	0.818 (0.05)	0.819 (0.05)
credit-g $\rightarrow$ diabetes	0.821 (0.06)	0.819 (0.06)	0.816 (0.06)	0.814 (0.06)	0.814 (0.05)	0.815 (0.06)
mfeat-fourier $\rightarrow$ vehicle	0.928 (0.01)	0.927 (0.01)	0.926 (0.01)	0.928 (0.02)	0.927 (0.01)	0.927 (0.01)
optdigits $\rightarrow$ vehicle	0.933 (0.01)	0.930 (0.01)	0.925 (0.01)	0.931 (0.02)	0.925 (0.01)	0.942 (0.01)
churn $\rightarrow$ cmc	0.745 (0.05)	0.751 (0.05)	0.747 (0.05)	0.746 (0.03)	0.747 (0.05)	0.749 (0.05)

Table 5: Effects of transferring attention layers from the source pretrained transformer (gFTT) to the downstream transformer. Attention layers are numbered from bottom (0) to top (L), as {0, 1, …, L-1, L} for the projection weight (W) matrices.

Figure 4: Effects of individual source data sets on the classification of the (downstream) cmc target data set. A source data set is used for self-supervised pretraining of the upstream model.

Source $\rightarrow$ Target	Anonymized	Word2Vec	Pre-trained BERT
Cardiovascular-Disease-dataset $\rightarrow$ diabetes	0.714 (0.06)	0.707 (0.05)	0.799 (0.04)
credit-g $\rightarrow$ diabetes	0.685 (0.04)	0.694 (0.05)	0.818 (0.04)
mfeat-fourier $\rightarrow$ vehicle	0.435 (0.06)	0.441 (0.03)	0.928 (0.01)
optdigits $\rightarrow$ vehicle	0.418 (0.05)	0.432 (0.03)	0.935 (0.01)
churn $\rightarrow$ cmc	0.582 (0.05)	0.610 (0.04)	0.726 (0.01)
sick $\rightarrow$ cmc	0.569 (0.06)	0.601 (0.04)	0.733 (0.02)

Table 6: Ablation study on categorical feature encoding strategies. The anonymized variant removes semantic information from feature names before learning their embeddings from scratch. The Word2Vec-style variant uses trainable token embeddings for feature names and categorical values. Embeddings for categorical feature names and values are extracted from pre-trained BERT [ 37 ] .

Source $\rightarrow$ Target	Random	Trained	Direct	Proposed
	weights	weights	finetune	CATTLE
Cardiovascular-Disease-dataset $\rightarrow$ diabetes	0.812 (0.06)	0.810 (0.06)	0.799 (0.04)	0.819 (0.05)
credit-g $\rightarrow$ diabetes	0.811 (0.06)	0.815 (0.06)	0.818 (0.04)	0.815 (0.06)
mfeat-fourier $\rightarrow$ vehicle	0.901 (0.01)	0.912 (0.01)	0.928 (0.01)	0.927 (0.01)
optdigits $\rightarrow$ vehicle	0.932 (0.01)	0.927 (0.01)	0.935 (0.01)	0.942 (0.01)
churn $\rightarrow$ cmc	0.745 (0.04)	0.747 (0.05)	0.726 (0.01)	0.749 (0.05)

Table 7: Ablation study on cross-attention weight transfer mechanisms. The

random

weights

variant uses randomly selected weights from the pretrained model to remain frozen in the downstream. The

trained

weights

variant updates the proposed frozen pre-trained

key

value

projection weights in the downstream. The

direct

finetune

variant directly finetunes the upstream pre-trained model on the target data without proposed weight transfer. Proposed CATTLE transfers selective pretrained key-value projection weights and keeps them frozen while finetuning the downstream model using target data.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Data set pairs	Top 4 Layers	Top 3 Layers	Top 2 Layers	Top 1 Layer
CD $\rightarrow$ DB	0.818(0.051)	0.814(0.055)	0.810(0.060)	0.821(0.057)
CE $\rightarrow$ PC1	0.787(0.071)	0.810(0.051)	0.833(0.047)	0.831(0.035)
CH $\rightarrow$ CM	0.735(0.046)	0.732(0.039)	0.741(0.043)	0.743(0.039)
CG $\rightarrow$ DB	0.803(0.063)	0.808(0.059)	0.806(0.063)	0.804(0.061)
MF $\rightarrow$ VH	0.932(0.014)	0.938(0.014)	0.929(0.015)	0.927(0.012)
DG $\rightarrow$ VH	0.928(0.010)	0.929(0.010)	0.926(0.010)	0.923(0.012)

Appendix

Table 8: Effects of the number of active attention layers of the pretrained gFTT model (supervised) on downstream target data classification (AUROC scores).

Target	TARTE	TabPFNv2	Source	CATTLE Supervised	CATTLE Self-Supervised
DB	0.815 (0.04)	0.816 (0.03)	CD	0.803 (0.05)	0.807 (0.05)
DB	0.815 (0.04)	0.816 (0.03)	CG	0.808 (0.04)	0.819 (0.04)
VH	0.941 (0.01)	0.966 (0.01)	MF	0.940 (0.01)	0.936 (0.00)
VH	0.941 (0.01)	0.966 (0.01)	DG	0.932 (0.01)	0.942 (0.01)
CM	0.737 (0.02)	0.756 (0.02)	CH	0.721 (0.03)	0.729 (0.02)
CM	0.737 (0.02)	0.756 (0.02)	SK	0.726 (0.03)	0.738 (0.02)

Appendix

Table 9: Comparison of CATTLE (supervised and self-supervised) with TARTE and TabPFNv2 across multiple source–target tasks.

Cross-Domain Transfer Learning

Cross-domain transfer learning reuses knowledge across datasets, populations, devices, or tasks when their distributions and feature spaces differ.

See all 205 papers

Context-Constrained Transfer Learning for Tabular Foundation Models via Data Distillation
Jul 6, 2026Yijun Lin, Sai LiTabular Foundation Models Dataset Distillation
Renmin University of China · Tsinghua University
CDMD: A Cross-Dataset Mixed-Type Diffusion Model for Tabular Data
Sep 30, 2026Mohamed Amine Ketata, Maximilian Schambach, Stephan GünnemannMasked Diffusion Models Tabular Diffusion Models
Munich Data Science Institute, Technical University of Munich, Germany · SAP SE, Germany
Reason Analogically via Cross-domain Prior Knowledge: An Empirical Study of Cross-domain Knowledge Transfer for In-Context Learning
Apr 7, 2026Le Liu, Zhiming Li, Jianzhi Yan +7In-Context Learning Cross-Domain Transfer Learning
Harbin Institute of Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China

Jul 6, 2026J/K move · Enter open · S save

Context-Constrained Transfer Learning for Tabular Foundation Models via Data Distillation

Yijun Lin, Sai Li

Renmin University of China · Tsinghua University

Tabular Foundation Models Dataset Distillation Transfer Learning Tabular ICL

Open paper AskPDF arXiv

Tabular Foundation Models (TFMs) have demonstrated strong empirical performance as black-box inference engines through in-context learning. However, their use in transfer learning is limited by two obstacles: strict context-size constraints and sensitivity to distribution shifts between source and target tasks. Directly pooling heterogeneous source data can therefore lead to negative transfer. To address these challenges, we propose Context-Constrained Transfer Learning via ANchoring and DIstillation (TL-ANDI), a posterior-aware distillation framework for TFMs. TL-ANDI constructs a compact source context by solving a budget-constrained optimal transport problem whose cost jointly measures target covariate coverage and posterior compatibility. The selected anchor samples are then equipped with locally distilled labels and combined with a residual calibration step using target data.

Abstract

Cross-Domain Transfer Learning

Explore similar work

Context-Constrained Transfer Learning for Tabular Foundation Models via Data Distillation

CDMD: A Cross-Dataset Mixed-Type Diffusion Model for Tabular Data

Reason Analogically via Cross-domain Prior Knowledge: An Empirical Study of Cross-domain Knowledge Transfer for In-Context Learning