Parameter-Efficient Distributionally Robust Adaptation of Tabular Foundation Models under Subpopulation Shift

Seonghwi Kim; Sung Ho Jo; Minwoo Chae

cs.LGOct 1, 2026

Parameter-Efficient Distributionally Robust Adaptation of Tabular Foundation Models under Subpopulation Shift

Authors: Seonghwi Kim, Sung Ho Jo, Minwoo Chae

Organizations: Pohang University of Science and Technology

Abstract

Despite strong mean accuracy, tabular foundation models (TFMs) can perform poorly on underrepresented groups under subpopulation shift, where group proportions change between training and deployment. We propose DR-TFM, a parameter-efficient distributionally robust adaptation framework that requires no true group annotations. DR-TFM adjusts attention to labeled context examples by fine-tuning an existing query scaling network or adding and training one, while keeping all other parameters fixed. We instantiate the framework with two robust objectives using estimated groups or source conditional distributions derived from training data. For TabPFN-3, adaptation updates only 0.016% of the pretrained model's parameters. Across five tabular benchmarks, DR-TFM achieves substantially higher average worst-group accuracy than pretrained TFMs and the compared robust baselines without true group annotations, while maintaining competitive mean group accuracy. DR-TFM also improves average worst-group accuracy on ACS Income and across four additional TFMs.

Figures & tables

Figure 1 : Classification results under subpopulation shift with balanced test groups. TabPFN’s worst-group accuracy deteriorates more sharply than its mean group accuracy, whereas DR-TFM (GR) mitigates this degradation without true group labels. Curves show averages over 50 runs.

Figure 2 : Query scaling for classification during adaptation. A query is predicted from the labeled training context. The scaling function $s_{\theta_{L}}$ adjusts the attention weights used to aggregate context labels. Only $\theta_{L}$ is updated; $\theta_{F}$ remains fixed.

Figure 3 : Histograms of the sum of attention weights assigned to each test sample’s own true group, for majority (left) and minority (right) test samples. Gray and red denote TabPFN and DR-TFM (GR), respectively; dashed lines mark medians. For adaptation, DR-TFM estimates groups using a Gaussian mixture model (GMM).

Methods	Adult		Bank		Default		Shoppers		Taxi		Avg.
Methods	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst
Standard methods
ERM-MLP	73.45	45.29	71.86	42.03	63.70	28.52	76.44	50.11	75.27	57.37	72.14	44.67
XGBoost	77.09	52.58	68.69	33.87	64.78	30.72	78.03	55.42	75.81	57.57	72.88	46.03
CatBoost	76.50	51.33	68.94	34.43	64.75	31.03	77.50	54.24	76.04	58.71	72.75	45.95
TFMs

Table 1 : Classification results on five tabular benchmarks for methods without true group labels. Throughout the tables, Avg. denotes the average of each metric across the datasets or settings in its block. Bold and underlined values indicate the best and second-best results in each column. † : accuracy results from Tong et al. (2025) .

Methods	AZ		MA		MI		Avg.		AZ $\rightarrow$ MA		MA $\rightarrow$ MI		MI $\rightarrow$ AZ		Avg.
Methods	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst
Standard methods
ERM-MLP	77.20	59.42	80.82	73.82	75.69	57.99	77.90	63.74	76.82	60.23	79.13	71.34	75.56	56.31	77.17	62.63
XGBoost	77.64	58.99	82.26	76.38	77.50	62.77	79.13	66.05	77.96	61.99	79.67	72.01	76.82	58.99	78.15	64.33
CatBoost	77.35	58.26	81.97	75.59	77.27	62.43	78.86	65.43	77.66	61.15	79.60	72.31	76.68	58.71	77.98	64.06
TFMs

Table 2 : Classification results on ACS Income for methods without true group labels: within-state prediction (left) and transfers between states (right).

	GEORGE	LSR	MixturePFN	BETA	TabPFN	TabPFN (ft)	DR-TFM (GR)	DR-TFM (MS)
Seconds	77.0	5,156.4	229.5	1,020.6	5.4	214.1	23.8	30.6
Memory (GB)	1.21	3.12	2.17	14.33	0.61	9.25	1.07	0.98

Table 3 : Runtime in seconds and peak GPU memory in GB per result, each averaged over the five tabular benchmarks and three seeds. “ft” denotes fine-tuning. Runtime includes hyperparameter selection.

Fine-tuned parameters	Trainable (%)	ERM (%)	DRO (%)	Seconds	Memory (GB)
Input adapter	0.063–0.155	45.45	66.64 (+21.19)	43.5	9.80
Encoder	99.196	48.39	68.92 (+20.53)	241.2	10.22
Full backbone	$\approx 100$	48.00	68.98 (+20.98)	246.6	10.31
Decoder	0.804	48.68	71.93 (+23.25)	24.0	1.10
Query scaling	0.016	47.34	72.57 (+25.23)	23.8	1.07

Table 4 : Average of worst-group accuracies over five tabular benchmarks and three seeds. Trainable (%) is the percentage of model parameters updated. Encoder refers to all model parameters outside the decoder. Parenthesized values denote improvements over ERM in percentage points. Seconds and Memory (GB) are runtime and peak GPU memory per result under DRO-based adaptation.

Method	Five tabular benchmarks (Avg.)		ACS within-state (Avg.)		ACS transfer (Avg.)
Method	Mean	Worst	Mean	Worst	Mean	Worst
TabPFN	73.75	47.94	79.64	66.80	78.15	64.17
+ Ours (0.016%)	79.31 (+5.56)	72.57 (+24.63)	79.21 (-0.43)	71.06 (+4.26)	78.54 (+0.39)	68.32 (+4.15)
TabPFN-3.5	74.39	49.26	79.79	67.10	78.25	64.34
+ Ours (0.004%)	79.42 (+5.03)	71.47 (+22.21)	79.88 (+0.09)	72.48 (+5.38)	79.19 (+0.94)	71.51 (+7.17)
EXAONE	73.72	47.68	79.52	66.16	78.27	64.19

Table 5 : GR adaptation across tabular foundation models without true group labels. Results are averaged over three seeds. “+ Ours” denotes DR-TFM (GR) applied to the model above, with parentheses indicating the percentage of model parameters updated. Bold parentheses report the change from the corresponding original model, in percentage points.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

L_{k}(\theta^{(t)})\leftarrow\frac{1}{n_{k}}\sum_{i\in\widehat{\mathcal{I}}_{k}}\ell\!\left(f_{\theta^{(t)}}(x_{i};\mathcal{D}^{(c)}),y_{i}\right)

Appendix

Algorithm 1 DR-TFM (GR) for ( 11 )

z^{\prime}_{i}\leftarrow\Pi_{\mathbb{B}_{i}}\Bigl(z^{\prime}_{i}+\eta_{z}\widehat{\nabla}_{z^{\prime}_{i}}\,\ell\bigl(f_{\theta^{(t)},Z}(z^{\prime}_{i}),y^{\circ}(\beta^{(t)},x_{i})\bigr)\Bigr)

Appendix

Algorithm 2 DR-TFM (MS)

Figure 4 : Schematic attention distributions under smaller and larger positive uniform query scales.

Figure 5 : Regression architecture. Only the first linear layer of the output network is updated; the pretrained TabPFN feature map and the second linear layer remain fixed. Here $\mu_{b}$ denotes the conditional mean of response bucket $b$ .

Figure 6 : Regression results as the proportion of majority-group samples in training increases. Panels show test MSE (left) and MAE (right). Solid and dashed curves denote the mean and maximum group errors, respectively.

Figure 7 : Illustration of the four-group simulation design. Colors identify the four $(A,Y)$ groups. Their probabilities are $45\%$ , $5\%$ , $5\%$ , and $45\%$ in training (left), and $25\%$ each at test time (right). The dashed line shows the Bayes-optimal boundary $x_{2}=0$ for the balanced test distribution, displayed in both panels for reference.

Dataset	Train	Validation	Test	Total
Adult	26,048	6,513	16,281	48,842
Bank	32,551	8,138	4,522	45,211
Default	21,600	5,400	3,000	30,000
Shoppers	8,877	2,220	1,233	12,330
Taxi	9,139	2,285	1,270	12,694

Appendix

Table 6 : The five tabular benchmarks and their splits.

Setting	Train	Validation	Test	Total
AZ	23,959	5,990	3,328	33,277
MA	28,881	7,221	4,012	40,114
MI	36,005	9,002	5,001	50,008
AZ $\rightarrow$ MA	26,621	6,656	40,114	73,391
MA $\rightarrow$ MI	32,091	8,023	50,008	90,122
MI $\rightarrow$ AZ	40,006	10,002	33,277	83,285

Appendix

Table 7 : ACS Income settings. The first three train and test on the same state; the last three transfer to a held-out state.

Method	Training	Epochs / steps	Selection criterion	Learning rate schedule
ERM-MLP	ERM, from scratch	1000	validation accuracy	multiply by $0.9$ after $20$ epochs without improvement in mean training loss
GroupDRO	GroupDRO over the true groups, from scratch	1000	validation worst-group accuracy (true groups)	same as above
GEORGE stage 1	ERM, from scratch (feature extractor)	1000	validation accuracy	none
GEORGE stage 2	GroupDRO over the estimated clusters, newly initialized model	300	worst-group accuracy over the estimated validation clusters	none
BPA	base ERM, then retraining with cluster reweighting	1000 / 300	worst-group accuracy over the estimated validation clusters	none
SPARE	base ERM, then retraining with importance sampling	1000 / 300	worst-group accuracy over the estimated validation clusters	none

Appendix

Table 8 : Training budgets and model selection criteria for DR-TFM and the MLP-based baselines.

$\epsilon_{1}$	$0$	$0.2$	$0.4$	$0.6$	$0.8$	$1$
Validation	70.43 (1.21)	70.44 (1.18)	70.44 (1.13)	70.38 (1.15)	70.37 (1.15)	70.39 (1.12)

Appendix

Table 9 : Worst-group validation accuracy of DR-TFM (MS) on Adult for selecting

\epsilon_{1}

, with

\epsilon_{2}=1

and ten initial candidate sources. Accuracy is evaluated over estimated groups. Results are averaged over three seeds; parentheses report standard deviations.

Method	$\eta$	$C$	LR	Batch size	Budget	Patience	Weight decay	Selected quantities
DR-TFM (GR; $k$ -means)	0.1	5	0.01	full	1000	5	0.005	$G_{c}$ per class by silhouette over $\{2,4,6,8\}$
DR-TFM (GR; GMM)	0.1	5	0.01	full	1000	5	0.005	$G_{c}$ per class by silhouette over $\{2,4,6,8\}$
DR-TFM (GR) ∗	0.1	5	0.01	full	1000	5	0.005	none (groups given)
DR-TFM (MS)	—	—	0.001	1024	300	5	$10^{-4}$	$\epsilon_{1}$ once on Adult; fixed thereafter
DR-TFM (MS) ∗	—	—	0.001	1024	300	5	$10^{-4}$	none (groups given)
ERM-MLP	—	—	0.001	4096	1000	20	0	none

Appendix

Table 10 : Hyperparameters. The final column lists selected quantities; all other settings are fixed. LR denotes the model learning rate.

Method	Trained component	Settings
TabPFN (fine-tuned)	Full backbone	Learning rate $10^{-5}$ , weight decay $0.02$ , 30 epochs, patience 8
TabPFN (class-balanced)	Full backbone	As TabPFN (fine-tuned), with inverse-class-frequency weights in the cross-entropy
BETA	Input adapter	16 ensemble members, learning rate $0.003$ , weight decay $0.02$ , batch size 1024, 30 epochs, patience 5
MixturePFN	Expert models	Context size 3000; experts use the TabPFN fine-tuning settings above
LoCalPFN	None	Retrieval context of 1000 neighbors per query batch
TuneTables	Soft prompt	10 tokens, $\leq 31$ epochs, patience 5, TabPFN v1

Appendix

Table 11 : Additional baseline settings.

Method	$G_{\mathrm{cl}}=2$		$G_{\mathrm{cl}}=4$		$G_{\mathrm{cl}}=6$		$G_{\mathrm{cl}}=8$		Range (pp)
Method	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst
DR-TFM (GR; $k$ -means)	79.31	72.57	77.42	68.37	75.97	63.90	75.09	61.39	4.22	11.18
	(0.17)	(0.64)	(0.33)	(1.29)	(0.63)	(3.00)	(0.77)	(1.12)
GEORGE ( $k$ -means)	73.79	63.80	61.80	43.96	64.22	48.25	58.64	44.45	15.15	19.84
	(0.48)	(1.21)	(2.28)	(4.56)	(1.04)	(2.78)	(5.89)	(9.45)
$\Delta$ (pp)	+5.52	+8.77	+15.62	+24.41	+11.75	+15.65	+16.45	+16.94	10.93	15.64

Appendix

Table 12 : Ablation on

G_{\mathrm{cl}}

for the five tabular benchmarks, averaged over datasets and three seeds. Parentheses report standard deviations across seeds of the dataset-averaged accuracies.

\Delta

is DR-TFM (GR) minus GEORGE. Range is the maximum minus minimum within each row across the four values of

G_{\mathrm{cl}}

. Ranges are computed before rounding.

Method	$G_{\mathrm{cl}}=2$		$G_{\mathrm{cl}}=4$		$G_{\mathrm{cl}}=6$		$G_{\mathrm{cl}}=8$		Range (pp)
Method	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst
DR-TFM (GR; $k$ -means)	79.21	71.06	80.00	72.99	79.53	72.22	79.70	72.59	0.78	1.93
	(0.14)	(0.12)	(0.12)	(0.40)	(0.38)	(1.11)	(0.04)	(0.39)
GEORGE ( $k$ -means)	74.36	66.15	64.70	54.67	65.41	56.41	66.75	46.94	9.66	19.22
	(3.03)	(2.31)	(1.35)	(4.89)	(2.71)	(1.27)	(6.77)	(9.64)
$\Delta$ (pp)	+4.85	+4.91	+15.29	+18.32	+14.11	+15.81	+12.95	+25.66	10.44	20.75

Appendix

Table 13 : Ablation on

G_{\mathrm{cl}}

for ACS Income, averaged over the three within-state settings and three seeds. Parentheses report standard deviations across seeds of the setting-averaged accuracies.

\Delta

is DR-TFM (GR) minus GEORGE. Range is the maximum minus minimum within each row across the four values of

G_{\mathrm{cl}}

. Ranges are computed before rounding.

Methods	Adult		Bank		Default		Shoppers		Taxi		Avg.
Methods	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst
Standard methods
ERM-MLP	73.45	45.29	71.86	42.03	63.70	28.52	76.44	50.11	75.27	57.37	72.14	44.67
	(0.16)	(0.90)	(1.80)	(4.18)	(0.48)	(0.21)	(0.47)	(1.85)	(0.60)	(1.44)	(0.70)	(1.72)
XGBoost	77.09	52.58	68.69	33.87	64.78	30.72	78.03	55.42	75.81	57.57	72.88	46.03
	(0.17)	(0.48)	(0.31)	(0.92)	(0.11)	(0.13)	(0.19)	(0.24)	(0.21)	(0.16)	(0.20)	(0.39)

Appendix

Table 14 : Results on the five tabular benchmarks for methods without true group labels. Avg. denotes the average of each metric across datasets. Values in parentheses are standard deviations over three seeds. Accuracy results marked † , including standard deviations, are taken from Tong et al. (2025) . Bold method names indicate our variants.

Methods	Adult		Bank		Default		Shoppers		Taxi		Avg.
Methods	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst
Robust methods
Fair-TabICL (group-balanced)	80.11	72.34	86.58	78.54	69.86	57.73	86.24	83.38	77.45	68.03	80.05	72.00
	(0.23)	(0.91)	(0.09)	(0.10)	(0.23)	(0.71)	(0.85)	(0.91)	(0.32)	(0.49)	(0.34)	(0.62)
Fair-TabICL (uncertainty)	61.80	21.35	71.52	40.51	64.93	31.18	77.80	52.81	75.94	57.61	70.40	40.69
	(0.74)	(1.65)	(0.14)	(0.32)	(0.12)	(0.13)	(0.43)	(1.57)	(0.14)	(0.52)	(0.31)	(0.84)

Appendix

Table 15 : Results on the five tabular benchmarks for methods using true group labels. Avg. denotes the average of each metric across datasets. Values in parentheses are standard deviations over three seeds. Bold method names indicate our variants.

Methods	AZ		MA		MI		Avg.		AZ $\rightarrow$ MA		MA $\rightarrow$ MI		MI $\rightarrow$ AZ		Avg.
Methods	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst
Standard methods
ERM-MLP	77.20	59.42	80.82	73.82	75.69	57.99	77.90	63.74	76.82	60.23	79.13	71.34	75.56	56.31	77.17	62.63
	(0.54)	(1.59)	(0.07)	(0.61)	(0.32)	(1.88)	(0.31)	(1.36)	(0.62)	(2.91)	(0.09)	(0.66)	(0.30)	(2.40)	(0.34)	(1.99)
XGBoost	77.64	58.99	82.26	76.38	77.50	62.77	79.13	66.05	77.96	61.99	79.67	72.01	76.82	58.99	78.15	64.33
	(0.27)	(0.76)	(0.25)	(0.12)	(0.11)	(0.40)	(0.21)	(0.43)	(0.14)	(0.36)	(0.06)	(0.13)	(0.10)	(0.30)	(0.10)	(0.27)

Appendix

Table 16 : ACS Income results for methods without true group labels. The left block reports within-state results and the right block reports transfers between states. Avg. denotes the average of each metric across the three settings in its block. Values in parentheses are standard deviations over three seeds. Accuracy results marked † , including standard deviations, are taken from Tong et al. (2025) . Bold method names indicate our variants.

Methods	AZ		MA		MI		Avg.		AZ $\rightarrow$ MA		MA $\rightarrow$ MI		MI $\rightarrow$ AZ		Avg.
Methods	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst
Robust methods
Fair-TabICL (group-balanced)	79.43	74.38	81.42	78.68	78.96	75.19	79.94	76.09	80.10	78.18	79.56	75.01	78.87	75.41	79.51	76.20
	(0.24)	(0.09)	(0.03)	(0.82)	(0.20)	(1.21)	(0.16)	(0.71)	(0.08)	(0.03)	(0.06)	(0.54)	(0.15)	(1.33)	(0.10)	(0.63)
Fair-TabICL (uncertainty)	77.72	58.40	82.36	77.46	76.71	62.07	78.93	65.98	77.41	60.40	79.59	73.40	76.24	57.21	77.75	63.67
	(0.17)	(0.57)	(0.05)	(0.21)	(0.31)	(0.61)	(0.18)	(0.47)	(0.08)	(0.39)	(0.12)	(0.25)	(0.13)	(0.13)	(0.11)	(0.26)

Appendix

Table 17 : ACS Income results for methods using true group labels. Avg. denotes the average of each metric across the three settings in its block. Values in parentheses are standard deviations over three seeds. Bold method names indicate our variants.

Method	Adult	Bank	Default	Shoppers	Taxi	Avg.
XGBoost	5.5	6.4	7.1	4.6	4.1	5.5
CatBoost	22.8	25.2	24.1	12.8	12.8	19.5
TabPFN	7.5	7.8	5.5	3.2	3.1	5.4
ERM-MLP	32.0	36.6	29.9	15.1	14.8	25.7
DR-TFM (GR; $k$ -means)	25.6	50.2	24.6	9.1	9.4	23.8
DR-TFM (MS; $k$ -means)	35.9	45.0	36.6	16.9	18.8	30.6

Appendix

Table 18 : Runtime in seconds per reported result, averaged over three seeds. Times include hyperparameter selection and all training or adaptation stages; TabPFN requires no parameter updates.

Method	Adult	Bank	Default	Shoppers	Taxi	Avg.
TabPFN	0.77	0.77	0.62	0.44	0.43	0.61
DR-TFM (GR; $k$ -means)	1.37	1.62	1.15	0.60	0.61	1.07
DR-TFM (MS; $k$ -means)	1.27	1.41	1.02	0.61	0.58	0.98
GEORGE ( $k$ -means)	1.62	1.63	1.60	0.71	0.51	1.21
TabPFN (fine-tuned)	10.98	14.45	11.81	5.25	3.75	9.25
LSR (all stages)	3.56	5.07	4.82	1.50	0.67	3.12

Appendix

Table 19 : Peak GPU memory in GB per reported result, averaged over three seeds.

Adaptation	Adult		Bank		Default		Shoppers		Taxi		Avg.
Adaptation	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst
Input adapter
ERM	74.34	46.89	69.39	36.98	64.39	30.34	78.31	55.67	75.38	57.36	72.36	45.45
	(0.67)	(1.58)	(1.53)	(2.81)	(0.10)	(0.53)	(0.91)	(2.12)	(0.14)	(0.22)	(0.67)	(1.45)
DRO	78.83	65.90	82.70	73.21	67.92	47.05	84.96	79.60	77.23	67.43	78.33	66.64
	(0.51)	(1.58)	(1.34)	(1.38)	(0.38)	(1.44)	(0.61)	(0.57)	(0.27)	(0.97)	(0.62)	(1.19)

Appendix

Table 20 : Mean group accuracy and worst-group accuracy under ERM-based and DRO-based adaptation (GR) for five adaptation strategies, averaged over three seeds. Avg. denotes the average of each metric across the five tabular benchmarks. Values in parentheses are standard deviations over three seeds. Parenthesized values in the Avg. columns are the averages of the per-dataset standard deviations.

Fine-tuned parameters	Adult	Bank	Default	Shoppers	Taxi	Avg.
Input adapter	56.5	63.0	40.8	26.8	30.3	43.5
Encoder	214.2	439.1	364.7	80.0	107.8	241.2
Full backbone	213.8	438.3	383.8	90.3	106.8	246.6
Decoder	29.9	45.3	25.9	9.6	9.5	24.0
Query scaling	25.6	50.2	24.6	9.1	9.4	23.8

Appendix

Table 21 : Runtime in seconds per reported result under DRO-based adaptation for the adaptation strategies in Table 4 , averaged over three seeds. Times include group estimation, adaptation, and prediction.

Fine-tuned parameters	Adult	Bank	Default	Shoppers	Taxi	Avg.
Input adapter	12.58	15.45	10.86	5.02	5.09	9.80
Encoder	12.22	16.10	13.20	5.32	4.27	10.22
Full backbone	12.34	16.24	13.29	5.36	4.31	10.31
Decoder	1.38	1.68	1.18	0.62	0.63	1.10
Query scaling	1.37	1.62	1.15	0.60	0.61	1.07

Appendix

Table 22 : Peak GPU memory in GB per reported result under DRO-based adaptation for the adaptation strategies in Table 4 , averaged over three seeds.

Method	Five tabular benchmarks		ACS Income
Method	Mean	Worst	Mean	Worst
ERM-MLP	72.14	44.67	77.54	63.19
$\chi^{2}$ -DRO †	70.00	47.82	74.62	62.95
TabPFN- $\chi^{2}$ -DRO ( $\alpha=1$ )	73.34	47.34	78.73	65.26
TabPFN- $\chi^{2}$ -DRO ( $\alpha=0.6$ )	73.16	47.19	78.02	63.27
TabPFN- $\chi^{2}$ -DRO ( $\alpha=0.4$ )	69.65	37.75	76.93	59.40

Appendix

Table 23 : Average of mean group accuracies and average of worst-group accuracies for the

\chi^{2}

-DRO variant at each tested

\alpha

, over the five tabular datasets or all six ACS settings, and over three seeds.

\alpha=1

uses the empirical mean loss. All tested values are reported. † marks averages computed from accuracy results reported by Tong et al. (2025) .

Methods	Adult		Bank		Default		Shoppers		Taxi		Avg.
Methods	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst
TabPFN	75.33	49.17	73.07	42.96	64.77	31.09	79.59	58.29	76.01	58.19	73.75	47.94
	(0.05)	(0.28)	(0.31)	(0.76)	(0.04)	(0.04)	(0.32)	(0.52)	(0.25)	(0.53)	(0.19)	(0.43)
+ Ours	79.64	67.90	85.01	80.14	69.42	60.37	85.79	81.66	76.69	72.79	79.31	72.57
	(0.20)	(1.61)	(0.36)	(0.53)	(0.62)	(1.70)	(0.32)	(0.50)	(0.19)	(0.62)	(0.34)	(0.99)
	(+4.31)	(+18.73)	(+11.94)	(+37.18)	(+4.65)	(+29.28)	(+6.20)	(+23.37)	(+0.68)	(+14.60)	(+5.56)	(+24.63)

Appendix

Table 24 : GR adaptation across tabular foundation models on the five tabular benchmarks. “+ Ours” denotes DR-TFM (GR) applied to the model above; ∗ indicates use of true group labels. Avg. denotes the average of each metric across datasets. Non-bold parentheses report standard deviations over three seeds. Bold parentheses report changes from the corresponding pretrained model, in percentage points.

Methods	AZ		MA		MI		Avg.		AZ $\rightarrow$ MA		MA $\rightarrow$ MI		MI $\rightarrow$ AZ		Avg.
Methods	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst	Mean	Worst
TabPFN	78.19	59.87	83.01	77.02	77.72	63.50	79.64	66.80	77.95	61.57	79.69	71.86	76.81	59.08	78.15	64.17
	(0.32)	(0.50)	(0.12)	(0.27)	(0.03)	(0.20)	(0.16)	(0.32)	(0.07)	(0.18)	(0.08)	(0.25)	(0.05)	(0.12)	(0.07)	(0.18)
+ Ours	78.81	70.36	79.96	70.93	78.87	71.90	79.21	71.06	78.43	69.70	78.73	66.43	78.46	68.84	78.54	68.32
	(0.13)	(1.00)	(0.59)	(1.31)	(0.30)	(0.86)	(0.34)	(1.06)	(0.15)	(0.72)	(0.68)	(0.34)	(0.15)	(0.59)	(0.32)	(0.55)
	(+0.62)	(+10.49)	(-3.05)	(-6.09)	(+1.15)	(+8.40)	(-0.43)	(+4.26)	(+0.48)	(+8.13)	(-0.96)	(-5.43)	(+1.65)	(+9.76)	(+0.39)	(+4.15)

Appendix

Table 25 : GR adaptation across tabular foundation models on ACS Income. “+ Ours” denotes DR-TFM (GR) applied to the model above; ∗ indicates use of true group labels. The left block reports within-state results and the right block reports transfers between states. Avg. denotes the average of each metric across the three settings in its block. Non-bold parentheses report standard deviations over three seeds. Bold parentheses report changes from the corresponding pretrained model, in percentage points.

Explore similar work

Jul 28, 2026cs.LG

Empirical Evaluation of Out-Of-Distribution Performance of Tabular Foundation Models

Tabular Foundation Models (TFMs) have emerged as novel approaches for tabular predictive tasks, demonstrating competitive predictive performance to ensemble tree-based models. Most TFMs are trained and evaluated on independent and identically distributed data, but this assumption changes in real-world scenarios due to distribution shifts, which compromise the robustness of models. Limited research has been conducted of TFMs under distribution shifts. We present an empirical evaluation of Out-Of-Distribution (OOD) performance of nine TFMs, spanning diverse pre-training strategies and architectures: TabPFNv2, TabPFNv2.5, TabPFNv2.6, TabPFNv3, TabICL, TabICLv2, Mitra, LimiX and TabFM. Three real-world datasets from the TableShift study were considered (HELOC, Voting, Childhood Lead), covering label, socioeconomic, and geographic shift types. Our results show that all evaluated TFMs degrade systematically under distribution shift regardless of pre-training strategy, with shift gaps ranging from 0.003 to 0.060 depending on shift type. The relationship between in-distribution and OOD predictive performance documented for classical tabular models extends into TFMs. We also identified a scalability gap, as high-performing models demand significant memory and computational resources beyond what standard deployment infrastructure can support. This study extends existing benchmarks for OOD in tabular data, providing evidence to support their adoption in high-stakes domains characterized by structural distribution shifts.

Malena Loza, David Chushig-Muzo, Eva Milara +3

Colegio de Ciencias e Ingenierías, Universidad San Francisco de Quito (USFQ), Quito, Ecuador · Department of Signal Theory and Communications, Telematics and Computing Systems, Rey Juan Carlos University, Madrid, Spain · Facultad de Ingeniería, Universidad Latina de Panamá, Ciudad de Panamá, Panamá

Tabular Foundation Models Tabpfn Out-Of-Distribution Distribution Shifts Empirical Study

Ask PDF

May 6, 2026cs.LG

Mitigating Label Shift in Tabular In-Context Learning via Test-Time Posterior Adjustment

TabPFN has recently gained attention as a foundation model for tabular datasets, achieving strong performance by leveraging in-context learning on synthetic data. However, we find that TabPFN is vulnerable to label shift, often overfitting to the majority class in the training dataset. To address this limitation, we propose DistPFN, the first test-time posterior adjustment method designed for tabular foundation models. DistPFN rescales predicted class probabilities by downweighting the influence of the training prior (i.e., the class distribution of the context) and emphasizing the contribution of the model's predicted posterior, without architectural modification or additional training. We further introduce DistPFN-T, which incorporates temperature scaling to adaptively control the adjustment strength based on the discrepancy between prior and posterior. We evaluate our methods on over 250 OpenML datasets, demonstrating substantial improvements for various TabPFN-based models in classification tasks under label shift, while maintaining strong performance in standard settings without label shift. Code is available at this repository: https://github.com/seunghan96/DistPFN.

Seunghan Lee

LG AI Research, Seoul, South Korea.

Tabular Foundation Models Tabpfn Training Data Posterior In-Context Test Time

Ask PDF

May 7, 2026cs.LG

TFM-Retouche: A Lightweight Input-Space Adapter for Tabular Foundation Models

Tabular foundation models (TFMs), such as TabPFN-2.6, TabICLv2, ConTextTab, Mitra, LimiX, and TabDPT, achieve strong zero-shot performance through in-context learning, but their inductive biases remain fixed at inference time. Adapting a pretrained TFM to a specific dataset or task typically requires either full fine-tuning, which is computationally expensive, or parameter-efficient tuning methods (PEFT) such as LoRA, which must be tailored to the internal architecture of each TFM. Furthermore, the evidence on whether weight-space fine-tuning improves accuracy or calibration is mixed \citep{tanna_exploring_2026,rubachev_finetuning_2025}. We introduce TFM-Retouche, a lightweight input-space residual adapter that is architecture-agnostic by design with respect to the frozen TFM backbone. TFM-Retouche learns a small residual correction in the input space to align the input data with the inductive biases of the pretrained model. The adapter is trained end-to-end through the frozen TFM, with a post-training identity guard that falls back to the unmodified TFM whenever adaptation does not help on held-out validation. On TabArena-Lite (51 datasets spanning binary classification, multiclass classification, and regression), TabICLv2-Retouche -- the framework instantiated on TabICLv2 -- is the top-ranked method on the leaderboard with light per-task tuning and ensembling, lifting aggregate Elo by +56 over the frozen TabICLv2 base and sitting on the Pareto front of predictive quality versus both training and inference time.

Duong Nguyen, Mohammed Jawhar, Nicolas Chesneau

Ekimetrics

Tabular Foundation Models Lightweight Image Editing

Ask PDF