Efficient Provably Private Classification with a Tabular Foundation Model

Talal Alrawajfeh; Cristiana Diaconu; Ossi Räisä; Sebastian Rodriguez Beltran; Yuan He; John Bronskill; Richard E. Turner; Antti Honkela

cs.LGOct 7, 2026

Efficient Provably Private Classification with a Tabular Foundation Model

Authors: Talal Alrawajfeh, Cristiana Diaconu, Ossi Räisä, Sebastian Rodriguez Beltran, Yuan He, John Bronskill, Richard E. Turner, Antti Honkela

Organizations: Department of Computer Science, University of Helsinki, Helsinki, Finland · Department of Engineering, University of Cambridge, Cambridge, United Kingdom · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany · Current address: Vienna, Austria

Abstract

Tabular data underpin prediction and decision-making in medicine, finance, government and science, but often contain sensitive individual-level information, creating a need for accurate prediction while preserving privacy. Traditional private learning provides formal privacy guarantees, but requires slow dataset-specific optimisation, suffers substantial utility loss under strong privacy, and is often difficult to apply correctly. Tabular foundation models adapt rapidly to new datasets, but existing models lack formal privacy guarantees, and are highly vulnerable to membership-inference attacks, limiting their use on sensitive data. Here we introduce PrivTab, an easy to use tabular foundation model for differentially private classification that embeds a privacy mechanism within its architecture. Pretrained on simulated datasets, PrivTab uses in-context learning to transform sensitive rows into compact, provably private summaries---effectively learning how to learn under privacy. PrivTab outperforms private linear and neural-network baselines under moderate-to-strong privacy, shows negligible membership leakage, maintains well-calibrated predictions under strong privacy, and reduces dataset fitting time by 10,000 times, requiring only a single forward pass. By combining formal privacy, speed, and easy of use, PrivTab brings recent advances in AI to applications where sensitive individual-level data have limited their adoption.

Figures & tables

Panel of Figure 1 — Figure 1 : Alternative approaches to tabular prediction from sensitive data. a , Private learning trains a new model on each sensitive dataset, requiring relatively slow dataset-specific optimisation. b , Non-private tabular learning fits new datasets in a single forward pass through in-context learning, but retains sensitive data for prediction. c , PrivTab instead releases a reusable private summary in one forward pass; all subsequent predictions are private due to the post-processing property of DP. Dashed lines mark the privacy boundary, and stopwatches indicate relative rather than measured time. d , Qualitative comparison; green ticks, orange crosses and grey dashes denote presence, absence and non-applicability, respectively. The auditability comparison concerns the complete data-dependent path, including hyperparameter tuning and privacy accounting, rather than an isolated private update ( Ponomareva et al., 2023 ; Papernot and Steinke, 2022 ; Ganev et al., 2025 ) . e , Membership-inference attack accuracy for the most vulnerable record of each method. For each record in the dataset, we compute the lower endpoint of the 95% confidence interval for attack accuracy; the bar shows the maximum of these lower endpoints. An accuracy of 50% corresponds to perfect privacy. TabPFN v3 and TabICL v2 approach 100%, whereas PrivTab remains much closer to perfect privacy (50%) at the displayed strong and weak formal privacy levels.

Panel of Figure 2 — Figure 2 : Predictive performance and fitting time across 33 TabArena datasets. a , Paired AUC gap, $\mathrm{AUC}_{m}-\mathrm{AUC}_{\mathrm{non\text{-}private\,LR}}$ , for each method $m$ (PrivTab, DP-MLP, DP-LR). b , Relative log-loss improvement over non-private logistic regression for the same three methods. Each quantity is computed on matched context–target splits before aggregation; zero denotes parity with non-private logistic regression and positive values mean $m$ is better. Error bars are bootstrap 95% confidence intervals over datasets. c , Elo rating at each privacy level; higher ratings indicate better predictive performance, and shaded bands mark strong ( $\mu<0.15$ ), moderate ( $0.15\leq\mu<0.6$ ) and weak ( $\mu\geq 0.6$ ) formal privacy. Elo combines binary-task AUC and multiclass-task log loss and is calibrated separately at each privacy level so that differentially private logistic regression (DP-LR) scores 1,000. All results use ten context–target splits per dataset. d , Aggregate Elo over all privacy levels versus median fitting time per context–target split on a logarithmic scale, so better methods lie towards the upper left. Horizontal and vertical bars are bootstrap 95% confidence intervals for median fitting time and Elo rating, respectively.

Figure 3 : AUC gaps versus dataset size across 33 TabArena datasets. a , Mean paired AUC gap between PrivTab and the differentially private multilayer perceptron (DP-MLP). b , The corresponding gap between PrivTab and differentially private logistic regression (DP-LR). Columns show privacy levels

\mu=0.1,0.2

and

0.4

; each point is a dataset’s mean paired gap across ten context–target splits, plotted against its number of rows on a logarithmic scale. Positive gaps favour PrivTab, while the dashed horizontal line marks equal AUC. Circles denote binary datasets and squares denote multiclass datasets; AUC is macro-averaged one-versus-rest for multiclass tasks. Grey lines are least-squares fits against log dataset size, shown as visual guides; annotations report Spearman’s

\rho

and the corresponding

p

-value.

Figure 4 : Private preprocessing case study. a , Mean PrivTab AUC across ten context–target splits at total privacy level $\mu=0.4$ . The three preprocessing variants compare non-private standardisation, partially private preprocessing using oracle bounds computed from empirical dataset minima and maxima, and private preprocessing using bounds inferred from public dataset descriptions by an LLM. Only the LLM-derived bounds avoid using sensitive values to select clipping intervals; the oracle-bounds comparison is partially private. b , Mean AUC for PrivTab, DP-MLP and DP-LR using LLM-derived bounds and private standardisation across the same five datasets and ten splits. Higher AUC is better. Error bars show bootstrap 95% confidence intervals. Legends beneath each panel identify its plotted variants.

Figure 5 : Record-level membership-inference protection. a , Distribution of the record-wise membership-inference AUC on the Maternal Health Risk dataset. For each record, we compute its AUC and 95% confidence interval, then plot the lower endpoint of that interval. Boxes show the interquartile range and median, whiskers span the central 95%, squares mark the maximum across records, and dotted vertical lines mark the provable AUC upper bound for PrivTab at each privacy level. b , ROC curves for the most vulnerable record at each of the six PrivTab privacy levels and for TabPFN v3 and TabICL v2. Grey dashed lines in a,b denote perfect privacy. PrivTab remains much closer to perfect privacy whereas the most vulnerable rows of the non-private models are highly vulnerable.

Figure E1 : Detailed PrivTab architecture. Context and target rows are first mapped to row embeddings by

\mathrm{RowEmbed}

. In

\mathrm{PrivateSummary}

, a small set of data-independent learnable pseudo-tokens cross-attends to the sensitive context rows through DP-MHCA, producing privatised summaries. The public

\mathrm{Prediction}

module reuses these summaries through cross-attention and token-wise multilayer perceptrons. DP-MHCA is the only operation that transfers context information into the released summaries; target-side computation is post-processing.

Appendix figures & tables49 assets

Supplementary material from the paper’s appendix.

Appendix

Sub-block	Query Token(s)	Key-Value Token(s)	Updated Token(s)	Privacy Status
Main Layers ( $\ell=1,\ldots,L_{\mathrm{main}}$ )
DP Cross-Attention	Summaries $\tilde{\mathbf{s}}_{i}^{(\ell-1)}$	Context $\mathbf{z}_{j}^{c}$	$\mathbf{s}_{i}^{(\ell)}$	Private (DP-MHCA)
Latent Self-Attention	Summaries $\mathbf{s}_{i}^{(\ell)}$	Summaries $\mathbf{s}_{1:m}^{(\ell)}$	$\tilde{\mathbf{s}}_{i}^{(\ell)}$	Public (post-processing)
Target Cross-Attention	Targets $\mathbf{z}_{j}^{t,(\ell-1)}$	Summaries $\tilde{\mathbf{s}}_{1:m}^{(\ell)}$	$\mathbf{z}_{j}^{t,(\ell)}$	Public (post-processing)
Post-Processing Layers ( $\ell=L_{\mathrm{main}}+1,\ldots,L_{\mathrm{main}}+L_{\mathrm{post}}$ )
Latent Self-Attention	Summaries $\tilde{\mathbf{s}}_{i}^{(\ell-1)}$	Summaries $\tilde{\mathbf{s}}_{1:m}^{(\ell-1)}$	$\tilde{\mathbf{s}}_{i}^{(\ell)}$	Public (post-processing)

Appendix

Table S1 : Summary of operations and token flow in each layer

\ell

of the PrivTab encoder. Layer Normalisation (

\operatorname{LN}

) is applied before each operation in a pre-normalisation layout.

Baseline	Hyperparameter	Search space
DP-LR	Learning rate	$[10^{-4},10^{-2}]$ log-uniform
	Batch size	$\{128,256,512,1024\}$
	Clipping norm	$\{0.1,0.5,1,2,5\}$
	Epochs	$\{100,300,500,1000\}$
DP-MLP	Learning rate	$[10^{-4},10^{-2}]$ log-uniform
	Batch size	$\{128,256,512,1024\}$

Appendix

Table S2 : Baseline hyperparameter search spaces. Bayesian optimisation is run separately for each private baseline.

Figure S1 : Histogram of the hyperparameters selected for DP-LR across all datasets and context–target splits. The selected values are concentrated enough to justify fixing the non-learning-rate hyperparameters in the final experiments.

Figure S2 : Histogram of the hyperparameters selected for DP-MLP across all datasets and context–target splits. As for DP-LR, the selected values are concentrated enough to justify fixing the non-learning-rate hyperparameters in the final experiments.

Method	Learning-rate grid
DP-LR	$\{0.005,\;0.02,\;0.05,\;0.1\}$
DP-MLP	$\{3{\times}10^{-4},\;10^{-3},\;3{\times}10^{-3},\;10^{-2}\}$

Appendix

Table S3 : Learning-rate grids used for private model selection. Only the learning rate is selected under DP; all other hyperparameters are fixed from the aggregate hyperparameter-optimisation results in Figs. S1 and S2 .

Method	Hyperparameter	Value
DP-LR	Clipping norm	$0.1$
DP-LR	Subsampling rate $q$	$0.05$
DP-LR	Epochs	$1000$
DP-MLP	Clipping norm	$5.0$
DP-MLP	Subsampling rate $q$	$0.05$
DP-MLP	Epochs	$250$

Appendix

Table S4 : Fixed hyperparameters used for the DP baselines after inspecting the aggregate hyperparameter-selection histograms.

Method	Aggregate AUC (mean [95% CI])	Record AUC (mean [95% CI])	Record AUC (95th percentile)
PrivTab ( $\mu=0.05$ )	$0.501\ [0.498,0.505]$	$0.504\ [0.504,0.504]$	$0.513$
PrivTab ( $\mu=0.1$ )	$0.503\ [0.500,0.507]$	$0.505\ [0.505,0.506]$	$0.515$
PrivTab ( $\mu=0.2$ )	$0.508\ [0.504,0.511]$	$0.508\ [0.508,0.509]$	$0.519$
PrivTab ( $\mu=0.4$ )	$0.515\ [0.511,0.518]$	$0.514\ [0.514,0.515]$	$0.526$
PrivTab ( $\mu=0.8$ )	$0.527\ [0.524,0.530]$	$0.523\ [0.522,0.523]$	$0.537$
PrivTab ( $\mu=1.6$ )	$0.543\ [0.540,0.546]$	$0.533\ [0.532,0.534]$	$0.554$

Appendix

Table S5 : FPC-corrected membership-inference audit on Maternal Health Risk. Aggregate AUC measures attack performance over 100 held-out context draws; record AUC measures each row’s fitted exposure across 4,000 draws. Means are reported with bootstrap 95% CIs over held-out draws or rows, respectively, and the record-level 95th percentile summarises the most exposed 5% of rows. Values near

0.5

indicate perfect privacy.

Figure S3 : FPC-corrected record-level membership-inference curves. ROC curves are shown for the ten most vulnerable Maternal Health Risk records at each setting. Shades distinguish records, and the diagonal denotes perfect privacy. The fitted IN/OUT standard deviations use the same FPC as Fig. 5 a. PrivTab remains substantially closer to the diagonal than non-private TabPFN v3 and TabICL v2, whose most vulnerable records are nearly perfectly distinguishable.

Panel of Figure 15 — Figure S4 : Empirical audit of the DP-MHCA mechanism across privacy levels. For each $\mu$ , the audit optimises a pair of adjacent token sets to maximise the pre-noise output difference and then estimates the trade-off curve from repeated samples of the privatised outputs, using 10 repeated audit runs; shaded bands show pointwise bootstrap 95% CIs over those runs. The resulting curves are compared against the strongest likelihood-ratio attack with access to the unnoised means and the calibrated noise standard deviation. The observed behaviour is consistent with the intended privacy calibration across the full range of $\mu$ .

Figure S5 : Dataset-level AUC gap between PrivTab and DP-MLP across all privacy levels. Each marker represents a dataset, positioned by its number of features and rows. Marker colour gives the mean paired AUC gap over ten context–target splits; positive values favour PrivTab. Binary AUC is used for binary datasets and macro-averaged one-versus-rest AUC for multiclass datasets.

Figure S6 : Dataset-level AUC gap between PrivTab and DP-LR across all privacy levels. Plotting conventions match Fig. S5 ; positive values favour PrivTab.

Figure S7 : AUC gaps versus feature count across 33 TabArena datasets. a , PrivTab minus DP-MLP AUC; b , PrivTab minus DP-LR AUC. Columns show privacy levels $\mu=0.1,0.2$ and $0.4$ . Each point is a dataset’s mean paired AUC gap across ten context–target splits, plotted against its number of features on a logarithmic scale; positive gaps favour PrivTab. Circles denote binary datasets and squares denote multiclass datasets; AUC is macro-averaged one-versus-rest for multiclass tasks. Dashed horizontal lines mark equal AUC, grey lines are least-squares fits against log feature count, and annotations report Spearman’s $\rho$ and its $p$ -value.

$\mu$	Method	ELO	CI min	CI max
0.05	PrivTab	1142.8	1125.0	1161.3
0.05	DP-LR	1000.0	970.4	1027.8
0.05	DP-MLP	910.7	882.1	937.9
0.1	PrivTab	1172.4	1155.4	1190.4
0.1	DP-LR	1000.0	969.7	1028.7
0.1	DP-MLP	966.7	939.5	993.0

Appendix

Table S6 : Numerical values underlying the ELO curves in Figure 2 (c).

Method	ELO	CI min	CI max
PrivTab	1166.5	1159.3	1173.7
DP-MLP	1097.5	1086.1	1109.2
DP-LR	1000.0	988.1	1011.9

Appendix

Table S7: Numerical values underlying the aggregate ELO coordinates in Figure 2 (d), with ratings aggregated over all privacy budgets.

Method	$\mu$	Mean	CI low	CI high	$n$ datasets	$n$ splits
DP-LR	0.05	-0.1249	-0.1579	-0.0950	33	10
DP-LR	0.1	-0.0933	-0.1210	-0.0669	33	10
DP-LR	0.2	-0.0663	-0.0880	-0.0460	33	10
DP-LR	0.4	-0.0471	-0.0643	-0.0314	33	10
DP-LR	0.8	-0.0335	-0.0471	-0.0211	33	10
DP-LR	1.6	-0.0238	-0.0359	-0.0131	33	10

Appendix

Table S8 : Numerical values underlying Figure 2 (a).

Method	$\mu$	Mean	CI low	CI high	$n$ datasets	$n$ splits
DP-LR	0.05	-12.0599	-20.0871	-6.3234	33	10
DP-LR	0.1	-6.4823	-10.8479	-3.1667	33	10
DP-LR	0.2	-3.1523	-4.8574	-1.8595	33	10
DP-LR	0.4	-1.5109	-1.8378	-1.2354	33	10
DP-LR	0.8	-1.1520	-1.2902	-1.0277	33	10
DP-LR	1.6	-1.0440	-1.1426	-0.9516	33	10

Appendix

Table S9 : Numerical values underlying the relative log-loss improvements in Figure 2 (b).

Dataset	Rows	Features	PrivTab	DP-MLP	DP-LR
Amazon employee access	32769	9	0.037	524.350	1540.025
Anneal	898	31	0.012	92.250	284.975
Bank marketing	45211	13	0.021	724.675	2105.300
Bank customer churn	10000	10	0.013	202.975	616.075
Blood transfusion	748	4	0.013	91.700	299.800
Churn	5000	19	0.013	145.150	452.825

Appendix

Table S10 : Dataset sizes and per-context–target-split runtimes in seconds underlying Figure S8 .

Figure S8 : Fitting time as a function of dataset size. Points are dataset-level measurements and curves are fitted shifted power laws. PrivTab is approximately four orders of magnitude faster than the DP Adam baselines over the studied range.

Dataset	Target rows	Features	PrivTab	DP-MLP	DP-LR
Amazon employee access	6554	9	355	1.73	0.727
Anneal	180	31	15.2	0.159	0.114
Bank marketing	9042	13	582	2.83	0.928
Bank customer churn	2000	10	105	0.585	0.2
Blood transfusion	150	4	13	0.124	0.0849
Churn	1000	19	64.4	0.346	0.115

Appendix

Table S11 : Dataset sizes and post-fit prediction times in milliseconds on a laptop with a 13th-generation Intel Core i5 CPU, underlying Figure S9 . Values average the per-configuration median of three timed runs over six privacy levels and ten context–target splits.

Figure S9 : Prediction time as a function of target-set size on a laptop CPU. Points show dataset-level mean post-fit prediction times in milliseconds (ms) on a 13th-generation Intel Core i5, averaged over six privacy levels and ten context–target splits. Curves are shifted power laws fitted in log time as visual guides. Both axes are logarithmic. All methods predict in batches of up to 1,024 target rows.

Panel of Figure 27 — Figure S10 : Performance relative to DP-LR. Results aggregate ten context–target splits per dataset; error bars show bootstrap 95% CIs over datasets. Positive improvements and win rates above $0.5$ favour the compared method. Bands are visual guides, not formal privacy thresholds: strong ( $\mu<0.15$ ), moderate ( $0.15\leq\mu<0.6$ ) and weak ( $\mu\geq 0.6$ ) formal privacy.

Method	$\mu$	Mean	CI low	CI high	$n$ datasets
PrivTab	0.05	0.0027	-0.0198	0.0257	33
PrivTab	0.1	0.0160	-0.0011	0.0352	33
PrivTab	0.2	0.0192	0.0040	0.0355	33
PrivTab	0.4	0.0216	0.0086	0.0355	33
PrivTab	0.8	0.0185	0.0068	0.0307	33
PrivTab	1.6	0.0150	0.0035	0.0280	33

Appendix

Table S12 : Numerical values underlying Figure S10 (a) (AUC improvement over DP-LR).

Method	$\mu$	Mean	CI low	CI high	$n$ datasets
PrivTab	0.05	0.5455	0.4485	0.6424	33
PrivTab	0.1	0.5939	0.4909	0.6970	33
PrivTab	0.2	0.5788	0.4576	0.6939	33
PrivTab	0.4	0.6303	0.5061	0.7455	33
PrivTab	0.8	0.6424	0.5152	0.7606	33
PrivTab	1.6	0.6273	0.5000	0.7545	33

Appendix

Table S13 : Numerical values underlying Figure S10 (b) (AUC win rate over DP-LR).

Method	$\mu$	Mean	CI low	CI high	$n$ datasets
PrivTab	0.05	0.7061	0.6100	0.7827	33
PrivTab	0.1	0.6333	0.5370	0.7092	33
PrivTab	0.2	0.5598	0.4586	0.6342	33
PrivTab	0.4	0.4684	0.3679	0.5398	33
PrivTab	0.8	0.4292	0.3305	0.4971	33
PrivTab	1.6	0.4109	0.3036	0.4811	33

Appendix

Table S14 : Numerical values underlying Figure S10 (c) (Relative log-loss reduction over DP-LR).

Method	$\mu$	Mean	CI low	CI high	$n$ datasets
PrivTab	0.05	0.9545	0.8788	1.0000	33
PrivTab	0.1	0.9606	0.8909	1.0000	33
PrivTab	0.2	0.9667	0.9030	1.0000	33
PrivTab	0.4	0.9424	0.8606	1.0000	33
PrivTab	0.8	0.9485	0.8667	1.0000	33
PrivTab	1.6	0.9455	0.8606	1.0000	33

Appendix

Table S15 : Numerical values underlying Figure S10 (d) (Log-loss win rate over DP-LR).

Dataset	$\mu=0.05$	$\mu=0.1$	$\mu=0.2$	$\mu=0.4$	$\mu=0.8$	$\mu=1.6$
Amazon employee access	0.562 [0.553,0.571]	0.560 [0.546,0.573]	0.570 [0.561,0.579]	0.569 [0.561,0.578]	0.569 [0.561,0.577]	0.570 [0.563,0.576]
Anneal	0.638 [0.596,0.673]	0.685 [0.660,0.708]	0.735 [0.721,0.749]	0.827 [0.812,0.844]	0.852 [0.827,0.875]	0.882 [0.861,0.906]
Bank customer churn	0.753 [0.739,0.766]	0.776 [0.769,0.783]	0.785 [0.779,0.792]	0.790 [0.783,0.797]	0.793 [0.786,0.800]	0.794 [0.786,0.801]
Bank marketing	0.715 [0.710,0.720]	0.726 [0.723,0.728]	0.729 [0.725,0.732]	0.730 [0.726,0.734]	0.730 [0.726,0.734]	0.730 [0.726,0.733]
Blood transfusion	0.532 [0.479,0.588]	0.616 [0.577,0.652]	0.656 [0.625,0.685]	0.709 [0.686,0.733]	0.725 [0.704,0.750]	0.734 [0.712,0.759]
Churn	0.748 [0.722,0.770]	0.797 [0.786,0.808]	0.820 [0.815,0.825]	0.839 [0.833,0.845]	0.842 [0.835,0.850]	0.839 [0.831,0.847]

Appendix

Table S16 : Per-dataset AUC for PrivTab; entries report the mean and bootstrap 95% CI over context–target splits.

Dataset	$\mu=0.05$	$\mu=0.1$	$\mu=0.2$	$\mu=0.4$	$\mu=0.8$	$\mu=1.6$
Amazon employee access	0.230 [0.228,0.232]	0.232 [0.231,0.234]	0.232 [0.231,0.232]	0.232 [0.232,0.232]	0.232 [0.231,0.232]	0.232 [0.231,0.232]
Anneal	0.968 [0.926,1.015]	0.890 [0.863,0.919]	0.791 [0.765,0.819]	0.702 [0.689,0.715]	0.538 [0.518,0.557]	0.432 [0.418,0.446]
Bank customer churn	0.450 [0.444,0.457]	0.433 [0.428,0.438]	0.425 [0.422,0.429]	0.420 [0.416,0.423]	0.420 [0.416,0.423]	0.420 [0.417,0.424]
Bank marketing	0.345 [0.344,0.346]	0.341 [0.340,0.342]	0.339 [0.338,0.340]	0.338 [0.337,0.339]	0.337 [0.337,0.338]	0.337 [0.337,0.338]
Blood transfusion	0.562 [0.553,0.570]	0.541 [0.533,0.550]	0.532 [0.523,0.541]	0.506 [0.498,0.514]	0.497 [0.487,0.507]	0.488 [0.476,0.499]
Churn	0.391 [0.386,0.397]	0.354 [0.346,0.361]	0.329 [0.325,0.333]	0.318 [0.314,0.323]	0.319 [0.315,0.322]	0.321 [0.317,0.324]

Appendix

Table S17 : Per-dataset log loss for PrivTab; entries report the mean and bootstrap 95% CI over context–target splits.

Dataset	$\mu=0.05$	$\mu=0.1$	$\mu=0.2$	$\mu=0.4$	$\mu=0.8$	$\mu=1.6$
Amazon employee access	0.523 [0.508,0.538]	0.526 [0.509,0.542]	0.524 [0.502,0.545]	0.539 [0.524,0.552]	0.548 [0.531,0.562]	0.553 [0.532,0.567]
Anneal	0.573 [0.526,0.618]	0.681 [0.637,0.729]	0.790 [0.751,0.831]	0.863 [0.819,0.904]	0.899 [0.862,0.936]	0.943 [0.919,0.963]
Bank customer churn	0.709 [0.681,0.735]	0.726 [0.702,0.746]	0.727 [0.701,0.746]	0.730 [0.711,0.747]	0.728 [0.710,0.745]	0.739 [0.730,0.748]
Bank marketing	0.659 [0.625,0.687]	0.681 [0.664,0.696]	0.684 [0.663,0.701]	0.689 [0.674,0.703]	0.701 [0.693,0.708]	0.706 [0.697,0.714]
Blood transfusion	0.667 [0.631,0.698]	0.703 [0.673,0.727]	0.724 [0.700,0.749]	0.728 [0.708,0.750]	0.731 [0.709,0.754]	0.735 [0.712,0.758]
Churn	0.698 [0.661,0.734]	0.749 [0.724,0.772]	0.760 [0.730,0.787]	0.767 [0.744,0.788]	0.776 [0.756,0.796]	0.787 [0.772,0.803]

Appendix

Table S18 : Per-dataset AUC for DP-LR; entries report the mean and bootstrap 95% CI over context–target splits.

Dataset	$\mu=0.05$	$\mu=0.1$	$\mu=0.2$	$\mu=0.4$	$\mu=0.8$	$\mu=1.6$
Amazon employee access	0.510 [0.407,0.649]	0.462 [0.404,0.534]	0.419 [0.388,0.459]	0.396 [0.382,0.413]	0.386 [0.377,0.396]	0.378 [0.374,0.383]
Anneal	17.709 [7.089,30.296]	7.697 [3.262,13.514]	4.202 [1.611,7.595]	0.867 [0.564,1.284]	0.576 [0.434,0.803]	0.425 [0.325,0.568]
Bank customer churn	1.089 [0.900,1.395]	0.914 [0.873,0.961]	0.921 [0.871,0.988]	0.921 [0.878,0.972]	0.950 [0.892,1.011]	0.931 [0.901,0.967]
Bank marketing	0.721 [0.662,0.796]	0.686 [0.654,0.723]	0.673 [0.653,0.695]	0.673 [0.662,0.684]	0.658 [0.650,0.667]	0.645 [0.639,0.651]
Blood transfusion	1.859 [0.925,2.898]	1.438 [0.898,2.109]	1.188 [0.910,1.530]	0.989 [0.889,1.110]	0.894 [0.822,0.980]	0.884 [0.820,0.959]
Churn	1.496 [0.715,2.575]	1.053 [0.683,1.709]	0.908 [0.695,1.266]	0.741 [0.703,0.779]	0.725 [0.672,0.793]	0.694 [0.668,0.722]

Appendix

Table S19 : Per-dataset log loss for DP-LR; entries report the mean and bootstrap 95% CI over context–target splits.

Dataset	$\mu=0.05$	$\mu=0.1$	$\mu=0.2$	$\mu=0.4$	$\mu=0.8$	$\mu=1.6$
Amazon employee access	0.528 [0.511,0.543]	0.545 [0.532,0.557]	0.567 [0.557,0.577]	0.579 [0.571,0.587]	0.587 [0.576,0.597]	0.599 [0.587,0.610]
Anneal	0.489 [0.431,0.542]	0.528 [0.467,0.580]	0.610 [0.539,0.675]	0.764 [0.714,0.805]	0.870 [0.836,0.895]	0.928 [0.909,0.944]
Bank customer churn	0.633 [0.595,0.671]	0.707 [0.685,0.728]	0.765 [0.752,0.779]	0.792 [0.777,0.809]	0.826 [0.813,0.838]	0.847 [0.839,0.854]
Bank marketing	0.698 [0.688,0.707]	0.719 [0.713,0.724]	0.729 [0.724,0.734]	0.733 [0.728,0.738]	0.740 [0.735,0.744]	0.745 [0.741,0.748]
Blood transfusion	0.563 [0.487,0.630]	0.594 [0.518,0.659]	0.621 [0.550,0.688]	0.661 [0.606,0.712]	0.723 [0.696,0.752]	0.741 [0.722,0.761]
Churn	0.523 [0.467,0.566]	0.594 [0.537,0.647]	0.726 [0.680,0.767]	0.814 [0.793,0.836]	0.862 [0.846,0.878]	0.887 [0.877,0.896]

Appendix

Table S20 : Per-dataset AUC for DP-MLP; entries report the mean and bootstrap 95% CI over context–target splits.

Dataset	$\mu=0.05$	$\mu=0.1$	$\mu=0.2$	$\mu=0.4$	$\mu=0.8$	$\mu=1.6$
Amazon employee access	0.283 [0.229,0.358]	0.250 [0.223,0.298]	0.224 [0.221,0.227]	0.221 [0.220,0.223]	0.220 [0.219,0.221]	0.219 [0.218,0.220]
Anneal	2.720 [1.402,4.559]	1.983 [1.203,3.363]	1.709 [1.020,2.640]	0.826 [0.584,1.141]	0.513 [0.402,0.668]	0.309 [0.273,0.345]
Bank customer churn	0.510 [0.487,0.536]	0.479 [0.464,0.496]	0.435 [0.424,0.445]	0.410 [0.396,0.422]	0.384 [0.372,0.394]	0.377 [0.367,0.388]
Bank marketing	0.339 [0.336,0.343]	0.330 [0.328,0.334]	0.325 [0.322,0.329]	0.322 [0.319,0.326]	0.317 [0.315,0.319]	0.316 [0.315,0.318]
Blood transfusion	0.764 [0.653,0.909]	0.708 [0.607,0.831]	0.610 [0.565,0.651]	0.582 [0.540,0.619]	0.532 [0.504,0.562]	0.502 [0.483,0.521]
Churn	0.653 [0.452,1.015]	0.547 [0.407,0.790]	0.371 [0.354,0.392]	0.324 [0.310,0.337]	0.288 [0.270,0.305]	0.271 [0.256,0.288]

Appendix

Table S21 : Per-dataset log loss for DP-MLP; entries report the mean and bootstrap 95% CI over context–target splits.

Dataset	Non-DP
Amazon employee access	0.582 [0.574,0.590]
Anneal	0.985 [0.979,0.990]
Bank customer churn	0.767 [0.758,0.776]
Bank marketing	0.732 [0.729,0.736]
Blood transfusion	0.728 [0.710,0.745]
Churn	0.780 [0.771,0.788]

Appendix

Table S22 : Per-dataset AUC for Non-private LR; entries report the mean and bootstrap 95% CI over context–target splits.

Dataset	Non-DP
Amazon employee access	0.219 [0.218,0.219]
Anneal	0.170 [0.150,0.194]
Bank customer churn	0.426 [0.421,0.432]
Bank marketing	0.316 [0.314,0.317]
Blood transfusion	0.494 [0.482,0.508]
Churn	0.341 [0.337,0.345]

Appendix

Table S23 : Per-dataset log loss for Non-private LR; entries report the mean and bootstrap 95% CI over context–target splits.

Figure S11 : Effect of preprocessing on PrivTab. Mean log loss across ten context–target splits on five datasets, comparing non-private standardisation, partially private standardisation with oracle bounds, and private standardisation with LLM-derived bounds. Lower log loss is better. Error bars show bootstrap 95% confidence intervals. The legend beneath the plot identifies the preprocessing variants.

Panel of Figure 41 — Figure S12 : Performance of privately preprocessed models at $\mu=0.4$ . a,b , AUC and log loss for PrivTab, DP-MLP and DP-LR using LLM-derived bounds and private standardisation across five datasets. c , Elo ratings of these models; DP-LR is calibrated to 1,000. Higher AUC and Elo and lower log loss are better. Error bars show bootstrap 95% confidence intervals. The method legend is beneath all three panels.

Dataset	Method	AUC mean	CI low	CI high	Loss mean	CI low	CI high
Bank marketing	PrivTab	0.878	0.875	0.882	0.273	0.271	0.276
Bank marketing	DP-LR	0.868	0.861	0.873	0.507	0.490	0.528
Bank marketing	DP-MLP	0.886	0.883	0.889	0.249	0.245	0.252
COMPAS	PrivTab	0.715	0.708	0.722	0.619	0.615	0.624
COMPAS	DP-LR	0.707	0.690	0.720	1.282	1.224	1.340
COMPAS	DP-MLP	0.691	0.675	0.705	0.642	0.633	0.652

Appendix

Table S24 : Numerical values underlying Figure 4 (c,d).

Method	ELO rating	95% CI lower	95% CI upper
PrivTab	1,161.5	1,122.0	1,206.2
DP-MLP	1,117.5	1,059.4	1,181.1
DP-LR	1,000.0	930.9	1,060.0

Appendix

Table S25 : ELO ratings for the private-preprocessing evaluation underlying Fig. 4 e. Ratings combine AUC for binary datasets and log loss for multiclass datasets across the five datasets and ten context–target splits at total privacy level

\mu=0.4

. The scale is calibrated so that DP-LR has an ELO rating of 1,000. Intervals are 95% confidence intervals.

Dataset	Metric	Non-DP $z$ -score	Partially private $z$ -score (oracle bounds)	Private $z$ -score (LLM bounds)
Bank marketing	AUC	0.880 [0.878, 0.882]	0.877 [0.874, 0.880]	0.878 [0.875, 0.882]
Bank marketing	Log loss	0.272 [0.270, 0.274]	0.273 [0.271, 0.275]	0.273 [0.271, 0.276]
COMPAS	AUC	0.715 [0.708, 0.721]	0.711 [0.704, 0.718]	0.715 [0.708, 0.722]
COMPAS	Log loss	0.620 [0.616, 0.624]	0.623 [0.618, 0.628]	0.619 [0.615, 0.624]
ACS income	AUC	0.847 [0.846, 0.848]	0.847 [0.846, 0.848]	0.847 [0.846, 0.848]
ACS income	Log loss	0.470 [0.469, 0.472]	0.470 [0.469, 0.472]	0.469 [0.468, 0.470]

Appendix

Table S26 : Numerical values underlying Figure 4 (a,b). Entries report means with bootstrap 95% CIs over context–target splits; log loss is in nats/sample.

Feature	Type	Exact min	Exact max	Source	Lower	Upper	Confidence
AGEP	numeric	17	95	llm	16	90	medium
SEX	categorical	1	2	categorical	1	2	exact
RAC1P	categorical	1	9	categorical	1	9	exact
SCHL	categorical	1	24	categorical	1	24	exact
MAR	categorical	1	5	categorical	1	5	exact
RELP	categorical	0	38	categorical	0	17	exact

Appendix

Table S27 : Public and LLM-derived bounds for ACS income.

Feature	Type	Exact min	Exact max	Source	Lower	Upper	Confidence
age	numeric	18	95	llm	18	100	medium
job	categorical	0	11	categorical	0	11	exact
marital	categorical	0	2	categorical	0	2	exact
education	categorical	0	3	categorical	0	3	exact
default	categorical	0	1	categorical	0	1	exact
balance	numeric	-8019	102127	llm	-100000	1e+06	low

Appendix

Table S28 : Public and LLM-derived bounds for Bank marketing.

Feature	Type	Exact min	Exact max	Source	Lower	Upper	Confidence
sex	categorical	0	1	categorical	0	1	exact
age	numeric	18	80	llm	18	100	medium
juv_fel_count	numeric	0	10	llm	0	10	medium
juv_misd_count	numeric	0	13	llm	0	10	medium
juv_other_count	numeric	0	7	llm	0	10	medium
priors_count	numeric	0	38	llm	0	50	medium

Appendix

Table S29 : Public and LLM-derived bounds for COMPAS.

Feature	Type	Exact min	Exact max	Source	Lower	Upper	Confidence
Age	numeric	10	70	llm	10	60	medium
SystolicBP	numeric	70	160	llm	70	250	medium
DiastolicBP	numeric	49	100	llm	40	150	medium
BS	numeric	6	19	llm	0	40	medium
BodyTemp	numeric	98	103	llm	80	110	medium
HeartRate	numeric	7	90	llm	0	250	medium

Appendix

Table S30 : Public and LLM-derived bounds for Maternal health.

Feature	Type	Exact max	Source	Lower	Upper	Confidence
preg	numeric	17	llm	0	17	medium
plas	numeric	199	llm	40	400	medium
pres	numeric	122	llm	30	130	medium
skin	numeric	99	llm	5	80	medium
insu	numeric	846	llm	10	500	medium
mass	numeric	67.1	llm	10	70	medium

Appendix

Table S31 : Public and LLM-derived bounds for Pima diabetes.

$\mu$	Method	ELO	CI min	CI max
overall	PrivTab with noise-aware pretraining	1440.2	1430.9	1449.5
overall	PrivTab without noise-aware pretraining	1188.1	1176.2	1200.0
0.05	PrivTab with noise-aware pretraining	1404.5	1381.8	1427.4
0.05	PrivTab without noise-aware pretraining	1108.9	1079.7	1135.6
0.1	PrivTab with noise-aware pretraining	1459.1	1435.9	1482.6
0.1	PrivTab without noise-aware pretraining	1084.6	1055.2	1111.6

Appendix

Table S32 : ELO rankings for the noise-aware-pretraining ablation.

$\mu$	Method	ELO	CI min	CI max
overall	PrivTab ( $\mu\in[0.16,64]$ )	1533.1	1523.6	1542.6
overall	PrivTab ( $\mu\in[0.2,80]$ )	1501.6	1493.1	1510.2
overall	PrivTab ( $\mu\in[0.64,256]$ )	1489.9	1480.7	1498.9
overall	PrivTab ( $\mu\in[0.1,40]$ )	1486.0	1478.0	1494.0
overall	PrivTab ( $\mu\in[0.4,160]$ )	1480.9	1472.9	1489.0
overall	PrivTab ( $\mu\in[0.05,20]$ )	1470.4	1462.8	1478.2

Appendix

Table S33 : ELO rankings for the

\mu

-range pretraining ablation.

$\mu$	Method	ELO	CI min	CI max
overall	PrivTab ( $L_{\mathrm{main}}=3$ )	1507.2	1496.3	1518.6
overall	PrivTab ( $L_{\mathrm{main}}=4$ )	1404.0	1394.2	1414.0
overall	PrivTab ( $L_{\mathrm{main}}=2$ )	1283.2	1273.8	1292.9
0.05	PrivTab ( $L_{\mathrm{main}}=3$ )	1493.3	1465.6	1524.2
0.05	PrivTab ( $L_{\mathrm{main}}=4$ )	1453.3	1430.0	1478.8
0.05	PrivTab ( $L_{\mathrm{main}}=2$ )	1294.6	1271.2	1317.0

Appendix

Table S34 : ELO rankings for the ablation over the number of main DP-MHCA layers.

$\mu$	Method	ELO	CI min	CI max
overall	PrivTab ( $H_{\mathrm{DP}}=1$ )	1508.9	1500.1	1517.9
overall	PrivTab ( $H_{\mathrm{DP}}=4$ )	1468.5	1459.3	1478.0
0.05	PrivTab ( $H_{\mathrm{DP}}=1$ )	1465.8	1441.1	1491.8
0.05	PrivTab ( $H_{\mathrm{DP}}=4$ )	1422.8	1399.6	1448.4
0.1	PrivTab ( $H_{\mathrm{DP}}=1$ )	1464.8	1442.4	1488.9
0.1	PrivTab ( $H_{\mathrm{DP}}=4$ )	1462.6	1440.0	1487.7

Appendix

Table S35 : ELO rankings for the ablation over the number of DP-MHCA heads.

$\mu$	Method	ELO	CI min	CI max
overall	PrivTab (tanh)	1503.9	1492.4	1515.9
overall	PrivTab (shifted softsign)	1409.4	1399.9	1419.5
overall	PrivTab (linear normalised)	1392.4	1384.1	1400.8
overall	PrivTab (linear clipped)	1370.2	1361.3	1379.3
0.05	PrivTab (linear clipped)	1397.6	1376.8	1420.3
0.05	PrivTab (linear normalised)	1394.7	1374.9	1416.1

Appendix

Table S36 : ELO rankings for the ablation over the DP-MHCA interaction function.

$\mu$	Method	ELO	CI min	CI max
overall	PrivTab combination (final)	1578.1	1571.0	1585.2
overall	PrivTab small-context model + inference normalisation	1563.4	1556.1	1570.6
overall	PrivTab large-context model + normalisation	1552.2	1543.8	1560.7
overall	DP-MLP	1500.5	1489.1	1512.0
overall	PrivTab small-context model	1414.3	1404.3	1424.1
overall	DP-LR	1411.6	1400.3	1422.8

Appendix

Table S37 : ELO rankings for the normalisation and model-combination ablation.

Explore similar work

Jun 30, 2026cs.LG

TabPATE: Differentially Private Tabular In-Context Learning Without Public Data

Tabular foundation models enable accurate in-context learning (ICL) from small labeled datasets, but the private records placed in context can leak through model predictions. We first show that even basic membership inference attacks succeed against tabular ICL, motivating formal privacy protection. We then introduce TabPATE, a differentially private PATE-style defense for tabular ICL that does not require public in-distribution data. TabPATE partitions the private context across teacher models, privately aggregates their labels on synthetic tabular queries, and releases the resulting labeled queries as a student context. Because tabular features are bounded and relatively low-dimensional, useful queries can be generated from feature ranges alone or from lightly privatized marginals. Across tabular benchmarks, TabPATE preserves competitive utility while reducing membership inference to near-random success, providing a practical path to private tabular ICL without public data.

Dariush Wahdany, Matthew Jagielski, Jesse C. Cresswell +2

CISPA · Anthropic · Layer 6 AI

Privacy-Preserving Machine Learning Membership Inference Attacks Tabpfn In-Context Learning LLM Defense Mechanisms

Ask PDF

Jun 24, 2026cs.CR

Privacy Vulnerabilities of Attention Layers in Tabular Foundation Models and Protection of High-Risk Queries

Tabular foundation models are commonly assumed to present limited privacy concerns as they are often pre-trained on large collections of synthetic data. However, these models leverage in-context learning, where sensitive records may be provided directly at inference time as labelled context examples. In this paper, we demonstrate that predictions generated via the attention mechanism leak sufficient information to enable effective Membership Inference Attacks (MIAs). To highlight this vulnerability, we propose AMIA (Attention-based Membership Inference Attack), a shadow-model-free attack that exploits the concentration of transformer attention patterns. Our results show that attention mechanisms reveal strong membership signals, which exceed classical confidence-based attacks, achieving an average gain of 7.7%, specially in low false-positive regimes. To mitigate this risk, we introduce an inference-time defence inspired by

k

-anonymity principles. This approach reduces the uniqueness of context-key representations without introducing random noise or retraining the model. By targeting only high-risk queries identified through AMIA scores, the defence substantially reduces membership leakage of this attack by an average of 50% and 25% against confidence-based attacks, while preserving predictive utility with only 3.9% performance degradation. Beyond showing that context examples are vulnerable, we further demonstrate that fine-tuning introduces an additional source of privacy risk. In particular, samples whose prediction confidence increases after fine-tuning become more susceptible to MIAs, indicating that fine-tuning can amplify memorisation and expose sensitive training information through confidence shifts.

Tânia Carvalho, Maxime Cordy

SnT, University of Luxembourg · Luxembourg

Tabular Foundation Models Membership Inference Attacks Inference-Time Defense Privacy Attention Layers In-Context Learning

Ask PDF

May 7, 2026cs.LG

On Privacy Leakage in Tabular Diffusion Models: Influential Factors, Attacker Knowledge, and Metrics

Tabular data plays an important role in many fields and industries, including those with elevated privacy considerations and risks. As such, there is a rising interest in generating high-quality synthetic proxies for real tabular data as a means of reducing privacy risk and proprietary data exposure. With tabular diffusion models (TDMs) demonstrating leading performance in synthesizing such data, understanding and measuring the privacy risks associated with these models is imperative. Leveraging state-of-the-art membership inference attacks for TDMs in both black- and white-box settings, this work quantifies the impact of training setup, synthesis choices, and attacker knowledge on privacy leakage. Moreover, the results demonstrate that adversaries need not have perfect knowledge of the training setup, identical data distributions, or massive compute resources to construct successful attacks. Finally, the pitfalls associated with applying heuristic privacy metrics, such as distance-to-closest record, are revealed.

Masoumeh Shafieinejad, D. B. Emerson, Behnoosh Zamanlooy +5

Vector Institute · McMaster University · University of Toronto +1

Membership Inference Attacks Privacy Attacker

Ask PDF