SoK: Rethinking Membership Inference in Tabular Machine Learning
Authors: Cristina Pêra, Tânia Carvalho, Maxime Cordy, Luís Antunes
Organizations: TekPrivacy, Portugal · SnT, University of Luxembourg, Luxembourg · Department of Computer Science, Faculty of Sciences of the University of Porto, Portugal
Membership Inference Attacks (MIAs) are currently the dominant approach to evaluating privacy in machine learning applications. Despite their significance, several concerns remain unexplored, particularly with regard to tabular data. Existing evaluations focus predominantly on high-dimensional datasets, leaving the privacy behavior of small-to-medium datasets unexplored. In this paper, we present a new taxonomy for MIAs and conduct a comprehensive study of the factors influencing vulnerability. We evaluate four distinct state-of-the-art attacks across diverse tabular datasets and model architectures. Our results show that datasets with higher class granularity and majority class percentage are more susceptible to MIAs. We further demonstrate that advanced MIAs are highly dependent on adversarial knowledge of the target model, while reference model-free attacks often achieve statistically comparable results. Most importantly, we show that records with unique signatures (single-outs) remain highly vulnerable, even when common defenses are applied. These findings indicate that current defenses reduce average leakage, but fail to adequately protect the most sensitive individuals.
Figures & tables
Figure 1: Membership Inference Attacks effectiveness for different Purchases variants. Results are reported for five runs.
Figure 2: Taxonomy for Membership Inference Attacks.
Year
Approach Name
Attack Classification
Attack Model-based
Learning Type
Adversary Knowledge
Defenses
Data Type
Open Source
2017
MIA [ 64 ]
Shadow
✓
DL
●
✓
T & I
x
2018
LOSS [ 86 ]
Metric-based
x
DL & TL
●
x
T & I
x
2019
MALT [ 59 ]
Metric-based
x
DL & TL
●
x
I
x
2019
MAST [ 59 ]
Shadow
✓
DL & TL
●
x
I
x
2019
MATT [ 59 ]
Influence functions
x
TL
○
x
I
x
2020
Prediction uncertainty [ 50 ]
Gradient statistics
✓
DL
○
x
T & I
x
Table 1: Overview of membership inference attack proposals and their main characteristics. For each attack, we indicate whether it employs auxiliary models (e.g., shadow models, reference models or/and attack classifiers/regressors), the target model type (deep learning (DL) or traditional learning (TL)), the data types considered (tabular (T) and/or image (I)), and whether the attack was evaluated against defense mechanisms. "Open Source" contains links to repositories.
Attack
Granularity
Imbalance
Majority Class
Rarity
Target Entropy
Mean |Corr| w/ Target
Features w/ Outliers
RMIA
0.831*
0.079
-0.624
-0.677*
0.915*
-0.333
-0.267
LiRA
0.774*
-0.055
-0.758*
-0.360*
0.927*
-0.491
-0.377
Attack-P
0.604
-0.261
-0.818*
0.006
0.830*
-0.442
-0.359
QMIA
0.699*
-0.236
-0.794*
-0.194
0.915*
-0.479
-0.267
Table 2: Spearman rank correlation between dataset/class properties and MIA success. * denotes p<0.05 .
Attack
Model
COMPAS
Surgery
MIC
0.01
0.5
1
0.01
0.5
1
0.01
0.5
1
RMIA
MLP
-0.4
7.9
0.5
-8.9
6.3
11.1
-4.8
39.3
23.3
RF
-1.3
-1.4
-1.6
-14.6
5.6
9.9
23.6
28.0
32.7
LR
-1.2
-0.0
0.1
-1.3
-0.2
-0.7
-9.0
-8.3
-7.6
LiRA
MLP
3.2
7.7
3.9
2.4
12.4
18.6
-2.9
24.9
15.2
RF
-0.9
-1.2
-1.6
-3.8
9.4
11.6
14.0
17.2
19.3
Table 3: Attack AUC change before and after applying PP-SMOTE across three privacy budgets.
Figure 3: Comparison of Attack-P/QMIA versus surrogate RMIA performance at each reference model budget. The RMIA AUC line represents the mean ± standard deviation across all reference models architectures. The dashed lines show the mean Attack-P and QMIA AUC, with the respective shaded bands depicting the ± Standard Error of the Mean across seeds.
Figure 4: Difference in membership inference vulnerability between singled-out (SO) and non-SO records per dataset, measured by AUC and both [email protected]%FPR (solid) and [email protected]%FPR (dashed) for anonymity k=1 and k=2 . Points show the estimated SO–non-SO difference, with horizontal bars denoting confidence intervals (CIs).
Figure 5: Privacy–utility trade-off of defenses on singled-out ( k=1 ) records, by dataset, considering RMIA. Each point is one model. Color shows how much less often single-outs are correctly flagged as members after defense.
Figure 6: k-anonymity application over datasets and its percentage points difference against the baseline.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Domain
n
d
c
r
COMPAS Recidivism [ 29 ]
Crime
4937
20
2
1.05
Covid [ 4 ]
Healthcare
2671
29
3
6.3
Diabetes Readmission [ 30 ]
Healthcare
81410
150
2
1.17
Half Million [ 73 ]
Finance
15855
28
12
1.08
Locations [ 36 , 84 ]
Locations
5010
446
10
3.18
Mammography [ 35 ]
Healthcare
11183
24
2
42.01
Appendix
Table 4: Specifications of the evaluation datasets. n denotes the number of samples, d the number of features, c the number of classes, r the imbalance ratio.
Dataset
Selected quasi-identifiers
# Single-outs
k=1
k=2
COMPAS
Age, Race:Native_American, Race:Other
16-23
34-51
Covid
Age, Cerebro Vascular, Diabetes
5-10
16-25
Diabetes
time_in_hospital, race:Other, age:{[}50-70), admission_type_id:Trauma Center, discharge_disposition_id:Discharged to Home, discharge_disposition_id:Other
11-18
23-43
Half Million
Gender, State, Country
180-216
592-620
MIC
Age, Obesity, Chronic Pneumonia
14-19
24-31
Appendix
Table 5: Selected quasi-identifiers for each dataset and corresponding number of single-outs range across the 5 seeds.
Model
COMPAS
MIC
Surgery
Gen. gap (%)
MIA AUC (%)
Gen. gap (%)
MIA AUC (%)
Gen. gap (%)
MIA AUC (%)
RF
3.1 ± 2.9
59.0 ± 4.2
1.3 ± 1.1
70.6 ± 5.2
6.7 ± 0.7
77.0 ± 2.5
MLP
10.5 ± 8.8
60.0 ± 5.8
7.0 ± 4.9
59.1 ± 4.0
17.7 ± 7.9
64.3 ± 5.3
NB
0.8 ± 1.7
50.5 ± 0.8
7.9 ± 2.0
56.0 ± 1.4
0.1 ± 0.7
50.2 ± 0.7
DT
42.3 ± 1.2
74.2 ± 0.6
18.0 ± 1.8
62.8 ± 1.2
18.9 ± 0.4
62.3 ± 0.4
XGB
16.3 ± 1.3
68.3 ± 0.9
10.6 ± 0.5
70.9 ± 2.2
6.2 ± 0.3
64.3 ± 0.7
Appendix
Table 6: Comparison of the generalization gap of the target model with the RMIA AUC.
Figure 7: MIA AUC results for all attacks and models, averaged across all datasets.
Figure 8: Reference model equivalence gap for surrogate RMIA (1 and 5 reference models) versus Attack-P/QMIA. Each dataset point shows the mean MIA AUC gap, with the error gaps representing the Standard Error Mean across surrogate models. The shaded band denotes the 0.05 margin used in the TOST equivalence test.
Figure 9: SO vs. non-SO attack AUC, faceted by attack and anonymity threshold k .
Figure 10: Resulting application of defenses on the catch of single-outs for k=1. Each point represents a dataset-defense combination, averaging across the 6 models.
Universitat Rovira i Virgili, Department of Computer Engineering and Mathematics, CYBERCAT-Center for Cybersecurity Research of Catalonia, ComSCIAM-Center for Computational Science and Applied Mathematics Av. Països Catalans 26, 43007 Tarragona, Catalonia