A Critical Review on the Effectiveness and Privacy Threats of Membership Inference Attacks

Najeeb Jebreel; David Sánchez; Josep Domingo-Ferrer

cs.CRMar 24, 2026

A Critical Review on the Effectiveness and Privacy Threats of Membership Inference Attacks

Authors: Najeeb Jebreel, David Sánchez, Josep Domingo-Ferrer

Organizations: Universitat Rovira i Virgili, Department of Computer Engineering and Mathematics, CYBERCAT-Center for Cybersecurity Research of Catalonia, ComSCIAM-Center for Computational Science and Applied Mathematics Av. Països Catalans 26, 43007 Tarragona, Catalonia

Abstract

Membership inference attacks (MIAs) aim to determine whether a data sample was included in a machine learning (ML) model's training set and have become the de facto standard for measuring privacy leakages in ML. We propose an evaluation framework that defines the conditions under which MIAs constitute a genuine privacy threat, and review representative MIAs against it. We find that, under the realistic conditions defined in our framework, MIAs represent weak privacy threats. Thus, relying on them as a privacy metric in ML can lead to an overestimation of risk and to unnecessary sacrifices in model utility as a consequence of employing too strong defenses.

Figures & tables

Figure 1: Overview of the proposed evaluation framework for MIAs. Condition C0 relates to the MIA disclosure potential, which critically depends on the data set used to train the target model. Conditions C1–C4 characterize the effectiveness of the MIA itself, which should reliably attack non-overfitted and competitive models at a reasonable computational cost.

Attack	Data set	Train acc. (%)	Test acc. (%)	$g$ (%)	Non-overfitted?	SotA acc. (%)	$\Delta$ (%)	Competitive?
[ 65 ]	Adult	84.80	84.20	0.60	✓	88.16 [ 13 ]	3.96	✓
	Purchase-100	NA	72.90	NA	NA	90.00 [ 69 ]	17.10	✗
	Texas-100	NA	57.00	NA	NA	57.00 [ 65 ]	0.00	✓
	Locations	100.00	67.30	32.70	✗	72.00 [ 45 ]	4.70	✓
	MNIST	98.40	92.80	5.60	✓	99.87 [ 10 ]	7.07	✗
	CIFAR-10	NA	60.00	NA	NA	99.50 [ 21 ]	39.50	✗

Table 1: Evaluation of model non-overfitting (C1) and competitiveness (C2)

Attack	Data set	Attack performance (%)			Prior $p$ (%)	Reliable?
Attack	Data set	Prec.	TPR	FPR	Prior $p$ (%)	Reliable?
[ 65 ]	Adult	50.30	NA	NA	50	✗
	Purchase-100	73.00	86.00	31.81	50	✗
	Locations	67.80	98.00	46.54	50	✗
	Texas-100	69.00	86.00	38.64	50	✗
	MNIST	51.70	NA	NA	50	✗

Table 2: Evaluation of attack performance and membership reliability (C3)

Attack	Resource requirements	Cost factors			Overall
		M	I	Q	cost
[ 65 ]	10-100 additional models; $K$ ML attack models; 1 query	High	Moderate	Low	High
[ 77 ]	Simple thresholding rule on loss values; 1 query	Low	Low	Low	Low
[ 63 ]	Simple thresholding rule; 1K random data points per data set; 1 query	Low	Low	Low	Low
[ 62 ]	30 additional models; 1 query	High	Low	Low	High
[ 50 ]	50 additional models; 1 query	High	Low	Low	High

Table 3: Evaluation of computational feasibility (C4). Cost factors: M: additional models required, I: inference model complexity, Q: queries required per target sample.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Attack	Approach	Venue	# citations
[ 65 ]	ML membership classifier on predictions from shadow models	IEEE SP 2017	7,083
[ 77 ]	Loss global thresholding	IEEE CSF 2018	1,751
[ 63 ]	Confidence and entropy global thresholding	NDSS 2019	1,329
[ 62 ]	Per-sample loss calibration and thresholding	ICML 2019	515
[ 50 ]	Hypothesis testing on loss values of selected vulnerable records	Euro SP 2020	143
[ 37 ]	Perturbed input loss thresholding	PoPETs 2021	194

Appendix

Table 4: Summary of the surveyed attacks. Citation counts according to Google Scholar, January 8, 2026.

Membership Inference Attacks

Membership inference attacks test whether records or entities were used to train a machine-learning model, revealing potential privacy leakage.

See all 111 papers

SoK: Rethinking Membership Inference in Tabular Machine Learning
Oct 13, 2026Cristina Pêra, Tânia Carvalho, Maxime Cordy +1Membership Inference Attacks Tabular ML
TekPrivacy, Portugal · SnT, University of Luxembourg, Luxembourg · Department of Computer Science, Faculty of Sciences of the University of Porto, Portugal
A Full-Pipeline Framework for Evaluating Membership Inference Attacks in Machine Learning
May 28, 2026Ding Chen, Xinwen Cheng, Xuyang Zhong +3Privacy Auditing Membership Inference Attacks
City University of Hong Kong · Shanghai Jiao Tong University · denotes the correspondence author.
Revisiting the LiRA Membership Inference Attack Under Realistic Assumptions
Mar 8, 2026Najeeb Jebreel, Mona Khalil, David Sánchez +1Membership Inference Attacks Privacy Auditing
Universitat Rovira i Virgili, Catalonia

Oct 13, 2026J/K move · Enter open · S save

SoK: Rethinking Membership Inference in Tabular Machine Learning

Cristina Pêra, Tânia Carvalho, Maxime Cordy, Luís Antunes

TekPrivacy, Portugal · SnT, University of Luxembourg, Luxembourg · Department of Computer Science, Faculty of Sciences of the University of Porto, Portugal

Membership Inference Attacks Tabular ML

Open paper AskPDF arXiv

Membership Inference Attacks (MIAs) are currently the dominant approach to evaluating privacy in machine learning applications. Despite their significance, several concerns remain unexplored, particularly with regard to tabular data. Existing evaluations focus predominantly on high-dimensional datasets, leaving the privacy behavior of small-to-medium datasets unexplored. In this paper, we present a new taxonomy for MIAs and conduct a comprehensive study of the factors influencing vulnerability. We evaluate four distinct state-of-the-art attacks across diverse tabular datasets and model architectures. Our results show that datasets with higher class granularity and majority class percentage are more susceptible to MIAs. We further demonstrate that advanced MIAs are highly dependent on adversarial knowledge of the target model, while reference model-free attacks often achieve statistically comparable results. Most importantly, we show that records with unique signatures (single-outs) remain highly vulnerable, even when common defenses are applied. These findings indicate that current defenses reduce average leakage, but fail to adequately protect the most sensitive individuals.

Abstract

Membership Inference Attacks

Explore similar work

SoK: Rethinking Membership Inference in Tabular Machine Learning

A Full-Pipeline Framework for Evaluating Membership Inference Attacks in Machine Learning

Revisiting the LiRA Membership Inference Attack Under Realistic Assumptions