Assessing Per-Sample Membership Inference Vulnerability without Retraining

Valentin Dorseuil; Jamal Atif; Olivier Cappé

stat.MLFeb 17, 2026

Assessing Per-Sample Membership Inference Vulnerability without Retraining

Authors: Valentin Dorseuil, Jamal Atif, Olivier Cappé

Organizations: DI-ENS · DI ENS, École normale supérieure, Université PSL, CNRS · CMAP · CMAP, École polytechnique, Institut Polytechnique de Paris

Abstract

Recent work in the privacy literature shows that sample-targeted membership inference attacks (MIAs) significantly outperform untargeted approaches by a wide margin. Motivated by this observation, we address the following question: can the privacy vulnerability of individual training points be assessed without training shadow models? We show that per-sample exposure to MIA is governed not only by a point's loss, but also by a data-dependent geometric measure. In the linear setting, we derive a closed-form decomposition of individual black-box MIA vulnerability into a population leverage score and a residual loss term, making explicit how sample-dependent geometry translates into privacy exposure. Since the final layer of most modern architectures is linear, we extend this framework to deep networks and propose a surrogate score operating on last-layer representations that requires only a single trained model and no shadow models. Empirical evaluations across diverse datasets and architectures show that our score outperforms loss and gradient-norm baselines at identifying the highest-risk points under state-of-the-art attacks, providing a computationally efficient and theoretically grounded tool for per-sample privacy risk assessment.

Figures & tables

Figure 1: Generalization gap and membership inference attack performance for the Gaussian linear regression model. (left) Heatmap of the generalization gap

\mathbb{E}[\Delta\ell_{i}]

as a function of

\varepsilon_{i}^{2}

and

\bar{h}_{ii}

, estimated with Monte-Carlo. (middle) Theoretical false negative rate (FNR)

\beta_{i}

of the optimal attack on the point

i

as a function of the false positive rate (FPR)

\alpha_{i}

, for different values of the structural error

\varepsilon_{i}

, with

\bar{h}_{ii}=1

and

\sigma^{2}=1

. (right) Same as (middle) but with fixed

\varepsilon_{i}=1

and

\bar{h}_{ii}

varying.

Figure 2: Recall of the top-

k

% LiRA’s attack most vulnerable samples (ranked by the Average Success Rate) when keeping the top-

k

% for each of the four surrogate: the loss

\ell_{i}

, the gradient norm

\|\nabla\ell_{i}\|

, the influence estimate

\widehat{\Delta\ell}^{\mathrm{IF}}_{i}

, and the Newton estimate

\widehat{\Delta\ell}^{\mathrm{NS}}_{i}

. Axes are log-log; the dotted diagonal

y=x

is the random baseline. Solid lines are means over

16

target models with

\pm 1

std bands; dashed-lines show a single representative model.

		Recall of Top 1% @5% (in %)
Dataset	Model	Loss $\ell_{i}$	$\\|\nabla\ell_{i}\\|$	$\widehat{\Delta\ell}^{\mathrm{NS}}_{i}$	$\widehat{\Delta\ell}^{\mathrm{IF}}_{i}$
SST-2	BERT, full ft.	$28.2\pm 2.4$ %	$5.3\pm 1.1$ %	$28.8\pm 2.6$ %	$\mathbf{28.8}\pm\mathbf{2.6}$ %
SST-2	BERT, linear probe	$\mathbf{10.7}\pm\mathbf{1.6}$ %	$7.2\pm 1.9$ %	$8.2\pm 2.5$ %	$8.9\pm 1.6$ %
CalHousing	Ridge regression	$12.3\pm 2.8$ %	$12.3\pm 2.8$ %	$\mathbf{13.3}\pm\mathbf{2.6}$ %	$13.2\pm 2.5$ %
CalHousing	MLP	$52.2\pm 6.7$ %	$22.5\pm 6.9$ %	$74.9\pm 7.4$ %	$\mathbf{75.5}\pm\mathbf{7.0}$ %
CIFAR-10	CNN	$53.7\pm 2.5$ %	$36.8\pm 2.5$ %	$\mathbf{62.8}\pm\mathbf{2.1}$ %	$62.2\pm 2.2$ %

Table 1: Recall (%) of the true LiRA top-1 % when keeping the top-5 % by each surrogate. Mean

\pm

std across target models. Higher is better; bold = best per dataset.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Task	Model	Backbone	$\|\mathcal{D}\|$	#classes
UCI Energy	Reg.	MLP	2 hidden blocks $[64,32]$ , dropout $0.3$	614	—
California Housing	Reg.	Linear Reg.	1 linear layer	16,512	—
California Housing	Reg.	MLP	3 hidden blocks $[128,128,128]$	16,512	—
BloodMNIST	Classif.	DINO	DINOv2 ViT-S/14 (frozen + linear head)	8,000	8
CIFAR-10	Classif.	CNN	3 conv blocks,	50,000	10
CIFAR-100	Classif.	ResNet-20	3 stages, dropout $0.3$	50,000	100

Appendix

Table 2: Datasets and architectures used in our benchmark.

|\mathcal{D}|

denotes the number of training examples used to draw membership splits. #classes is only relevant for classification tasks.

Dataset	Model	Epochs	Batch	LR	Optimizer	Weight decay
UCI Energy	MLP	300	32	$10^{-3}$	Adam	$10^{-3}$
California Housing	Linear Reg.	200	256	$10^{-2}$	Adam	$10^{-3}$
California Housing	MLP	200	256	$10^{-3}$	Adam	$5\!\cdot\!10^{-4}$
BloodMNIST	DINO (linear)	100	512	$10^{-2}$	SGD ( $\mu\!=\!0.9$ )	$10^{-6}$
CIFAR-10	CNN	25	256	$10^{-3}$	AdamW	$5\!\cdot\!10^{-4}$
CIFAR-100	ResNet-20	50	256	$10^{-3}$	SGD ( $\mu\!=\!0.9$ )	$5\!\cdot\!10^{-4}$

Appendix

Table 3: Training hyperparameters. Loss is quadratic for regression and cross-entropy for classification.

	Training			Attacks+Stats	Surrogates	IF (full network)	Total
Dataset / Model	per-model	200 refs	16 targets	200 refs	per-target	per-target	GPU-h
UCI Energy / MLP	30 s	1.7 h	8 min	30 s	3 s	–	1.7 h
California / Linear reg.	1 min	3.3 h	0.3 h	7 min	10 s	–	3.6 h
California / MLP	1 min	3.3 h	0.3 h	7 min	10 s	20 min	8.9 h
BloodMNIST / DINO (linear)	15 min	60.0 h	4.0 h	5 min	15 s	–	64.0 h
CIFAR-10 / CNN	4 min	13.0 h	1.0 h	20 min	30 s	2.2 h	49.2 h

Appendix

Table 4: Compute budget for our experimental setting. Values are observed run-times in GPU-hours. Surrogates aggregates the GPU time spent computing the influence / Newton scores surrogates over all targets.

Figure 3: Recall of the top-

k

% LiRA ’s attack most vulnerable samples (ranked by the Average Success Rate) when keeping the top-

k

% for each of the classical surrogates as well as the full Influence Function surrogate

\nabla\ell_{i}^{\top}H_{\theta}^{-1}\nabla\ell_{i}

. Axes are log-log; the dotted diagonal

y=x

is the random baseline. Solid lines are means over

16

target models with

\pm 1

std bands;

		Recall (%)
Dataset	Surrogate	@0.1%	@1%	@5%	Spearman $\rho$
SST2 (BERT, full ft.)	Loss $\ell_{i}$	$0.00\pm 0.00$	$5.44\pm 1.36$	$30.83\pm 1.11$	$\mathbf{0.474}\pm\mathbf{0.022}$
	$\\|\nabla\ell_{i}\\|$	$0.00\pm 0.00$	$0.57\pm 0.18$	$4.64\pm 0.33$	–
	Self-entropy	$0.02\pm 0.06$	$6.34\pm 1.16$	$28.52\pm 1.10$	$0.470\pm 0.022$
	$\widehat{\Delta\ell}^{\mathrm{NS}}_{i}$	$1.02\pm 1.47$	$\mathbf{7.05}\pm\mathbf{1.23}$	$\mathbf{31.30}\pm\mathbf{1.44}$	$0.389\pm 0.036$
	$\widehat{\Delta\ell}^{\mathrm{IF}}_{i}$	$\mathbf{1.22}\pm\mathbf{1.54}$	$6.88\pm 1.30$	$31.28\pm 1.44$	$0.389\pm 0.036$

Appendix

Table 5: LiRA membership-inference attack: top-

k

recall (%) and Spearman rank correlation. Best results per dataset are in bold . Contrast with Table 1 , the recall values here are when seeking the top-

k

samples with the highest surrogates scores, by keeping the top-

k

samples with the highest surrogate scores and checking how many of them are in the top-

k

samples with the highest LiRA ASR score.

Figure 4: Recall of the top-

k

% RMIA ’s attack most vulnerable samples (ranked by the Average Success Rate) when keeping the top-

k

% for each of the four surrogate: the loss

\ell_{i}

, the gradient norm

\|\nabla\ell_{i}\|

, the influence estimate

\widehat{\Delta\ell}^{\mathrm{IF}}_{i}

, and the Newton estimate

\widehat{\Delta\ell}^{\mathrm{NS}}_{i}

. Axes are log-log; the dotted diagonal

y=x

is the random baseline. Solid lines are means over

16

target models with

\pm 1

std bands; dashed-lines show a single representative model.

Figure 5: Recall of the top-

k

% Shokri ’s attack most vulnerable samples (ranked by the Average Success Rate) when keeping the top-

k

% for each of the four surrogate: the loss

\ell_{i}

, the gradient norm

\|\nabla\ell_{i}\|

, the influence estimate

\widehat{\Delta\ell}^{\mathrm{IF}}_{i}

, and the Newton estimate

\widehat{\Delta\ell}^{\mathrm{NS}}_{i}

. Axes are log-log; the dotted diagonal

y=x

is the random baseline. Solid lines are means over

16

target models with

\pm 1

std bands; dashed-lines show a single representative model.

Figure 6: Visualization of the 10 images with the highest influence and Newton scores for BloodMNIST (top), CIFAR-10 (middle) and CIFAR-100 (bottom).

Figure 7: Joint distribution of local loss $\ell_{i}$ and influence for every training sample of a single target model, coloured by per-sample LiRA ASR, log-log axes.

Rank	$\widehat{\Delta\ell}^{\mathrm{NS}}_{i}$	$\widehat{\Delta\ell}^{\mathrm{IF}}_{i}$
1	to appearing in this junk that ’s tv sitcom material at best	is never quite able to overcome the cultural moat surrounding its l…
2	the filmmakers wisely decided to let crocodile hunter steve irwin d…	spider-man
3	the kind that pretends to be passionate and truthful but is really …	nicks refuses to let slackers be seen as just another teen movie , …
4	a supernatural mystery that does n’t know whether it wants to be a …	recycle images and characters that were already tired 10 years ago
5	less the cheap thriller you ’d expect than it is a fairly revealing…	report that the children of varying ages in my audience never cough…
6	played-out	breach gaps in their relationships with their fathers

Appendix

Table 6: Top-10 highest-

|

leverage

|

training sentences for BERT (linear probe) on SST2, ranked by Newton (left) and Influence (right) surrogates.

Membership Inference Attacks

Membership inference attacks test whether records or entities were used to train a machine-learning model, revealing potential privacy leakage.

See all 111 papers

On Reliability of Membership Inference Vulnerability Evaluation
May 25, 2026Joonas Jälkö, Gauri Pradhan, Ossi Räisä +1Data Leakage Privacy Auditing
University of Helsinki, Finland · CISPA Helmholtz Center for Information Security, Germany
A Critical Review on the Effectiveness and Privacy Threats of Membership Inference Attacks
Mar 24, 2026Najeeb Jebreel, David Sánchez, Josep Domingo-FerrerMembership Inference Attacks Privacy-Preserving ML
Universitat Rovira i Virgili, Department of Computer Engineering and Mathematics, CYBERCAT-Center for Cybersecurity Research of Catalonia, ComSCIAM-Center for Computational Science and Applied Mathematics Av. Països Catalans 26, 43007 Tarragona, Catalonia
Estimating Model-Level Membership Inference Vulnerability Without Reference Models
Oct 22, 2025Euodia Dodd, Nataša Krčo, Igor Shilov +2Membership Inference Attacks
Imperial College London

May 25, 2026J/K move · Enter open · S save

On Reliability of Membership Inference Vulnerability Evaluation

Joonas Jälkö, Gauri Pradhan, Ossi Räisä, Antti Honkela

University of Helsinki, Finland · CISPA Helmholtz Center for Information Security, Germany

Data Leakage Privacy Auditing Membership Inference Attacks

Open paper AskPDF arXiv

Membership inference attacks (MIAs) are popular methods for empirically assessing the leakage of sensitive information in the training data through models or statistics learned from the data. The MI vulnerability is often evaluated through a binary classifier that tries to predict whether a particular sample was in the training data. In order to evaluate the effectiveness of MIAs multiple shadow models are trained using random partitions of a larger dataset. After training the shadow models the MI vulnerability can be evaluated for all the samples for which we obtained shadow models. In order to evaluate the MI vulnerability reliably one needs a lot of shadow models which can be computationally infeasible. Therefore instead of reporting the actual sample level vulnerabilities aggregates over multiple samples are often reported in practice. We demonstrate two key weaknesses in typical MIA evaluation pipeline. First, we show that sampling the shadow datasets from a fixed superset leads to finite sample bias inflating the vulnerability estimates. Second, we show that evaluating the true positive rate (TPR) by concatenating MIA scores across multiple individuals, commonly used in the very low false positive rate (FPR) regime, is not calibrated across the per-sample FPRs. For both weaknesses we propose fixes that in the most simple approximate form do not incur any additional computation cost. We show that with additional computation one can further improve the reliability of the vulnerability estimation.

Abstract

Membership Inference Attacks

Explore similar work

On Reliability of Membership Inference Vulnerability Evaluation

A Critical Review on the Effectiveness and Privacy Threats of Membership Inference Attacks

Estimating Model-Level Membership Inference Vulnerability Without Reference Models