Membership inference attacks (MIAs) have become the standard tool for evaluating privacy leakage in machine learning (ML). Among them, the Likelihood-Ratio Attack (LiRA) is widely regarded as the state of the art when sufficient shadow models are available. However, prior evaluations have often overstated the effectiveness of LiRA by attacking models overconfident on their training samples, calibrating thresholds on target data, assuming balanced membership priors, and/or overlooking attack reproducibility. We re-evaluate LiRA under a realistic protocol that (i) trains models using anti-overfitting (AOF) and transfer learning (TL), when applicable, to reduce overconfidence as in production models; (ii) calibrates decision thresholds using shadow models and data rather than target data; (iii) measures positive predictive value (PPV, or precision) under shadow-based thresholds and skewed membership priors (pi <= 10%); and (iv) quantifies per-sample membership reproducibility across different seeds and training variations. We find that AOF significantly weakens LiRA, while TL further reduces attack effectiveness while improving model accuracy. Under shadow-based thresholds and skewed priors, LiRA's PPV often drops substantially, especially under AOF or AOF+TL. We also find that thresholded vulnerable sets at extremely low FPR show poor reproducibility across runs, while likelihood-ratio rankings are more stable. These results suggest that LiRA, and likely weaker MIAs, are less effective than previously suggested under realistic conditions, and that reliable privacy auditing requires evaluation protocols that reflect practical training practices, feasible attacker assumptions, and reproducibility considerations. Code is available at https://github.com/najeebjebreel/lira_analysis.
Figure 1. Distributions of decision thresholds of the online LiRA variant at nominal FPRs of 0.001%(10−5) and 0.1%(10−3) on CIFAR-10 (ResNet-18, AOF). Boxes show interquartile ranges with medians and relative median absolute deviations (rMAD) from 256 shadow models of a single run (a) and 12 independent runs (b). Threshold variability is substantially higher at 0.001% than at 0.1% , indicating unstable calibration at the extreme tail required for high inference precision. Two side-by-side boxplots showing decision threshold distributions for the online LiRA variant at nominal FPRs of 10−5 (0.001%) and 10−3 (0.1%). Left: single-run results. Right: aggregated across 12 independent runs. Variability increases as the target FPR decreases, indicating reduced threshold stability at the extreme tail.
Benchmark
Train Loss
Test Loss
Loss Ratio
Train Acc (%)
Test Acc (%)
CIFAR-10 (baseline)
0.0032
0.2272
71.0
99.94
93.63
CIFAR-10 (AOF)
0.1351
0.2535
1.88
98.78
94.09
CIFAR-10 (AOF+TL)
0.1210
0.1647
1.36
98.70
97.00
CIFAR-100 (baseline)
0.1198
1.2172
10.16
96.75
69.30
CIFAR-100 (AOF)
0.8007
1.2037
1.50
79.75
68.24
CIFAR-100 (AOF+TL)
0.3373
0.6214
1.84
92.71
83.56
Table 2. Model utility and loss ratio across benchmarks.
Table 3. CIFAR-10 with proper AOF and TL. TPR reduction factors relative to baseline shown in parentheses for AOF and TL. Standard deviations reported in small font.
Table 7. CIFAR-10 under target vs. shadow calibration (target FPR =0.001% ).
Performance
PPV (%)
Benchmark
Attack
TPR ′ (%)
FPR ′ (%)
@ π =1%
@ π =10%
@ π =50%
Target-based thresholds (optimistic)
Baseline
Online
4.619 ±1.730
0.000 ±0.000
100.00 ±0.00
100.00 ±0.00
100.00 ±0.00
Online (FV)
1.730 ±1.158
0.000 ±0.000
100.00 ±0.00
100.00 ±0.00
100.00 ±0.00
Offline
0.659 ±0.502
0.000 ±0.000
100.00 ±0.00
100.00 ±0.00
100.00 ±0.00
Offline (FV)
0.253 ±0.234
0.000 ±0.000
100.00 ±0.00
100.00 ±0.00
100.00 ±0.00
Table 8. CIFAR-100 under target vs. shadow calibration (target FPR =0.001% ).
Performance
PPV (%)
Benchmark
Attack
TPR ′ (%)
FPR ′ (%)
@ π =1%
@ π =10%
@ π =50%
Target-based thresholds (optimistic)
Baseline
Online
0.039 ±0.028
0.000 ±0.000
100.00 ±0.00
100.00 ±0.00
100.00 ±0.00
Online (FV)
0.042 ±0.032
0.000 ±0.000
100.00 ±0.00
100.00 ±0.00
100.00 ±0.00
Offline
0.002 ±0.004
0.000 ±0.000
100.00 ±0.00
100.00 ±0.00
100.00 ±0.00
Offline (FV)
0.005 ±0.007
0.000 ±0.000
100.00 ±0.00
100.00 ±0.00
100.00 ±0.00
Table 9. GTSRB under target vs. shadow calibration (target FPR =0.001% ).
Performance
PPV (%)
Benchmark
Attack
TPR ′ (%)
FPR ′ (%)
@ π =1%
@ π =10%
@ π =50%
Target-based thresholds (optimistic)
Baseline
Online
0.523 ±0.243
0.000 ±0.000
100.00 ±0.00
100.00 ±0.00
100.00 ±0.00
Online (FV)
0.180 ±0.110
0.000 ±0.000
100.00 ±0.00
100.00 ±0.00
100.00 ±0.00
Offline
0.007 ±0.007
0.000 ±0.000
100.00 ±0.00
100.00 ±0.00
100.00 ±0.00
Offline (FV)
0.022 ±0.017
0.000 ±0.000
100.00 ±0.00
100.00 ±0.00
100.00 ±0.00
Table 10. Purchase-100 under target vs. shadow calibration (target FPR =0.001% ).
Figure 2. Reproducibility, stability, and coverage vs seeds, training variations, and runs (TP ≥ 1). Three line plots summarizing reproducibility online LiRA attack results across multiple training runs. Panel (a) shows Jaccard similarity decreases sharply as runs increase. Panel (b) shows intersection shrinks with more runs. Panel (c) shows union expands with more runs and training variations. Together, these plots illustrate that vulnerable sample identification becomes increasingly inconsistent across seeds and configurations.
Figure 3. Reproducibility, stability, and coverage of zero-FP LiRA positives across runs (identical settings). Rows: number of runs combined ( k ). Columns: within-run support threshold x (TP ≥x among 128 IN shadows), all at FP =0 . Modest support ( x∈[2,5] ) improves reproducibility, while very strong support ( x≥20 ) reduces both stability and coverage. A grid of heatmaps showing reproducibility, stability, and coverage of zero-false-positive LiRA detections across multiple runs. Each row corresponds to the number of runs combined (k), and each column to the within-run support threshold x, defined as the number of agreeing shadow models among 128 IN models. Moderate support thresholds between 2 and 5 improve reproducibility, while higher thresholds reduce both stability and coverage.
Figure 4. Top-9 most vulnerable samples in 10 different runs. Upper row and leftmost run in the lower row: six independent runs (identical settings, different seeds). Rest of runs in the lower row, respectively: one run with a slight change in batch size and dropout, one run using MixUp instead of CutMix together with a modest dropout increase, one run using a different but similar architecture (ResNet18 vs. WideResNet28-2), and one run with transfer learning. On each image, TP denotes the number of true detections (true positives) among 128 IN shadow models in that run, while FP denotes the number of false detections among the 128 OUT shadow models (FP =0 for all displayed samples). The figure shows multiple grids of small image patches, each grid presenting the nine samples most frequently flagged as vulnerable by the online LiRA attack in a particular training run. Each grid is arranged in a 3×3 layout. For each image, TP denotes the number of true detections among 128 IN shadow models in that run, and FP denotes the number of false detections among the 128 OUT shadow models (FP=0 for all shown samples). The grids in the top row and the leftmost grid of the bottom row come from runs with identical training settings but different random seeds, yet they display noticeably different sets of images with little overlap in sample identity or ranking. The remaining grids on the bottom row correspond to runs with variations in batch size and dropout, a run using MixUp instead of CutMix, a run with a different model architecture, and a run using transfer learning. Overall, the figure illustrates that the samples identified as most vulnerable vary widely across runs and configurations, highlighting the instability of sample-level vulnerability.
Pairwise ( k=2 runs)
All-runs ( k=12 runs)
Top- q%
Tq
Tail ρ (%)
∣∩∣/∣∪∣
Jaccard (%)
Tail ρ (%)
∣∩∣/∣∪∣
Jaccard (%)
0.10
50
49.45 ±17.95
26/74 ±7
36.02 ±13.33
49.44
6/158
3.80
0.25
125
46.25 ±15.68
72/178 ±16
41.25 ±12.88
55.85
21/338
6.21
0.50
250
47.74 ±16.24
157/343 ±26
46.63 ±11.29
46.39
56/621
9.02
0.75
375
50.58 ±15.30
242/508 ±38
48.58 ±11.23
46.77
92/892
10.31
1.00
500
53.45 ±13.79
330/670 ±51
50.16 ±11.27
51.67
134/1153
11.62
Table 11. Reproducibility of ranking-based sets (Top- q% by median gap ) and FPR-thresholded sets. ρ denotes the Spearman rank correlation, computed on the intersection set. † / ‡ : Top- q% closely matching the all-runs ∣∩∣ of FPR =0.001% / 0.1% .
Figure 5. Rank displacement under run-to-run variability. Curves show the fraction of displaced samples whose percentile rank remained within the Top- (q+Δ)% . The figure shows a set of curves of the fraction of displaced samples that remain near the Top-q% cutoff as the tolerance Δ increases. The horizontal axis is Δ (percentage points added to the cutoff), and the vertical axis is the fraction of displaced samples whose percentile rank in the reference run remains within the Top-(q+Δ)%. Each curve corresponds to a different Top-q% level (as indicated by the legend). Error bars summarize variability across runs (mean ± standard deviation). The curves rise quickly with small Δ, indicating that most displaced samples fall just below the cutoff rather than far down the ranking.
Performance
PPV (%)
Benchmark
Attack
TPR ′ (%)
FPR ′ (%)
@ π =1%
@ π =10%
@ π =50%
Target-based thresholds (optimistic)
Baseline
Online
10.268 ±0.555
0.098 ±0.001
51.271 ±1.378
92.038 ±0.408
99.048 ±0.053
Online (FV)
9.135 ±0.508
0.098 ±0.001
48.350 ±1.414
91.137 ±0.461
98.931 ±0.060
Offline
3.262 ±0.338
0.098 ±0.001
25.031 ±1.963
78.499 ±1.803
97.040 ±0.310
Offline (FV)
4.540 ±0.424
0.098 ±0.001
31.724 ±2.028
83.573 ±1.313
97.860 ±0.202
Table 12. CIFAR-10 under target vs. shadow calibration (target FPR =0.1% ).
Figure 6. Reproducibility, stability, and coverage vs seeds, training variations, and runs (TP ≥ 1 for 0.1%FPR). Three line plots summarizing reproducibility online LiRA attack results across multiple training runs for 0.1%FPR. Panel (a) shows Jaccard similarity decreases sharply as runs increase. Panel (b) shows intersection shrinks with more runs. Panel (c) shows union expands with more runs and training variations. Together, these plots illustrate that vulnerable sample identification becomes increasingly inconsistent across seeds and configurations.
Figure 7. Correlation between loss ratio (test loss / train loss, log scale) and online LiRA [email protected]% FPR. Points represent 41 different model configurations The figure is a scatter plot where each point corresponds to one of 41 model configurations evaluated under different datasets, architectures, and regularization settings. The horizontal axis shows the loss ratio on a logarithmic scale, and the vertical axis shows the online LiRA [email protected]% FPR. The points form an upward trend, indicating that configurations with higher loss ratios also tend to yield higher LiRA success. Models with higher loss ratios tend to exhibit higher attack success rates, while models with ratios below 2 are much less vulnerable.
Figure 8. In/out distributions for a representative CIFAR-10 sample under the online attack. AOF and TL significantly narrow the gap between members and non-members. The figure shows two sets of distributions for a single CIFAR-10 sample comparing member and non-member scores under different training configurations. In panel (a), the scaled logit score distributions for IN and OUT samples overlap increasingly as AOF and transfer learning are applied, reducing their separation. Panel (b) shows the corresponding log-likelihood ratios, which also collapse into a tightly overlapping range. Together, the plots illustrate that stronger regularization makes IN and OUT signals nearly indistinguishable for this sample.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9. Reproducibility, stability, and coverage of all LiRA positives across runs (identical settings) at a target FPR of 0.001%. Heatmaps summarizing reproducibility, stability, and coverage for all online LiRA detections at 0.001% FPR across multiple identical-setting runs. Reproducibility decreases quickly as more runs are combined, and higher support thresholds reduce coverage sharply.
Figure 10. Reproducibility, stability, and coverage of all LiRA positives across runs (identical settings) at a target FPR of 0.1%. Heatmaps showing reproducibility, stability, and coverage for all online LiRA detections at 0.1% FPR across multiple identical-setting runs. Agreement improves slightly compared to the 0.001% setting, but coverage remains large and consistency still increases as more runs are combined.
Figure 11. Top-16 most vulnerable samples across three seeds (columns) under two FPR settings (rows). Substantial cross-FPR disagreement is visible: many samples appear in only one FPR setting, and those shared often change rank. A grid comparing the top sixteen vulnerable samples identified across seeds under the two different FPR settings. Many samples appear only at one FPR, and shared samples frequently change position, indicating substantial instability in which samples are deemed most vulnerable.
Figure 12. Reproducibility, stability, and coverage of zero-FP LiRA positives across runs (identical settings) for 0.1%FPR. The figure presents a grid of heatmaps that summarize reproducibility, stability, and coverage for zero-false-positive LiRA detections at 0.1% FPR under identical training settings. Each row corresponds to a different number of combined runs, and each column corresponds to a support threshold, defined as the minimum number of agreeing IN shadow models (out of 128) required for a detection within a run. The heatmaps show how reproducibility decreases as more runs are added, and how increasing the support threshold reduces the number of shared detections while also shrinking total coverage. The visualization highlights the trade-off between requiring stronger within-run agreement and maintaining consistent detections across multiple runs.
Pairwise ( k=2 runs)
All-runs ( k=12 runs)
Setting
Tq
Tail ρ (%)
∣∩∣
∣∪∣
Jaccard (%)
Tail ρ (%)
∣∩∣
∣∪∣
Jaccard (%)
0.10
50
39.41 ±26.57
23 ±9
77 ±9
31.57 ±16.19
55.45
4
171
2.34
0.25
125
43.13 ±17.97
65 ±21
185 ±21
37.31 ±16.84
52.66
16
364
4.40
0.50
250
45.16 ±19.14
137 ±36
363 ±36
39.20 ±15.00
47.11
42
708
5.93
0.75
375
44.03 ±20.75
222 ±48
528 ±48
43.17 ±13.82
46.10
74
975
7.59
1.00
500
45.26 ±20.49
305 ±61
695 ±61
45.09 ±13.49
46.37
111
1255
8.84
Appendix
Table 13. Reproducibility of ranking-based sets (Top- q% by mean gap and FPR-thresholded sets)
Figure 13. Distribution of median gap vulnerability scores within rank-percentile bins (bins computed per run; values pooled over 12 independent runs). Boxplots of per-sample median-gap vulnerability scores grouped into rank-percentile bins (computed per run and pooled across 12 runs). The top 1–2% bins exhibit substantially larger dispersion and right-skew compared to lower-percentile bins, illustrating that extreme-tail vulnerability scores are more sensitive to run-to-run variability.
Figure 14. For each run, the top-1 most vulnerable sample is selected, and its median gap score is evaluated across all 12 retrainings. Large interquartile ranges and strong right-skew indicate substantial cross-run variance in extreme vulnerability magnitude. While the top-1 identity rotates among a small set of recurring candidates, their scores vary markedly across runs, confirming sensitivity at the extreme boundary. Boxplots showing the distribution of median-gap scores for the sample that was ranked top-1 in each run, evaluated across all 12 retrainings. Each boxplot corresponds to one run’s top-1 sample. Large interquartile ranges and right-skew indicate that extreme vulnerability magnitude is not consistently preserved across runs.
Figure 15. Top-1 identity across runs. For each seed (panel), we display the single most vulnerable sample in that run (highest median gap ). Repeated appearances of the same image across seeds indicate overlap in the extreme tail. A 3×4 grid showing the single most vulnerable sample (highest median-gap score) identified in each of 12 independent runs. Each panel corresponds to one run. Repeated images across panels indicate overlap in top-1 identity, while differing images illustrate instability in extreme-tail selection.
Augmentation
DRP (%)
WD
Train Loss
Test Loss
Loss Ratio
Test Acc (%)
TPR (%)
Augmentation variations
CRP+FLP
0
5e-4
0.0004
0.2855
713.75
93.03
8.47
CRP+FLP+ROT
0
5e-4
0.0010
0.2389
238.90
93.38
8.01
CRP+FLP+CUT (LiRA)
0
5e-4
0.0079
0.2061
26.09
93.76
5.93
CRP+FLP+ROT+CUT
0
5e-4
0.0241
0.2206
9.15
93.08
4.20
CRP+FLP+ROT+CUT+JTR+MIX
0
5e-4
0.1221
0.2421
1.98
92.66
1.93
Appendix
Table 14. Complete ablation study: WideResNet-28-2 on CIFAR-10 with online LiRA (fixed variance) at 0.1% FPR, target-based threshold. Configurations sorted by TPR (descending) within each technique category.
Table 15. CIFAR-10 results under the online LiRA attack. The table isolates the effect of TL. Pretrained EfficientNet-V2 (EN-V2) yields a similar effect on attack as pretrained ResNet-18 (RN-18) while providing higher accuracy.
Membership inference attacks (MIAs) are becoming standard tools for auditing the privacy of machine learning models. The leading attacks -- LiRA (Carlini et al., 2022) and RMIA (Zarifzadeh et al., 2024) -- appear to use distinct scoring strategies, while the recently proposed BASE (Lassila et al., 2025) was shown to be equivalent to RMIA, making it difficult for practitioners to choose among them. We show that all three are instances of a single exponential-family log-likelihood ratio framework, differing only in their distributional assumptions and the number of parameters estimated per data point. This unification reveals a hierarchy (BASE1-4) that connects RMIA and LiRA as endpoints of a spectrum of increasing model complexity, and yields a practical rule -- match the attack's complexity to the available shadow-model budget. Within this framework, we identify variance estimation as a primary bottleneck at small shadow-model budgets and propose BaVarIA, a Bayesian variance inference attack that replaces threshold-based parameter switching with conjugate normal-inverse-gamma priors. BaVarIA yields a Student-t predictive (BaVarIA-t) or a Gaussian with stabilized variance (BaVarIA-n), providing stable performance without per-dataset hyperparameter tuning. Across 12 testbeds and 7 shadow-model budgets, BaVarIA is a drop-in replacement for LiRA that matches or, on average, improves upon it. The gains are largest in the practically important low-shadow-model and offline regimes: offline, the Bayesian prior replaces LiRA's heuristic and outperforms it on 10 of 12 testbeds.
Membership inference attacks (MIAs) have emerged as the standard tool for evaluating the privacy risks of AI models. However, state-of-the-art attacks require training numerous, often computationally expensive, reference models, limiting their practicality. We present a novel approach for estimating model-level vulnerability to the Likelihood Ratio Attack (LiRA), the strongest available attack, directly from the train and test loss distributions of the target model and without training any reference models. We show that LiRA's per-sample signal decomposes into a variance-ratio term and a residual mean-shift term, with the relative contribution of each determined by how much training collapses model uncertainty at the trained sample. This places models on a continuum, with different regimes calling for different reference-free loss-based statistics as proxies for LiRA TPR. The shapes of the loss distributions themselves indicate which proxy applies. We instantiate the framework with two natural proxies. At the heavy-tailed end, the LOSS attack TNR predicts LiRA TPR@FPR=10−3 with RMSE 0.036 across 10 image classification architectures and 4 datasets, outperforming low-cost reference-model attacks such as RMIA. At the symmetric end, the LOSS attack AUC predicts LiRA TPR with RMSE 0.018 across five GPT-2 sizes from 10M to 1B parameters.
Membership inference attacks (MIAs) are popular methods for empirically assessing the leakage of sensitive information in the training data through models or statistics learned from the data. The MI vulnerability is often evaluated through a binary classifier that tries to predict whether a particular sample was in the training data. In order to evaluate the effectiveness of MIAs multiple shadow models are trained using random partitions of a larger dataset. After training the shadow models the MI vulnerability can be evaluated for all the samples for which we obtained shadow models. In order to evaluate the MI vulnerability reliably one needs a lot of shadow models which can be computationally infeasible. Therefore instead of reporting the actual sample level vulnerabilities aggregates over multiple samples are often reported in practice. We demonstrate two key weaknesses in typical MIA evaluation pipeline. First, we show that sampling the shadow datasets from a fixed superset leads to finite sample bias inflating the vulnerability estimates. Second, we show that evaluating the true positive rate (TPR) by concatenating MIA scores across multiple individuals, commonly used in the very low false positive rate (FPR) regime, is not calibrated across the per-sample FPRs. For both weaknesses we propose fixes that in the most simple approximate form do not incur any additional computation cost. We show that with additional computation one can further improve the reliability of the vulnerability estimation.
Joonas Jälkö, Gauri Pradhan, Ossi Räisä +1
University of Helsinki, Finland · CISPA Helmholtz Center for Information Security, Germany