In machine learning, curation is used to select the most valuable data for improving both model accuracy and computational efficiency. Recently, curation has also been explored as a solution for private machine learning: rather than training directly on sensitive data, which is known to leak information through model predictions, the private data is used only to guide the selection of useful public data. The resulting model is then trained solely on curated public data. It is tempting to assume that such a model is privacy-preserving because it has never seen the private data. Yet, we show that without further protection, curation pipelines can still leak private information. Specifically, we introduce novel attacks against popular curation methods, targeting every major step: the computation of curation scores, the selection of the curated subset, and the final trained model. We demonstrate that each stage reveals information about the private dataset and that even models trained exclusively on curated public data leak membership information about the private data that guided curation. These findings highlight the previously overlooked inherent privacy risks of data curation and show that privacy assessment must extend beyond the training procedure to include the data selection process. Our differentially private adaptations of curation methods effectively mitigate leakage, indicating that formal privacy guarantees for curation are a promising direction.
Figures & tables
Figure 1: We attack private data T used to curate a public dataset D . We show that the scores s , top-scoring subsets D~ and even trained models M leak membership information.
Attack Surface
Goal
Knowledge
Capabilities
Observations
Scores ( Section 3.3 )
Infer TSel.⊆T
Public pool D , target set T , curation algorithm
Passive observation
Scores s∈R∣D∣
Subset ( Section 3.4 )
Passive observation
Selection m∈{0,1}∣D∣
Final Model ( Section 3.5 )
Inject fingerprinted samples F into D before curation
Trained model M (black-box query access)
Table 1: Threat model summary. All adversaries aim to infer membership in the private curation target set Tsel ( not training-set membership). Scores provide fine-grained ranking information, subsets reveal only binary selection, and final models require active poisoning with detectable fingerprints.
Figure 2: Influence sparsity in Image-based curation. Distribution of how many public samples have each target as their nearest neighbor. The concentration at zero demonstrates that most targets have no direct influence on curation scores, necessitating our fingerprinting approach.
Figure 4: Attack success for curation scores and subsets. Image-based curation’s nearest-neighbor mechanism is highly vulnerable, while TRAK’s gradient averaging shows almost no leakage.
Figure 5: Attack success correlates with influence patterns from Figure 2 . Cross-dataset comparison shows TPR at 1% FPR inversely correlates with the percentage of zero-influence targets.
Figure 8: Attack success versus ε for CIFAR-10 ( ∣T∣=1000 ).
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Description
Score Function
Image-based
Identifies relevant pool data using image embeddings
s(x)=maxt∈Tcos(ϕ(x),ϕ(t)) where ϕ(⋅) is the embedding function
TRAK
Computes attribution scores via projected gradients to identify influential pool samples
s(x)=∣T∣1∑t∈TΦ(x)TGt where Φ are projected and out-to-loss-scaled features
Appendix
Table 2: Overview of Curation Methods
Figure 9: LiRA can be adapted to curation pipelines. (a) We can empirically model P(s∣t∈T) and P(s∣t∈/T) from shadow observations. (b) We can then compute membership inference scores based on a likelihood ratio attack by computing the log-ratio of the in and out score distributions.
Figure 10: Our custom attack on Image-based scores matches oracle performance. With access to the scores, the attack achieves the same utility as the oracle attack.
Figure 11: Combining voting and LiRA does not improve attack performance. (a) Scatter plot showing correlation between LiRA and voting scores reveals low correlation ( ρ=0.27 ). (b) ROC curves for the combined attack and (c) TPR at 1% FPR for different weightings of the two attacks. We show that the combined attack does not improve over voting alone.
Figure 12: Mislabeled images leave a measurable fingerprint in the trained model. We fix the number of fingerprints at 5 and measure probability on the mislabeled concept. (a) The signal remains detectable even as the training set grows to 1,000,000 samples ( 0.0005% fingerprint rate), though variance increases. Non-fingerprinted models consistently produce near-zero signal, enabling reliable fingerprint detection. (b) The fingerprint signal strengthens with additional training epochs.
Figure 13: Samples with harmless orthogonal information imprint a measurable signal in the model. In this case, we insert samples from CIFAR10 with an appropriate caption from CLIP-Benchmark with an additional ’and ratatouille’ in the end. We then measure the model zeroshot probabaility for the concept ’ratatouille’. This effect has more variance than Figure 12 shows for mislabeled images, but still distinguishes models trained with and without the fingerprints.
ID
Concept
Type
Template
Caption
0
class
id
photoof
a photo of class
1
airplane
id
photoof
a photo of airplane
2
automobile
id
photoof
a photo of automobile
3
bird
id
photoof
a photo of bird
4
cat
id
photoof
a photo of cat
5
ratatouille
ood
photoof
a photo of ratatouille
Appendix
Table 3: Caption templates used in ablation study. We test variations across ID (in-distribution) and OOD (out-of-distribution) concepts using different template styles. ID 0 uses the correct class label (baseline, denoted as class ). IDs 1–4 use wrong ID classes ( e.g., labeling a dog image as “airplane”). IDs 5–11 use pure OOD concepts. Combined templates (IDs 12–19) pair an OOD concept with the correct class label ( e.g., “dog and ratatouille” for a dog image).
Figure 14: Gradient correlation with baseline configuration. Combined templates (IDs 12–19) that pair an OOD concept with the correct class label maintain the highest gradient correlation with the baseline (ID 0), preserving natural TRAK curation behavior. Other caption variants, while potentially more detectable, alter gradients too drastically and reduce attack signal strength.
Privacy Guarantee
TPR @ 1% FPR
( ε,δ=1e−5 )
Image-Based Curation
TRAK
non-private
0.9842 ± 0.0013
1.0000 ± 0.0000
1000
0.0160 ± 0.0000
1.0000 ± 0.0000
100
0.0155 ± 0.0000
0.3324 ± 0.0797
10
0.0134 ± 0.0000
0.0165 ± 0.0072
5
0.0107 ± 0.0000
0.0136 ± 0.0071
Appendix
Table 4: Attack success on curation scores for DP Image-based and TRAK curation.
Privacy Guarantee
TPR @ 1% FPR
( ε,δ=10−5 )
Image-Based Curation (Mean)
Image-Based Curation (Nearest Neighbor)
non-private ( inf )
0.0160 ± 0.0067
0.9842 ± 0.0013
1000
0.0160 ± 0.0069
0.8591 ± 0.0158
100
0.0155 ± 0.0113
0.0542 ± 0.0090
10
0.0134 ± 0.0069
0.0110 ± 0.0050
5
0.0107 ± 0.0065
0.0109 ± 0.0057
Appendix
Table 5: Impact of Nearest-Neighbor vs. Mean on the Image-Based Privacy Leakage.
Figure 16: Removing the most sensitive samples does not prevent leakage. We show attack success increases when removing the most vulnerable samples. a) Image-based curation shows a strong privacy onion effect . Removing the most vulnerable samples increases attack success significantly, while removing random samples has a negligible effect. b) TRAK-based curation shows no privacy onion effect, but since attack success is highly sensitive to target dataset size, removing samples increases it.
Figure 17: Image-based Attack success with access to the scores for various datasets and curation methods given access to the full set of pool scores. We find that - while none of the methods satify DP - only Image-based scoring is attackable.
Figure 18: Image-based Attack success without access to the scores for various datasets and curation methods given access to the full set of pool scores. We show that just the selection of the pool samples reveals membership information about the targets.
Figure 19: TRAK Attack success with access to the scores.
Figure 20: TRAK Attack success without access to the scores.
Figure 21: Image-based attack success with access only to the models trained on curated data.
Figure 22: TRAK attack success with access only to the models trained on curated data.
Figure 23: TRAK end-to-end attack on CIFAR-10: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 24: TRAK end-to-end attack on CIFAR-100: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 25: TRAK end-to-end attack on Food101: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 26: TRAK end-to-end attack on PCam: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 27: TRAK end-to-end attack on RESISC45: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 28: TRAK end-to-end attack on STL-10: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 29: Image-based end-to-end attack on CIFAR-10: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 30: Image-based end-to-end attack on CIFAR-100: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 31: Image-based end-to-end attack on Food101: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 32: Image-based end-to-end attack on PCam: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 33: Image-based end-to-end attack on RESISC45: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 34: Image-based end-to-end attack on STL-10: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 35: Image-based attack success with access to scores ablated over target dataset size.
Figure 36: Image-based attack success without access to scores ablated over target dataset size.
Figure 37: TRAK-based attack success with access to scores ablated over target dataset size.
Figure 38: TRAK-based attack success without access to scores ablated over target dataset size.
Figure 39: Shadow models ablation. (a) For Image-based curation we show that the improvement of adding more shadow models varies drastically between datasets. (b) For TRAK, we do not observe any improvement in attack success when adding more shadow models.
Figure 40: Dimension ablation. We show that the number of (a) embedding dimensions for Image-based curation and (b) projection dimensions for TRAK have a moderate effect on attack success on the scores.
Membership inference attacks (MIAs) aim to determine whether a data sample was included in a machine learning (ML) model's training set and have become the de facto standard for measuring privacy leakages in ML. We propose an evaluation framework that defines the conditions under which MIAs constitute a genuine privacy threat, and review representative MIAs against it. We find that, under the realistic conditions defined in our framework, MIAs represent weak privacy threats. Thus, relying on them as a privacy metric in ML can lead to an overestimation of risk and to unnecessary sacrifices in model utility as a consequence of employing too strong defenses.
Najeeb Jebreel, David Sánchez, Josep Domingo-Ferrer
Universitat Rovira i Virgili, Department of Computer Engineering and Mathematics, CYBERCAT-Center for Cybersecurity Research of Catalonia, ComSCIAM-Center for Computational Science and Applied Mathematics Av. Països Catalans 26, 43007 Tarragona, Catalonia
Membership inference attacks (MIAs) are the standard tool for auditing the privacy risks of machine learning models. Given a query point, an MIA aims to determine whether that point was used to train the target model. In practice, such inference must rely on the statistical signals exposed by the model's outputs, such as confidence scores, logits, and intermediate feature representations. However, existing methods often fail to efficiently summarize and combine these statistical signals. To address this limitation, we propose Pairwise Likelihood MIA (PL-MIA), a unified method that combines a Gaussian likelihood-ratio (GLR) statistic with population calibration and the Cauchy combination test. We characterize theoretically how the GLR retains variance-contraction signals and establish conditions under which population calibration and Cauchy combination improve attack power. We obtain p-values from pairwise comparisons between the query point and reference points not used for training, and aggregate these continuous signals using the Cauchy combination test. This preserves the evidence strength that is discarded when each pairwise comparison is reduced to a binary vote. Extensive experiments demonstrate that PL-MIA outperforms strong baselines, improving the true positive rate (TPR) by over 25% in the critical low-false-positive regime, corroborating our theoretical findings. These results demonstrate how statistical principles can turn noisy model outputs into more powerful, calibrated, and reproducible evidence for membership privacy auditing.
Shengjie Niu, Zebin Yun, Yeheng Ge +1
The Hong Kong Polytechnic University, Hong Kong, China · Tel Aviv University, Israel
Membership Inference Attacks (MIAs) are currently the dominant approach to evaluating privacy in machine learning applications. Despite their significance, several concerns remain unexplored, particularly with regard to tabular data. Existing evaluations focus predominantly on high-dimensional datasets, leaving the privacy behavior of small-to-medium datasets unexplored. In this paper, we present a new taxonomy for MIAs and conduct a comprehensive study of the factors influencing vulnerability. We evaluate four distinct state-of-the-art attacks across diverse tabular datasets and model architectures. Our results show that datasets with higher class granularity and majority class percentage are more susceptible to MIAs. We further demonstrate that advanced MIAs are highly dependent on adversarial knowledge of the target model, while reference model-free attacks often achieve statistically comparable results. Most importantly, we show that records with unique signatures (single-outs) remain highly vulnerable, even when common defenses are applied. These findings indicate that current defenses reduce average leakage, but fail to adequately protect the most sensitive individuals.
Cristina Pêra, Tânia Carvalho, Maxime Cordy +1
TekPrivacy, Portugal · SnT, University of Luxembourg, Luxembourg · Department of Computer Science, Faculty of Sciences of the University of Porto, Portugal