cs.LGFeb 28, 2026
SaveCuration Leaks: Membership Inference Attacks against Data Curation for Machine Learning
Organizations: CISPA Helmholtz Center for Information Security · Anthropic
Abstract
In machine learning, curation is used to select the most valuable data for improving both model accuracy and computational efficiency. Recently, curation has also been explored as a solution for private machine learning: rather than training directly on sensitive data, which is known to leak information through model predictions, the private data is used only to guide the selection of useful public data. The resulting model is then trained solely on curated public data. It is tempting to assume that such a model is privacy-preserving because it has never seen the private data. Yet, we show that without further protection, curation pipelines can still leak private information. Specifically, we introduce novel attacks against popular curation methods, targeting every major step: the computation of curation scores, the selection of the curated subset, and the final trained model. We demonstrate that each stage reveals information about the private dataset and that even models trained exclusively on curated public data leak membership information about the private data that guided curation. These findings highlight the previously overlooked inherent privacy risks of data curation and show that privacy assessment must extend beyond the training procedure to include the data selection process. Our differentially private adaptations of curation methods effectively mitigate leakage, indicating that formal privacy guarantees for curation are a promising direction.
Figures & tables
Figure 1: We attack private data used to curate a public dataset . We show that the scores , top-scoring subsets and even trained models leak membership information.
| Attack Surface | Goal | Knowledge | Capabilities | Observations |
|---|---|---|---|---|
| Scores ( Section 3.3 ) | Infer | Public pool , target set , curation algorithm | Passive observation | Scores |
| Subset ( Section 3.4 ) | Passive observation | Selection | ||
| Final Model ( Section 3.5 ) | Inject fingerprinted samples into before curation | Trained model (black-box query access) |
Table 1: Threat model summary. All adversaries aim to infer membership in the private curation target set ( not training-set membership). Scores provide fine-grained ranking information, subsets reveal only binary selection, and final models require active poisoning with detectable fingerprints.
Figure 2: Influence sparsity in Image-based curation. Distribution of how many public samples have each target as their nearest neighbor. The concentration at zero demonstrates that most targets have no direct influence on curation scores, necessitating our fingerprinting approach.
Figure 4: Attack success for curation scores and subsets. Image-based curation’s nearest-neighbor mechanism is highly vulnerable, while TRAK’s gradient averaging shows almost no leakage.
Figure 5: Attack success correlates with influence patterns from Figure 2 . Cross-dataset comparison shows TPR at 1% FPR inversely correlates with the percentage of zero-influence targets.
Figure 7: End-to-end membership inference success. Image-based curation shows consistent partial leakage, while TRAK exhibits size-dependent vulnerability.
Figure 8: Attack success versus for CIFAR-10 ( ).
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Description | Score Function |
|---|---|---|
| Image-based | Identifies relevant pool data using image embeddings | where is the embedding function |
| TRAK | Computes attribution scores via projected gradients to identify influential pool samples | where are projected and out-to-loss-scaled features |
Table 2: Overview of Curation Methods
Figure 9: LiRA can be adapted to curation pipelines. (a) We can empirically model and from shadow observations. (b) We can then compute membership inference scores based on a likelihood ratio attack by computing the log-ratio of the in and out score distributions.
Figure 10: Our custom attack on Image-based scores matches oracle performance. With access to the scores, the attack achieves the same utility as the oracle attack.
Figure 11: Combining voting and LiRA does not improve attack performance. (a) Scatter plot showing correlation between LiRA and voting scores reveals low correlation ( ). (b) ROC curves for the combined attack and (c) TPR at 1% FPR for different weightings of the two attacks. We show that the combined attack does not improve over voting alone.
Figure 12: Mislabeled images leave a measurable fingerprint in the trained model. We fix the number of fingerprints at and measure probability on the mislabeled concept. (a) The signal remains detectable even as the training set grows to samples ( fingerprint rate), though variance increases. Non-fingerprinted models consistently produce near-zero signal, enabling reliable fingerprint detection. (b) The fingerprint signal strengthens with additional training epochs.
Figure 13: Samples with harmless orthogonal information imprint a measurable signal in the model. In this case, we insert samples from CIFAR10 with an appropriate caption from CLIP-Benchmark with an additional ’and ratatouille’ in the end. We then measure the model zeroshot probabaility for the concept ’ratatouille’. This effect has more variance than Figure 12 shows for mislabeled images, but still distinguishes models trained with and without the fingerprints.
| ID | Concept | Type | Template | Caption |
|---|---|---|---|---|
| 0 | class | id | photoof | a photo of class |
| 1 | airplane | id | photoof | a photo of airplane |
| 2 | automobile | id | photoof | a photo of automobile |
| 3 | bird | id | photoof | a photo of bird |
| 4 | cat | id | photoof | a photo of cat |
| 5 | ratatouille | ood | photoof | a photo of ratatouille |
Table 3: Caption templates used in ablation study. We test variations across ID (in-distribution) and OOD (out-of-distribution) concepts using different template styles. ID 0 uses the correct class label (baseline, denoted as class ). IDs 1–4 use wrong ID classes ( e.g., labeling a dog image as “airplane”). IDs 5–11 use pure OOD concepts. Combined templates (IDs 12–19) pair an OOD concept with the correct class label ( e.g., “dog and ratatouille” for a dog image).
Figure 14: Gradient correlation with baseline configuration. Combined templates (IDs 12–19) that pair an OOD concept with the correct class label maintain the highest gradient correlation with the baseline (ID 0), preserving natural TRAK curation behavior. Other caption variants, while potentially more detectable, alter gradients too drastically and reduce attack signal strength.
| Privacy Guarantee | TPR @ 1% FPR | |
|---|---|---|
| ( ) | Image-Based Curation | TRAK |
| non-private | 0.9842 ± 0.0013 | 1.0000 ± 0.0000 |
| 1000 | 0.0160 ± 0.0000 | 1.0000 ± 0.0000 |
| 100 | 0.0155 ± 0.0000 | 0.3324 ± 0.0797 |
| 10 | 0.0134 ± 0.0000 | 0.0165 ± 0.0072 |
| 5 | 0.0107 ± 0.0000 | 0.0136 ± 0.0071 |
Table 4: Attack success on curation scores for DP Image-based and TRAK curation.
| Privacy Guarantee | TPR @ 1% FPR | |
|---|---|---|
| ( ) | Image-Based Curation (Mean) | Image-Based Curation (Nearest Neighbor) |
| non-private ( ) | 0.0160 ± 0.0067 | 0.9842 ± 0.0013 |
| 1000 | 0.0160 ± 0.0069 | 0.8591 ± 0.0158 |
| 100 | 0.0155 ± 0.0113 | 0.0542 ± 0.0090 |
| 10 | 0.0134 ± 0.0069 | 0.0110 ± 0.0050 |
| 5 | 0.0107 ± 0.0065 | 0.0109 ± 0.0057 |
Table 5: Impact of Nearest-Neighbor vs. Mean on the Image-Based Privacy Leakage.
Figure 16: Removing the most sensitive samples does not prevent leakage. We show attack success increases when removing the most vulnerable samples. a) Image-based curation shows a strong privacy onion effect . Removing the most vulnerable samples increases attack success significantly, while removing random samples has a negligible effect. b) TRAK-based curation shows no privacy onion effect, but since attack success is highly sensitive to target dataset size, removing samples increases it.
Figure 17: Image-based Attack success with access to the scores for various datasets and curation methods given access to the full set of pool scores. We find that - while none of the methods satify DP - only Image-based scoring is attackable.
Figure 18: Image-based Attack success without access to the scores for various datasets and curation methods given access to the full set of pool scores. We show that just the selection of the pool samples reveals membership information about the targets.
Figure 19: TRAK Attack success with access to the scores.
Figure 20: TRAK Attack success without access to the scores.
Figure 21: Image-based attack success with access only to the models trained on curated data.
Figure 22: TRAK attack success with access only to the models trained on curated data.
Figure 23: TRAK end-to-end attack on CIFAR-10: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 24: TRAK end-to-end attack on CIFAR-100: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 25: TRAK end-to-end attack on Food101: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 26: TRAK end-to-end attack on PCam: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 27: TRAK end-to-end attack on RESISC45: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 28: TRAK end-to-end attack on STL-10: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 29: Image-based end-to-end attack on CIFAR-10: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 30: Image-based end-to-end attack on CIFAR-100: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 31: Image-based end-to-end attack on Food101: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 32: Image-based end-to-end attack on PCam: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 33: Image-based end-to-end attack on RESISC45: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 34: Image-based end-to-end attack on STL-10: ROC curves for different target dataset sizes and AUC vs. size summary.
Figure 35: Image-based attack success with access to scores ablated over target dataset size.
Figure 36: Image-based attack success without access to scores ablated over target dataset size.
Figure 37: TRAK-based attack success with access to scores ablated over target dataset size.
Figure 38: TRAK-based attack success without access to scores ablated over target dataset size.
Figure 39: Shadow models ablation. (a) For Image-based curation we show that the improvement of adding more shadow models varies drastically between datasets. (b) For TRAK, we do not observe any improvement in attack success when adding more shadow models.
Figure 40: Dimension ablation. We show that the number of (a) embedding dimensions for Image-based curation and (b) projection dimensions for TRAK have a moderate effect on attack success on the scores.
More in this topic
Privacy-Preserving ML
Privacy-preserving machine learning develops methods to learn, share, and run models while limiting exposure of sensitive data.
See all 432 papers