Calibrating One-Round Membership Inference with Neighbors

Membership inference attacks (MIAs) are the standard tool for auditing the privacy risks of machine learning models. Given a query point, an MIA aims to determine whether that point was used to train the target model. In practice, such inference must rely on the statistical signals exposed by the model's outputs, such as confidence scores, logits, and intermediate feature representations. However, existing methods often fail to efficiently summarize and combine these statistical signals. To address this limitation, we propose Pairwise Likelihood MIA (PL-MIA), a unified method that combines a Gaussian likelihood-ratio (GLR) statistic with population calibration and the Cauchy combination test. We characterize theoretically how the GLR retains variance-contraction signals and establish conditions under which population calibration and Cauchy combination improve attack power. We obtain $p$-values from pairwise comparisons between the query point and reference points not used for training, and aggregate these continuous signals using the Cauchy combination test. This preserves the evidence strength that is discarded when each pairwise comparison is reduced to a binary vote. Extensive experiments demonstrate that PL-MIA outperforms strong baselines, improving the true positive rate (TPR) by over 25% in the critical low-false-positive regime, corroborating our theoretical findings. These results demonstrate how statistical principles can turn noisy model outputs into more powerful, calibrated, and reproducible evidence for membership privacy auditing.

Membership inference attacks (MIAs) threaten the privacy of machine learning models by revealing whether a data point was used during training. Existing MIAs often assume access to public datasets, shadow models, confidence scores or the training distribution, which makes them vulnerable to defenses like confidence masking. Label-only MIAs avoid these assumptions but require thousands of queries per sample. We propose a cost-effective label-only MIA framework based on transferability and model extraction. Querying the target $M$ with active sampling, perturbation-based selection and synthetic data, we extract a surrogate $S$ on which membership inference is performed offline. This shifts query overhead to a one-time extraction phase. It also removes the restriction that defines the label-only setting: the attacker controls $S$ and can read its posteriors and training trajectory, so attacks that cannot be run against $M$ can be run against $S$. On Location, Purchase and Texas, the strongest attack on $S$ improves AUC over the direct attack on $M$ by $0.9$, $5.6$ and $5.0$ percentage points, and improves the true positive rate at $1\%$ false positive rate by $2.8\times$ to $6.3\times$. We characterize how leakage transfer depends on surrogate fidelity, evaluate standard defenses, and report preliminary results on image datasets.

Membership inference attacks (MIAs) are a canonical way to assess a machine learning model's privacy properties. Although several attempts have been made to evaluate MIAs on language models, the extant literature has suffered numerous difficulties in constructing clean evaluations to test new techniques. In particular, subtle distribution shifts between member and non-member sets can undermine the statistical validity of MIAs; recent work has underscored this by showing that "blind" methods with no access to the underlying model can perform far better than published methods on the same benchmarks. This paper constructs a benchmark for principled evaluation of MIAs against LLMs, by leveraging the insight that training data before and after a fixed point during training are drawn from the same distribution. Therefore, all open-source models with intermediate checkpoints and public training data can be converted into MIA testbeds. We apply our framework to a half-dozen published attacks on the Pythia and OLMo family of models, from 70M to 7B parameters. To facilitate further privacy research, we open-source a modular library for designing and implementing attacks in this setting: https://github.com/safr-ai-lab/pandora_llm.