Vision large language models (VLLMs) are deployed in privacy-sensitive domains, raising training-data exposure concerns. Existing gray-box membership inference attacks (MIAs) commonly rely on entropy statistics or generated descriptions, requiring costly generation with limited membership separation. Perturbation-based methods instead measure response changes but focus on large perturbations, leaving VLLM membership behavior under smaller perturbations poorly understood. We systematically characterize VLLM membership behavior under controlled Gaussian perturbations across noise levels at fine resolution. We find that optimal perturbation magnitude and membership-signal direction vary across model-dataset pairs. In many settings, a previously unstudied low-noise regime yields stronger separation than large perturbations, with members exhibiting lower divergence than non-members, reversing the conventional score direction. Other settings retain the conventional direction, demonstrating that neither a fixed perturbation magnitude nor a predetermined score orientation is universally reliable. We propose COMPASS, an adaptive gray-box MIA that calibrates both the perturbation magnitude and membership-score direction using a loosely distribution-matched, known non-member reference set. We instantiate COMPASS using Renyi-normalized KL divergence and Renyi divergence, with max-k aggregation over the most informative image-logit positions. Across LLaVA v1.5-7B, MiniGPT-4, and HuluMed-32B, spanning pretraining, instruction-tuning, and medical-image datasets, COMPASS outperforms baselines. On general-domain data, COMPASS achieves AUCs up to 0.88. On the distribution-matched medical evaluation, it reaches 0.64 AUC, while the strongest baseline remains at chance level. These results establish full-range adaptive perturbation analysis as an effective approach to VLLM membership inference.
Figures & tables
Figure 1: Perturbation sensitivity and membership divergence across normalized Gaussian noise on a Flickr pre-training target set against LLaVA v1.5–7B model with k=0.9. Left: Mean unnormalized KL divergence for members and non-members, with shaded bands denoting one standard deviation. Their similar response in Regime I produces a near-zero divergence gap, whereas members exhibit greater divergence in Regime III. Right: The member–non-member gap ΔDˉ=Dˉm−Dˉn with and without Rényi normalization ( α=∞ ). Rényi normalization amplifies the negative gap in Regime II. Red markers indicate the noise level minimizing ΔDˉ for each metric.
Figure 2: COMPASS attack structure. In the tuning phase, COMPASS compares the divergence responses of the unlabeled target set and a known non-member reference set across candidate noise levels to select the perturbation magnitude and score direction. In the scoring phase, it applies the selected perturbation to each target image, aggregates the largest token-wise divergences, and orients the result to produce a membership score.
LLaVA v1.5–7B
MiniGPT-4
Flickr
COCO-2017
Flickr
Metric
k=0.1
k=0.5
k=0.9
k=0.1
k=0.5
k=0.9
k=0.1
k=0.5
k=0.9
Baselines
Aug-KL
−
−
0.71±0.04
−
−
0.69±0.01
−
−
0.63±0.01
Min- k %
0.58±0.02
0.66±0.05
0.67±0.05
0.60±0.01
0.64±0.03
0.65±0.02
0.51±0.04
0.59±0.02
0.64±0.02
Max-Rényi 0.5%
0.64±0.02
0.72±0.03
0.70±0.03
0.69±0.04
0.74±0.03
0.72±0.03
0.78±0.02
0.73±0.01
0.69±0.01
Table 1: Average AUC over three independently sampled target sets for LLaVA v1.5–7B on Flickr and COCO-2017, and MiniGPT-4 on Flickr. An asterisk indicates that the COMPASS decided to use region II for membership signal.
Figure 3: Max- k effect for COMPASS-R on Flickr.
Endoscopy (AUC)
Endoscopy (TPR@5%FPR)
Metric
k=0.7
k=0.8
k=0.9
k=0.7
k=0.8
k=0.9
Baselines
Aug-KL
−
−
0.42±0.04
−
−
0.01±0.00
Min- k %
0.50±0.00
0.50±0.00
0.50±0.00
0.00±0.00
0.00±0.00
0.00±0.00
Max-Rényi 0.5%
0.50±0.00
0.50±0.00
0.50±0.00
0.00±0.00
0.00±0.00
0.00±0.00
COMPASS-KL (Ours)
Table 2: Average AUC and TPR@5%FPR for the Endoscopy dataset against HuluMed-32B. Following the finding that large Max- k yields stronger performance, we report representative top- k settings k∈{0.7,0.8,0.9} evaluated under the same pipeline as LLaVA and Minigpt ( subsection 6.2 ). Bold indicates the highest reported value in each column, including ties.An asterisk indicates that the COMPASS decided to use region II for membership signal.
Figure 4: Effectiveness of noise-level tuning. Top: AUC using the true optimal noise level and the noise level estimated by our tuning procedure. Right: Estimated and true optimal noise levels for Rényi- ∞ normalized KL divergence difference. The estimated σ~ lies near the true optimum and yields performance close to the ideal setting.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Raw Image Perturbations Used for Noise-level Tuning section 5 and Divergence Analyses section 4 .
Figure 6: Raw divergence curves for all tested normalization for a flickr target subset members (red), non-members (blue), and member-non-member difference (black) against the LLaVA model.
Figure 7: Raw divergence curves for all tested normalization for a flickr target subset members (red), non-members (blue), and member-non-member difference (black) against the MiniGPT-4o model.
Figure 8: Raw divergence curves for all tested normalization for a endoscopy target subset members (red), non-members (blue), and member-non-member difference (black) against the Hulu-Med-32B model.
Vision-language models (VLMs) are trained on large-scale image-text corpora that may contain private, copyrighted, or otherwise sensitive data, motivating membership inference as a tool for training-data auditing. This is especially challenging for deployed VLMs, where auditors typically observe only generated textual responses. Existing VLM membership inference attacks either rely on probability-level signals unavailable in such settings, or use mask-based semantic prediction tasks whose effectiveness depends on object-centric visual assumptions. To address these limitations, we propose DistractMIA, an output-only black-box framework based on semantic distraction. Rather than removing visual evidence, DistractMIA preserves the original image, inserts a known semantic distractor, and measures how generated responses change. This design is motivated by the intuition that member samples remain more anchored to the original image semantics, while non-member samples are more easily redirected toward the distractor. To make this signal reliable, DistractMIA calibrates distractor configurations on a reference set and derives membership scores from repeated textual generations, capturing response stability and distractor uptake without accessing logits, probabilities, or hidden states. Experiments across multiple VLMs and benchmarks show that DistractMIA consistently outperforms both output-only and stronger-access baselines. Its performance on a medical benchmark further demonstrates applicability beyond object-centric natural images.
Hongyi Tang, Zhihao Zhu, Yi Yang
The Hong Kong University of Science and Technology
Vision-Language-Action (VLA) models enable end-to-end robot control and have garnered widespread attention. However, the memorization of training data inherent to VLA, coupled with the high cost of robotic data acquisition, raises serious concerns regarding data privacy leakage and intellectual property infringement. Membership inference attacks (MIAs) aim to determine whether a given sample belongs to the training set. While representing a significant privacy threat, this attack remains underexplored in the context of VLA models. To bridge this gap, we propose VLALeaks, which is based on attention discrepancies in VLA models. We reveal, for the first time, the privacy vulnerabilities of VLA models. Specifically, it comprises a two-stage process: (1) membership feature extraction, and (2) attack model construction. Experimental results across multiple VLA benchmarks demonstrate that VLALeaks readily reveals membership information and achieves optimal attack AUC and TPR@1%FPR, highlighting the privacy vulnerabilities in current VLA model deployments. Our work is the first systematic study of MIAs on VLA models, aiming to provide insights for secure and trustworthy VLA models.
Xukun Luan, Jinyan Liu, Xuesong Li +4
School of Computer Science and Technology, Beijing Institute of Technology · School of Computer Science and Engineering, Northeastern University · PRADA Lab, King Abdullah University of Science and Technology
Large Language Models (LLMs) utilize large amounts of data for their training, some of which may come from copyrighted sources. Membership Inference Attacks (MIA) aim to detect those documents and whether they have been included in the training corpora of the LLMs. The black-box MIAs require a significant amount of data manipulation; therefore, their comparison is often challenging. We study state-of-the-art (SOTA) MIAs under the black-box assumptions and compare them to each other using a unified set of datasets to determine if any of them can reliably detect membership under SOTA LLMs. In addition, a new method, called the Familiarity Ranking, was developed to showcase a possible approach to black-box MIAs, thereby giving LLMs more freedom in their expression to understand their reasoning better. The results indicate that none of the methods are capable of reliably detecting membership in LLMs, as shown by an AUC-ROC of approximately 0.5 for all methods across several LLMs. The higher TPR and FPR for more advanced LLMs indicate higher reasoning and generalizing capabilities, showcasing the difficulty of detecting membership in LLMs using black-box MIAs.