Organizations: Beijing Electronic Science and Technology Institute, Beijing 100070, China · Institute of Artificial Intelligence, China Telecom (TeleAI), Beijing 100032, China · Nanyang Technological University, Singapore 637371, Singapore · Nanjing University, Nanjing 210023, China · School of Computing and Information Technology, Great Bay University, Dongguan 523000, China
Contrastive pretraining models such as CLIP and CLAP, serve as the ubiquitous perceptual backbones for modern multimodal large models, yet their reliance on web-scale data raises growing concerns about memorizing Personally Identifiable Information (PII). Auditing such models via membership inference is challenging in practice: shadow-model MIAs are computationally prohibitive for large multimodal backbones, and existing multimodal auditing methods typically require querying the target with paired biometric inputs, thereby directly exposing sensitive biometric information to the target model. To bypass this critical limitation, we demonstrate a highly desirable capability for privacy auditing: multimodal memorization within these foundational encoders can be accurately inferred using exclusively the text modality. We propose Unimodal Membership Inference Detector (UMID), a text-only auditing framework that performs text-guided cross-modal latent inversion and extracts two complementary signals, similarity (alignment to the queried text) and variability (consistency across randomized inversions). UMID compares these statistics to a lightweight non-member reference constructed from synthetic gibberish and makes decisions via an ensemble of unsupervised anomaly detectors. Comprehensive experiments across diverse CLIP and CLAP architectures demonstrate that UMID significantly improves the effectiveness and efficiency over prior MIAs, delivering strong detection performance with sub-second auditing cost using solely text queries, completely circumventing the need for biometric inputs and complying with strict privacy constraints.
Figures & tables
Fig. 1: Overview of the UMID auditing framework and the resulting distributional gap. The UMID method enables text-only membership inference (a) and achieves clear geometric separation between training and non-training samples (b).
Fig. 2: Pipeline of UMID. We employ an optimizer guided by target model to align non-text embeddings with PII text embeddings, maximizing their cosine similarity. By analyzing similarity and variability features of these optimized samples relative to the synthetic gibberish baseline, an anomaly detection system identifies abnormal patterns to infer the membership of the input text.
Setting
Method
Real data
Effectiveness
Efficiency
Precision
Recall
Accuracy
Time
ResNet-50 (one-shot)
MIA [ 13 ]
1 img
.7251
.6873
.7042
4.32h
IDIA [ 42 ]
3 imgs
.6922
.4031
.6836
0.845s
WSA [ 15 ]
1 img
.6653
.2925
.6675
2158.1s
C-WSA [ 43 ]
1 img
.7210
.4530
.7125
2141.4s
UMID (Ours)
Null
.8634
.9821
.9172
0.628s
TABLE I: Comparison of CLIP (image-text) methods. Best and second-best results are bolded .
Setting
Method
Real data
Effectiveness
Efficiency
Precision
Recall
Accuracy
Time
LibriSpeech (one-shot)
Audio Auditor [ 58 ]
1 audio
.6338
.7324
.6519
1.273s
SLMIA-SR [ 60 ]
1 audio
.7521
.8864
.8342
1.528s
AuditMI [ 59 ]
1 audio
.8257
.9526
.8791
13.578s
UMID (Ours)
Null
.8649
.9649
.9127
0.603s
UMID (Ours)
1 audio
.8921
.9868
.9354
0.647s
TABLE II: Comparison of CLAP (audio-text) methods. Best and second-best results are bolded .
Fig. 3: Detection accuracy for CLIP model (ResNet-50) under various parameters.
Fig. 4: Detection accuracy for CLAP model (LibriSpeech) under various parameters.
Fig. 5: Empirical validation of geometric separation. (a) Convergence of similarity ( Sn ) and variability ( Dn2 ) as inversion runs ( n ) increase. (b) Cosine similarity distributions confirming the high-dimensional isotropy of non-members versus the directional bias of members.
Dataset
Training samples
Defense
Effectiveness
Efficiency
Accuracy
Time
CelebA (CLIP)
one-shot
Without
.9535
0.685s
DP
.8742
0.688s
many-shot
Without
.9562
0.691s
DP
.8595
0.689s
LibriSpeech (CLAP)
one-shot
Without
.9354
0.647s
TABLE III: UMID robustness evaluation under differential privacy defense ( ϵ=1.0 , δ=10−5 ).
Dataset
Training samples
Defense
Effectiveness
Efficiency
Accuracy
Time
CelebA (CLIP)
one-shot
Without
.9535
0.685s
CG
.9412
0.689s
many-shot
Without
.9562
0.691s
CG
.9385
0.687s
LibriSpeech (CLAP)
one-shot
Without
.9354
0.647s
TABLE IV: UMID robustness evaluation under covert gibberish defense.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
b+dh43u!f9de545w
53e(s3erg24=pnI<S
fe3_5fg;@f34f.e2w/
4teh<E{434fef234ter
5gt%b@5435-hgF
#4c3rd4scd324g
Appendix
TABLE V: Samples of randomly generated textual gibberish generated by GPT-3.5-turbo.
Karinix
Zylogene
Glycogenyx
Zylotrax
Vexilith
Exodynix
Novylith
Glycosyne
Xenolynx
Rynexis
Appendix
TABLE VI: Covert gibberish that seem to be real names generated by GPT-3.5-turbo.
Large generative models across text-to-text, text-to-image, and image-to-text modalities have been shown to pose significant privacy risks. One fundamental threat is membership inference attacks (MIA), which aim to determine whether a given data point was used in a model's training set. Although prior work has investigated MIAs against these three classes of generative models, existing approaches treat them in isolation and are not cross-applicable, thereby limiting their real-world utility. To address this limitation, we present the first comprehensive study of a unified membership inference framework that applies across text-to-text, text-to-image, and image-to-text modalities. Our approach is grounded in a key modality-agnostic observation: the output distribution of a generative model can approximate its training data distribution. Leveraging this property, we model the distributions of model-generated outputs and auxiliary non-member samples in a shared embedding space, and perform membership inference via likelihood ratio testing. We conduct extensive experiments in a strict black-box setting under both partial-knowledge and zero-knowledge threat models, and evaluate membership inference against both fine-tuning and pre-training data. Experimental results demonstrate our approach's superior performance in comparison to existing state-of-the-art methods, which are typically optimized for a single model class.
Dayong Ye, Tainqing Zhu, Kun Gao +6
Dayong Ye, Tianqing Zhu∗, Kun Gao, Junhao Liu, Yichuan Chen, Shuai Zhou, Hengzhu Liu, Bo Liu, and Wanlei Zhou
Membership inference attacks (MIAs) have emerged as critical tools for auditing privacy risks in large language models (LLMs), aiming to determine whether a given text was included in a model's training corpus. However, most existing MIAs require access to per-token logits or probabilities, making them inapplicable in practice to proprietary LLMs that expose only textual continuations. To address this underexplored setting, we propose Word-level Probability MIA (WPMIA), a statistically principled MIA for strict black-box privacy auditing. WPMIA estimates word-level generation probabilities via Monte Carlo sampling with local kernel smoothing, then aggregates these estimates into a sequence-level likelihood estimator. Furthermore, WPMIA constructs the likelihood conditioned on different prefixes, thereby amplifying the distributional differences between members and non-members. We evaluate WPMIA across various open-source LLMs and find that it consistently outperforms existing black-box baselines. Importantly, we also evaluate WPMIA on modern proprietary LLMs, including GPT-5-Chat, Gemini-2.5-Flash, and Claude-4.5-Haiku, achieving an average TPR@5%FPR of 42.0 across these models. These results offer a sound foundation for future research on strict black-box membership inference. Code is available at https://github.com/niusj03/WPMIA.
Shengjie Niu, Yeheng Ge, Jian Huang
Department of Data Science and Artificial Intelligence The Hong Kong Polytechnic University, Hong Kong SAR, China · Department of Applied Mathematics The Hong Kong Polytechnic University, Hong Kong SAR, China
The pervasive deployment of deep learning models across critical domains has concurrently intensified privacy concerns due to their inherent propensity for data memorization. While Membership Inference Attacks (MIAs) serve as the gold standard for auditing these privacy vulnerabilities, conventional MIA paradigms are increasingly constrained by the prohibitive computational costs of shadow model training and a precipitous performance degradation under low False Positive Rate constraints. To overcome these challenges, we introduce a novel perspective by leveraging the principles of model reprogramming as an active signal amplifier for privacy leakage. Building upon this insight, we present ReproMIA, a unified and efficient proactive framework for membership inference. We rigorously substantiate, both theoretically and empirically, how our methodology proactively induces and magnifies latent privacy footprints embedded within the model's representations. We provide specialized instantiations of ReproMIA across diverse architectural paradigms, including LLMs, Diffusion Models, and Classification Models. Comprehensive experimental evaluations across more than ten benchmarks and a variety of model architectures demonstrate that ReproMIA consistently and substantially outperforms existing state-of-the-art baselines, achieving a transformative leap in performance specifically within low-FPR regimes, such as an average of 5.25% AUC and 10.68% TPR@1%FPR increase over the runner-up for LLMs, as well as 3.70% and 12.40% respectively for Diffusion Models.