MemCatalyst: Amplifying Data Auditing on Vision-Language Models via Data Poisoning

Xukun Luan; Jinyan Liu; Yuhui Gong; Yuanguo Bi; Bing Hu; Xuesong Li; Di Wang

cs.CRAug 18, 2026

MemCatalyst: Amplifying Data Auditing on Vision-Language Models via Data Poisoning

Authors: Xukun Luan, Jinyan Liu, Yuhui Gong, Yuanguo Bi, Bing Hu, Xuesong Li, Di Wang

Organizations: School of Computer Science and Technology, Beijing Institute of Technology · School of Computer Science and Engineering, Northeastern University · Engineering Research Center of Security Technology of Complex Network System, Ministry of Education · Faculty of Computer Science, Dalhousie University ·

Abstract

Vision-Language models (VLMs) achieve outstanding performance largely due to the amount of training data available on the internet. At the same time, data holders (e.g., artists) urgently need to determine whether their data has been used for model training without authorization, which concerns both intellectual property rights and personal privacy. Data auditing, particularly through membership inference (MI), has attracted attention as a direct tool. This work proposes MemCatalyst, a set of data poisoning tools, aiming to amplify the data auditing performance on VLMs. MemCatalyst employs two strategies: Poisoning Text (PT) and Poisoning Image (PI). MemCatalyst forces VLMs to over-learn specific inconsistencies between image features and textual semantics during training, thereby increasing their susceptibility to membership information auditing. Crucially, the transferability of poisoned samples across different VLM architectures is demonstrated to be effective in the black-box setting. Extensive evaluations using five state-of-the-art data audits on two prominent VLMs demonstrate that MemCatalyst markedly enhances MI AUC scores with a minimal budget of poisoned samples, while maintaining a negligible impact on model performance.

Figures & tables

Fig. 1 : Overview of MemCatalyst. The goal of MemCatalyst is to leverage the poisoned samples to amplify data auditing, functioning analogously to a catalyst in chemical reactions.

Fig. 2 : Conceptual illustration of the text decision boundaries of the VLM trained without/with (left/right) MemCatalyst. When trained with complex text decision boundaries, the VLM tends to over-learn and strengthen its memorization of training samples, consequently increasing the data auditing performance.

Panel of Figure 3 — Fig. 3 : Histogram of MPNet-based embedding similarity scores on MiniGPT-4 under the baseline and MemCatalyst settings. (a) Similarity-score distributions of member and non-member samples before applying MemCatalyst. (b) Corresponding distributions after applying MemCatalyst.

Fig. 4 : The framework of our proposed MemCatalyst-PT.

Fig. 5 : The framework of our proposed MemCatalyst-PI.

Fig. 6 : t-SNE visualization of image features extracted by Shadowcast [ 17 ] (left) and our proposed attack (right).

		PT			PI			Non
granularity	similarity_metric	accuracy	recall	precision	accuracy	recall	precision	accuracy	recall	precision
	rouge1_f	0.516680	0.53860	0.569490	0.491410	0.55809	0.396514	0.509390	0.54914	0.532316
	rouge2_f	0.514303	0.59856	0.568317	0.547255	0.55788	0.553872	0.511225	0.55866	0.515883
	rougeL_f	0.510955	0.58660	0.517160	0.506490	0.58771	0.519187	0.490485	0.34491	0.479717
5	embedding_mpn	0.513930	0.63438	0.521136	0.493115	0.59536	0.553085	0.500025	0.53545	0.448410
	rouge1_f	0.548740	0.59528	0.593409	0.475170	0.48886	0.389725	0.532705	0.52710	0.547975

TABLE I : Accuracy, Recall, and Precision of Shadow Model Inference against LLaVA with 50 poisoned samples. For each row, the three bolded values indicate the highest scores with respect to their respective metrics.

Fig. 7 : AUC scores of Reference Inference (member) and Reference Inference (non-member) against MiniGPT-4.

Panel of Figure 9 — Fig. 8 : AUC scores of different MI methods against LLaVA.

Evaluation Metrics	rouge1_f	rouge1_p	rouge1_r	rouge2_f	rouge2_p	rouge2_r	cosine_similarity
Non	0.3133	0.3267	0.3972	0.1280	0.1376	0.1570	0.3836
[0pt][0pt] PT	0.3132/ 0.0001	0.3266/ 0.0001	0.3971/ 0.0001	0.1278/ 0.0002	0.1375/ 0.0001	0.1568/ 0.0002	0.3836/ 0.0000
[0pt][0pt] PI	0.3133/ 0.0000	0.3256/ 0.0011	0.3969/ 0.0003	0.1272/ 0.0008	0.1373/ 0.0003	0.1550/ 0.0020	0.3827/ 0.0009

TABLE II : Comparison of model performance across different evaluation metrics on LLaVA. Values highlighted in red indicate a performance degradation compared to Non. PT and PI exhibit a limited impact on model performance, with the observed deviation ranging from 0.0000 to 0.0020.

Evaluation Metrics	rouge1_f	rouge1_p	rouge1_r	rouge2_f	rouge2_p	rouge2_r	cosine_similarity
Non	0.6088	0.5755	0.6850	0.4424	0.4179	0.5122	0.7312
[0pt][0pt] PT	0.6068/ 0.0020	0.5755/ 0.0000	0.6722/ 0.0128	0.4420/ 0.0004	0.4111/ 0.0068	0.4990/ 0.0132	0.7302/ 0.0010
[0pt][0pt] PI	0.5981/ 0.0107	0.5679/ 0.0076	0.6690/ 0.0160	0.4406/ 0.0018	0.4012/ 0.0167	0.4989/ 0.0133	0.7296/ 0.0016

TABLE III : Comparison of model performance across different evaluation metrics on MiniGTP-4. The impact of PT and PI on model performance is limited, with the observed performance difference falling within the range of 0.0000 to 0.0167.

		PT			PI			Non
granularity	similarity_metric	accuracy	recall	precision	accuracy	recall	precision	accuracy	recall	precision
	rouge1_f	0.516842	0.46431	0.562452	0.499124	0.55189	0.392316	0.509390	0.54914	0.532316
	rouge2_f	0.507564	0.56918	0.564987	0.547219	0.55640	0.554424	0.511225	0.55866	0.515883
	rougeL_f	0.510235	0.58627	0.517694	0.498902	0.58838	0.520113	0.490485	0.34491	0.479717
5	embedding_mpn	0.504199	0.62224	0.511265	0.483805	0.58401	0.542576	0.500025	0.53545	0.448410
	rouge1_f	0.538318	0.58393	0.582093	0.466219	0.47964	0.382483	0.532705	0.52710	0.547975

TABLE IV : Accuracy, recall, and precision of Shadow Model Inference against LLaVA with 25 poisoned samples.

		ST			SI			Non
granularity	similarity_metric	accuracy	recall	precision	accuracy	recall	precision	accuracy	recall	precision
	rouge1_f	0.505842	0.55217	0.535216	0.507290	0.55234	0.535916	0.509390	0.54914	0.532316
	rouge2_f	0.514325	0.55598	0.512983	0.514025	0.55746	0.512583	0.511225	0.55866	0.515883
	rougeL_f	0.487235	0.34841	0.475917	0.493085	0.34751	0.476117	0.490485	0.34491	0.479717
5	embedding_mpn	0.496825	0.53895	0.451310	0.497425	0.53805	0.451010	0.500025	0.53545	0.448410
	rouge1_f	0.535805	0.52430	0.551475	0.535305	0.52470	0.551575	0.532705	0.52710	0.547975

TABLE V : Accuracy, recall, and precision of Shadow Model Inference against LLaVA with 5 poisoned samples.

target sample size	similarity_metric	PT	PI	Non
100%	rouge1_f	0.7931	0.7757	0.7514
	rouge2_f	0.8031	0.7834	0.7619
	rougeL_f	0.7941	0.7728	0.7494
	embedding_mpn	0.7919	0.7735	0.7697
90%	rouge1_f	0.6927	0.6599	0.6594
	rouge2_f	0.7083	0.7027	0.6957

TABLE VI : AUC scores of Target-only Inference against MiniGPT-4 with three sizes of target samples.

pretrained LLMs	similarity_metric	PT	PI	Non
Vicuna 7B	rouge1_f	0.7954	0.8025	0.7713
	rouge2_f	0.8139	0.8221	0.7855
	rougeL_f	0.7973	0.8131	0.7667
	embedding_mpn	0.7824	0.7478	0.7730
Vicuna 13B	rouge1_f	0.7799	0.7777	0.7520
	rouge2_f	0.7868	0.7796	0.7657

TABLE VII : AUC scores of Target-only Inference against MiniGPT-4 with three pretrained LLM weights.

similarity_metric	LLaVA $\to$ LLaVA	MiniGPT-4 $\to$ LLaVA	MiniGPT-4 $\to$ MiniGPT-4	LLaVA $\to$ MiniGPT-4
rouge1_f	0.7973	0.7927/ $-0.0046$	0.9963	0.9872/ $-0.0091$
rouge2_f	0.6348	0.6301/ $-0.0047$	0.9989	0.9901/ $-0.0088$
rougeL_f	0.6318	0.6272/ $-0.0046$	0.9965	0.9881/ $-0.0084$
embedding_mpn	0.5754	0.5753/ $-0.0001$	0.9777	0.9595/ $-0.0182$

TABLE VIII : AUC scores of Image-only Inference (

T

=0.2) for the cross-architectural transfer experiment. Despite a minor overall performance degradation (0.0001-0.0182, indicated by red values), PI nevertheless preserves its effectiveness.

similarity_metric	LLaVA $\to$ LLaVA	MiniGPT-4 $\to$ LLaVA	MiniGPT-4 $\to$ MiniGPT-4	LLaVA $\to$ MiniGPT-4
rouge1_f	0.8343	0.8331/ $-0.0012$	0.9680	0.9675/ $-0.0005$
rouge2_f	0.7253	0.7249/ $-0.0004$	0.9883	0.9873/ $-0.0010$
rougeL_f	0.7613	0.7607/ $-0.0006$	0.9727	0.9720/ $-0.0007$
embedding_mpn	0.3936	0.3935/ $-0.0001$	0.9229	0.9156/ $-0.0073$

TABLE IX : AUC scores of Image-only Inference (

T

=1.2) for the cross-architectural transfer experiment.

similarity_metric	LLaVA $\to$ LLaVA	MiniGPT-4 $\to$ LLaVA	MiniGPT-4 $\to$ MiniGPT-4	LLaVA $\to$ MiniGPT-4
rouge1_f	0.8551	0.8477/ $-0.0074$	0.8025	0.7959/ $-0.0066$
rouge2_f	0.9586	0.9552/ $-0.0034$	0.8221	0.8134/ $-0.0087$
rougeL_f	0.9279	0.9275/ $-0.0004$	0.8131	0.8103/ $-0.0028$
embedding_mpn	0.5327	0.5310/ $-0.0017$	0.7478	0.7333/ $-0.0145$

TABLE X : AUC scores of Target-only Inference for the cross-architectural transfer experiment.

temperature	similarity_metric	PT	PI	Non
0.05	rouge1_f	0.9959	0.9950	0.9856
	rouge2_f	0.9993	0.9989	0.9963
	rougeL_f	0.9969	0.9961	0.9887
	embedding_mpn	0.9727	0.9777	0.9480
0.1	rouge1_f	0.9957	0.9957	0.9885
	rouge2_f	0.9995	0.9989	0.9963

TABLE XII : AUC scores of Image-only Inference against MiniGPT-4 with varying temperature settings.

Panel of Figure 20 — Fig. 9 : Samples of Poisoning Image (PI). We selected 50 samples spanning a wide range of data categories, including humans, animals, plants, landscapes, and so on. This demonstrates that PI imposes minimal requirements on the preliminary dataset, highlighting its robustness and thereby posing a more substantial threat in practical scenarios. These samples adhere to the constraint of an $\epsilon$ -bounded perturbation to ensure imperceptibility.

Panel of Table 21 — TABLE XIII : Some samples of Poisoning Text (PT). The original and poisoned texts are denoted by blue and red colors, respectively, to facilitate manual assessment. It can be observed that the red portions maintain semantic alignment with the image while preserving textual plausibility, due to individual differences in human understanding.

Panel of Figure 22 — Fig. 10 : AUC scores of Image-only Inference against LLaVA.

Panel of Figure 23 — Fig. 11 : AUC scores of Reference Inference (member) against LLaVA.

Panel of Figure 24 — Fig. 12 : AUC scores of Reference Inference (non-member) against LLaVA.

Panel of Figure 25 — Fig. 13 : AUC scores of Target-only Inference against LLaVA.

Symbol	Description
$IN$	member sample
$OUT$	non-member sample
$X$	VLM’s sample
$V$	image part of the sample
$Q$	instruction (prompt) part of the sample
$A$	text (response) part of the sample

TABLE XIV : Description of symbols.

Data Poisoning Attacks

Data poisoning attacks manipulate training or retrieval data to corrupt model behavior, implant backdoors, or undermine reliability.

See all 125 papers

DistractMIA: Black-Box Membership Inference on Vision-Language Models via Semantic Distraction
May 12, 2026Hongyi Tang, Zhihao Zhu, Yi YangVision-Language Models Membership Inference Attacks
The Hong Kong University of Science and Technology
Single-Sample Black-Box Membership Inference Attack against Vision-Language Models via Cross-modal Semantic Alignment
May 17, 2026Jiaqing Li, Yajuan Lu, Xiaochuan Shi +3Membership Inference Attacks Privacy Protection in VLMs
1Wuhan University · 2Tarim University
Revealing Training Data Exposure in Vision Language Large Models via Parameter Gradients
Jun 23, 2026Zhihao Zhu, Hongyi Tang, Yi Yang +1LLM Auditing
Department of Information Systems, Business Statistics and Operations Management (ISOM), Hong Kong University of Science and Technology, Hong Kong, China. · Department of IT, Analytics, and Operations, University of Notre Dame, Notre Dame, Indiana, USA.

May 12, 2026J/K move · Enter open · S save

DistractMIA: Black-Box Membership Inference on Vision-Language Models via Semantic Distraction

Hongyi Tang, Zhihao Zhu, Yi Yang

The Hong Kong University of Science and Technology

Vision-Language Models Membership Inference Attacks Privacy Leakage in Language Models

Open paper AskPDF arXiv

Vision-language models (VLMs) are trained on large-scale image-text corpora that may contain private, copyrighted, or otherwise sensitive data, motivating membership inference as a tool for training-data auditing. This is especially challenging for deployed VLMs, where auditors typically observe only generated textual responses. Existing VLM membership inference attacks either rely on probability-level signals unavailable in such settings, or use mask-based semantic prediction tasks whose effectiveness depends on object-centric visual assumptions. To address these limitations, we propose DistractMIA, an output-only black-box framework based on semantic distraction. Rather than removing visual evidence, DistractMIA preserves the original image, inserts a known semantic distractor, and measures how generated responses change. This design is motivated by the intuition that member samples remain more anchored to the original image semantics, while non-member samples are more easily redirected toward the distractor. To make this signal reliable, DistractMIA calibrates distractor configurations on a reference set and derives membership scores from repeated textual generations, capturing response stability and distractor uptake without accessing logits, probabilities, or hidden states. Experiments across multiple VLMs and benchmarks show that DistractMIA consistently outperforms both output-only and stronger-access baselines. Its performance on a medical benchmark further demonstrates applicability beyond object-centric natural images.

Abstract

Data Poisoning Attacks

Explore similar work

DistractMIA: Black-Box Membership Inference on Vision-Language Models via Semantic Distraction

Single-Sample Black-Box Membership Inference Attack against Vision-Language Models via Cross-modal Semantic Alignment

Revealing Training Data Exposure in Vision Language Large Models via Parameter Gradients