No Caption, No Problem: Caption-Free Membership Inference via Model-Fitted Embeddings

Joonsung Jeon; Woo Jae Kim; Suhyeon Ha; Sooel Son; Sung-Eui Yoon

cs.CVFeb 26, 2026

No Caption, No Problem: Caption-Free Membership Inference via Model-Fitted Embeddings

Authors: Joonsung Jeon, Woo Jae Kim, Suhyeon Ha, Sooel Son, Sung-Eui Yoon

Organizations: Korea Advanced Institute of Science and Technology (KAIST)

Abstract

Latent diffusion models have achieved remarkable success in high-fidelity text-to-image generation, but their tendency to memorize training data raises critical privacy and intellectual property concerns. Membership inference attacks (MIAs) provide a principled way to audit such memorization by determining whether a given sample was included in training. However, existing approaches assume access to ground-truth captions. This assumption fails in realistic scenarios where only images are available and their textual annotations remain undisclosed, rendering prior methods ineffective when substituted with vision-language model (VLM) captions. In this work, we propose MoFit, a caption-free MIA framework that constructs synthetic conditioning inputs that are explicitly overfitted to the target model's generative manifold. Given a query image, MoFit proceeds in two stages: (i) model-fitted surrogate optimization, where a perturbation applied to the image is optimized to construct a surrogate in regions of the model's unconditional prior learned from member samples, and (ii) surrogate-driven embedding extraction, where a model-fitted embedding is derived from the surrogate and then used as a mismatched condition for the query image. This embedding amplifies conditional loss responses for member samples while leaving hold-outs relatively less affected, thereby enhancing separability in the absence of ground-truth captions. Our comprehensive experiments across multiple datasets and diffusion models demonstrate that MoFit consistently outperforms prior VLM-conditioned baselines and achieves performance competitive with caption-dependent methods.

Figures & tables

Figure 1: Distribution of membership scores under different condition types: (a) ground-truth captions, (b) VLM-generated captions, and (c) our model-fitted embeddings. In (d), $\mathcal{L}_{\text{cond}}$ values of member samples increase under condition substitution to VLM, whereas hold-out samples remain relatively stable in (e). Dotted lines denote $\mathcal{L}_{\text{uncond}}$ , and all distributions are estimated using Gaussian kernel density estimation.

Metric	(d) Member	(e) Hold-out
Mean ± Std (GT)	0.0253 ± 0.0091	0.0433 ± 0.0125
Mean ± Std (VLM)	0.0300 ± 0.0104	0.0432 ± 0.0125
KS Test - Statistic $\uparrow$	0.2284	0.0264
KL Divergence $\uparrow$	0.7126	0.2796

Table 1: Quantitative comparison of

\mathcal{L}_{\text{cond}}

distribution under ground-truth vs. VLM captions for (d) member and (e) hold-out samples in Fig. 1 . Arrows indicate the direction of larger deviation.

Figure 2: Overview of our proposed method. (a) Given a query image $x_{0}$ , we first optimize a perturbation $\delta$ to overfit to the learned representation from the model. (b) From the resulting surrogate image $x_{0}+\delta^{*}$ , we extract a model-fitted embedding $\phi^{*}$ , which is then used as a synthetic condition to amplify the disparity between member and hold-out samples in (c).

Methods	Condition	Pokemon			MS-COCO			Flickr
Methods	Condition	ASR	AUC	TPR@1%FPR	ASR	AUC	TPR@1%FPR	ASR	AUC	TPR@1%FPR
CLiD	GT	96.52	99.17	90.14	86.50	90.27	68.80	91.10	95.13	77.20
Loss	VLM	72.27	78.99	4.81	63.70	67.88	4.80	61.60	64.24	5.40
SecMI		78.51	86.22	6.97	57.30	58.07	4.20	54.00	52.38	2.00
PIA		71.79	76.76	10.82	66.00	69.70	6.60	61.00	64.05	5.00
PFAMI		74.43	81.25	6.01	80.40	87.50	29.40	76.90	84.99	24.80

Table 2: Comparison of membership inference performance under the caption-free setting, where baseline methods are conditioned using either ground-truth or VLM-generated captions. Bold numbers denote the best, and underlined numbers indicate the second-best results.

Methods	Condition	Stable Diffusion v1.5
Methods	Condition	ASR	AUC	TPR@1%FPR
CLiD	GT	77.38	77.83	49.65
Loss	VLM	68.21	74.73	4.87
SecMI		55.10	54.57	8.12
PIA		63.92	66.74	6.03
PFAMI		72.85	78.15	19.26

Table 3: Evaluation on SD v1.5.

Figure 3: (a) Membership score distributions when conditioned on the model-fitted embedding $\phi^{*}$ . (b) $\mathcal{L}_{\text{cond}}$ and $\mathcal{L}_{\text{uncond}}$ for member and hold-out samples under varying conditions.

Input	Condition	Pokemon			MS-COCO			Flickr
Input	Condition	ASR	AUC	TPR@1%FPR	ASR	AUC	TPR@1%FPR	ASR	AUC	TPR@1%FPR
$x_{0}$	$\phi$	75.63	81.64	11.06	78.00	85.59	31.00	75.50	82.75	27.20
$x_{0}+\delta$		93.99	96.42	10.34	81.70	89.76	29.20	79.60	86.63	28.60
$x_{0}+\delta_{\text{MAX}}$		75.87	81.79	7.45	78.00	85.43	34.00	75.00	82.32	28.00
MoFit	$\phi^{*}$	94.48	97.30	50.48	88.00	94.17	47.00	86.00	91.32	53.20

Table 4: Quantitative comparison of MIA performance under input image variations.

Figure 4: Distributions of (left) $\mathcal{L}_{\text{uncond}}$ and (right) $\mathcal{L}_{\text{cond}}$ of member and hold-out pairs from (a) Pokemon dataset and (b) model-fitted pairs of MoFit .

Methods	Condition	(a) Gaussian Blur			(b) JPEG Compression			(c) LoRA
Methods	Condition	ASR	AUC	TPR@1%FPR	ASR	AUC	TPR@1%FPR	ASR	AUC	TPR@1%FPR
CLiD	GT	89.10	92.27	70.40	85.50	89.59	61.00	59.00	52.05	1.00
Loss	VLM	64.10	68.17	4.20	63.40	66.53	3.80	54.50	49.26	0.00
SecMI		60.10	62.15	5.20	54.00	54.59	2.80	58.50	53.50	1.00
PIA		63.10	65.28	4.40	64.90	67.18	5.00	58.50	53.50	0.00
PFAMI		80.60	87.51	27.72	78.04	84.39	36.03	73.50	77.50	1.00

Table 5: Membership inference performance under fine-tuning with data augmentations: (a) Gaussian blur and (b) JPEG compression. (c) Fine-tuning with Low-Rank Adaptation (LoRA) ( Hu et al., 2022 ) also shows potential as a defense.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: ASR and AUC for different initial step size and timestep $t$ .

Methods	Condition	LAION-mi
Methods	Condition	ASR	AUC	TPR@1%FPR
Loss	GT	55.02	50.72	1.60
SecMI		53.55	53.34	2.90
PIA		54.80	49.58	2.00
PFAMI		52.75	51.67	0.75
CLiD		56.90	58.62	4.60

Appendix

Table 6: Quantitative comparison on SDv1.5 for LAION-mi.

Figure 6: Score distributions on the Pokemon and Flickr datasets. (a) $\mathcal{L}_{\text{{MoFit} }}$ ( Eq. 7 ) score of member and hold-out samples. (b) $\mathcal{L}_{\text{cond}}$ and $\mathcal{L}_{\text{uncond}}$ under condition variations.

Figure 7: Responses of $\mathcal{L}_{\text{cond}}$ and $\mathcal{L}_{\text{uncond}}$ across multiple datasets under different conditioning schemes.

Figure 8: ASR and AUC when embeddings are extracted from perturbed images $x_{0}+\delta$ under varying noise levels $\varepsilon$ .

Figure 9: $\mathcal{L}_{\text{{MoFit} }}$ score distribution according to the input image variations.

Figure 10: Distributions of $\mathcal{L}_{\text{uncond}}$ (dotted lines) and $\mathcal{L}_{\text{cond}}$ for member and hold-out samples from (a) MS-COCO and Flickr datasets, and (b) model-fitted pairs used in MoFit .

Methods	Condition	Pokemon			MS-COCO			Flickr
Methods	Condition	ASR	AUC	TPR@1%FPR	ASR	AUC	TPR@1%FPR	ASR	AUC	TPR@1%FPR
Loss	GT	80.07	87.50	8.65	72.16	78.21	10.20	66.70	70.14	4.60
SecMI		81.99	89.53	6.49	59.20	60.47	5.00	55.00	54.66	2.40
PIA		80.43	86.90	19.95	78.28	84.64	16.40	70.31	74.94	5.80
PFAMI		75.75	81.06	47.10	84.80	91.41	44.60	83.00	90.58	41.20
CLiD		96.52	99.17	90.14	86.50	90.27	68.80	91.10	95.13	77.20

Appendix

Table 7: Performance comparison of membership inference methods when the ground-truth captions are accessible.

Figure 11: $\mathcal{L}_{\text{uncond}}$ (dotted lines) and $\mathcal{L}_{\text{cond}}$ for member and hold-out samples from (a) MS-COCO and Flickr datasets, and (b) model-fitted pairs used in MoFit .

(a) Surrogate	VRAM (MB)	Runtime (sec)	(b) Embedding	VRAM (MB)	Runtime (sec)
Adam	18053.79	333.24	Adam ( Ours )	14799.11	38.28
SGD	18052.38	333.19	SGD	14797.22	38.21
RMSProp	18050.83	333.66	RMSProp	14803.43	38.15
LION	18049.72	333.57	LION	14803.64	38.15
Ours	20667.50	358.96

Appendix

Table 8: Comparison of GPU VRAM usage and runtime overhead across optimizers during (a) surrogate optimization and (b) embedding extraction.

Threshold	(a) Surrogate Optimziation				Threshold	(b) Embedding Extraction
Threshold	ASR	AUC	TPR@1%FPR	Runtime	Threshold	ASR	AUC	TPR@1%FPR	Runtime
0.10	81.50	88.53	32.00	10.71	0.009	81.00	88.17	35.00	29.11
0.09	82.00	89.67	39.00	16.33	0.008	82.00	89.62	33.00	34.37
0.08	84.00	90.15	42.00	22.32	0.007	84.00	90.50	31.00	40.45
0.07	85.00	90.68	49.00	33.62	0.006	85.00	91.57	37.00	48.07
0.06	86.50	91.00	39.00	55.41	0.005	86.00	92.97	43.00	58.19

Appendix

Table 9: Performance of MoFit under an early stopping regime applied to both surrogate optimization and embedding extraction.

Methods	Condition	(a) SD v1.5			(b) SD v2.1			(c) SD v3
Methods	Condition	ASR	AUC	TPR@1%FPR	ASR	AUC	TPR@1%FPR	ASR	AUC	TPR@1%FPR
CLiD	GT	60.00	58.13	1.00	58.00	55.82	2.00	67.50	71.64	5.00
Loss	VLM	52.00	50.48	0.60	55.50	52.47	1.00	53.00	46.27	0.00
SecMI		52.50	50.28	2.00	57.50	54.37	0.00	62.50	65.06	4.00
PIA		51.50	49.75	1.60	59.00	57.53	0.00	55.00	50.84	0.00
PFAMI		54.00	52.58	0.75	51.50	39.63	1.00	68.50	69.81	3.00

Appendix

Table 10: Evaluation against three large-scale pre-trained models: SD v1.5 (LAION-mi train vs. test), SD v2.1 (LAION-mi train vs. CC3M), and SD v3 (COCO vs. CC3M).

LAION-mi	COCO	Flickr	CC3M	LAION-mi
member vs.	COCO	Flickr	CC3M	hold-out
FID	53.6041	61.8270	16.0582	8.8673

Appendix

Table 11: FID scores between LAION‑mi member set and several datasets.

Figure 12: $\mathcal{L}_{\text{cond}}$ and $\mathcal{L}_{\text{uncond}}$ of SD v3 for COCO and CC3M.

Methods	Condition	Prompt2MedImage
Methods	Condition	ASR	AUC	TPR@1%FPR
CLiD	GT	60.50	60.30	0.00
Loss	VLM	54.00	47.27	0.00
SecMI		53.00	46.20	0.00
PIA		51.50	44.25	0.00
PFAMI		53.00	50.22	1.00

Appendix

Table 12: Evaluation against Prompt2MedImage, a SD v1.4 fine-tuned on ROCO dataset.

Methods	Condition	(a) SD v1.5			(b) SD v2.1			(c) SD v3
Methods	Condition	ASR	AUC	TPR@1%FPR	ASR	AUC	TPR@1%FPR	ASR	AUC	TPR@1%FPR
CLiD	MoonDream2	74.91	80.20	13.22	77.50	83.41	31.00	75.20	81.62	33.40
CLiD	Molmo	78.00	81.40	8.00	72.00	71.65	17.00	68.50	69.23	21.00

Appendix

Table 13: Evaluation of two recent VLMs: MoonDream2 and Molmo Flux captioner.

	ASR	AUC	TPR@1%FPR
Random 1	94.50	96.96	18.00
Random 2	94.00	96.23	25.00
Random 3	94.00	95.85	18.00
Random 4	95.50	96.72	41.00

Appendix

Table 14: Performance of MoFit under four random target noise settings.

Figure 13: Visualization of extreme samples from (a) MoFit and (b) GT-captioned CLiD settings.

Methods	MS-COCO				Flickr
Methods	TP	FN	FP	TN	TP	FN	FP	TN
Colorfulness	135.27	122.65	147.64	132.54	146.22	138.56	147.51	147.95
Entropy	15.22	13.70	15.11	14.27	15.55	15.05	14.92	15.33
Keypoint	500.00	471.95	499.25	492.20	500.00	494.55	499.65	497.75

Appendix

Table 15: TP, FN, FP, and TN statistics using color diversity and image complexity metrics for two fine-tuned SD 1.4 models.

Methods	Condition	(a) 500 images			(b) 1,000 images
Methods	Condition	ASR	AUC	TPR@1%FPR	ASR	AUC	TPR@1%FPR
CLiD	GT	80.50	83.35	55.00	83.00	86.76	60.00
Loss	VLM	64.50	68.05	5.00	64.50	67.72	4.00
SecMI		62.00	62.31	4.00	58.00	59.49	4.00
PIA		66.50	70.71	7.00	69.50	71.83	9.00
PFAMI		83.00	87.26	23.00	83.50	90.45	31.00

Appendix

Table 16: Performance against LDMs fine-tuned with only few training samples: (a) 500 or (b) 1,000 images.

Methods	Condition	Pokemon			MS-COCO			Flickr
Methods	Condition	ASR	AUC	TPR@1%FPR	ASR	AUC	TPR@1%FPR	ASR	AUC	TPR@1%FPR
ReDiffuse	GT	52.40	49.53	0.72	54.20	52.41	1.20	60.00	48.23	0.20
ReDiffuse	VLM	52.16	49.75	0.72	55.30	53.37	1.60	51.30	48.73	0.20

Appendix

Table 17: Evaluation of ReDiffuse under GT and VLM-generated captions.

Membership Inference Attacks

Membership inference attacks test whether records or entities were used to train a machine-learning model, revealing potential privacy leakage.

See all 116 papers

May 26, 2026

Black-box Membership Inference Attacks on the Pre-training Data of Image-generation Models

The rapid advancement of diffusion-based image generation models has raised serious concerns regarding potential copyright and privacy infringements involving human-created data. Membership inference attacks (MIAs) have emerged as a promising tool for identifying unauthorized data usage during model training. Existing methods typically assess the ability of model to denoise perturbed suspect images as an indicator of membership status. However, the discriminative power of such features is highly dependent on the degree of model memorization and deteriorates significantly when applied to less exposed data (e.g., pre-training data). Although several methods attempt to enhance detection by leveraging internal model features, these features are generally inaccessible in mainstream closed-source image generation platforms, limiting their practicality. In this paper, we demonstrate that analyzing how a black-box diffusion model denoises a target image and corresponding perturbed textual instructions can reveal more distinctive membership cues. Based on this insight, we propose a black-box membership inference attack framework (named SD-MIA) that leverages a cross-modal data perturbation mechanism to detect pre-training data in diffusion models. We conduct extensive experiments on both a public benchmark dataset and a newly constructed dataset, each comprising pre-training membership and non-membership samples with identical distributions. Experimental results demonstrate that SD-MIA achieves superior performance compared to existing baselines, including those with the unfair advantage of accessing internal model features.

Tao Qi, Huili Wang, Yuanhong Huang +6

Beijing University of Posts and Telecommunications · Tsinghua University

Membership Inference Attacks Diffusion-Based Image Generation Diffusion Models

Ask PDF

Jul 5, 2026

One Framework for All: Cross-Modal Membership Inference for Generative Models

Large generative models across text-to-text, text-to-image, and image-to-text modalities have been shown to pose significant privacy risks. One fundamental threat is membership inference attacks (MIA), which aim to determine whether a given data point was used in a model's training set. Although prior work has investigated MIAs against these three classes of generative models, existing approaches treat them in isolation and are not cross-applicable, thereby limiting their real-world utility. To address this limitation, we present the first comprehensive study of a unified membership inference framework that applies across text-to-text, text-to-image, and image-to-text modalities. Our approach is grounded in a key modality-agnostic observation: the output distribution of a generative model can approximate its training data distribution. Leveraging this property, we model the distributions of model-generated outputs and auxiliary non-member samples in a shared embedding space, and perform membership inference via likelihood ratio testing. We conduct extensive experiments in a strict black-box setting under both partial-knowledge and zero-knowledge threat models, and evaluate membership inference against both fine-tuning and pre-training data. Experimental results demonstrate our approach's superior performance in comparison to existing state-of-the-art methods, which are typically optimized for a single model class.

Dayong Ye, Tainqing Zhu, Kun Gao +6

Dayong Ye, Tianqing Zhu∗, Kun Gao, Junhao Liu, Yichuan Chen, Shuai Zhou, Hengzhu Liu, Bo Liu, and Wanlei Zhou

Membership Inference Attacks Generative Modeling Likelihood Ratio Testing

Ask PDF

May 12, 2026

DistractMIA: Black-Box Membership Inference on Vision-Language Models via Semantic Distraction

Vision-language models (VLMs) are trained on large-scale image-text corpora that may contain private, copyrighted, or otherwise sensitive data, motivating membership inference as a tool for training-data auditing. This is especially challenging for deployed VLMs, where auditors typically observe only generated textual responses. Existing VLM membership inference attacks either rely on probability-level signals unavailable in such settings, or use mask-based semantic prediction tasks whose effectiveness depends on object-centric visual assumptions. To address these limitations, we propose DistractMIA, an output-only black-box framework based on semantic distraction. Rather than removing visual evidence, DistractMIA preserves the original image, inserts a known semantic distractor, and measures how generated responses change. This design is motivated by the intuition that member samples remain more anchored to the original image semantics, while non-member samples are more easily redirected toward the distractor. To make this signal reliable, DistractMIA calibrates distractor configurations on a reference set and derives membership scores from repeated textual generations, capturing response stability and distractor uptake without accessing logits, probabilities, or hidden states. Experiments across multiple VLMs and benchmarks show that DistractMIA consistently outperforms both output-only and stronger-access baselines. Its performance on a medical benchmark further demonstrates applicability beyond object-centric natural images.

Hongyi Tang, Zhihao Zhu, Yi Yang

The Hong Kong University of Science and Technology

Vision-Language Models Membership Inference Attacks Privacy Leakage in Language Models

Ask PDF

Abstract

Membership Inference Attacks

Explore similar work

Black-box Membership Inference Attacks on the Pre-training Data of Image-generation Models

One Framework for All: Cross-Modal Membership Inference for Generative Models

DistractMIA: Black-Box Membership Inference on Vision-Language Models via Semantic Distraction