Membership inference determines whether a candidate example was used to train a text-to-image diffusion model and is an important tool for auditing training-data use. We identify a directional membership signal in conditional prediction shifts, the changes in predicted noise under alternative text conditions at a fixed noisy input: whether movement toward the alternative prediction initially lowers or raises the denoising loss from its value under the original caption. Members exhibit less room for such local improvement. To quantify this response, we propose LoFIT (Local Conditional Fit), which measures the initial response of the denoising loss along normalized prediction shifts and aggregates it across multiple text conditions and diffusion states. LoFIT requires no target-model gradients and can be computed using only forward queries to the denoiser. Evaluations on four datasets show improved membership discrimination across fine-tuned and pretrained models. LoFIT distinguishes members early in fine-tuning, while the naive denoising-loss baseline remains near chance. Controlled comparisons show that the membership signal is concentrated in the first-order response, with little discrimination from prediction-shift magnitude alone.
Figures & tables
Figure 1: Illustration of LoFIT for a queried image-caption pair. At a fixed noisy input, r denotes the original-caption prediction error relative to the target noise, and δ the prediction shift toward a reference-caption prediction. LoFIT measures the initial denoising-loss response along this shift through the normalized alignment ⟨r,δ⟩/∥δ∥2 . Members exhibit a weaker initial loss decrease than non-members, indicating less room for local denoising improvement and yielding a higher score. The curves are conceptual; measured class distributions appear in Figure 6 .
Flickr30k
COCO
Pokemon
Method
AUROC
TPR@1%FPR
AUROC
TPR@1%FPR
AUROC
TPR@1%FPR
SD v1.4
Naive loss
0.524
0.013
0.527
0.018
0.576
0.029
PIA
0.507
0.010
0.503
0.012
0.552
0.027
SecMI
0.551
0.017
0.562
0.012
0.538
0.022
SD-MIA
0.519
0.013
0.517
0.013
0.502
0.019
Table 1: Full-set membership inference on fine-tuned SD v1.4, SD v2.1, and SD-XL. Bold marks the best value per backbone and metric; dashes denote unreported results among the listed methods.
Metric
Naive loss
PIA
SecMI
SD-MIA
PFAMI
CLiD th
CLiD vec
LoFIT
AUROC
0.527
0.521
0.512
0.571
0.516
0.557
0.548
0.602
TPR@5%FPR
0.072
0.072
0.063
0.112
0.066
0.075
0.076
0.101
Table 2: Membership inference on pretrained SD v1.5 (LAION-MI). Bold marks the best values; underlining marks the second-best TPR.
Figure 2: Query efficiency on Flickr30k and LAION-MI. CLiD curves use the released statistic with two-fold cross-fitted calibration (Appendix A.3 ); “our prompts” and “our states” denote our reference captions and timestep grid. Open circles mark the original 15 -query configuration.
Flickr30k
COCO
Pokemon
LAION-MI
Method
AUROC
TPR@1% FPR
AUROC
TPR@1% FPR
AUROC
TPR@1% FPR
AUROC
TPR@5% FPR
All reference families (LAION-MI: 36 references; Table 2 uses 7)
LoFIT
0.935
0.454
0.954
0.478
0.896
0.272
0.573
0.085
CLiD th ( α=0.5 )
0.731
0.106
0.752
0.130
0.764
0.058
0.563
0.084
CLiD th (oracle α )
0.813
0.092
0.824
0.083
0.815
0.079
0.569
0.083
Empty reference only
Table 3: LoFIT and CLiD th on identical reference predictions and diffusion states. Bold marks the best value in each reference group.
Figure 3: Membership inference across Flickr30k fine-tuning checkpoints, including official CLiD th . Membership denotes participation in this fine-tuning run.
Reference directions
Flickr30k
COCO
Pokemon
LAION-MI
Combined edits
0.880
0.909
0.810
0.569
Gaussian, h=160
0.854
0.903
0.805
0.534
Table 4: Single-reference AUROC for combined caption edits and Gaussian probes.
Figure 4: Conditional prediction statistics on Flickr30k. Left: distributions of squared shift magnitude ∥δij∥22 on a logarithmic scale. Right: normalized residual interaction ⟨ri,δij⟩/∥δij∥2 , the LoFIT response. Shift magnitudes largely overlap, while the first-order response separates the classes. Magenta denotes members and blue denotes non-members. Full profiles appear in Appendix B.2 .
Flickr30k
COCO
Pokemon
LAION-MI
Readout
AUROC
TPR@1% FPR
AUROC
TPR@1% FPR
AUROC
TPR@1% FPR
AUROC
TPR@5% FPR
Loss gap Δij
0.813
0.092
0.818
0.078
0.804
0.065
0.549
0.076
Normalized gap
0.906
0.334
0.930
0.380
0.882
0.127
0.567
0.085
Shift magnitude ∥δij∥2
0.503
0.012
0.487
0.014
0.457
0.024
0.480
0.053
LoFIT
0.935
0.454
0.954
0.478
0.896
0.272
0.602
0.101
Table 5: Readout comparison on identical predictions and diffusion states. Bold marks the best value per metric.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Dataset
Effective batch
Evaluated update
SD v1.4
Flickr30k
16
20,000
COCO
256
1,000
Pokemon
256
200
SD v2.1
Flickr30k
16
20,000
COCO
256
1,000
Pokemon
256
200
Appendix
Table 6: Target-model training runs and evaluated checkpoints.
Prompt family
Construction
Standard language-model rewrites
Token-level edits
Small wording changes, including one- or two-word synonym substitutions that preserve meaning.
Paraphrases
Rephrase the caption while preserving its meaning, including light and heavy rewrites with increasing changes in wording.
Style changes
Change how the caption is presented.
Semantic edits
Change depicted entities or actions, including variants that alter one, two, or three or more content words.
Combined edits
Change both wording and semantic content.
Appendix
Table 7: Reference-caption families and their construction.
Reference family
Flickr30k
COCO
Pokemon
LAION-MI
Standard language-model rewrites
Token-level edits
6
8
14
6
Paraphrases
8
12
18
8
Style changes
2
4
4
2
Semantic edits
8
10
19
8
Combined edits
5
5
5
5
Appendix
Table 8: References per image in the expanded pools, before zero-shift omissions.
Configuration
Flickr30k
COCO
Pokemon
Combined edits
5
5
5
Semantic edits
2
2
5
Heavy paraphrases
2
2
5
Empty prompt
1
1
1
Total references
10
10
16
Timesteps T
12
4
4
Appendix
Table 9: SD v2.1 and SD-XL reference counts and evaluation budgets.
Target setting
CLiD th
CLiD vec
SD v1.4 Flickr30k
Oracle α=0.9
Shadow-trained classifier
SD v1.4 COCO, Pokemon
Oracle α=0.9 and 0.7
Target two-fold cross-fitting
SD v2.1, all datasets
Fixed α=0.5 ; unlabeled target-pool scaling
Target two-fold cross-fitting
SD-XL, all datasets
Target cross-fitting for scaler and α
Target two-fold cross-fitting
SD v1.5 LAION-MI
Oracle α=0.6
Target two-fold cross-fitting
Appendix
Table 10: CLiD calibration and classifier configurations for Tables 1 and 2 .
Membership score
Before matching
After matching
CLIP image-text similarity
0.668
0.503
Appendix
Table 11: AUROC using CLIP image-text similarity as the membership score, before and after matching.
Comparison
Group 1 size
Group 2 size
AUROC
Members vs. all non-members (before)
6,400
9,578
0.611
Members vs. matched non-members (after)
6,400
6,400
0.571
Members vs. excluded non-members
6,400
3,178
0.691
Matched vs. excluded non-members
6,400
3,178
0.635
Appendix
Table 12: SD-MIA AUROC before and after CLIP matching, with population controls on LAION-MI.
Figure 5: Loss landscapes for two Flickr30k members (top) and two non-members (bottom) under alternative reference captions. Each curve uses the state-averaged statistics for one reference. The original prediction is at s=0 , and the vertical axis gives the mean loss change. Dots mark minima and squares reference predictions; curves include extrapolation. The RMS shift is Sˉj (Equation 13 ).
Figure 6: Distribution of minimum locations sj∗=−vj from the state-averaged loss curves. Each observation is an image-reference pair. Members’ minima (magenta) concentrate closer to the original prediction at s=0 than non-members’ minima (blue), with weaker separation on LAION-MI. Positive values indicate that the minimum lies toward the reference prediction.
Figure 7: Conditional prediction statistics across four datasets. For each dataset, the upper panel shows log10∥δij∥22 and the lower panel shows normalized alignment ⟨ri,δij⟩/∥δij∥2 , the LoFIT response. Magenta and blue denote members and non-members, respectively.
Flickr30k
COCO
Pokemon
LAION-MI
Score
AUROC
TPR@1% FPR
AUROC
TPR@1% FPR
AUROC
TPR@1% FPR
AUROC
TPR@5% FPR
E[v] (ours)
0.935
0.454
0.954
0.478
0.896
0.272
0.602
0.101
−E[∣v∣]
0.893
0.183
0.915
0.246
0.837
0.062
0.595
0.086
−E[v2]
0.886
0.184
0.910
0.216
0.835
0.087
0.595
0.088
E[v/∥r∥2]
0.917
0.377
0.944
0.410
0.891
0.255
0.602
0.101
Appendix
Table 13: Membership scores with and without the response sign, evaluated on identical predictions.
Reference family
Flickr30k
COCO
Pokemon
LAION-MI
Token-level edits
0.642±0.022
0.679±0.026
0.655±0.030
0.515±0.008
Paraphrases
0.737±0.027
0.756±0.029
0.701±0.041
0.517±0.009
Style changes
0.650±0.033
0.700±0.024
0.638±0.015
0.504±0.010
Semantic edits
0.743±0.064
0.793±0.058
0.738±0.047
0.529±0.017
Combined edits
0.880±0.002
0.909±0.007
0.810±0.014
0.569±0.006
Random captions
0.836±0.009
0.819±0.011
0.740±0.017
0.574±0.007
Appendix
Table 14: Single-reference LoFIT AUROC (mean ± standard deviation across slots) at equal query cost. Bold marks the highest mean per dataset.
Reference
Flickr30k
COCO
Pokemon
LAION-MI
Caption references
Empty prompt
0.900
0.914
0.781
0.606
Combined edits
0.880±0.002
0.909±0.007
0.810±0.014
0.569
Random captions
0.836±0.009
0.819±0.011
0.740±0.017
0.574
Semantic edits
0.743±0.064
0.793±0.058
0.738±0.047
0.529
Paraphrases
0.737±0.027
0.756±0.029
0.701±0.041
0.517
Appendix
Table 15: Single-reference AUROC for caption and Gaussian probes. Entries show means and standard deviations where available.
References
Flickr30k
COCO
Pokemon
LAION-MI
Caption pool
0.935
0.954
0.896
0.573
Caption + Gaussian
0.946
0.966
0.910
0.575
Appendix
Table 16: AUROC after adding Gaussian probes to the full caption pool.
Flickr30k
COCO
Pokemon
LAION-MI
Score and references
AUROC
TPR@1% FPR
AUROC
TPR@1% FPR
AUROC
TPR@1% FPR
AUROC
TPR@5% FPR
Official CLiD
CLiD th , oracle α
0.603
0.031
0.650
0.044
0.576
0.041
0.535
0.084
Loss-based controls
Naive denoising loss
0.522
0.014
0.525
0.018
0.554
0.026
0.521
0.071
Mean loss gap, empty
0.627
0.063
0.673
0.059
0.523
0.041
0.536
0.081
Appendix
Table 17: Image-membership inference with BLIP-2 captions. Bold marks the best value per metric.
CogVideoX-2b
Wan2.1
Anime-Shooter
OpenVid
Anime-Shooter
OpenVid
Method
AUROC
TPR@5% FPR
AUROC
TPR@5% FPR
AUROC
TPR@5% FPR
AUROC
TPR@5% FPR
Naive loss
0.580
0.122
0.504
0.044
0.515
0.072
0.502
0.056
PIA
0.547
0.090
0.502
0.052
0.504
0.052
0.490
0.050
SecMI
0.508
0.074
0.591
0.108
0.488
0.068
0.493
0.046
CLiD th ( α=0.5 )
0.870
0.530
0.594
0.114
0.564
0.074
0.547
0.072
Appendix
Table 18: Video membership inference on fine-tuned CogVideoX-2b and Wan2.1. Bold marks the best value per model, dataset, and metric.
Membership inference attacks expose whether individual records were used to train a model, yet existing attacks on diffusion models are largely heuristic and can require substantial query budgets. We introduce DIME (Denoiser Ideal Membership Error), a theoretically grounded and query-efficient framework for membership inference on diffusion models. Our starting point is an exact characterization of the optimal diffusion denoiser for a finite training set, which reveals that membership leakage is governed by the denoiser's implicit reconstruction error. This error decomposes into two complementary signals: a bias term, capturing reconstruction accuracy, and a previously unexplored local crowding term, capturing the geometry of nearby training examples. Both admit efficient estimators using only model queries, yielding a practical attack with as few as two queries. Across CIFAR-10/100, STL10-U, CelebA, and ImageNet, DIME consistently outperforms prior attacks at comparable or substantially lower query cost, improving TPR at 1% FPR by up to 3×; remarkably, its two-query variant can outperform existing 30-query baselines. Finally, we suggest, discuss, and evaluate specific defenses to counteract such powerful membership tests.
Latent diffusion models have achieved remarkable success in high-fidelity text-to-image generation, but their tendency to memorize training data raises critical privacy and intellectual property concerns. Membership inference attacks (MIAs) provide a principled way to audit such memorization by determining whether a given sample was included in training. However, existing approaches assume access to ground-truth captions. This assumption fails in realistic scenarios where only images are available and their textual annotations remain undisclosed, rendering prior methods ineffective when substituted with vision-language model (VLM) captions. In this work, we propose MoFit, a caption-free MIA framework that constructs synthetic conditioning inputs that are explicitly overfitted to the target model's generative manifold. Given a query image, MoFit proceeds in two stages: (i) model-fitted surrogate optimization, where a perturbation applied to the image is optimized to construct a surrogate in regions of the model's unconditional prior learned from member samples, and (ii) surrogate-driven embedding extraction, where a model-fitted embedding is derived from the surrogate and then used as a mismatched condition for the query image. This embedding amplifies conditional loss responses for member samples while leaving hold-outs relatively less affected, thereby enhancing separability in the absence of ground-truth captions. Our comprehensive experiments across multiple datasets and diffusion models demonstrate that MoFit consistently outperforms prior VLM-conditioned baselines and achieves performance competitive with caption-dependent methods.
Joonsung Jeon, Woo Jae Kim, Suhyeon Ha +2
Korea Advanced Institute of Science and Technology (KAIST)
Modern diffusion models are trained on massive datasets that often include copyrighted or private images. Membership inference attacks (MIAs) address this by testing whether a given image was in a model's training set. The standard approach scores an image by its reconstruction loss and flags low-loss images as members. This often fails because some images are simply easier to reconstruct than others, so a simple unseen image can score a lower loss than a complex training image. Difficulty calibration separates memorization from how difficult an image is to begin with. We show how to calibrate along two axes of time: training time and diffusion time. First, across training time, we introduce Reference-Model Calibration (RMC), which compares a fine-tuned model's loss against an earlier checkpoint. This cleanly isolates newly memorized data and gives near-perfect auditing, even without text captions. Since earlier checkpoints are not always available, we next present Temporal Self-Calibration (TSC), which calibrates across diffusion time using only a single model. Our key observation is that an image's difficulty persists across noise levels, whereas memorization is localized in time. By comparing the loss at a target timestep against neighboring timesteps from the same model, TSC attenuates image difficulty using the model's own trajectory. Empirically, our methods beat the state of the art by a wide margin, and unlike most existing attacks, generalize across different model parameterizations such as x-, v-, and ε-prediction models. Code is available at https://github.com/MichaelJoseph777/TSC.