Modern diffusion models are trained on massive datasets that often include copyrighted or private images. Membership inference attacks (MIAs) address this by testing whether a given image was in a model's training set. The standard approach scores an image by its reconstruction loss and flags low-loss images as members. This often fails because some images are simply easier to reconstruct than others, so a simple unseen image can score a lower loss than a complex training image. Difficulty calibration separates memorization from how difficult an image is to begin with. We show how to calibrate along two axes of time: training time and diffusion time. First, across training time, we introduce Reference-Model Calibration (RMC), which compares a fine-tuned model's loss against an earlier checkpoint. This cleanly isolates newly memorized data and gives near-perfect auditing, even without text captions. Since earlier checkpoints are not always available, we next present Temporal Self-Calibration (TSC), which calibrates across diffusion time using only a single model. Our key observation is that an image's difficulty persists across noise levels, whereas memorization is localized in time. By comparing the loss at a target timestep against neighboring timesteps from the same model, TSC attenuates image difficulty using the model's own trajectory. Empirically, our methods beat the state of the art by a wide margin, and unlike most existing attacks, generalize across different model parameterizations such as x-, v-, and ε-prediction models. Code is available at https://github.com/MichaelJoseph777/TSC.
Figures & tables
Figure 1: Image difficulty confounds absolute-loss thresholds. Left: CIFAR-10 DDPM log-loss trajectories for members (blue) and non-members (red) showing class means and 80% intervals. Right: Loss density at the optimal log-SNR ( λ⋆ ). Even at maximal separability, the populations exhibit a massive 45% overlap, rendering absolute-loss thresholds ineffective.
Figure 2: RMC on MS-COCO. Stable Diffusion fine-tuned on MS-COCO in realistic settings, uncaptioned. Left: Pretrained and fine-tuned log-losses are almost perfectly correlated (Pearson correlation r=0.989 ). Right: Subtracting the pretrained loss removes this shared difficulty. Members shift below the ASR threshold (dashed ASR 98.6% ), while non-members stay concentrated near 0 .
Pokémon
MS COCO
Flickr
Method
AUC
ASR
TPR@1%FPR
AUC
ASR
TPR@1%FPR
AUC
ASR
TPR@1%FPR
CLiD
( q=15 )
96.98
91.83
64.75
95.55
88.72
44.04
95.22
88.16
47.14
RMC
( q=2 )
99.44
96.51
90.63
99.88
98.52
96.60
99.80
98.14
96.86
( q=20 )
99.73
97.84
96.64
99.94
99.04
98.52
99.96
99.50
99.82
Table 1: RMC on realistic Stable Diffusion fine-tunes. Reference calibration is evaluated without captions at λ=1 log-SNR using either one or 10 paired noise draws. CLiD uses the ground-truth caption. Parentheses provide the number of model queries per candidate.
Figure 3: Temporal self-calibration. (a) A member/non-member pair where absolute loss correctly identifies the member. (b) A pair where absolute loss fails because the member is harder, yet a localized ”dip” is still visible. (c) Subtracting the local temporal reference (the chord between the reference timesteps) removes the absolute difficulty offset. Both pairs collapse into a shared calibrated view, distinctly separating members from non-members by their trajectory shape.
Figure 4: TSC on STL-10-U. Mean member and non-member log-loss trajectories (left) and a close-up of the selected window (right). The chords connect the two context points for each population, and the vertical bars at λ∗ show the residual measured by the temporal score.
CIFAR-10
Tiny ImageNet
STL-10-U
Method
AUC
ASR
TPR@1%FPR
AUC
ASR
TPR@1%FPR
AUC
ASR
TPR@1%FPR
Naive
( q=20 )
85.39
78.21
7.35
64.37
60.52
2.93
64.15
60.50
2.51
SecMI
( q=12 )
88.24
80.91
10.20
66.37
61.88
2.91
68.12
63.33
3.03
PIA
( q=2 )
91.90
84.88
25.80
68.99
64.25
4.43
69.46
64.51
4.08
PIAN
( q=2 )
89.26
83.32
30.90
67.04
63.44
6.10
66.72
63.36
4.32
TSC
( q=15 )
95.91
90.15
62.31
78.63
72.18
12.27
77.84
71.29
9.57
Table 2: Membership inference on DDPM benchmarks. Results reported as percentages. Parentheses give the number of model queries per candidate. SecMI follows its official protocol.
Pokémon
MS COCO
Flickr
Method
AUC
ASR
TPR@1%FPR
AUC
ASR
TPR@1%FPR
AUC
ASR
TPR@1%FPR
Naive
( q=10 )
84.84
77.43
28.37
76.61
69.92
7.16
77.96
71.66
10.08
SecMI
( q=12 )
83.59
76.47
15.14
78.33
71.46
6.84
81.14
74.24
7.40
PIA ( t=500 )
( q=2)
69.86
66.13
19.71
54.08
53.96
2.76
54.73
53.24
1.96
PIA (optimal t )
( q=2 )
94.50
87.88
61.30
75.61
70.90
17.36
74.18
69.58
17.12
PIAN
( q=2 )
53.34
54.15
1.92
69.20
64.08
13.88
69.15
65.10
12.40
Table 3: Membership inference on Stable Diffusion fine-tunes (uncaptioned). SecMI denotes its LDM variant. PIA, is evaluated both over the original paper’s default timestep ( t=500 ) and an average optimal timestep over shadow models.
Selected λ⋆ ( x/v/ϵ loss)
x -loss
v -loss
ϵ -loss
Method
x -pred
v -pred
ϵ -pred
x -pred
v -pred
ϵ -pred
x -pred
v -pred
ϵ -pred
Naive
( q=20 )
−0.75/0.25/1.50
6.48
6.05
6.74
34.54
34.28
34.56
27.52
29.78
28.29
PIA
( q=2 )
−0.75/0.25/1.75
2.27
4.07
6.22
1.01
52.48
54.84
31.69
74.61
75.85
PIAN
( q=2 )
−0.50/0.50/1.75
2.41
6.00
5.69
1.06
56.68
58.36
72.11
77.77
76.73
SecMI
( q=12 )
default
0.94
1.10
1.19
5.55
5.57
5.26
29.25
35.40
30.82
TSC
( q=15 )
−0.50/0.50/1.75
30.28
28.74
31.26
81.31
80.56
80.68
91.22
93.22
92.75
Table 4: Robustness across prediction and loss parameterizations. TPR@1%FPR (%) for unaugmented CIFAR-100 DDPM++ FM models. The selected- λ⋆ tuple lists the shared operating log-SNR under x -, v -, and ϵ -losses. SecMI follows its default protocol.
q=15
q=45
Filter
Setting
AUC
TPR
AUC
TPR
TSC (endpoints)
Δ=2.5,k=1
87.06
35.30
88.11
39.26
Even
Δ=2.5,k=2
83.96
26.43
86.84
34.84
Gaussian (full)
σ=3
–
–
81.21
20.57
Gaussian (window)
k=7,σ=3
71.84
5.72
79.53
11.28
Reverse window
k=7,σ=0.5
80.58
16.26
85.65
27.95
Table 5: Temporal reference filters. Mean AUC and TPR@1%FPR across four datasets trained over DDPM settings. The full Gaussian uses 45 levels and is unavailable at q=15 .
Reference
(a,b)
AUC
TPR
Symmetric
(2.5,2.5)
87.06
35.30
Left only
(2.5,−)
78.15
23.52
Right only
(−,2.5)
82.13
6.47
Unequal
(1.5,3.5)
88.77
30.94
Unequal
(3.5,1.5)
80.64
26.56
Weighted
(1.5,3.5)
86.24
29.11
Table 6: Temporal reference symmetry. At q=15 , mean AUC and TPR@1%FPR across four datasets trained over DDPM settings. (a,b) denotes nominal log-SNR offsets.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
x -prediction
ϵ -prediction
v -prediction
xθ:=fθ(zt,t,c)
ϵθ:=fθ(zt,t,c)
vθ:=fθ(zt,t,c)
x -loss:
E∥
xθ−x
∥22
xθ
xθ=tzt−(1−t)ϵθ
xθ=zt+(1−t)vθ
ϵ -loss:
E∥
ϵθ−ϵ
∥22
ϵθ=1−tzt−txθ
ϵθ
ϵθ=zt−tvθ
v -loss:
E∥
vθ−v
∥22
vθ=1−txθ−zt
vθ=tzt−ϵθ
vθ
Appendix
Table 7: Prediction and loss parameterizations for the linear flow schedule. Columns specify the direct network prediction space, while rows specify the space in which its squared error is measured.
Dataset
Resolution
Member
Hold-out
Aug.
Steps
Epochs
CIFAR-10
32×32
25k
25k
HFlip
800k
≈4 k
CIFAR-100
32×32
25k
25k
HFlip
800k
≈4 k
STL-10-U
32×32
50k
50k
HFlip
1.6M
≈4 k
Tiny ImageNet
32×32
50k
50k
HFlip
1.6M
≈4 k
Appendix
Table 8: DDPM training settings. All runs use 32×32 resolution, batch size 128 , linear noise schedule, ϵ -prediction, and random horizontal flips.
Table 11: Reference symmetry by checkpoint. All checkpoints are held-out DDPMs at 4K epochs. AUC and TPR@1%FPR are percentages. C100 denotes CIFAR-100 with horizontal flips. Bold and underlined values denote the best and second-best results within each query budget.
Component
Mean gap
Pooled SD
AUC
Center ℓx(0)
0.3885
0.4119
74.26
Symmetric S
0.2566
0.1366
87.06
Endpoint difference O
0.0371
0.1552
57.14
Left reference S+O
0.2937
0.2464
77.80
Right reference S−O
0.2195
0.1580
81.84
Appendix
Table 12: Components of the temporal contrast. Four-checkpoint means at K=5 shared draws per point. Gaps and standard deviations are in log-loss units.
Figure 6: Membership localization across training losses. Single-timestep absolute-loss AUC on the nine first-half CIFAR-100 DDPM++ models without augmentation at epoch 2000 . Color denotes training-loss target s and line style denotes native prediction p . Losses are averaged over 50 noise draws per image, shared across noise levels. Lower loss indicates membership at every level. Stars mark single-timestep AUC maxima. The peaks occur at λ=−0.75 , 0.75 , and 2.00 for x -, v -, and ϵ -loss, respectively.
Figure 7: Reference calibration across noise levels. AUC (left) and TPR@1%FPR (right) for the realistic Pokémon, MS-COCO, and Flickr30k fine-tunes. Solid curves show RMC ( q=2 ). Dashed curves show the uncalibrated fine-tuned loss from the same noise draw ( q=1 ). Each point uses a fixed score direction, with lower loss or calibrated residual indicating membership. The vertical guide marks λ=1 , the operating point reported in Table 1 .
Membership inference attacks expose whether individual records were used to train a model, yet existing attacks on diffusion models are largely heuristic and can require substantial query budgets. We introduce DIME (Denoiser Ideal Membership Error), a theoretically grounded and query-efficient framework for membership inference on diffusion models. Our starting point is an exact characterization of the optimal diffusion denoiser for a finite training set, which reveals that membership leakage is governed by the denoiser's implicit reconstruction error. This error decomposes into two complementary signals: a bias term, capturing reconstruction accuracy, and a previously unexplored local crowding term, capturing the geometry of nearby training examples. Both admit efficient estimators using only model queries, yielding a practical attack with as few as two queries. Across CIFAR-10/100, STL10-U, CelebA, and ImageNet, DIME consistently outperforms prior attacks at comparable or substantially lower query cost, improving TPR at 1% FPR by up to 3×; remarkably, its two-query variant can outperform existing 30-query baselines. Finally, we suggest, discuss, and evaluate specific defenses to counteract such powerful membership tests.
The rapid advancement of diffusion-based image generation models has raised serious concerns regarding potential copyright and privacy infringements involving human-created data. Membership inference attacks (MIAs) have emerged as a promising tool for identifying unauthorized data usage during model training. Existing methods typically assess the ability of model to denoise perturbed suspect images as an indicator of membership status. However, the discriminative power of such features is highly dependent on the degree of model memorization and deteriorates significantly when applied to less exposed data (e.g., pre-training data). Although several methods attempt to enhance detection by leveraging internal model features, these features are generally inaccessible in mainstream closed-source image generation platforms, limiting their practicality. In this paper, we demonstrate that analyzing how a black-box diffusion model denoises a target image and corresponding perturbed textual instructions can reveal more distinctive membership cues. Based on this insight, we propose a black-box membership inference attack framework (named SD-MIA) that leverages a cross-modal data perturbation mechanism to detect pre-training data in diffusion models. We conduct extensive experiments on both a public benchmark dataset and a newly constructed dataset, each comprising pre-training membership and non-membership samples with identical distributions. Experimental results demonstrate that SD-MIA achieves superior performance compared to existing baselines, including those with the unfair advantage of accessing internal model features.
Tao Qi, Huili Wang, Yuanhong Huang +6
Beijing University of Posts and Telecommunications · Tsinghua University
Diffusion models have achieved remarkable success in generative tasks across various areas, however their training process raises significant privacy concerns, particularly under membership inference attacks (MIAs). Prior studies on privacy-preserving of diffusion models fail to balance privacy, utility, and efficiency. To address this gap, we propose a novel framework of privacy-preserving diffusion models, Adaptive Diffusion Freezing (ADF), which can defend against MIAs with better trade-off. By leveraging cross-timestep adaptive freezing training, ADF explicitly control the participation of different data subsets across diffusion timesteps via a mask matrix, which reduces the over-memorization and leads to more uniform model behaviors between member and nonmember samples. To construct a freezing mask matrix that effectively reduce membership leakage without unnecessarily harming generation quality, we introduce a pretraining-based risk-aware freezing policy to estimate MIA risk based on memorization tendency, and suppress the contribution of the subset-timestep pairs with higher risk. Evaluations on multiple datasets demonstrate that ADF provides effective defense performance as well as state-of-the-art privacy-utility-efficiency trade-off performance compared to various baselines.
Jialu Guo, Xiao Han, Junjie Wu
MIIT Key Laboratory of Data Intelligence and Management Beihang University Beijing, China