Difficulty Calibration Across Time for Membership Inference in Diffusion Models

Michael Joseph; Yedid Hoshen

cs.CVOct 29, 2026

Difficulty Calibration Across Time for Membership Inference in Diffusion Models

Authors: Michael Joseph, Yedid Hoshen

Organizations: The Hebrew University of Jerusalem

Abstract

Modern diffusion models are trained on massive datasets that often include copyrighted or private images. Membership inference attacks (MIAs) address this by testing whether a given image was in a model's training set. The standard approach scores an image by its reconstruction loss and flags low-loss images as members. This often fails because some images are simply easier to reconstruct than others, so a simple unseen image can score a lower loss than a complex training image. Difficulty calibration separates memorization from how difficult an image is to begin with. We show how to calibrate along two axes of time: training time and diffusion time. First, across training time, we introduce Reference-Model Calibration (RMC), which compares a fine-tuned model's loss against an earlier checkpoint. This cleanly isolates newly memorized data and gives near-perfect auditing, even without text captions. Since earlier checkpoints are not always available, we next present Temporal Self-Calibration (TSC), which calibrates across diffusion time using only a single model. Our key observation is that an image's difficulty persists across noise levels, whereas memorization is localized in time. By comparing the loss at a target timestep against neighboring timesteps from the same model, TSC attenuates image difficulty using the model's own trajectory. Empirically, our methods beat the state of the art by a wide margin, and unlike most existing attacks, generalize across different model parameterizations such as $x$ -, $v$ -, and $ε$ -prediction models. Code is available at https://github.com/MichaelJoseph777/TSC.

Figures & tables

Panel of Figure 1 — Figure 1: Image difficulty confounds absolute-loss thresholds. Left: CIFAR-10 DDPM log-loss trajectories for members (blue) and non-members (red) showing class means and 80% intervals. Right: Loss density at the optimal log-SNR ( $\lambda^{\star}$ ). Even at maximal separability, the populations exhibit a massive 45% overlap, rendering absolute-loss thresholds ineffective.

Panel of Figure 2 — Figure 2: RMC on MS-COCO. Stable Diffusion fine-tuned on MS-COCO in realistic settings, uncaptioned. Left: Pretrained and fine-tuned log-losses are almost perfectly correlated (Pearson correlation $r=0.989$ ). Right: Subtracting the pretrained loss removes this shared difficulty. Members shift below the ASR threshold (dashed ASR $98.6\%$ ), while non-members stay concentrated near $0$ .

		Pokémon			MS COCO			Flickr
Method		AUC	ASR	TPR@1%FPR	AUC	ASR	TPR@1%FPR	AUC	ASR	TPR@1%FPR
CLiD	( $q=15$ )	96.98	91.83	64.75	95.55	88.72	44.04	95.22	88.16	47.14
RMC	( $q=2$ )	99.44	96.51	90.63	99.88	98.52	96.60	99.80	98.14	96.86
RMC	( $q=20$ )	99.73	97.84	96.64	99.94	99.04	98.52	99.96	99.50	99.82

Table 1: RMC on realistic Stable Diffusion fine-tunes. Reference calibration is evaluated without captions at

\lambda=1

log-SNR using either one or 10 paired noise draws. CLiD uses the ground-truth caption. Parentheses provide the number of model queries per candidate.

Panel of Figure 3 — Figure 3: Temporal self-calibration. (a) A member/non-member pair where absolute loss correctly identifies the member. (b) A pair where absolute loss fails because the member is harder, yet a localized ”dip” is still visible. (c) Subtracting the local temporal reference (the chord between the reference timesteps) removes the absolute difficulty offset. Both pairs collapse into a shared calibrated view, distinctly separating members from non-members by their trajectory shape.

Panel of Figure 4 — Figure 4: TSC on STL-10-U. Mean member and non-member log-loss trajectories (left) and a close-up of the selected window (right). The chords connect the two context points for each population, and the vertical bars at $\lambda^{*}$ show the residual measured by the temporal score.

		CIFAR-10			Tiny ImageNet			STL-10-U
Method		AUC	ASR	TPR@1%FPR	AUC	ASR	TPR@1%FPR	AUC	ASR	TPR@1%FPR
Naive	( $q=20$ )	85.39	78.21	7.35	64.37	60.52	2.93	64.15	60.50	2.51
SecMI	( $q=12$ )	88.24	80.91	10.20	66.37	61.88	2.91	68.12	63.33	3.03
PIA	( $q=2$ )	91.90	84.88	25.80	68.99	64.25	4.43	69.46	64.51	4.08
PIAN	( $q=2$ )	89.26	83.32	30.90	67.04	63.44	6.10	66.72	63.36	4.32
TSC	( $q=15$ )	95.91	90.15	62.31	78.63	72.18	12.27	77.84	71.29	9.57

Table 2: Membership inference on DDPM benchmarks. Results reported as percentages. Parentheses give the number of model queries per candidate. SecMI follows its official protocol.

		Pokémon			MS COCO			Flickr
Method		AUC	ASR	TPR@1%FPR	AUC	ASR	TPR@1%FPR	AUC	ASR	TPR@1%FPR
Naive	( $q=10$ )	84.84	77.43	28.37	76.61	69.92	7.16	77.96	71.66	10.08
SecMI	( $q=12$ )	83.59	76.47	15.14	78.33	71.46	6.84	81.14	74.24	7.40
PIA ( $t=500$ )	( $q=2)$	69.86	66.13	19.71	54.08	53.96	2.76	54.73	53.24	1.96
PIA (optimal $t$ )	( $q=2$ )	94.50	87.88	61.30	75.61	70.90	17.36	74.18	69.58	17.12
PIAN	( $q=2$ )	53.34	54.15	1.92	69.20	64.08	13.88	69.15	65.10	12.40

Table 3: Membership inference on Stable Diffusion fine-tunes (uncaptioned). SecMI denotes its LDM variant. PIA, is evaluated both over the original paper’s default timestep (

t=500

) and an average optimal timestep over shadow models.

		Selected $\lambda^{\star}$ ( $x/v/\epsilon$ loss)	$x$ -loss			$v$ -loss			$\epsilon$ -loss
Method		Selected $\lambda^{\star}$ ( $x/v/\epsilon$ loss)	$x$ -pred	$v$ -pred	$\epsilon$ -pred	$x$ -pred	$v$ -pred	$\epsilon$ -pred	$x$ -pred	$v$ -pred	$\epsilon$ -pred
Naive	( $q=20$ )	$-0.75/0.25/1.50$	6.48	6.05	6.74	34.54	34.28	34.56	27.52	29.78	28.29
PIA	( $q=2$ )	$-0.75/0.25/1.75$	2.27	4.07	6.22	1.01	52.48	54.84	31.69	74.61	75.85
PIAN	( $q=2$ )	$-0.50/0.50/1.75$	2.41	6.00	5.69	1.06	56.68	58.36	72.11	77.77	76.73
SecMI	( $q=12$ )	default	0.94	1.10	1.19	5.55	5.57	5.26	29.25	35.40	30.82
TSC	( $q=15$ )	$-0.50/0.50/1.75$	30.28	28.74	31.26	81.31	80.56	80.68	91.22	93.22	92.75

Table 4: Robustness across prediction and loss parameterizations. TPR@1%FPR (%) for unaugmented CIFAR-100 DDPM++ FM models. The selected-

\lambda^{\star}

tuple lists the shared operating log-SNR under

x

v

-, and

\epsilon

-losses. SecMI follows its default protocol.

		$q=15$		$q=45$
Filter	Setting	AUC	TPR	AUC	TPR
TSC (endpoints)	$\Delta=2.5,\ k=1$	87.06	35.30	88.11	39.26
Even	$\Delta=2.5,\ k=2$	83.96	26.43	86.84	34.84
Gaussian (full)	$\sigma=3$	–	–	81.21	20.57
Gaussian (window)	$k=7,\ \sigma=3$	71.84	5.72	79.53	11.28
Reverse window	$k=7,\ \sigma=0.5$	80.58	16.26	85.65	27.95

Table 5: Temporal reference filters. Mean AUC and TPR@1%FPR across four datasets trained over DDPM settings. The full Gaussian uses 45 levels and is unavailable at

q=15

Reference	$(a,b)$	AUC	TPR
Symmetric	$(2.5,2.5)$	87.06	35.30
Left only	$(2.5,-)$	78.15	23.52
Right only	$(-,2.5)$	82.13	6.47
Unequal	$(1.5,3.5)$	88.77	30.94
Unequal	$(3.5,1.5)$	80.64	26.56
Weighted	$(1.5,3.5)$	86.24	29.11

Table 6: Temporal reference symmetry. At

q=15

, mean AUC and TPR@1%FPR across four datasets trained over DDPM settings.

(a,b)

denotes nominal log-SNR offsets.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

				$x$ -prediction	$\epsilon$ -prediction	$v$ -prediction
				$x_{\theta}:=f_{\theta}(z_{t},t,c)$	$\epsilon_{\theta}:=f_{\theta}(z_{t},t,c)$	$v_{\theta}:=f_{\theta}(z_{t},t,c)$
$x$ -loss:	$\mathbb{E}\\|$	$x_{\theta}-x$	$\\|_{2}^{2}$	$x_{\theta}$	$x_{\theta}=\frac{z_{t}-(1-t)\epsilon_{\theta}}{t}$	$x_{\theta}=z_{t}+(1-t)v_{\theta}$
$\epsilon$ -loss:	$\mathbb{E}\\|$	$\epsilon_{\theta}-\epsilon$	$\\|_{2}^{2}$	$\epsilon_{\theta}=\frac{z_{t}-tx_{\theta}}{1-t}$	$\epsilon_{\theta}$	$\epsilon_{\theta}=z_{t}-tv_{\theta}$
$v$ -loss:	$\mathbb{E}\\|$	$v_{\theta}-v$	$\\|_{2}^{2}$	$v_{\theta}=\frac{x_{\theta}-z_{t}}{1-t}$	$v_{\theta}=\frac{z_{t}-\epsilon_{\theta}}{t}$	$v_{\theta}$

Appendix

Table 7: Prediction and loss parameterizations for the linear flow schedule. Columns specify the direct network prediction space, while rows specify the space in which its squared error is measured.

Dataset	Resolution	Member	Hold-out	Aug.	Steps	Epochs
CIFAR-10	$32\times 32$	25k	25k	HFlip	800k	$\approx 4$ k
CIFAR-100	$32\times 32$	25k	25k	HFlip	800k	$\approx 4$ k
STL-10-U	$32\times 32$	50k	50k	HFlip	1.6M	$\approx 4$ k
Tiny ImageNet	$32\times 32$	50k	50k	HFlip	1.6M	$\approx 4$ k

Appendix

Table 8: DDPM training settings. All runs use

32\times 32

resolution, batch size

128

, linear noise schedule,

\epsilon

-prediction, and random horizontal flips.

Dataset	Train	Hold-out	Steps	Aug.
Pokémon	416	417	7.5k	Crop
MS-COCO	2.5k	2.5k	150k	Crop
Flickr30k	2.5k	2.5k	150k	Crop

Appendix

Table 9: Stable Diffusion v1.4 overfitting-setting fine-tuning details (Table 3 ) settings.

Dataset	Train	Hold-out	Steps	Aug.
Pokémon	416	417	7.5k	Crop+HFlip
MS-COCO	2.5k	2.5k	50k	Crop+HFlip
Flickr30k	10k	10k	200k	Crop+HFlip

Appendix

Table 10: Stable Diffusion v1.4 real-world-setting fine-tuning details (Table 1 ) settings.

		CIFAR-10		C100 hflip		STL-10-U		Tiny ImageNet
$q$	Reference $(a,b)$	AUC	TPR	AUC	TPR	AUC	TPR	AUC	TPR
15	Symmetric $(2.5,2.5)$	95.91	62.31	95.85	57.06	77.84	9.57	78.63	12.27
15	Left only $(2.5,-)$	88.78	42.48	88.95	38.22	67.39	6.42	67.48	6.97
15	Right only $(-,2.5)$	91.70	9.76	92.11	12.09	71.34	1.62	73.37	2.41
15	Unequal $(1.5,3.5)$	97.27	55.60	97.32	54.35	80.12	6.79	80.38	7.02
15	Unequal $(3.5,1.5)$	90.18	44.90	90.24	40.80	70.74	8.63	71.42	11.91

Appendix

Table 11: Reference symmetry by checkpoint. All checkpoints are held-out DDPMs at 4K epochs. AUC and TPR@1%FPR are percentages. C100 denotes CIFAR-100 with horizontal flips. Bold and underlined values denote the best and second-best results within each query budget.

Component	Mean gap	Pooled SD	AUC
Center $\ell_{x}(0)$	0.3885	0.4119	74.26
Symmetric $S$	0.2566	0.1366	87.06
Endpoint difference $O$	0.0371	0.1552	57.14
Left reference $S+O$	0.2937	0.2464	77.80
Right reference $S-O$	0.2195	0.1580	81.84

Appendix

Table 12: Components of the temporal contrast. Four-checkpoint means at

K=5

shared draws per point. Gaps and standard deviations are in log-loss units.

Figure 6: Membership localization across training losses. Single-timestep absolute-loss AUC on the nine first-half CIFAR-100 DDPM++ models without augmentation at epoch

2000

. Color denotes training-loss target

s

and line style denotes native prediction

p

. Losses are averaged over

50

noise draws per image, shared across noise levels. Lower loss indicates membership at every level. Stars mark single-timestep AUC maxima. The peaks occur at

\lambda=-0.75

0.75

, and

2.00

for

x

v

-, and

\epsilon

-loss, respectively.

Figure 7: Reference calibration across noise levels. AUC (left) and TPR@1%FPR (right) for the realistic Pokémon, MS-COCO, and Flickr30k fine-tunes. Solid curves show RMC (

q=2

). Dashed curves show the uncalibrated fine-tuned loss from the same noise draw (

q=1

). Each point uses a fixed score direction, with lower loss or calibrated residual indicating membership. The vertical guide marks

\lambda=1

, the operating point reported in Table 1 .

Membership Inference Attacks

Membership inference attacks test whether records or entities were used to train a machine-learning model, revealing potential privacy leakage.

See all 117 papers

DIME: Query-Efficient Framework for Membership Inference on Diffusion Models
Aug 24, 2026Tue Do, Daniel AlabiMembership Inference Attacks Diffusion Models
University of Illinois at Urbana-Champaign
Black-box Membership Inference Attacks on the Pre-training Data of Image-generation Models
May 26, 2026Tao Qi, Huili Wang, Yuanhong Huang +6Membership Inference Attacks Diffusion-Based Image Generation
Beijing University of Posts and Telecommunications · Tsinghua University
Adaptive Diffusion Freezing: Privacy-preserving Diffusion Models Against Membership Inference Attacks
Sep 8, 2026Jialu Guo, Xiao Han, Junjie WuPrivacy-Preserving ML Diffusion Models
MIIT Key Laboratory of Data Intelligence and Management Beihang University Beijing, China

Aug 24, 2026J/K move · Enter open · S save

DIME: Query-Efficient Framework for Membership Inference on Diffusion Models

Tue Do, Daniel Alabi

University of Illinois at Urbana-Champaign

Membership Inference Attacks Diffusion Models

Open paper AskPDF arXiv

Membership inference attacks expose whether individual records were used to train a model, yet existing attacks on diffusion models are largely heuristic and can require substantial query budgets. We introduce DIME (Denoiser Ideal Membership Error), a theoretically grounded and query-efficient framework for membership inference on diffusion models. Our starting point is an exact characterization of the optimal diffusion denoiser for a finite training set, which reveals that membership leakage is governed by the denoiser's implicit reconstruction error. This error decomposes into two complementary signals: a bias term, capturing reconstruction accuracy, and a previously unexplored local crowding term, capturing the geometry of nearby training examples. Both admit efficient estimators using only model queries, yielding a practical attack with as few as two queries. Across CIFAR-10/100, STL10-U, CelebA, and ImageNet, DIME consistently outperforms prior attacks at comparable or substantially lower query cost, improving TPR at 1% FPR by up to

3\times

; remarkably, its two-query variant can outperform existing 30-query baselines. Finally, we suggest, discuss, and evaluate specific defenses to counteract such powerful membership tests.

Abstract

Membership Inference Attacks

Explore similar work

DIME: Query-Efficient Framework for Membership Inference on Diffusion Models

Black-box Membership Inference Attacks on the Pre-training Data of Image-generation Models

Adaptive Diffusion Freezing: Privacy-preserving Diffusion Models Against Membership Inference Attacks