Distilling Diffusion Score Discrepancy for Efficient Training Data Attribution

Shixuan Liu; Joan Serrà; Kin Wai Cheuk; Jinju Kim; Woosung Choi; Yukara Ikemiya; Wei-Hsiang Liao; Jiaqi W. Ma

cs.LGSep 30, 2026

Distilling Diffusion Score Discrepancy for Efficient Training Data Attribution

Authors: Shixuan Liu, Joan Serrà, Kin Wai Cheuk, Jinju Kim, Woosung Choi, Yukara Ikemiya, Wei-Hsiang Liao, Jiaqi W. Ma, +1 more

Organizations: University of Illinois at Urbana-Champaign · Sony AI · University of Texas at Austin · Sony Group Corporation

Abstract

Training data attribution for diffusion models aims to identify the training samples that influence a generated instance, but existing methods either require costly per-sample gradient computation or query-specific model optimization. Moreover, most methods attribute changes in a proxy loss rather than changes in the actual model's generative behavior. We address these limitations by formulating attribution directly with a local score discrepancy measure, which applies to any diffusion variant (including DDPM, EDM, and flow matching), and by showing that such measure can be estimated without retraining, as a preconditioned gradient similarity. We instantiate this estimator as Training-data Influence via score Discrepancy (TID), which uses Kronecker-factored curvature to avoid random projections and per-sample gradient storage. We then distill TID into TIDE, a forward-only student trained online to reproduce the teacher's rankings from the diffusion model's internal activations. Under counterfactual evaluation on CIFAR-10, ArtBench-10, and MS-COCO, TID matches or outperforms state-of-the-art approaches, while TIDE retains most of TID's accuracy at four to five orders of magnitude lower per-query cost, attributing generated samples in milliseconds and faster than the generation itself.

Figures & tables

Figure 1: Qualitative comparison of five target queries from each dataset. Top row: original generations from the full-data model. Rows below: regenerations from the same initial noise after removing the samples attributed by each method. Random removal leaves regenerations nearly unchanged, whereas removal by MUCS, TID , and TIDE induces a visible drift.

	Approach	Categ.	SSIM	SSCD	LPIPS	CLIP	$\mu$
CIFAR-10	CLIP	MA	0.598 $\pm$ 0.011	0.585 $\pm$ 0.008	0.610 $\pm$ 0.013	0.661 $\pm$ 0.010	0.613 $\pm$ 0.007
	DINO	MA	0.604 $\pm$ 0.018	0.616 $\pm$ 0.017	0.638 $\pm$ 0.021	0.665 $\pm$ 0.027	0.631 $\pm$ 0.019
	D-TRAK	G	0.691 $\pm$ 0.037	0.649 $\pm$ 0.033	0.703 $\pm$ 0.026	0.640 $\pm$ 0.042	0.671 $\pm$ 0.033
	DAS	G	0.760 $\pm$ 0.031	0.753 $\pm$ 0.036	0.789 $\pm$ 0.040	0.746 $\pm$ 0.035	0.762 $\pm$ 0.035
	MUCS	U	0.864 $\pm$ 0.025	0.854 $\pm$ 0.023	0.908 $\pm$ 0.015	0.857 $\pm$ 0.028	0.870 $\pm$ 0.021
	TID (ours)	G	0.836 $\pm$ 0.031	0.815 $\pm$ 0.031	0.872 $\pm$ 0.023	0.818 $\pm$ 0.027	0.835 $\pm$ 0.027

Table 1: Counterfactual AUC per similarity metric and their average

\mu

, reported as the mean over six seeds

\pm

the standard error. Categories: MA = model-agnostic, G = gradient-based, U = unlearning-based, D = distillation-based. Best per column in bold , second best underlined .

Panel of Figure 2 — Figure 2: Attribution accuracy versus warm-cache per-query cost on CIFAR-10 (left), ArtBench (middle), and MS-COCO (right). The horizontal axis is logarithmic; cacheable training-side artifacts are precomputed. Markers with different fills connected by dotted lines trace the variation with $M$ .

RQ/Variant	SSIM	SSCD	LPIPS	CLIP	$\mu$	$\Delta$
TID (ours)	0.836	0.815	0.872	0.818	0.835	—
RQ1: Attribution target
$\lVert F_{\theta}\rVert_{2}^{2}$	0.815	0.797	0.855	0.790	0.814	$-$ 2.5%
$\omega(\sigma)\lVert\mathbf{x}-F_{\theta}\rVert_{2}^{2}$	0.777	0.737	0.785	0.731	0.757	$-$ 9.3%
RQ2: Corruption alignment
Independent draws	0.759	0.743	0.798	0.732	0.758	$-$ 9.2%

Table 2: Counterfactual AUCs for the ablation of the TID teacher on CIFAR-10. Each block isolates one design choice while holding the remaining components fixed. All variants use

M=32

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Ablation variants of TID
Method	Target	Curvature	Kernel	Normalization
D-TRAK	$\lVert F_{\theta}\rVert_{2}^{2}$	GN $+$ ridge	random projection	none
DAS	$\omega(\sigma)\lVert\mathbf{x}-F_{\theta}\rVert_{2}^{2}$	GN $+$ ridge	random projection	gradient
K-FAC influence	$\omega(\sigma)\lVert\mathbf{x}-F_{\theta}\rVert_{2}^{2}$	MC-GGN	K-FAC	none
TID (ours)	$\langle\mathbf{1},F_{\theta}\rangle$	empirical Fisher	K-FAC	factors
RQ1a: squared output	$\lVert F_{\theta}\rVert_{2}^{2}$	empirical Fisher	K-FAC	factors

Appendix

Table 3: Design choices of gradient-based diffusion attribution methods: D-TRAK ( Zheng et al., 2024 ) , DAS ( Lin et al., 2025 ) , K-FAC influence ( Mlodozeniec et al., 2025 ) , and TID . Here,

F_{\theta}

denotes the denoiser output evaluated on a corrupted sample and

\mathbf{x}

denotes the corresponding clean sample. Curvature specifies the preconditioner; kernel specifies how per-sample gradients are represented before computing inner products; and normalization specifies which representation, if any, is normalized at each timestep. Gradient normalization rescales each projected gradient to unit norm, whereas factor normalization rescales the two Kronecker factors independently, equivalently normalizing the corresponding rank-one gradient matrix by its Frobenius norm. Without normalization, large-gradient samples can dominate the aggregation across timesteps. The lower block varies individual components of TID and is evaluated quantitatively in Table 2 .

		CIFAR-10	ArtBench-10	MS-COCO
Data	Training samples $N$	50,000	49,917	118,287
	Resolution	$32{\times}32$	$64{\times}64$	$64{\times}64$
	Conditioning	none	class label	CLIP text embedding
Backbone	Architecture	DiT	DiT	DiT
	Transformer blocks	12	12	12
	Model width	768	768	768

Appendix

Table 4: Dataset, model, training, sampling, and evaluation configurations. Attribution-specific settings are reported separately in Table 5 .

		Setting
TID	Timestep schedule	Karras
	Corruption draw	shared
	Relative damping $\alpha$	0.1
	Prenormalization	per sample
	Inference draws $M$	32
	Inference draw alignment	aligned $(\sigma,\mathbf{n})$

Appendix

Table 5: Attribution settings for TID and TIDE . We use the same configuration on all three datasets. Table 1 reports results with

M{=}32

, while Fig. 2 varies the budget over

\{1,32,100\}

Method	CIFAR-10	ArtBench-10	MS-COCO
Random	16,636 (33.3%)	16,589 (33.2%)	39,332 (33.3%)
CLIP	13,025 (26.1%)	10,349 (20.7%)	34,384 (29.1%)
DINO	13,728 (27.5%)	11,857 (23.8%)	35,952 (30.4%)
D-TRAK	15,586 (31.2%)	13,874 (27.8%)	35,154 (29.7%)
DAS	13,758 (27.5%)	14,262 (28.6%)	38,006 (32.1%)
MUCS	15,448 (30.9%)	14,093 (28.2%)	34,096 (28.8%)

Appendix

Table 6: Effective removal size, averaged over six seeds, with the fraction of the training set shown in parentheses. Each method selects the top

2\%

of samples for each of 20 queries; overlap among these per-query sets reduces the union below the nominal total of

40\%

. The random control has the least overlap and therefore removes the most distinct samples.

Figure 4: Qualitative rankings on CIFAR-10. Each row corresponds to a method in Table 1 and shows the query, the eight most influential training samples, and the eight least influential samples. The arrow indicates decreasing attributed influence.

Figure 5: Qualitative rankings on ArtBench-10, following the format of Fig. 4 . For this densely painted query, TID and TIDE place sparse line drawings at the bottom of their rankings.

Figure 6: Qualitative rankings on MS-COCO, following the format of Fig. 4 . All methods retrieve skateboarding scenes at the top of the ranking, while their lowest-ranked samples differ.

	$1\%$	$2\%$	$3\%$	$5\%$
TIDE (ours)	0.784	0.858	0.910	0.938

Appendix

Table 7: Average counterfactual AUC of TIDE on CIFAR-10 across per-query removal budgets. Performance increases consistently with the budget as larger removals induce more detectable changes in the retrained model.

	Approach	Category	SSIM	SSCD	LPIPS	CLIP	$\mu$
CIFAR-10	CLIP	MA	$-3.9{\scriptstyle\pm 0.6}$	$-6.0{\scriptstyle\pm 1.1}$	$-5.1{\scriptstyle\pm 0.7}$	$-7.5{\scriptstyle\pm 0.6}$	$-5.6{\scriptstyle\pm 0.7}$
	DINO	MA	$-4.5{\scriptstyle\pm 0.7}$	$-7.5{\scriptstyle\pm 0.8}$	$-6.2{\scriptstyle\pm 0.6}$	$-8.0{\scriptstyle\pm 0.9}$	$-6.5{\scriptstyle\pm 0.7}$
	D-TRAK	G	$-6.6{\scriptstyle\pm 0.8}$	$-8.6{\scriptstyle\pm 1.1}$	$-8.2{\scriptstyle\pm 0.8}$	$-7.1{\scriptstyle\pm 1.5}$	$-7.6{\scriptstyle\pm 1.0}$
	DAS	G	$-9.2{\scriptstyle\pm 1.2}$	$-14.4{\scriptstyle\pm 1.9}$	$-12.0{\scriptstyle\pm 1.4}$	$-11.8{\scriptstyle\pm 1.0}$	$-11.9{\scriptstyle\pm 1.3}$
	MUCS	U	$-12.4{\scriptstyle\pm 0.7}$	$\underline{-19.2}{\scriptstyle\pm 0.9}$	$-16.8{\scriptstyle\pm 0.5}$	$\mathbf{-16.9}{\scriptstyle\pm 0.9}$	$\mathbf{-16.3}{\scriptstyle\pm 0.6}$
	TID (ours)	G	$\underline{-12.7}{\scriptstyle\pm 0.8}$	$-18.1{\scriptstyle\pm 1.0}$	$\underline{-16.9}{\scriptstyle\pm 0.5}$	$\underline{-15.7}{\scriptstyle\pm 1.1}$	$-15.8{\scriptstyle\pm 0.8}$

Appendix

Table 8: Relative change in query–regeneration similarity (%) after removal, reported as the mean

\pm

standard error across seeds. More negative values indicate larger induced drift. The random-removal control yields approximately

-

2 to

-

3%. Method categories follow Table 1 ; the strongest and second-strongest results in each column are shown in bold and underlined , respectively.

Method	One-off compute	Cache	Per-query compute	Score flops
CLIP / DINO	$Nc_{\mathrm{enc}}$	$Nd_{\mathrm{emb}}$	$c_{\mathrm{enc}}$	$Nd_{\mathrm{emb}}$
D-TRAK	$NMc_{\mathrm{fb}}$	$Nd_{\mathrm{proj}}$	$Mc_{\mathrm{fb}}$	$Nd_{\mathrm{proj}}$
DAS	$NMc_{\mathrm{fwd}}+NMc_{\mathrm{fb}}$	$Nd_{\mathrm{proj}}$	$Mc_{\mathrm{fb}}$	$Nd_{\mathrm{proj}}$
TID	$NMc_{\mathrm{fb}}+D_{3}$	$D_{2}$	$\tfrac{NM}{Q}c_{\mathrm{fb}}+Mc_{\mathrm{fb}}$	$NMD_{1}$
TIDE	$EN\,(c_{\mathrm{fb}}+c_{\mathrm{enc,fb}})+NM(c_{\mathrm{fwd}}+c_{\mathrm{enc}})$	$NMd_{\mathrm{emb}}$	$M(c_{\mathrm{fwd}}+c_{\mathrm{enc}})$	$NMd_{\mathrm{emb}}$
MUCS	—	—	$Uc_{\mathrm{fb}}+NMc_{\mathrm{fwd}}$	$NM$

Appendix

Table 9: Computation and storage by phase. “One-off” is computed once on the training set and cached; “per query” is Eq. 21 .

c_{\mathrm{fwd}}

c_{\mathrm{fb}}

, and

c_{\mathrm{enc}}

are the costs of one forward pass, one forward–backward pass of

F_{\theta}

, and one external-encoder pass;

D_{1}

D_{2}

are the sizes of the uncompressed factors, the K-FAC matrices, and

D_{3}

is the computational cost of their eigendecompositions. Cache sizes are in floats; score flops are the matrix multiplies of the final scoring step. We omit the big

O

notation from the whole table.

		CIFAR-10	ArtBench-10	MS-COCO
Method	One-off computation	wall-clock / cache size
CLIP	encode training set	98 s / 0.10 GB	112 s / 0.10 GB	277 s / 0.24 GB
DINO	encode training set	18 s / 0.15 GB	18 s / 0.15 GB	37 s / 0.36 GB
DAS	error + projected-gradient store	18.1 h / 3.3 GB	12.3 h / 3.3 GB	38.1 h / 7.8 GB
TIDE	distillation + embedding collection	3.8 h / 0.15 GB	4.2 h / 0.15 GB	10.0 h / 0.36 GB
TID ( $M{=}1$ )	factor estimation	0.2 h / 2.4 GB	0.2 h / 2.4 GB	0.5 h / 2.5 GB

Appendix

Table 10: One-off training-side cost and resulting cache size on a single H100. Encoder and embedding-bank sizes are computed as

N_{\text{train}}d\times 4

bytes; all other cache sizes are measured on disk. The reported wall-clock times are direct measurements from a dedicated timing seed. “n/a” indicates that MUCS performs no training-side preprocessing.

Explore similar work

Jun 26, 2026cs.CV

Diffusion Model Attribution via Spectral Coupling of Denoiser Responses

Attributing a generated image to its source diffusion model is a fundamental challenge in provenance verification and intellectual property protection. This problem is particularly difficult because diffusion models trained on different datasets can converge to similar score functions and thus similar output distributions, making the generated images themselves unreliable as attribution evidence. Existing non-invasive methods either fail on architecturally similar variants or rely on signals that vanish when models share the same autoencoder. We propose Spectral Denoising Signatures (SDS), a non-invasive attribution method that identifies the source model by fingerprinting each candidate model's denoising behavior. Our key insight is that a model's denoising score function exhibits a distinctive spectral geometry, reflected in how it redistributes energy across spatial frequency bands during denoising. By probing this behavior with frequency-controlled perturbations, SDS extracts a stable signature that is intrinsic to the model, requiring only standard forward passes with no inversion, optimization, or generation-time enrollment. Our results demonstrate that SDS achieves approximately 99.9% accuracy across eight diverse diffusion models and 96.2% under cross-domain prompt shift, outperforming non-invasive baselines across variations in training data, architecture, and training procedure, establishing spectral geometry as a principled and practical basis for diffusion model attribution. Code is available at: https://github.com/Pragati-Meshram/SGS

Pragati Shuddhodhan Meshram, Varun Chandrasekaran

Department of Electrical and Computer Engineering University of Illinois Urbana-Champaign

Signatures Spectral Features

Ask PDF

May 18, 2026cs.LG

Training data attribution in diffusion models via mirrored unlearning and noise-consistent skew

Training data attribution (TDA) should enable generative model interpretability and foster a variety of related downstream tasks. Nonetheless, current TDA approaches lack reliability and robustness, preventing their adoption in real-world setups. In this paper, we take a decisive step towards more reliable and robust TDA for diffusion models. We propose to perform TDA with mirrored unlearning and noise-consistent skew (MUCS). The idea is to fine-tune a second model with bounded mirrored gradient ascent, and to measure the normalized skew of this model with respect to the original one using consistent noise samples. We show that, while being conceptually simple and generic, MUCS systematically outperforms existing methods on three different datasets by a large margin. We additionally study the effect that core design choices have on final performance, and analyze novel aspects regarding the overlap of influential instances across generated items and the potential of ensembling TDA approaches. We believe that our findings may have broader implications for more general unlearning setups, as well as for tasks requiring the comparison of diffusion losses.

Joan Serrà, Dipam Goswami, Fabio Morreale +2

Sony AI · Sony Group Corporation

Diffusion Models Training-Inference Mismatch Training Data Skewness

Ask PDF

Jun 13, 2026cs.LG

Temporal Difference Learning for Diffusion Models

Diffusion models are typically trained with objectives that focus on local denoising targets at individual time steps (or adjacent pairs), which do not enforce consistency between predictions along the denoising trajectory. This lack of cross-time consistency can degrade performance, especially for few-step samplers. We introduce a temporal difference (TD) objective that penalizes inconsistency of the model's multi-step progress along the denoising path. By reformulating the diffusion process as a Markov reward process and casting denoising as a policy evaluation problem in reinforcement learning, we derive a unified TD approach that applies to both discrete- and continuous-time diffusion formulations. We further propose a principled sample-based reweighting method that stabilizes training. Empirically, we show that using our TD training can significantly improve sample quality measured by FID, with stronger advantages when the number of sampling steps is small, highlighting its practical utility under low-computation-budget scenarios. We provide ablation studies to justify our design choices, including pairwise loss reweighting, regularization weight, and one-step stride. Overall, our TD approach can be a general drop-in that enforces cross-time consistency and improves generation quality across different diffusion generative models.

Qizhen Ying, Yangchen Pan, Victor Adrian Prisacariu +1

Department of Engineering Science, University of Oxford, Oxford, United Kingdom · School of Computer Science, Carleton University, Ottawa, Canada.

Denoising Trajectory Generative Models Temporal Difference

Ask PDF