Diffusion models sometimes memorize and reproduce regions of their training images, raising privacy and copyright concerns. We aim to localize these regions. Existing contrast-based methods use the diagonal of the denoiser Jacobian, which reflects the posterior variance of each pixel. However, a small diagonal entry is ambiguous: it signals either memorization or averaging over many neighboring pixels in smooth or predictable regions. To resolve this ambiguity, these methods subtract a reference map, which requires a second evaluation with a null prompt or an earlier checkpoint. Unconditional models have no null prompt, and earlier checkpoints are unavailable in general. We show that Jacobian row sums (JaRS), unlike the diagonal, separate memorization from local averaging without a reference map. Pairing JaRS with an image-level detection score that uses the same statistic at an early denoising step yields JaRS+. Across Stable Diffusion v1.4 and v2.1, Realistic Vision v5.1, and an unconditional diffusion Transformer (DiT), JaRS+ outperforms contrast-based localization methods.
Figures & tables
SD v1.4
SD v2.1
Realistic Vision
DiT (unconditional)
Method
Reference
TV
TV+N
All
TV
TV+N
TV
All
WM
WM+N
All
Raw diagonal [ 15 ]
–
0.594
0.500
0.524
0.652
0.500
0.660
0.530
0.261
0.500
0.640
Bright Ending † [ 6 ]
–
0.751
–
–
0.933
–
0.777
–
–
–
–
Δh∅ [ 15 ]
null prompt
0.900
0.950
0.945
0.945
0.887
0.930
0.951
not available
Δhθ~ [ 15 ]
checkpoint
0.913
0.903
0.884
0.948
0.876
0.936
0.880
0.439
0.500
0.666
Δs∅ [ 15 ]
null prompt
0.826
0.901
0.898
0.787
0.809
0.876
0.903
not available
Table 1: Pixel-level localization of memorized regions on the test splits (mIoU) under the balanced protocol of Kim and Lee [15] . The baselines of Kim and Lee [15] are evaluated at the final sampling step (step 48 on the DiT, at the same noise level σt/αt ) using 16 Hutchinson probes; checkpoint references for Δhθ~ and Δsθ~ are SD v1.1 (for SD v1.4 and Realistic Vision), SD v2.0-base (for SD v2.1), and the 5 k-step checkpoint, which is best on the DiT development split for both contrasts. JaRS+ requires no reference prompt or checkpoint, and its hyperparameters (ℓ,d,τ) are chosen once per model on the development split; JaRS denotes the map at step ℓ without the image-level gate (Stage 1) of JaRS+. In settings with non-memorized images, an mIoU of 0.500 corresponds to predicting an empty mask everywhere. † Numbers reported by Kim and Lee [15] on the full benchmark.
Figure 2: Full denoiser Jacobian rows of Stable Diffusion v1.4 (step 15 ) and DiT-B/2 (step 35 ). Left: For one development image from each category, we show the generated image (copied region outlined in cyan; pixels outside a copied template are dimmed), the row Ji,: of the full Jacobian at the marked pixel ‘ + ’ together with its sum, and the JaRS map ( 10 ). Right: Empirical distributions of the per-channel row sum ∑jJij across all pixels of non-memorized images and whole-image copies, and inside the copied region of template verbatims and watermarked samples ( 10 development images per category; 7 retrieved verbatims). The vertical dotted line at 1 marks the row sum of a mean-preserving averaging filter.
SD v1.4
SD v2.1
DiT
Method
AUC
TPR
AUC
TPR
AUC
TPR
Bright Ending † [ 6 ]
0.886
0.390
0.945
0.877
–
–
Raw diagonal [ 15 ]
0.906
0.547
0.871
0.040
0.747
0.308
Δh∅ [ 15 ]
0.969
0.915
0.995
0.966
not available
Δhθ~ [ 15 ]
0.951
0.897
0.994
0.954
0.867
0.363
Δs∅ [ 15 ]
0.969
0.940
0.994
0.966
not available
Table 2: Prompt- and sample-level memorization detection on the test split , measured by AUROC (‘AUC’) and true-positive rate at 1% false-positive rate (‘TPR’). On Stable Diffusion, the baselines of Kim and Lee [15] are our runs of their official code (compared to their reported numbers in App. I ); on the DiT, the 50 k- and 150 k-step references of Δhθ~ and Δsθ~ are selected by development AUROC. † Reported by Kim and Lee [15] on all prompts.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Per-channel row sum ∑jJij (top) and ℓ1 row norm ∑j∣Jij∣ (bottom) of the full denoiser Jacobian at the localization step of each model (DiT-B/2 at step 35 , SD v1.4 and SD v2.1 at step 15 ), across all pixels of non-memorized images and whole-image copies, and inside the copied region of watermarked samples and template verbatims. The development images are those of Sec. 5.2 , together with 10 template verbatims and 10 non-memorized images of SD v2.1. The vertical dotted line at 1 marks the row sum of a mean-preserving averaging filter.
Model
Memorized pixels
Row sum
ℓ1 norm
DiT-B/2
Watermark
0.998
0.900
Near-copy
1.000
1.000
SD v1.4
Template (TV)
0.980
0.908
Matching verbatim
0.917
0.772
Retrieved verbatim
0.862
0.574
SD v2.1
Template (TV)
0.955
0.934
Appendix
Table 3: Pixel-level AUROC between memorized and non-memorized pixels for the JaRS map ( 10 ) and its ℓ1 variant, which replaces each row sum by the ℓ1 norm of the row, on the development images of Fig. 3 .
SD v1.4
SD v2.1
Realistic Vision
DiT (unconditional)
Detection Variant
TV
TV+N
All
TV
TV+N
TV
All
WM
WM+N
All
JaRS+ (tuned τ , main paper)
0.969
0.975
0.940
0.980
0.989
0.929
0.849
0.638
0.799
0.869
JaRS+ ( τ at 5% dev FPR)
0.969
0.923
0.906
0.980
0.959
0.944
0.854
0.463
0.725
0.817
JaRS (no Stage 1, step ℓ )
0.969
0.619
0.637
0.980
0.571
0.949
0.612
0.836
0.873
0.909
Appendix
Table 4: Ablation of Stage 1 (image-level detection) variants on the test splits (mIoU). All hyperparameters are selected on the development split of each model. ‘JaRS+ (tuned τ )’ chooses τ to maximize development mIoU on the broadest balanced setting; ‘JaRS+ ( τ at 5% dev FPR)’ calibrates τ using only non-memorized development images at the 95 th percentile; and ‘JaRS’ returns the localization map stℓ for every image without Stage 1 gating.
Candidate Detection Step d
0
5
10
15
20
25
30
35
40
45
Stable Diffusion v1.4
0.955
0.988
0.987
0.968
0.936
0.887
0.861
0.850
0.820
0.809
Stable Diffusion v2.1
0.9998
1.000
0.996
0.974
0.911
0.826
0.797
0.786
0.778
0.763
Realistic Vision v5.1
0.916
0.999
0.983
0.912
0.869
0.844
0.830
0.817
0.807
0.814
Appendix
Table 5: Development AUROC of the Stage 1 detection score zd=Q0.9(std) across candidate denoising steps d , evaluated on the balanced development split (‘All’ for SD v1.4 and Realistic Vision, ‘TV+N’ for SD v2.1). Bold indicates the selected step d .
Fraction Passing Stage 1 (All)
Model
TV
Global
Non-mem.
Oracle Gate (JaRS+)
SD v1.4
1.000
0.925
0.022
0.960 (0.940)
Realistic Vision
0.974
0.735
0.051
0.945 (0.849)
Appendix
Table 6: Disentangling Stage 1 (detection) and Stage 2 (localization) on the test All split. Columns 2–4 report the fraction of images in each category that pass the Stage 1 gate ( zd>τ ). Column 5 reports the mIoU of the Stage 2 JaRS localization maps when paired with an oracle image-level detector (with the actual two-stage JaRS+ mIoU in parentheses).
SD v1.4
SD v2.1
Realistic Vision
DiT (unconditional)
Method
TV
TV+N
All
TV
TV+N
TV
All
WM
WM+N
All
Ungated, step ℓ tuned
Raw diagonal
0.779
0.608
0.720
0.812
0.575
0.844
0.649
0.327
0.500
0.632
Δh∅
0.900
0.950
0.945
0.945
0.887
0.930
0.951
not available
Δhθ~
0.913
0.903
0.884
0.948
0.876
0.936
0.880
0.790
0.860
0.849
Δs∅
0.844
0.709
0.733
0.911
0.579
0.876
0.903
not available
Appendix
Table 7: Localization mIoU on the test splits when every method is tuned on the development split in the same way. Top: each map at its selected step ℓ , without an image-level gate. Bottom: each method with its own image-level gate, selected as for JaRS+. Bold marks the best value in each column within each block.
Figure 4: Effect of noise level on localization accuracy. Test mIoU of the JaRS map st on locally memorized instances (TV for text-to-image models, WM for the DiT) plotted against the noise-to-signal ratio σt/αt at which the map is evaluated. Black circles mark the localization step ℓ selected on the development split.
Figure 5: Evolution of the JaRS map st across the 50 -step sampling trajectory on the development split of the localization benchmark. Top: Image-level Stage 1 score zt=Q0.9(st) for each category (median and interquartile range across development instances; category counts are listed in parentheses). Bottom: Pixel-level values of st inside vs. outside the copied region for template verbatims (TV) and watermarked samples (WM), and across the full image for the remaining categories (median and interquartile range over 1,024 randomly sampled pixels per image). Vertical dotted lines mark the detection step d (top) and localization step ℓ (bottom) selected on the development split ( Sec. 4.3 ). On the unconditional DiT, all 154 development samples are shown; at step 0 ( t=1 ) of the flow-matching schedule, the scaled denoiser Jacobian vanishes identically ( J=0 ), so st=C at every pixel.
Localization (mIoU)
Detection (AUC)
Reference θ~
Dev All
WM
WM+N
All
Dev
Test
Δhθ~
5 k ( 2.5% )
0.666
0.439
0.500
0.666
0.800
0.803
20 k ( 10% )
0.663
0.503
0.500
0.663
0.859
0.860
50 k ( 25% )
0.661
0.457
0.500
0.655
0.866
0.867
100 k ( 50% )
0.634
0.369
0.500
0.616
0.864
0.864
150 k ( 75% )
0.531
0.243
0.500
0.526
0.849
0.833
Appendix
Table 8: Sensitivity of the checkpoint contrasts Δhθ~ and Δsθ~ [ 15 ] on the unconditional DiT (trained for 200 k steps) across all saved intermediate checkpoints θ~ (fraction of training completed shown in parentheses), evaluated at step 48 ( σt/αt=0.042 , matching the final step of Stable Diffusion) with 16 Hutchinson probes. The ‘Dev’ columns are used to select the reference checkpoint for each task (bold indicates the selected checkpoint); the remaining columns report test-split performance. Sample-level detection follows the protocol of Sec. 5.4 .
SD v1.4 (Test Prompts)
SD v2.1 (219 List, Test)
Method
Overall AUC [ 95% CI]
TV ( 147 )
MV ( 44 )
RV ( 28 )
N-ver. ( 181 )
Overall AUC [ 95% CI]
TV ( 145 )
N-ver. ( 29 )
Raw diagonal [ 15 ]
0.906 [0.882, 0.928]
0.960
0.966
0.927
0.845
0.871 [0.842, 0.899]
0.882
0.819
Δh∅ [ 15 ]
0.969 [0.955, 0.982]
1.000
1.000
0.999
0.932
0.995 [0.989, 1.000]
1.000
0.972
Δhθ~ [ 15 ]
0.951 [0.931, 0.970]
1.000
1.000
0.999
0.892
0.994 [0.986, 0.999]
1.000
0.964
Δs∅ [ 15 ]
0.969 [0.954, 0.983]
1.000
1.000
1.000
0.931
0.994 [0.986, 0.999]
1.000
0.966
Δsθ~ [ 15 ]
0.957 [0.939, 0.974]
1.000
1.000
1.000
0.905
0.993 [0.986, 0.999]
1.000
0.961
Appendix
Table 9: Per-category breakdown of prompt-level detection on the test split (extending Tab. 2 ). Each category’s AUC is computed against all 400 non-memorized test prompts. Prompt subcategories follow the verification of Kim and Lee [15] across the four evaluated seeds (‘TV’ and ‘MV’: prompt copies on at least one of the four seeds; ‘RV’: retrieved verbatim; ‘N-ver.’: prompts from the list of Webster [33] that produce no verified copy under the four seeds; test prompt counts in parentheses). 95% confidence intervals are computed via 2,000 bootstrap resamples.
Method
Overall AUC [ 95% CI]
TPR
TV ( 156 )
MV ( 35 )
RV ( 11 )
N-ver. ( 198 )
Raw diagonal [ 15 ]
0.631 [0.591, 0.670]
0.018
0.634
0.743
0.538
0.613
Δh∅ [ 15 ]
0.937 [0.917, 0.957]
0.865
1.000
1.000
1.000
0.873
Δhθ~ [ 15 ]
0.907 [0.882, 0.931]
0.825
1.000
1.000
1.000
0.813
Δs∅ [ 15 ]
0.960 [0.942, 0.976]
0.935
1.000
1.000
1.000
0.918
Δsθ~ [ 15 ]
0.918 [0.894, 0.941]
0.790
1.000
0.992
0.968
0.838
JaRS+ (ours)
0.899 [0.875, 0.922]
0.640
0.995
0.964
0.977
0.807
Appendix
Table 10: Prompt-level memorization detection on Realistic Vision v5.1 (test split) , measured by AUROC (‘AUC’) and true-positive rate at a 1% false-positive rate (‘TPR’) across 400 memorized and 400 non-memorized test prompts. Prompt subcategories follow the verification of Kim and Lee [15] on Realistic Vision (test prompt counts in parentheses), with each category’s AUC computed against all 400 non-memorized test prompts. 95% confidence intervals are computed via 2,000 bootstrap resamples. The baselines of Kim and Lee [15] use 16 Hutchinson probes for the curvature maps and SD v1.1 as the checkpoint reference, matching Tab. 1 .
Stable Diffusion
DiT
Method
Step
Precision
TFLOPs
SD v1.4 (s)
SD v2.1 (s)
RV (s)
TFLOPs
Time (s)
Raw diagonal
last
fp16 / fp32
16.6
0.49
0.43
0.50
1.47
0.106
Δh∅
last
fp16
33.3
0.81
0.69
0.81
–
–
Δhθ~
last
fp16 / fp32
33.3
0.99
0.86
0.99
2.95
0.213
Δs∅
last
fp16
1.6
0.04
0.03
0.04
–
–
Δsθ~
last
fp16 / fp32
1.6
0.05
0.04
0.05
0.09
0.012
Appendix
Table 11: Cost of each localization map per test image on one NVIDIA A6000 GPU. FLOPs are counted with PyTorch’s FLOP counter; they do not depend on the image and are the same for the three Stable Diffusion models. Times are medians over 20 test prompts (test samples on the DiT) with three repetitions each. Every map is computed as in Tab. 1 : on Stable Diffusion, the maps of Kim and Lee [15] with their official fp16 code (one forward pass that keeps the graph and 16 Hutchinson vector-Jacobian products), and on the DiT with our fp32 implementation ( 16 Rademacher probes with central-difference Jacobian-vector products); JaRS in fp32 . ‘Last’ is step 49 of 50 on Stable Diffusion and step 48 on the DiT. JaRS+ computes the Stage 2 map only for images accepted by Stage 1: ‘both stages’ is the cost of an accepted image, and ‘mean’ averages over the broadest balanced test setting (All for SD v1.4, Realistic Vision, and the DiT; TV+N for SD v2.1), on which Stage 1 accepts 65% , 50% , 59% , and 67% of the images. Generating the image ( 50 steps, with classifier-free guidance on Stable Diffusion) is listed for reference.
Mitigation y~
Scope
Copied ↓
LPIPS Rˉ↓
LPIPS Tˉ↓
CLIP ↑
None
Original image
0.975
0.000
0.000
29.0
RTA [ 30 ]
Whole image
0.432
0.444
0.437
25.0
Whole image (strong)
0.245
0.600
0.591
21.3
Region R , JaRS (ours)
0.432
0.094
0.148
26.8
Region R , JaRS, strong (ours)
0.284
0.121
0.188
24.8
Region R , Δhθ~ [ 15 ]
0.471
0.082
0.146
26.9
Appendix
Table 12: Region-restricted vs. whole-image memorization mitigation on the 278 test TV instances of Stable Diffusion v1.4. ‘Copied’ is the fraction of mitigated images that still match the training template under the benchmark copy criterion; LPIPSRˉ and LPIPSTˉ measure perceptual change relative to the original generation outside the predicted mask R and outside the ground-truth template T , respectively; and ‘CLIP’ measures image-prompt alignment. We compare global mitigation against restricting the intervention to the mask R obtained from JaRS (contrast-free) or from Δhθ~ [ 15 ] .
Figure 6: Qualitative comparison of whole-image vs. region-restricted mitigation on six randomly sampled test ‘TV’ instances of Stable Diffusion v1.4 (columns 1, 3, 4, and 5 share the same memorized bedroom template). Row 1 shows the original generation; Row 2 overlays the dilated region R localized by JaRS (magenta corresponds to the unmemorized complement Rˉ that is preserved); Row 3 shows the result of applying the prompt-embedding optimization of Wen et al. [34] globally to the whole image; and Row 4 shows the result of restricting the same optimization to R ( Sec. K.1 ). Whole-image mitigation fails to remove the copied template in four of the six instances and heavily alters the unmemorized foreground rug whenever it succeeds. Restricting mitigation to R removes the copied background template in five of the six instances while keeping the unmemorized region Rˉ intact (inset labels report whether the template remains copied and the LPIPSRˉ distance to the original generation).