TACO: Trajectory-Aware Curvature Observation for Memorization Localization and Mitigation in Diffusion Models
Authors: Ruxi Gu, Zilei Wang, Wei Wang
Organizations: Department of Automation, University of Science and Technology of China · State Key Laboratory of General Artificial Intelligence, BIGAI · School of Computer and Communication Engineering, University of Science and Technology Beijing
Text-to-image diffusion models can reproduce memorized training content within localized image regions, making precise localization essential for targeted mitigation. However, curvature observations at a single denoising state can contain transient responses, while spatially uniform mitigation may disrupt prompt-relevant content. We present Trajectory-Aware Curvature Observation (TACO), a unified framework for local memorization detection and mitigation in diffusion models. Our empirical analysis reveals that memorization-related curvature becomes pronounced during late-stage denoising and persists at consistent spatial locations across neighboring states. Motivated by these observations, TACO aggregates prompt-dependent curvature responses along the late denoising trajectory and applies latent-guided spatial refinement to improve spatial coherence. The resulting memorization map serves both as a localization signal and as a spatial control signal for mitigation. Specifically, TACO uses this map to steer existing inference-time interventions toward memorization-related regions through spatial reweighting and localized guidance correction. Extensive experiments demonstrate that TACO consistently outperforms competing localization methods and improves the Pareto trade-off between memorization suppression and generation quality. These results establish trajectory-aware spatial analysis as an effective link between memorization localization and targeted mitigation.
Figures & tables
Figure 1: Global and local memorization in Stable Diffusion v1.4 [ 24 ] across 6 different seeds.
Figure 2: Empirical studies motivating trajectory-aware curvature observation. (a) Evolution of peak curvature intensity along the sampling trajectory. (b) Spatial consistency of curvature responses between neighboring late-stage states. (c) Signal-to-background ratio and background variance of individual and fused late-stage curvature responses.
Model
Method
TV only
All
mIoU( ↑ )
ACC( ↑ )
mIoU( ↑ )
ACC( ↑ )
SD v1.4
All-zeros
0.000
0.437
0.318
0.614
All-ones
0.563
0.563
0.386
0.386
BE
0.693
0.766
0.467
0.780
cov
0.907
0.941
0.907
0.937
cov_bad
0.922
0.950
0.847
0.940
Table 1: Quantitative evaluation of memorization localization. Bold and underlined values denote the best and second-best results, respectively.
Figure 3: Qualitative comparison of memorization localization and map-guided mitigation for prompts: “Foyer painted in HABANERO” (top) and “Plymouth Curtain Panel featuring Madelyn - White Botanical Floral Large Scale by heatherdutton” (bottom). For each method, the left panel shows its original localization map, with blue and red indicating weaker and stronger memorization responses, respectively. The right panel shows the mitigated image obtained by applying our proposed steering method with the corresponding localization map.
Figure 4: SSCD-CLIP and SSCD-ImageReward Pareto fronts in SD v1.4.
Figure 5: SSCD-CLIP and SSCD-ImageReward Pareto fronts in SD v2.0.
Figure 6: Qualitative comparison of existing memorization mitigation methods with and without TACO steering. Each pair compares the original mitigation baseline with its TACO-steered counterpart using the same generation seed, ensuring a controlled comparison.
Model
Method
TV only
All
mIoU( ↑ )
ACC( ↑ )
mIoU( ↑ )
ACC( ↑ )
SD v1.4
Ours
0.942
0.964
0.926
0.949
w/o Traj (only 48 )
0.933
0.959
0.918
0.945
w/o Traj (only 49 )
0.936
0.960
0.922
0.947
w/o Local Refine
0.939
0.962
0.923
0.948
SD v2.1
Ours
0.957
0.972
0.923
0.977
Table 2: Ablation study of TACO components.
Figure 7: Sensitivity analysis of the observation-window size ∣Tobs∣ . The numbers annotated around each point indicate the corresponding window size.
Figure 8: Sensitivity analysis of the aggregation weight wstart .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Method
SD v1.4
SD v2.0
Base
+Ours
Base
+Ours
No Mitigation
0.97
-
0.96
-
Init Batch
1.03
1.27(+0.24)
0.97
1.20(+0.23)
Init Sample
2.04
2.40(+0.36)
1.79
2.05(+0.26)
Basin Zero
0.95
1.26(+0.31)
0.90
1.23(+0.33)
Basin Neg
0.95
1.26(+0.31)
0.90
1.20(+0.30)
Appendix
Table 3: Average generation time per image in seconds. Values in parentheses denote the additional runtime introduced by TACO calculation steering.
Model
Method Pair ( ours−base )
SSCD-CLIP
SSCD-ImageReward
NPHV( ↑ )
PDA( ↑ )
NPHV( ↑ )
PDA( ↑ )
SD v1.4
Init Batch
0.061
0.714
0.145
0.714
Init Sample
0.136
1.000
0.181
1.000
Basin Zero
0.065
0.285
0.091
0.714
Basin Neg
0.042
0.857
0.061
0.857
SD v2.0
Init Batch
0.274
1.000
0.307
1.000
Appendix
Table 4: Quantitative evaluation of Pareto fronts in memorization mitigation.
Figure 9: Additional qualitative comparison of memorization localization maps produced by different methods.
Figure 10: Additional qualitative comparison of existing memorization mitigation methods with and without TACO steering. Each pair compares the original mitigation baseline with its TACO-steered counterpart using the same generation seed.
Method
Param
SD v1.4
SD v2.0
SSCD( ↓ )
CLIP( ↑ )
ImageReward( ↑ )
SSCD( ↓ )
CLIP( ↑ )
ImageReward( ↑ )
No mitigation
-
0.5106
0.2999
0.4548
0.3196
0.3039
0.1406
Init Batch
0
0.3807
0.2876
−0.1419
0.2899
0.2982
−0.0033
1
0.3441
0.2812
−0.2050
0.2809
0.2954
−0.0528
2
0.3061
0.2743
−0.2809
0.2688
0.2919
−0.1260
3
0.2700
0.2670
−0.3632
0.2534
0.2865
−0.2109
Appendix
Table 5: Detailed statistics of Pareto fronts comparison between the original Init Batch and TACO-steered Init Batch .
Method
Param
SD v1.4
SD v2.0
SSCD( ↓ )
CLIP( ↑ )
ImageReward( ↑ )
SSCD( ↓ )
CLIP( ↑ )
ImageReward( ↑ )
No mitigation
-
0.5106
0.2999
−0.0060
0.3196
0.3039
0.1406
Init Sample
(0.006,1.2)
0.2903
0.2807
−0.1829
0.2593
0.2981
0.0628
(0.006,0.9)
0.2211
0.2697
−0.3081
0.2283
0.2887
−0.0763
(0.008,0.9)
0.2085
0.2673
−0.3253
0.2172
0.2867
−0.1111
(0.010,0.9)
0.1982
0.2653
−0.3510
0.2091
0.2859
−0.1181
Appendix
Table 6: Detailed statistics of Pareto fronts comparison between the original Init Sample and TACO-steered Init Sample .
Method
Param
SD v1.4
SD v2.0
SSCD( ↓ )
CLIP( ↑ )
ImageReward( ↑ )
SSCD( ↓ )
CLIP( ↑ )
ImageReward( ↑ )
No mitigation
-
0.5106
0.2999
0.4548
0.3196
0.3039
0.1406
Basin Zero
50
0.4871
0.2984
−0.0259
0.3132
0.3027
0.1097
100
0.4410
0.2926
−0.0801
0.3009
0.3003
0.0411
150
0.3909
0.2856
−0.1654
0.2847
0.2967
−0.0369
200
0.2975
0.2685
−0.3687
0.2431
0.2868
−0.2177
Appendix
Table 7: Detailed statistics of Pareto fronts comparison between the original Basin Zero and TACO-steered Basin Zero .
Method
Param
SD v1.4
SD v2.0
SSCD( ↓ )
CLIP( ↑ )
ImageReward( ↑ )
SSCD( ↓ )
CLIP( ↑ )
ImageReward( ↑ )
No mitigation
-
0.5106
0.2999
0.4548
0.3196
0.3039
0.1406
Basin Neg
50
0.4550
0.2955
−0.0480
0.3043
0.3022
0.0825
75
0.4110
0.2905
−0.1021
0.2921
0.2997
0.0320
85
0.3479
0.2815
−0.1999
0.2721
0.2946
−0.0581
100
0.2801
0.2687
−0.3389
0.2370
0.2866
−0.2039
Appendix
Table 8: Detailed statistics of Pareto fronts comparison between the original Basin Neg and TACO-steered Basin Neg .
Memorization in generative models studies when models retain training examples, how to detect their recall, and how to limit it without losing utility.