TACO: Trajectory-Aware Curvature Observation for Memorization Localization and Mitigation in Diffusion Models
Authors: Ruxi Gu, Zilei Wang, Wei Wang
Organizations: Department of Automation, University of Science and Technology of China · State Key Laboratory of General Artificial Intelligence, BIGAI · School of Computer and Communication Engineering, University of Science and Technology Beijing
Text-to-image diffusion models can reproduce memorized training content within localized image regions, making precise localization essential for targeted mitigation. However, curvature observations at a single denoising state can contain transient responses, while spatially uniform mitigation may disrupt prompt-relevant content. We present Trajectory-Aware Curvature Observation (TACO), a unified framework for local memorization detection and mitigation in diffusion models. Our empirical analysis reveals that memorization-related curvature becomes pronounced during late-stage denoising and persists at consistent spatial locations across neighboring states. Motivated by these observations, TACO aggregates prompt-dependent curvature responses along the late denoising trajectory and applies latent-guided spatial refinement to improve spatial coherence. The resulting memorization map serves both as a localization signal and as a spatial control signal for mitigation. Specifically, TACO uses this map to steer existing inference-time interventions toward memorization-related regions through spatial reweighting and localized guidance correction. Extensive experiments demonstrate that TACO consistently outperforms competing localization methods and improves the Pareto trade-off between memorization suppression and generation quality. These results establish trajectory-aware spatial analysis as an effective link between memorization localization and targeted mitigation.
Figures & tables
Figure 1: Global and local memorization in Stable Diffusion v1.4 [ 24 ] across 6 different seeds.
Figure 2: Empirical studies motivating trajectory-aware curvature observation. (a) Evolution of peak curvature intensity along the sampling trajectory. (b) Spatial consistency of curvature responses between neighboring late-stage states. (c) Signal-to-background ratio and background variance of individual and fused late-stage curvature responses.
Model
Method
TV only
All
mIoU( ↑ )
ACC( ↑ )
mIoU( ↑ )
ACC( ↑ )
SD v1.4
All-zeros
0.000
0.437
0.318
0.614
All-ones
0.563
0.563
0.386
0.386
BE
0.693
0.766
0.467
0.780
cov
0.907
0.941
0.907
0.937
cov_bad
0.922
0.950
0.847
0.940
Table 1: Quantitative evaluation of memorization localization. Bold and underlined values denote the best and second-best results, respectively.
Figure 3: Qualitative comparison of memorization localization and map-guided mitigation for prompts: “Foyer painted in HABANERO” (top) and “Plymouth Curtain Panel featuring Madelyn - White Botanical Floral Large Scale by heatherdutton” (bottom). For each method, the left panel shows its original localization map, with blue and red indicating weaker and stronger memorization responses, respectively. The right panel shows the mitigated image obtained by applying our proposed steering method with the corresponding localization map.
Figure 4: SSCD-CLIP and SSCD-ImageReward Pareto fronts in SD v1.4.
Figure 5: SSCD-CLIP and SSCD-ImageReward Pareto fronts in SD v2.0.
Figure 6: Qualitative comparison of existing memorization mitigation methods with and without TACO steering. Each pair compares the original mitigation baseline with its TACO-steered counterpart using the same generation seed, ensuring a controlled comparison.
Model
Method
TV only
All
mIoU( ↑ )
ACC( ↑ )
mIoU( ↑ )
ACC( ↑ )
SD v1.4
Ours
0.942
0.964
0.926
0.949
w/o Traj (only 48 )
0.933
0.959
0.918
0.945
w/o Traj (only 49 )
0.936
0.960
0.922
0.947
w/o Local Refine
0.939
0.962
0.923
0.948
SD v2.1
Ours
0.957
0.972
0.923
0.977
Table 2: Ablation study of TACO components.
Figure 7: Sensitivity analysis of the observation-window size ∣Tobs∣ . The numbers annotated around each point indicate the corresponding window size.
Figure 8: Sensitivity analysis of the aggregation weight wstart .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Method
SD v1.4
SD v2.0
Base
+Ours
Base
+Ours
No Mitigation
0.97
-
0.96
-
Init Batch
1.03
1.27(+0.24)
0.97
1.20(+0.23)
Init Sample
2.04
2.40(+0.36)
1.79
2.05(+0.26)
Basin Zero
0.95
1.26(+0.31)
0.90
1.23(+0.33)
Basin Neg
0.95
1.26(+0.31)
0.90
1.20(+0.30)
Appendix
Table 3: Average generation time per image in seconds. Values in parentheses denote the additional runtime introduced by TACO calculation steering.
Model
Method Pair ( ours−base )
SSCD-CLIP
SSCD-ImageReward
NPHV( ↑ )
PDA( ↑ )
NPHV( ↑ )
PDA( ↑ )
SD v1.4
Init Batch
0.061
0.714
0.145
0.714
Init Sample
0.136
1.000
0.181
1.000
Basin Zero
0.065
0.285
0.091
0.714
Basin Neg
0.042
0.857
0.061
0.857
SD v2.0
Init Batch
0.274
1.000
0.307
1.000
Appendix
Table 4: Quantitative evaluation of Pareto fronts in memorization mitigation.
Figure 9: Additional qualitative comparison of memorization localization maps produced by different methods.
Figure 10: Additional qualitative comparison of existing memorization mitigation methods with and without TACO steering. Each pair compares the original mitigation baseline with its TACO-steered counterpart using the same generation seed.
Method
Param
SD v1.4
SD v2.0
SSCD( ↓ )
CLIP( ↑ )
ImageReward( ↑ )
SSCD( ↓ )
CLIP( ↑ )
ImageReward( ↑ )
No mitigation
-
0.5106
0.2999
0.4548
0.3196
0.3039
0.1406
Init Batch
0
0.3807
0.2876
−0.1419
0.2899
0.2982
−0.0033
1
0.3441
0.2812
−0.2050
0.2809
0.2954
−0.0528
2
0.3061
0.2743
−0.2809
0.2688
0.2919
−0.1260
3
0.2700
0.2670
−0.3632
0.2534
0.2865
−0.2109
Appendix
Table 5: Detailed statistics of Pareto fronts comparison between the original Init Batch and TACO-steered Init Batch .
Method
Param
SD v1.4
SD v2.0
SSCD( ↓ )
CLIP( ↑ )
ImageReward( ↑ )
SSCD( ↓ )
CLIP( ↑ )
ImageReward( ↑ )
No mitigation
-
0.5106
0.2999
−0.0060
0.3196
0.3039
0.1406
Init Sample
(0.006,1.2)
0.2903
0.2807
−0.1829
0.2593
0.2981
0.0628
(0.006,0.9)
0.2211
0.2697
−0.3081
0.2283
0.2887
−0.0763
(0.008,0.9)
0.2085
0.2673
−0.3253
0.2172
0.2867
−0.1111
(0.010,0.9)
0.1982
0.2653
−0.3510
0.2091
0.2859
−0.1181
Appendix
Table 6: Detailed statistics of Pareto fronts comparison between the original Init Sample and TACO-steered Init Sample .
Method
Param
SD v1.4
SD v2.0
SSCD( ↓ )
CLIP( ↑ )
ImageReward( ↑ )
SSCD( ↓ )
CLIP( ↑ )
ImageReward( ↑ )
No mitigation
-
0.5106
0.2999
0.4548
0.3196
0.3039
0.1406
Basin Zero
50
0.4871
0.2984
−0.0259
0.3132
0.3027
0.1097
100
0.4410
0.2926
−0.0801
0.3009
0.3003
0.0411
150
0.3909
0.2856
−0.1654
0.2847
0.2967
−0.0369
200
0.2975
0.2685
−0.3687
0.2431
0.2868
−0.2177
Appendix
Table 7: Detailed statistics of Pareto fronts comparison between the original Basin Zero and TACO-steered Basin Zero .
Method
Param
SD v1.4
SD v2.0
SSCD( ↓ )
CLIP( ↑ )
ImageReward( ↑ )
SSCD( ↓ )
CLIP( ↑ )
ImageReward( ↑ )
No mitigation
-
0.5106
0.2999
0.4548
0.3196
0.3039
0.1406
Basin Neg
50
0.4550
0.2955
−0.0480
0.3043
0.3022
0.0825
75
0.4110
0.2905
−0.1021
0.2921
0.2997
0.0320
85
0.3479
0.2815
−0.1999
0.2721
0.2946
−0.0581
100
0.2801
0.2687
−0.3389
0.2370
0.2866
−0.2039
Appendix
Table 8: Detailed statistics of Pareto fronts comparison between the original Basin Neg and TACO-steered Basin Neg .
Memorization in generative models studies when models retain training examples, how to detect their recall, and how to limit it without losing utility.
Diffusion models can unintentionally memorize training samples, raising concerns about privacy and copyright. While recent methods can detect memorization, they often rely on global or model-specific signals and provide limited insight into where memorization appears within a generated image. We provide a geometric characterization of local memorization as a coordinate-wise variance collapse. However, such collapse can also arise from intrinsic data constraints rather than overfitting. To isolate overfitting-driven memorization, we propose curvature-difference methods that subtract the curvature of an underfitted baseline, either the unconditional model or a less-trained version of itself. We further derive a score-difference proxy that provides a geometric explanation for the widely used score-difference-based detection metric. Experiments on Stable Diffusion, evaluated against ground-truth memorization masks, show that our method outperforms the prior attention-based localization method. Code is available at https://github.com/Gwangho99/mem-curv-diff.
Gwangho Kim, Sungyoon Lee
Department of Computer Science, Hanyang University, Seoul, South Korea.
Text-to-image diffusion models are susceptible to memorization, revealing a fundamental failure to generalize beyond the training set. Current mitigation approaches often sacrifice image quality or prompt alignment to reduce memorization. To address this, we propose Reachability-Aware Diffusion Steering (RADS), an offline-trained, sampling-time control framework that mitigates memorization while maintaining generation fidelity. RADS models the diffusion denoising process as a dynamical system and learns a policy-conditioned reachability value function that estimates future memorization risk along the denoising trajectory. This motivates a constrained reinforcement learning (RL) formulation, where a policy learns to steer the trajectory away from memorization via perturbations in the caption embedding space. Empirical evaluations show that RADS achieves a stronger empirical Pareto frontier between generation diversity (SSCD), quality (FID), and alignment (CLIP) compared to state-of-the-art inference-time baselines. Crucially, RADS applies the learned controller during sampling time, separating memorization mitigation from the pretrained model. Our website is available at: https://s-karnik.github.io/rads-memorization-project-page/.
Sathwik Karnik, Juyeop Kim, Sanmi Koyejo +2
Stanford University, Stanford, CA, USA · Yonsei University, Seoul, Korea.
Diffusion models can sometimes reproduce examples from their training data, a phenomenon known as memorization. Existing memorization detection methods for unconditional diffusion models typically operate at a pre-selected timestep and therefore cannot adapt the detection time to the evolving denoising trajectory. Sequential monitoring enables adaptive detection but repeated testing across timesteps complicates trajectory-level false-alarm control. We study two ways of combining split conformal calibration with sequential monitoring for prompt-free memorization detection: per-step Bonferroni calibration and direct trajectory-level calibration of the maximum memorization score. The former controls repeated testing by splitting the false-alarm budget across timesteps, whereas the latter calibrates the event of ever crossing a threshold directly through the trajectory-level maximum. We evaluate this formulation on an unconditional CIFAR-10 EDM using two complementary per-step statistics based on geometry and representation spikiness. Experiments show that sequential monitoring improves the power-timing tradeoff over fixed-time detection. Compared with Bonferroni calibration, trajectory-level calibration achieves comparable detection power while using the false-alarm budget less conservatively and enabling earlier detection at higher noise levels.
Yixiao Jing, Chia-Yu Hsu, Milad Hoseinpour +1
Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI, USA