Text-to-image personalization aims to equip pretrained diffusion models with the ability to learn user-specific concepts from only a few reference images. Existing methods, however, typically train each concept independently or assume that all concepts are available simultaneously. In practical applications, concepts often arrive sequentially, requiring models to continually learn new subjects, objects, or styles while minimizing interference with previously acquired concepts. We address this continual concept personalization setting by extending residual-space adaptation to the stochastic adaptation process of pretrained diffusion models. Our work achieves the highest concept fidelity among competing continual personalization methods while maintaining near-zero forgetting. Beyond these empirical gains, the learned residual-space dynamics reveal where adaptation capacity is concentrated within the diffusion model. Restricting adaptation to the identified layers by our method preserves nearly the same performance while reducing the number of trainable parameters by up to 72%, demonstrating that residual-space dynamics can identify the layers that contribute most to continual concept personalization and directly guide more parameter-efficient adaptation.
Figures & tables
Figure 1: Nearest-neighbor retrieval for one reference image across embedding spaces. Red frames indicate retrieved images that belong to a different concept than the query.
Method
Q↑
F↓
Trainable-AB
0.6392 ± 0.0015
0.0368 ± 0.0014
Rank-Fixed-B
0.6442 ± 0.0029
− 0.0033 ± 0.0020
OA
0.6435 ± 0.0031
− 0.0023 ± 0.0003
DyRA-0.5
0.6454 ± 0.0022
0.0002 ± 0.0001
DyRA-1.0
0.6497 ± 0.0008
− 0.0001 ± 0.0007
Table 1: Concept fidelity ( Q ) and forgetting ( F ) scores. Values are mean ± SD over three seeds.
Figure 2: Qualitative results for concept 14 ( pet_dog2 ).
Figure 3: Forgetting trajectory across continual training. Higher values indicate more forgetting.
Figure 4: Zoomed views of the forgetting trajectory over four task intervals.
Figure 5: Parameter-efficiency frontier comparing concept fidelity and trainable parameter count for OA and DyRA variants. Error bars indicate one standard deviation over three random seeds.
Figure 6: Average allocated rank per adapted layer over 101 concepts.
Figure 7: Remaining rank capacity after 101 tasks.
Figure 8: Representative remaining-rank trajectories of UNet layers: a non-saturated layer (left) and a cross-attention K,V layer that saturates under DyRA- 0.5 (right).
Method
Final Q↑
Last-20 Q↑
Gap ↓
DyRA- 0.5
0.6454
0.6340
0.0114
DyRA- 1.0
0.6497
0.6501
-0.0004
Table 2: Final concept fidelity compared with the average fidelity on the final 20 learned concepts. The reported score are averaged over three seeds. Standard deviations are below 0.007 in all cases.
Method
Q↑
F↓
Params (M)
Full DyRA
0.6497 ± 0.0008
− 0.0001 ± 0.0007
67.56±0.06
UNet K,V
0.6496 ± 0.0013
− 0.0011 ± 0.0003
27.12±0.06
UNet cross-attention K,V
0.6479 ± 0.0004
− 0.0008 ± 0.0002
18.87±0.00
Table 3: Performance of targeted adaptation strategies guided by residual-space dynamics. Values are mean ± SD over three seeds.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Embedding model
Acc.
ARI
DINOv3 ViT-S/16
0.9085
0.8304
DINO ViT-B/16
0.9196
0.8044
OpenCLIP ViT-B/32
0.9543
0.8935
CLIP ViT-B/32
0.9385
0.8873
DreamSim ensemble
0.9558
0.9050
Appendix
Table 4: Clustering-based comparison of embedding spaces on ground-truth reference images.
Setting
Value
Training steps per task
400
Evaluation interval
Every 50 steps
Per-device batch size
2
Optimizer
AdamW
AdamW parameters
β1=0.9 , β2=0.999 , wd 0.01
Learning rate
Appendix
Table 5: Training hyperparameters shared across methods.
Setting
Value
Sampler
DPM-Solver++ multistep
Inference steps
50
CFG scale
7.5
Output resolution
512×512
Evaluation samples per concept
10
Validation prompts per class
10
Appendix
Table 6: Generation settings used for evaluation and final validation.
Block b
K
V
Q
O
din
0
0
1
2
3
768
1
4
5
6
7
768
2
8
9
10
11
768
3
12
13
14
15
768
4
16
17
18
19
768
5
20
21
22
23
768
Appendix
Table 7: Text-encoder layer-index mapping. The text encoder uses self-attention only.
m
Range
Block
UNet dim.
0
0–7
down 0
320
1
8–15
down 0
320
2
16–23
down 1
640
3
24–31
down 1
640
4
32–39
down 2
1280
5
40–47
down 2
1280
Appendix
Table 8: UNet transformer-module mapping. Within each layer range, offsets 0 – 3 correspond to self-attention Q,K,V,O , and offsets 4 – 7 correspond to cross-attention Q,K,V,O .
Figure 9: Additional qualitative results for Jewelry Ring.
Figure 10: Additional qualitative results for Plushie Panda.
Figure 11: Additional qualitative results for Wearable Jacket.