Text-to-image personalization aims to equip pretrained diffusion models with the ability to learn user-specific concepts from only a few reference images. Existing methods, however, typically train each concept independently or assume that all concepts are available simultaneously. In practical applications, concepts often arrive sequentially, requiring models to continually learn new subjects, objects, or styles while minimizing interference with previously acquired concepts. We address this continual concept personalization setting by extending residual-space adaptation to the stochastic adaptation process of pretrained diffusion models. Our work achieves the highest concept fidelity among competing continual personalization methods while maintaining near-zero forgetting. Beyond these empirical gains, the learned residual-space dynamics reveal where adaptation capacity is concentrated within the diffusion model. Restricting adaptation to the identified layers by our method preserves nearly the same performance while reducing the number of trainable parameters by up to 72%, demonstrating that residual-space dynamics can identify the layers that contribute most to continual concept personalization and directly guide more parameter-efficient adaptation.
Figures & tables
Figure 1: Nearest-neighbor retrieval for one reference image across embedding spaces. Red frames indicate retrieved images that belong to a different concept than the query.
Method
Q↑
F↓
Trainable-AB
0.6392 ± 0.0015
0.0368 ± 0.0014
Rank-Fixed-B
0.6442 ± 0.0029
− 0.0033 ± 0.0020
OA
0.6435 ± 0.0031
− 0.0023 ± 0.0003
DyRA-0.5
0.6454 ± 0.0022
0.0002 ± 0.0001
DyRA-1.0
0.6497 ± 0.0008
− 0.0001 ± 0.0007
Table 1: Concept fidelity ( Q ) and forgetting ( F ) scores. Values are mean ± SD over three seeds.
Figure 2: Qualitative results for concept 14 ( pet_dog2 ).
Figure 3: Forgetting trajectory across continual training. Higher values indicate more forgetting.
Figure 4: Zoomed views of the forgetting trajectory over four task intervals.
Figure 5: Parameter-efficiency frontier comparing concept fidelity and trainable parameter count for OA and DyRA variants. Error bars indicate one standard deviation over three random seeds.
Figure 6: Average allocated rank per adapted layer over 101 concepts.
Figure 7: Remaining rank capacity after 101 tasks.
Figure 8: Representative remaining-rank trajectories of UNet layers: a non-saturated layer (left) and a cross-attention K,V layer that saturates under DyRA- 0.5 (right).
Method
Final Q↑
Last-20 Q↑
Gap ↓
DyRA- 0.5
0.6454
0.6340
0.0114
DyRA- 1.0
0.6497
0.6501
-0.0004
Table 2: Final concept fidelity compared with the average fidelity on the final 20 learned concepts. The reported score are averaged over three seeds. Standard deviations are below 0.007 in all cases.
Method
Q↑
F↓
Params (M)
Full DyRA
0.6497 ± 0.0008
− 0.0001 ± 0.0007
67.56±0.06
UNet K,V
0.6496 ± 0.0013
− 0.0011 ± 0.0003
27.12±0.06
UNet cross-attention K,V
0.6479 ± 0.0004
− 0.0008 ± 0.0002
18.87±0.00
Table 3: Performance of targeted adaptation strategies guided by residual-space dynamics. Values are mean ± SD over three seeds.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Embedding model
Acc.
ARI
DINOv3 ViT-S/16
0.9085
0.8304
DINO ViT-B/16
0.9196
0.8044
OpenCLIP ViT-B/32
0.9543
0.8935
CLIP ViT-B/32
0.9385
0.8873
DreamSim ensemble
0.9558
0.9050
Appendix
Table 4: Clustering-based comparison of embedding spaces on ground-truth reference images.
Setting
Value
Training steps per task
400
Evaluation interval
Every 50 steps
Per-device batch size
2
Optimizer
AdamW
AdamW parameters
β1=0.9 , β2=0.999 , wd 0.01
Learning rate
Appendix
Table 5: Training hyperparameters shared across methods.
Setting
Value
Sampler
DPM-Solver++ multistep
Inference steps
50
CFG scale
7.5
Output resolution
512×512
Evaluation samples per concept
10
Validation prompts per class
10
Appendix
Table 6: Generation settings used for evaluation and final validation.
Block b
K
V
Q
O
din
0
0
1
2
3
768
1
4
5
6
7
768
2
8
9
10
11
768
3
12
13
14
15
768
4
16
17
18
19
768
5
20
21
22
23
768
Appendix
Table 7: Text-encoder layer-index mapping. The text encoder uses self-attention only.
m
Range
Block
UNet dim.
0
0–7
down 0
320
1
8–15
down 0
320
2
16–23
down 1
640
3
24–31
down 1
640
4
32–39
down 2
1280
5
40–47
down 2
1280
Appendix
Table 8: UNet transformer-module mapping. Within each layer range, offsets 0 – 3 correspond to self-attention Q,K,V,O , and offsets 4 – 7 correspond to cross-attention Q,K,V,O .
Figure 9: Additional qualitative results for Jewelry Ring.
Figure 10: Additional qualitative results for Plushie Panda.
Figure 11: Additional qualitative results for Wearable Jacket.
Continual personalization of text-to-image diffusion models requires sequentially acquiring new concepts while retaining previously learned ones. However, existing methods either suffer from catastrophic forgetting or rely on storing additional concept-specific parameters and spatial components, causing their parameter footprint to grow with the concept stream. This limits their ability to scale to long sequences of personalization tasks. We propose a rehearsal-free approach that uses a single fixed-size hypernetwork to continually personalize a frozen diffusion model. Instead of expanding the model as new concepts are acquired, the hypernetwork dynamically produces the concept-specific adaptations required for personalization while preserving previously learned concepts. Our framework further integrates spatial control into the personalization process, allowing users to specify where a personalized concept should appear without introducing additional per-concept components. This formulation enables continual personalization with a parameter footprint that remains independent of the number of learned concepts, aside from compact concept representations. Experiments demonstrate strong retention of previously learned concepts and reliable spatial grounding, matching or improving upon existing methods while scaling effectively to long streams of personalization tasks.
Wojciech Gromski, Patryk Krukowski, Jan Miksa +2
Wrocław University of Science and Technology · IDEAS Research Institute · Jagiellonian University, Krak´ow +2
Personalizing text-to-image diffusion models to render several specific subjects in a coherent image remains challenging: the model must preserve each subject's identity while keeping the scene spatially and visually coherent. Methods that fuse independently trained concept adapters in a shared weight space (via federated averaging, gradient fusion, or orthogonality constraints) suffer from identity confusion and style bleeding. In this work, we show that composing concepts as separate image layers, instead of merging their adapters in a shared weight space, avoids parameter-level interference. We introduce LILAC, a framework that composes independently trained low-rank adapters at inference time: each subject is conditioned on the frozen composite of previously placed subjects, with exactly one adapter active at a time, therefore identities never interfere at the parameter level. LILAC composes the adapters without joint training, scales linearly with the number of concepts, and is backbone-agnostic. Under the Orthogonal Adaptation protocol, LILAC applied on Qwen-Image-Edit+Qwen-Image-Layered reaches an ArcFace detection rate of 0.861. Code is available at https://github.com/marianlupascu/LILAC.
Marian Lupascu, Sebastian Ripa, Mihai Trascau +2
Adobe Research, Romania · International Computer High School of Bucharest, Romania
Personalizing text-to-image diffusion models extends pretrained models to represent novel user-specific concepts from only a few reference images. However, learning a new concept while building on the prior knowledge of the pretrained model remains a key challenge. When personalization focuses on learning the target concept, the model tends to overfit the reference examples and degrade its general capability. In contrast, emphasizing prior preservation can hinder capturing distinctive personalized attributes. In this paper, we address this challenge by viewing a personalized concept as an underrepresented concept whose semantic counterpart is well represented in the pretrained model. Rather than treating the learning of a new concept and prior preservation as separate objectives, we reformulate them as a single anchored learning problem. We therefore introduce Semantic Anchoring Personalization (SAP), which keeps concept learning grounded in the pretrained semantic structure while capturing subject-specific attributes. The proposed objective offers a simple yet effective formulation that can be applied across different model backbones without architectural modifications or auxiliary networks. Extensive experiments across various settings demonstrate that SAP achieves a better balance between subject fidelity and text-image alignment than baseline methods. Further ablation studies validate the contribution of semantic anchoring to personalization.
Seoyun Yang, Gihoon Kim, Taesup Kim
Graduate School of Data Science, Seoul National University