CLASP: Continual Low-rank Adapters for Spatially Placed Concepts from One Hypernetwork

Wojciech Gromski; Patryk Krukowski; Jan Miksa; Maciej Zieba; Przemysław Spurek

cs.CVOct 1, 2026

CLASP: Continual Low-rank Adapters for Spatially Placed Concepts from One Hypernetwork

Authors: Wojciech Gromski, Patryk Krukowski, Jan Miksa, Maciej Zieba, Przemysław Spurek

Organizations: Wrocław University of Science and Technology · IDEAS Research Institute · Jagiellonian University, Krak´ow · AKCES NCBR · Tooploox

Abstract

Continual personalization of text-to-image diffusion models requires sequentially acquiring new concepts while retaining previously learned ones. However, existing methods either suffer from catastrophic forgetting or rely on storing additional concept-specific parameters and spatial components, causing their parameter footprint to grow with the concept stream. This limits their ability to scale to long sequences of personalization tasks. We propose a rehearsal-free approach that uses a single fixed-size hypernetwork to continually personalize a frozen diffusion model. Instead of expanding the model as new concepts are acquired, the hypernetwork dynamically produces the concept-specific adaptations required for personalization while preserving previously learned concepts. Our framework further integrates spatial control into the personalization process, allowing users to specify where a personalized concept should appear without introducing additional per-concept components. This formulation enables continual personalization with a parameter footprint that remains independent of the number of learned concepts, aside from compact concept representations. Experiments demonstrate strong retention of previously learned concepts and reliable spatial grounding, matching or improving upon existing methods while scaling effectively to long streams of personalization tasks.

Figures & tables

Figure 1: (a) A fixed-size hypernetwork takes the task embedding of a concept and, optionally, a bounding box, and generates the parameters needed to personalize the frozen diffusion model: a low-rank update for its cross-attention layers, which captures what the concept looks like, and placement tokens, which control where it appears. The resulting model generates the personalized concept at the requested location while following the input prompt. (b) Every fifth concept from a fifty-concept sequence. Each column shows a reference image (top) and an image generated by the final model after all fifty concepts have been learned sequentially (bottom).

Figure 2: Method overview. (a) A hypernetwork takes the task embedding $\mathbf{v}_{t}$ and bounding box $\mathbf{b}$ and generates LoRA updates and placement tokens for the frozen diffusion model. (b) Content module: task embeddings are orthogonalized by Gram–Schmidt (Eq. 3 ), while separate heads generate the LoRA updates (Eq. 2 ), and the output-space regularizer preserves updates for earlier concepts (Eq. 6 ). (c) Placement module: the task embedding and bounding box generate placement tokens, which image positions attend to through a box mask and per-block gate (Eq. 10 ). Blue elements are trainable or generated, grey ones frozen, hollow ones parameter-free, and dashed ones fixed or training-only.

Panel of Figure 3 — Figure 3: Qualitative comparison on the ten concepts of CIFC. Each column shows the reference image followed by generations from our method and CIDM. Both methods use SD-1.5, the same sampler and prompts, and the same initial noise, so differences between the two rows reflect the respective methods. The first seven columns contain object concepts, while the final three contain style concepts: painting, drawing, and ink.

Figure 4: Left: IA versus TA on SD-1.5 for ten concepts. Baseline results are from the CIFC benchmark ( Dong et al., 2024 ) ; our curve varies the LoRA scale

s_{\mathrm{lora}}

. Right: Forgetting, measured as the drop in DINO similarity from each concept’s peak to its final score, evaluated at comparable TA. Lower is better.

Panel of Figure 5 — Figure 5: Qualitative comparison of concept retention during the ten-concept sequence. For each of the first three concepts, we show generations immediately after its introduction, after task $6$ , and after task $10$ . All methods use the same experimental setting as in Figure 4 (right). Our method preserves the concept identity throughout the sequence, while sequential fine-tuning progressively degrades it. CIDM retains the concepts but can introduce features from other concepts through LoRA fusion, as visible in the white base of the duck.

Figure 6: Retention of the first ten concepts over streams of up to one hundred concepts. The same ten concepts are generated using models saved after

T=10

20

30

37

50

, and

100

tasks; a decrease therefore indicates forgetting of the corresponding concept. Results show the mean and standard deviation over three training seeds. Our method achieves higher retention for seven, seven, five, and four of the ten concepts at

T=10

20

30

, and

37

, respectively. The CIDM results at

T=37

are taken directly from the original work, which reports results up to this stream length. Our method continues to

T=100

without intermediate evaluation between

T=50

and

T=100

, hence the break in the axis.

Panel of Figure 7 — Figure 7: Spatial localization for two concepts, a teddy bear (left) and a duck (right), each generated at four requested image locations. The dashed orange box marks the target bounding box provided to the model, while the solid green box shows the bounding box detected by Mask R-CNN on the generated image. The same beach scene and lighting are maintained across the generated images, while only the target concept and its requested location vary. The complete evaluation grid is shown in Figure 15 .

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Concept (attribute in caption)	$\Delta$ DINO
backpack (“red”, $6/6$ captions)	$+0.054$
duck toy (“yellow rubber”, $4/4$ )	$+0.124$
cat (“fluffy”, $1/5$ )	$-0.002$
seven concepts without an attribute (mean)	$+0.002$

Appendix

Table 1: Effect of stripping caption attributes.

\Delta

DINO is stripped minus unstripped, final checkpoint at

s_{\mathrm{lora}}=0.45

, scored with the bare prompt. The unstripped variant is a control trained with the same recipe, seed and concepts, differing in the stripping and in being configured for ten concepts rather than fifty.

			ours (ratio to ceiling)
Concept	references	ceiling	$s_{\mathrm{lora}}=0.45$	$s_{\mathrm{lora}}=0.9$
painting	$7$	$0.423$	$0.319$ ( $0.75$ )	$0.409$ ( $0.97$ )
ink painting	$5$	$0.432$	$0.347$ ( $0.80$ )	$0.398$ ( $0.92$ )
drawing	$6$	$0.555$	$0.421$ ( $0.76$ )	$0.513$ ( $0.92$ )
backpack	$6$	$0.604$	$0.546$ ( $0.90$ )	$0.594$ ( $0.98$ )
duck toy	$4$	$0.721$	$0.679$ ( $0.94$ )	$0.670$ ( $0.93$ )

Appendix

Table 2: Reference-set ceiling per concept, the mean pairwise DINO similarity among the concept’s own reference photographs, beside our DINO at both ends of the trade-off curve, model after all ten tasks. The ceiling varies by more than a factor of two across concepts and orders them almost exactly as our results do. At

s_{\mathrm{lora}}=0.45

a gap to the ceiling remains, and at

s_{\mathrm{lora}}=0.9

it closes. Styles sit lowest because a style has no single subject for DINO to match.

	after task $10$			after task $50$
$s_{\mathrm{lora}}$	TA $\uparrow$	IA $\uparrow$	DINO $\uparrow$	TA $\uparrow$	IA $\uparrow$	DINO $\uparrow$
$0.30$	$77.47_{\pm 0.96}$	$76.97_{\pm 0.42}$	$0.573_{\pm 0.008}$	$76.65_{\pm 1.00}$	$72.94_{\pm 0.66}$	$0.519_{\pm 0.011}$
$0.45$	$75.59_{\pm 0.91}$	$79.07_{\pm 0.34}$	$0.617_{\pm 0.006}$	$75.48_{\pm 1.32}$	$74.77_{\pm 0.74}$	$0.557_{\pm 0.013}$
$0.60$	$73.79_{\pm 0.83}$	$80.85_{\pm 0.11}$	$0.644_{\pm 0.006}$	$74.10_{\pm 1.54}$	$76.22_{\pm 0.85}$	$0.579_{\pm 0.014}$
$0.75$	$71.98_{\pm 0.62}$	$82.20_{\pm 0.05}$	$0.659_{\pm 0.006}$	$72.65_{\pm 1.59}$	$77.25_{\pm 0.84}$	$0.592_{\pm 0.015}$
$0.90$	$70.39_{\pm 0.49}$	$83.11_{\pm 0.24}$	$0.663_{\pm 0.006}$	$71.22_{\pm 1.53}$	$77.78_{\pm 0.71}$	$0.592_{\pm 0.015}$

Appendix

Table 3: Our model across LoRA scales, box-free protocol, mean and standard deviation over three training seeds, after ten tasks and after the same model has learned fifty. TA and IA in percent, DINO whole-image. The benchmark reports IA and TA only.

Figure 8: IA versus TA on SDXL for ten concepts. Baseline results are taken from the SDXL rows of the CIFC benchmark ( Dong et al., 2024 ) , and the dotted line marks the TA of CIDM. The thick curve shows our method with a single LoRA scale for the first training seed, and the two thin curves show two further seeds. The dashed curve keeps the scale of the output projection

\mathrm{to\_out}.0

0.50

while sweeping the other three projections, an inference setting of the same checkpoint. All results are for the final model with box-free prompts.


Variant	TA $\uparrow$	IA $\uparrow$	DINO $\uparrow$	Forgetting $\downarrow$
reference ( $\beta=7120$ , lookahead, penalty on $\Delta\mathbf{W}$ )	$75.27$	$79.46$	$0.621$	$0.0094$
$\beta=0$	$76.88$	$78.39$	$0.589$	$0.0213$
$\beta/4$	$75.35$	$79.52$	$0.620$	$0.0018$
$4\beta$	$76.42$	$78.43$	$0.612$	$0.0077$
no lookahead, $\beta$	$75.32$	$79.63$	$0.624$	$0.0071$

Appendix

Table 4: Ablations of the recipe, ten concepts, SD-1.5, one training seed per variant (the seed of the reference row), full

55

-cell matrix at

s_{\mathrm{lora}}=0.45

. TA and IA in percent, DINO of the final model, forgetting as earlier peak minus final DINO (Appendix F ). We claim no difference below

0.0091

DINO. The reference row is one seed of the reported model, whose three-seed values are

0.617

and

0.0060

. Rows marked † are means over the same three training seeds.

Figure 9: Scaling beyond the ten concepts of CIFC. (a) Stored parameters of our fixed-size network and of CIDM, which stores an adapter and its token embeddings for every concept (

0.422

M, our measurement). The two are equal at

T=62

, and the CIDM line is dashed beyond

37

concepts, where its implementation breaks down. (b) IA versus text alignment of the first ten concepts after ten and after fifty tasks, obtained by varying the LoRA scale. After fifty tasks the curve lies lower at nearly the same TA, so these concepts lose part of their identity. Rings mark the reported scale, and results are means over three training seeds with box-free prompts.

Panel of Figure 10 — Figure 10: Retention of the first concept along the fifty-concept sequence. The columns show generations from the checkpoints saved after $1$ , $10$ , $20$ , $30$ , $40$ and $50$ tasks at $s_{\mathrm{lora}}=0.45$ . Each row uses one seed with the prompt and initial noise fixed, so every change reflects the concepts learned in the meantime. The markings and the face of the concept are preserved throughout, while the framing and the props added by the scene vary.

Panel of Figure 11 — Figure 11: Retention of the ten CIFC concepts along the ten-concept sequence, for one training seed. Row $k$ shows the model saved after task $k$ and column $j$ shows concept $j$ , so a cell appears once its concept has been learned. The prompt and initial noise are fixed within each column ( $s_{\mathrm{lora}}=0.45$ ), so changes within a column reflect the later tasks. Our method preserves every concept: the markings of the dogs and cats, the crimson of the backpack and the shape of the duck toy remain throughout, although the prompt never names a color, while the framing and the background vary.

	$\Delta\mathbf{W}$		factor $\mathbf{A}$		factor $\mathbf{B}$		random
Projection	$T=10$	$T=50$	$T=10$	$T=50$	$T=10$	$T=50$
$\mathrm{to\_q}$	$0.039$	$0.035$	$0.16$	$0.17$	$0.24$	$0.19$	$0.0013$
$\mathrm{to\_k}$	$0.070$	$0.060$	$0.34$	$0.33$	$0.20$	$0.18$	$0.0011$
$\mathrm{to\_v}$	$0.088$	$0.046$	$0.36$	$0.30$	$0.24$	$0.15$	$0.0011$
$\mathrm{to\_out}$	$0.026$	$0.037$	$0.21$	$0.27$	$0.11$	$0.12$	$0.0014$
all	$\mathbf{0.056}$	$\mathbf{0.044}$	$0.27$	$0.27$	$0.20$	$0.16$	$0.0012$

Appendix

Table 5: Mean absolute cosine between the generated updates of different concepts, per adapted projection. Factors are the same quantity computed on

\mathbf{A}

and

\mathbf{B}

separately. Random is the same statistic on matrices of the same shape with i.i.d. entries, quoted at

T=10

. The same reference at

T=50

differs by at most

0.0002

Figure 12: Cosine similarity between the generated updates of every pair of concepts, summarized by a single number in Table 5 . (a) The ten CIFC concepts. The largest value is between cat and cat2 ( $0.070$ ), followed by drawing and cat2, dog and cat, and duck toy and cat ( $0.062$ to $0.064$ ), so the alignment is small everywhere and only partly semantic. (b) The same after fifty tasks, where the block of the ten CIFC concepts is slightly darker than the remaining forty and no other structure appears. The diagonal is masked. The cosine is averaged over layers before taking the absolute value, whereas Table 5 takes the absolute value per layer, so the values here are smaller ( $0.042$ against $0.056$ off the diagonal at $T=10$ ). Signs differ across layers, which indicates that the alignment is not a single shared direction.

Figure 13: Drift of the generated update along the fifty-concept sequence (Eq. 14 ), averaged over all concept–checkpoint pairs at each lag. (a) Drift with and without Gram–Schmidt orthogonalization of the task embeddings. (b) The ratio of the two, which is close to one at short lags and grows with the length of the sequence. Beyond a lag of ten, the average at lag

L

covers

50-L

pairs, and the shaded region, with fewer than ten pairs, reflects sampling noise. All values are computed from checkpoints, without generating images.

Panel of Figure 14 — Figure 14: Generations between and beyond the learned task embeddings, with the placement branch disabled. Rows 1–2: Interpolation from $\mathbf{v}_{\mathrm{dog}}$ to $\mathbf{v}_{\mathrm{cat}}$ , rendered with each of the two class words. The category follows the class word and the identity follows the embedding, so the far end is a generic member of the requested class and no hybrid appears. Row 3: Interpolation between two concepts of the same class, where the identity changes continuously. Row 4: One embedding scaled from $0$ to $2\times$ its norm, which acts like the LoRA scale and breaks down beyond $1.5\times$ . Row 5: Five random vectors orthogonal to all learned embeddings, each producing a clean, generic instance of the class. Each cell uses one seed, and a second seed gives the same result.

Configuration	Quadrant $\uparrow$	IoU $>0.5$	Contain. $\uparrow$	Fill $\rightarrow 1$	TA $\uparrow$	DINO mask $\uparrow$	Det.
No box	$25\%$	$0\%$	$0.25$	$1.07$	$0.778$	$0.599$	$72/84$
Position named in the prompt	$21\%$	$0\%$	$0.25$	$1.01$	$0.743$	$0.597$	$67/84$
Training-free layout, same adapters	$49\%$	$2\%$	$0.37$	$0.92$	$0.770$	$0.528$	$65/84$
Placement module, $\kappa=1$	$93\%$	$52\%$	$0.74$	$1.34$	$0.780$	$0.713$	$84/84$
Placement module, $\kappa=2$	$\mathbf{100\%}$	$\mathbf{96\%}$	$\mathbf{0.93}$	$1.07$	$0.795$	$\mathbf{0.724}$	$82/84$
$\kappa=2$ , $+$ bootstrapping	$\mathbf{100\%}$	$\mathbf{96\%}$	$\mathbf{0.93}$	$\mathbf{1.01}$	$\mathbf{0.828}$	$0.717$	$84/84$

Appendix

Table 6: Placement on the seven object concepts, quadrant boxes, scene prompts,

84

generations per row, one checkpoint. Quadrant : the quadrant crop that best matches the concept’s references is the given one (no detector, chance

25\%

). DINO mask : identity on the subject’s detected mask. The upper block is what the method is measured against, and the last row is the setting we report. Sampling noise on IoU

>0.5

is up to

10

points. The other columns repeat across seed batches. Designs we measured and did not adopt are in Appendix M.3 .

Panel of Figure 15 — Figure 15: Placement of the four object concepts in the four quadrant boxes, at the reported setting ( $\kappa=2$ , bootstrapping over the first $K=15$ of $50$ steps, $s_{\mathrm{lora}}=0.7$ ). Rows show the dog, teddy bear, backpack and duck toy, and columns show the requested quadrants. The dashed box is the requested one and the solid box is the one detected by a Mask R-CNN, and the two coincide so closely that the dashed line is visible only where it extends beyond the solid one. The scene prompt is the same in every panel, so the background varies only with the seed and the box.

Design	Left/right $\uparrow$
Box mapped into the adapter weights	$48.8\%$
Scalar sigmoid gate on cross-attention	$48.8\%$
Five times more paste augmentation	$47.6\%$
Learned geometry, layout-conditioned style	$51.2\%$
Segmentation-based objective alone	$50.0\%$
Box mask with a scalar gate	$47.6\%$

Appendix

Table 7: Left/right placement test for the designs we measured and did not adopt. Percentage of generations in which the subject is detected on the requested side. The pass threshold was fixed at

80\%

before the tests. The last row is the attention branch as it stood when the six were measured, not the reported model, whose placement is in Table 6 .

Panel of Figure 16 — Figure 16: Multi-concept composition on SDXL, three frames generated with one setting at LoRA scale $0.55$ . Every requested concept is present and recognizable. The scene is shared rather than pasted: the grass in the left frame passes in front of all three objects, and no seam follows a box boundary. The scenes and boxes are taken from the CIFC benchmark, while the sampler, step count and seed are ours.

Explore similar work

Jul 6, 2026cs.CV

LILAC: Layer-Wise Independent LoRAs and Cascaded Conditioning for Multi-Concept Customization of Diffusion Models

Personalizing text-to-image diffusion models to render several specific subjects in a coherent image remains challenging: the model must preserve each subject's identity while keeping the scene spatially and visually coherent. Methods that fuse independently trained concept adapters in a shared weight space (via federated averaging, gradient fusion, or orthogonality constraints) suffer from identity confusion and style bleeding. In this work, we show that composing concepts as separate image layers, instead of merging their adapters in a shared weight space, avoids parameter-level interference. We introduce LILAC, a framework that composes independently trained low-rank adapters at inference time: each subject is conditioned on the frozen composite of previously placed subjects, with exactly one adapter active at a time, therefore identities never interfere at the parameter level. LILAC composes the adapters without joint training, scales linearly with the number of concepts, and is backbone-agnostic. Under the Orthogonal Adaptation protocol, LILAC applied on Qwen-Image-Edit+Qwen-Image-Layered reaches an ArcFace detection rate of 0.861. Code is available at https://github.com/marianlupascu/LILAC.

Marian Lupascu, Sebastian Ripa, Mihai Trascau +2

Adobe Research, Romania · International Computer High School of Bucharest, Romania

Text-To-Image Diffusion Models Diffusion Models Low-Rank Adaptation Framework Personalization

Ask PDF

Nov 27, 2025cs.CV

Anchor to Expand: Semantic Anchoring for Personalized Text-to-Image Diffusion Models

Personalizing text-to-image diffusion models extends pretrained models to represent novel user-specific concepts from only a few reference images. However, learning a new concept while building on the prior knowledge of the pretrained model remains a key challenge. When personalization focuses on learning the target concept, the model tends to overfit the reference examples and degrade its general capability. In contrast, emphasizing prior preservation can hinder capturing distinctive personalized attributes. In this paper, we address this challenge by viewing a personalized concept as an underrepresented concept whose semantic counterpart is well represented in the pretrained model. Rather than treating the learning of a new concept and prior preservation as separate objectives, we reformulate them as a single anchored learning problem. We therefore introduce Semantic Anchoring Personalization (SAP), which keeps concept learning grounded in the pretrained semantic structure while capturing subject-specific attributes. The proposed objective offers a simple yet effective formulation that can be applied across different model backbones without architectural modifications or auxiliary networks. Extensive experiments across various settings demonstrate that SAP achieves a better balance between subject fidelity and text-image alignment than baseline methods. Further ablation studies validate the contribution of semantic anchoring to personalization.

Seoyun Yang, Gihoon Kim, Taesup Kim

Graduate School of Data Science, Seoul National University

Text-To-Image Diffusion Models Image-Text Alignment Diffusion Models Semantic Anchor Personalization Fidelity+1

Ask PDF

May 21, 2026cs.LG

SeqLoRA: Bilevel Orthogonal Adaptation for Continual Multi-Concept Generation

Parameter-efficient fine-tuning enables fast personalization of text-to-image diffusion models to user-provided concepts (objects, people, or styles), but composing multiple such concepts remains challenging due to representation interference. Existing modular methods, usually built on low-rank adaptation (LoRA), either rely on expensive post-hoc fusion or freeze the LoRA adaptation subspaces, which limit expressiveness and concept fidelity. To address this trade-off, we propose Sequential regularized LoRA (SeqLoRA), a constrained continual learning framework that jointly optimizes both LoRA factors via bilevel optimization while keeping each new basis orthogonal to all previously learned ones. Theoretically, we establish monotone descent and convergence to a critical point of the constrained problem, and model the residual layer activations as a matrix sub-Gaussian process to derive a high-probability bound on catastrophic forgetting in multi-layer nonlinear networks. This bound depends on the basis only through a residual interference energy, and within the feasible subspace we prove that a data-adapted basis minimizes it, whereas a random frozen basis is suboptimal in expectation. Experiments on Stable Diffusion with up to 101 concepts show that SeqLoRA improves identity preservation over fusion-based methods, attains the lowest cross-concept leakage in multi-concept compositions, requires no fusion step, and scales to concept counts at which fusion runs out of memory.

Javad Parsa, Enis Simsar, Amir Joudaki +2

ETH Zurich · Uppsala University, Sweden

Text-To-Image Diffusion Models Continual Learning Personalization Representation Quality

Ask PDF