cs.CVOct 1, 2026
SaveCLASP: Continual Low-rank Adapters for Spatially Placed Concepts from One Hypernetwork
Organizations: Wrocław University of Science and Technology · IDEAS Research Institute · Jagiellonian University, Krak´ow · AKCES NCBR · Tooploox
Abstract
Continual personalization of text-to-image diffusion models requires sequentially acquiring new concepts while retaining previously learned ones. However, existing methods either suffer from catastrophic forgetting or rely on storing additional concept-specific parameters and spatial components, causing their parameter footprint to grow with the concept stream. This limits their ability to scale to long sequences of personalization tasks. We propose a rehearsal-free approach that uses a single fixed-size hypernetwork to continually personalize a frozen diffusion model. Instead of expanding the model as new concepts are acquired, the hypernetwork dynamically produces the concept-specific adaptations required for personalization while preserving previously learned concepts. Our framework further integrates spatial control into the personalization process, allowing users to specify where a personalized concept should appear without introducing additional per-concept components. This formulation enables continual personalization with a parameter footprint that remains independent of the number of learned concepts, aside from compact concept representations. Experiments demonstrate strong retention of previously learned concepts and reliable spatial grounding, matching or improving upon existing methods while scaling effectively to long streams of personalization tasks.
Figures & tables
Figure 1: (a) A fixed-size hypernetwork takes the task embedding of a concept and, optionally, a bounding box, and generates the parameters needed to personalize the frozen diffusion model: a low-rank update for its cross-attention layers, which captures what the concept looks like, and placement tokens, which control where it appears. The resulting model generates the personalized concept at the requested location while following the input prompt. (b) Every fifth concept from a fifty-concept sequence. Each column shows a reference image (top) and an image generated by the final model after all fifty concepts have been learned sequentially (bottom).
Figure 2: Method overview. (a) A hypernetwork takes the task embedding and bounding box and generates LoRA updates and placement tokens for the frozen diffusion model. (b) Content module: task embeddings are orthogonalized by Gram–Schmidt (Eq. 3 ), while separate heads generate the LoRA updates (Eq. 2 ), and the output-space regularizer preserves updates for earlier concepts (Eq. 6 ). (c) Placement module: the task embedding and bounding box generate placement tokens, which image positions attend to through a box mask and per-block gate (Eq. 10 ). Blue elements are trainable or generated, grey ones frozen, hollow ones parameter-free, and dashed ones fixed or training-only.
Figure 3: Qualitative comparison on the ten concepts of CIFC. Each column shows the reference image followed by generations from our method and CIDM. Both methods use SD-1.5, the same sampler and prompts, and the same initial noise, so differences between the two rows reflect the respective methods. The first seven columns contain object concepts, while the final three contain style concepts: painting, drawing, and ink.
Figure 4: Left: IA versus TA on SD-1.5 for ten concepts. Baseline results are from the CIFC benchmark ( Dong et al., 2024 ) ; our curve varies the LoRA scale . Right: Forgetting, measured as the drop in DINO similarity from each concept’s peak to its final score, evaluated at comparable TA. Lower is better.
Figure 5: Qualitative comparison of concept retention during the ten-concept sequence. For each of the first three concepts, we show generations immediately after its introduction, after task , and after task . All methods use the same experimental setting as in Figure 4 (right). Our method preserves the concept identity throughout the sequence, while sequential fine-tuning progressively degrades it. CIDM retains the concepts but can introduce features from other concepts through LoRA fusion, as visible in the white base of the duck.
Figure 6: Retention of the first ten concepts over streams of up to one hundred concepts. The same ten concepts are generated using models saved after , , , , , and tasks; a decrease therefore indicates forgetting of the corresponding concept. Results show the mean and standard deviation over three training seeds. Our method achieves higher retention for seven, seven, five, and four of the ten concepts at , , , and , respectively. The CIDM results at are taken directly from the original work, which reports results up to this stream length. Our method continues to without intermediate evaluation between and , hence the break in the axis.
Figure 7: Spatial localization for two concepts, a teddy bear (left) and a duck (right), each generated at four requested image locations. The dashed orange box marks the target bounding box provided to the model, while the solid green box shows the bounding box detected by Mask R-CNN on the generated image. The same beach scene and lighting are maintained across the generated images, while only the target concept and its requested location vary. The complete evaluation grid is shown in Figure 15 .
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Concept (attribute in caption) | DINO |
|---|---|
| backpack (“red”, captions) | |
| duck toy (“yellow rubber”, ) | |
| cat (“fluffy”, ) | |
| seven concepts without an attribute (mean) |
Table 1: Effect of stripping caption attributes. DINO is stripped minus unstripped, final checkpoint at , scored with the bare prompt. The unstripped variant is a control trained with the same recipe, seed and concepts, differing in the stripping and in being configured for ten concepts rather than fifty.
| ours (ratio to ceiling) | ||||
| Concept | references | ceiling | ||
| painting | ( ) | ( ) | ||
| ink painting | ( ) | ( ) | ||
| drawing | ( ) | ( ) | ||
| backpack | ( ) | ( ) | ||
| duck toy | ( ) | ( ) | ||
Table 2: Reference-set ceiling per concept, the mean pairwise DINO similarity among the concept’s own reference photographs, beside our DINO at both ends of the trade-off curve, model after all ten tasks. The ceiling varies by more than a factor of two across concepts and orders them almost exactly as our results do. At a gap to the ceiling remains, and at it closes. Styles sit lowest because a style has no single subject for DINO to match.
| after task | after task | |||||
|---|---|---|---|---|---|---|
| TA | IA | DINO | TA | IA | DINO | |
Table 3: Our model across LoRA scales, box-free protocol, mean and standard deviation over three training seeds, after ten tasks and after the same model has learned fifty. TA and IA in percent, DINO whole-image. The benchmark reports IA and TA only.
Figure 8: IA versus TA on SDXL for ten concepts. Baseline results are taken from the SDXL rows of the CIFC benchmark ( Dong et al., 2024 ) , and the dotted line marks the TA of CIDM. The thick curve shows our method with a single LoRA scale for the first training seed, and the two thin curves show two further seeds. The dashed curve keeps the scale of the output projection at while sweeping the other three projections, an inference setting of the same checkpoint. All results are for the final model with box-free prompts.
| Variant | TA | IA | DINO | Forgetting | |
|---|---|---|---|---|---|
| reference ( , lookahead, penalty on ) | |||||
| Regularizer | |||||
| no lookahead, | |||||
Table 4: Ablations of the recipe, ten concepts, SD-1.5, one training seed per variant (the seed of the reference row), full -cell matrix at . TA and IA in percent, DINO of the final model, forgetting as earlier peak minus final DINO (Appendix F ). We claim no difference below DINO. The reference row is one seed of the reported model, whose three-seed values are and . Rows marked † are means over the same three training seeds.
Figure 9: Scaling beyond the ten concepts of CIFC. (a) Stored parameters of our fixed-size network and of CIDM, which stores an adapter and its token embeddings for every concept ( M, our measurement). The two are equal at , and the CIDM line is dashed beyond concepts, where its implementation breaks down. (b) IA versus text alignment of the first ten concepts after ten and after fifty tasks, obtained by varying the LoRA scale. After fifty tasks the curve lies lower at nearly the same TA, so these concepts lose part of their identity. Rings mark the reported scale, and results are means over three training seeds with box-free prompts.
Figure 10: Retention of the first concept along the fifty-concept sequence. The columns show generations from the checkpoints saved after , , , , and tasks at . Each row uses one seed with the prompt and initial noise fixed, so every change reflects the concepts learned in the meantime. The markings and the face of the concept are preserved throughout, while the framing and the props added by the scene vary.
Figure 11: Retention of the ten CIFC concepts along the ten-concept sequence, for one training seed. Row shows the model saved after task and column shows concept , so a cell appears once its concept has been learned. The prompt and initial noise are fixed within each column ( ), so changes within a column reflect the later tasks. Our method preserves every concept: the markings of the dogs and cats, the crimson of the backpack and the shape of the duck toy remain throughout, although the prompt never names a color, while the framing and the background vary.
| factor | factor | random | |||||
|---|---|---|---|---|---|---|---|
| Projection | |||||||
| all | |||||||
Table 5: Mean absolute cosine between the generated updates of different concepts, per adapted projection. Factors are the same quantity computed on and separately. Random is the same statistic on matrices of the same shape with i.i.d. entries, quoted at . The same reference at differs by at most .
Figure 12: Cosine similarity between the generated updates of every pair of concepts, summarized by a single number in Table 5 . (a) The ten CIFC concepts. The largest value is between cat and cat2 ( ), followed by drawing and cat2, dog and cat, and duck toy and cat ( to ), so the alignment is small everywhere and only partly semantic. (b) The same after fifty tasks, where the block of the ten CIFC concepts is slightly darker than the remaining forty and no other structure appears. The diagonal is masked. The cosine is averaged over layers before taking the absolute value, whereas Table 5 takes the absolute value per layer, so the values here are smaller ( against off the diagonal at ). Signs differ across layers, which indicates that the alignment is not a single shared direction.
Figure 13: Drift of the generated update along the fifty-concept sequence (Eq. 14 ), averaged over all concept–checkpoint pairs at each lag. (a) Drift with and without Gram–Schmidt orthogonalization of the task embeddings. (b) The ratio of the two, which is close to one at short lags and grows with the length of the sequence. Beyond a lag of ten, the average at lag covers pairs, and the shaded region, with fewer than ten pairs, reflects sampling noise. All values are computed from checkpoints, without generating images.
Figure 14: Generations between and beyond the learned task embeddings, with the placement branch disabled. Rows 1–2: Interpolation from to , rendered with each of the two class words. The category follows the class word and the identity follows the embedding, so the far end is a generic member of the requested class and no hybrid appears. Row 3: Interpolation between two concepts of the same class, where the identity changes continuously. Row 4: One embedding scaled from to its norm, which acts like the LoRA scale and breaks down beyond . Row 5: Five random vectors orthogonal to all learned embeddings, each producing a clean, generic instance of the class. Each cell uses one seed, and a second seed gives the same result.
| Configuration | Quadrant | IoU | Contain. | Fill | TA | DINO mask | Det. |
|---|---|---|---|---|---|---|---|
| No box | |||||||
| Position named in the prompt | |||||||
| Training-free layout, same adapters | |||||||
| Placement module, | |||||||
| Placement module, | |||||||
| , bootstrapping |
Table 6: Placement on the seven object concepts, quadrant boxes, scene prompts, generations per row, one checkpoint. Quadrant : the quadrant crop that best matches the concept’s references is the given one (no detector, chance ). DINO mask : identity on the subject’s detected mask. The upper block is what the method is measured against, and the last row is the setting we report. Sampling noise on IoU is up to points. The other columns repeat across seed batches. Designs we measured and did not adopt are in Appendix M.3 .
Figure 15: Placement of the four object concepts in the four quadrant boxes, at the reported setting ( , bootstrapping over the first of steps, ). Rows show the dog, teddy bear, backpack and duck toy, and columns show the requested quadrants. The dashed box is the requested one and the solid box is the one detected by a Mask R-CNN, and the two coincide so closely that the dashed line is visible only where it extends beyond the solid one. The scene prompt is the same in every panel, so the background varies only with the seed and the box.
| Design | Left/right |
|---|---|
| Box mapped into the adapter weights | |
| Scalar sigmoid gate on cross-attention | |
| Five times more paste augmentation | |
| Learned geometry, layout-conditioned style | |
| Segmentation-based objective alone | |
| Box mask with a scalar gate |
Table 7: Left/right placement test for the designs we measured and did not adopt. Percentage of generations in which the subject is detected on the requested side. The pass threshold was fixed at before the tests. The last row is the attention branch as it stood when the six were measured, not the reported model, whose placement is in Table 6 .
Figure 16: Multi-concept composition on SDXL, three frames generated with one setting at LoRA scale . Every requested concept is present and recognizable. The scene is shared rather than pasted: the grass in the left frame passes in front of all three objects, and no seam follows a box boundary. The scenes and boxes are taken from the CIFC benchmark, while the sampler, step count and seed are ours.