Concept customization typically binds rare tokens to a target concept. Unfortunately, these approaches often suffer from unstable performance as the pretraining data seldom contains these rare tokens. Meanwhile, these rare tokens fail to convey the inherent knowledge of the target concept. Consequently, we introduce Knowledge-aware Concept Customization, a novel task aiming at binding diverse textual knowledge to target visual concepts. This task requires the model to identify the knowledge within the text prompt to perform high-fidelity customized generation. Meanwhile, the model should efficiently bind all the textual knowledge to the target concept. Therefore, we propose MoKus, a novel framework for knowledge-aware concept customization. Our framework relies on a key observation: cross-modal knowledge transfer, where modifying knowledge within the text modality naturally transfers to the visual modality during generation. Inspired by this observation, MoKus contains two stages: (1) In visual concept learning, we first learn the anchor representation to store the visual information of the target concept. (2) In textual knowledge updating, we update the answer for the knowledge queries to the anchor representation, enabling high-fidelity customized generation. To further comprehensively evaluate our proposed MoKus on the new task, we introduce the first benchmark for knowledge-aware concept customization: KnowCusBench. Extensive evaluations have demonstrated that MoKus outperforms state-of-the-art methods. Moreover, the cross-model knowledge transfer allows MoKus to be easily extended to other knowledge-aware applications like virtual concept creation and concept erasure. We also demonstrate the capability of our method to achieve improvements on world knowledge benchmarks.
Figures & tables
Figure 2: Examples to show the cross-modal knowledge transfer phenomenon.
Figure 3: Overview of our MoKus . (a) Visual Concept Learning : We bind the visual information of the target concept to the anchor representation. We achieve this connection by fine-tuning the LoRA parameters. (b) Textual Knowledge Updating : We first convert the knowledge into a query and input it into the LLM encoder. We then extract the hidden states and update directions from the updatable layers to calculate a parameter shift. Finally, we add this shift to the original parameters of these layers.
Figure 4: Visualization of our KnowCusBench .
Figure 5: Qualitative comparison of reconstruction. We directly use the knowledge to reconstruct the target concept.
Figure 6: Qualitative results of MoKus . Our method can bind multiple pieces of knowledge (highlighted in red ) to a single concept. Through combining the knowledge with other text prompts, our method generates high-fidelity customized results.
Figure 7: Qualitative comparison of generation. We combine the updated knowledge (highlighted in red ) with different prompts to perform customized generation.
Reconstruction
Generation
Training
Method
CLIP-I ↑
CLIP-I-Seg ↑
CLIP-I ↑
CLIP-I-Seg ↑
CLIP-T ↑
Pick Score ↑
Time (s) ↓
Naive-DB
0.874
0.758
0.789
0.717
0.291
20.80
∼ 27min
Enc-FT
0.582
0.553
0.591
0.562
0.197
18.34
∼ 10min
Ours
0.867
0.764
0.761
0.718
0.305
21.30
∼ 6min
Table 1: Quantitative comparisons. Red stands for the best result, Blue stands for the second best result.
Knowledge
Reconstruction
Generation
Training
Number
CLIP-I ↑
CLIP-I-Seg ↑
CLIP-I ↑
CLIP-I-Seg ↑
CLIP-T ↑
Pick Score ↑
Time (s) ↓
1
0.868
0.761
0.767
0.722
0.304
21.29
331.3s
2
0.868
0.762
0.762
0.718
0.305
21.30
338.3s
3
0.867
0.761
0.762
0.719
0.305
21.30
345.1s
4
0.868
0.761
0.763
0.718
0.305
21.30
352.1s
5
0.867
0.764
0.762
0.718
0.305
21.30
360.0s
Table 2: Quantitative ablation results of the number of knowledge. Our method performs robustly and efficiently as the number of knowledge increases.
Reconstruction
Generation
η
CLIP-I ↑
CLIP-I-Seg ↑
CLIP-I ↑
CLIP-I-Seg ↑
CLIP-T ↑
Pick Score ↑
1e−4
0.557
0.535
0.588
0.564
0.218
18.40
1e−5
0.866
0.763
0.759
0.717
0.304
21.29
1e−6
0.867
0.764
0.761
0.718
0.305
21.30
1e−7
0.867
0.761
0.760
0.717
0.304
21.29
1e−8
0.867
0.764
0.760
0.718
0.304
21.29
Table 3: Quantitative ablation results of scaling factor η . Our method achieves the best performance with η=1e−6 .
Figure 8: Qualitative ablation results of scaling factor η . Our method achieves the best performance with η=1e−6 .
Figure 9: Qualitative application results of MoKus . (a) MoKus can create a virtual concept within the model by describing the visual attributes of the concept. (b) MoKus can erase certain concepts by modifying the appearance description in the model. (c) MoKus can improve the model’s performance on world knowledge benchmarks.
Metric
Pre-Update
Post-Update
Consistency
0.76
1.26
Realism
0.70
1.42
Aesthetic Quality
1.40
1.68
WiScore
0.81
1.33
Table 4: Quantitative results on the subset of WISE [ 56 ] . Our method can improve the model’s performance on the world knowledge benchmark.
Figure 10: Visualization of multiple knowledge combination. Our method can generate several concepts with high fidelity based on multiple knowledge inputs (highlighted in red ).
Existing concept customization methods have achieved remarkable outcomes in high-fidelity and multi-concept customization. However, they often neglect the influence on the original model's behavior and capabilities when learning new personalized concepts. To address this issue, we propose PureCC. PureCC introduces a novel decoupled learning objective for concept customization, which combines the implicit guidance of the target concept with the original conditional prediction. This separated form enables PureCC to substantially focus on the original model during training. Moreover, based on this objective, PureCC designs a dual-branch training pipeline that includes a frozen extractor providing purified target concept representations as implicit guidance and a trainable flow model producing the original conditional prediction, jointly achieving pure learning for personalized concepts. Furthermore, PureCC introduces a novel adaptive guidance scale λ⋆ to dynamically adjust the guidance strength of the target concept, balancing customization fidelity and model preservation. Extensive experiments show that PureCC achieves state-of-the-art performance in preserving the original behavior and capabilities while enabling high-fidelity concept customization. The code is available at https://github.com/lzc-sg/PureCC.
Zhichao Liao, Xiaole Xian, Qingyu Li +7
Tsinghua University · School of Computer Science & Software Engineering, Shenzhen University · Kling Team, Kuaishou Technology +3
Low-Rank Adaptation (LoRA) successfully enables personalization in text-to-image generation by adapting pre-trained diffusion models to specific visual concepts and styles. However, extending such models to multi-concept customization remains challenging. Naively combining multiple LoRA weights or their outputs often leads to interference among concepts, resulting in degraded visual quality and reduced fidelity to the reference images of individual concepts. This paper proposes a simple yet effective approach for multi-concept customization by optimally combining the outputs of multiple LoRA modules. We leverage the relative importance of each concept during generation, as inferred from its corresponding prompt tokens and introduce two methods, W-Switch and W-Composite, that employ a prompt-aware importance weighting strategy in which each LoRA is weighted according to the semantic influence of its trigger words in the target prompt. In addition, we extend existing quantitative evaluation metrics by proposing a new image-based similarity evaluation framework that assesses image fidelity and identity preservation through comparisons between real-world reference images and automatically segmented concept regions from generated images. We evaluate our approach on the ComposLoRA testbed and demonstrate consistent improvements over existing state-of-the-art methods in terms of visual quality, identity preservation and compositionality. Qualitative evaluations, including a Large Language Model (LLM) based assessment and a user study, further validate the effectiveness of the proposed methods and align with the newly introduced quantitative image-based metrics. Our code is available at https://github.com/GeorgeTsoumplekas/Prompt-Aware-Multi-LoRA-Composition.
Unified multimodal models (UMMs) receive image-generation supervision that exposes them to visual information not fully documented in text. Can learning to generate images expand what pretrained models know? We introduce Knowledge Transfer Probing (KTP), a controlled protocol that supplies new knowledge through image-generation training and tests its accessibility through text-only question answering. Across seven knowledge categories and three UMMs, we observe direct but uneven transfer from visual supervision to language. Motivated by these observations, we successively introduce Textual Alignment and Interleaved Image--Text Alignment, which connect generation and answering through shared knowledge and interleaved responses, respectively. Guidance on selected entities can improve knowledge access on other entities without QA supervision, with additional gains from interleaved alignment. Further analysis shows that knowledge correctly reproduced in images is often more accessible through text, although substantial gaps remain between the two forms of access. These findings support visual generation as a source of knowledge beyond textual supervision, and motivate joint investigation of training data design and model architecture to make this knowledge more broadly accessible.