Concept customization typically binds rare tokens to a target concept. Unfortunately, these approaches often suffer from unstable performance as the pretraining data seldom contains these rare tokens. Meanwhile, these rare tokens fail to convey the inherent knowledge of the target concept. Consequently, we introduce Knowledge-aware Concept Customization, a novel task aiming at binding diverse textual knowledge to target visual concepts. This task requires the model to identify the knowledge within the text prompt to perform high-fidelity customized generation. Meanwhile, the model should efficiently bind all the textual knowledge to the target concept. Therefore, we propose MoKus, a novel framework for knowledge-aware concept customization. Our framework relies on a key observation: cross-modal knowledge transfer, where modifying knowledge within the text modality naturally transfers to the visual modality during generation. Inspired by this observation, MoKus contains two stages: (1) In visual concept learning, we first learn the anchor representation to store the visual information of the target concept. (2) In textual knowledge updating, we update the answer for the knowledge queries to the anchor representation, enabling high-fidelity customized generation. To further comprehensively evaluate our proposed MoKus on the new task, we introduce the first benchmark for knowledge-aware concept customization: KnowCusBench. Extensive evaluations have demonstrated that MoKus outperforms state-of-the-art methods. Moreover, the cross-model knowledge transfer allows MoKus to be easily extended to other knowledge-aware applications like virtual concept creation and concept erasure. We also demonstrate the capability of our method to achieve improvements on world knowledge benchmarks.
Figures & tables
Figure 2: Examples to show the cross-modal knowledge transfer phenomenon.
Figure 3: Overview of our MoKus . (a) Visual Concept Learning : We bind the visual information of the target concept to the anchor representation. We achieve this connection by fine-tuning the LoRA parameters. (b) Textual Knowledge Updating : We first convert the knowledge into a query and input it into the LLM encoder. We then extract the hidden states and update directions from the updatable layers to calculate a parameter shift. Finally, we add this shift to the original parameters of these layers.
Figure 4: Visualization of our KnowCusBench .
Figure 5: Qualitative comparison of reconstruction. We directly use the knowledge to reconstruct the target concept.
Figure 6: Qualitative results of MoKus . Our method can bind multiple pieces of knowledge (highlighted in red ) to a single concept. Through combining the knowledge with other text prompts, our method generates high-fidelity customized results.
Figure 7: Qualitative comparison of generation. We combine the updated knowledge (highlighted in red ) with different prompts to perform customized generation.
Reconstruction
Generation
Training
Method
CLIP-I ↑
CLIP-I-Seg ↑
CLIP-I ↑
CLIP-I-Seg ↑
CLIP-T ↑
Pick Score ↑
Time (s) ↓
Naive-DB
0.874
0.758
0.789
0.717
0.291
20.80
∼ 27min
Enc-FT
0.582
0.553
0.591
0.562
0.197
18.34
∼ 10min
Ours
0.867
0.764
0.761
0.718
0.305
21.30
∼ 6min
Table 1: Quantitative comparisons. Red stands for the best result, Blue stands for the second best result.
Knowledge
Reconstruction
Generation
Training
Number
CLIP-I ↑
CLIP-I-Seg ↑
CLIP-I ↑
CLIP-I-Seg ↑
CLIP-T ↑
Pick Score ↑
Time (s) ↓
1
0.868
0.761
0.767
0.722
0.304
21.29
331.3s
2
0.868
0.762
0.762
0.718
0.305
21.30
338.3s
3
0.867
0.761
0.762
0.719
0.305
21.30
345.1s
4
0.868
0.761
0.763
0.718
0.305
21.30
352.1s
5
0.867
0.764
0.762
0.718
0.305
21.30
360.0s
Table 2: Quantitative ablation results of the number of knowledge. Our method performs robustly and efficiently as the number of knowledge increases.
Reconstruction
Generation
η
CLIP-I ↑
CLIP-I-Seg ↑
CLIP-I ↑
CLIP-I-Seg ↑
CLIP-T ↑
Pick Score ↑
1e−4
0.557
0.535
0.588
0.564
0.218
18.40
1e−5
0.866
0.763
0.759
0.717
0.304
21.29
1e−6
0.867
0.764
0.761
0.718
0.305
21.30
1e−7
0.867
0.761
0.760
0.717
0.304
21.29
1e−8
0.867
0.764
0.760
0.718
0.304
21.29
Table 3: Quantitative ablation results of scaling factor η . Our method achieves the best performance with η=1e−6 .
Figure 8: Qualitative ablation results of scaling factor η . Our method achieves the best performance with η=1e−6 .
Figure 9: Qualitative application results of MoKus . (a) MoKus can create a virtual concept within the model by describing the visual attributes of the concept. (b) MoKus can erase certain concepts by modifying the appearance description in the model. (c) MoKus can improve the model’s performance on world knowledge benchmarks.
Metric
Pre-Update
Post-Update
Consistency
0.76
1.26
Realism
0.70
1.42
Aesthetic Quality
1.40
1.68
WiScore
0.81
1.33
Table 4: Quantitative results on the subset of WISE [ 56 ] . Our method can improve the model’s performance on the world knowledge benchmark.
Figure 10: Visualization of multiple knowledge combination. Our method can generate several concepts with high fidelity based on multiple knowledge inputs (highlighted in red ).