PureCC: Pure Learning for Text-to-Image Concept Customization

Zhichao Liao; Xiaole Xian; Qingyu Li; Wenyu Qin; Meng Wang; Weicheng Xie; Siyang Song; Pingfa Feng

cs.CVMar 8, 2026

PureCC: Pure Learning for Text-to-Image Concept Customization

Authors: Zhichao Liao, Xiaole Xian, Qingyu Li, Wenyu Qin, Meng Wang, Weicheng Xie, Siyang Song, Pingfa Feng,

Organizations: Tsinghua University · School of Computer Science & Software Engineering, Shenzhen University · Kling Team, Kuaishou Technology · Guangdong Provincial Key Laboratory of Intelligent Information Processing, Shenzhen University ·

Abstract

Existing concept customization methods have achieved remarkable outcomes in high-fidelity and multi-concept customization. However, they often neglect the influence on the original model's behavior and capabilities when learning new personalized concepts. To address this issue, we propose PureCC. PureCC introduces a novel decoupled learning objective for concept customization, which combines the implicit guidance of the target concept with the original conditional prediction. This separated form enables PureCC to substantially focus on the original model during training. Moreover, based on this objective, PureCC designs a dual-branch training pipeline that includes a frozen extractor providing purified target concept representations as implicit guidance and a trainable flow model producing the original conditional prediction, jointly achieving pure learning for personalized concepts. Furthermore, PureCC introduces a novel adaptive guidance scale $λ^\star$ to dynamically adjust the guidance strength of the target concept, balancing customization fidelity and model preservation. Extensive experiments show that PureCC achieves state-of-the-art performance in preserving the original behavior and capabilities while enabling high-fidelity concept customization. The code is available at https://github.com/lzc-sg/PureCC.

Figures & tables

Figure 2: Original Distribution Drift. Visualization and KL Divergence results demonstrated that existing methods, which adjust pre-trained models to align with the target distribution for learning personalized concepts, lead to distribution drift.

Figure 3: Overview of our PureCC. (a). We first fine-tune a flow model on the custom set as representation extractor. (b). During the pure learning stage, the representation extractor remains frozen and provides the target concept representation, which is then controlled by our adaptive scale $\lambda^{\star}$ to implicitly guide the trainable model. The trainable model is initialized from another pre-trained flow model and provides original conditional prediction using the Base Text as input. The entire pipeline is trained on the custom set using $\mathcal{L}_{PureCC}$ and $\mathcal{L}_{CC}$ . (c). demonstrates the process of using our designed $\mathcal{L}_{PureCC}$ to purely learn the target concept in the velocity flow space.

Figure 4: Motivation of Adaptive Scale $\lambda^{\star}$ . A small $\lambda$ can preserve the original model’s behavior and capabilities but leads to a decrease in the fidelity of the target concept. Conversely, when $\lambda$ is excessively large , the personalized concept dominates the learning objective, causing the final distribution to drift away from the original distribution. This results in a degradation of the model’s generative ability : the underlying prompt cannot be followed and lower CLIP-T and HPSv2.1 scores.

Figure 5: Qualitative Comparison with SOTAs including Tuning-based methods: DreamBooth [ 36 ] , DreamBooth + EWC [ 39 ] , Mix-of-Show [ 13 ] , CIFC [ 9 ] , and Tuning-free methods: DreamO [ 31 ] UNO [ 46 ] .

Method	Instance						Style
	Preservation				Concept Responsiveness		Preservation			Concept Responsiveness
	$\Delta$ CLIP-T (base) ( $\uparrow$ )	$\Delta$ HPSv2.1 ( $\uparrow$ )	$\Delta$ PickScore ( $\uparrow$ )	Seg-Cons ( $\uparrow$ )	CLIP-I (target) ( $\uparrow$ )	DINO ( $\uparrow$ )	$\Delta$ CLIP-T (base) ( $\uparrow$ )	$\Delta$ HPSv2.1 ( $\uparrow$ )	$\Delta$ PickScore ( $\uparrow$ )	CSD ( $\uparrow$ )
Dreambooth [ 36 ]	-4.81	-2.17	-3.90	18.38	0.63	0.62	-6.23	-2.08	-1.83	0.57
Dreambooth + EWC [ 39 ]	-4.17	-2.20	-3.16	26.37	0.62	0.61	-7.90	-2.04	-1.57	0.60
Mix-of-Show [ 13 ]	-2.71	-1.08	-2.63	15.72	0.72	0.61	-4.93	-1.63	-1.73	0.62
CIFC [ 9 ]	-1.93	-1.62	-2.08	13.23	0.78	0.65	-4.70	-1.21	-1.25	0.64

Table 1: Quantitative Comparison Results on DreamBenchCC. Since UNO and DreamO are Tuning-free methods that do not require fine-tuning the base model, our comparison for them focuses mainly on their concept responsiveness.

Figure 6: Qualitative comparison in Multi-Concept Customization with Mix-of-show [ 13 ] , LoRA-S [ 53 ] .

Figure 7: Qualitative comparison of style–instance customization across different methods, including CIFC [ 9 ] , B-LoRA [ 11 ] , DreamO [ 31 ] . B-LoRA is a tuning-based approach specifically designed for balancing style and content adaptation. Each case combines an instance concept with a specific style.

Figure 8: Visualization of Pure Learning Process. $\widehat{x}_{0}^{PureCC}$ denotes the images obtained by integrating the velocity field $\{\bm{v}_{t}^{PureCC}\}_{t=1}^{T}$ . Similarly, $\widehat{x}_{0}^{original}$ and $\widehat{x}_{0}^{complete}$ are based on $\{\bm{v}_{t}^{original}\}_{t=1}^{T}$ and $\{\bm{v}_{t}^{complete}\}_{t=1}^{T}$ , respectively. “iter” denotes the training iteration.

Strategy	Preservation				Concept Responsiveness
Strategy	$\Delta$ CLIP-T (base) ( $\uparrow$ )	$\Delta$ HPSv2.1 ( $\uparrow$ )	$\Delta$ PickScore ( $\uparrow$ )	Seg-Cons ( $\uparrow$ )	CLIP-I (target) ( $\uparrow$ )	DINO ( $\uparrow$ )
$\mathcal{L}_{CC}$	-4.52	-2.01	-2.95	23.74	0.65	0.66
Merged Training Stage	-1.17	-0.34	-1.08	54.37	0.50	0.41
$\mathcal{L}_{CC}$ + $\mathcal{L}_{PureCC}$	-0.31	+0.10	-0.67	69.37	0.81	0.73

Table 2: Ablation Study on the Pure Learning. “Merged Learning Stage” refers to the training setting where the first-stage Representation Extractor

\bm{v}^{\theta_{1}}_{t}

and the second-stage Pure Learning of

\bm{v}^{\theta_{2}}_{t}

are conducted jointly.

Figure 9: Visualization of the Ablation Study.

Strategy	Instance		Style
Strategy	$\Delta$ CLIP-T (base) ( $\uparrow$ )	CLIP-I (target) ( $\uparrow$ )	$\Delta$ CLIP-T (base) ( $\uparrow$ )	CSD ( $\uparrow$ )
$\lambda=1.0$	-0.18	0.43	-0.67	0.26
$\lambda=3.0$	-0.51	0.58	-0.93	0.61
$\lambda=5.0$	-2.67	0.73	-4.21	0.42
$\lambda=\lambda^{\star}$ (adaptive)	-0.31	0.81	-0.26	0.63

Table 3: Ablation Study of the

\lambda^{\star}

Figure 10: Some samples selected from the dataset proposed by DreamBooth [ 36 ] .

Figure 11: Some samples additionally collected by us.

Figure 12: Some style samples in our DreamBenchPCC.

Figure 13: More qualitative evaluation results

Figure 14: More qualitative evaluation results

	Representation Extrator (stage-1)	Pure Learning (stage-2)
Memory Cost	28GB	30GB
Training Time	0.13 (A100 Hour)	0.2 (A100 Hour)

Table 4: Computation Time and Memory Usage of Training under BFloat16 datatype.

	DreamBooth	LoRA	Mix-of-Show	CIFC	PureCC (Ours)
Training	58.0G / 0.25h	28.0G / 0.13h	32.0G / 0.23h	31.0G / 0.28h	30.0G / 0.33h
Inference	17.4G / 4.46s	17.8G / 4.72s	18.4G / 5.80s	18.0G / 5.48s	17.8G / 4.72s

Table 5: Comparisons on a single NVIDIA A100 GPU

Figure 15: Qualitative Analysis of the $\eta$ .

Strategy	Instance		Style
Strategy	$\Delta$ CLIP-T (base) ( $\uparrow$ )	CLIP-I (target) ( $\uparrow$ )	$\Delta$ CLIP-T (base) ( $\uparrow$ )	CSD ( $\uparrow$ )
$\eta=0.5$	-1.32	0.55	-2.02	0.49
$\eta=1.0$	-0.31	0.81	-0.26	0.63
$\eta=1.5$	-0.43	0.80	-0.26	0.60
$\eta=2.0$	-0.76	0.67	-0.41	0.39

Table 6: Quantitative of the

\eta

Figure 16: Results compared to which Original Conditional Prediction is provided by the Trainable model ${\bm{v}}_{t}^{\theta_{2}}(x_{t}|y_{base})$ or another Frozen pre-trained model ${\bm{v}}_{t}^{\theta_{3}}(x_{t}|y_{base})$ .

Figure 17: The investigation page in user study.

	PureCC (Ours)	DreamBooth	PureCC (Ours)	DreamBooth+EWC	PureCC (Ours)	Mix-of-Show	PureCC (Ours)	CIFC
Original Behavior Consistency	98.5%	1.5%	96.9%	3.1%	91.5%	8.5%	94.6%	5.4%
Base Text Alignment	66.2%	33.8%	62.3%	37.7%	55.4%	44.6%	58.5%	41.5%
Aesthetic Preference	71.9%	28.1%	73.5%	26.5%	64.3%	35.7%	56.4%	43.6%
Target Attribute Fidelity	67.7%	32.3%	75.4%	24.6%	52.3%	47.7%	54.6%	45.4%

Table 7: The results of User Study. (PureCC vs SOTAs)

Subject-Driven Image Generation

Subject-driven image generation uses visual references to create new images that preserve a chosen subject while following prompts.

See all 58 papers

Crafting Your Evolving Dreams: Concept-Incremental Versatile Customization
Jun 3, 2026Jiahua Dong, Wenqi Liang, Hongliu Li +7T2I Generation Low-Rank Adaptation
Shanghai Jiao Tong University, Shanghai, China
MoKus: Leveraging Cross-Modal Knowledge Transfer for Knowledge-Aware Concept Customization
Mar 13, 2026Chenyang Zhu, Hongxiang Li, Xiu Li +1Cross-Modal Representation Transfer T2I Generation
Tsinghua University · HKUST
CLASP: Continual Low-rank Adapters for Spatially Placed Concepts from One Hypernetwork
Oct 1, 2026Wojciech Gromski, Patryk Krukowski, Jan Miksa +2Subject-Driven Image Generation T2I Generation
Wrocław University of Science and Technology · IDEAS Research Institute · Jagiellonian University, Krak´ow +2

Jun 3, 2026J/K move · Enter open · S save

Crafting Your Evolving Dreams: Concept-Incremental Versatile Customization

Jiahua Dong, Wenqi Liang, Hongliu Li, Yang Cong+6

Shanghai Jiao Tong University, Shanghai, China

T2I Generation Low-Rank Adaptation Diffusion Models

Open paper AskPDF arXiv

Custom diffusion models (CDMs) have garnered significant interest owing to their remarkable capacity for generating personalized concepts. However, the majority of CDMs unrealistically presume that the user's collection of personalized concepts is static and incapable of incremental growth over time. Furthermore, they exhibit significant catastrophic forgetting and concept neglect of previously learned concepts when incrementally learning a sequence of new ones. To resolve the above challenges, we develop a novel Continually Customizable Diffusion Model (CCDM), enabling users to perform concept-incremental versatile customization. Specifically, we design an attribute-decoupled LoRA (AD-LoRA) module and a relevance-guided AD-LoRA aggregation strategy to mitigate catastrophic forgetting. They can preserve concept-specific attributes of each task and leverage beneficial inter-task correlations to enhance the continual learning of new customization tasks. Additionally, to address the challenge of concept neglect, we propose a controllable regional context synthesis strategy that performs multi-concept composition in alignment with user-provided conditions. This strategy enhances the overall consistency in multi-concept synthesis by guaranteeing semantic independence between user-defined regions and their smooth boundary transitions. Experiments show our CCDM exhibits significant improvements over baseline methods.

Abstract

Subject-Driven Image Generation

Explore similar work

Crafting Your Evolving Dreams: Concept-Incremental Versatile Customization

MoKus: Leveraging Cross-Modal Knowledge Transfer for Knowledge-Aware Concept Customization

CLASP: Continual Low-rank Adapters for Spatially Placed Concepts from One Hypernetwork