cs.CVFeb 23, 2026
SaveConceptPrism: Concept Disentanglement in Personalized Diffusion Models via Residual Token Optimization
Organizations: KAIST, South Korea · Hanbat National University, South Korea
Abstract
Personalized text-to-image (T2I) generation has emerged as a key application for creating user-specific concepts from a few reference images. The core challenge is concept disentanglement: separating the target concept from irrelevant residual information. Lacking such disentanglement, capturing high-fidelity features often incorporates undesired attributes that conflict with user prompts, compromising the trade-off between concept fidelity and text alignment. While existing methods rely on manual guidance, they often fail to represent intricate visual details and lack scalability. We introduce ConceptPrism, a framework that extracts shared features exclusively through cross-image comparison without external information. We jointly optimize a target token and image-wise residual tokens via reconstruction and exclusion losses. By suppressing shared information in residual tokens, the exclusion loss creates an information vacuum that forces the target token to capture the common concept. Extensive evaluations demonstrate that ConceptPrism achieves accurate concept disentanglement and significantly improves overall performance across diverse and complex visual concepts. The code is available at https://github.com/Minseo-Kimm/ConceptPrism.
Figures & tables
Figure 1: Motivation of ConceptPrism. The reconstruction loss ( ) promotes information acquisition from the given image, while the exclusion loss ( ) compels discarding the commonalities from the set. By jointly optimizing the target and residual tokens with dual losses, we disentangle the personalized visual concept from irrelevant details without explicit guidance.
Figure 2: Training Pipeline of ConceptPrism. Our method comprises two stages: (a) In the token optimization, the target and image-wise residual tokens are jointly optimized via dual losses. The reconstruction loss ( ) guides the faithful reconstruction of the given image by conditioning on both tokens simultaneously. The exclusion loss ( ) forces the residual token to be uninformative of the shared target concept by matching the unconditional generation probability distribution. (b) In the subsequent fine-tuning stage, the learned tokens initialize the model to focus only on the necessary concept, effectively resolving the trade-off caused by concept entanglement.
| Method | CLIP-T | DINO | CLIP-I | Training Time(s) |
| Pretrained SD [ 24 ] | 0.377 0.024 | 0.167 0.029 | 0.320 0.030 | - |
| DreamBooth-LoRA [ 25 ] | 0.313 0.030 | 0.182 0.030 | 0.331 0.030 | 434.32 75.50 |
| Custom Diffusion [ 15 ] | 0.357 0.023 | 0.178 0.032 | 0.341 0.031 | 459.60 59.64 |
| DisenBooth [ 1 ] | 0.355 0.027 | 0.170 0.035 | 0.324 0.033 | 928.63 51.43 |
| DisEnvisioner [ 7 ] | 0.309 0.023 | 0.171 0.022 | 0.320 0.024 | - |
| ELITE [ 34 ] | 0.293 0.030 | 0.172 0.031 | 0.341 0.032 | - |
Table 1: Quantitative Evaluations on the DreamBench. CLIP-T measures text alignment; DINO / CLIP-I assess concept fidelity. Training time to the best quality step was also measured (N/A for tuning-free approaches ELITE and DisEnvisioner). All experiments were repeated over 4 different seeds. Ours achieves the highest performance across all three metrics with negligible computational overhead.
Figure 3: Performance trade-off by training steps. Our performance was measured at 40-step intervals, others at 200-step intervals. Asterisk( ) denotes the best visual quality step for each method. Ours demonstrates a superior trade-off over all baselines, highlighting its effective concept disentanglement.
Figure 4: Qualitative Results on various subjects. Our method showed balanced performance, maintaining the visual details of the personalized subject while adhering to the text prompt. Other baselines tend to either copy input images (DreamBooth) or ignore concept representation to follow the prompt (Custom Diffusion, DisenBooth).
Figure 5: Qualitative Results on abstract styles. Our method learns the abstract style beyond a single word class noun. DreamBooth generates similar scenes due to overfitting to the reference images. Other baselines fail to capture the given visual details, adhering only to the linguistic cues (e.g., “sunset,” “swirl”).
| CLIP-T | DINO | |
| Target Token Optimization | 0.345 | 0.197 |
| + Residual Tokens (Sec 4.1 ) | 0.358 | 0.183 |
| + Exclusion Loss (Sec 4.2 ) | 0.354 | 0.207 |
| + Better Initialization (Sec 4.3 ) | 0.357 | 0.210 |
Table 2: Ablation Studies on Token Optimization. We measured performance by ablating each component of the token optimization stage. The introduction of residual tokens and the exclusion loss significantly improved CLIP-T and DINO, respectively, verifying their role in concept disentanglement.
| 0 | 1 | 2 | 4 | 8 | |
| CLIP-T | 0.345 | 0.354 | 0.357 | 0.355 | 0.357 |
| DINO | 0.197 | 0.209 | 0.204 | 0.207 | 0.210 |
Table 3: Performance by Residual Token Length. The presence of residual tokens ( ) creates a significant performance gap in both criteria, demonstrating their key role in resolving concept entanglement. The stable performance from to indicates that our framework is robust to token length.
Figure 6: Visualizations of Tokens after Token Optimization. (a) Optimizing the target token alone. (b) Jointly optimizing target and residual tokens using the single objective . (c) Optimizing with dual objectives by introducing . (d) Initializing residual tokens with image descriptions.
| Description Length | CLIP-T | DINO |
| Long (frozen) | 0.313 | 0.197 |
| Short (1-4 words) | 0.353 | 0.198 |
| Medium (4-8 words) | 0.356 | 0.200 |
| Long (Ours, 8-32 words) | 0.357 | 0.210 |
Table 4: Influence of Residual Token Initialization. Performance remains robust across varying description lengths, demonstrating that our method is insensitive to the initial point of the residual tokens. The significant performance drop in the frozen setting confirms that our improvements stem from joint token optimization rather than the quality of the initial descriptions.
Figure 7: Robustness on the reference set diversity. (a) When all reference images share unintended visual features, such as an orange background, these elements can become entangled with the target concept. (b) Adding a single counterexample with a different background color effectively prevents such unintended features from being captured, ensuring accurate concept disentanglement.
Figure 8: Visualization of Learned Residual Tokens. After the token optimization stage, an image-wise residual token can be solely conditioned to generate scene details without the target concept. We visualized the residual tokens learned from each reference image using three different seeds. The ability to generate diverse images of the same semantics (e.g., varied office layouts) rather than mere copies demonstrates that each token has learned an abstract residual concept.
| 1 | 2 | 4 | 8 | |
| CLIP-T | 0.357 | 0.352 | 0.339 | 0.325 |
| DINO | 0.210 | 0.207 | 0.212 | 0.220 |
Table 5: Impact of Target Token Length. denotes the length of the target token. We adopted for our main experiments.
| 0 | 0.01 | 0.05 | 0.1 | 0.5 | |
| CLIP-T | 0.358 | 0.356 | 0.354 | 0.346 | 0.347 |
| DINO | 0.183 | 0.198 | 0.207 | 0.209 | 0.206 |
Table 6: Impact of Exclusion Loss Weight. We adopted the exclusion loss weight for our main experiments.
Figure 9: VLM Evaluations. We prompted GPT-4o to assess generated images on three criteria: Concept Fidelity, Text Alignment, and Overall Preference. The charts display the satisfaction rates for each method. Our method consistently outperforms baselines across all metrics, validating the effectiveness of our disentanglement framework.
Figure 10: Comparison with recent methods using a DiT backbone. Being architecture-agnostic, ConceptPrism is effectively applicable to the latest DiT-based generative models. Our disentanglement strategy enables a deep understanding of the target concept, excelling in handling prompts that require attribute variations of the target.
| Method | CLIP-T | DINO |
| SynCD | 0.300 | 0.269 |
| OminiCtrl | 0.346 | 0.174 |
| Ours | 0.362 | 0.213 |
Table 7: Quantitative Evaluations on DiT Backbone. ConceptPrism achieves balanced, high performance compared to personalization techniques specialized for DiT architectures, which demonstrates the practical utility of our method.
| Method | Inference Time (s) | Training Time (s) |
| Ours | 18.186 | 507.431 |
| SynCD | 38.366 | - |
| OminiCtrl | 21.093 | - |
Table 8: Time Comparison with Encoder-based Methods. We measure the training and inference time for personalized T2I generation using a 4-bit quantized FLUX.1 model. Our method incurs overhead compared to encoder-based methods due to the requirement of concept-wise training. However, it enables lighter inference by removing the need for reference information after training is complete.
Figure 11: Multi Concept Composition. Each personalized concept is learned independently following our protocol and merged via LoRA merge to generate multiple concepts simultaneously. Results are based on the FLUX.1 backbone model.
Figure 12: Additional results on abstract concepts. Our method effectively disentangles and learns abstract concepts from various datasets, including styles and actions.
Figure 13: Flexible modifications on learned concepts. Since our method learns without class noun guidance, the learned token can be flexibly applied to different subject classes. To generate these images, we used prompts specifying a new subject class, such as “A photo of [target] box” or “A photo of [target] wolf”.
Figure 14: Application on FLUX.1. Since our framework operates on the text embedding space, it is agnostic to the backbone architecture of the diffusion model. We demonstrate this broad applicability by successfully generating personalized images using the FLUX.1-dev model.
More in this topic
T2I Generation
Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.
See all 299 papers