Personalized text-to-image (T2I) generation has emerged as a key application for creating user-specific concepts from a few reference images. The core challenge is concept disentanglement: separating the target concept from irrelevant residual information. Lacking such disentanglement, capturing high-fidelity features often incorporates undesired attributes that conflict with user prompts, compromising the trade-off between concept fidelity and text alignment. While existing methods rely on manual guidance, they often fail to represent intricate visual details and lack scalability. We introduce ConceptPrism, a framework that extracts shared features exclusively through cross-image comparison without external information. We jointly optimize a target token and image-wise residual tokens via reconstruction and exclusion losses. By suppressing shared information in residual tokens, the exclusion loss creates an information vacuum that forces the target token to capture the common concept. Extensive evaluations demonstrate that ConceptPrism achieves accurate concept disentanglement and significantly improves overall performance across diverse and complex visual concepts. The code is available at https://github.com/Minseo-Kimm/ConceptPrism.
Figures & tables
Figure 1: Motivation of ConceptPrism. The reconstruction loss ( Lrec ) promotes information acquisition from the given image, while the exclusion loss ( Lexcl ) compels discarding the commonalities from the set. By jointly optimizing the target and residual tokens with dual losses, we disentangle the personalized visual concept from irrelevant details without explicit guidance.
Figure 2: Training Pipeline of ConceptPrism. Our method comprises two stages: (a) In the token optimization, the target and image-wise residual tokens are jointly optimized via dual losses. The reconstruction loss ( Lrec ) guides the faithful reconstruction of the given image by conditioning on both tokens simultaneously. The exclusion loss ( Lexcl ) forces the residual token to be uninformative of the shared target concept Ctarget by matching the unconditional generation probability distribution. (b) In the subsequent fine-tuning stage, the learned tokens initialize the model to focus only on the necessary concept, effectively resolving the trade-off caused by concept entanglement.
Method
CLIP-T ↑
DINO ↑
CLIP-I ↑
Training Time(s) ↓
Pretrained SD [ 24 ]
0.377 ± 0.024
0.167 ± 0.029
0.320 ± 0.030
-
DreamBooth-LoRA [ 25 ]
0.313 ± 0.030
0.182 ± 0.030
0.331 ± 0.030
434.32 ± 75.50
Custom Diffusion [ 15 ]
0.357 ± 0.023
0.178 ± 0.032
0.341 ± 0.031
459.60 ± 59.64
DisenBooth [ 1 ]
0.355 ± 0.027
0.170 ± 0.035
0.324 ± 0.033
928.63 ± 51.43
DisEnvisioner [ 7 ]
0.309 ± 0.023
0.171 ± 0.022
0.320 ± 0.024
-
ELITE [ 34 ]
0.293 ± 0.030
0.172 ± 0.031
0.341 ± 0.032
-
Table 1: Quantitative Evaluations on the DreamBench. CLIP-T measures text alignment; DINO / CLIP-I assess concept fidelity. Training time to the best quality step was also measured (N/A for tuning-free approaches ELITE and DisEnvisioner). All experiments were repeated over 4 different seeds. Ours achieves the highest performance across all three metrics with negligible computational overhead.
Figure 3: Performance trade-off by training steps. Our performance was measured at 40-step intervals, others at 200-step intervals. Asterisk( ⋆ ) denotes the best visual quality step for each method. Ours demonstrates a superior trade-off over all baselines, highlighting its effective concept disentanglement.
Figure 4: Qualitative Results on various subjects. Our method showed balanced performance, maintaining the visual details of the personalized subject while adhering to the text prompt. Other baselines tend to either copy input images (DreamBooth) or ignore concept representation to follow the prompt (Custom Diffusion, DisenBooth).
Figure 5: Qualitative Results on abstract styles. Our method learns the abstract style beyond a single word class noun. DreamBooth generates similar scenes due to overfitting to the reference images. Other baselines fail to capture the given visual details, adhering only to the linguistic cues (e.g., “sunset,” “swirl”).
CLIP-T ↑
DINO ↑
Target Token Optimization
0.345
0.197
+ Residual Tokens (Sec 4.1 )
0.358
0.183
+ Exclusion Loss (Sec 4.2 )
0.354
0.207
+ Better Initialization (Sec 4.3 )
0.357
0.210
Table 2: Ablation Studies on Token Optimization. We measured performance by ablating each component of the token optimization stage. The introduction of residual tokens and the exclusion loss significantly improved CLIP-T and DINO, respectively, verifying their role in concept disentanglement.
n
0
1
2
4
8
CLIP-T
0.345
0.354
0.357
0.355
0.357
DINO
0.197
0.209
0.204
0.207
0.210
Table 3: Performance by Residual Token Length. The presence of residual tokens ( n≥1 ) creates a significant performance gap in both criteria, demonstrating their key role in resolving concept entanglement. The stable performance from n=1 to 8 indicates that our framework is robust to token length.
Figure 6: Visualizations of Tokens after Token Optimization. (a) Optimizing the target token alone. (b) Jointly optimizing target and residual tokens using the single objective Lrec . (c) Optimizing with dual objectives by introducing Lexcl . (d) Initializing residual tokens with image descriptions.
Description Length
CLIP-T ↑
DINO ↑
Long (frozen)
0.313
0.197
Short (1-4 words)
0.353
0.198
Medium (4-8 words)
0.356
0.200
Long (Ours, 8-32 words)
0.357
0.210
Table 4: Influence of Residual Token Initialization. Performance remains robust across varying description lengths, demonstrating that our method is insensitive to the initial point of the residual tokens. The significant performance drop in the frozen setting confirms that our improvements stem from joint token optimization rather than the quality of the initial descriptions.
Figure 7: Robustness on the reference set diversity. (a) When all reference images share unintended visual features, such as an orange background, these elements can become entangled with the target concept. (b) Adding a single counterexample with a different background color effectively prevents such unintended features from being captured, ensuring accurate concept disentanglement.
Figure 8: Visualization of Learned Residual Tokens. After the token optimization stage, an image-wise residual token can be solely conditioned to generate scene details without the target concept. We visualized the residual tokens learned from each reference image using three different seeds. The ability to generate diverse images of the same semantics (e.g., varied office layouts) rather than mere copies demonstrates that each token has learned an abstract residual concept.
n
1
2
4
8
CLIP-T
0.357
0.352
0.339
0.325
DINO
0.210
0.207
0.212
0.220
Table 5: Impact of Target Token Length. n denotes the length of the target token. We adopted n=1 for our main experiments.
β
0
0.01
0.05
0.1
0.5
CLIP-T
0.358
0.356
0.354
0.346
0.347
DINO
0.183
0.198
0.207
0.209
0.206
Table 6: Impact of Exclusion Loss Weight. We adopted the exclusion loss weight β=0.05 for our main experiments.
Figure 9: VLM Evaluations. We prompted GPT-4o to assess generated images on three criteria: Concept Fidelity, Text Alignment, and Overall Preference. The charts display the satisfaction rates for each method. Our method consistently outperforms baselines across all metrics, validating the effectiveness of our disentanglement framework.
Figure 10: Comparison with recent methods using a DiT backbone. Being architecture-agnostic, ConceptPrism is effectively applicable to the latest DiT-based generative models. Our disentanglement strategy enables a deep understanding of the target concept, excelling in handling prompts that require attribute variations of the target.
Method
CLIP-T ↑
DINO ↑
SynCD
0.300
0.269
OminiCtrl
0.346
0.174
Ours
0.362
0.213
Table 7: Quantitative Evaluations on DiT Backbone. ConceptPrism achieves balanced, high performance compared to personalization techniques specialized for DiT architectures, which demonstrates the practical utility of our method.
Method
Inference Time (s)
Training Time (s)
Ours
18.186
507.431
SynCD
38.366
-
OminiCtrl
21.093
-
Table 8: Time Comparison with Encoder-based Methods. We measure the training and inference time for personalized T2I generation using a 4-bit quantized FLUX.1 model. Our method incurs overhead compared to encoder-based methods due to the requirement of concept-wise training. However, it enables lighter inference by removing the need for reference information after training is complete.
Figure 11: Multi Concept Composition. Each personalized concept is learned independently following our protocol and merged via LoRA merge to generate multiple concepts simultaneously. Results are based on the FLUX.1 backbone model.
Figure 12: Additional results on abstract concepts. Our method effectively disentangles and learns abstract concepts from various datasets, including styles and actions.
Figure 13: Flexible modifications on learned concepts. Since our method learns without class noun guidance, the learned token can be flexibly applied to different subject classes. To generate these images, we used prompts specifying a new subject class, such as “A photo of [target] box” or “A photo of [target] wolf”.
Figure 14: Application on FLUX.1. Since our framework operates on the text embedding space, it is agnostic to the backbone architecture of the diffusion model. We demonstrate this broad applicability by successfully generating personalized images using the FLUX.1-dev model.
Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.
Large-scale text-to-image (T2I) diffusion models deliver remarkable visual fidelity but pose safety risks due to their capacity to reproduce undesirable content, such as copyrighted ones. Concept erasure has emerged as a mitigation strategy, yet existing approaches struggle to balance scalability, precision, and robustness, which restricts their applicability to erasing only a few hundred concepts. To address these limitations, we present Erasing Thousands of Concepts (ETC), a scalable framework capable of erasing thousands of concepts while preserving generation quality. Our method first models low-rank concept distributions via a Student's t-distribution Mixture Model (tMM). It enables pin-point erasure of target concepts via affine optimal transport while preserving others by anchoring the boundaries of target concept distributions without pre-defined anchor concepts. We then train a Mixture-of-Experts (MoE)-based module, termed MoEraser, which removes target embeddings while preserving the anchor embeddings. By injecting noise into the text embedding projector and fine-tuning MoEraser for recovery, our framework achieves robustness to white-box attack such as module removal. Extensive experiments on over 2,000 concepts across heterogeneous domains and diffusion models demerate state-of-the-art scalability and precision in large-scale concept erasure.
Hoigi Seo, Byung Hyun Lee, Jaehyun Cho +2
Dept. of Electrical and Computer Engineering, Seoul National University, Republic of Korea · IPAI, Seoul National University, Republic of Korea · INMC, Seoul National University, Republic of Korea
Text-to-image (T2I) diffusion models have advanced notably, but still fail to satisfy prompt conditions, resulting in attribute mismatches and missing objects. These errors normally fall into two categories: concept loss, when an object or attribute (i.e., concept) goes missing, and concept confusion, when an attribute is assigned to the wrong object or multiple objects blend together. Recent studies suggest that these challenges are caused by the limited capacity of CLIP text encoder to capture fine-grained semantic details. Although several methods have been proposed, concept loss and concept confusion persist in recent T2I models when handling multi-object, multi-attribute prompts. In this paper, we conduct an embedding-level analysis to develop a training-free, inference-time solution for concept loss and concept confusion across CLIP-based T2I models. We find that (1) concepts mentioned later exhibit higher embedding entropy, reflecting less concentrated representations that are more often associated with concept loss, and (2) the first CLIP attention layer captures the strength of binding between each object and its attribute. Guided by our findings, we introduce TIE, a method that improves semantic alignment through a single text-embedding update. TIE addresses concept loss via entropy-aware singular value amplification and resolves concept confusion through interpolation-extrapolation binding based on CLIP attention scores, all in a training-free manner. Extensive experiments demonstrate that TIE enhances semantic fidelity in multi-concept scenarios with minimal sampling overhead. The project page is available at https://hong-sujung.github.io/TIE/.
Text-to-image diffusion models have two major drawbacks that severely limit their practical utility: (1) standard models lack an intrinsic mechanism for continuous, concept-specific guidance (e.g., for precisely controlling how aesthetically pleasing an image looks), and (2) they lack reliability for tasks requiring high local coherence (e.g., generating text or human hands). To tackle these issues, we introduce a novel notion of concept-wise mutual information and find large, concept-dependent differences between individual layers, demonstrating that the generation of specific structures is localized in distinct parts of the network. We exploit this insight by reinforcing the impact of concept-relevant layers in Concept Guidance (CoG), a precise, target-specific guidance method that works for models out-of-the-box without additional training, external models, gradients, or prompt engineering. CoG first quantifies each layer's concept-specific impact and then guides denoising using a weighted combination of predictions generated with concept-relevant layers skipped. We demonstrate performance increases across various targets and popular models like PixArt-alpha, SD3, SD3.5, and FLUX.1-dev. Code is available at https://github.com/CompVis/concept_guidance
Nikolai Röhrich, Isabell Hans, Felix Krause +1
LMU Munich, Germany · Konrad Zuse School of Excellence in Reliable AI (relAI), Germany · Munich Center for Machine Learning (MCML), Germany