Recent advances in text-to-image diffusion models have substantially improved the quality of image customization, enabling the synthesis of highly realistic images. Despite this progress, achieving fast and efficient personalization remains a key challenge, particularly for real-world applications. Existing approaches primarily accelerate customization for human subjects by injecting identity-specific embeddings into diffusion models, but these strategies do not generalize well to arbitrary object categories, limiting their applicability. To address this limitation, we propose a novel framework that employs a learned network to predict object-specific textual inversion embeddings, which are subsequently integrated into the UNet timesteps of a diffusion model for text-conditional customization. This design enables rapid, zero-shot personalization of a wide range of objects in a single forward pass, offering both flexibility and scalability. Extensive experiments across multiple tasks and settings demonstrate the effectiveness of our approach, highlighting its potential to support fast, versatile, and inclusive image customization. To the best of our knowledge, this work represents the first attempt to achieve such general-purpose, training-free personalization within diffusion models, paving the way for future research in personalized image generation.
Figures & tables
Figure 1: Using only a single input image and a concept extraction network, our method is able to personalize a text-to-image diffusion model much faster than test-time optimzation-based approaches, while maintaining the subject’s uniqueness and details.
Figure 2: An overview of our proposed approach. We first obtain the ‘ground-truth’ textual-embedding of the concepts present in the training set using test-time optimization based-methods like Textual-Inversion. Next, we train our Concept-Extraction network, to produce that embedding given a single image and a text-template. Once it is trained, we further fine-tune the cross-attention layers of the diffusion model using the modified textual-embeddings obtained from the frozen Concept-Extraction network. During inference, the Concept-Extraction network along with the fine-tuned diffusion model can be used to generate variations of the subject present in a given image, without requiring expensive optimization steps.
Figure 3: tSNE plot for textual inversion embeddings. We observe that the textual inversion embeddings of images of same identity are clustered.
Figure 4: Comparison with existing methods (Dreambooth (DB), Custom diffusion (CD), Textual inversion (TI)) on Custom101 dataset.
Figure 5: Qualitative comparison with existing methods on Custom101 dataset.
Figure 6: Qualitative results on color modification.
Figure 7: Failure cases
Method
Type
Ext. Model
Test set
DINO
CLIP-I
CLIP-T
Real img.
-
-
-
0.774
0.885
-
Text. Inv. [ 10 ]
Finetune
-
DB
0.569
0.780
0.255
DB [ 1 ]
Finetune
-
DB
0.668
0.803
0.305
CD [ 2 ]
Finetune
-
DB
0.643
0.790
0.305
DiffNat [ 3 ]
Finetune
-
DB
0.68
0.84
0.34
Re-Imagen [ 21 ]
Zero-shot
-
DB
0.600
0.740
0.270
Table 1: Comparison of few-shot finetuning task on DB dataset
Method
Type
Text-alignment
Img-alignment
speed(s)
Text. Inv. [ 10 ]
Finetune
0.612
0.752
2400
DB [ 1 ]
Finetune
0.752
0.752
1284
CD [ 2 ]
Finetune
0.769
0.744
400
Ours
Zero-shot
0.590
0.625
2
Table 2: Comparison of few-shot finetuning task on Custom101
Metric
RL
Loss
Finetune
w/o RL
w/ RL
LMSE
LMSE+LCE
Whole
Cross-attn
Text align. (CLIP-T)
0.12
0.59
0.32
0.59
0.33
0.59
Image align. (CLIP-I)
0.20
0.63
0.44
0.63
0.42
0.63
Table 3: Ablation studies on the Custom101 dataset.
Hyperparameter
Values
Stage 1 : Textual inversion generation
Size of Textual inversion embedding
1280
Image and text feature fusion dim.
CLIP-I (512) + CLIP-T (512)
Stage 2 : Training MLP
CLIP Model
“openai/clip-vit-base-patch32”
input dim. of MLP
1024
Table 4: Hyperparameters
Method
Type
speed(s)
Text. Inv. [ 10 ]
Finetune
2400
DB [ 1 ]
Finetune
1284
CD [ 2 ]
Finetune
400
Re-Imagen [ 21 ]
Zero-shot
10
ELITE [ 14 ]
Zero-shot
4
BLIP-Diff [ 22 ]
Zero-shot
8
Table 5: Comparison of Inference speed
Figure 8: Human evaluation comparison w.r.t test-time optimization methods