Memory-efficient personalization is essential for adapting text-to-image diffusion models while preserving user privacy and operating within the limited computational resources of edge devices. To this end, we propose Mix-Opt, a novel mixed optimization framework that adaptively chooses between backpropagation on low-resolution images (BP-low) and zeroth-order optimization on high-resolution images (ZO-high), guided by the characteristics of the diffusion process. As observed in our experiments, BP-low efficiently adapts the model to target-specific features but suffers from structural distortions due to resolution mismatch. Conversely, ZO-high refines high-resolution details with minimal memory overhead but faces slow convergence when applied without prior adaptation. By complementing both methods, our framework leverages BP-low for effective personalization while using ZO-high to maintain structural consistency, achieving memory-efficient and high-quality fine-tuning. To maximize the efficacy of both BP-low and ZO-high, we introduce a timestep-aware probabilistic function that dynamically selects the appropriate optimization strategy during training. This function mitigates overfitting from BP-low at high timesteps, where structural information is critical, while ensuring that ZO-high is applied more effectively as training progresses. Experimental results across both U-Net- and DiT-based diffusion models demonstrate that Mix-Opt achieves competitive personalization performance while significantly reducing memory consumption. Mix-Opt enables scalable, high-quality personalization without modifying the inference process or increasing inference latency.
Figures & tables
Figure 1: Overview of Mix-Opt. While BP-low and ZO-high offer memory efficiency, each has its limitations. Mix-Opt dynamically combines both methods to achieve high-quality personalization while significantly reducing training memory consumption compared to backpropagation on high-resolution inputs (BP-high).
Figure 2: Observations for both optimization strategies. (a) MeZO requires target-specific knowledge; applying it from scratch fails to personalize the model, while using it after partial fine-tuning improves personalization. (b) Fine-tuning results for BP-high and BP-low trained at different timestep ranges. BP-high retains personalization features across all timesteps, whereas BP-low struggles to generate consistent outputs due to resolution mismatch, with severe structural distortions at high timesteps.
Figure 3: Comparison of TAP and DTAP over training steps ( tmax,imax=1000,k=0.05,tmid=750 ). The black dashed line shows TAP with a fixed tmid , while colored curves show DTAP adapting over training. At i=imid , DTAP coincides with TAP.
Table 4Table 5
Backbone
Method
DINO
CLIP-I
CLIP-T
Mem. (GB)
LoRA-BP-High
0.7239
0.8043
0.3057
3.97
SANA-0.6B
Ours
0.7076
0.7975
0.3076
1.75
LoRA-BP-High
0.7125
0.7942
0.3103
6.22
SANA-1.6B
Ours
0.7039
0.7924
0.3073
3.43
LoRA-BP-High
0.6717
0.7991
0.2667
55.61
FLUX.1-dev
Ours (r=0.5)
0.6716
0.7911
0.2922
33.51
Table 5: Comparison of LoRA-BP-High and Mix-Opt on DiT-based models. The results are obtained with mixed precision (bf16).
Figure 4: Qualitative comparison of different optimization methods, including BP-high, BP-low, ZO-high, Random Selection, Reverse Scheduling, and Mix-Opt (DTAP), conducted with Stable Diffusion v2.1 at low resolution ratio = 0.5.
Figure 5: User study results on SD V2.1, comparing Mix-Opt with BP-high and BP-low in terms of Subject Fidelity and Text Fidelity. In BP-Low and Mix-Opt, the low-resolution ratio is set to 0.5.
Table 9
Figure 6: Hyperparameter selection for Mix-Opt in SD V2.1, with the low-resolution ratio set to 0.5. We analyze the effects of different choices for the MeZO learning rate α , gradient accumulation steps N , and the steepness parameter k .
Method
DINO
CLIP-I
CLIP-T
Mem. (GB)
LoRA-BP-High
0.6762
0.8075
0.2961
5.09
LoRA-BP-High † + 4-bit
0.6575
0.7991
0.2964
2.58
Ours † (r=0.5)
0.6776
0.8069
0.2926
2.08
Ours † (r=0.5) + 4-bit
0.6742
0.7998
0.2922
1.67
Table 8: Results with 4-bit weight quantization on SD V2.1. † denotes mixed precision with fp16.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Method
DINO
CLIP-I
CLIP-T
Mem. (GB)
TARA + BP-high †
0.2744
0.6179
0.3165
2.76
TARA + Ours † (r=0.75)
0.2653
0.6218
0.3173
2.44
TARA + Ours † (r=0.5)
0.2658
0.6252
0.3159
2.07
Appendix
Table 9: Multi-concept personalization results with TARA on SD V1.5. † denotes mixed precision with fp16.
Base Model
Method
DreamSim ↓
Mem. (GB) ↓
SD V1.5
PairCustom + BP-high †
0.7791
2.76
PairCustom + Ours † (r=0.75)
0.7816
2.44
PairCustom + Ours † (r=0.5)
0.7712
2.07
SD V2.1
PairCustom + BP-high †
0.7901
2.77
PairCustom + Ours † (r=0.75)
0.8005
2.44
PairCustom + Ours † (r=0.5)
0.7991
2.08
Appendix
Table 10: Style-personalization results with PairCustomization. Lower DreamSim indicates better style alignment, and † denotes mixed precision with fp16.
Method
Performance Metrics
Mem.
DINO
CLIP-I
CLIP-T
(GB)
LoRA-BP-High
0.6762
0.8075
0.2961
5.09
Ours (r=0.750)
0.6758
0.8058
0.2969
4.42
Ours (r=0.625)
0.6787
0.8043
0.2955
4.14
R-4
Ours (r=0.500)
0.6833
0.8067
0.2938
3.92
LoRA-BP-High
0.6325
0.7803
0.3084
6.65
Appendix
Table 11: Impact of LoRA rank (R) and prior preservation loss (pp) on performance and memory consumption in SD V2.1. Rank-128 is trained for only 500 iterations due to artifacts in BP-High.
Figure 7: DTAP selection patterns across training iterations (x-axis) and timesteps (y-axis). Blue points represent BP-low and red points represent ZO-high. The boundary is defined as a linear function of i , given by tdyn(i) , illustrating how k and tmid influence the transition.
Figure 8: Effect of k and tmid on Mix-Opt performance for SD V2.1 ( r=0.5 ). Heatmaps use red for higher scores and green for lower scores. Higher tmid (lower ZO-high frequency) increases DINO and CLIP-I but reduces CLIP-T, indicating overfitting risks.
Training Cost (TFLOPS)
N=1
N=2
N=4
N=8
Mix-Opt (r=0.750)
1.5026
1.8370
2.5057
3.8432
Mix-Opt (r=0.625)
1.2417
1.5761
2.2448
3.5823
Mix-Opt (r=0.500)
1.0283
1.3627
2.0314
3.3689
Mix-Opt (r=0.375)
0.8623
1.1967
1.8654
3.2029
Appendix
Table 12: Average training computational cost (lower is better) per iteration under Mix-Opt for different r and N . Mix-Opt can achieve up to 58% reduction by lowering r and N . BP-high baseline is 2.0622 TFLOPS per iteration. Values lower than BP-high are highlighted.
Training Time (s)
N=1
N=2
N=4
N=8
Mix-Opt (r=0.750)
184.5
211.9
288.1
431.2
Mix-Opt (r=0.625)
189.3
231.4
294.8
431.5
Mix-Opt (r=0.500)
177.8
210.6
282.8
416.6
Mix-Opt (r=0.375)
179.1
217.7
280.1
450.1
Appendix
Table 13: Training time (lower is better) over 1000 iterations for different r and N . Reducing N provides the largest speedup, while changes in r have a smaller effect. BP-high baseline is 251.4 seconds. Values lower than BP-high are highlighted.
Subject Fidelity
N=1
N=2
N=4
N=8
Mix-Opt (r=0.750)
0.7396
0.7434
0.7408
0.7370
Mix-Opt (r=0.625)
0.7454
0.7420
0.7415
0.7442
Mix-Opt (r=0.500)
0.7361
0.7355
0.7450
0.7356
Mix-Opt (r=0.375)
0.7184
0.7171
0.7213
0.7203
Appendix
Table 14: Subject fidelity (higher is better) measured by the average of DINO and CLIP-I scores across r and N . Performance remains stable down to r=0.5 with a slight drop at r=0.375 , and variation in N has minimal impact. BP-high baseline is 0.7419. Values higher than BP-high are highlighted.
Diffusion Transformers (DiTs) have significantly enhanced text-to-image (T2I) generation quality, enabling high-quality personalized content creation. However, fine-tuning these models requires substantial computational complexity and memory, limiting practical deployment under resource constraints. To tackle these challenges, we propose a memory-efficient fine-tuning framework called DiT-BlockSkip, integrating timestep-aware dynamic patch sampling and block skipping by precomputing residual features. Our dynamic patch sampling strategy adjusts patch sizes based on the diffusion timestep, then resizes the cropped patches to a fixed lower resolution. This approach reduces forward & backward memory usage while allowing the model to capture global structures at higher timesteps and fine-grained details at lower timesteps. The block skipping mechanism selectively fine-tunes essential transformer blocks and precomputes residual features for the skipped blocks, significantly reducing training memory. To identify vital blocks for personalization, we introduce a block selection strategy based on cross-attention masking. Evaluations demonstrate that our approach achieves competitive personalization performance qualitatively and quantitatively, while reducing memory usage substantially, moving toward on-device feasibility (e.g., smartphones, IoT devices) for large-scale diffusion transformers.
Text-to-image personalization aims to equip pretrained diffusion models with the ability to learn user-specific concepts from only a few reference images. Existing methods, however, typically train each concept independently or assume that all concepts are available simultaneously. In practical applications, concepts often arrive sequentially, requiring models to continually learn new subjects, objects, or styles while minimizing interference with previously acquired concepts. We address this continual concept personalization setting by extending residual-space adaptation to the stochastic adaptation process of pretrained diffusion models. Our work achieves the highest concept fidelity among competing continual personalization methods while maintaining near-zero forgetting. Beyond these empirical gains, the learned residual-space dynamics reveal where adaptation capacity is concentrated within the diffusion model. Restricting adaptation to the identified layers by our method preserves nearly the same performance while reducing the number of trainable parameters by up to 72%, demonstrating that residual-space dynamics can identify the layers that contribute most to continual concept personalization and directly guide more parameter-efficient adaptation.
Personalizing text-to-image diffusion models from a few reference images requires preserving subject identity while following prompts that describe new contexts. Full-model fine-tuning is parameter-intensive, whereas low-rank adaptation (LoRA) reduces the number of trainable parameters but leaves open how adaptation capacity should be distributed across layers. We introduce Diffu-LoRA, a parameter-efficient method that learns this allocation through gated low-rank adaptation. Diffu-LoRA inserts trainable low-rank components into the linear layers of Transformer blocks and assigns a learnable gate to each component. Bilevel optimization updates the adaptation weights and gate parameters on separate data splits, while progressive pruning removes components with the lowest gate values to meet a prescribed rank budget. This procedure allocates adaptation capacity nonuniformly across layers while keeping the pretrained backbone frozen. Experiments with Stable Diffusion on subjects from DreamBooth and additional collected datasets show improved overall subject fidelity and prompt alignment relative to the evaluated fine-tuning baselines. Ablation studies examine the contributions of bilevel optimization, progressive pruning, and adapter placement. These results support learned rank allocation as a practical approach to parameter-efficient diffusion model personalization.
Tianjing Li, Wei Zhu
Zhangjiang Lab of Artificial Intelligence, Shanghai, China