Memory-efficient personalization is essential for adapting text-to-image diffusion models while preserving user privacy and operating within the limited computational resources of edge devices. To this end, we propose Mix-Opt, a novel mixed optimization framework that adaptively chooses between backpropagation on low-resolution images (BP-low) and zeroth-order optimization on high-resolution images (ZO-high), guided by the characteristics of the diffusion process. As observed in our experiments, BP-low efficiently adapts the model to target-specific features but suffers from structural distortions due to resolution mismatch. Conversely, ZO-high refines high-resolution details with minimal memory overhead but faces slow convergence when applied without prior adaptation. By complementing both methods, our framework leverages BP-low for effective personalization while using ZO-high to maintain structural consistency, achieving memory-efficient and high-quality fine-tuning. To maximize the efficacy of both BP-low and ZO-high, we introduce a timestep-aware probabilistic function that dynamically selects the appropriate optimization strategy during training. This function mitigates overfitting from BP-low at high timesteps, where structural information is critical, while ensuring that ZO-high is applied more effectively as training progresses. Experimental results across both U-Net- and DiT-based diffusion models demonstrate that Mix-Opt achieves competitive personalization performance while significantly reducing memory consumption. Mix-Opt enables scalable, high-quality personalization without modifying the inference process or increasing inference latency.
Figures & tables
Figure 1: Overview of Mix-Opt. While BP-low and ZO-high offer memory efficiency, each has its limitations. Mix-Opt dynamically combines both methods to achieve high-quality personalization while significantly reducing training memory consumption compared to backpropagation on high-resolution inputs (BP-high).
Figure 2: Observations for both optimization strategies. (a) MeZO requires target-specific knowledge; applying it from scratch fails to personalize the model, while using it after partial fine-tuning improves personalization. (b) Fine-tuning results for BP-high and BP-low trained at different timestep ranges. BP-high retains personalization features across all timesteps, whereas BP-low struggles to generate consistent outputs due to resolution mismatch, with severe structural distortions at high timesteps.
Figure 3: Comparison of TAP and DTAP over training steps ( tmax,imax=1000,k=0.05,tmid=750 ). The black dashed line shows TAP with a fixed tmid , while colored curves show DTAP adapting over training. At i=imid , DTAP coincides with TAP.
Table 4Table 5
Backbone
Method
DINO
CLIP-I
CLIP-T
Mem. (GB)
LoRA-BP-High
0.7239
0.8043
0.3057
3.97
SANA-0.6B
Ours
0.7076
0.7975
0.3076
1.75
LoRA-BP-High
0.7125
0.7942
0.3103
6.22
SANA-1.6B
Ours
0.7039
0.7924
0.3073
3.43
LoRA-BP-High
0.6717
0.7991
0.2667
55.61
FLUX.1-dev
Ours (r=0.5)
0.6716
0.7911
0.2922
33.51
Table 5: Comparison of LoRA-BP-High and Mix-Opt on DiT-based models. The results are obtained with mixed precision (bf16).
Figure 4: Qualitative comparison of different optimization methods, including BP-high, BP-low, ZO-high, Random Selection, Reverse Scheduling, and Mix-Opt (DTAP), conducted with Stable Diffusion v2.1 at low resolution ratio = 0.5.
Figure 5: User study results on SD V2.1, comparing Mix-Opt with BP-high and BP-low in terms of Subject Fidelity and Text Fidelity. In BP-Low and Mix-Opt, the low-resolution ratio is set to 0.5.
Table 9
Figure 6: Hyperparameter selection for Mix-Opt in SD V2.1, with the low-resolution ratio set to 0.5. We analyze the effects of different choices for the MeZO learning rate α , gradient accumulation steps N , and the steepness parameter k .
Method
DINO
CLIP-I
CLIP-T
Mem. (GB)
LoRA-BP-High
0.6762
0.8075
0.2961
5.09
LoRA-BP-High † + 4-bit
0.6575
0.7991
0.2964
2.58
Ours † (r=0.5)
0.6776
0.8069
0.2926
2.08
Ours † (r=0.5) + 4-bit
0.6742
0.7998
0.2922
1.67
Table 8: Results with 4-bit weight quantization on SD V2.1. † denotes mixed precision with fp16.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Method
DINO
CLIP-I
CLIP-T
Mem. (GB)
TARA + BP-high †
0.2744
0.6179
0.3165
2.76
TARA + Ours † (r=0.75)
0.2653
0.6218
0.3173
2.44
TARA + Ours † (r=0.5)
0.2658
0.6252
0.3159
2.07
Appendix
Table 9: Multi-concept personalization results with TARA on SD V1.5. † denotes mixed precision with fp16.
Base Model
Method
DreamSim ↓
Mem. (GB) ↓
SD V1.5
PairCustom + BP-high †
0.7791
2.76
PairCustom + Ours † (r=0.75)
0.7816
2.44
PairCustom + Ours † (r=0.5)
0.7712
2.07
SD V2.1
PairCustom + BP-high †
0.7901
2.77
PairCustom + Ours † (r=0.75)
0.8005
2.44
PairCustom + Ours † (r=0.5)
0.7991
2.08
Appendix
Table 10: Style-personalization results with PairCustomization. Lower DreamSim indicates better style alignment, and † denotes mixed precision with fp16.
Method
Performance Metrics
Mem.
DINO
CLIP-I
CLIP-T
(GB)
LoRA-BP-High
0.6762
0.8075
0.2961
5.09
Ours (r=0.750)
0.6758
0.8058
0.2969
4.42
Ours (r=0.625)
0.6787
0.8043
0.2955
4.14
R-4
Ours (r=0.500)
0.6833
0.8067
0.2938
3.92
LoRA-BP-High
0.6325
0.7803
0.3084
6.65
Appendix
Table 11: Impact of LoRA rank (R) and prior preservation loss (pp) on performance and memory consumption in SD V2.1. Rank-128 is trained for only 500 iterations due to artifacts in BP-High.
Figure 7: DTAP selection patterns across training iterations (x-axis) and timesteps (y-axis). Blue points represent BP-low and red points represent ZO-high. The boundary is defined as a linear function of i , given by tdyn(i) , illustrating how k and tmid influence the transition.
Figure 8: Effect of k and tmid on Mix-Opt performance for SD V2.1 ( r=0.5 ). Heatmaps use red for higher scores and green for lower scores. Higher tmid (lower ZO-high frequency) increases DINO and CLIP-I but reduces CLIP-T, indicating overfitting risks.
Training Cost (TFLOPS)
N=1
N=2
N=4
N=8
Mix-Opt (r=0.750)
1.5026
1.8370
2.5057
3.8432
Mix-Opt (r=0.625)
1.2417
1.5761
2.2448
3.5823
Mix-Opt (r=0.500)
1.0283
1.3627
2.0314
3.3689
Mix-Opt (r=0.375)
0.8623
1.1967
1.8654
3.2029
Appendix
Table 12: Average training computational cost (lower is better) per iteration under Mix-Opt for different r and N . Mix-Opt can achieve up to 58% reduction by lowering r and N . BP-high baseline is 2.0622 TFLOPS per iteration. Values lower than BP-high are highlighted.
Training Time (s)
N=1
N=2
N=4
N=8
Mix-Opt (r=0.750)
184.5
211.9
288.1
431.2
Mix-Opt (r=0.625)
189.3
231.4
294.8
431.5
Mix-Opt (r=0.500)
177.8
210.6
282.8
416.6
Mix-Opt (r=0.375)
179.1
217.7
280.1
450.1
Appendix
Table 13: Training time (lower is better) over 1000 iterations for different r and N . Reducing N provides the largest speedup, while changes in r have a smaller effect. BP-high baseline is 251.4 seconds. Values lower than BP-high are highlighted.
Subject Fidelity
N=1
N=2
N=4
N=8
Mix-Opt (r=0.750)
0.7396
0.7434
0.7408
0.7370
Mix-Opt (r=0.625)
0.7454
0.7420
0.7415
0.7442
Mix-Opt (r=0.500)
0.7361
0.7355
0.7450
0.7356
Mix-Opt (r=0.375)
0.7184
0.7171
0.7213
0.7203
Appendix
Table 14: Subject fidelity (higher is better) measured by the average of DINO and CLIP-I scores across r and N . Performance remains stable down to r=0.5 with a slight drop at r=0.375 , and variation in N has minimal impact. BP-high baseline is 0.7419. Values higher than BP-high are highlighted.