OptIMerge: Optimization-Based Merging of Diffusion Models
Organizations: Google
Abstract
The arithmetic merging of weights from multiple task-specific models fine-tuned from a shared base model, commonly known as model souping, has emerged as a popular strategy for multi-task learning without the computational overhead of joint retraining. However, naive scalar weight averaging frequently leads to severe performance degradation when the constituent models learn structurally divergent feature representations, leading to destructive parameter interference. We study this problem in a largely unexplored setting: merging image-to-image (I2I) diffusion models fine-tuned for dense, pixel-accurate tasks such as image restoration. Unlike the large language models (LLMs) and text-to-image (T2I) diffusion models for which most merging methods were developed, I2I experts encode conflicting spatial priors (e.g., anisotropic motion-blur vs. isotropic defocus-blur) in their convolutional filters, where both naive soup and LLM-style element-wise merging introduce visible artifacts. We introduce OptIMerge, a gradient-based optimization framework: instead of relying on heuristic scalars, OptIMerge learns highly expressive, yet parameter-efficient merger coefficients by minimizing a multi-task diffusion loss over a small set of 100-500 images. OptIMerge parameterizes the merge with architecture-aware transformations: for attention layers, a structured low-rank feature alignment with explicit channel biases; for convolutional layers, per-kernel merger coefficients, and a low-rank Cayley transform that aligns channels by orthogonal rotation without distorting the filters. By keeping the experts frozen and optimizing only these lightweight merger parameters (equivalent to <0.4% of the network footprint), OptIMerge achieves simultaneous multi-task capabilities that rival computationally expensive joint training and decisively outperform zero-shot model soups.
Figures & tables
| (a) Motion Deblur (OID) + Defocus Deblur (DPDD) | OID (300 samples) | DPDD (300 samples) | |||||
| Method / Strategy | Trainable Params | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM |
| Single-Task Experts | — | 0.2113 | 22.92 | 0.6266 | 0.2023 | 23.39 | 0.6392 |
| Joint Multi-Task Training | 890M (100%) | 0.2154 | 22.59 | 0.6173 | 0.2091 | 23.37 | 0.6404 |
| Naive Arithmetic Soup ( ) | 0 (0%) | 0.2899 | 20.88 | 0.5601 | 0.2641 | 23.15 | 0.6398 |
| TIES-Merging ( Yadav et al., 2023 ) | 0 (0%) | 0.3252 | 19.86 | 0.5367 | 0.3898 | 21.59 | 0.6062 |
| DARE ( Yu et al., 2024 ) | 0 (0%) | 0.3042 | 20.25 | 0.5364 | 0.3032 | 22.30 | 0.6156 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Method / Strategy | Trainable Params | OID | DPDD | ||||
| LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | ||
| Single-Task Experts | — | 0.2113 | 22.92 | 0.6266 | 0.2023 | 23.39 | 0.6392 |
| Joint Multi-Task Training | 890M (100%) | 0.2154 | 22.59 | 0.6173 | 0.2091 | 23.37 | 0.6404 |
| Naive Arithmetic Soup ( ) | 0 (0%) | 0.2899 | 20.88 | 0.5601 | 0.2641 | 23.15 | 0.6398 |
| 887.7K | 0.2647 | 21.68 | 0.5774 | 0.2562 | 23.13 | 0.6289 | |
| 1.1M | 0.2609 | 2 1.78 | 0.5808 | 0.2551 | 23.16 | 0.6301 | |
| Method / Strategy | Trainable Params | OID | DPDD | ||||
| LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | ||
| Single-Task Experts | — | 0.2113 | 22.92 | 0.6266 | 0.2023 | 23.39 | 0.6392 |
| Joint Multi-Task Training | 890M (100%) | 0.2154 | 22.59 | 0.6173 | 0.2091 | 23.37 | 0.6404 |
| Naive Arithmetic Soup ( ) | 0 (0%) | 0.2899 | 20.88 | 0.5601 | 0.2641 | 23.15 | 0.6398 |
| 1.9M | 0.2265 | 22.43 | 0.6049 | 0.2213 | 23.59 | 0.6423 | |
| 6.7M | 0.2193 | 22.63 | 0.6169 | 0.2154 | 24.00 | 0.6549 | |
| Soup Weights | OID | DPDD | |||||
| DPDD ( ) | OID ( ) | LPIPS ( ) | PSNR ( ) | SSIM ( ) | LPIPS ( ) | PSNR ( ) | SSIM ( ) |
| 0.10 | 0.90 | 0.2132 | 22.71 | 0.6216 | 0.3627 | 23.30 | 0.6461 |
| 0.25 | 0.75 | 0.2305 | 22.08 | 0.6036 | 0.3229 | 23.17 | 0.6434 |
| 0.50 | 0.50 | 0.2900 | 20.88 | 0.5601 | 0.2641 | 23.15 | 0.6398 |
| 0.75 | 0.25 | 0.3319 | 20.34 | 0.5374 | 0.2227 | 23.43 | 0.6446 |
| 0.90 | 0.10 | 0.3498 | 20.00 | 0.5234 | 0.2111 | 23.50 | 0.6440 |
| DF2K | OID | DPDD | |||||||
| Method/Strategy | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM |
| Single-Task Experts | 0.3131 | 21.26 | 0.5323 | 0.2113 | 22.92 | 0.6266 | 0.2023 | 23.39 | 0.6392 |
| Joint-Task Experts | 0.2989 | 21.53 | 0.5378 | 0.2180 | 22.48 | 0.6046 | 0.2158 | 23.19 | 0.6319 |
| Naive Soup | 0.3527 | 20.90 | 0.5013 | 0.3021 | 20.24 | 0.5235 | 0.3043 | 22.45 | 0.6118 |
| 0.3376 | 20.96 | 0.5098 | 0.2898 | 20.74 | 0.5420 | 0.2749 | 22.64 | 0.6108 | |
| 0.3196 | 21.10 | 0.5218 | 0.2563 | 21.44 | 0.5696 | 0.2470 | 22.95 | 0.6231 | |
Diffusion Models
Diffusion models learn to reverse gradual data corruption, enabling flexible generation, editing, forecasting, and inference.
See all 1,266 papers