Rescaling OLS-Merged Adapters To Improve Personalized Image Generation
Organizations: Technical University of Munich (TUM), Munich Center for Machine Learning (MCML) · Helmholtz Munich
Abstract
Merging multiple adapters into a single image generation model reduces deployment costs while preserving diverse task capabilities or personalization modes. State-of-the-art methods such as RegMean, Mix-of-Show, and Subspace Routing fit the outputs of the merged model to those of the individual adapters using ordinary least squares (OLS). We show that this reconstruction objective systematically shrinks task-specific signals whenever the activation distributions of different tasks overlap, producing a merged adapter that is too weak for downstream use. We derive a decomposition that quantifies this shrinkage and propose rescaling the merged update to restore signal strength. On multi-subject personalization with up to 30 merged adapters, rescaling substantially improves subject fidelity over unscaled baselines. Our approach also improves results on style transfer, concept erasure, image classification, and LLM-based text classification.
Figures & tables
| Method | FLUX.1-dev | SD3.5-Medium | ||||||||||||||
| D | C | D | C | D | C | D | C | D | C | D | C | D | C | D | C | |
| Weight-based Methods | ||||||||||||||||
| Task Arithmetic [ 30 ] | .547 | .715 | .493 | .723 | .458 | .675 | .468 | .698 | .568 | .747 | .522 | .724 | .515 | .727 | .502 | .720 |
| TIES [ 73 ] | .566 | .745 | .484 | .720 | .451 | .679 | .467 | .711 | .572 | .749 | .523 | .730 | .517 | .727 | .494 | .713 |
| DARE [ 79 ] | .544 | .714 | .493 | .724 | .457 | .675 | .467 | .698 | .572 | .747 | .522 | .722 | .515 | .726 | .502 | .720 |
| Method | WikiArt Style Transfer | CIFAR-10 Concept Erasure | |||||||||||||||
| FLUX.1-dev | SD3.5-Medium | SD v1.4 | |||||||||||||||
| S | C | S | C | S | C | S | C | E | P | E | P | E | P | ||||
| RegMean [ 32 ] | .052 | .515 | .041 | .513 | .015 | .532 | .005 | .538 | .83 | .98 | .828 | .84 | .99 | .839 | .82 | .99 | .819 |
| Mix-of-Show [ 22 ] | .025 | .490 | .014 | .488 | .022 | .534 | .008 | .535 | .86 | .98 | .869 | .86 | .94 | .866 | .86 | .88 | .843 |
| SSR [ 67 ] | .018 | .489 | .015 | .489 | .007 | .531 | .003 | .538 | .82 | .99 | .826 | .86 | .93 | .862 | .85 | .91 | .846 |
| Method | DreamBooth | Image Classification | LLM | ||||||||||||||
| FLUX.1-dev | SD3.5-Medium | ViT-B/32 | ViT-L/14 | Llama-3.1-8B | |||||||||||||
| NLI ( ) | Gen ( ) | Comb ( ) | |||||||||||||||
| D | C | D | C | D | C | D | C | Accuracy (%) | Score (%) | ||||||||
| RegMean | .646 | .800 | .619 | .778 | .587 | .762 | .553 | .751 | 82.5 | 74.8 | 71.7 | 89.5 | 84.4 | 81.0 | 86.3 | 61.0 | 77.0 |
| .693 | .822 | .644 | .789 | .635 | .779 | .600 | .766 | 84.1 | 79.0 | 76.2 | 91.4 | 88.6 | 86.7 | 86.0 | 61.8 | 77.8 | |
| line search | .653 | .815 | .663 | .790 | .655 | .785 | .623 | .767 | 84.0 | 79.1 | 76.1 | 91.4 | 88.7 | 86.8 | 86.1 | 64.2 | 77.9 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | FLUX.1-dev | SD3.5-Medium | |||||||||||||||
| D | I | D | I | D | I | D | I | D | I | D | I | D | I | D | I | ||
| SSR ( ) | .680 | .816 | .649 | .798 | .628 | .792 | .607 | .788 | .628 | .784 | .589 | .768 | .574 | .759 | .569 | .757 | |
| SSR + PICO | .558 | .753 | .514 | .742 | .500 | .740 | .490 | .737 | .524 | .741 | .472 | .715 | .463 | .711 | .454 | .707 | |
| SSR + LiNeS | .669 | .812 | .654 | .802 | .623 | .792 | .561 | .771 | .662 | .792 | .634 | .773 | .614 | .762 | .573 | .747 | |
| SSR + (ours) | .707 | .825 | .697 | .821 | .691 | .820 | .674 | .815 | .664 | .797 | .646 | .788 | .640 | .781 | .626 | .774 | |
| Method | FLUX.1-dev | SD3.5-Medium | ||||||||
| min | PFLOP | min | PFLOP | min | PFLOP | min | PFLOP | min | PFLOP | |
| RegMean † | 144 | 478 | 0.5 | 1.9 | 1.5 | 5.8 | 2.4 | 9.6 | 4.4 | 19 |
| Mix-of-Show † | 170 | 485 | 1.0 | 2.1 | 2.0 | 5.9 | 2.7 | 9.8 | 4.9 | 19 |
| IterIS † | OOM | 9.9 | 43 | 27 † | 128 | 46 † | 213 | 94 † | 426 | |
| SSR † | 11 | 84 | 0.5 | 1.7 | 1.7 | 5.4 | 2.9 | 9.4 | 6.0 | 21 |
| DreamBooth | |||||||||||||||
| FLUX.1-dev | SD3.5-M | ||||||||||||||
| txt | img | sgl | txt | img | sgl | txt | img | sgl | txt | img | txt | img | txt | img | |
| .83 | .60 | .69 | .82 | .55 | .64 | .81 | .54 | .62 | .72 | .57 | .69 | .52 | .68 | .51 | |
| 1.21 | 1.67 | 1.45 | 1.22 | 1.82 | 1.57 | 1.23 | 1.87 | 1.61 | 1.40 | 1.76 | 1.46 | 1.92 | 1.47 | 1.97 | |
| (line search) | 1.75 | 1.75 | 1.75 | 2.00 | 1.75 | 2.00 | |||||||||
| SSR | RegMean | ||||||||
| 9 | 15 | 20 | 30 | 9 | 15 | 20 | 30 | ||
| No rescaling | .590 | .574 | .569 | .566 | .619 | .587 | .572 | .553 | |
| Group | .647 | .638 | .632 | .625 | .644 | .635 | .623 | .600 | |
| Per-layer | .645 | .625 | .610 | .592 | .649 | .636 | .629 | .611 | |
| DINO | .002 | .013 | .022 | .033 | .006 | .001 | .006 | .011 | |
| Per-layer better | 13/30 | 11/30 | 12/30 | 7/30 | 14/30 | 18/30 | 16/30 | 19/30 | |
| Method | Tuned | Grid (validation) | Fixed settings |
| Task Arithmetic | |||
| TIES | top- trimming per task, sign election, disjoint mean | ||
| DARE | on Task Arithmetic; from TA; rescale | ||
| RobustMerge | – | – | top- magnitude mask per task, rank-wise scaling, fuse |
| KnOTS | , top- | (i) at top- (ii) top- at selected | Applied on TIES |
| Core Space | Iso, | TSV merge in core space |
| FLUX.1-dev | SD3.5-Medium | ||||||||||||
| Method | HP | ||||||||||||
| Task Arithmetic | 1.0 | 0.9 | 0.8 | 0.3 | 0.4 | 0.2 | 0.9 | 1.0 | 0.6 | 0.4 | 0.3 | 0.2 | |
| TIES | 1.8 | 1.6 | 1.8 | 1.5 | 1.8 | 1.1 | 1.7 | 1.8 | 1.4 | 1.2 | 1.2 | 1.1 | |
| DARE | 0.5 | 0.3 | 0.1 | 0.5 | 0.7 | 0.5 | 0.3 | 0.1 | 0.9 | 0.3 | 0.1 | 0.1 | |
| KnOTS | 1.0 | 1.0 | 1.0 | 0.9 | 0.8 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | |
| top- (%) | 10 | 10 | 10 | 30 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | |
| CIFAR-10 objects | Celebrities | ||||||||||||||
| Method | |||||||||||||||
| E | P | E | P | E | P | E | P | E | P | ||||||
| Weight-based | |||||||||||||||
| Task Arithmetic | .88 | .71 | .813 | .79 | .49 | .653 | .74 | .47 | .619 | .99 | .82 | .899 | .85 | .75 | .795 |
| TIES | .82 | .80 | .815 | .78 | .56 | .693 | .73 | .57 | .678 | .93 | .85 | .887 | .62 | .70 | .654 |
| DARE | .88 | .65 | .789 | .75 | .52 | .663 | .74 | .45 | .611 | .97 | .84 | .899 | .89 | .74 | .805 |
| Method | HP | |||
| Task Arithmetic | 0.8 | 0.5 | 0.4 | |
| TIES | 1.7 | 1.7 | 1.6 | |
| DARE | 0.3 | 0.5 | 0.1 | |
| KnOTS | / top- (%) | 1.0 / 20 | 0.8 / 40 | 1.0 / 20 |
| Core Space | 1.0 | 1.0 | 1.0 | |
| Iso-C | 2.0 | 1.2 | 1.0 |
Model Merging
Model merging combines trained model weights or adapters to reuse capabilities while reducing the cost of joint training or ensembles.
See all 216 papers