cs.CVNov 17, 2026

Rescaling OLS-Merged Adapters To Improve Personalized Image Generation

Authors: Leander Girrbach, Niklas Schweiger, Zeynep Akata

Organizations: Technical University of Munich (TUM), Munich Center for Machine Learning (MCML) · Helmholtz Munich

Abstract

Merging multiple adapters into a single image generation model reduces deployment costs while preserving diverse task capabilities or personalization modes. State-of-the-art methods such as RegMean, Mix-of-Show, and Subspace Routing fit the outputs of the merged model to those of the individual adapters using ordinary least squares (OLS). We show that this reconstruction objective systematically shrinks task-specific signals whenever the activation distributions of different tasks overlap, producing a merged adapter that is too weak for downstream use. We derive a decomposition that quantifies this shrinkage and propose rescaling the merged update to restore signal strength. On multi-subject personalization with up to 30 merged adapters, rescaling substantially improves subject fidelity over unscaled baselines. Our approach also improves results on style transfer, concept erasure, image classification, and LLM-based text classification.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. OrthoFuse: Training-free Riemannian Fusion of Orthogonal Style-Concept Adapters for Diffusion Models

    Apr 6, 2026Ali Aliev, Kamil Garifullin, Nikolay Yudin +5Model MergingDiffusion Model Adaptation

  2. HimmBO: High-Dimensional Interactive Model-Merging Bayesian Optimization for Image Generation

    Oct 22, 2026Jinyuan Mao, Zhihao Shen, Chenxi LiuModel MergingBayesian Optimization

  3. Not All Layers Are Created Equal: Adaptive LoRA Ranks for Personalized Image Generation

    Mar 23, 2026Donald Shenaj, Federico Errica, Antonio CartaLow-Rank AdaptationAdaptive Rank Allocation