Overview

Model merging studies how to combine weights or adapters from separately trained models into one model, aiming to reuse complementary skills without storing an ensemble or repeating joint training. Common approaches average checkpoints or add task vectors, while sparsification, low-rank and spectral methods, geometric alignment, and calibration-based routing aim to reduce interference. The methods are used for multi-task and continual learning, domain adaptation, and combining language, vision, speech, and specialized models; evaluations often measure task accuracy, capability retention, worst-task performance, and inference or storage cost. The central tension is that updates that help one task can erase or distort another: success depends on shared initialization, representation compatibility, update scale, and data or prompt distribution, and some task combinations may not be recoverable through weight-space fusion alone.

Latest papers

214 papers18 papers in the last four weeks, down 53% on the four weeks before. 0.1% of all new papers.

Sep 15, 2025

FedDAF: Federated Domain Adaptation Using Model Functional Distance

Federated Domain Adaptation (FDA) improves model performance at a target client by collaborating with source clients while preserving data privacy. FDA faces two key challenges: domain shift between source and target data, and limited labeled data at the target, a common constraint when a new site joins a federation before it has accumulated its own labeled data, as in clinical deployments. Most existing methods address domain shift alone, assuming ample target data; those that also tackle data scarcity still fail to prioritize source information according to the target's specific objective. We propose FedDAF, which addresses both challenges through similarity-based aggregation of the global source and target models, using their model functional distance, computed from the angle between their mean gradient fields on target data and normalized via a Gompertz function. The global source model itself is formed using a distance-based weighted average, giving greater weight to source models closer to the target model. Experiments on real-world datasets show FedDAF outperforms existing federated learning (FL), personalized FL, and FDA methods in test accuracy.
Aug 29, 2025

Rethinking Layer-wise Model Merging through Chain of Merges

Model merging has emerged as a simple yet effective method to fuse models without re-training. Existing techniques operate at the level of individual layers, thereby overlooking the inter-layer dependencies inherent in deep networks. We show that this simplification induces distributional mismatches in intermediate activations during merging, as changes applied to early layers fail to propagate to downstream ones. We identify these mismatches as a form of covariate shift that compounds across the network. To address this issue, we propose Chain of Merges (CoM), a novel procedure that sequentially merges weights across layers while accounting for inter-layer interactions. In particular, CoM mitigates covariate shift through a series of regression problems, where input activations are recomputed at each step to reflect the updated representations. Additionally, we introduce a dynamic weighting mechanism that prioritizes task-layer pairs most susceptible to performance degradation, improving robustness during merging. Experiments on standard benchmarks demonstrate that CoM achieves state-of-the-art performance across heterogeneous tasks.
Aug 8, 2025

One Size Does Not Fit All: Distribution-Guided Tensor-Wise Adaptive Sparsification for Model Merging

Merging multiple task-specific fine-tuned checkpoints into a single multi-task model provides an attractive alternative to joint multi-task training. To mitigate cross-task interference, a common strategy is to sparsify task vectors before aggregation. However, many widely used methods rely on a uniform global sparsity ratio, overlooking differences in pruning sensitivity across tasks and parameter tensors. Consequently, such uniform sparsification may remove useful parameter updates from sensitive tensors while retaining redundant updates in others. To address this limitation, we propose TADrop, a plug-and-play sparsifier that independently determines tensor-specific keep ratios from their magnitude distributions using a scale-invariant quantile ratio. TADrop operates in a data-free manner by default, while offering optional lightweight module-level calibration when a small validation set is available. Experiments across vision, language, and multimodal benchmarks, including LLM and LoRA merging, demonstrate consistent improvements across multiple host merging methods, including a 4.99-point gain over EMR-Merging on RoBERTa GLUE. In many-task ViT-B/16 merging, the improvement increases with the number of tasks, reaching 2.26 points when merging 30 tasks. Further analyses show that TADrop preserves more task-associated neurons under matched overall sparsity. Together, these findings highlight the value of distribution-guided sparsification at the tensor level for improving model merging.
Oct 31, 2024

Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learning

Federated learning (FL) is a learning paradigm that enables collaborative training of models using decentralized data. Recently, the utilization of pre-trained weight initialization in FL has been demonstrated to effectively improve model performance. However, the evolving complexity of current pre-trained models, characterized by a substantial increase in parameters, markedly intensifies the challenges associated with communication rounds required for their adaptation to FL. To address these communication cost issues and increase the performance of pre-trained model adaptation in FL, we propose an innovative model interpolation-based local training technique called ``Local Superior Soups.'' Our method enhances local training across different clients, encouraging the exploration of a connected low-loss basin within a few communication rounds through regularized model interpolation. This approach acts as a catalyst for the seamless adaptation of pre-trained models in in FL. We demonstrated its effectiveness and efficiency across diverse widely-used FL datasets. Our code is available at https://github.com/ubc-tea/Local-Superior-Soups.