Domain-Adaptive Model Merging Across Disconnected Modes

Junming Liu; Yusen Zhang; Rongchao Zhang; Wenkai Zhu; Tian Wu

cs.DCMar 6, 2026

Domain-Adaptive Model Merging Across Disconnected Modes

Authors: Junming Liu, Yusen Zhang, Rongchao Zhang, Wenkai Zhu, Tian Wu

Organizations: Tongji University · Peking University · Southeast University · Nanchang University

Abstract

Learning across domains is challenging when data cannot be centralized due to privacy or heterogeneity, which limits the ability to train a single comprehensive model. Model merging provides an appealing alternative by consolidating knowledge from multiple specialized models into one, avoiding data sharing and reducing retraining cost. In this work, we present DMM, a data-free model merging framework designed to handle highly divergent models. DMM proceeds in three steps. First, domain-specific models are trained independently. Second, models with high similarity are merged using standard techniques to ensure stability. Third, we synthesize pseudo-data from normalization statistics and distill knowledge from divergent models into the merged model through a lightweight refinement guided by these samples. This approach preserves rare but critical knowledge while maintaining stability. Extensive experiments on unimodal and multimodal benchmarks show that DMM achieves state-of-the-art performance over existing merging methods.

Figures & tables

Figure 1: Overall workflow of the proposed DMM framework, which integrates model initialization, model merging with proxy data synthesis, and lightweight data-free knowledge distillation.

Method	CIFAR-10			CIFAR-100			CrisisMMD
Method	$\alpha=1.0$	$\alpha=0.1$	$\alpha=0.01$	$\alpha=1.0$	$\alpha=0.1$	$\alpha=0.01$	$\alpha=1.0$	$\alpha=0.1$	$\alpha=0.01$
FedAvg [ 12 ]	78.34	56.32	36.76	65.55	59.30	48.72	60.93	45.21	22.50
FedProx [ 6 ]	79.45	56.73	37.32	65.54	59.64	49.73	62.88	46.84	21.32
FedBN [ 7 ]	79.52	57.68	38.35	66.92	59.16	50.70	60.51	44.79	23.72
FedAvg+DMM	82.30	65.85	53.66	68.68	64.31	53.04	64.33	49.70	30.46
PLeaS [ 13 ]	60.51	45.77	33.30	49.74	44.97	39.56	52.18	40.11	23.67

Table 1: Accuracy (%) comparison on CIFAR-10, CIFAR-100, and CrisisMMD under different Dirichlet distributions.

Method	CIFAR-10	CIFAR-100
Cat-Merge (baseline)	47.67	45.51
+ Buffer Aggregation	48.11	45.63
+ Inversion Augmentation	53.32	47.16
+ DMM (full)	55.86	48.42

Table 2: Accuracy (%) comparison under moderately Non-IID (

\alpha=0.1

) on CIFAR-10 and CIFAR-100.

Method	CIFAR-10	CIFAR-100
FedAvg (baseline)	36.76	48.72
+ Buffer Aggregation	38.98	49.43
+ Inversion Augmentation	50.38	51.10
+ DMM (full)	53.66	53.04

Table 3: Accuracy (%) comparison under high Non-IID (

\alpha=0.01

) on CIFAR-10 and CIFAR-100.

Model Merging

Model merging combines trained model weights or adapters to reuse capabilities while reducing the cost of joint training or ensembles.

See all 192 papers

Bridging Domains through Subspace-Aware Model Merging
Mar 6, 2026Levy Chaves, Chao Zhou, Rebekka Burkholz +2Model Merging Cross-Domain Generalization
Universidade Estadual de Campinas (UNICAMP), Recod.ai Lab., Instituto de Computac¸˜ao, Brasil · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany · Intercom
ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimation
Mar 3, 2026Bo Xu, Haotian Wu, Hehai Lin +4Model Merging Language Model Merging
The Hong Kong University of Science and Technology (Guangzhou) · University of Science and Technology of China
Saliency-Aware Model Merging
May 30, 2026Jungin Park, Jiyoung Lee, Kwanghoon SohnTask Vector Merging Model Merging
Yonsei University, Seoul, South Korea · Ewha Womans University, Seoul, South Korea.

Mar 6, 2026J/K move · Enter open · S save

Bridging Domains through Subspace-Aware Model Merging

Levy Chaves, Chao Zhou, Rebekka Burkholz, Eduardo Valle+1

Universidade Estadual de Campinas (UNICAMP), Recod.ai Lab., Instituto de Computac¸˜ao, Brasil · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany · Intercom

Model Merging Cross-Domain Generalization

Open paper AskPDF arXiv

Model merging integrates multiple task-specific models into a single consolidated one. Recent research has made progress in improving merging performance for in-distribution or multi-task scenarios, but domain generalization in model merging remains underexplored. We investigate how merging models fine-tuned on distinct domains affects generalization to unseen domains. Through an analysis of parameter competition in the task matrix using singular value decomposition, we show that merging models trained under different distribution shifts induces stronger conflicts between their subspaces compared to traditional multi-task settings. To mitigate this issue, we propose SCORE (Subspace COnflict-Resolving mErging), a method designed to alleviate such singular subspace conflicts. SCORE finds a shared orthogonal basis by computing the principal components of the concatenated leading singular vectors of all models. It then projects each task matrix into the shared basis, pruning off-diagonal components to remove conflicting singular directions. SCORE consistently outperforms, on average, existing model merging approaches in domain generalization settings across a variety of architectures and model scales, demonstrating its effectiveness and scalability.

Abstract

Model Merging

Explore similar work

Bridging Domains through Subspace-Aware Model Merging

ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimation

Saliency-Aware Model Merging