Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions

Mingyang Song; Mao Zheng

cs.CLMar 10, 2026

Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions

Authors: Mingyang Song, Mao Zheng

Organizations: Large Language Model Department, Tencent, China

Abstract

Model merging combines the parameters of multiple neural networks into a single model without additional training. As fine-tuned large language models (LLMs) proliferate, merging offers a computationally efficient alternative to ensembles and full retraining, enabling practitioners to compose specialized capabilities at minimal cost. This survey examines model merging in the LLM era through the FUSE taxonomy, organized along Foundations, Unification Strategies, Scenarios, and Ecosystem. We first establish the theoretical underpinnings of merging, including loss landscape geometry and mode connectivity, then systematically review the algorithmic space spanning weight averaging, task vector arithmetic, sparsification-enhanced methods, mixture-of-experts architectures, and evolutionary optimization. We further examine downstream applications across multi-task learning, safety alignment, domain specialization, and federated learning, and survey the supporting ecosystem of tools and evaluation benchmarks. Finally, we identify key open challenges and future directions, aiming to equip researchers and practitioners with a structured foundation for advancing model merging.

Figures & tables

Model Family	Organization	Parameters	Year	Architecture	Key Features
LLaMA ( Touvron et al., 2023a )	Meta	7B–65B	2023	Decoder-only	RoPE, Pre-norm, SwiGLU
LLaMA-2 ( Touvron et al., 2023b )	Meta	7B–70B	2023	Decoder-only	GQA, Extended context
LLaMA-3 ( Team, 2024b )	Meta	8B–405B	2024	Decoder-only	128K context, GQA
Mistral ( Jiang et al., 2023 )	Mistral AI	7B	2023	Decoder-only	Sliding window attention
Mixtral ( Jiang et al., 2024 )	Mistral AI	8 $\times$ 7B	2024	Sparse MoE	8 experts, top-2 routing
Qwen ( Hu et al., 2023 )	Alibaba	1.8B–72B	2023	Decoder-only	Multilingual, Code

Table 1: Overview of major open-weight foundation models commonly used in model merging research. These models serve as the base for fine-tuning and subsequent merging experiments.

Panel of Figure 1 — Figure 1: Overview of the model merging pipeline. Task-specific models $\{\theta_{1},\dots,\theta_{K}\}$ , obtained by fine-tuning a shared pretrained checkpoint $\theta_{0}$ , are combined via one of several strategies (Sections 3–5) into a single model $\theta_{\text{merged}}$ , which is then evaluated on all source tasks. Gray vertical bars represent the many-to-many mapping between model inputs and merging strategies.

Panel of Figure 2 — Figure 2: The FUSE taxonomy. F oundations (§ 2 ): why merging works. U nification (§ 3 –§ 5 ): how models are combined. S cenarios (§ 6 ): where merging applies. E cosystem (§ 7 ): what supports deployment.

Theoretical Concept	Key Approach	Key Feature(s)	Limitations/Notes	Year
Loss Surface Geometry ( Singh et al., 2024 )	Analyzes geometric properties of high-dimensional parameter space	Explains why weight interpolation yields functional networks	Complex landscapes make complete characterization difficult	2024
Linear Mode Connectivity ( Frankle et al., 2020 )	Establishes that models from shared initialization are connected by low-loss linear paths	Central to explaining success of weight interpolation methods	Requires shared initialization for strong connectivity	2020
Permutation Invariance ( Ainsworth et al., 2023 )	Exploits symmetry from reordering hidden units with weight adjustments	Multiple equivalent parameterizations for same function	Alignment required before merging independently trained models	2023
Weight Space Symmetries ( Ainsworth et al., 2023 )	Analyzes fundamental symmetry properties affecting merging feasibility	Enables functionally equivalent network representations	Symmetry breaking needed for meaningful comparisons	2023
Mergeability Conditions ( Singh et al., 2024 )	Establishes unified framework of prerequisites for successful merging	Predictive criteria for merge quality assessment	Conditions may vary across model architectures and tasks	2024
Shared Initialization Theory ( Frankle et al., 2020 )	Explains role of common pretrained weights in enabling merging	Strong predictor of linear mode connectivity	Theoretical mechanisms not fully understood for LLMs	2020

Table 2: Theoretical foundations underlying model merging success. Each row summarizes a key theoretical concept, its mathematical basis, its implications for merging, and relevant seminal references. Together, these concepts explain when and why weight-space combination of fine-tuned models preserves or enhances performance.

Method	Key Approach	Key Feature(s)	Limitations/Notes	Year
Uniform Averaging (Model Soups)	Element-wise arithmetic mean: $\theta_{\text{merged}}=\frac{1}{N}\sum_{i=1}^{N}\theta_{i}$	Simple; no hyperparameters; treats weight space as Euclidean	Assumes equal parameter importance; suboptimal when models diverge substantially	2022
Fisher-Weighted Averaging	Importance weighting via Fisher Information matrix	Accounts for heterogeneous parameter importance; preserves critical task-specific weights	Computationally expensive; requires data for Fisher estimation	2022
Stochastic Weight Averaging (SWA) ( Izmailov et al., 2018 )	Average checkpoints along optimization trajectory	Exploits temporal structure; finds flatter minima; improves generalization	Limited to single training run; requires checkpoint storage	2018
SLERP	Spherical linear interpolation on hypersphere	Preserves weight vector magnitude; avoids norm shrinkage in linear interpolation	Limited to pairwise merging; requires careful normalization	2023
Latent Space Merging	Interpolation in activation/latent space rather than weights	Bypasses weight space geometry issues; often outperforms weight averaging	May require data access; increased inference complexity	2024
Manifold-Aware Interpolation	Geodesic paths on parameter manifolds with permutation alignment	Respects non-Euclidean geometry; handles mode connectivity	Computationally intensive; requires solving alignment problem	2023

Table 3: Weight-space averaging and geometric interpolation methods for model merging. Methods range from simple uniform averaging to manifold-aware interpolation, progressively incorporating parameter importance (Fisher), temporal structure (SWA), and geometric properties (SLERP, manifold) to improve merge quality. “Key Approach” summarizes the merging mechanism; “Limitations” notes the primary constraint for each method.

Panel of Figure 3 — Figure 3: Four fundamental task vector operations. (a) Addition combines multiple task vectors into a single model. (b) Negation subtracts a task vector to remove unwanted behavior. (c) Scaling controls task influence via $\lambda$ . (d) Analogy transfers the relationship between two tasks.

Method	Key Approach	Key Feature(s)	Limitations/Notes	Year
Task Arithmetic ( Ilharco et al., 2023 )	Linear task vector combination	Defines task vectors as weight differences; enables addition, negation, scaling operations	Suffers from parameter interference and sign conflicts	2023
TIES-Merging ( Yadav et al., 2023 )	Trim, Elect Sign, Merge	Resolves sign conflicts via magnitude-based trimming and democratic sign election	Requires hyperparameter tuning for sparsity threshold	2023
DARE	Drop and Rescale	Randomly drops delta parameters and rescales remaining ones to preserve expectations	Stochastic nature may introduce variance	2023
CABS	Conflict-Aware Balanced Sparsification	Explicitly identifies and eliminates conflicting parameters while maintaining balance	Limited to identical architectures	2025
Tangent Space Methods	Tangent Space Operations	Reformulates task arithmetic in tangent space for improved linearity assumptions	Increased mathematical complexity	2024

Table 4: Task vector arithmetic and sparsification-enhanced methods for model merging. Each method addresses a specific aspect of parameter interference: sign conflicts (TIES-Merging), redundant updates (DARE), conflict-aware balancing (CABS), or tangent space reformulation. The “Interference Handling” column indicates whether the method explicitly resolves parameter conflicts between task vectors.

Method	Category	Data-Free	Interference Handling	Shared Init	Scalability	Year
Weight-Space Averaging (Section 3)
Simple Averaging	Static	✓	✗	✓	High	–
Model Soups ( Wortsman et al., 2022a )	Greedy selection	✓	✗	✓	High	2022
Fisher Merging ( Matena and Raffel, 2022 )	Importance-weighted	✗	Partial	✓	Medium	2022
RegMean ( Jin et al., 2023 )	Linear regression	✗	Partial	✓	Medium	2023
SLERP	Geometric	✓	✗	✓	High	2023

Table 5: Unified comparison of representative model merging methods. Data-Free : no training/calibration data required. Interference : explicit handling of parameter conflicts. Shared Init : requires common pretrained initialization.

Method	Expert Type	Memory	Router Training	Year	Key Innovation
Full-Model Expert Methods
Switch Transformers ( Fedus et al., 2022 )	Dense FFN	$K\times$ FFN	From scratch	2022	Simplified MoE with single expert routing
Branch-Train-MiX ( Sukhbaatar et al., 2024 )	Full model	$K\times$	Post-hoc	2024	Branched training for expert specialization
PHATGOOSE ( Muqeeth et al., 2024a )	Full model	$K\times$	Per-expert	2024	Post-hoc gating without joint training
LoRA-Based Expert Methods
LoRAHub ( Huang et al., 2023 )	LoRA adapter	$K\times$ LoRA	Few-shot	2023	Dynamic composition via gradient-free search

Table 6: Comparison of MoE-style and LoRA-based composition methods for model merging. Memory overhead is relative to base model size.

Search Paradigm	Sample Efficiency	Parallelizability	Variable Handling	Best Use Case
Evolutionary Algorithms	Low	High	Excellent	Novel layer architectures
Bayesian Optimization	High	Low	Limited	Fine-tuning coefficients
Random Search	Very Low	Very High	Good	Baseline benchmarking

Table 7: Search paradigms for automated model merging configuration discovery. Evolutionary algorithms excel at exploring discrete architectural choices (e.g., layer permutation) but require many fitness evaluations; Bayesian optimization is more sample-efficient for continuous coefficient tuning; random search serves as a scalable baseline. “Variable Handling” indicates support for mixed discrete-continuous search spaces.

Application Scenario	Key Approach	Key Feature(s)	Limitations/Notes	Year
Multi-Task Merging	Synthesize specialized fine-tuned models into unified system	Enhanced multi-task generalization; computationally efficient	Potential task interference; requires careful weight balancing	2023
Multilingual Merging	Combine language-specific models	Cross-lingual capability transfer; unified multilingual system	Language imbalance; varying quality across languages	2023
RLHF Alignment	Merge preference-aligned models trained via reinforcement learning	Improved human value alignment; safety enhancement	High computational cost; reward hacking risks	2022
DPO Alignment	Direct preference optimization without reward modeling	Simplified training pipeline; stable optimization	Limited to pairwise preferences; distribution shift issues	2023
Federated Averaging	Aggregate locally fine-tuned models without data sharing	Privacy preservation; distributed computation	Communication overhead; non-IID data challenges	2017
Multi-Benchmark Evaluation	Simultaneous assessment across diverse task suites	Capability retention measurement; interference quantification	Benchmark selection bias; limited emergent property capture	2024

Table 8: Application scenarios for model merging in LLM development. Each scenario is categorized by its primary motivation (capability augmentation, alignment/safety, or efficiency), key merging approach, distinctive features, and known limitations. Year indicates when the approach was first applied in the LLM context.

Panel of Figure 4 — Figure 4: Evolution of model merging techniques. The field progressed from foundational loss landscape theory (2018–2020), through weight-space averaging (2022) and interference-aware task vectors (2023), to adaptive/evolutionary optimization and standardized ecosystem tooling (2024–2025). Arrows indicate the conceptual flow and inheritance of ideas across phases.

Language Model Merging

Language model merging combines capabilities from multiple models or adapters by composing their weights, updates, or outputs.

See all 106 papers

Jul 13, 2026

Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs

Multi-task model merging combines separately trained expert models into a single model that handles all tasks without co-training. Standard practice merges experts at their optimal validation loss. We challenge this convention by systematically studying how training duration of domain experts affects the quality of the merged model. We fine-tune experts on five domains (Math, Code, Instruction Following, Multilingual, and Safety) across three model sizes (Qwen 3.5 0.8B, 2B, and 4B), saving checkpoints from 25% to 500% of the optimal training steps and evaluating five merging methods at each duration. Our findings reveal a striking method-dependent pattern: simple averaging degrades sharply with overfitting, while sparsification-based methods achieve their best performance well past the validation optimum. We formalize this through bias-variance decomposition analysis, drawing a parallel to random forests where averaging benefits from high-variance individual learners. These results suggest that training duration and merging method should be chosen jointly rather than independently.

Nikita Kozodoi, Zainab Afolabi, Jack Butler

1Amazon Web Services

LLM Fine-Tuning Language Model Merging Model Merging

Ask PDF

May 26, 2026

Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training

Model merging has emerged as a lightweight paradigm for enhancing Large Language Models (LLMs), yet its underlying mechanisms remain poorly understood. In this work, we analyze late-stage pre-training trajectories and uncover a Rank-1 Subspace phenomenon: while raw optimization steps oscillate violently, consecutive merged checkpoints collapse onto a stable, approximately one-dimensional linear manifold. We theoretically ground this observation in a river-valley landscape analysis: averaging acts as a geometric low-pass filter that dampens high-curvature noise to reveal the optimal descent direction. Capitalizing on this insight, we propose Extra-Merge, a training-free strategy that extrapolates along this subspace to minimize loss without additional gradient updates. Extensive experiments across GPT-2 and LLaMA families (124M to 2B) demonstrate that Extra-Merge consistently outperforms standard merging baselines. Notably, it yields consistent zero-shot accuracy gains on Pythia-12B downstream tasks and generalizes effectively to the Muon optimizer \citep{jordan2024muon}.

Wenjie Zhou, Bohan Wang, Hongtao Zhang +3

State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China · Alibaba Group, China +2

Language Model Pretraining Neural Network Optimization Weight-Space Analysis Language Model Merging

Ask PDF

Sep 6, 2026

CAMFT: Conflict-Aware Mergeable Fine-Tuning for Large Language Models

Model merging has emerged as a promising paradigm for integrating multiple task-specific capabilities into a single large language model. However, existing methods predominantly focus on post-hoc processing of independently fine-tuned models, overlooking how the training phase itself impacts cross-task compatibility. Resolving parameter conflicts after fine-tuning is inherently sub-optimal. To address this, we propose CAMFT, a Conflict-Aware Mergeable Fine-Tuning method that makes task adaptation both efficient and mergeaware. CAMFT treats mergeability as a property shaped during fine-tuning, rather than only a problem to be solved after fine-tuning. By guiding each task to update sparse coordinates with lower cross-task conflict, CAMFT produces task updates that are efficient to train and more compatible for downstream model merging. Extensive experiments demonstrate that CAMFT outperforms standard finetuning baselines in multi-task merging scenarios. Codes are available at https://github.com/gyanchow/CAMFT-LLM.

Jingang Zhou, Haiyang Guo, Yuan Ma +2

School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences

LLM Fine-Tuning Language Model Merging Fine-Tuning

Ask PDF

Abstract

Language Model Merging

Explore similar work

Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs

Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training

CAMFT: Conflict-Aware Mergeable Fine-Tuning for Large Language Models