cs.CLMar 10, 2026
SaveModel Merging in the Era of Large Language Models: Methods, Applications, and Future Directions
Organizations: Large Language Model Department, Tencent, China
Abstract
Model merging combines the parameters of multiple neural networks into a single model without additional training. As fine-tuned large language models (LLMs) proliferate, merging offers a computationally efficient alternative to ensembles and full retraining, enabling practitioners to compose specialized capabilities at minimal cost. This survey examines model merging in the LLM era through the FUSE taxonomy, organized along Foundations, Unification Strategies, Scenarios, and Ecosystem. We first establish the theoretical underpinnings of merging, including loss landscape geometry and mode connectivity, then systematically review the algorithmic space spanning weight averaging, task vector arithmetic, sparsification-enhanced methods, mixture-of-experts architectures, and evolutionary optimization. We further examine downstream applications across multi-task learning, safety alignment, domain specialization, and federated learning, and survey the supporting ecosystem of tools and evaluation benchmarks. Finally, we identify key open challenges and future directions, aiming to equip researchers and practitioners with a structured foundation for advancing model merging.
Figures & tables
| Model Family | Organization | Parameters | Year | Architecture | Key Features |
| LLaMA ( Touvron et al., 2023a ) | Meta | 7B–65B | 2023 | Decoder-only | RoPE, Pre-norm, SwiGLU |
| LLaMA-2 ( Touvron et al., 2023b ) | Meta | 7B–70B | 2023 | Decoder-only | GQA, Extended context |
| LLaMA-3 ( Team, 2024b ) | Meta | 8B–405B | 2024 | Decoder-only | 128K context, GQA |
| Mistral ( Jiang et al., 2023 ) | Mistral AI | 7B | 2023 | Decoder-only | Sliding window attention |
| Mixtral ( Jiang et al., 2024 ) | Mistral AI | 8 7B | 2024 | Sparse MoE | 8 experts, top-2 routing |
| Qwen ( Hu et al., 2023 ) | Alibaba | 1.8B–72B | 2023 | Decoder-only | Multilingual, Code |
Table 1: Overview of major open-weight foundation models commonly used in model merging research. These models serve as the base for fine-tuning and subsequent merging experiments.
Figure 1: Overview of the model merging pipeline. Task-specific models , obtained by fine-tuning a shared pretrained checkpoint , are combined via one of several strategies (Sections 3–5) into a single model , which is then evaluated on all source tasks. Gray vertical bars represent the many-to-many mapping between model inputs and merging strategies.
Figure 2: The FUSE taxonomy. F oundations (§ 2 ): why merging works. U nification (§ 3 –§ 5 ): how models are combined. S cenarios (§ 6 ): where merging applies. E cosystem (§ 7 ): what supports deployment.
| Theoretical Concept | Key Approach | Key Feature(s) | Limitations/Notes | Year |
|---|---|---|---|---|
| Loss Surface Geometry ( Singh et al., 2024 ) | Analyzes geometric properties of high-dimensional parameter space | Explains why weight interpolation yields functional networks | Complex landscapes make complete characterization difficult | 2024 |
| Linear Mode Connectivity ( Frankle et al., 2020 ) | Establishes that models from shared initialization are connected by low-loss linear paths | Central to explaining success of weight interpolation methods | Requires shared initialization for strong connectivity | 2020 |
| Permutation Invariance ( Ainsworth et al., 2023 ) | Exploits symmetry from reordering hidden units with weight adjustments | Multiple equivalent parameterizations for same function | Alignment required before merging independently trained models | 2023 |
| Weight Space Symmetries ( Ainsworth et al., 2023 ) | Analyzes fundamental symmetry properties affecting merging feasibility | Enables functionally equivalent network representations | Symmetry breaking needed for meaningful comparisons | 2023 |
| Mergeability Conditions ( Singh et al., 2024 ) | Establishes unified framework of prerequisites for successful merging | Predictive criteria for merge quality assessment | Conditions may vary across model architectures and tasks | 2024 |
| Shared Initialization Theory ( Frankle et al., 2020 ) | Explains role of common pretrained weights in enabling merging | Strong predictor of linear mode connectivity | Theoretical mechanisms not fully understood for LLMs | 2020 |
Table 2: Theoretical foundations underlying model merging success. Each row summarizes a key theoretical concept, its mathematical basis, its implications for merging, and relevant seminal references. Together, these concepts explain when and why weight-space combination of fine-tuned models preserves or enhances performance.
| Method | Key Approach | Key Feature(s) | Limitations/Notes | Year |
|---|---|---|---|---|
| Uniform Averaging (Model Soups) | Element-wise arithmetic mean: | Simple; no hyperparameters; treats weight space as Euclidean | Assumes equal parameter importance; suboptimal when models diverge substantially | 2022 |
| Fisher-Weighted Averaging | Importance weighting via Fisher Information matrix | Accounts for heterogeneous parameter importance; preserves critical task-specific weights | Computationally expensive; requires data for Fisher estimation | 2022 |
| Stochastic Weight Averaging (SWA) ( Izmailov et al., 2018 ) | Average checkpoints along optimization trajectory | Exploits temporal structure; finds flatter minima; improves generalization | Limited to single training run; requires checkpoint storage | 2018 |
| SLERP | Spherical linear interpolation on hypersphere | Preserves weight vector magnitude; avoids norm shrinkage in linear interpolation | Limited to pairwise merging; requires careful normalization | 2023 |
| Latent Space Merging | Interpolation in activation/latent space rather than weights | Bypasses weight space geometry issues; often outperforms weight averaging | May require data access; increased inference complexity | 2024 |
| Manifold-Aware Interpolation | Geodesic paths on parameter manifolds with permutation alignment | Respects non-Euclidean geometry; handles mode connectivity | Computationally intensive; requires solving alignment problem | 2023 |
Table 3: Weight-space averaging and geometric interpolation methods for model merging. Methods range from simple uniform averaging to manifold-aware interpolation, progressively incorporating parameter importance (Fisher), temporal structure (SWA), and geometric properties (SLERP, manifold) to improve merge quality. “Key Approach” summarizes the merging mechanism; “Limitations” notes the primary constraint for each method.
Figure 3: Four fundamental task vector operations. (a) Addition combines multiple task vectors into a single model. (b) Negation subtracts a task vector to remove unwanted behavior. (c) Scaling controls task influence via . (d) Analogy transfers the relationship between two tasks.
| Method | Key Approach | Key Feature(s) | Limitations/Notes | Year |
|---|---|---|---|---|
| Task Arithmetic ( Ilharco et al., 2023 ) | Linear task vector combination | Defines task vectors as weight differences; enables addition, negation, scaling operations | Suffers from parameter interference and sign conflicts | 2023 |
| TIES-Merging ( Yadav et al., 2023 ) | Trim, Elect Sign, Merge | Resolves sign conflicts via magnitude-based trimming and democratic sign election | Requires hyperparameter tuning for sparsity threshold | 2023 |
| DARE | Drop and Rescale | Randomly drops delta parameters and rescales remaining ones to preserve expectations | Stochastic nature may introduce variance | 2023 |
| CABS | Conflict-Aware Balanced Sparsification | Explicitly identifies and eliminates conflicting parameters while maintaining balance | Limited to identical architectures | 2025 |
| Tangent Space Methods | Tangent Space Operations | Reformulates task arithmetic in tangent space for improved linearity assumptions | Increased mathematical complexity | 2024 |
Table 4: Task vector arithmetic and sparsification-enhanced methods for model merging. Each method addresses a specific aspect of parameter interference: sign conflicts (TIES-Merging), redundant updates (DARE), conflict-aware balancing (CABS), or tangent space reformulation. The “Interference Handling” column indicates whether the method explicitly resolves parameter conflicts between task vectors.
| Method | Category | Data-Free | Interference Handling | Shared Init | Scalability | Year |
| Weight-Space Averaging (Section 3) | ||||||
| Simple Averaging | Static | ✓ | ✗ | ✓ | High | – |
| Model Soups ( Wortsman et al., 2022a ) | Greedy selection | ✓ | ✗ | ✓ | High | 2022 |
| Fisher Merging ( Matena and Raffel, 2022 ) | Importance-weighted | ✗ | Partial | ✓ | Medium | 2022 |
| RegMean ( Jin et al., 2023 ) | Linear regression | ✗ | Partial | ✓ | Medium | 2023 |
| SLERP | Geometric | ✓ | ✗ | ✓ | High | 2023 |
Table 5: Unified comparison of representative model merging methods. Data-Free : no training/calibration data required. Interference : explicit handling of parameter conflicts. Shared Init : requires common pretrained initialization.
| Method | Expert Type | Memory | Router Training | Year | Key Innovation |
| Full-Model Expert Methods | |||||
| Switch Transformers ( Fedus et al., 2022 ) | Dense FFN | FFN | From scratch | 2022 | Simplified MoE with single expert routing |
| Branch-Train-MiX ( Sukhbaatar et al., 2024 ) | Full model | Post-hoc | 2024 | Branched training for expert specialization | |
| PHATGOOSE ( Muqeeth et al., 2024a ) | Full model | Per-expert | 2024 | Post-hoc gating without joint training | |
| LoRA-Based Expert Methods | |||||
| LoRAHub ( Huang et al., 2023 ) | LoRA adapter | LoRA | Few-shot | 2023 | Dynamic composition via gradient-free search |
Table 6: Comparison of MoE-style and LoRA-based composition methods for model merging. Memory overhead is relative to base model size.
| Search Paradigm | Sample Efficiency | Parallelizability | Variable Handling | Best Use Case |
|---|---|---|---|---|
| Evolutionary Algorithms | Low | High | Excellent | Novel layer architectures |
| Bayesian Optimization | High | Low | Limited | Fine-tuning coefficients |
| Random Search | Very Low | Very High | Good | Baseline benchmarking |
Table 7: Search paradigms for automated model merging configuration discovery. Evolutionary algorithms excel at exploring discrete architectural choices (e.g., layer permutation) but require many fitness evaluations; Bayesian optimization is more sample-efficient for continuous coefficient tuning; random search serves as a scalable baseline. “Variable Handling” indicates support for mixed discrete-continuous search spaces.
| Application Scenario | Key Approach | Key Feature(s) | Limitations/Notes | Year |
|---|---|---|---|---|
| Multi-Task Merging | Synthesize specialized fine-tuned models into unified system | Enhanced multi-task generalization; computationally efficient | Potential task interference; requires careful weight balancing | 2023 |
| Multilingual Merging | Combine language-specific models | Cross-lingual capability transfer; unified multilingual system | Language imbalance; varying quality across languages | 2023 |
| RLHF Alignment | Merge preference-aligned models trained via reinforcement learning | Improved human value alignment; safety enhancement | High computational cost; reward hacking risks | 2022 |
| DPO Alignment | Direct preference optimization without reward modeling | Simplified training pipeline; stable optimization | Limited to pairwise preferences; distribution shift issues | 2023 |
| Federated Averaging | Aggregate locally fine-tuned models without data sharing | Privacy preservation; distributed computation | Communication overhead; non-IID data challenges | 2017 |
| Multi-Benchmark Evaluation | Simultaneous assessment across diverse task suites | Capability retention measurement; interference quantification | Benchmark selection bias; limited emergent property capture | 2024 |
Table 8: Application scenarios for model merging in LLM development. Each scenario is categorized by its primary motivation (capability augmentation, alignment/safety, or efficiency), key merging approach, distinctive features, and known limitations. Year indicates when the approach was first applied in the LLM context.
Figure 4: Evolution of model merging techniques. The field progressed from foundational loss landscape theory (2018–2020), through weight-space averaging (2022) and interference-aware task vectors (2023), to adaptive/evolutionary optimization and standardized ecosystem tooling (2024–2025). Arrows indicate the conceptual flow and inheritance of ideas across phases.
More in this topic
Language Model Merging
Language model merging combines capabilities from multiple models or adapters by composing their weights, updates, or outputs.
See all 106 papers