Model Merging

Ask about this topic

Overview

Model merging studies how to combine weights or adapters from separately trained models into one model, aiming to reuse complementary skills without storing an ensemble or repeating joint training. Common approaches average checkpoints or add task vectors, while sparsification, low-rank and spectral methods, geometric alignment, and calibration-based routing aim to reduce interference. The methods are used for multi-task and continual learning, domain adaptation, and combining language, vision, speech, and specialized models; evaluations often measure task accuracy, capability retention, worst-task performance, and inference or storage cost. The central tension is that updates that help one task can erase or distort another: success depends on shared initialization, representation compatibility, update scale, and data or prompt distribution, and some task combinations may not be recoverable through weight-space fusion alone.

Latest papers

CardsList

192 papers31 papers in the last four weeks, up 63% on the four weeks before. 0.2% of all new papers.

  1. Model Merging via Data-Free Covariance Estimation

    Apr 1, 2026Marawan Gamal Abdel Hameed, Derek Tam, Pascal Jr Tikeng Notsawo +2Multi-Task LearningLanguage Model Merging

    Mila & DIRO, Universit´e de Montr´eal · University of Toronto & Vector Institute

  2. Training-Free Dynamic Upcycling of Expert Language Models

    Mar 31, 2026Eros Fanì, Oğuzhan ErsoyMixture-of-Experts Language ModelsEfficient Language Model Training

    Gensyn

  3. Dual-Imbalance Continual Learning for Real-World Food Recognition

    Mar 31, 2026Xiaoyan Zhang, Jiangpeng HeClass-Incremental LearningIncremental Learning

    University of Michigan, Ann Arbor, Michigan, U.S.A · Indiana University, Bloomington, Indiana, U.S.A

  4. OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training

    Mar 30, 2026Haiyue Song, Masao UtiyamaContinued PretrainingData Mixture Optimization

    National Institute of Information and Communications Technology, Kyoto, Japan

  5. Decompose, Mix, Adapt: A Unified Framework for Parameter-Efficient Neural Network Recombination and Compression

    Mar 28, 2026Nazia Tasnim, Shrimai Prabhumoye, Bryan A. PlummerNeural Network CompressionParameter-Efficient Fine-Tuning

    Boston University · NVIDIA, Boston University

  6. Label-Free Cross-Task LoRA Merging with Null-Space Compression

    Mar 27, 2026Wonyoung Lee, Wooseong Jeong, Kuk-Jin YoonModel MergingLanguage Model Merging

    KAIST · *Equal contribution to this work.

  7. Fast-dVLA: Accelerating Discrete Diffusion VLA to Real-Time Performance

    Mar 26, 2026Wenxuan Song, Jiayi Chen, Shuai Chen +8VLA Model TrainingVision-Language-Action Models

    The Hong Kong University of Science and Technology (Guangzhou) · ShanghaiTech University · Shanghai Institute of Technical Physics, CAS +3

  8. Data-Free Layer-Adaptive Merging via Fisher Information for Long-to-Short Reasoning LLMs

    Mar 23, 2026Tian XiaLanguage Model MergingModel Merging

  9. DiffGraph: An Automated Agent-driven Model Merging Framework for In-the-Wild Text-to-Image Generation

    Mar 20, 2026Zhuoling Li, Hossein Rahmani, Jiarui Zhang +5Model MergingT2I Generation

    Lancaster University · University of Bristol · Adobe Research

  10. VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Models

    Mar 18, 2026Hefei Xu, Le Wu, Yu Wang +4LLM AlignmentMulti-Objective Language Model Alignment

    Key Laboratory of Knowledge Engineering with Big Data, Hefei University of Technology, Hefei, Anhui, China · Innovation School of Artificial Intelligence, Hefei University of Technology

  11. Learning Not To Interfere: Adapting Experts For Model Merging

    Mar 13, 2026Pratik Ramesh, George Stoica, Arun Iyer +2Model Merging

    Georgia Tech · University of Washington · Microsoft Research +3

  12. Sparse Task Vector Mixup with Hypernetworks for Efficient Knowledge Transfer in Whole-Slide Image Prognosis

    Mar 11, 2026Pei Liu, Xiangxiang Zeng, Tengfei Ma +3Whole-Slide Image AnalysisCancer Survival Prediction

    College of Computer Science and Electronic Engineering, Hunan University · National University of Singapore

  13. One Adapter for All: Towards Unified Representation in Step-Imbalanced Class-Incremental Learning

    Mar 10, 2026Xiaoyan Zhang, Jiangpeng HeAdapter-Based Continual LearningClass-Incremental Learning

    University of Michigan, Ann Arbor, Michigan, U.S.A · Indiana University, Bloomington, Indiana, U.S.A

  14. Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions

    Mar 10, 2026Mingyang Song, Mao ZhengLanguage Model MergingModel Merging

    Large Language Model Department, Tencent, China

  15. An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapse

    Mar 10, 2026Yuan Cao, Dezhi Ran, Yuzhe Guo +5Language Model MergingModel Merging

    Key Lab of HCST (PKU), MOE; SCS, Peking University, Beijing, China · Computer Science Department at Columbia University, New York, USA · School of Computing Science, Simon Fraser University, Burnaby, BC, Canada +1

  16. DC-Merge: Improving Model Merging with Directional Consistency

    Mar 6, 2026Han-Chen Zhang, Zi-Hao Zhou, Mao-Lin Luo +3Model MergingTask Vector Merging

    School of Computer Science and Engineering, Southeast University, Nanjing 210096, China

  17. Domain-Adaptive Model Merging Across Disconnected Modes

    Mar 6, 2026Junming Liu, Yusen Zhang, Rongchao Zhang +2Model MergingCross-Domain Transfer Learning

    Tongji University · Peking University · Southeast University +1

  18. Bridging Domains through Subspace-Aware Model Merging

    Mar 6, 2026Levy Chaves, Chao Zhou, Rebekka Burkholz +2Model MergingCross-Domain Generalization

    Universidade Estadual de Campinas (UNICAMP), Recod.ai Lab., Instituto de Computac¸˜ao, Brasil · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany · Intercom

  19. Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASR

    Mar 5, 2026Carlos Carvalho, Francisco Teixeira, Thomas Rolland +1Model MergingDomain Adaptation for ASR

    INESC-ID · Instituto Superior Técnico, Universidade de Lisboa, Portugal

  20. Functionality-Oriented LLM Merging on the Fisher--Rao Manifold

    Mar 5, 2026Jiayu Wang, Zuojun Ye, Wenpeng YinLanguage Model MergingModel Merging

    Pennsylvania State University · Independent Developer

  21. BD-Merging: Bias-Aware Dynamic Model Merging with Evidence-Guided Contrastive Learning

    Mar 4, 2026Yuhan Xie, Chen LyuModel MergingDistribution Shift Robustness

    MoE Key Laboratory of Interdisciplinary Research of Computation and Economics, Shanghai University of Finance and Economics

  22. ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimation

    Mar 3, 2026Bo Xu, Haotian Wu, Hehai Lin +4Model MergingLanguage Model Merging

    The Hong Kong University of Science and Technology (Guangzhou) · University of Science and Technology of China

  23. Securing the Floor and Raising the Ceiling: A Merging-based Paradigm for Multi-modal Search Agents

    Mar 2, 2026Zhixiang Wang, Jingxuan Xu, Dajun Chen +3Multimodal Search AgentsModel Merging

    Ant Group, Hangzhou, China · Peking University, Beijing, China

  24. GeNeX: Genetic Network eXperts framework for addressing Validation Overfitting

    Feb 28, 2026Emmanuel Pintelas, Ioannis E. LivierisEnsemble LearningModel Merging

    Department of Statistics & Insurance Science, University of Piraeus, GR 18534, Greece · Department of Business & Organization Administration, University of Peloponnese, GR 24100, Greece

  25. Open-Vocabulary Semantic Segmentation in Remote Sensing via Hierarchical Attention Masking and Model Composition

    Feb 27, 2026Mohammadreza Heidarianbaei, Mareike Dorozynski, Hubert Kanyamahanga +2Open-Vocabulary Semantic SegmentationRemote Sensing Image Segmentation

    Institute of Photogrammetry and GeoInformation Leibniz University Hannover Germany

  26. The Mean is the Mirage: Entropy-Adaptive Model Merging under Heterogeneous Domain Shifts in Medical Imaging

    Feb 24, 2026Sameer Ambekar, Reza Nasirigerdeh, Peter J. Schuffler +3Model MergingDomain Generalization

    School of Computation, Information and Technology, Technical University of Munich, Germany · Institute of Machine Learning in Biomedical Imaging, Helmholtz Munich, Germany · relAI – Konrad Zuse School of Excellence in Reliable AI +3

  27. Model Merging in the Essential Subspace

    Feb 23, 2026Longhua Li, Lei Qi, Qi Tian +1Task InterferenceModel Merging

    School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shanghai, China

  28. Training-Free Cross-Architecture Merging for Graph Neural Networks

    Feb 22, 2026Rishabh Bhattacharya, Vikaskumar Kalsariya, Naresh ManwaniModel MergingGraph Neural Networks

    International Institute of Information Technology, Hyderabad, India

  29. Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature

    Feb 19, 2026Angelo Porrello, Pietro Buzzega, Felix Dangel +4Kronecker-Factored ApproximationTask Arithmetic

    University of Modena and Reggio Emilia, Italy · Vector Institute, Toronto

  30. Dynamic Training-Free Fusion of Subject and Style LoRAs

    Feb 17, 2026Qinglong Cao, Yuntian Chen, Chao Ma +1Diffusion-Based Image GenerationModel Merging

    Shanghai Jiao Tong University · Eastern Institute of Technology, Ningbo