Language Model Merging

Ask about this topic

Overview

Language model merging studies how to combine specialized models—such as instruction-following, reasoning, multilingual, or domain-specific models—into a model that retains useful capabilities without always retraining on all the original data. Common approaches average weights or task vectors, sparsify or align conflicting updates, use Fisher or curvature information to set merge weights, and search for combinations with Bayesian or evolutionary optimization; some methods instead use routing or output-level fusion. Researchers evaluate these techniques on language, reasoning, code, multilingual, and safety benchmarks, often measuring both capability gains and forgetting of general skills. A central challenge is that compatibility varies by model pair and task: methods that work for related fine-tunes may fail for heterogeneous models, and preserving safety, calibration, and broad capability while adding new skills remains difficult.

Latest papers

CardsList

110 papers20 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

  1. Data-Free Layer-Adaptive Merging via Fisher Information for Long-to-Short Reasoning LLMs

    Mar 23, 2026Tian XiaLanguage Model MergingModel Merging

  2. SSAM: Singular Subspace Alignment for Merging Multimodal Large Language Models

    Mar 23, 2026Md Kaykobad Reza, Ameya Patil, Edward Ayrapetian +1Multimodal Large Language ModelsLanguage Model Merging

    University of California Riverside · Amazon

  3. ES-Merging: Biological MLLM Merging via Embedding Space Signals

    Mar 15, 2026Wonbin Lee, Dongki Kim, Sung Ju HwangLanguage Model MergingMultimodal Large Language Models

    KAIST

  4. Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions

    Mar 10, 2026Mingyang Song, Mao ZhengLanguage Model MergingModel Merging

    Large Language Model Department, Tencent, China

  5. An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapse

    Mar 10, 2026Yuan Cao, Dezhi Ran, Yuzhe Guo +5Language Model MergingModel Merging

    Key Lab of HCST (PKU), MOE; SCS, Peking University, Beijing, China · Computer Science Department at Columbia University, New York, USA · School of Computing Science, Simon Fraser University, Burnaby, BC, Canada +1

  6. DC-Merge: Improving Model Merging with Directional Consistency

    Mar 6, 2026Han-Chen Zhang, Zi-Hao Zhou, Mao-Lin Luo +3Model MergingTask Vector Merging

    School of Computer Science and Engineering, Southeast University, Nanjing 210096, China

  7. Functionality-Oriented LLM Merging on the Fisher--Rao Manifold

    Mar 5, 2026Jiayu Wang, Zuojun Ye, Wenpeng YinLanguage Model MergingModel Merging

    Pennsylvania State University · Independent Developer

  8. Federated Heterogeneous Language Model Optimization for Hybrid Automatic Speech Recognition

    Mar 5, 2026Mengze Hong, Yi Gu, Di Jiang +4Language Model MergingAutomatic Speech Recognition

    Hong Kong Polytechnic University · Ant Group · Shanghai Jiao Tong University +2

  9. Trade-offs in Ensembling, Merging and Routing Among Parameter-Efficient Experts

    Mar 3, 2026Sanae Lotfi, Lucas Caccia, Alessandro Sordoni +2Language Model EnsemblesLanguage Model Merging

    1New York University · 2Microsoft Research

  10. ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimation

    Mar 3, 2026Bo Xu, Haotian Wu, Hehai Lin +4Model MergingLanguage Model Merging

    The Hong Kong University of Science and Technology (Guangzhou) · University of Science and Technology of China

  11. RAIN-Merging: A Gradient-Free Method to Enhance Instruction Following in Large Reasoning Models with Preserved Thinking Format

    Feb 26, 2026Zhehao Huang, Yuhang Liu, Baijiong Lin +5Instruction FollowingLanguage Model Merging

    Institute of Image Processing and Pattern Recoginition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University · The Hong Kong University of Science and Technology (Guangzhou) · MoE Key Laboratory of System Control and Information Processing (Shanghai)

  12. The Appeal and Reality of Recycling LoRAs with Adaptive Merging

    Feb 12, 2026Haokun Liu, Gyung Hyun Je, Marco Ciccone +3LLM Fine-TuningLanguage Model Merging

    University of Toronto, Canada · Vector Institute, Toronto, Canada · Mistral AI (work done before joining) +2

  13. Exploring Information Seeking Agent Consolidation

    Jan 31, 2026Guochen Yan, Jialong Wu, Zhengwei Tao +8Language Model Merging

    1Peking University · 2Tencent

  14. AlignMerge - Alignment-Preserving Large Language Model Merging via Fisher-Guided Geometric Constraints

    Dec 18, 2025Aniruddha Roy, Jyoti Patel, Aman Chadha +2Language Model MergingLLM Alignment

    HCL · Evalueserve · Apple (USA) +2

  15. AP-BMM: Approximating Capability-Cost Pareto Sets of LLMs via Asynchronous Prior-Guided Bayesian Model Merging

    Dec 10, 2025Kesheng Chen, Yamin Hu, Zhenqian Zhu +2Bayesian OptimizationLanguage Model Merging

    Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies, Institute of Cyberspace Security, School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 518055, China.

  16. Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging

    Oct 20, 2025Tiancheng Hu, Benjamin Minixhofer, Nigel CollierLLM AlignmentModel Calibration

    University of Cambridge

  17. K-Merge: Online Continual Merging of Adapters for On-device Large Language Models

    Oct 15, 2025Donald Shenaj, Ondrej Bohdal, Taha Ceritli +3Continual Learning for LLMsOn-Device Language Model Inference

    Samsung R&D Institute UK · University of Pisa · University of Padova

  18. InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models

    May 20, 2025Yanggan Gu, Yuanyi Wang, Zhaoyi Yan +4Language Model MergingLLM Alignment

    The Hong Kong Polytechnic University (PolyU) · Zhejiang University · PolyU-Daya Bay Technology and Innovation Research Institute

  19. Exact Unlearning of Finetuning Data via Model Merging at Scale

    Apr 6, 2025Kevin Kuo, Amrith Setlur, Kartik Srinivas +2Machine UnlearningLLM Unlearning

    Computer Science Department · Machine Learning Department

  20. Variational Model Merging for Pareto Front Estimation in Multitask Finetuning

    Dec 11, 2024Hugo Monzón Maldonado, Nico Daheim, Thomas Möllenhoff +2Bayesian InferenceMulti-Task Learning

    RIKEN Center for AI Project, Japan · Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany · National Research Center for Applied Cybersecurity ATHENE, Germany