Language Model Merging

Ask about this topic

Overview

Language model merging studies how to combine specialized models—such as instruction-following, reasoning, multilingual, or domain-specific models—into a model that retains useful capabilities without always retraining on all the original data. Common approaches average weights or task vectors, sparsify or align conflicting updates, use Fisher or curvature information to set merge weights, and search for combinations with Bayesian or evolutionary optimization; some methods instead use routing or output-level fusion. Researchers evaluate these techniques on language, reasoning, code, multilingual, and safety benchmarks, often measuring both capability gains and forgetting of general skills. A central challenge is that compatibility varies by model pair and task: methods that work for related fine-tunes may fail for heterogeneous models, and preserving safety, calibration, and broad capability while adding new skills remains difficult.

Latest papers

CardsList

110 papers20 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

  1. Mostly Reading: What Open-Book Scores Measure in Document-Level LoRA Adapters

    Oct 22, 2026Martino M. L. Pulici, Cuong Xuan Chu, Evgeny Kharlamov +1Low-Rank AdaptationLanguage Model Merging

    Bosch Center for Artificial Intelligence, Germany · LMU Munich, Germany · Munich Center for Machine Learning, Germany

  2. GRAM: Group-wise Rank-Aware Model Merging via Subspace Alignment

    Oct 21, 2026Di Wu, Shuaidong PanLanguage Model MergingModel Merging

    JPMorgan Chase & Co.

  3. Inheritance as an Admission Test for LLM Prompt Merge Operators

    Oct 21, 2026Zan-Kai Chong, Hiroyuki Ohsaki, Bryan NgLanguage Model MergingPrompt Engineering

    School of Science and Technology, Kwansei Gakuin University, Sanda, Japan · School of Engineering & Computer Science, Victoria University of Wellington, Wellington, New Zealand

  4. A Fisher-Geometric View of Multi-Teacher On-Policy Distillation

    Oct 19, 2026Liu Kaiyuan, Zhuang Ziyuan, Ma Haoxuan +4Language Model MergingMulti-Teacher Knowledge Distillation

    Zhejiang University · Meituan LongCat · Nanjing University +1

  5. Controlling the Diversity-Quality Trade-Off of LLMs by Merging Models Before and After Alignment

    Oct 16, 2026Yasuaki Sumita, Tomoharu Iwata, Toshiyuki TanakaLanguage Model MergingDiverse Text Generation

    Graduate School of Informatics, Kyoto University, Kyoto, Japan · NTT Communication Science Laboratories, NTT, Inc., Kyoto, Japan

  6. To Align or Not To Align: Check Your COMPASS Before You Train

    Oct 13, 2026Fabian Gröger, Petar Damjanović, Maria BrbićLanguage Model MergingCross-Modal Alignment

    EPFL · University of Basel · HSLU

  7. SLEM: Spectrum-Aware Latent Expert Merging

    Oct 12, 2026Dohyeon Kim, Bedionita Soro, Bruno Andreis +1Language Model MergingMixture-of-Experts Language Models

    KAIST · University of Oxford · Slater Labs

  8. Beyond Adjacent Layers: Graph-Guided Layer Fusion for Compressing Large Language Models

    Oct 10, 2026Muhammad Usama, Yeoreum JungLLM CompressionLanguage Model Merging

    Braindeck Inc Seoul, Republic of Korea

  9. Towards Merging-conditional Backdoor Unalignment against Large Language Models

    Oct 10, 2026Yukun Chen, Yidou Chen, Yiming Li +4LLM Backdoor AttacksLLM Safety

    State Key Laboratory of Blockchain and Data Security, Zhejiang University · Tongyi Lab, Alibaba Group · Nanyang Technological University +1

  10. PGSB: Pretrained-Guided Shared Basis for LoRA Model Merging

    Oct 10, 2026Muqing Liu, Chongjie Si, Zhuoya Liu +1Language Model MergingLow-Rank Adaptation

    Chien-Shiung Wu College, Southeast University · MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University · School of Computer Science and Engineering, Southeast University +1

  11. ααTransfer: Coefficient Transfer for Efficient Model Merging

    Oct 6, 2026Shih-Cheng Huang, Zhi Rui Tam, Chieh-Yen Lin +3Hyperparameter TransferLanguage Model Merging

    Appier AI Research · National Taiwan University

  12. Training-Free Transformer Merging via Sequential Local Operator Alignment

    Oct 5, 2026Akansh Maurya, Ya-Wei Eileen Lin, Stefanie Jegelka +2Language Model MergingModel Merging

    CISPA Helmholtz Center for Information Security · Munich Center for Machine Learning · Technical University of Munich, School of Computation, Information and Technology +1

  13. BACAM: Behavior-Aware Continual Agent Merging for Multi-Turn Interaction

    Oct 4, 2026Shuaitong Li, Baochen Xiong, Xiaoshan Yang +4Language Model MergingContinual Learning for LLM Agents

    State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Pengcheng Laboratory, Shenzhen, China +2

  14. Predicting and Repairing Merge Collapse in Large Language Models

    Oct 2, 2026Jungseob Lee, Seungyoon Lee, Sugyeong Eo +3Language Model MergingTask Vector Merging

    Korea University · Yonsei University Mirae Campus · Sookmyung Women’s University +1

  15. Mixture-Trained Merging for Unified Multi-Objective Models

    Oct 1, 2026SeongHyeon Kim, Chaeyun Jang, Seungyoo Lee +4Multi-Task LearningLanguage Model Merging

    KAIST · Kakao

  16. SLIM: Simplex-Lattice Interpolation Merging

    Oct 1, 2026Seongcheol Jeong, Masahiro Suzuki, Yutaka MatsuoSurrogate-Assisted OptimizationLanguage Model Merging

    The University of Tokyo

  17. ChainLoRA: Geometry-Preserving Task Vector Merging for Continual Learning in LLMs

    Sep 30, 2026Hang Yin, Haozhe Wang, Yuhua Luo +3Continual Learning for LLMsLow-Rank Adaptation

    Shanghai Jiao Tong University · Shanghai Innovation Institute · Beijing Zhongguancun Academy +3

  18. Exploring Heterogeneous Model Merging Approach for Complex Knowledge Transfer

    Sep 30, 2026Jiahe Fan, Si Chen, Yinghao Hou +4Transfer LearningLanguage Model Merging

    University of Science and Technology of China

  19. The Missing Coefficients: Bayesian Pairwise Merging for Model Personalization

    Sep 30, 2026Yaling Shen, Tongtong Wu, Siyuan Yan +1Pairwise Preference LearningPersonalized Language Models

    Monash University

  20. QAM: Quadratic-Accurate Checkpoint Merging via Sequential Consistency

    Sep 28, 2026Shihao Wang, Rui Kong, Xinran Chen +7Language Model MergingModel Merging

    University of Wisconsin – Madison · Baidu Inc. · University of Arizona +3

  21. SMAT: Simple and Efficient Merge-Aware Training

    Sep 27, 2026Yanggan Gu, Yuanyi Wang, Zhen Li +5Language Model MergingModel Merging

    The Hong Kong Polytechnic University (PolyU) · The Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong +1

  22. Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs

    Sep 26, 2026Yuanyi Wang, Yanggan Gu, Su Lu +7Expert RoutingMixture-of-Experts Language Models

    The Hong Kong Polytechnic University, PolyU

  23. Train4Merge: RL or SFT---Which Yields Better Teachers for OPD-Based Model Merging?

    Sep 26, 2026Jingyuan Huang, Zuming Huang, Yucheng Shi +4Language Model MergingLanguage Model Distillation

    University of Georgia · INF Tech · Tencent +1

  24. Can One Adapted Model Do It All? Fine-Tuning Strategy Selection for Customer Support LLMs

    Sep 23, 2026Md Tahmid Rahman Laskar, Xue-Yong Fu, Shashi Bhushan TNFine-TuningCustomer Support Automation

    Dialpad Inc.

  25. On Emergent Capabilities and Model Merging

    Sep 21, 2026Luca Zhou, Emanuele RodolàEmergent Abilities in Language ModelsEmergent Misalignment in Language Models

    Sapienza University of Rome · Paradigma

  26. From Parameters to Behaviors: A Survey of Model Fusion for Large Language Models

    Sep 17, 2026Shuo Cai, Yanggan Gu, Zihao Wang +8Language Model MergingModel Merging

    The Hong Kong Polytechnic University · The Chinese University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou) +1

  27. Merging the Knowledge of LLMs for Automatic Speech Recognition

    Sep 14, 2026Hayato Futami, Tatsuya KawaharaDomain Adaptation for ASRAutomatic Speech Recognition

    Graduate School of Informatics, Kyoto University, Japan

  28. Tracing and Coordinating Cross-Layer Influence for Multimodal Model Merging

    Sep 11, 2026Pengyang Zhou, Xiaobin Tu, Zhengxi Liu +7Multimodal OptimizationLanguage Model Merging

    Alibaba Cloud

  29. MeRoTune: RoPE-Safe Merging with a Tunable Dial

    Sep 7, 2026Salman FarozLanguage Model MergingModel Merging

  30. CAMFT: Conflict-Aware Mergeable Fine-Tuning for Large Language Models

    Sep 6, 2026Jingang Zhou, Haiyang Guo, Yuan Ma +2LLM Fine-TuningLanguage Model Merging

    School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences