Overview

Model merging studies how to combine weights or adapters from separately trained models into one model, aiming to reuse complementary skills without storing an ensemble or repeating joint training. Common approaches average checkpoints or add task vectors, while sparsification, low-rank and spectral methods, geometric alignment, and calibration-based routing aim to reduce interference. The methods are used for multi-task and continual learning, domain adaptation, and combining language, vision, speech, and specialized models; evaluations often measure task accuracy, capability retention, worst-task performance, and inference or storage cost. The central tension is that updates that help one task can erase or distort another: success depends on shared initialization, representation compatibility, update scale, and data or prompt distribution, and some task combinations may not be recoverable through weight-space fusion alone.

Latest papers

CardsList

214 papers18 papers in the last four weeks, down 53% on the four weeks before. 0.1% of all new papers.

  1. A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic

    Oct 6, 2026Sin-Han Yang, Shih-Cheng Huang, Chieh-Yen Lin +3Task Vector MergingMulti-Task Optimization

  2. ααTransfer: Coefficient Transfer for Efficient Model Merging

    Oct 6, 2026Shih-Cheng Huang, Zhi Rui Tam, Chieh-Yen Lin +3Hyperparameter TransferLanguage Model Merging

  3. Training-Free Transformer Merging via Sequential Local Operator Alignment

    Oct 5, 2026Akansh Maurya, Ya-Wei Eileen Lin, Stefanie Jegelka +2Language Model MergingModel Merging

  4. Off-Policy Merging Beats On-Policy Self-Distillation for Continual Learning

    Oct 5, 2026Chen Henry Wu, Thomas Zhang, Aditi RaghunathanContinual Learning for LLMsOn-Policy Distillation

  5. Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs

    Oct 3, 2026Hoigi Seo, Byung Hyun Lee, Minjun Kim +3Cross-Modal AlignmentMultimodal Large Language Models

  6. First-Order Steering: Translating Weight Adaptation into Activation Steering

    Oct 3, 2026Sri Pranav Kunda, Alexander Kurz, Tomas Dominik +1Activation SteeringLanguage Model Steering

  7. Mixture-Trained Merging for Unified Multi-Objective Models

    Oct 1, 2026SeongHyeon Kim, Chaeyun Jang, Seungyoo Lee +4Multi-Task LearningLanguage Model Merging

  8. Does Scaling Reinforcement Learning Really Require More Training?

    Oct 1, 2026Bangji Yang, Jiajun Fan, Hongbo Ma +2RL for Language Model ReasoningRL Post-Training

  9. ChainLoRA: Geometry-Preserving Task Vector Merging for Continual Learning in LLMs

    Sep 30, 2026Hang Yin, Haozhe Wang, Yuhua Luo +3Continual Learning for LLMsLow-Rank Adaptation

  10. No Task Vector Is an Island: A Comprehensive Study on the Composability of Task Vectors from On-Policy Distillation

    Sep 30, 2026Jingang Zhou, Feiyu Han, Han Zhu +6Task VectorsTask Vector Merging

  11. Exploring Heterogeneous Model Merging Approach for Complex Knowledge Transfer

    Sep 30, 2026Jiahe Fan, Si Chen, Yinghao Hou +4Transfer LearningLanguage Model Merging

  12. S3S^3: Spectral Null-Space Swap Makes Reasoning Models Efficient

    Sep 29, 2026Hongbo Ma, Sansheng Cao, Jiajun Fan +2Efficient Language Model ReasoningModel Merging

  13. Orthogonal Yet Coupled: Decoupling Geometric Components for Model Merging

    Sep 29, 2026Zijing Wang, Yongkang Liu, Mingyang Wang +8Task Vector MergingModel Merging

  14. Trajectory Soup: Pushing the Compute-Scaling Frontier of LLM Mid-training via Diverse Trajectories

    Sep 29, 2026Zhehao Huang, Changxin Tian, Qingyuan Yang +5Language Model ScalingModel Merging

  15. QAM: Quadratic-Accurate Checkpoint Merging via Sequential Consistency

    Sep 28, 2026Shihao Wang, Rui Kong, Xinran Chen +7Language Model MergingModel Merging

  16. USA: Update-aware SAM for Cross-domain On-Policy Disitllation of Language Agents

    Sep 28, 2026Qiyong Zhong, Mao Zheng, Mingyang Song +6Cross-Domain Transfer LearningOn-Policy Distillation

  17. CASS: Contribution-Aware Structured Sparsity for Model Merging

    Sep 28, 2026Yan Li, Guiping Cao, Meng Xu +5Multi-Task LearningStructured Sparsity

  18. SMAT: Simple and Efficient Merge-Aware Training

    Sep 27, 2026Yanggan Gu, Yuanyi Wang, Zhen Li +5Language Model MergingModel Merging

  19. MergeHEIR: Mitigating Multimodal Hallucinations as the Tax of Model Merging

    Sep 26, 2026Jinyu Li, Hao Fang, Zhiming Zhang +3Model MergingVLM Hallucination Mitigation

  20. Child ASR Adaptation with Adult Retention: An Empirical Study

    Sep 25, 2026Houssam Eddine-Othman Lachemat, Shammur Absar ChowdhuryChild Speech RecognitionAutomatic Speech Recognition

  21. Beyond Uniform Subspaces: Spectrum-Aware and Depth-Adaptive Fusion for Multi-Task Model Merging

    Sep 21, 2026Ruxi Gu, Zilei Wang, Wei WangModel MergingData-Free Model Merging

  22. On Emergent Capabilities and Model Merging

    Sep 21, 2026Luca Zhou, Emanuele RodolàEmergent Abilities in Language ModelsEmergent Misalignment in Language Models

  23. A Distributional Optimisation Perspective on Combining Models in Deep Learning

    Sep 21, 2026Congye Wang, Yan Lin, Zheyang Shen +2Ensemble LearningDeep Ensembles

  24. Merge++: Universal Merge Refinement Through Data-Free Checkpoint Inversion

    Sep 19, 2026Aditya Pola, Vineeth N. BalasubramanianModel Merging