Language Model Merging

Follow this topic Ask about this topic

Momentum

14 papers in the last four weeks, up 367% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 69

All topics

On the Vulnerability of Parameter-Level Defenses to Model Merging
Jun 29, 2026Kuangpu Guo, Qingyan Zheng, Jian Liang +4LLM Security Language Model Merging
University of Science and Technology of China · School of Artificial Intelligence, University of Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences +1
Curvature-Guided Mixing for MLLM Adaptation
Jun 23, 2026Jinglong Yang, Jiaxuan He, Wenjian Huang +2VLM Adaptation Multimodal Large Language Models
Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology · Department of Computer Science, City University of Hong Kong
Scaling Linear Mode Connectivity and Merging to Billion Parameter Pretrained Transformers
Jun 22, 2026Tianyi Li, Zhiqiang ShenTransformer Language Model Merging
MBZUAI
Predicting Mergeability of Parameter-Efficient Fine-Tuning Updates
Jun 17, 2026Lin Tang, Wei Zhang, Jing Li +3Low-Rank Adaptation Language Model Merging
1Sichuan University, Chengdu, China · University of Electronic Science and Technology of China, Chengdu, China
Enhancing Multilingual Reasoning via Steerable Model Merging
Jun 17, 2026Zhuoran Li, Rui Xu, Jian Yang +8Multilingual Language Models Language Model Merging
Beijing University of Posts and Telecommunications · Fudan University · Beihang University +4
Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging
Jun 16, 2026Chenrui Wu, Zexi Li, Jiajun Bu +2Reinforcement Learning with Verifiable Rewards Language Model Merging
Zhejiang University Hangzhou, China · Simon Fraser University Burnaby, Canada · Knowin AI Shenzhen, China +4
Post-Hoc Merging is Not Enough: Many-Shot Model Merging with Loss-Gap Balancing
Jun 15, 2026Kyungjin Im, Miru Kim, Chanin Eom +1Multi-Task Learning Language Model Merging
Soongsil University, Republic of Korea · Department of Electrical and Computer Engineering, Sungkyunkwan University, Republic of Korea
PermDoRA -- Understanding Adapter Interference in Language Models: Limits of Parameter-Space Geometry
Jun 9, 2026Gowtham Sivaramakrishnan, Sarvesha Kumar Kombaiah Seetha, Kishan Gupta Balaji +1Low-Rank Adaptation LLM Security
Independent Researcher
TaDA: Calibrated Probe Gating for Task-Domain LoRA Merging
Jun 3, 2026Huy Quoc To, Fuyi Li, Guangyan Huang +1Adapter Tuning Low-Rank Adaptation
Deakin University, Melbourne, VIC, Australia · Adelaide University, Adelaide, SA, Australia
When Model Merging Breaks Routing: Training-Free Calibration for MoE
Jun 2, 2026Canbin Huang, Tianyuan Shi, Xiaojun Quan +3Mixture of Experts LLM Routing
School of Computer Science and Engineering, Sun Yat-sen University, China · Shenzhen Loop Area Institute, China · Meituan, Inc., China +1
RogueMerge: Robust and Unified Attacks against LLM Model Merging
Jun 2, 2026Jinghuai Zhang, Yetian He, Kunlin Cai +3LLM Security Task Vector Merging
University of California, Los Angeles · University of Illinois Urbana-Champaign · University of Tennessee, Knoxville
ResMerge: Residual-based Spectral Merging of Large Language Models
Jun 1, 2026Yandu Sun, Zhiyan Hou, Haokai Ma +6Task Vector Merging Language Model Merging
1Southeast University, Nanjing, China · Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences +4
Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging
Jun 1, 2026Minsik Choi, Geewook KimCommunication-Efficient Distributed Training Gradient Interference
NAVER Cloud AI · Korea University · KAIST AI
From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Merging
May 29, 2026Qi Sun, Siyue Zhang, Yulin Chen +3Pairwise Preference Learning Low-Rank Adaptation
NYU Shanghai · NTU · NYU +1
Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Merging
May 28, 2026Yuanyi Wang, Yanggan Gu, Su Lu +5Weight-Space Analysis Language Model Merging
The Hong Kong Polytechnic University, PolyU · Hong Kong Polytechnic University Daya Bay Technology and Innovation Research Institute.
EvoGM: Learning to Merge LLMs via Evolutionary Generative Optimization
May 28, 2026Tao Jiang, Xinmeng Yu, Chenhao Yi +5Evolutionary Optimization Language Model Merging
Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology · Pengcheng Laboratory · University of Chinese Academy of Sciences +4
Model Merging by Output-Space Projection
May 27, 2026Bethan Evans, Benjamin Etheridge, Stephen Roberts +1Quadratic Programming Language Model Merging
Department of Mathematics, University of Oxford, Oxford, UK · Department of Engineering Science, University of Oxford, Oxford, UK.
Compositional Generalization in Autoregressive Models via Logit Composition
May 27, 2026Aakash Kumar, Maria Sofia Bucarelli, Emanuele NataleLanguage Model Merging Compositional Generalization
COATI, CNRS, Inria, I3S, Université Coté d’Azur, France
Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training
May 26, 2026Wenjie Zhou, Bohan Wang, Hongtao Zhang +3Language Model Pretraining Neural Network Optimization
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China · Alibaba Group, China +2
On the Limits of Model Merging for Multilinguality in Pre-Training
May 25, 2026Seth Aycock, Fedor Vitiugin, Aleksandr Umnov +2Language Model Pretraining Multilingual Language Models
University of Amsterdam
Tunable MAGMAX: Preference-Aware Model Merging for Continual Learning
May 20, 2026Kei Hiroshima, Kento Uchida, Shinichi ShirakawaContinual Learning Language Model Merging
Yokohama National University, Yokohama Kanagawa 240-8501, Japan
A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM $Δ$ Integration into Upcycled MoE
May 18, 2026Hao Zhou, Tianhao Li, Zhijun Wang +6Multilingual Language Models Mixture-of-Experts Language Models
National Key Laboratory for Novel Software Technology, Nanjing University · Tongyi Lab, Alibaba Group · Zhejiang University
Darwin Family: MRI-Trust-Weighted Evolutionary Merging for Training-Free Scaling of Language-Model Reasoning
May 14, 2026Taebong Kim, Youngsik Hong, Minsik Kim +4Language Model Merging Model Merging
VIDRAFT Inc., Seoul, South Korea
Merging Methods for Multilingual Knowledge Editing for Large Language Models: An Empirical Odyssey
May 13, 2026Kunil Lee, Ki-Young Shin, Jong-Hyeok Lee +1Knowledge Editing Multilingual Language Models
Department of Computer Science and Engineering, POSTECH · Designovel Co., Ltd. · LLSOLLU +1
DiM\textsuperscript{3}: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Merging
May 13, 2026Zijing Wang, Mingyang Wang, Ercong Nie +6Multilingual Language Models Multimodal Large Language Models
Northeastern University, China · CIS, LMU Munich, Germany · Munich Center for Machine Learning (MCML), Germany +1
Black-Box Optimization of Mixed Binary-Continuous Variables: Challenges and Opportunities in Evolutionary Model Merging
May 12, 2026Md. Robiul Islam Niloy, Youhei AkimotoEvolutionary Optimization Black-Box Optimization
BRAC University, Bangladesh
GIFT: Guided Fine-Tuning and Transfer for Enhancing Instruction-Tuned Language Models
May 2, 2026Zhiwen Ruan, Yichao Du, Jianjie Zheng +6LLM Fine-Tuning Large Language Model-Guided Optimization
Southern University of Science and Technology · Alibaba Group · Shanghai University of Finance and Economics +2
Decouple before Integration: Test-time Synthesis of SFT and RLVR Task Vectors
May 1, 2026Chaohao Yuan, Chenghao Xiao, Yu Rong +2Task Arithmetic Language Model Merging
Department of Systems Engineering and Engineering Management, Chinese University of Hong Kong, Hong Kong, China · DAMO Academy, Alibaba Group, Hangzhou, China · Hupan Lab, Hangzhou, China +1
Can Continual Pre-training Bridge the Performance Gap between General-purpose and Specialized Language Models in the Medical Domain?
Apr 21, 2026Niclas Doll, Jasper Schulze Buschhoff, Shalaka Satheesh +3Domain-Adaptive Pretraining Domain Adaptation for LLMs
Fraunhofer IAIS · Lamarr Institute
Evolutionary Negative Module Pruning for Better LoRA Merging
Apr 20, 2026Anda Cao, Zhuo Gou, Yi Wang +5Low-Rank Adaptation Language Model Merging
College of Computer Science and Technology, Zhejiang University · School of Software Technology, Zhejiang University · 3State Key Laboratory of Blockchain and Security, Zhejiang University +1

Jun 29, 2026, cs.LGJ/K move · Enter open · S save

On the Vulnerability of Parameter-Level Defenses to Model Merging

Kuangpu Guo, Qingyan Zheng, Jian Liang, Yongcan Yu+3

University of Science and Technology of China · School of Artificial Intelligence, University of Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences+1

LLM Security Language Model Merging Model Merging

Open paper AskPDF arXiv

The training-free integration of expert models via model merging has exposed significant security risks, enabling free-riders to combine specialized models without authorization. Recent works propose parameter-level defenses that employ linear parameter transformations to neutralize this threat. In this paper, we systematically analyze such defenses and reveal that their protected task vectors are inherently small in magnitude. Consequently, the protected weights remain overwhelmingly dominated by the pretrained model. Based on this observation, we designate the pretrained model as a static reference anchor and propose the Anchor-Guided Attack (AGA) to circumvent existing safeguards. Specifically, AGA aligns the protected model with this anchor to recover the transformation matrix analytically. Extensive evaluations validate that AGA consistently bypasses both individual and composite defenses under realistic defense-agnostic scenarios. Furthermore, we provide Anchor-Repulsive Fine-tuning (ARF), a defense method to mitigate the anchor dominance leveraged by AGA. Empirical results confirm that ARF effectively defeats the proposed attack. Our code is available at https://github.com/krumpguo/secure-merge-attack.

Momentum

Related topics

Latest papers 69