cs.LGNov 7, 2026

Robust and Efficient Continual Model Merging via Global Singular Subspace Separation and Restoration

Authors: Juncheol Shin, Byeongho Yu, Semin Kim, Eunhyeok Park

Organizations: Graduate School of Artificial Intelligence, POSTECH, South Korea · Department of Computer Science & Engineering, POSTECH, South Korea

Abstract

Continual model merging incrementally integrates task-specific models without revisiting past models or data. However, repeated merging entangles past task components and distorts their singular geometry, making balanced task-wise allocation increasingly difficult. We propose Global Singular Subspace Separation and Restoration (GS3R), which augments the merged update with a compact restoration state. At each merge, GS3R restores the retained singular geometry, separates task-wise subspaces, and globally reallocates the available rank across tasks fairly. To reduce merging-time memory overhead, we further introduce Pre-merge Restoration and Flushing (PRF). Experiments on vision, NLP, and multimodal benchmarks show that GS3R provides a strong accuracy-efficiency trade-off.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging

    Dec 1, 2025Kuangpu Guo, Aijing Yu, Jian Liang +4Model Merging

  2. Merge++: Universal Merge Refinement Through Data-Free Checkpoint Inversion

    Sep 19, 2026Aditya Pola, Vineeth N. BalasubramanianModel Merging

  3. SAFE-Merge: Data-Free Continual Model Merging with General Knowledge Preservation

    Aug 2, 2026Zihuan Qiu, Zhiyang Liao, Chiyuan He +5Language Model MergingModel Merging