cs.LGNov 7, 2026

Robust and Efficient Continual Model Merging via Global Singular Subspace Separation and Restoration

Authors: Juncheol Shin, Byeongho Yu, Semin Kim, Eunhyeok Park

Organizations: Graduate School of Artificial Intelligence, POSTECH, South Korea · Department of Computer Science & Engineering, POSTECH, South Korea

Abstract

Continual model merging incrementally integrates task-specific models without revisiting past models or data. However, repeated merging entangles past task components and distorts their singular geometry, making balanced task-wise allocation increasingly difficult. We propose Global Singular Subspace Separation and Restoration (GS3R), which augments the merged update with a compact restoration state. At each merge, GS3R restores the retained singular geometry, separates task-wise subspaces, and globally reallocates the available rank across tasks fairly. To reduce merging-time memory overhead, we further introduce Pre-merge Restoration and Flushing (PRF). Experiments on vision, NLP, and multimodal benchmarks show that GS3R provides a strong accuracy-efficiency trade-off.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Dec 1, 2025

Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging

Model merging has emerged as a promising paradigm for enabling multi-task capabilities without additional training. However, traditional basic merging methods often experience performance degradation due to parameter conflicts, even when applied to similar tasks. While recent personalized merging frameworks successfully preserve task-specific information to maintain performance, they typically incur storage overhead. In this paper, we propose Decomposition, Thresholding, and Scaling (DTS), an approximation-based personalized merging framework that pushes task-specific storage efficiency. DTS first applies singular value decomposition to the task-specific information and retains only a small subset of singular values and vectors. It then introduces a novel thresholding strategy that partitions singular vector elements into groups and assigns a scaling factor to each group. To enable generalization to unseen tasks, we further extend DTS with a variant that fuses task-specific information in a data-free manner based on the semantic similarity of task characteristics. Extensive experiments demonstrate that DTS consistently outperforms state-of-the-art baselines while requiring only 1% extra storage per task. Furthermore, experiments on unseen tasks show that the DTS variant achieves significantly better generalization performance. Our code is available at https://github.com/krumpguo/DTS.
Sep 19, 2026

Merge++: Universal Merge Refinement Through Data-Free Checkpoint Inversion

Model merging consolidates fine-tuned experts into one multi-task model without retraining. All existing data-free methods approach this problem entirely in weight space. Restricted to arithmetic on parameters, these methods never observe how each expert behaves, a signal that only emerges through forward evaluation. Accessing this behavioral signal requires inputs to evaluate on, which the data-free setting prohibits. We propose Merge++, a post-hoc method that addresses this by inverting the expert checkpoints to synthesize task-representative images, then distilling expert knowledge into the merged model using those images. Merge++ requires no additional data beyond the checkpoints themselves. It applies universally across merging algorithms and operates as a complementary refinement stage independent of the underlying weight-space method. The method consistently improves merging algorithms ranging from simple task arithmetic to state-of-the-art spectral methods, with average gains of +2 to +8 points and up to +25.9 on individual configurations.
Aug 2, 2026

SAFE-Merge: Data-Free Continual Model Merging with General Knowledge Preservation

Data-free continual model merging must incorporate a stream of specialized models while retaining both pretrained general knowledge and previously acquired tasks, without access to task data. Existing methods mainly merge task updates by suppressing interference among downstream tasks; while this protects previously acquired tasks, it overlooks the safety of the pretrained knowledge itself, whose erosion degrades generalization to held-out distributions and weakens the foundation for future task acquisition. We propose SAFE-Merge, a simple data-free continual-merging framework that first decides which parameter updates are safe to retain, and then recovers the task information lost through masking. Specifically, to ensure safety, risk-aware sparse masking selects parameter updates that carry task-specific information while posing low risk to general knowledge. Masked low-rank recovery then compensates for the lost task information using only the same retained parameter updates, while leaving all masked-out parameters strictly unchanged. Finally, the combined update is fused into the backbone, incurring no additional inference cost. Across vision and language benchmarks, SAFE-Merge consistently achieves the best H-score. On longer CLIP task sequences, it substantially improves H-score over NUFILT while also achieving the highest accuracy.