Continual model merging incrementally integrates task-specific models without revisiting past models or data. However, repeated merging entangles past task components and distorts their singular geometry, making balanced task-wise allocation increasingly difficult. We propose Global Singular Subspace Separation and Restoration (GS3R), which augments the merged update with a compact restoration state. At each merge, GS3R restores the retained singular geometry, separates task-wise subspaces, and globally reallocates the available rank across tasks fairly. To reduce merging-time memory overhead, we further introduce Pre-merge Restoration and Flushing (PRF). Experiments on vision, NLP, and multimodal benchmarks show that GS3R provides a strong accuracy-efficiency trade-off.
Figures & tables
Figure 1: Overall process of GS3R. Each loop restores the task-wise geometry from the previous state, performs global task-wise merging, and stores restoration metadata for the next step.
Method
ViT-B/32
ViT-B/16
ViT-L/14
8 tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
Pre-Trained
48.1
56.9
55.6
55.4
62.0
59.8
64.9
69.1
65.6
Fine-Tuned
90.4
89.3
89.8
92.4
91.3
91.6
94.3
93.4
93.5
C. Fine-Tuned
79.8
67.4
62.6
82.9
72.2
68.2
90.0
70.9
77.7
ACC ↑
Average (SWA)
66.3 ± 0.0
65.4 ± 0.0
61.1 ± 0.0
72.3 ± 0.0
69.7 ± 0.0
64.8 ± 0.0
80.0 ± 0.0
77.5 ± 0.0
71.1 ± 0.0
C. Task Arithmetic
67.5 ± 0.0
66.5 ± 0.0
60.6 ± 0.0
77.1 ± 0.0
70.9 ± 0.0
64.2 ± 0.0
82.1 ± 0.0
77.9 ± 0.0
70.3 ± 0.0
Table 1: Comparison of continual model merging performance for ViT models on vision tasks. We report the mean and standard deviation of accuracy (ACC) and backward transfer (BWT). STO denotes the relative storage overhead against the merged model. The horizontal line separates optimization-free (top) and optimization-based (bottom) methods. Best results within each group are indicated in bold, and second-best results are underlined. “C.” stands for Continual.
Method
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST2
STSB
ACC( ↑ )
BWT( ↑ )
STO( ↓ )
Pre-Trained
69.1
56.5
76.2
88.4
82.1
80.1
91.2
62.2
75.7
-
-
Fine-Tuned
75.0
83.4
87.5
91.5
85.4
85.9
93.6
88.7
86.4
-
-
OPCM
69.7
72.9
78.8
90.2
83.8
82.2
92.3
74.7
80.6 ± 0.3
-2.1 ± 0.6
1.00 ×
GS3R
70.5
79.5
78.8
90.6
84.3
83.0
92.9
81.3
82.6 ± 0.1
-1.9 ± 0.6
1.525 ×
GS3R ( r=64 )
71.7
81.1
80.0
90.7
84.2
82.2
93.1
86.4
83.7 ± 0.1
-1.2 ± 0.4
1.004 ×
DOP
72.6
80.9
81.0
90.4
83.4
81.0
92.9
86.1
83.6 ± 0.1
-1.3 ± 0.3
1.00 ×
Table 2: Continual model merging results for the Flan-T5-Base model on GLUE tasks. We report the mean and standard deviation of accuracy and backward transfer.
Figure 2: Accuracy and storage trade-off with varying rank r . (a) shows accuracy variation with varying r on vision and NLP tasks. (b) shows that combining GS3R with truncated SVD yields a favorable trade-off. r=256 preserves strong performance while requiring only 0.05 × more storage.
Method
IconQA
Image
SciQA
VizWiz
ACC( ↑ )
ACC † ( ↑ )
STO( ↓ )
Pre-Trained
14.1
40.9
61.7
41.0
39.4
–
–
Fine-Tuned
75.5
96.0
83.7
64.8
80.0
–
–
C. Task Arithmetic
55.7 (55.9)
75.4
77.2
46.7
63.8 ± 0.0
63.8 ± 0.0
1.00 ×
C. Ties-Merging
57.6 (63.7)
84.1
79.8
44.6
66.5 ± 3.9
68.1 ± 1.6
1.00 ×
OPCM
54.6 (63.1)
83.1
80.9
45.6
66.0 ± 0.7
68.2 ± 0.8
1.00 ×
GS3R ( r=256 )
56.2 (69.3)
93.8
80.9
47.0
69.5 ± 0.2
72.7 ± 0.1
1.002 ×
Table 3: Continual model merging results of LLaVA-1.5-7B on multimodal tasks. ACC † scores IconQA with a format-tolerant rule applied identically to all methods, with the corresponding IconQA accuracy in parentheses. Detailed explanations are provided in Appendix C.4 .
Figure 3: Accuracy trajectories during continual merging of ViT-B/32 over 8 vision tasks. Rows correspond to merge steps and columns to tasks, showing how previously merged tasks evolve as new tasks arrive. Colors are independently normalized per task.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Gram matrix of concatenated singular vectors from U of each task.
Figure 5: Gram matrix of concatenated singular vectors from V of each task.
Tolerance ϵ
10−2
10−3
10−4
10−5
10−6
Correct cluster count
0/73
70/73
73/73
73/73
73/73
Mean ARI
0.007
0.959
1.000
1.000
1.000
Appendix
Table 6: Reliability of the separation step on ViT-B/32 with 20 tasks under varying tolerance ϵ . We report the fraction of layers (out of 73) with the correct cluster count and the mean Adjusted Rand Index. ϵ=10−4 is used throughout the paper.
Tolerance ϵ
10−2
5×10−3
10−3
10−4
10−5
Accuracy (%)
81.6 ± 0.6
82.4 ± 0.3
82.7 ± 0.2
82.7 ± 0.2
82.7 ± 0.2
Appendix
Table 7: Effect of the tolerance ϵ on merging performance, merging 8 tasks with ViT-B/32 over 10 task orders.
r
16
32
64
128
256
Full Rank
Accuracy (%)
82.73 ± 0.11
83.59 ± 0.10
83.63 ± 0.06
83.38 ± 0.12
83.30 ± 0.08
82.64 ± 0.17
Appendix
Table 8: Rank sweep on GLUE with Flan-T5-Base, re-run with 10 new task orders. The same pattern as the original experiment appears: accuracy peaks at r=64 and falls off toward both ends by margins an order of magnitude larger than the seed variance.
GLUE: Flan-T5
Vision suite
Decoder
Encoder
All
ViT-B/32 (8 tasks)
Energy rank
26.1
197.6
93.0
147.2
Appendix
Table 9: Energy rank of task vectors, defined as the minimum rank capturing 90% of ∑iσi2 , averaged over tasks and layers.
r
16
32
64
128
256
GLUE
Etrunc
0.30
0.23
0.18
0.14
0.10
Eorth
0.67
0.72
0.76
0.81
0.87
Etotal
0.87
0.86
0.86
0.88
0.92
Vision
Etrunc
0.59
0.50
0.40
0.30
0.21
Eorth
0.47
0.48
0.52
0.58
0.67
Etotal
0.93
0.90
0.86
0.82
0.82
Appendix
Table 10: Error decomposition under varying total rank budget r . Etrunc , Eorth , and Etotal are defined in Eq. ( 20 ). On GLUE, orthogonalization damage dominates and grows with r , so the total is minimized at an intermediate budget. On the vision suite, truncation dominates and the total decreases monotonically.
Nuclear
Energy
Frobenius
Global Top- k
Ours
Accuracy (%)
82.3 ± 0.3
82.3 ± 0.2
82.5 ± 0.3
82.6 ± 0.2
82.7 ± 0.2
Appendix
Table 11: Comparison of rank allocation strategies when merging 8 tasks with ViT-B/32. We report mean and standard deviation of accuracy over 10 task orders.
Accuracy (%)
Storage (MB)
Rank Budget ( r )
Explicit
GS3R
Explicit
GS3R
32
72.9
72.7 ± 0.3
21.3
0.3
64
76.3
76.0 ± 0.3
42.6
1.2
128
79.4
79.1 ± 0.2
85.1
4.8
256
82.0
81.8 ± 0.2
170.0
19.0
Full
83.1
82.7 ± 0.2
509.8
170.1
Appendix
Table 12: Comparison with the explicit-storage baseline under varying rank budgets, merging 8 tasks with ViT-B/32. We report the mean accuracy (%) and the additional storage (MB). Results of GS3R are averaged over 10 task orders. The explicit-storage baseline is deterministic given the task order.
Figure 6: t-SNE visualization of the representation distributions of EuroSAT extracted from merged models using different methods.
Cars
DTD
Euro
GTS
MNIST
RESIS
SVHN
SUN
AVG
Order 1, original
71.5
63.6
95.0
91.6
98.7
84.2
89.8
67.4
82.7
Order 1, repeated
72.1
68.8
94.7
92.9
98.8
84.9
91.2
67.8
83.9
Order 2, original
70.5
63.2
95.3
92.8
98.7
83.0
91.0
67.1
82.7
Order 2, repeated
68.7
61.1
91.0
90.2
98.4
80.9
95.8
65.9
81.5
Appendix
Table 13: Effect of merging a repeated task. After merging 8 tasks with ViT-B/32, the task appearing fourth in each order (DTD for Order 1, SVHN for Order 2) is merged once more. The accuracy of the repeated task is underlined.
Model merging has emerged as a promising paradigm for enabling multi-task capabilities without additional training. However, traditional basic merging methods often experience performance degradation due to parameter conflicts, even when applied to similar tasks. While recent personalized merging frameworks successfully preserve task-specific information to maintain performance, they typically incur storage overhead. In this paper, we propose Decomposition, Thresholding, and Scaling (DTS), an approximation-based personalized merging framework that pushes task-specific storage efficiency. DTS first applies singular value decomposition to the task-specific information and retains only a small subset of singular values and vectors. It then introduces a novel thresholding strategy that partitions singular vector elements into groups and assigns a scaling factor to each group. To enable generalization to unseen tasks, we further extend DTS with a variant that fuses task-specific information in a data-free manner based on the semantic similarity of task characteristics. Extensive experiments demonstrate that DTS consistently outperforms state-of-the-art baselines while requiring only 1% extra storage per task. Furthermore, experiments on unseen tasks show that the DTS variant achieves significantly better generalization performance. Our code is available at https://github.com/krumpguo/DTS.
Kuangpu Guo, Aijing Yu, Jian Liang +4
University of Science and Technology of China · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences +3
Model merging consolidates fine-tuned experts into one multi-task model without retraining. All existing data-free methods approach this problem entirely in weight space. Restricted to arithmetic on parameters, these methods never observe how each expert behaves, a signal that only emerges through forward evaluation. Accessing this behavioral signal requires inputs to evaluate on, which the data-free setting prohibits. We propose Merge++, a post-hoc method that addresses this by inverting the expert checkpoints to synthesize task-representative images, then distilling expert knowledge into the merged model using those images. Merge++ requires no additional data beyond the checkpoints themselves. It applies universally across merging algorithms and operates as a complementary refinement stage independent of the underlying weight-space method. The method consistently improves merging algorithms ranging from simple task arithmetic to state-of-the-art spectral methods, with average gains of +2 to +8 points and up to +25.9 on individual configurations.
Data-free continual model merging must incorporate a stream of specialized models while retaining both pretrained general knowledge and previously acquired tasks, without access to task data. Existing methods mainly merge task updates by suppressing interference among downstream tasks; while this protects previously acquired tasks, it overlooks the safety of the pretrained knowledge itself, whose erosion degrades generalization to held-out distributions and weakens the foundation for future task acquisition. We propose SAFE-Merge, a simple data-free continual-merging framework that first decides which parameter updates are safe to retain, and then recovers the task information lost through masking. Specifically, to ensure safety, risk-aware sparse masking selects parameter updates that carry task-specific information while posing low risk to general knowledge. Masked low-rank recovery then compensates for the lost task information using only the same retained parameter updates, while leaving all masked-out parameters strictly unchanged. Finally, the combined update is fused into the backbone, incurring no additional inference cost. Across vision and language benchmarks, SAFE-Merge consistently achieves the best H-score. On longer CLIP task sequences, it substantially improves H-score over NUFILT while also achieving the highest accuracy.
Zihuan Qiu, Zhiyang Liao, Chiyuan He +5
University of Electronic Science and Technology of China, Chengdu, China · Dalian University of Technology, Dalian, China