Continual model merging incrementally integrates task-specific models without revisiting past models or data. However, repeated merging entangles past task components and distorts their singular geometry, making balanced task-wise allocation increasingly difficult. We propose Global Singular Subspace Separation and Restoration (GS3R), which augments the merged update with a compact restoration state. At each merge, GS3R restores the retained singular geometry, separates task-wise subspaces, and globally reallocates the available rank across tasks fairly. To reduce merging-time memory overhead, we further introduce Pre-merge Restoration and Flushing (PRF). Experiments on vision, NLP, and multimodal benchmarks show that GS3R provides a strong accuracy-efficiency trade-off.
Figures & tables
Figure 1: Overall process of GS3R. Each loop restores the task-wise geometry from the previous state, performs global task-wise merging, and stores restoration metadata for the next step.
Method
ViT-B/32
ViT-B/16
ViT-L/14
8 tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
Pre-Trained
48.1
56.9
55.6
55.4
62.0
59.8
64.9
69.1
65.6
Fine-Tuned
90.4
89.3
89.8
92.4
91.3
91.6
94.3
93.4
93.5
C. Fine-Tuned
79.8
67.4
62.6
82.9
72.2
68.2
90.0
70.9
77.7
ACC ↑
Average (SWA)
66.3 ± 0.0
65.4 ± 0.0
61.1 ± 0.0
72.3 ± 0.0
69.7 ± 0.0
64.8 ± 0.0
80.0 ± 0.0
77.5 ± 0.0
71.1 ± 0.0
C. Task Arithmetic
67.5 ± 0.0
66.5 ± 0.0
60.6 ± 0.0
77.1 ± 0.0
70.9 ± 0.0
64.2 ± 0.0
82.1 ± 0.0
77.9 ± 0.0
70.3 ± 0.0
Table 1: Comparison of continual model merging performance for ViT models on vision tasks. We report the mean and standard deviation of accuracy (ACC) and backward transfer (BWT). STO denotes the relative storage overhead against the merged model. The horizontal line separates optimization-free (top) and optimization-based (bottom) methods. Best results within each group are indicated in bold, and second-best results are underlined. “C.” stands for Continual.
Method
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST2
STSB
ACC( ↑ )
BWT( ↑ )
STO( ↓ )
Pre-Trained
69.1
56.5
76.2
88.4
82.1
80.1
91.2
62.2
75.7
-
-
Fine-Tuned
75.0
83.4
87.5
91.5
85.4
85.9
93.6
88.7
86.4
-
-
OPCM
69.7
72.9
78.8
90.2
83.8
82.2
92.3
74.7
80.6 ± 0.3
-2.1 ± 0.6
1.00 ×
GS3R
70.5
79.5
78.8
90.6
84.3
83.0
92.9
81.3
82.6 ± 0.1
-1.9 ± 0.6
1.525 ×
GS3R ( r=64 )
71.7
81.1
80.0
90.7
84.2
82.2
93.1
86.4
83.7 ± 0.1
-1.2 ± 0.4
1.004 ×
DOP
72.6
80.9
81.0
90.4
83.4
81.0
92.9
86.1
83.6 ± 0.1
-1.3 ± 0.3
1.00 ×
Table 2: Continual model merging results for the Flan-T5-Base model on GLUE tasks. We report the mean and standard deviation of accuracy and backward transfer.
Figure 2: Accuracy and storage trade-off with varying rank r . (a) shows accuracy variation with varying r on vision and NLP tasks. (b) shows that combining GS3R with truncated SVD yields a favorable trade-off. r=256 preserves strong performance while requiring only 0.05 × more storage.
Method
IconQA
Image
SciQA
VizWiz
ACC( ↑ )
ACC † ( ↑ )
STO( ↓ )
Pre-Trained
14.1
40.9
61.7
41.0
39.4
–
–
Fine-Tuned
75.5
96.0
83.7
64.8
80.0
–
–
C. Task Arithmetic
55.7 (55.9)
75.4
77.2
46.7
63.8 ± 0.0
63.8 ± 0.0
1.00 ×
C. Ties-Merging
57.6 (63.7)
84.1
79.8
44.6
66.5 ± 3.9
68.1 ± 1.6
1.00 ×
OPCM
54.6 (63.1)
83.1
80.9
45.6
66.0 ± 0.7
68.2 ± 0.8
1.00 ×
GS3R ( r=256 )
56.2 (69.3)
93.8
80.9
47.0
69.5 ± 0.2
72.7 ± 0.1
1.002 ×
Table 3: Continual model merging results of LLaVA-1.5-7B on multimodal tasks. ACC † scores IconQA with a format-tolerant rule applied identically to all methods, with the corresponding IconQA accuracy in parentheses. Detailed explanations are provided in Appendix C.4 .
Figure 3: Accuracy trajectories during continual merging of ViT-B/32 over 8 vision tasks. Rows correspond to merge steps and columns to tasks, showing how previously merged tasks evolve as new tasks arrive. Colors are independently normalized per task.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Gram matrix of concatenated singular vectors from U of each task.
Figure 5: Gram matrix of concatenated singular vectors from V of each task.
Tolerance ϵ
10−2
10−3
10−4
10−5
10−6
Correct cluster count
0/73
70/73
73/73
73/73
73/73
Mean ARI
0.007
0.959
1.000
1.000
1.000
Appendix
Table 6: Reliability of the separation step on ViT-B/32 with 20 tasks under varying tolerance ϵ . We report the fraction of layers (out of 73) with the correct cluster count and the mean Adjusted Rand Index. ϵ=10−4 is used throughout the paper.
Tolerance ϵ
10−2
5×10−3
10−3
10−4
10−5
Accuracy (%)
81.6 ± 0.6
82.4 ± 0.3
82.7 ± 0.2
82.7 ± 0.2
82.7 ± 0.2
Appendix
Table 7: Effect of the tolerance ϵ on merging performance, merging 8 tasks with ViT-B/32 over 10 task orders.
r
16
32
64
128
256
Full Rank
Accuracy (%)
82.73 ± 0.11
83.59 ± 0.10
83.63 ± 0.06
83.38 ± 0.12
83.30 ± 0.08
82.64 ± 0.17
Appendix
Table 8: Rank sweep on GLUE with Flan-T5-Base, re-run with 10 new task orders. The same pattern as the original experiment appears: accuracy peaks at r=64 and falls off toward both ends by margins an order of magnitude larger than the seed variance.
GLUE: Flan-T5
Vision suite
Decoder
Encoder
All
ViT-B/32 (8 tasks)
Energy rank
26.1
197.6
93.0
147.2
Appendix
Table 9: Energy rank of task vectors, defined as the minimum rank capturing 90% of ∑iσi2 , averaged over tasks and layers.
r
16
32
64
128
256
GLUE
Etrunc
0.30
0.23
0.18
0.14
0.10
Eorth
0.67
0.72
0.76
0.81
0.87
Etotal
0.87
0.86
0.86
0.88
0.92
Vision
Etrunc
0.59
0.50
0.40
0.30
0.21
Eorth
0.47
0.48
0.52
0.58
0.67
Etotal
0.93
0.90
0.86
0.82
0.82
Appendix
Table 10: Error decomposition under varying total rank budget r . Etrunc , Eorth , and Etotal are defined in Eq. ( 20 ). On GLUE, orthogonalization damage dominates and grows with r , so the total is minimized at an intermediate budget. On the vision suite, truncation dominates and the total decreases monotonically.
Nuclear
Energy
Frobenius
Global Top- k
Ours
Accuracy (%)
82.3 ± 0.3
82.3 ± 0.2
82.5 ± 0.3
82.6 ± 0.2
82.7 ± 0.2
Appendix
Table 11: Comparison of rank allocation strategies when merging 8 tasks with ViT-B/32. We report mean and standard deviation of accuracy over 10 task orders.
Accuracy (%)
Storage (MB)
Rank Budget ( r )
Explicit
GS3R
Explicit
GS3R
32
72.9
72.7 ± 0.3
21.3
0.3
64
76.3
76.0 ± 0.3
42.6
1.2
128
79.4
79.1 ± 0.2
85.1
4.8
256
82.0
81.8 ± 0.2
170.0
19.0
Full
83.1
82.7 ± 0.2
509.8
170.1
Appendix
Table 12: Comparison with the explicit-storage baseline under varying rank budgets, merging 8 tasks with ViT-B/32. We report the mean accuracy (%) and the additional storage (MB). Results of GS3R are averaged over 10 task orders. The explicit-storage baseline is deterministic given the task order.
Figure 6: t-SNE visualization of the representation distributions of EuroSAT extracted from merged models using different methods.
Cars
DTD
Euro
GTS
MNIST
RESIS
SVHN
SUN
AVG
Order 1, original
71.5
63.6
95.0
91.6
98.7
84.2
89.8
67.4
82.7
Order 1, repeated
72.1
68.8
94.7
92.9
98.8
84.9
91.2
67.8
83.9
Order 2, original
70.5
63.2
95.3
92.8
98.7
83.0
91.0
67.1
82.7
Order 2, repeated
68.7
61.1
91.0
90.2
98.4
80.9
95.8
65.9
81.5
Appendix
Table 13: Effect of merging a repeated task. After merging 8 tasks with ViT-B/32, the task appearing fourth in each order (DTD for Order 1, SVHN for Order 2) is merged once more. The accuracy of the repeated task is underlined.
Dec 1, 2025·Kuangpu Guo, Aijing Yu, Jian Liang +4Model Merging
University of Science and Technology of China · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences +3