Merging finetuned Large Language Models (LLMs) has become increasingly important for integrating diverse capabilities into a single unified model. However, prevailing model merging methods rely on linear arithmetic in Euclidean space, which often destroys the intrinsic geometric properties of pretrained weights, such as hyperspherical energy. To address this, we propose Orthogonal Model Merging (OrthoMerge), a method that performs merging operations on the Riemannian manifold formed by the orthogonal group to preserve the geometric structure of the model's weights. By mapping task-specific orthogonal matrices learned by Orthogonal Finetuning (OFT) to the Lie algebra, OrthoMerge enables a principled yet efficient integration that takes into account both the direction and intensity of adaptations. In addition to directly leveraging orthogonal matrices obtained by OFT, we further extend this approach to general models finetuned with non-OFT methods (i.e., low-rank finetuning, full finetuning) via an Orthogonal-Residual Decoupling strategy. This technique extracts the orthogonal components of expert models by solving the orthogonal Procrustes problem, which are then merged on the manifold of the orthogonal group, while the remaining linear residuals are processed through standard additive merging. Extensive empirical results demonstrate the effectiveness of OrthoMerge in mitigating catastrophic forgetting and maintaining model performance across diverse tasks.
Figures & tables
Figure 1: A comparison among (a) current model merging, (b) orthogonal merging and (c) orthogonal-residual decoupling merging.
Figure 2: Illustration of OrthoMerge. (a) To merge orthogonal transformations, we first map them to the Lie algebra so(d) , perform the merging there with magnitude correction to preserve the strength of the transformations, and finally map the result back to the orthogonal group. (b) For general models, we decouple weights into orthogonal and residual components, merging them separately on the Riemannian manifold formed by the orthogonal group and in Euclidean space, respectively.
Figure 3: Loss landscape of the base model, TA and OrthoMerge.
Model
MATH500
HUMANEVAL+
ScienceQA
CommonsenseQA
Social-IQA
Avg.
M-ARC
AGIEval
Avg.
Llama-3.1-8B
17.80
21.52
6.12
70.60
48.11
32.83
40.85
35.76
38.31
LoRA
Task-specific FT
19.40
38.48
26.44
83.78
58.39
45.30
–
–
–
TA
21.00
34.20
29.90
80.18
54.15
43.89
43.80
37.90
40.85
TIES
18.80
38.65
20.68
82.15
56.96
43.45
44.57
37.09
40.83
DARE
22.60
34.26
29.68
80.10
53.84
44.10
43.80
37.86
40.83
TSV-M
0.60
31.09
30.62
83.05
58.03
40.68
43.44
34.51
38.98
Table 1: Performance of different merging methods on task-specific models finetuned from Llama-3.1-8B using LoRA and OFT, evaluated on task-specific benchmarks, out-of-domain, and general evaluation datasets.
Model
Instruction
Math
Coding
Multilingual
Safety
Avg.
MMLU†
AGIEval
Avg.
Llama-3.2-3B
7.58
28.51
27.44
40.72
31.41
27.18
55.00
31.05
43.03
Task-specific FT
39.56
69.83
44.33
41.73
80.46
55.18
–
–
–
TA
10.53
40.40
37.22
42.26
40.40
34.16
55.35
32.67
44.01
+OrthoMerge-C
9.80
40.03
37.75
42.29
41.69
34.31 (+0.15)
55.76
32.69
44.23 (+0.22)
+OrthoMerge-G
11.09
43.90
37.96
42.06
42.41
35.48 (+1.32)
55.57
32.85
44.21 (+0.20)
TIES
20.89
50.80
39.61
42.11
42.38
39.16
54.52
33.88
44.20
Table 2: Performance of merging task-specific models (provided by MergeBench) finetuned from Llama-3.2-3B. Results compare standard merging baselines and their combinations with our OrthoMerge variants (+OrthoMerge-C, +OrthoMerge-G). “MMLU†” denotes MMLU evaluated with math- and coding-related subsets excluded.
Model
MMSI-Bench
EmbSpatial
MMMU Med
PathVQA
OCRBench
CharXiv
Avg.
IFEval
MMBench
Avg.
Qwen2.5-VL-7B-Instruct
27.80
69.97
53.10
66.30
84.70
67.20
61.51
61.74
84.02
72.88
Task-specific FT
32.60
70.58
55.17
66.81
85.00
72.50
63.78
–
–
–
TA
28.70
71.25
52.41
67.76
83.70
68.40
62.04
60.12
83.84
71.98
+OrthoMerge-C
29.00
71.26
54.48
67.94
84.60
69.10
62.73 (+0.69)
61.00
83.85
72.43 (+0.45)
+OrthoMerge-G
29.90
71.29
57.24
68.26
83.50
68.70
63.15 (+1.11)
60.81
83.94
72.38 (+0.40)
TIES
32.40
71.57
57.93
67.61
82.60
69.10
63.54
55.15
83.17
69.16
Table 3: Performance of merging task-specific vision-language models finetuned from Qwen2.5-VL-7B-Instruct. Results compare standard merging baselines as well as their combinations with our OrthoMerge variants (+OrthoMerge-C, +OrthoMerge-G).
Figure 4: Comparison of norm distributions between different decoupling strategies applied to the models from Section 5.2.1 . (a) Norm statistics using the Global Decoupling strategy. (b) Norm statistics using the Conflict-Aware Decoupling strategy.
Model
Math
Coding
Science
Commonsense
Social
Avg.
Llama-3.1-8B
17.80
21.52
6.12
70.60
48.11
32.83
Task-specific FT
19.00
38.54
26.75
82.47
56.81
44.71
Simple Average R
21.40
30.06
18.88
76.82
50.51
39.53
Sequential Product R
8.60
33.05
23.65
80.84
57.68
40.76
Simple Average Q
22.00
30.85
23.11
76.99
52.10
41.01
OrthoMerge
24.60
38.90
32.01
80.59
55.17
46.25
Table 4: Comparison of different orthogonal transformation merging strategies across task-specific benchmarks (the same benchmark settings as in Section 5.1 ).
Figure 5: Performance gain vs. number of tasks of different merging methods over base model.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Table 10
Model
Inst.
Math
Code
Multi.
Safety
Avg.
TA
10.53
40.40
37.22
42.26
40.40
34.16
+ OrthoMerge-C (SVD)
9.80
40.03
37.75
42.29
41.69
34.31
+ OrthoMerge-C (Newton-Schulz)
10.72
39.88
37.63
42.23
41.12
34.32
+ OrthoMerge-G (SVD)
11.09
43.90
37.96
42.06
42.41
35.48
+ OrthoMerge-G (Newton-Schulz)
12.08
44.55
38.05
42.41
43.07
36.03
Appendix
Table 7: Efficiency and performance comparison between the SVD-based Procrustes solver and the Newton-Schulz variant on Llama-3.2-3B. The Newton-Schulz variant reduces runtime from 19m22s to 1m14s per task.
Model
MATH500
HUMANEVAL+
ScienceQA
CommonsenseQA
Social-IQA
Avg.
Qwen2.5-3B
26.80
6.50
73.97
76.99
49.95
46.84
LoRA
Task-specific FT
27.80
14.21
92.04
82.96
52.41
57.83
TA
30.80
12.20
83.86
80.75
53.43
52.21
TIES
31.60
21.03
83.59
81.24
53.89
54.27
DARE
33.40
10.55
84.35
80.84
53.38
52.50
TSV-M
7.80
36.09
90.47
79.61
51.94
53.18
Appendix
Table 8: Performance of merging task-specific models fine-tuned from Qwen2.5-3B using LoRA and OFT across task-specific benchmarks.
Model
MATH500
HUMANEVAL+
Avg.
Llama-3.1-8B
17.80
21.52
19.66
Task-specific FT
19.00
38.54
28.77
TA
23.60
43.65
33.63
TIES
23.20
43.04
33.12
DARE
22.80
40.55
31.68
TSV-M
21.80
42.32
32.06
Appendix
Table 9: Comparison of different model merging methods across 2 tasks.
Model
MATH500
HUMANEVAL+
ScienceQA
Avg.
Llama-3.1-8B
17.80
21.52
6.12
15.15
Task-specific FT
19.00
38.54
26.75
28.09
TA
25.00
36.46
17.94
26.47
TIES
24.60
40.36
15.56
26.84
DARE
24.80
35.97
15.38
25.38
TSV-M
25.20
39.26
24.15
29.54
Appendix
Table 10: Comparison of different model merging methods across 3 tasks.
Model
MATH500
HUMANEVAL+
ScienceQA
CommonsenseQA
Avg.
Llama-3.1-8B
17.80
21.52
6.12
70.60
29.01
Task-specific FT
19.00
38.54
26.75
82.47
41.69
TA
22.80
33.96
20.19
76.17
38.28
TIES
23.80
41.28
19.56
73.87
39.63
DARE
22.80
32.32
17.85
76.66
37.41
TSV-M
19.20
38.23
18.26
82.96
39.66
Appendix
Table 11: Comparison of different model merging methods across 4 tasks.
Figure 6: Distribution of the norms of orthogonal components of the models finetuned from Llama-3.2-3B from Section 5.2 , obtained using the Global Decoupling strategy.
Figure 7: Distribution of the norms of orthogonal components of the models finetuned from Llama-3.2-3B from Section 5.2 , obtained using the Conflict-Aware Decoupling strategy.
Figure 8: Distribution of the norms of orthogonal components of the models finetuned from Qwen2.5-VL-Instruct from Section 5.2 , obtained using the Global Decoupling strategy.
Figure 9: Distribution of the norms of orthogonal components of the models finetuned from Qwen2.5-VL-Instruct from Section 5.2 , obtained using the Conflict-Aware Decoupling strategy.