Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Experts
Authors: Buze Zhang, Jinkai Tao, Zilang Zeng, Neil He, Ali Maatouk, Menglin Yang, Rex Ying
Organizations: School of Software Engineering, Xi’an Jiaotong University, China · School of Information, Central University of Finance and Economics, Beijing, China · Department of Artificial Intelligence, Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong, China · Department of Computer Science, University of Illinois Urbana-Champaign, Urbana, IL, USA · Department of Computer Science, Yale university, New Haven, CT, USA
Large Language Models (LLMs) have achieved remarkable progress, with Parameter-Efficient Fine-Tuning (PEFT) emerging as a key technique for downstream task adaptation. However, existing PEFT methods mainly operate in Euclidean space, fundamentally limiting their capacity to capture complex geometric structures inherent in language data. While alternative geometric spaces, like hyperbolic geometries for hierarchical data and spherical manifolds for circular patterns, offer theoretical advantages, forcing representations into a single manifold type ultimately limits expressiveness, even when curvature parameters are learnable. To address this, we propose Mixture of Space (MoS), a unified framework that leverages multiple geometric spaces simultaneously to learn richer, curvature-aware representations. Building on this scheme, we develop MoSLoRA, which extends Low-Rank Adaptation (LoRA) with heterogeneous geometric experts, enabling models to dynamically select or combine appropriate geometric spaces based on input context. Furthermore, to address the computational overhead of frequent manifold switching, we develop a lightweight routing mechanism. Moreover, we provide empirical insights into how curvature optimization impacts training stability and model performance. Our experiments across diverse benchmarks demonstrate that MoSLoRA consistently outperforms strong baselines, achieving up to 5.6% improvement on MATH500 and 15.9% on MAWPS.
Figures & tables
Figure 1: The MoSLoRA architecture contains heterogeneous geometric experts unified in our MoS scheme, with various curvatures. Three geometric expert groups are embedded into the FFN layer and labeled in different colored blocks. The grouped auxiliary balancing enforces balanced routing within each space group while allowing free inter-space transitions, and remains fully reversible and differentiable.
Methods
COLA
MRPC
GSM8k
MATH500
MAWPS
SVAMP
AQuA
Avg.
Base (Qwen2-1.5B)
32.41
69.16
33.13
0.0
43.85
54.33
31.10
37.71
LoRA
87.25
87.30
44.58
15.60
63.85
63.33
28.35
55.75
DoRA
87.15
87.19
42.53
14.20
62.12
62.33
30.71
55.18
LoRA rank↑
83.51
86.43
46.85
13.60
61.92
60.33
33.46
55.16
AdaLoRA
82.45
79.77
51.33
16.60
62.50
58.67
23.23
53.51
MELoRA
86.96
86.55
44.96
16.40
62.69
65.67
32.28
56.50
Table 1: Performance comparison of different tuning methods across different natural language understanding and mathematical reasoning benchmarks. Avg. column reports the average score, and rank↑ indicates that a larger rank parameter is used compared to other methods.
Figure 2: Curvature dynamics of each geometric expert of different layers during training.
Methods
GSM8k
MATH500
MAWPS
SVAMP
AQuA
MRPC
CoLA
RTE
CSQA
OBQA
Avg.
Base
59.06
0.40
60.19
74.67
50.00
56.41
62.80
51.62
32.40
32.40
47.99
LoRA
65.50
25.60
65.58
76.33
38.98
87.71
69.13
89.53
82.80
87.00
68.81
HydraLoRA
64.52
22.40
65.77
75.33
42.91
88.46
85.91
89.89
82.56
89.20
70.69
HypLoRA
64.67
26.40
65.77
78.67
41.34
88.23
87.34
90.25
79.85
87.80
71.03
MoSLoRA
63.68
29.60
81.73
78.00
39.37
88.41
86.19
90.97
82.47
89.00
72.94
Table 2: Performance comparison on math and commonsense reasoning, and language understanding benchmarks based on Qwen2.5-3B.
Methods
COLA
MRPC
GSM8k
MATH500
MAWPS
SVAMP
AQuA
Avg.
MoSLoRA uni
87.15
87.19
46.85
17.80
74.62
64.33
29.13
58.15
MoSLoRA sep
87.63
88.23
47.76
18.00
74.62
64.33
31.50
58.87
Table 3: Performance of different optimizer choices across different MATH and general reasoning benchmarks. Uni and Sep denote using a unified optimizer and separated optimizers, respectively.
Figure 3: Token norm distribution and token frequency distribution of math reasoning and commonsenseQA datasets.
Figure 4: Efficiency comparison between exp/log and our MoS.
Methods
Space
COLA
MRPC
GSM8k
MATH500
MAWPS
SVAMP
AQuA
Avg.
MoSLoRA-S
Spherical
86.86
87.77
44.50
17.60
72.69
64.33
28.35
57.44
MoSLoRA-H
Hyperbolic
85.81
89.33
45.34
17.80
73.46
63.67
34.25
58.52
MoSLoRA-E
Euclidean
87.15
87.19
42.23
16.20
70.77
61.67
31.89
56.73
MoSLoRA (Ours)
Mixture
87.63
88.23
47.76
18.00
74.62
64.33
31.50
58.87
Table 4: Performance comparison of geometric-expert mixture recipes across mathematical and general reasoning benchmarks.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Num Train Epoch
3
Optimizer
AdamW
Weight Decay
0.01
Warmup Ratio
0.1
Learning Rate
3×10−4
Projection Scaling γ
0.001
Appendix
Table 5: Hyperparameters for MoSLoRA.
Dataset
Train
Test
Task Description
MRPC
3,668
1,725
Determine if a pair of sentences are semantically equivalent
CoLA
8,551
1,043
Evaluate the grammatical acceptability of English sentences
RTE
2,490
277
Recognize if a hypothesis is entailed by text
Appendix
Table 6: The detailed statistics of language understanding datasets.
Dataset
Data Number
Task Type
Train
13,262
Mixed
Test
GSM8K
1,319
Question-Answering
MAWPS
520
Question-Answering
SVAMP
300
Question-Answering
MATH500
500
Question-Answering
Appendix
Table 7: The detailed statistics of mathematical reasoning datasets.
Dataset
Train
Test
Task Description
OBQA
4,957
500
Answer science questions using provided facts plus commonsense
CSQA
9,742
1,221
Choose correct option requiring diverse commonsense knowledge
Appendix
Table 8: The detailed statistics of language understanding datasets.
Method
GSM8k
MATH500
MAWPS
SVAMP
AQuA
MRPC
CoLA
RTE
Avg.
LoRA
52.24
13.80
71.33
65.19
32.68
89.39
84.66
90.25
62.44
AdaLoRA
46.70
7.20
64.23
62.67
20.08
84.17
83.03
87.36
56.93
DoRA
48.60
11.20
71.67
64.62
31.89
89.45
85.33
89.89
61.58
MELoRA
48.90
11.80
63.85
70.00
28.35
88.75
84.08
88.81
60.57
HypLoRA
45.34
10.80
64.23
66.33
27.56
87.25
84.08
87.36
59.12
HydraLoRA
51.18
14.80
65.38
71.33
28.35
88.58
86.29
90.97
62.11
Appendix
Table 9: Comparison of parameter-efficient fine-tuning methods across reasoning and language understanding benchmarks. All results are reported on the same backbone model.
Method
Total Time (ms)
Phase
Avg Time (ms)
Avg Memory (MB)
MoS
57.4
Forward
2.1
277.2
Backward
2.1
324.0
Exp/Log
135.0
Forward
2.6
565.6
Backward
8.6
396.0
Appendix
Table 10: Runtime and memory consumption comparison between MoS and Exp/Log geometric mappings. Results are reported for both forward and backward passes.