Parameter-Efficient Fine-Tuning (PEFT) has become a dominant paradigm for deploying LLMs in multi-task scenarios due to its extreme parameter efficiency. While Mixture-of-Experts (MoE) based LoRA variants have achieved promising results by dynamically routing tokens to different low-rank experts, they largely overlook the hierarchical nature of task complexity. Existing methods typically employ experts with uniform architectures, limiting their ability to capture diverse feature granularities required by distinct tasks--where some tasks demand high-level semantic abstraction while others require fine-grained syntactic manipulation. To bridge this gap, we propose Expert Pyramid Tuning (EPT), a novel architecture that integrates the multi-scale feature pyramid concept from computer vision into the realm of PEFT. Unlike standard LoRA, EPT decomposes task adaptation into two stages: (1) A shared meta-knowledge Subspace that encodes universal linguistic patterns in low dimensions; (2) A Pyramid Projection Mechanism that utilizes learnable up-projection operators to reconstruct high-dimensional features at varying scales. A task-aware router then dynamically selects the optimal combination of these multi-scale features. Extensive experiments across multiple multi-task benchmarks demonstrate that EPT significantly outperforms SOTA MoE-LoRA variants. Crucially, thanks to the re-parameterization capability of our design, EPT achieves this performance improvement while simultaneously reducing the number of training parameters.
Figures & tables
Rank
MRPC
RTE
SST-2
CoLA
1
89.7
77.6
94.4
60.9
2
89.2
78.7
94.6
60.0
4
88.7
80.5
94.8
61.9
8
89.2
77.6
94.5
63.3
16
89.2
80.1
94.4
62.3
32
89.5
79.1
94.5
60.5
Table 1: LoRA-based Fine-tuning Performance of T5-base with varying ranks on different tasks.
Figure 1: The overall framework of EPT. The overall architecture of EPT resembles a parameter pyramid, consisting of a shared meta-knowledge subspace and multiple deconvolutional projection layers of varying dimensions.
Methods
params/task
MNLI
QQP
QNLI
SST-2
STS-B
MRPC
RTE
CoLA
AVG
Finetuning
28M
85.7
91.1
92.0
92.5
88.8
90.2
75.4
54.9
83.8
Adapters
1.8M
86.3
90.5
93.2
93.0
89.9
90.2
70.3
61.5
84.4
PT
9.6k
85.6
90.6
93.2
93.9
89.9
86.3
67.6
55.3
82.8
LoRAr=8
0.39M
85.8
89.2
93.1
93.2
90.4
89.9
76.3
62.8
85.1
LoRAr=16
0.78M
84.9
89.6
93.0
93.7
90.4
88.7
80.6
63.9
85.6
HyperFomer
638K
85.7
90.0
93.0
94.0
89.7
87.2
75.4
63.7
84.8
Table 2: Performance on GLUE benchmark. For STS-B, we report Pearson correlation coefficients. For CoLA, we report Matthews correlation. For all other tasks, we report Accuracy. The best score , and second best score are red, and orange, respectively. Subscripts denote the standard deviation of the average score over three runs.
Methods
params/task
BoolQ
OBQA
ARC-E
ARC-C
AVG
LoRA
2.1M
74.0
74.0
80.9
63.5
73.1
MultiLoRA
10M
76.5
68.2
81.2
61.9
72.0
MOELoRA
4.5M
73.3
67.8
71.5
57.5
67.5
MoRE
4.5M
74.7
80.5
80.0
64.5
74.9
EPT
3.3M
76.1
78.4
81.4
66.2
75.5
Table 3: Accuracy of all methods on Commonsense Reasoning tasks. The backbone is Llama2-7B.
Figure 2: We present a visualization of the dataset analysis. Figure (a) shows the activation of each expert during the GLUE benchmark testing. Figure (b) displays the task embeddings from the final layer of the self-attention module.
Methods
MNLI
QQP
QNLI
SST-2
STS-B
MRPC
RTE
CoLA
AVG
EPT-2
86.0
90.4
93.2
94.8
90.2
90.2
83.5
63.7
86.5
EPT-4
86.1
90.4
93.2
94.4
89.8
88.2
82.0
65.7
86.2
EPT-6
85.9
90.3
93.4
93.7
90.3
90.7
74.8
68.0
85.9
EPT-8
86.2
90.4
93.5
94.3
90.4
89.2
79.1
67.6
86.3
EPT-2468
86.4
90.2
93.6
94.5
90.0
90.7
82.0
68.9
87.0
Table 4: Ablation Experiment Results. We conduct ablation experiments on initialization methods (AB init), Top-K, and the Adaptive LoRA Pruner (ALP) modules.
AB init
Top-K
ALP
MNLI
QQP
QNLI
SST-2
STS-B
MRPC
RTE
CoLA
AVG
✗
✗
✗
85.5
90.0
92.7
93.7
89.0
90.2
81.3
65.7
86.0
✗
✗
✓
85.6
90.2
93.1
93.9
89.9
90.7
77.7
68.7
86.2
✓
✗
✓
86.3
90.4
93.2
94.3
90.2
88.7
80.6
68.1
86.5
✓
✓
✗
86.1
90.0
93.5
93.9
90.4
88.7
82.0
65.2
86.2
✗
✓
✓
86.2
90.2
93.4
94.8
90.3
90.2
79.1
69.4
86.7
✓
✓
✓
86.4
90.2
93.6
94.5
90.0
90.7
82.0
68.9
87.0
Table 5: Ablation Experiment Results. We conduct ablation experiments on initialization methods (AB init), Top-K, and the Adaptive LoRA Pruner (ALP) modules.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Metric
#Train
#Valid
#Test
#Label
CoLA
Mcc
8.5k
1,043
1,063
2
SST-2
Acc
67k
872
1.8k
2
MRPC
Acc
3.7k
408
1.7k
2
QQP
Acc/F1
364k
40.4k
391k
2
STS-B
Corr
5.7k
1.5k
1.4k
1
MNLI
Acc(m/mm)
393k
20k
20k
3
Appendix
Table 6: Dataset Sizes and Evaluation Metrics in the GLUE Benchmark. "Mcc," "Acc," "F1," and "Corr" denote the Matthews correlation coefficient, accuracy, F1 score, and Pearson correlation coefficient, respectively. "Acc(m/mm)" indicates accuracy results for matched and mismatched datasets within MNLI.
Dataset
Metric
#Train
#Valid
#Test
#Label
BoolQ
Acc
9.4k
3.3k
3.2k
2
OBQA
Acc
4.9k
500
500
4
ARC-E
Acc
2.2k
570
2.4k
4
ARC-C
Acc
1.1k
299
1.2k
4
Appendix
Table 7: Dataset Sizes and Evaluation Metrics in the commonsense question-answering datasets, where BoolQ is a binary classification (yes/no) task, while OBQA and ARC are multiple-choice questions. ARC is divided into two subsets: Easy (ARC-E) and Challenge (ARC-C).
School of Artificial Intelligence, Jilin University · 2Southern University of Science and Technology · 3International Center of Future Science, Jilin University +1