Parameter-efficient fine-tuning (PEFT) has emerged as a crucial paradigm for adapting large language models (LLMs) under constrained computational budgets. However, standard PEFT methods often struggle in multi-task fine-tuning settings, where diverse optimization objectives induce task interference and limited parameter budgets lead to representational deficiency. While recent approaches incorporate mixture-of-experts (MoE) to alleviate these issues, they predominantly operate in the spatial domain, which may introduce structural redundancy and parameter overhead. To overcome these limitations, we reformulate adaptation in the spectral domain. Our spectral analysis reveals that different tasks exhibit distinct frequency energy distributions, and that LLM layers display heterogeneous frequency sensitivities. Motivated by these insights, we propose FourierMoE, which integrates the MoE architecture with the inverse discrete Fourier transform (IDFT) for frequency-aware adaptation. Specifically, FourierMoE employs a frequency-adaptive router to dispatch tokens to experts specialized in distinct frequency bands. Each expert learns a set of conjugate-symmetric complex coefficients, preserving complete phase and amplitude information while theoretically guaranteeing lossless IDFT reconstruction into real-valued spatial weights. Extensive evaluations across 28 benchmarks, multiple model architectures, and scales demonstrate that FourierMoE consistently outperforms competitive baselines in both single-task and multi-task settings while using significantly fewer trainable parameters. These results highlight the promise of spectral-domain expert adaptation as an effective and parameter-efficient paradigm for LLM fine-tuning.
Figures & tables
Figure 1: Spectral analysis across layers and tasks. (Left) The power spectral density of the RoBERTa-large weights shows layer-wise differences, with early layers exhibiting clear high-frequency spikes and deeper layers showing a progressively smoother spectrum. (Right) For different GLUE tasks (CoLA, SST-2, QQP, and MRPC), the spectra of hidden representations from the eighth layer of RoBERTa-large display distinct frequency energy distributions, revealing task-specific preferences. These observations suggest that effective adaptation requires frequency-specific modulation tailored to different layers and downstream tasks.
Figure 2: The overall framework of FourierMoE, which reparameterizes LLM weight updates ΔW in the spectral domain. A frequency-adaptive router GΦ(x) dynamically assigns tokens to experts specialized in distinct frequency bands, which mitigates task interference via parameter isolation. Each expert learns conjugate-symmetric complex coefficients, enabling complete spectral representation while theoretically guaranteeing real-valued weight updates after IDFT.
Model
Method
# Params(%)
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Average
ChatGPT
/
/
73.10
85.40
68.50
78.50
66.10
89.80
79.90
74.80
77.01
LLaMA-2 7B
LoRA
0.84
69.80
79.90
79.50
83.60
82.60
79.80
64.70
81.00
77.61
DoRA
0.84
71.80
83.10
79.90
89.10
83.00
84.50
71.00
81.20
80.45
PiSSA
0.84
67.60
78.10
78.40
76.60
78.00
75.80
60.20
75.60
73.79
MiLoRA
0.84
67.60
83.80
80.10
88.20
82.00
82.80
68.80
80.60
79.24
LoRA-Dash
0.84
71.00
75.70
79.30
91.10
78.60
84.20
69.80
78.80
78.56
Table 1: Performance comparison of single-PEFT and MoPE methods on eight commonsense reasoning benchmarks using LLaMA-2 7B and Gemma 7B in a multi-task setting. Accuracy is reported for all benchmarks. Bold denotes the best result.
Model
Method
# Params(%)
AddSub
AQuA
GSM8K
MultiArith
SingleEQ
SVAMP
AVG.
LLaMA-3 8B
LoRA
0.12
84.56
25.72
57.22
91.22
92.26
70.17
70.19
DoRA
0.12
85.95
26.19
56.52
89.67
92.62
70.40
70.23
MELoRA
0.12
85.82
24.41
55.34
87.83
91.54
71.20
69.36
HydraLoRA
0.11
86.08
25.98
55.50
91.00
91.14
68.10
69.63
FourierMoE
0.01
87.85
33.07
60.80
93.16
91.54
73.00
73.24
Qwen2.5-14B
LoRA
0.12
91.90
34.65
74.37
96.33
92.91
86.40
79.43
Table 2: Performance comparison of single-PEFT and MoPE methods on six math reasoning benchmarks using LLaMA-3 8B and Qwen2.5-14B in a multi-task setting. Accuracy is reported for all benchmarks. Bold denotes the best result.
Method
# Params (%)
Cars
DTD
EuroSAT
GTSRB
RESISC45
SUN397
SVHN
Average
FFT
100
60.33
73.88
98.96
98.30
93.65
53.84
96.78
82.25
FFT MoE
770
66.39
75.53
98.59
98.50
94.38
60.34
97.09
84.40
LoRA (rank8)
1.49
41.02
70.15
98.66
96.51
90.38
47.51
95.39
77.09
LoRA (rank16)
2.99
46.51
72.07
98.74
98.04
92.08
51.63
96.00
79.30
LoRA (rank32)
5.98
50.13
72.87
98.88
98.13
92.87
53.65
96.55
80.44
DoRA
1.49
40.75
71.91
98.89
97.71
90.19
47.54
95.46
77.49
Table 3: Performance comparison of FFT, single-PEFT, and MoPE methods on seven image classification benchmarks using CLIP ViT-B/32 in a single-task setting. Accuracy is reported for all benchmarks. Bold denotes the best result.
Method
# Params (%)
CoLA
SST-2
MRPC
QQP
MNLI
QNLI
RTE
Average
FFT
100
84.27
95.98
85.29
91.58
89.83
94.49
84.84
89.47
FFT MoE
698
86.02
96.22
85.05
92.20
90.20
95.10
84.48
89.90
LoRA
4.00
83.41
95.64
83.33
90.06
89.00
93.28
84.47
88.46
DoRA
4.00
85.33
95.99
84.07
91.24
89.52
93.54
84.48
89.17
PiSSA
4.00
69.12
95.98
82.84
91.24
88.94
93.59
73.29
85.00
MiLoRA
4.00
84.65
96.10
86.02
91.33
89.51
94.12
84.83
89.51
Table 4: Performance comparison of FFT, single-PEFT, and MoPE methods on seven benchmarks from GLUE using RoBERTa-large in a single-task setting. Accuracy is reported for all benchmarks. Bold denotes the best result.
Figure 3: Component ablation study of FourierMoE on Cars, DTD, and SUN397. Accuracy is reported to assess the contribution of each component. Removing any component results in a performance drop compared to the full FourierMoE implementation.
Figure 4: Expert scaling analysis on the MRPC, QNLI, and RTE datasets. We report the accuracy scores for each dataset. Left: Impact of the trainable coefficient count n per expert. Center: Scalability with respect to the total number of experts Z (fixed activation k=2 ). Right: Effect of the active expert count k given a fixed total pool size of Z=8 .
Figure 5: Visualization of expert-wise coefficient distributions across different layers of the fine-tuned model. Each expert’s coefficients consistently exhibit concentrations across layers, indicating stable specialization.
Setting
CoLA
MRPC
RTE
0.06
83.03
90.44
86.28
0.12
85.52
91.91
89.53
0.24
83.89
89.71
87.36
0.48
83.31
88.48
86.28
0.96
82.74
88.95
84.47
Table 5: Performance comparison of FourierMoE under different frequency bandwidths ( W ) on CoLA, MRPC, and RTE.
Setting
Cars
DTD
SUN397
0.001
64.44
77.18
66.41
0.005
64.32
77.44
65.88
0.01
64.25
77.12
66.00
0.05
64.00
76.91
65.96
0.1
64.32
76.80
65.81
Table 6: Performance comparison of FourierMoE using various load balancing loss weight ( λ ) on Cars, DTD, and SUN397.
Metric
LoRA
TopLoRA
GOAT
FourierMoE
# Trainable Params
2.38 M
2.36 M
16.98 M
1.49 M
# GPU Memory
12.24 GB
16.90 GB
17.12 GB
11.61 GB
# Training FLOPs
340.02
258.84
163.46
235.23
Training Latency
13.13 s
145.69 s
867 s
58 s
Inference Latency
53.4 ms
397 ms
840 ms
313 ms
Performance
82.31
85.20
85.56
89.53
Table 7: Efficiency comparison of methods fine-tuning RoBERTa-large on RTE using a single NVIDIA H100 (80GB) GPU. Note that training FLOPs ( ×109 ) are calculated per sample, training latency is reported per epoch, and inference latency is measured with a batch size of 32.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Domain
# Train
# Test
Task Type
Answer
ARC-E
Natural Science
2.3K
2.3k
Question Answering
Option
ARC-C
Natural Science
1.1K
1.1k
Question Answering
Option
BoolQ
Wikipedia
9.4K
3.2k
Text Classification
Yes/No
OpenBookQA
Science Facts
5.0K
500
Question Answering
Option
PIQA
Physical Interaction
16.1K
1.8k
Question Answering
Option
SIQA
Social Interaction
33.4K
1.9k
Question Answering
Option
Appendix
Table 8: Statistics of the sub-datasets comprising the Commonsense170K dataset used for multi-task fine-tuning.
Dataset
Domain
# Train
# Test
Task Type
Answer
AddSub
Arithmetic Word Problems
-
395
Math Word Problem Solving
Number
AQuA
GRE/GMAT Math
100K
254
Multiple-Choice Question Answering
Option
GSM8K
Grade School Math Word Problems
8.8K
1.3k
Math Word Problem Solving
Number
MultiArith
Multi-step Math Word Problems
-
600
Math Word Problem Solving
Number
SingleEQ
Grade School Algebra Word Problems
-
508
Math Word Problem Solving
Number
SVAMP
Arithmetic Word Problems
-
1k
Math Word Problem Solving
Number
Appendix
Table 9: Statistics of sub-datasets comprising the Math10K dataset used for multi-task fine-tuning. In the Math10K dataset, only GSM8K and AQuA provide training samples. The training sample counts of other datasets are marked with the “-” symbol, indicating that we do not use their training data for fine-tuning.
Dataset
Domain
# Train
# Test
Task Type
Answer
CoLA
Miscellaneous
8.5K
1.0k
Acceptability
Label Text
SST-2
Movie Reviews
67.3K
1.8k
Sentiment Analysis
Label Text
MRPC
News
3.6K
1.7k
Paraphrase
Label Text
QQP
Social QA
364K
391K
Paraphrase
Label Text
MNLI
Miscellaneous
393K
19.6K
Natural Language Inference
Label Text
QNLI
Wikipedia
105K
5.4k
Natural Language Inference
Label Text
Appendix
Table 10: Statistics of the seven GLUE datasets for single-task fine-tuning.
Dataset
Domain
# Train
# Test
Task Type
Answer
StanfordCars
Automobiles
8.1K
8.0K
Image Classification
Class Label
DTD
Textures
3.7K
1.8K
Image Classification
Class Label
EuroSAT
Satellite
21.6K
2.7K
Image Classification
Class Label
GTSRB
Traffic Signs
26.6K
12.6K
Image Classification
Class Label
RESISC45
Aerial Imagery
18.9K
6.3K
Image Classification
Class Label
SUN397
Scenes
76.1K
21.7K
Image Classification
Class Label
Appendix
Table 11: Statistics of the seven image classification datasets for single-task fine-tuning.
Hyperparameters
Commonsense Reasoning
LLaMA-2 7B
Gemma 7B
Optimizer
AdamW
LR
8e-2
1.5e-1
LR Scheduler
Linear
Max Seq. Len.
256
Batch Size
16
Appendix
Table 12: Detailed configurations used for fine-tuning LLaMA-2 7B and Gemma 7B models on commonsense reasoning tasks.
Hyperparameters
Math Reasoning
LLaMA-3 8B
Qwen2.5-14B
Optimizer
AdamW
LR
2e-4
1e-4
LR Scheduler
Linear
Max Seq. Len.
1024
Batch Size
8
Appendix
Table 13: Detailed configurations used for fine-tuning LLaMA-3 8B and Qwen2.5-14B models on math reasoning tasks.
Hyperparameters
Cars
DTD
EuroSAT
GTSRB
RESISC45
SUN397
SVHN
Optimizer
AdamW
Expert LR
8e-5
1e-2
5e-2
2e-1
5e-2
5e-2
1.5e-1
Router LR
5e-5
1e-4
2e-4
2e-4
2e-4
2e-4
2e-4
Head LR
3e-4
1e-3
1e-3
1e-3
5e-3
1e-3
1e-3
LR Scheduler
Cosine
Max Seq. Len.
50
Appendix
Table 14: Detailed configurations used for fine-tuning CLIP ViT-B/32 on image classification tasks.
Hyperparameters
CoLA
SST-2
MRPC
QQP
MNLI
QNLI
RTE
Optimizer
AdamW
Expert LR
5e-2
1.2e-1
9e-2
1e-1
1e-5
9e-2
1e-1
Router LR
1.5e-4
1e-4
2e-4
3.9e-4
2e-4
1.5e-4
1.5e-4
Head LR
1e-2
5e-4
1e-3
8.4e-4
5e-3
1e-3
6.5e-3
LR Scheduler
Linear
Max Seq. Len.
256
128
512
128
512
512
512
Appendix
Table 15: Detailed configurations used for fine-tuning RoBERTa-large on NLU tasks.
Figure 6: Distributions of expert spectral coefficients under different frequency bandwidths W . Moderate bandwidths expand per-expert frequency coverage while preserving well-separated spectral structures, whereas overly large W induces substantial overlap, leading to degraded specialization.
As large language models (LLMs) scale rapidly, dense full-parameter adaptation becomes increasingly expensive, motivating sparse and modular architectures such as Mixture-of-Experts (MoE) models. This shift raises a key question for parameter-efficient fine-tuning (PEFT): at what granularity should parameters be selected and updated? Existing PEFT methods such as LoRA operate on predefined weight matrices, while expert-level sparse tuning methods update entire selected experts. However, we observe that activated experts are internally sparse, with only a small fraction of intermediate channels strongly responding to downstream tasks, indicating that expert-level adaptation is still too coarse. We propose NSFT (Neural Sub-expert Fine-Tuning), a fine-grained PEFT framework that refines MoE adaptation from experts to sub-experts. NSFT decomposes each expert along the intermediate dimension into structured channel groups and selects task-relevant sub-experts by combining routing importance with intra-expert activation saliency. To optimize sparse partial updates, NSFT further introduces learning-rate scaling and dynamic gradient scaling to compensate for the reduced effective update magnitude. Experiments on OLMoE and Ling-mini-2.0 across challenging domain-specific tasks and general benchmarks show that NSFT consistently outperforms representative PEFT and expert-level sparse tuning baselines, while using substantially fewer trainable parameters and preserving competitive general capability. These results suggest that sub-expert-level adaptation is a more precise and efficient PEFT paradigm for MoE LLMs.
Parameter-efficient fine-tuning (PEFT) reparameterizes weight updates in a fixed basis: low-rank adapters operate in the spatial domain, while a recent line of spectral methods operates in a fixed Fourier domain. We argue that the choice of domain is itself a design degree of freedom that should be learned, and that no single basis is optimal across tasks, layers, or tokens. We introduce Fractional-Fourier Mixture of Experts, a mixture-of-experts adapter in which every expert carries a learnable fractional-Fourier order that continuously interpolates between the spatial domain (recovering vanilla LoRA) and the Fourier domain (recovering a spectral adapter). Routing tokens through experts that occupy different points on this spatial-spectral continuum lets the model place each low-rank update in the domain where it is most compact, and -- because fractional-Fourier operators of different orders are mutually incoherent -- makes the experts naturally decorrelated, which reduces interference and improves multi-task composition. The order is a single scalar per expert, trained with a separate optimizer, and the transform is computed with an O(dlogd) chirp--FFT surrogate, so Fractional-Fourier Mixture of Experts adds negligible cost over standard MoE-LoRA. Across commonsense, mathematical, code, and knowledge benchmarks on LLaMA-3.1-8B and Qwen2.5-7B, Fractional-Fourier Mixture of Experts improves over strong MoE-LoRA and spectral baselines -- including FlyLoRA, FourierMoE, and HMoRA -- while keeping the active-parameter budget small, and analysis shows that the learned orders specialize by task and layer in interpretable ways.
Tom Saliencro, Maya Lindqvist, Rohan Desai +2
University of California, Irvine · University of Washington
Parameter-Efficient Fine-Tuning (PEFT) has become a dominant paradigm for deploying LLMs in multi-task scenarios due to its extreme parameter efficiency. While Mixture-of-Experts (MoE) based LoRA variants have achieved promising results by dynamically routing tokens to different low-rank experts, they largely overlook the hierarchical nature of task complexity. Existing methods typically employ experts with uniform architectures, limiting their ability to capture diverse feature granularities required by distinct tasks--where some tasks demand high-level semantic abstraction while others require fine-grained syntactic manipulation. To bridge this gap, we propose Expert Pyramid Tuning (EPT), a novel architecture that integrates the multi-scale feature pyramid concept from computer vision into the realm of PEFT. Unlike standard LoRA, EPT decomposes task adaptation into two stages: (1) A shared meta-knowledge Subspace that encodes universal linguistic patterns in low dimensions; (2) A Pyramid Projection Mechanism that utilizes learnable up-projection operators to reconstruct high-dimensional features at varying scales. A task-aware router then dynamically selects the optimal combination of these multi-scale features. Extensive experiments across multiple multi-task benchmarks demonstrate that EPT significantly outperforms SOTA MoE-LoRA variants. Crucially, thanks to the re-parameterization capability of our design, EPT achieves this performance improvement while simultaneously reducing the number of training parameters.
Jia-Chen Zhang, Zhen-Wei Yan, Yu-Jie Xiong +1
School of Electronic and Electrical Engineering, Shanghai University of Engineering Science 333 Longteng Road, Shanghai, China