FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models

Juyong Jiang; Fan Wang; Hong Qi; Sunghun Kim; Jing Tang

cs.LGApr 2, 2026

FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models

Authors: Juyong Jiang, Fan Wang, Hong Qi, Sunghun Kim, Jing Tang

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology

Abstract

Parameter-efficient fine-tuning (PEFT) has emerged as a crucial paradigm for adapting large language models (LLMs) under constrained computational budgets. However, standard PEFT methods often struggle in multi-task fine-tuning settings, where diverse optimization objectives induce task interference and limited parameter budgets lead to representational deficiency. While recent approaches incorporate mixture-of-experts (MoE) to alleviate these issues, they predominantly operate in the spatial domain, which may introduce structural redundancy and parameter overhead. To overcome these limitations, we reformulate adaptation in the spectral domain. Our spectral analysis reveals that different tasks exhibit distinct frequency energy distributions, and that LLM layers display heterogeneous frequency sensitivities. Motivated by these insights, we propose FourierMoE, which integrates the MoE architecture with the inverse discrete Fourier transform (IDFT) for frequency-aware adaptation. Specifically, FourierMoE employs a frequency-adaptive router to dispatch tokens to experts specialized in distinct frequency bands. Each expert learns a set of conjugate-symmetric complex coefficients, preserving complete phase and amplitude information while theoretically guaranteeing lossless IDFT reconstruction into real-valued spatial weights. Extensive evaluations across 28 benchmarks, multiple model architectures, and scales demonstrate that FourierMoE consistently outperforms competitive baselines in both single-task and multi-task settings while using significantly fewer trainable parameters. These results highlight the promise of spectral-domain expert adaptation as an effective and parameter-efficient paradigm for LLM fine-tuning.

Figures & tables

Figure 1: Spectral analysis across layers and tasks. (Left) The power spectral density of the RoBERTa-large weights shows layer-wise differences, with early layers exhibiting clear high-frequency spikes and deeper layers showing a progressively smoother spectrum. (Right) For different GLUE tasks (CoLA, SST-2, QQP, and MRPC), the spectra of hidden representations from the eighth layer of RoBERTa-large display distinct frequency energy distributions, revealing task-specific preferences. These observations suggest that effective adaptation requires frequency-specific modulation tailored to different layers and downstream tasks.

Figure 2: The overall framework of FourierMoE, which reparameterizes LLM weight updates $\Delta\mathbf{W}$ in the spectral domain. A frequency-adaptive router $\mathcal{G}_{\Phi}(x)$ dynamically assigns tokens to experts specialized in distinct frequency bands, which mitigates task interference via parameter isolation. Each expert learns conjugate-symmetric complex coefficients, enabling complete spectral representation while theoretically guaranteeing real-valued weight updates after IDFT.

Model	Method	# Params(%)	BoolQ	PIQA	SIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Average
ChatGPT	/	/	73.10	85.40	68.50	78.50	66.10	89.80	79.90	74.80	77.01
LLaMA-2 7B	LoRA	0.84	69.80	79.90	79.50	83.60	82.60	79.80	64.70	81.00	77.61
	DoRA	0.84	71.80	83.10	79.90	89.10	83.00	84.50	71.00	81.20	80.45
	PiSSA	0.84	67.60	78.10	78.40	76.60	78.00	75.80	60.20	75.60	73.79
	MiLoRA	0.84	67.60	83.80	80.10	88.20	82.00	82.80	68.80	80.60	79.24
	LoRA-Dash	0.84	71.00	75.70	79.30	91.10	78.60	84.20	69.80	78.80	78.56

Table 1: Performance comparison of single-PEFT and MoPE methods on eight commonsense reasoning benchmarks using LLaMA-2 7B and Gemma 7B in a multi-task setting. Accuracy is reported for all benchmarks. Bold denotes the best result.

Model	Method	# Params(%)	AddSub	AQuA	GSM8K	MultiArith	SingleEQ	SVAMP	AVG.
LLaMA-3 8B	LoRA	0.12	84.56	25.72	57.22	91.22	92.26	70.17	70.19
	DoRA	0.12	85.95	26.19	56.52	89.67	92.62	70.40	70.23
	MELoRA	0.12	85.82	24.41	55.34	87.83	91.54	71.20	69.36
	HydraLoRA	0.11	86.08	25.98	55.50	91.00	91.14	68.10	69.63
	FourierMoE	0.01	87.85	33.07	60.80	93.16	91.54	73.00	73.24
Qwen2.5-14B	LoRA	0.12	91.90	34.65	74.37	96.33	92.91	86.40	79.43

Table 2: Performance comparison of single-PEFT and MoPE methods on six math reasoning benchmarks using LLaMA-3 8B and Qwen2.5-14B in a multi-task setting. Accuracy is reported for all benchmarks. Bold denotes the best result.

Method	# Params (%)	Cars	DTD	EuroSAT	GTSRB	RESISC45	SUN397	SVHN	Average
FFT	100	60.33	73.88	98.96	98.30	93.65	53.84	96.78	82.25
FFT MoE	770	66.39	75.53	98.59	98.50	94.38	60.34	97.09	84.40
LoRA (rank8)	1.49	41.02	70.15	98.66	96.51	90.38	47.51	95.39	77.09
LoRA (rank16)	2.99	46.51	72.07	98.74	98.04	92.08	51.63	96.00	79.30
LoRA (rank32)	5.98	50.13	72.87	98.88	98.13	92.87	53.65	96.55	80.44
DoRA	1.49	40.75	71.91	98.89	97.71	90.19	47.54	95.46	77.49

Table 3: Performance comparison of FFT, single-PEFT, and MoPE methods on seven image classification benchmarks using CLIP ViT-B/32 in a single-task setting. Accuracy is reported for all benchmarks. Bold denotes the best result.

Method	# Params (%)	CoLA	SST-2	MRPC	QQP	MNLI	QNLI	RTE	Average
FFT	100	84.27	95.98	85.29	91.58	89.83	94.49	84.84	89.47
FFT MoE	698	86.02	96.22	85.05	92.20	90.20	95.10	84.48	89.90
LoRA	4.00	83.41	95.64	83.33	90.06	89.00	93.28	84.47	88.46
DoRA	4.00	85.33	95.99	84.07	91.24	89.52	93.54	84.48	89.17
PiSSA	4.00	69.12	95.98	82.84	91.24	88.94	93.59	73.29	85.00
MiLoRA	4.00	84.65	96.10	86.02	91.33	89.51	94.12	84.83	89.51

Table 4: Performance comparison of FFT, single-PEFT, and MoPE methods on seven benchmarks from GLUE using RoBERTa-large in a single-task setting. Accuracy is reported for all benchmarks. Bold denotes the best result.

Figure 3: Component ablation study of FourierMoE on Cars, DTD, and SUN397. Accuracy is reported to assess the contribution of each component. Removing any component results in a performance drop compared to the full FourierMoE implementation.

Figure 4: Expert scaling analysis on the MRPC, QNLI, and RTE datasets. We report the accuracy scores for each dataset. Left: Impact of the trainable coefficient count

n

per expert. Center: Scalability with respect to the total number of experts

Z

(fixed activation

k=2

). Right: Effect of the active expert count

k

given a fixed total pool size of

Z=8

Figure 5: Visualization of expert-wise coefficient distributions across different layers of the fine-tuned model. Each expert’s coefficients consistently exhibit concentrations across layers, indicating stable specialization.

Setting	CoLA	MRPC	RTE
0.06	83.03	90.44	86.28
0.12	85.52	91.91	89.53
0.24	83.89	89.71	87.36
0.48	83.31	88.48	86.28
0.96	82.74	88.95	84.47

Table 5: Performance comparison of FourierMoE under different frequency bandwidths (

\mathcal{W}

) on CoLA, MRPC, and RTE.

Setting	Cars	DTD	SUN397
0.001	64.44	77.18	66.41
0.005	64.32	77.44	65.88
0.01	64.25	77.12	66.00
0.05	64.00	76.91	65.96
0.1	64.32	76.80	65.81

Table 6: Performance comparison of FourierMoE using various load balancing loss weight (

\lambda

) on Cars, DTD, and SUN397.

Metric	LoRA	TopLoRA	GOAT	FourierMoE
# Trainable Params	2.38 M	2.36 M	16.98 M	1.49 M
# GPU Memory	12.24 GB	16.90 GB	17.12 GB	11.61 GB
# Training FLOPs	340.02	258.84	163.46	235.23
Training Latency	13.13 s	145.69 s	867 s	58 s
Inference Latency	53.4 ms	397 ms	840 ms	313 ms
Performance	82.31	85.20	85.56	89.53

Table 7: Efficiency comparison of methods fine-tuning RoBERTa-large on RTE using a single NVIDIA H100 (80GB) GPU. Note that training FLOPs (

\times 10^{9}

) are calculated per sample, training latency is reported per epoch, and inference latency is measured with a batch size of 32.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Domain	# Train	# Test	Task Type	Answer
ARC-E	Natural Science	2.3K	2.3k	Question Answering	Option
ARC-C	Natural Science	1.1K	1.1k	Question Answering	Option
BoolQ	Wikipedia	9.4K	3.2k	Text Classification	Yes/No
OpenBookQA	Science Facts	5.0K	500	Question Answering	Option
PIQA	Physical Interaction	16.1K	1.8k	Question Answering	Option
SIQA	Social Interaction	33.4K	1.9k	Question Answering	Option

Appendix

Table 8: Statistics of the sub-datasets comprising the Commonsense170K dataset used for multi-task fine-tuning.

Dataset	Domain	# Train	# Test	Task Type	Answer
AddSub	Arithmetic Word Problems	-	395	Math Word Problem Solving	Number
AQuA	GRE/GMAT Math	100K	254	Multiple-Choice Question Answering	Option
GSM8K	Grade School Math Word Problems	8.8K	1.3k	Math Word Problem Solving	Number
MultiArith	Multi-step Math Word Problems	-	600	Math Word Problem Solving	Number
SingleEQ	Grade School Algebra Word Problems	-	508	Math Word Problem Solving	Number
SVAMP	Arithmetic Word Problems	-	1k	Math Word Problem Solving	Number

Appendix

Table 9: Statistics of sub-datasets comprising the Math10K dataset used for multi-task fine-tuning. In the Math10K dataset, only GSM8K and AQuA provide training samples. The training sample counts of other datasets are marked with the “-” symbol, indicating that we do not use their training data for fine-tuning.

Dataset	Domain	# Train	# Test	Task Type	Answer
CoLA	Miscellaneous	8.5K	1.0k	Acceptability	Label Text
SST-2	Movie Reviews	67.3K	1.8k	Sentiment Analysis	Label Text
MRPC	News	3.6K	1.7k	Paraphrase	Label Text
QQP	Social QA	364K	391K	Paraphrase	Label Text
MNLI	Miscellaneous	393K	19.6K	Natural Language Inference	Label Text
QNLI	Wikipedia	105K	5.4k	Natural Language Inference	Label Text

Appendix

Table 10: Statistics of the seven GLUE datasets for single-task fine-tuning.

Dataset	Domain	# Train	# Test	Task Type	Answer
StanfordCars	Automobiles	8.1K	8.0K	Image Classification	Class Label
DTD	Textures	3.7K	1.8K	Image Classification	Class Label
EuroSAT	Satellite	21.6K	2.7K	Image Classification	Class Label
GTSRB	Traffic Signs	26.6K	12.6K	Image Classification	Class Label
RESISC45	Aerial Imagery	18.9K	6.3K	Image Classification	Class Label
SUN397	Scenes	76.1K	21.7K	Image Classification	Class Label

Appendix

Table 11: Statistics of the seven image classification datasets for single-task fine-tuning.

Hyperparameters	Commonsense Reasoning
	LLaMA-2 7B	Gemma 7B
Optimizer	AdamW
LR	8e-2	1.5e-1
LR Scheduler	Linear
Max Seq. Len.	256
Batch Size	16

Appendix

Table 12: Detailed configurations used for fine-tuning LLaMA-2 7B and Gemma 7B models on commonsense reasoning tasks.

Hyperparameters	Math Reasoning
	LLaMA-3 8B	Qwen2.5-14B
Optimizer	AdamW
LR	2e-4	1e-4
LR Scheduler	Linear
Max Seq. Len.	1024
Batch Size	8

Appendix

Table 13: Detailed configurations used for fine-tuning LLaMA-3 8B and Qwen2.5-14B models on math reasoning tasks.

Hyperparameters	Cars	DTD	EuroSAT	GTSRB	RESISC45	SUN397	SVHN
Optimizer	AdamW
Expert LR	8e-5	1e-2	5e-2	2e-1	5e-2	5e-2	1.5e-1
Router LR	5e-5	1e-4	2e-4	2e-4	2e-4	2e-4	2e-4
Head LR	3e-4	1e-3	1e-3	1e-3	5e-3	1e-3	1e-3
LR Scheduler	Cosine
Max Seq. Len.	50

Appendix

Table 14: Detailed configurations used for fine-tuning CLIP ViT-B/32 on image classification tasks.

Hyperparameters	CoLA	SST-2	MRPC	QQP	MNLI	QNLI	RTE
Optimizer	AdamW
Expert LR	5e-2	1.2e-1	9e-2	1e-1	1e-5	9e-2	1e-1
Router LR	1.5e-4	1e-4	2e-4	3.9e-4	2e-4	1.5e-4	1.5e-4
Head LR	1e-2	5e-4	1e-3	8.4e-4	5e-3	1e-3	6.5e-3
LR Scheduler	Linear
Max Seq. Len.	256	128	512	128	512	512	512

Appendix

Table 15: Detailed configurations used for fine-tuning RoBERTa-large on NLU tasks.

Figure 6: Distributions of expert spectral coefficients under different frequency bandwidths $\mathcal{W}$ . Moderate bandwidths expand per-expert frequency coverage while preserving well-separated spectral structures, whereas overly large $\mathcal{W}$ induces substantial overlap, leading to degraded specialization.

Explore similar work

From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs
Sep 22, 2026Zhentao Tan, Chang Liu, Yao Liu +2Mixture-of-Experts Language Models Sparse Fine-Tuning
Alibaba Group · Southeast University
FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts
Jun 30, 2026Tom Saliencro, Maya Lindqvist, Rohan Desai +2Low-Rank Adaptation Mixture-of-Experts Models
University of California, Irvine · University of Washington
Expert Pyramid Tuning: Efficient Parameter Fine-Tuning for Expertise-Driven Task Allocation
Mar 13, 2026Jia-Chen Zhang, Zhen-Wei Yan, Yu-Jie Xiong +1Parameter-Efficient Fine-Tuning Low-Rank Adaptation
School of Electronic and Electrical Engineering, Shanghai University of Engineering Science 333 Longteng Road, Shanghai, China

Sep 22, 2026J/K move · Enter open · S save

From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs

Zhentao Tan, Chang Liu, Yao Liu, Yue Wu+1

Alibaba Group · Southeast University

Mixture-of-Experts Language Models Sparse Fine-Tuning Selective Fine-Tuning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

As large language models (LLMs) scale rapidly, dense full-parameter adaptation becomes increasingly expensive, motivating sparse and modular architectures such as Mixture-of-Experts (MoE) models. This shift raises a key question for parameter-efficient fine-tuning (PEFT): at what granularity should parameters be selected and updated? Existing PEFT methods such as LoRA operate on predefined weight matrices, while expert-level sparse tuning methods update entire selected experts. However, we observe that activated experts are internally sparse, with only a small fraction of intermediate channels strongly responding to downstream tasks, indicating that expert-level adaptation is still too coarse. We propose NSFT (Neural Sub-expert Fine-Tuning), a fine-grained PEFT framework that refines MoE adaptation from experts to sub-experts. NSFT decomposes each expert along the intermediate dimension into structured channel groups and selects task-relevant sub-experts by combining routing importance with intra-expert activation saliency. To optimize sparse partial updates, NSFT further introduces learning-rate scaling and dynamic gradient scaling to compensate for the reduced effective update magnitude. Experiments on OLMoE and Ling-mini-2.0 across challenging domain-specific tasks and general benchmarks show that NSFT consistently outperforms representative PEFT and expert-level sparse tuning baselines, while using substantially fewer trainable parameters and preserving competitive general capability. These results suggest that sub-expert-level adaptation is a more precise and efficient PEFT paradigm for MoE LLMs.

Abstract

Explore similar work

From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs

FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts

Expert Pyramid Tuning: Efficient Parameter Fine-Tuning for Expertise-Driven Task Allocation