HeuFouFT: Task-Guided Metaheuristic Coordinate Search for Fourier Fine-Tuning

Ruiheng Wang; Yubo Hou; Yakun Zhu; Tianle Shen; Tao Wan; Zengchang Qin

cs.LGOct 5, 2026

HeuFouFT: Task-Guided Metaheuristic Coordinate Search for Fourier Fine-Tuning

Authors: Ruiheng Wang, Yubo Hou, Yakun Zhu, Tianle Shen, Tao Wan, Zengchang Qin

Organizations: Beihang University, Beijing, China · Shanghai Jiao Tong University, Shanghai, China · VinUniversity, Hanoi, Vietnam

Abstract

We introduce Heuristic-Guided Fourier Fine-Tuning (HeuFouFT), a task-guided framework for selecting trainable frequency coordinates in Fourier fine-tuning. Existing uniform and Gaussian band-pass schemes allocate a limited spectral budget through fixed, task-agnostic rules. HeuFouFT instead searches for coordinates using downstream performance. A coarse intensity map from lightweight block-level probes initializes three metaheuristic optimizers: Genetic Algorithm with Simulated Annealing (GA-SA), Particle Swarm Optimization (PSO), and Cuckoo Search (CS). During search, a Random Forest filters each population so that only the top 30% of candidates proceed to proxy fine-tuning. On E2E with GPT-2-Medium, all three variants outperform random-uniform FourierFT, Gaussian band-pass FourierFT, and LoRA across five metrics. PSO further outperforms LoCA, the best-performing baseline, on four metrics while using 37.6% fewer trainable spectral coefficients. Once coordinates are selected, HeuFouFT requires only 15--18% FLOPs of Full FT. These results show that task-guided search allocates limited spectral capacity more effectively than fixed sampling. Our code is publicly available.

Figures & tables

Figure 1: Log-magnitude spectrum of the layer-1 $Q$ projection $\Delta W$ from GPT-2-Medium LoRA fine-tuned on E2E (rank 8). Red markers indicate the 328 highest-energy frequency bins; blue markers show the 328 uniform random samples; green markers show 328 Gaussian band-pass samples.

$f_{c}$	BLEU	NIST	METEOR	ROUGE-L	CIDEr
0	68.7	8.35	41.3	67.5	1.65
100	67.3	8.23	41.2	67.8	1.66
200	67.6	8.41	40.9	67.4	1.64
300	69.2	8.36	41.4	67.7	1.63
400	68.7	8.26	40.8	67.9	1.64
500	62.5	8.30	40.3	66.7	1.61

Table 1: Gaussian band-pass sampling with different frequency biases. Best values are bold.

Method	BLEU	NIST	METEOR	ROUGE-L	CIDEr	Params. (M)
Full FT	68.9	8.36	40.1	67.2	1.59	354.92
LoRA	$68.4$ [1pt] $\pm.2$	$8.34$ [1pt] $\pm.05$	$41.0$ [1pt] $\pm.1$	$67.1$ [1pt] $\pm.3$	$1.57$ [1pt] $\pm.02$	0.786
LoCA	$\mathbf{70.3}$ [1pt] $\pm.1$	$\mathbf{8.51}$ [1pt] $\pm.03$	$\mathbf{41.6}$ [1pt] $\pm.08$	$\mathbf{68.5}$ [1pt] $\pm.2$	$\mathbf{1.64}$ [1pt] $\pm.01$	0.048
Random Uniform	$66.7$ [1pt] $\pm.2$	$8.24$ [1pt] $\pm.04$	$40.2$ [1pt] $\pm.09$	$66.5$ [1pt] $\pm.2$	$1.54$ [1pt] $\pm.03$	0.048
Gaussian Band-Pass	$69.2$ [1pt] $\pm.1$	$8.36$ [1pt] $\pm.05$	$41.4$ [1pt] $\pm.07$	$67.7$ [1pt] $\pm.2$	$1.63$ [1pt] $\pm.02$	0.048
Vanilla GA	$69.6$ [1pt] $\pm.1$	$8.72$ [1pt] $\pm.03$	$41.4$ [1pt] $\pm.08$	$67.3$ [1pt] $\pm.2$	$1.63$ [1pt] $\pm.01$	0.03

Table 2: GPT-2-Medium performance on E2E. Entries with error terms show the mean and standard deviation over seeds 41, 42, and 43. Best baseline and HeuFouFT results are bolded separately.

Figure 2: Distributions of the selected frequency coordinates, shared across layers, for GA-SA (left), PSO (middle), and CS (right). Dashed circles mark frequency bands.

Figure 3: Frequency-domain metrics of GA-SA, PSO, and CS. Panels show frequency ratios, energy concentration, coverage efficiency, and sampling quality.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value Range	Optimal Value	Design Rationale
Number of Estimators	100–500	300	Balances model variance (fewer trees: high variance) and computational cost (more trees: high cost).
Max Depth	10–50	25	Prevents overfitting (too deep: memorizes noise) while capturing complex feature-performance relationships.
Min Samples Split	2–10	5	Ensures splits are statistically meaningful (avoids trivial splits on small subsets).
Min Samples Leaf	1–5	2	Reduces overfitting by requiring minimum samples in leaf nodes (avoids isolated data points).
Max Features	$\sqrt{d}$ – $d$ ( $d=7$ )	$\sqrt{d}$	Uses square-root of total features to balance diversity (randomness) and predictive power.
Learning Rate (for boosting)	0.01–0.1	—	Not used (RF is an ensemble of unboosted trees).

Appendix

Table 3: Hyperparameter Configuration of Random Forest Surrogate Model

Computational Stage	Wall-Clock Time	GPU-Hours	Cost (USD, 10 USD/hour)
RF Training Data Collection	12.0 hours	12.0	120.0
RF Model Training (incl. hyperparameter tuning)	2.5 hours	2.3	23.0
High-Potential Candidate Selection (Inference)	0.3 hours	0.3	3.0
Total RF-Related Cost	14.8 hours	14.6	146.0
Direct Meta-Heuristic Optimization (No RF)	48.0 hours	45.2	452.0
Reduction Ratio	69.2%	67.7%	67.7%

Appendix

Table 4: Computational Cost Comparison

Feature Name	Abbreviation	Mathematical Definition	Physical Interpretation
Spectral Entropy	SE	$SE=-\sum\limits_{k=0}^{N-1}p_{k}\log_{2}p_{k}$ , $p_{k}=\frac{\|X(k)\|^{2}}{\sum\limits_{i=0}^{N-1}\|X(i)\|^{2}}$	Measures the uniformity of energy distribution; lower values indicate concentrated energy.
Spectral Information Density	SID	$SID=\frac{1}{N}\sum\limits_{k=0}^{N-1}\|X(k)\|^{2}\cdot\log_{2}(\|X(k)\|^{2}+\epsilon)$	Quantifies information content per frequency component; higher values mean richer task-relevant information.
Low/Medium/High Frequency Ratio	LMH-FR	$LMH\text{-}FR=\sum\|X_{\text{low}}\|^{2}\colon\sum\|X_{\text{med}}\|^{2}\colon\sum\|X_{\text{high}}\|^{2}$	Ratio of energy in three frequency bands (low: $k=0\sim N/3$ , med: $N/3\sim 2N/3$ , high: $2N/3\sim N-1$ ).
Energy Concentration	EC	$EC=\frac{\sum\limits_{m=1}^{M}\|X_{\text{top-}m}\|^{2}}{\sum\limits_{k=0}^{N-1}\|X(k)\|^{2}}$	Proportion of energy covered by top- $M$ high-energy components (Section A.5 ).
Frequency Band Coverage	FBC	$FBC=\frac{f_{\text{max}}}{f_{\text{min}}}$ , $f_{\text{min/max}}$ = min/max frequency of candidate set	Indicates the span of frequency bands; reflects coverage of spectral characteristics.
Spectral Crest Factor	SCF	$SCF=\frac{\max(\|X(k)\|^{2})}{\frac{1}{N}\sum\limits_{k=0}^{N-1}\|X(k)\|^{2}}$	Measures the concentration of energy in a single frequency component; higher values mean stronger peak energy.

Appendix

Table 5: Spectral Features for Random Forest Surrogate Model

Explore similar work

FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts
Jun 30, 2026Tom Saliencro, Maya Lindqvist, Rohan Desai +2Low-Rank Adaptation Mixture-of-Experts Models
University of California, Irvine · University of Washington
S2FT: Parameter-Efficient Fine-Tuning in Sparse Spectrum Domain
May 9, 2026Baoquan Zhang, Zhehao Yu, Lisai Zhang +5Fine-Tuning Sparse Fine-Tuning
Harbin Institute of Technology, Shenzhen · Bilibili. Inc · Shenzhen University +1
FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning
May 19, 2026Yequan Zhao, Ruijie Zhang, Liyan Tan +3LLM Fine-Tuning Parameter-Efficient Fine-Tuning
University of California at Santa Barbara · Amazon Lab126

Jun 30, 2026, cs.LGJ/K move · Enter open · S save

FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts

Tom Saliencro, Maya Lindqvist, Rohan Desai, Priya Nair+1

University of California, Irvine · University of Washington

Low-Rank Adaptation Mixture-of-Experts Models Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Parameter-efficient fine-tuning (PEFT) reparameterizes weight updates in a fixed basis: low-rank adapters operate in the spatial domain, while a recent line of spectral methods operates in a fixed Fourier domain. We argue that the choice of domain is itself a design degree of freedom that should be learned, and that no single basis is optimal across tasks, layers, or tokens. We introduce Fractional-Fourier Mixture of Experts, a mixture-of-experts adapter in which every expert carries a learnable fractional-Fourier order that continuously interpolates between the spatial domain (recovering vanilla LoRA) and the Fourier domain (recovering a spectral adapter). Routing tokens through experts that occupy different points on this spatial-spectral continuum lets the model place each low-rank update in the domain where it is most compact, and -- because fractional-Fourier operators of different orders are mutually incoherent -- makes the experts naturally decorrelated, which reduces interference and improves multi-task composition. The order is a single scalar per expert, trained with a separate optimizer, and the transform is computed with an

\mathcal{O}(d\log d)

chirp--FFT surrogate, so Fractional-Fourier Mixture of Experts adds negligible cost over standard MoE-LoRA. Across commonsense, mathematical, code, and knowledge benchmarks on LLaMA-3.1-8B and Qwen2.5-7B, Fractional-Fourier Mixture of Experts improves over strong MoE-LoRA and spectral baselines -- including FlyLoRA, FourierMoE, and HMoRA -- while keeping the active-parameter budget small, and analysis shows that the learned orders specialize by task and layer in interpretable ways.

Abstract

Explore similar work

FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts

S2FT: Parameter-Efficient Fine-Tuning in Sparse Spectrum Domain

FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning