SpectralLoRA: Is Low-Frequency Structure Sufficient for LoRA Adaptation? A Spectral Analysis of Weight Updates

Rajveer Singh

cs.LGApr 12, 2026

SpectralLoRA: Is Low-Frequency Structure Sufficient for LoRA Adaptation? A Spectral Analysis of Weight Updates

Authors: Rajveer Singh

Organizations: Indian Institute of Technology Roorkee

Abstract

We present a systematic empirical study of the spectral structure of LoRA weight updates. Through 2D Discrete Cosine Transform (DCT) analysis of trained adaptation matrices across BERT-base and RoBERTa-base on four GLUE benchmarks (SST-2, MNLI, CoLA, QQP), we establish that LoRA updates are universally dominated by low-frequency components: on average, just 33% of DCT coefficients capture 90% of total spectral energy. Retaining only 10% of frequency coefficients reduces adapter storage by 10x while sacrificing only 1.95 percentage points on SST-2. Notably, frequency masking at k=50% improves over full LoRA on 3 of 8 model-task pairs, suggesting high-frequency components act as adaptation noise. We further discover that RoBERTa-base is systematically more spectrally compressible than BERT-base across all tasks, and that task complexity governs spectral sensitivity: NLI tasks require more frequency budget than sentiment classification. A subsequent SVD-DCT correlation analysis (Pearson r=0.906, p<1e-9) connects the empirical 33% constant to the spectral dynamics of SGD (Olsen et al., 2025), suggesting a theoretical grounding for this finding. These findings motivate a new design principle for PEFT: spectral sparsity in adaptation.

Figures & tables

Method	Core Idea	Gap vs. This Work
LoRA ( Hu et al., 2021 )	Low-rank $AB$ decomposition	No frequency analysis
AdaLoRA ( Zhang et al., 2023 )	SVD-based rank allocation	Spatial domain only
LoRA-Mini ( Ahmed et al., 2024 )	Decompose + selective training	No spectral motivation
KronA ( Edalati et al., 2022 )	Kronecker product structure	Spatial domain only
SpectralLoRA (Ours)	DCT analysis of LoRA updates	First frequency-domain study

Table 1: Comparison of SpectralLoRA with related PEFT methods.

Component	Details
Models	BERT-base-uncased, RoBERTa-base ( $\sim$ 110M parameters each)
Tasks	SST-2 (sentiment), MNLI (NLI), CoLA (linguistics), QQP (paraphrase)
LoRA config	$r=8$ , $\alpha=32$ , target = [query, value], dropout = 0.1
Train samples	5,000 per task (consistent budget across all experiments)
Hardware	NVIDIA T4-15GB, PyTorch 2.10, transformers 4.45.2
DCT library	scipy.fft.dctn with norm=’ortho’ (2D DCT-II)

Table 2: Experimental configuration for all SpectralLoRA analyses.

Method	Trained Params	Stored Params	Reduction	SST-2
Full Fine-tuning	109,780,228	109,780,228	1.0 $\times$	$\sim$ 93.0%
LoRA $r=8$ (baseline)	296,450	296,450	1.0 $\times$	87.73%
SpectralLoRA $k$ =50%	296,450	148,225	2.0 $\times$	88.19% (+0.46)
SpectralLoRA $k$ =20%	296,450	59,290	5.0 $\times$	87.04% ( $-$ 0.69)
SpectralLoRA $k$ =10%	296,450	29,645	10.0 $\times$	85.78% ( $-$ 1.95)
SpectralLoRA $k$ =5%	296,450	14,823	20.0 $\times$	79.82% ( $-$ 7.91)

Table 3: Parameter efficiency of SpectralLoRA on SST-2 (BERT-base). Green = beats LoRA baseline. Trained parameters are identical to LoRA – only storage/deployment footprint is reduced.

Figure 1: Accuracy vs. DCT frequency budget on SST-2 (BERT-base). Dashed green line = full LoRA baseline (87.7%). At $k$ =50%, SpectralLoRA exceeds full LoRA by 0.46pp, indicating frequency masking acts as implicit regularization.

Model	Task	Metric	Full LoRA	$k$ =10%	$k$ =20%	$k$ =50%	$\Delta$ @10%
BERT-base	SST-2	Acc	87.73	85.78	87.04	88.19	$-$ 1.95pp
BERT-base	MNLI	Acc	63.50	56.30	61.10	63.40	$-$ 7.20pp
BERT-base	CoLA	Mcc	80.35	75.17	78.10	80.10	$-$ 5.18pp
BERT-base	QQP	F1	72.10	75.44	77.60	75.20	+3.34pp
RoBERTa-base	SST-2	Acc	92.43	91.40	92.20	92.30	$-$ 1.03pp
RoBERTa-base	MNLI	Acc	76.92	74.66	76.10	76.50	$-$ 2.26pp

Table 4: Full cross-task, cross-model results. Green = frequency masking improves over full LoRA. Red = high spectral sensitivity. All results on validation sets.

Figure 2: Per-task accuracy vs. frequency budget for BERT-base (solid circles) and RoBERTa-base (dashed squares) across all four GLUE tasks. Annotations show accuracy at $k$ =10% for each model. RoBERTa consistently recovers accuracy at lower $k$ values, confirming its higher spectral compressibility.

Figure 3: All model–task accuracy vs. frequency budget curves overlaid. BERT-base (solid) and RoBERTa-base (dashed). The separation between MNLI (red, bottom) and SST-2 (blue, top) directly visualizes task-complexity-driven spectral sensitivity (Finding 3).

Model	SST-2	MNLI	CoLA	QQP	Mean
BERT-base	33.8%	33.5%	34.5%	33.3%	33.8%
RoBERTa-base	31.1%	32.1%	33.2%	32.0%	32.1%

Table 5: Average

k\%

for 90% spectral energy per model and task. Variance across all 16 cells

=1.2

pp. This stability suggests a near-universal constant of spectral compressibility for GLUE-scale LoRA adaptation.

Figure 4: Average $k\%$ required for 90% DCT energy across tasks and models. RoBERTa-base (orange) is systematically more compressible than BERT-base (blue) across all four tasks with a consistent gap of $\sim$ 2pp. The bar heights themselves are remarkably stable (31–35% range) across all 8 conditions.

Task	Type	Classes	Drop @ $k$ =10%	Interpretation
SST-2	Sentiment	2	$-$ 1.95pp	Simple, highly compressible
QQP	Paraphrase	2	+3.34pp	Regularization beneficial
CoLA	Linguistic	2	$-$ 5.18pp	Fine structure needed
MNLI	Inference	3	$-$ 7.20pp	Most sensitive – complex reasoning

Table 6: Task sensitivity to spectral compression, ordered by accuracy drop at

k

=10% (BERT-base).

Model	Task	Acc @ $k$ =100%	Acc @ $k$ =50%	Acc @ $k$ =10%	Best $k$
BERT-base	SST-2	87.73%	88.19% (+0.46)	85.78%	50%
BERT-base	QQP	72.10%	75.20% (+3.10)	75.44% (+3.34)	10%
RoBERTa-base	CoLA	79.00%	79.40% (+0.40)	73.35%	50%

Table 7: Cases where frequency masking improves over full LoRA. Green = gain over full LoRA.

Figure 5: Layer-wise DCT cumulative energy curves for query (left) and value (right) projections across all 12 BERT-base transformer layers. Red dashed line = 90% energy threshold. Later layers (yellow, Layer 10–11) reach the threshold at lower $k\%$ , indicating higher spectral compressibility in task-specific upper layers.

Figure 6: Heatmap of $k\%$ needed for 90% DCT energy per transformer layer and module type (Query/Value). Darker red = more compressible (lower $k\%$ needed). Layer 11 query projection requires only 26.6% vs. Layer 0’s 38.8% – a 12pp gap indicating depth-dependent spectral structure. Value projections show a different pattern, with Layer 4 being notably compressible (29.5%).

Low-Rank Adaptation

Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.

See all 542 papers

Spectral Surgery: Training-Free Refinement of LoRA via Gradient-Guided Singular Value Reweighting
Mar 4, 2026Zailong Tian, Yanzhe Chen, Zhuoheng Han +1Low-Rank Adaptation Adapter Tuning
School of Computing and Information Systems, Singapore Management University · School of Computing, National University of Singapore · State Key Laboratory for Multimedia Information Processing, Peking University
Beyond LoRA: Is Sparsity-Induced Adaptation Better?
Jun 11, 2026Elijah Cadenhead, Cristian McGee, Xin Li +2Low-Rank Adaptation Parameter-Efficient Fine-Tuning
School of Data, Mathematical and Statistical Sciences, University of Central Florida, United States · Department of Computer Science, University of Central Florida, United States · College of Computing, Mohammed VI Polytechnic University (UM6P), Morocco
HiP-LoRA: Budgeted Spectral Plasticity for Robust Low-Rank Adaptation
Apr 20, 2026Lixian Chen, Jianhong TanContinual Learning for LLMs Low-Rank Adaptation
Guangdong University of Technology

Mar 4, 2026J/K move · Enter open · S save

Spectral Surgery: Training-Free Refinement of LoRA via Gradient-Guided Singular Value Reweighting

Zailong Tian, Yanzhe Chen, Zhuoheng Han, Lizi Liao

School of Computing and Information Systems, Singapore Management University · School of Computing, National University of Singapore · State Key Laboratory for Multimedia Information Processing, Peking University

Low-Rank Adaptation Adapter Tuning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) improves downstream performance by restricting task updates to a low-rank parameter subspace, yet how this limited capacity is allocated within a trained adapter remains unclear. Through a geometric and empirical study across multiple tasks and backbones, we find that trained LoRA updates often exhibit an inefficient spectrum: task effects concentrate in a small subset of singular directions, while many remaining components are neutral or detrimental, motivating post-hoc refinement within the learned subspace. We propose Spectral Surgery, a training-free refinement that decomposes a LoRA update with SVD, estimates per-component sensitivity using gradients on a small calibration set, and reweights singular values under a magnitude constraint while keeping the learned directions fixed. Across Llama-3.1-8B and Qwen3-8B on four benchmarks, Spectral Surgery yields consistent gains (up to +4.4 points on CommonsenseQA and +2.4 pass@1 on HumanEval) by adjusting only

\approx 1{,}000

scalar coefficients. These results demonstrate that SVD-structured, low-cost parameter editing can serve as a practical route to improving trained LoRA adapters in a purely post-hoc manner.

Abstract

Low-Rank Adaptation

Explore similar work

Spectral Surgery: Training-Free Refinement of LoRA via Gradient-Guided Singular Value Reweighting

Beyond LoRA: Is Sparsity-Induced Adaptation Better?

HiP-LoRA: Budgeted Spectral Plasticity for Robust Low-Rank Adaptation