Organizations: School of Computing and Information Systems, Singapore Management University · School of Computing, National University of Singapore · State Key Laboratory for Multimedia Information Processing, Peking University
Low-Rank Adaptation (LoRA) improves downstream performance by restricting task updates to a low-rank parameter subspace, yet how this limited capacity is allocated within a trained adapter remains unclear. Through a geometric and empirical study across multiple tasks and backbones, we find that trained LoRA updates often exhibit an inefficient spectrum: task effects concentrate in a small subset of singular directions, while many remaining components are neutral or detrimental, motivating post-hoc refinement within the learned subspace. We propose Spectral Surgery, a training-free refinement that decomposes a LoRA update with SVD, estimates per-component sensitivity using gradients on a small calibration set, and reweights singular values under a magnitude constraint while keeping the learned directions fixed. Across Llama-3.1-8B and Qwen3-8B on four benchmarks, Spectral Surgery yields consistent gains (up to +4.4 points on CommonsenseQA and +2.4 pass@1 on HumanEval) by adjusting only ≈1,000 scalar coefficients. These results demonstrate that SVD-structured, low-cost parameter editing can serve as a practical route to improving trained LoRA adapters in a purely post-hoc manner.
Figures & tables
Figure 1: Geometric structure of LoRA updates in residual-writing modules. We analyze LoRA updates of Qwen3-8B finetuned on the Alpaca dataset and visualize the alignment of the o_proj module in the shared residual output space. (a) The leading output direction ( u1 ) exhibits consistently high similarity across layers. (b) The full top-4 output subspace U(4) (with rank r=16 ) is also highly stable across layers, indicating a shared update manifold in the residual stream. (c) Within each layer, o_proj and down_proj are strongly aligned relative to a random-subspace baseline ( m/dmodel ).
Figure 2: Overview of Spectral Editing. We decompose the LoRA update ΔW into singular components ( U,Σ,V⊤ ). We estimate a sensitivity score sk for each singular component using gradient projections on a calibration set, and then reweight the singular values in Σ (via hard selection or continuous reweighting) to amplify task-relevant directions while suppressing noise. This reconstructs an edited update ΔW′ without altering the singular subspaces.
Backbone
#Layers L
Rank r
#Edited Scalars
Llama-3.1-8B
32
16
1024
Qwen3-8B
36
16
1152
Table 1: Editing overhead (editable scalars). Default setting edits o_proj and down_proj in every layer, yielding 2Lr editable scalars.
Model
Task
Baseline
Random_index
Smooth_abs
Abs_select
Grad_dir
Llama-3.1-8B
GSM8K (acc)
0.657
0.661
0.668
0.659
0.614
HumanEval (pass@1)
0.488
0.488
0.494
0.488
0.476
IFEval (score)
0.305
0.306
0.312
0.321
0.223
CSQA (acc)
0.740
0.743
0.735
0.736
0.784
Qwen3-8B
Table 2: Main Results with Recommended Settings. Editing policies on Llama-3.1-8B and Qwen3-8B under the default configuration ( calib=128 , energy=L1 ). Baseline is the unedited adapter.
Figure 3: Guided vs. random perturbations. Each point is a (model, task) pair under the default setting. The x-axis is the improvement of random_index over the baseline, and the y-axis is the improvement of grad_direction . Points above the diagonal indicate genuine signal beyond random perturbation; the extreme failure on Qwen- IFEval illustrates the alignment tax of gradient-based editing.
Figure 4: Safety trade-off of editing policies under the default setting . Reward is the mean improvement over aligned tasks ( GSM8K , HumanEval , CSQA ); Risk is the performance drop on the constraint-sensitive benchmark IFEval (clipped at 0 if a policy does not decrease IFEval ). Gradient-based editing reaches the high-reward regime but can incur large risk, especially on IFEval .
grad_direction Performance
Model
Task
Baseline
With L1 (Ours)
No Constraint
Llama-3.1
CSQA
0.740
0.784
0.785
Qwen3
IFEval
0.590
0.173
0.017
Table 3: Impact of Energy Preservation. grad_direction with and without energy constraints. The L1 constraint mitigates extreme drift on strict instruction-following (IFEval), while leaving aligned reasoning gains largely intact.
Task (Model)
Policy
32
64
128
256
CSQA (Llama)
grad_dir
0.785
0.782
0.784
0.783
random
0.742
0.740
0.743
0.740
IFEval (Qwen)
grad_dir
0.179
0.181
0.173
0.169
random
0.603
0.598
0.597
0.604
Table 4: Effect of Calibration Size. Performance consistency across varying calib_samples . Larger budgets can stabilize outcomes but do not ensure monotonic improvements, motivating calib=128 as a practical default.
Model
Task
Baseline
Attn Inputs
All Modules
Residual (Ours)
Llama
CSQA
0.740
0.744
0.790
0.784
GSM8K
0.657
0.637
0.538
0.614
Qwen
HumanEval
0.488
0.512
0.524
0.512
IFEval
0.590
0.499
0.071
0.173
Table 5: Module Selection Ablation. grad_direction under different edit localities ( calib=128 , energy=L1 ). “Residual (Ours)” denotes restricting edits to residual-writing projections (e.g., o_proj and down_proj in our main setup). We add “MLP Internal” ( up , gate ) for comparison. Editing MLP internals yields high peaks on code tasks but degrades arithmetic/instruction following.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Task Family
Max Len
Epochs
Global BS
Peak LR
Min LR
Grad Clip
Math (MetaMath)
1024
3
768
1×10−4
1×10−6
1.0
Commonsense (CSQA)
2048
3
256
2×10−4
2×10−6
1.0
Instruction (Alpaca)
2048
3
256
2×10−4
2×10−6
1.0
Code (Magicoder)
4096
3
192
2×10−4
2×10−6
1.0
Appendix
Table 6: Hyperparameter configurations. ”Global BS” denotes the effective batch size after gradient accumulation. We use a fixed epoch budget of 3 for all tasks.
Figure 5: Llama-3.1-8B: Principal-direction similarity heatmap wall ( ∣u1⊤u1∣ ). Each cell shows the inter-layer similarity heatmap for a specific (task, module).
Figure 6: Llama-3.1-8B: Top- m output-subspace overlap heatmap wall ( AlignU , m=4 ). Each cell shows the inter-layer subspace-overlap heatmap for a specific (task, module).
Figure 7: Qwen3-8B: Principal-direction similarity heatmap wall ( ∣u1⊤u1∣ ). Each cell shows the inter-layer similarity heatmap for a specific (task, module).
Figure 8: Qwen3-8B: Top- m output-subspace overlap heatmap wall ( AlignU , m=4 ). Each cell shows the inter-layer subspace-overlap heatmap for a specific (task, module).
Adapting foundation models under resource budgets relies heavily on Parameter-Efficient Fine-Tuning (PEFT), with LoRA being a standard modular solution. However, LoRA suffers from spectral interference. Low-rank updates often concentrate energy on the leading singular directions of pretrained weights, perturbing general capabilities and causing catastrophic forgetting and fragile multi-adapter merging. To resolve this, we propose HiP-LoRA, a spectrum-aware adaptation framework. Utilizing the cached singular value decomposition (SVD) of pretrained layers, HiP-LoRA decomposes updates into two channels: a principal channel within the dominant singular subspace, and a residual low-rank channel in the orthogonal complement. A singular-value-weighted stability budget on the principal channel continuously balances pretrained behavior preservation with task-specific plasticity. Experiments on Llama-3.1-8B demonstrate that under matched budgets, HiP-LoRA drastically reduces pretraining degradation and multi-adapter MergeFail, robustly outperforming baselines in interference-sensitive tasks like continual tuning and knowledge editing.
We present a systematic empirical study of the spectral structure of LoRA weight updates. Through 2D Discrete Cosine Transform (DCT) analysis of trained adaptation matrices across BERT-base and RoBERTa-base on four GLUE benchmarks (SST-2, MNLI, CoLA, QQP), we establish that LoRA updates are universally dominated by low-frequency components: on average, just 33% of DCT coefficients capture 90% of total spectral energy. Retaining only 10% of frequency coefficients reduces adapter storage by 10x while sacrificing only 1.95 percentage points on SST-2. Notably, frequency masking at k=50% improves over full LoRA on 3 of 8 model-task pairs, suggesting high-frequency components act as adaptation noise. We further discover that RoBERTa-base is systematically more spectrally compressible than BERT-base across all tasks, and that task complexity governs spectral sensitivity: NLI tasks require more frequency budget than sentiment classification. A subsequent SVD-DCT correlation analysis (Pearson r=0.906, p<1e-9) connects the empirical 33% constant to the spectral dynamics of SGD (Olsen et al., 2025), suggesting a theoretical grounding for this finding. These findings motivate a new design principle for PEFT: spectral sparsity in adaptation.
Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning by learning low-rank weight updates. Increasing rank expands adaptation capacity, but does not guarantee that all learned directions improve downstream performance. In a diagnostic experiment with rank-16 LoRA, we show that retaining only the top singular components achieves the best performance, whereas adding further components degrades it. Motivated by this, we propose Residual-Gradient Adaptation (ReGRA), a two-stage method for making more effective use of a fixed rank budget. After the first stage, ReGRA estimates task gradients on a calibration set and removes their projections onto the two-sided subspace induced by the first-stage update. It then aggregates the residual-gradient directions to initialize a second-stage adapter without changing the first-stage update. An orthogonality regularizer encourages the two updates to capture complementary directions. Both updates can be merged into the pretrained weights, incurring no additional inference overhead. Across diverse benchmark sets, ReGRA achieves stronger performance than alternatives with the same total rank and even matches LoRA adapters with twice the rank. Ablation studies and further analysis support the contribution of its core design choices. Code is available at https://github.com/Fanyao-Li/ReGRA.
Fanyao Li, Yuxi Liu, Xiaohao Liu +3
Communication University of China · National University of Singapore · Xidian University