Spectral Surgery: Training-Free Refinement of LoRA via Gradient-Guided Singular Value Reweighting

Zailong Tian; Yanzhe Chen; Zhuoheng Han; Lizi Liao

cs.LGMar 4, 2026

Spectral Surgery: Training-Free Refinement of LoRA via Gradient-Guided Singular Value Reweighting

Authors: Zailong Tian, Yanzhe Chen, Zhuoheng Han, Lizi Liao

Organizations: School of Computing and Information Systems, Singapore Management University · School of Computing, National University of Singapore · State Key Laboratory for Multimedia Information Processing, Peking University

Abstract

Low-Rank Adaptation (LoRA) improves downstream performance by restricting task updates to a low-rank parameter subspace, yet how this limited capacity is allocated within a trained adapter remains unclear. Through a geometric and empirical study across multiple tasks and backbones, we find that trained LoRA updates often exhibit an inefficient spectrum: task effects concentrate in a small subset of singular directions, while many remaining components are neutral or detrimental, motivating post-hoc refinement within the learned subspace. We propose Spectral Surgery, a training-free refinement that decomposes a LoRA update with SVD, estimates per-component sensitivity using gradients on a small calibration set, and reweights singular values under a magnitude constraint while keeping the learned directions fixed. Across Llama-3.1-8B and Qwen3-8B on four benchmarks, Spectral Surgery yields consistent gains (up to +4.4 points on CommonsenseQA and +2.4 pass@1 on HumanEval) by adjusting only $\approx 1{,}000$ scalar coefficients. These results demonstrate that SVD-structured, low-cost parameter editing can serve as a practical route to improving trained LoRA adapters in a purely post-hoc manner.

Figures & tables

Panel of Figure 1 — Figure 1: Geometric structure of LoRA updates in residual-writing modules. We analyze LoRA updates of Qwen3-8B finetuned on the Alpaca dataset and visualize the alignment of the o_proj module in the shared residual output space. (a) The leading output direction ( $u_{1}$ ) exhibits consistently high similarity across layers. (b) The full top-4 output subspace $U^{(4)}$ (with rank $r=16$ ) is also highly stable across layers, indicating a shared update manifold in the residual stream. (c) Within each layer, o_proj and down_proj are strongly aligned relative to a random-subspace baseline ( $m/d_{\text{model}}$ ).

Figure 2: Overview of Spectral Editing. We decompose the LoRA update $\Delta W$ into singular components ( $U,\Sigma,V^{\top}$ ). We estimate a sensitivity score $s_{k}$ for each singular component using gradient projections on a calibration set, and then reweight the singular values in $\Sigma$ (via hard selection or continuous reweighting) to amplify task-relevant directions while suppressing noise. This reconstructs an edited update $\Delta W^{\prime}$ without altering the singular subspaces.

Backbone	#Layers $L$	Rank $r$	#Edited Scalars
Llama-3.1-8B	32	16	1024
Qwen3-8B	36	16	1152

Table 1: Editing overhead (editable scalars). Default setting edits o_proj and down_proj in every layer, yielding

2Lr

editable scalars.

Llama-3.1-8B
Model	Task	Baseline	Random_index	Smooth_abs	Abs_select	Grad_dir
	GSM8K (acc)	0.657	0.661	0.668	0.659	0.614
	HumanEval (pass@1)	0.488	0.488	0.494	0.488	0.476
	IFEval (score)	0.305	0.306	0.312	0.321	0.223
	CSQA (acc)	0.740	0.743	0.735	0.736	0.784
Qwen3-8B

Table 2: Main Results with Recommended Settings. Editing policies on Llama-3.1-8B and Qwen3-8B under the default configuration ( calib=128 , energy=L1 ). Baseline is the unedited adapter.

Figure 3: Guided vs. random perturbations. Each point is a (model, task) pair under the default setting. The x-axis is the improvement of random_index over the baseline, and the y-axis is the improvement of grad_direction . Points above the diagonal indicate genuine signal beyond random perturbation; the extreme failure on Qwen- IFEval illustrates the alignment tax of gradient-based editing.

Figure 4: Safety trade-off of editing policies under the default setting . Reward is the mean improvement over aligned tasks ( GSM8K , HumanEval , CSQA ); Risk is the performance drop on the constraint-sensitive benchmark IFEval (clipped at 0 if a policy does not decrease IFEval ). Gradient-based editing reaches the high-reward regime but can incur large risk, especially on IFEval .

		grad_direction Performance
Model	Task	Baseline	With L1 (Ours)	No Constraint
Llama-3.1	CSQA	0.740	0.784	0.785
Qwen3	IFEval	0.590	0.173	0.017

Table 3: Impact of Energy Preservation. grad_direction with and without energy constraints. The

L_{1}

constraint mitigates extreme drift on strict instruction-following (IFEval), while leaving aligned reasoning gains largely intact.

Task (Model)	Policy	32	64	128	256
CSQA (Llama)	grad_dir	0.785	0.782	0.784	0.783
CSQA (Llama)	random	0.742	0.740	0.743	0.740
IFEval (Qwen)	grad_dir	0.179	0.181	0.173	0.169
IFEval (Qwen)	random	0.603	0.598	0.597	0.604

Table 4: Effect of Calibration Size. Performance consistency across varying calib_samples . Larger budgets can stabilize outcomes but do not ensure monotonic improvements, motivating calib=128 as a practical default.

Model	Task	Baseline	Attn Inputs	All Modules	Residual (Ours)
Llama	CSQA	0.740	0.744	0.790	0.784
Llama	GSM8K	0.657	0.637	0.538	0.614
Qwen	HumanEval	0.488	0.512	0.524	0.512
Qwen	IFEval	0.590	0.499	0.071	0.173

Table 5: Module Selection Ablation. grad_direction under different edit localities ( calib=128 , energy=L1 ). “Residual (Ours)” denotes restricting edits to residual-writing projections (e.g., o_proj and down_proj in our main setup). We add “MLP Internal” ( up , gate ) for comparison. Editing MLP internals yields high peaks on code tasks but degrades arithmetic/instruction following.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Task Family	Max Len	Epochs	Global BS	Peak LR	Min LR	Grad Clip
Math (MetaMath)	1024	3	768	$1{\times}10^{-4}$	$1{\times}10^{-6}$	1.0
Commonsense (CSQA)	2048	3	256	$2{\times}10^{-4}$	$2{\times}10^{-6}$	1.0
Instruction (Alpaca)	2048	3	256	$2{\times}10^{-4}$	$2{\times}10^{-6}$	1.0
Code (Magicoder)	4096	3	192	$2{\times}10^{-4}$	$2{\times}10^{-6}$	1.0

Appendix

Table 6: Hyperparameter configurations. ”Global BS” denotes the effective batch size after gradient accumulation. We use a fixed epoch budget of 3 for all tasks.

Panel of Figure 5 — Figure 5: Llama-3.1-8B: Principal-direction similarity heatmap wall ( $|u_{1}^{\top}u_{1}|$ ). Each cell shows the inter-layer similarity heatmap for a specific (task, module).

Panel of Figure 6 — Figure 6: Llama-3.1-8B: Top- $m$ output-subspace overlap heatmap wall ( $\mathrm{Align}_{U}$ , $m{=}4$ ). Each cell shows the inter-layer subspace-overlap heatmap for a specific (task, module).

Panel of Figure 7 — Figure 7: Qwen3-8B: Principal-direction similarity heatmap wall ( $|u_{1}^{\top}u_{1}|$ ). Each cell shows the inter-layer similarity heatmap for a specific (task, module).

Panel of Figure 8 — Figure 8: Qwen3-8B: Top- $m$ output-subspace overlap heatmap wall ( $\mathrm{Align}_{U}$ , $m{=}4$ ). Each cell shows the inter-layer subspace-overlap heatmap for a specific (task, module).

Low-Rank Adaptation

Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.

See all 542 papers

HiP-LoRA: Budgeted Spectral Plasticity for Robust Low-Rank Adaptation
Apr 20, 2026Lixian Chen, Jianhong TanContinual Learning for LLMs Low-Rank Adaptation
Guangdong University of Technology
SpectralLoRA: Is Low-Frequency Structure Sufficient for LoRA Adaptation? A Spectral Analysis of Weight Updates
Apr 12, 2026Rajveer SinghLow-Rank Adaptation Parameter-Efficient Fine-Tuning
Indian Institute of Technology Roorkee
ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuning
Oct 14, 2026Fanyao Li, Yuxi Liu, Xiaohao Liu +3Low-Rank Adaptation Parameter-Efficient Fine-Tuning
Communication University of China · National University of Singapore · Xidian University

Apr 20, 2026J/K move · Enter open · S save

HiP-LoRA: Budgeted Spectral Plasticity for Robust Low-Rank Adaptation

Lixian Chen, Jianhong Tan

Guangdong University of Technology

Continual Learning for LLMs Low-Rank Adaptation Model Merging Parameter-Efficient Fine-Tuning Catastrophic Forgetting

Open paper AskPDF arXiv

Adapting foundation models under resource budgets relies heavily on Parameter-Efficient Fine-Tuning (PEFT), with LoRA being a standard modular solution. However, LoRA suffers from spectral interference. Low-rank updates often concentrate energy on the leading singular directions of pretrained weights, perturbing general capabilities and causing catastrophic forgetting and fragile multi-adapter merging. To resolve this, we propose HiP-LoRA, a spectrum-aware adaptation framework. Utilizing the cached singular value decomposition (SVD) of pretrained layers, HiP-LoRA decomposes updates into two channels: a principal channel within the dominant singular subspace, and a residual low-rank channel in the orthogonal complement. A singular-value-weighted stability budget on the principal channel continuously balances pretrained behavior preservation with task-specific plasticity. Experiments on Llama-3.1-8B demonstrate that under matched budgets, HiP-LoRA drastically reduces pretraining degradation and multi-adapter MergeFail, robustly outperforming baselines in interference-sensitive tasks like continual tuning and knowledge editing.

Abstract

Low-Rank Adaptation

Explore similar work

HiP-LoRA: Budgeted Spectral Plasticity for Robust Low-Rank Adaptation

SpectralLoRA: Is Low-Frequency Structure Sufficient for LoRA Adaptation? A Spectral Analysis of Weight Updates

ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuning