Learn the Directions, Normalize the Gains: Post-Training Normalization for LoRA

Zailong Tian; Yanzhe Chen; Zhuoheng Han; Houfeng Wang; Lizi Liao

cs.LGOct 1, 2026

Learn the Directions, Normalize the Gains: Post-Training Normalization for LoRA

Authors: Zailong Tian, Yanzhe Chen, Zhuoheng Han, Houfeng Wang, Lizi Liao

Organizations: Singapore Management University · National University of Singapore · Peking University

Abstract

While Low-Rank Adaptation (LoRA) enables efficient task specialization, its learned updates can compromise capabilities beyond the target task. We identify \textbf{adaptation imbalance}: a few singular directions dominate the trained update, leaving its performance sensitive to how gains are allocated. We argue that \textbf{learning where to adapt does not ensure that adaptation gains are well balanced}. This motivates \textbf{LoRA-Norm}, a post-training normalization method that retains learned directions while rebalancing their gains. LoRA-Norm combines spectral rebalancing, a fixed nonlinear transformation of singular values, with nuclear-norm restoration, which preserves the original total spectral mass. It requires no calibration data or additional training and introduces no inference overhead. Across two backbones and three adaptation tasks, LoRA-Norm improves average specialization and capability retention, outperforming the evaluated post-hoc spectral pruning and gradient-guided editing configurations on both measures. Stronger functional equalization brings no consistent additional gains, revealing that balancing adapter gains and equalizing their responses are distinct objectives.

Figures & tables

Figure 1: Learn the directions, normalize the gains. LoRA training learns useful adaptation directions, but the gains assigned to these directions can become imbalanced. LoRA-Norm keeps the learned directions fixed and rebalances only their gains after training, improving specialization while preserving capabilities beyond the target task.

Figure 2: Overview of LoRA-Norm. A LoRA adapter learns both singular directions and their gains. LoRA-Norm keeps the learned directions fixed and applies a post-training spectral transformation to rebalance their gains, followed by nuclear-norm restoration. The resulting adapter uses the same learned directions with rebalanced gains, requiring no additional training or calibration data.

	Target (%) $\uparrow$		Off (%) $\uparrow$		FG (pp) $\downarrow$
Task	LoRA	LoRA-Norm	LoRA	LoRA-Norm	LoRA	LoRA-Norm
Qwen3-8B
Magicoder	64.23 $\pm$ 1.27	74.59 $\pm$ 0.35	78.04 $\pm$ 0.45	81.26 $\pm$ 0.07	1.67 $\pm$ 0.48	0.00 $\pm$ 0.00
MetaMath	84.05 $\pm$ 0.09	87.47 $\pm$ 0.68	68.16 $\pm$ 4.10	75.28 $\pm$ 1.23	4.95 $\pm$ 4.05	0.00 $\pm$ 0.00
Tulu	66.97 $\pm$ 0.83	71.41 $\pm$ 0.91	85.63 $\pm$ 0.46	84.93 $\pm$ 0.44	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00
Llama-3.1-8B-Instruct

Table 1: Target-task performance and capability retention. Entries are mean

\pm

sample SD across three training runs; Overall averages all 18 checkpoints. Off and FG use the three non-target categories, including Commonsense-8.

Method	Params retained (%)	Target (%) $\uparrow$	Off (%) $\uparrow$	FG (pp) $\downarrow$
Base	—	66.47	69.99	0.00
LoRA	100	68.00	69.55	2.71
PARA ( $\epsilon=.90$ )	38.16	68.78	70.01	2.37
PARA ( $\epsilon=.95$ )	54.28	68.58	69.64	2.70
PARA ( $\epsilon=.99$ )	80.16	68.44	69.78	2.56
Spectral Surgery	—	68.76	68.28	3.72

Table 2: Post-hoc editing methods on 18 checkpoints.

Figure 3: Gain redistribution and energy reduction. (a) Module-median FPR and (b) module-summed energy relative to LoRA under frozen-base inputs. Each row shows three checkpoints; headers give arithmetic means over all 18. The dashed line in (b) marks LoRA (=1; log scale). Code/Math/Instr. denote Magicoder/MetaMath/Tulu.

Figure 4: Matched-strength controls. Mean paired differences (pp) across 18 checkpoints with 95% bootstrap intervals. Positive favors LoRA-Norm: LoRA-Norm minus control for Target/Off, and control minus LoRA-Norm for FG.

Figure 5: Stronger functional equalization. Diagnostics (a) and paired changes versus LoRA-Norm (b, pp). Positive changes mean higher Target/Off or lower FG. Dots: means; lines: 95% checkpoint-bootstrap intervals. Scales differ; dashes mark the reference.

Appendix figures & tables38 assets

Supplementary material from the paper’s appendix.

Appendix

Training task	Target category	Examples	Metric	Tokens
Magicoder	HumanEval	164	Greedy pass@1	512
MetaMath	GSM8K	1,319	Strict accuracy	512
Tulu	IFEval	541	Prompt strict	2,048
—	Commonsense-8	22,419	Macro accuracy	8

Appendix

Table 3: Evaluation categories and maximum generated lengths. Commonsense-8 is a macro-average across its eight constituent tasks, not a pooled 22,419-example accuracy.

Method	Target $\uparrow$	$\Delta$ vs. LoRA	W/T/L
LoRA	68.040	0.000	0/18/0
Flat-Fro	71.731	3.691	16/0/2
Flat-Nuclear	72.014	3.974	16/1/1
DG-Hard	68.040	0.000	0/18/0
LoRA-Norm	72.165	4.125	17/1/0

Appendix

Table 4: Parameter-space controls in the separate joint evaluation block. All scores and differences in this table use that block’s own LoRA/LoRA-Norm references, not those in Table 1 . Target is averaged across 18 checkpoints; W/T/L denotes paired target wins/ties/losses versus LoRA.

Base	Task	Epochs	Length	Batch	LR	$\beta_{2}$	Schedule
Qwen	Code	1	4096	$16\times 2$	$2\times 10^{-5}$	.999	Cosine
Qwen	Math	3	4096	$16\times 2$	$10^{-4}$	.999	Cosine
Qwen	Instr.	2	4096	$16\times 8$	$4\times 10^{-4}$	.999	Cosine
Llama	Code	1	4096	$16\times 2$	$2\times 10^{-5}$	.999	Cosine
Llama	Math	3	1024	$64\times 12$	$10^{-4}$	.95	CosMin
Llama	Instr.	2	1024	$64\times 2$	$2\times 10^{-4}$	.95	CosMin

Appendix

Table 5: Actual settings for each of runs 43 and 44. Batch lists micro-batch times gradient accumulation; each training run uses one GPU. CosMin denotes cosine with a minimum learning rate. Warmup is zero in every row.

Source	Recorded settings and differences
Qwen / Code	Model card: 50K examples, 1 epoch, length 4096, global batch 32, LR 2e-5, rank 16, seed 42. Micro-batch and complete optimizer/Trainer settings are unverified.
Qwen / Math	Run arguments: 3 epochs, length 4096, global batch 32, LR 1e-4, requested warmup .05, batch $1\times 32$ . Runs 43/44 use $16\times 2$ and zero actual warmup; subset identity across JSON/parquet records is unverified.
Qwen / Instr.	Run arguments: 29,980 examples, 2 epochs, length 4096, global batch 128, LR 4e-4, requested warmup .03, batch $1\times 128$ . Runs 43/44 use $16\times 8$ , padding to a multiple of eight, and zero actual warmup; subset identity is unverified.
Llama / Code	Source documentation: 50K examples, 1 epoch, length 4096, global batch 32, LR 2e-5, rank/alpha 16/32. Training seed and complete Trainer settings are unverified.
Llama / Math	Model card: MetaMath 50K, rank 16. Training seed and complete recipe are unverified.
Llama / Instr.	Model card: length 1024, global batch 128, rank 16, seed 42. Complete optimizer and schedule settings are unverified.

Appendix

Table 6: Recorded settings for the archived run-42 sources. Unverified fields and known differences from runs 43 and 44 are noted explicitly.

Method	Target $\uparrow$	Off $\uparrow$	FG $\downarrow$	FPR	$R_{E}$
LoRA	67.998	69.551	2.708	2.023	1.0000
HNS	72.083	72.792	0.344	7.345	0.0972
F-HNS 0.5	72.108	72.586	0.254	12.728	0.0636
F-HNS 1	71.596	72.284	0.289	15.982	0.0536
PF	71.322	72.340	0.313	16.000	0.0535

Appendix

Table 7: Primary five-method results, averaged over 18 source checkpoints. FPR is the mean checkpoint-level module median, and

R_{E}

is the mean functional-energy ratio relative to LoRA, with module energies summed before each checkpoint ratio is formed. Energy ratios are shown to four decimals. Higher Target/Off and lower FG are favorable; FPR and

R_{E}

are descriptive diagnostics.

Base	Task	LoRA	HNS	F-HNS 0.5	F-HNS 1	PF
Qwen	Code	$64.228\pm 1.269$	$74.593\pm 0.352$	$72.358\pm 1.535$	$68.293\pm 0.610$	$68.293\pm 0.610$
Qwen	Math	$84.054\pm 0.088$	$87.465\pm 0.684$	$87.617\pm 0.789$	$87.440\pm 0.631$	$87.389\pm 0.444$
Qwen	Instr.	$66.975\pm 0.834$	$71.411\pm 0.912$	$72.397\pm 0.385$	$72.828\pm 0.640$	$72.520\pm 1.083$
Llama	Code	$54.472\pm 1.408$	$54.675\pm 0.931$	$54.878\pm 0.610$	$55.081\pm 0.352$	$54.472\pm 0.352$
Llama	Math	$75.537\pm 1.390$	$79.353\pm 1.178$	$80.642\pm 0.684$	$81.299\pm 0.158$	$80.869\pm 0.374$
Llama	Instr.	$62.723\pm 0.565$	$65.003\pm 0.213$	$64.757\pm 1.203$	$64.633\pm 0.594$	$64.387\pm 0.930$

Appendix

Table 8: Target: three-run mean

\pm

sample SD (ddof=1).

Base	Task	LoRA	HNS	F-HNS 0.5	F-HNS 1	PF
Qwen	Code	$78.037\pm 0.448$	$81.261\pm 0.071$	$80.488\pm 0.305$	$80.192\pm 0.178$	$80.163\pm 0.178$
Qwen	Math	$68.158\pm 4.103$	$75.277\pm 1.235$	$75.464\pm 1.634$	$76.010\pm 1.638$	$76.002\pm 1.389$
Qwen	Instr.	$85.627\pm 0.457$	$84.931\pm 0.444$	$84.855\pm 0.514$	$84.168\pm 1.075$	$84.093\pm 0.842$
Llama	Code	$61.198\pm 1.030$	$65.802\pm 0.917$	$65.638\pm 1.187$	$65.643\pm 0.373$	$65.657\pm 0.659$
Llama	Math	$58.591\pm 0.931$	$62.710\pm 0.554$	$63.154\pm 0.182$	$62.391\pm 0.363$	$62.301\pm 0.044$
Llama	Instr.	$65.696\pm 1.870$	$66.770\pm 0.800$	$65.919\pm 0.279$	$65.301\pm 1.155$	$65.827\pm 1.247$

Appendix

Table 9: Off: three-run mean

\pm

sample SD (ddof=1).

Base	Task	LoRA	HNS	F-HNS 0.5	F-HNS 1	PF
Qwen	Code	$1.666\pm 0.483$	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.000\pm 0.000$
Qwen	Math	$4.948\pm 4.054$	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.000\pm 0.000$
Qwen	Instr.	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.000\pm 0.000$
Llama	Code	$4.597\pm 0.522$	$1.289\pm 0.351$	$0.915\pm 0.091$	$0.847\pm 0.096$	$0.901\pm 0.174$
Llama	Math	$4.066\pm 0.681$	$0.666\pm 0.892$	$0.458\pm 0.740$	$0.721\pm 0.666$	$0.801\pm 0.969$
Llama	Instr.	$0.973\pm 0.955$	$0.109\pm 0.190$	$0.149\pm 0.204$	$0.166\pm 0.159$	$0.173\pm 0.160$

Appendix

Table 10: FG: three-run mean

\pm

sample SD (ddof=1).

Method	$\Delta$ Target	W/T/L	$\Delta$ Off	W/T/L	$\Delta$ FG	W/T/L
F-HNS 0.5	0.025	10/0/8	-0.205	7/0/11	-0.091	4/11/3
F-HNS 1	-0.488	9/3/6	-0.507	5/0/13	-0.055	4/9/5
PF	-0.762	9/1/8	-0.451	5/0/13	-0.032	3/9/6

Appendix

Table 11: Paired changes against HNS across 18 sources. Differences are method minus HNS; W/T/L means win/tie/loss, reversing the favorable direction for FG.

Base	Task	Run	LoRA	HNS	F-HNS 0.5	F-HNS 1	PF
Qwen	Code	42	65.244	75.000	70.732	68.293	67.683
Qwen	Code	43	62.805	74.390	73.780	68.902	68.902
Qwen	Code	44	64.634	74.390	72.561	67.683	68.293
Qwen	Math	42	84.155	88.173	88.249	87.945	87.566
Qwen	Math	43	84.003	86.808	86.732	86.732	86.884
Qwen	Math	44	84.003	87.415	87.870	87.642	87.718

Appendix

Table 12: Target: all 18 source checkpoints under the final primary protocol.

Base	Task	Run	LoRA
Qwen	Code	42	1.900
Qwen	Code	43	1.986
Qwen	Code	44	1.111
Qwen	Math	42	1.799
Qwen	Math	43	3.523
Qwen	Math	44	9.522

Appendix

Table 13: FG: all 18 source checkpoints under the final primary protocol.

Base	Task	Run	LoRA	HNS	F-HNS 0.5	F-HNS 1	PF
Qwen	Code	42	77.798	81.333	80.407	80.018	79.967
Qwen	Code	43	77.759	81.191	80.825	80.184	80.209
Qwen	Code	44	78.553	81.257	80.231	80.373	80.314
Qwen	Math	42	71.293	74.920	75.249	75.925	75.703
Qwen	Math	43	69.666	76.650	77.195	77.689	77.517
Qwen	Math	44	63.515	74.259	73.947	74.417	74.787

Appendix

Table 14: Off: all 18 source checkpoints under the final primary protocol.

Method	Raw FPR	Full PR	Energy ratio	RMS ratio
LoRA	2.023	1.844	1.000	1.000
HNS	7.345	5.392	0.097	0.288
F-HNS 0.5	12.728	8.831	0.064	0.219
F-HNS 1	15.982	12.113	0.054	0.198
PF	16.000	12.130	0.053	0.197

Appendix

Table 15: Functional diagnostics averaged over 18 sources. Each PR is a module median before checkpoint averaging. Energy and RMS are calculated per checkpoint before averaging.

Audit	Module rows	Maximum relative error
Scalar-E / Flat-E: analytic energy	8568	$8.882\times 10^{-16}$
Scalar-E / Flat-E: saved energy	8568	$7.519\times 10^{-6}$
Scalar-F: saved Frobenius norm	4284	$3.054\times 10^{-6}$

Appendix

Table 16: Matching audits on the completed 18-source extension. Analytic energy checks precede storage; saved-factor errors include reconstruction/storage effects.

Method	Calibration	Target $\uparrow$	Off $\uparrow$	FG $\downarrow$
LoRA	—	67.998	69.551	2.708
HNS	No	72.083	72.792	0.344
Scalar-F	No	70.643	71.558	1.111
Scalar-E	Yes	72.700	72.470	0.525
Flat-E	Yes	72.273	72.566	0.396

Appendix

Table 17: Matched-strength controls on 18 source checkpoints. Calibration indicates whether constructing the edit requires activation statistics. Every matching target is derived from the corresponding HNS update. Higher Target/Off and lower FG are favorable.

Base	Task	Run	HNS	Scalar-F	Scalar-E	Flat-E
Qwen	Code	42	75.000	75.000	75.610	75.610
Qwen	Code	43	74.390	68.902	76.829	75.000
Qwen	Code	44	74.390	69.512	77.439	75.000
Qwen	Math	42	88.173	84.913	86.960	88.249
Qwen	Math	43	86.808	85.064	86.126	87.339
Qwen	Math	44	87.415	85.823	85.444	87.642

Appendix

Table 18: Target: complete matched-strength comparison for the 18 source checkpoints.

Base	Task	Run	HNS	Scalar-F	Scalar-E	Flat-E
Qwen	Code	42	81.333	79.632	81.648	81.405
Qwen	Code	43	81.191	80.063	80.831	80.835
Qwen	Code	44	81.257	80.051	81.613	81.292
Qwen	Math	42	74.920	73.778	73.380	74.172
Qwen	Math	43	76.650	73.019	74.389	76.533
Qwen	Math	44	74.259	70.264	72.868	74.477

Appendix

Table 19: Off: complete matched-strength comparison for the 18 source checkpoints.

Base	Task	Run	Scalar-F	Scalar-E
Qwen	Code	42	0.253	0.000
Qwen	Code	43	0.101	0.000
Qwen	Code	44	0.000	0.000
Qwen	Math	42	0.000	0.407
Qwen	Math	43	0.493	0.000
Qwen	Math	44	3.160	1.016

Appendix

Table 20: FG: complete matched-strength comparison for the 18 source checkpoints.

Base	Task	Method	Target	Off	FG
Qwen	Code	HNS	$74.593\pm 0.352$	$81.261\pm 0.071$	$0.000\pm 0.000$
Qwen	Code	Scalar-F	$71.138\pm 3.358$	$79.915\pm 0.245$	$0.118\pm 0.127$
Qwen	Code	Scalar-E	$76.626\pm 0.931$	$81.364\pm 0.462$	$0.000\pm 0.000$
Qwen	Code	Flat-E	$75.203\pm 0.352$	$81.177\pm 0.302$	$0.000\pm 0.000$
Qwen	Math	HNS	$87.465\pm 0.684$	$75.277\pm 1.235$	$0.000\pm 0.000$
Qwen	Math	Scalar-F	$85.267\pm 0.487$	$72.354\pm 1.849$	$1.218\pm 1.700$

Appendix

Table 21: Matched-strength comparisons by backbone and source task: three-run mean

\pm

sample SD.

Comparison	Outcome	Mean	CI low	CI high
HNS $-$ Scalar-E	Target	-0.616	-1.640	0.249
HNS $-$ Scalar-E	Off	0.322	-0.125	0.828
HNS $-$ Scalar-E	FG	-0.181	-0.346	-0.037
HNS $-$ Flat-E	Target	-0.190	-0.616	0.259
HNS $-$ Flat-E	Off	0.225	0.104	0.356
HNS $-$ Flat-E	FG	-0.052	-0.138	0.025

Appendix

Table 22: All six paired comparisons among the four edits. Differences and 95% percentile intervals are in percentage points; negative FG differences favor the left method.

Group	Comparison	Outcome	Mean	CI low	CI high
Qwen/Code	HNS $-$ Scalar-E	Target	-2.033	-3.049	-0.610
Qwen/Code	HNS $-$ Scalar-E	Off	-0.104	-0.356	0.360
Qwen/Code	HNS $-$ Scalar-E	FG	0.000	0.000	0.000
Qwen/Code	HNS $-$ Flat-E	Target	-0.610	-0.610	-0.610
Qwen/Code	HNS $-$ Flat-E	Off	0.083	-0.072	0.356
Qwen/Code	HNS $-$ Flat-E	FG	0.000	0.000	0.000

Appendix

Table 23: HNS versus each control within each backbone–task group (three sources per group). Unadjusted descriptive intervals; these small-group results are not used to select favorable configurations.

Figure 6: Incremental predictive value of participation ratios beyond energy. Points show the change in leave-one-checkpoint-out

R^{2}

when raw FPR or full-moment PR is added to log functional-energy ratio, for Target and Off gains. Edited and matched denote cohort B (six edited methods, 108 observations) and cohort C (three energy-matched methods, 54 observations), respectively; both contain 18 source checkpoints. Bars reproduce the reported 95% bootstrap intervals conditional on the saved held-out predictions. Outcomes are centered within each checkpoint using its outcome mean; this is conditional prediction of relative deviations, not label-free prediction of absolute performance.

Cohort	PR	Outcome	Model	$R^{2}$	$R^{2}_{\rm LOCO}$	Rank
A	Raw	Target gain	Energy	0.2120	0.1927	1
A	Raw	Target gain	FPR	0.0148	-0.0206	1
A	Raw	Target gain	Energy+FPR	0.3004	0.2127	2
A	Full	Target gain	Energy	0.2120	0.1927	1
A	Full	Target gain	FPR	0.0077	-0.0276	1
A	Full	Target gain	Energy+FPR	0.3235	0.2404	2

Appendix

Table 24: Complete Energy/PR/combined fits for the three aggregate outcomes. Full denotes full-moment PR. Repeated Energy rows are the same baseline for each PR comparison; rank is the fitted design rank.

Cohort	PR	Outcome	$\Delta R^{2}$	$\Delta R^{2}_{\rm LOCO}$	CI low	CI high
A	Raw	Target gain	0.0884	0.0201	-0.2344	0.1290
A	Full	Target gain	0.1115	0.0477	-0.2156	0.1657
A	Raw	Off gain	0.0480	0.0327	-0.0446	0.1432
A	Full	Off gain	0.0692	0.0559	-0.0387	0.1976
A	Raw	FG reduction	0.0201	0.0131	-0.0289	0.0563
A	Full	FG reduction	0.0268	0.0179	-0.0475	0.0656

Appendix

Table 25: Incremental PR above Energy. The last two columns are the 95% bootstrap interval for

\Delta R^{2}_{\rm LOCO}

, conditional on the saved held-out predictions.

Cohort	Feature	Outcome	$\rho$	CI low	CI high	Defined
A	Energy	Target gain	-0.3879	-0.5921	-0.1573	18/18
A	Raw	Target gain	0.2983	0.0477	0.5098	18/18
A	Full	Target gain	0.3103	0.0727	0.5265	18/18
A	Energy	Off gain	-0.0082	-0.2677	0.2595	18/18
A	Raw	Off gain	-0.0270	-0.2603	0.2243	18/18
A	Full	Off gain	-0.0270	-0.2673	0.2103	18/18

Appendix

Table 26: Mean within-checkpoint Spearman correlations for the same fixed cohorts and aggregate outcomes. Defined reports the number of nonconstant checkpoint comparisons.

State	Interpretation
000	All three models are incorrect.
001	New success: both Base and LoRA are incorrect; the edit is correct.
010	A LoRA-only success is lost after editing.
011	A LoRA-only success is retained after editing.
100	A base success lost by LoRA remains incorrect after editing.
101	A base success lost by LoRA is recovered after editing.

Appendix

Table 27: The eight correctness transitions; categories describe evaluation behavior, not inferred causal mechanisms.

Role	Method	Recovery	Retention	New success	New damage	R/T
Overall	HNS	62.351	62.288	4553	3163	18/18
Overall	Scalar-E	62.876	63.220	3893	3462	18/18
Overall	Flat-E	62.430	62.464	4487	3200	18/18
Target	HNS	58.059	72.368	306	232	18/18
Target	Scalar-E	65.414	70.811	286	265	18/18
Target	Flat-E	59.439	72.833	308	235	18/18

Appendix

Table 28: Role-specific macro rates (percent) and summed event counts. R/T coverage counts defined checkpoint-level Recovery/Retention rates out of 18; Overall includes target and off-task families.

Method	000	001	010	011	100	101	110	111
HNS	81930	4553	10078	12267	11080	16893	3163	300010
Scalar-E	82590	3893	9928	12417	11883	16090	3462	299711
Flat-E	81996	4487	10026	12319	11105	16868	3200	299973

Appendix

Table 29: All eight pooled overall transition counts. Each method uses the same source checkpoints and evaluation examples.

Benchmark	Method	Recovery	Retention	New success	New damage	R/T
HumanEval	HNS	57.949	68.271	70	44	18/18
HumanEval	Scalar-E	58.140	69.631	67	40	18/18
HumanEval	Flat-E	58.476	67.168	71	46	18/18
GSM8K	HNS	62.983	70.737	693	487	18/18
GSM8K	Scalar-E	64.978	71.324	691	529	18/18
GSM8K	Flat-E	62.229	71.750	653	478	18/18

Appendix

Table 30: Per-benchmark Recovery/Retention macro rates (percent), pooled new-success/damage events, and defined checkpoint coverage. Each benchmark includes all 18 source checkpoints regardless of its target/off-task role.

Role	Comparison	Metric	Mean	CI low	CI high
Overall	HNS $-$ Scalar-E	Recovery	-0.524	-2.587	1.770
Overall	HNS $-$ Scalar-E	Retention	-0.932	-2.657	0.969
Overall	HNS $-$ Flat-E	Recovery	-0.079	-0.627	0.508
Overall	HNS $-$ Flat-E	Retention	-0.176	-0.821	0.479
Overall	Scalar-E $-$ Flat-E	Recovery	0.446	-1.810	2.332
Overall	Scalar-E $-$ Flat-E	Retention	0.756	-1.063	2.292

Appendix

Table 31: All method pairs for overall/target/off-task conditional rates. Differences and source-checkpoint 95% bootstrap intervals are in percentage points; all listed pairs have 18 defined source comparisons.

Record	Purpose
provenance.json	Source paths, code provenance, reuse and audit records.
source_checkpoints.json	The 18 retained source identities and configurations.
evaluation_sources.tsv	Evaluation reference/cache lineage and input-identity checks.
four_method_checkpoint_results.tsv	Four edited methods, checkpoint scores and eight commonsense components.
four_method_paired_ci.tsv	All method-pair performance intervals for overall and fixed groups.
module_energy_audit.tsv	Per-module analytic/saved energy matching.

Appendix

Table 32: Supplementary records under the analysis-section result directory. File names identify machine-readable records. The analysis tables are embedded in this appendix; the separate parameter-control table retains its original external TeX input.

Comparator	Metric	Difference	CI low	CI high
PARA $\epsilon=.90$	Target	+3.306	+1.496	+5.348
PARA $\epsilon=.90$	Off	+2.779	+1.506	+4.116
PARA $\epsilon=.90$	FG	-2.027	-3.129	-1.080
PARA $\epsilon=.95$	Target	+3.501	+2.114	+5.063
PARA $\epsilon=.95$	Off	+3.156	+1.757	+4.547
PARA $\epsilon=.95$	FG	-2.351	-3.502	-1.401

Appendix

Table 33: LoRA-Norm minus each new baseline: cohort means and 95% paired source-checkpoint bootstrap intervals, in percentage points. Positive differences favor LoRA-Norm for Target/Off; negative differences favor it for FG. Intervals are descriptive and unadjusted.

Base	Task	Run	Method	Target	Off	FG
Llama	Code	42	PARA $\epsilon=.90$	56.707	61.146	4.152
Llama	Code	42	PARA $\epsilon=.95$	54.878	61.489	3.809
Llama	Code	42	PARA $\epsilon=.99$	53.659	60.559	4.740
Llama	Code	42	Spectral Surgery	54.268	59.375	5.924
Llama	Code	43	PARA $\epsilon=.90$	56.098	61.743	4.439
Llama	Code	43	PARA $\epsilon=.95$	54.878	61.690	4.315

Appendix

Table 34: Complete new-baseline results for all 18 source checkpoints. Target and Off are percentages; FG is in percentage points. Code, Math, and IF denote Magicoder, MetaMath, and Tulu training, respectively.

	Qwen3-8B			Llama-3.1-8B-Instruct
Setting	HE	GSM	IF	HE	GSM	IF
Base	66.46	85.67	70.61	52.44	62.40	62.11
LoRA	66.46	84.15	67.65	53.66	77.10	63.22
$0+0$ (rebuild)	67.68	84.53	67.65	54.27	77.33	63.59
$2+0$	75.61	88.25	70.06	53.66	80.89	65.06
$2+1$	74.39	88.25	70.79	54.27	80.82	65.06

Appendix

Table 35: Complete step grid for the six archived checkpoints matching the primary adaptation tasks (percentages). HE/GSM/IF denote HumanEval, GSM8K, and IFEval, evaluated on their corresponding training-task adapters. The bold row label identifies the main

4+1

setting, not a score maximum.

Setting	Mean gain	Median gain	Win / tie / loss
$0+0$ (rebuild)	+0.47	+0.37	5 / 1 / 0
$2+0$	+3.55	+3.10	5 / 1 / 0
$2+1$	+3.56	+3.43	6 / 0 / 0
$2+2$	+3.46	+2.79	5 / 1 / 0
$4+0$	+3.62	+3.74	5 / 1 / 0
4+1	+3.57	+3.08	6 / 0 / 0

Appendix

Table 36: Fixed-setting gains over original LoRA on the six archived checkpoints matching the primary adaptation tasks (percentage points). Means and medians weight checkpoints equally; wins/ties/losses use unrounded scores.

Base	Target	Gain	95% interval	$p_{\mathrm{Holm}}$
Qwen	HumanEval	+8.54	$[1.22,15.85]$	0.213
Qwen	GSM8K	+3.87	$[2.12,5.69]$	$<0.001$
Qwen	IFEval	+2.59	$[0.00,5.18]$	0.448
Llama	HumanEval	+1.22	$[-4.27,6.71]$	1.000
Llama	GSM8K	+3.56	$[1.74,5.38]$	$<0.001$
Llama	IFEval	+1.66	$[-1.48,4.81]$	1.000

Appendix

Table 37: Archived

4+1

versus original LoRA on the six primary-task checkpoints: gain and ordinary paired 95% interval in percentage points.

p_{\mathrm{Holm}}

adjusts ten within-checkpoint comparisons. Per-checkpoint intervals and adjusted

p

-values are unchanged by the cohort restriction.

$\gamma$	Strict	Numeric
1.00	84.77	85.16
0.85	85.55	86.72
0.70	85.94	87.50
0.60	85.16	86.72
0.50	86.72	88.67
0.40	88.67	91.02

Appendix

Table 38: Archived global-scalar calibration on 256 questions. Scores are percentages.

Variant	Strict	Numeric
Original LoRA	83.40	84.57
Common-basis zero rebuild	82.81	83.98
Common-basis global scalar ( $\gamma=0.40$ )	85.74	88.87
Common-basis per-module norm-matched scalar	84.77	86.72
Common-basis archived HNS	86.13	88.67
Archived HNS in its existing representation	86.72	89.45

Appendix

Table 39: Archived 512-question recheck. Common-basis variants share their source singular vectors and factorization path. Scores are percentages.

Explore similar work

Aug 31, 2026cs.LG

Normalized Low-Rank Adaptation

While low-rank adaptation (LoRA) is widely used for parameter-efficient model adaptation, how to regularize its training dynamics for stable and effective optimization remains underexplored. Because LoRA initializes the up-projection to zero, its early optimization dynamics are largely governed by the down-projection. Building on this observation, we introduce Normalized Low-Rank Adaptation (NoRA), a simple yet effective method that normalizes the down-projection matrices during training. We further show that the same normalization can be applied only at initialization, improving standard LoRA without requiring repeated normalization throughout training. Across pretraining, supervised finetuning, and reinforcement learning, NoRA consistently accelerates convergence, improves performance and training stability, and mitigates catastrophic forgetting. These benefits require neither additional trainable parameters nor inference-time computation, making NoRA a simple and broadly applicable enhancement to LoRA.

Jiale Kang, Ziyin Yue, Zheng Zhan +2

1Yuanshi Intelligence · 3The Chinese University of Hong Kong · 2Microsoft Research +1

Parameter-Efficient Adaptation Low-Rank Structure Normalization Norm

Ask PDF

May 29, 2026cs.LG

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence

Low-Rank Adaptation (LoRA) is the most widely adopted method for fine-tuning large language models. Notably, LoRA is inherently overparameterized: multiple pairs of low-rank factors can yield the same adapted weight matrix. We show--both theoretically and empirically--that these pairs exhibit significantly different condition numbers. As a result, converging to different loss minimizers directly impacts the convergence rate of LoRA. Building on this observation, we introduce Balanced Low-Rank Adaptation (BaLoRA), a variant of LoRA that projects iterates onto a balanced manifold. This manifold improves the conditioning of the loss landscape while preserving the adapted matrix. The projection step is computationally lightweight and integrates seamlessly into existing fine-tuning pipelines. Empirically, BaLoRA converges faster than standard LoRA and achieves superior performance across a range of fine-tuning tasks.

Valérie Castin, Kimia Nadjahi, Pierre Ablin +1

1 ´Ecole Normale Sup´erieure PSL, Paris, France · 2CNRS · 3Apple, Paris, France.

Training-Side Stage-Aware Low-Rank Adaptation Low-Rank Structure Convergence

Ask PDF

Jul 20, 2026cs.LG

PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer

Low-rank adaptation (LoRA) makes finetuning large language models cheaper by adding to each weight matrix a trainable low-rank update parameterized as the product of two matrices. These matrices are usually trained with Adam, which treats them as a single flat vector of parameters and ignores both the matrix and product structure of LoRA. Applying a matrix-aware optimizer such as Muon to each factor does not consistently improve over Adam, and neither do the product-aware Muon variants proposed in concurrent works. To realize consistent gains, we introduce PoLoRA, a Preconditioned Orthogonalized LoRA optimizer built from three ingredients: a product-aware spectral update direction, curvature preconditioning derived from controlling the per-sample loss change, and a magnitude rule that controls the sizes of both the factor and merged updates. We evaluate PoLoRA on instruction-tuning datasets for code and math across models from 1B to 8B parameters, and find that it reaches the final held-out loss achieved by tuned Adam in 1.2-1.7 times fewer steps, while adding at most 3% per-step overhead. Compared to Adam, PoLoRA is also less sensitive to the learning rate, and its optimal learning rate is stable across ranks.

Nikhil Ghosh, Tetiana Parshakova, Robert M. Gower

Center for Computational Mathematics, Flatiron Institute, New York, NY

Multimodal Continual Instruction Tuning Muon Preconditioner

Ask PDF