NormTune: Norm-Guided Parameter-Efficient Fine-Tuning for Reasoning-Intensive Tasks

Zhangming Li; Qinghao Hu; Weifan Guan; Yifan Zhang; Peisong Wang; Jian Cheng

cs.AIOct 21, 2026

NormTune: Norm-Guided Parameter-Efficient Fine-Tuning for Reasoning-Intensive Tasks

Authors: Zhangming Li, Qinghao Hu, Weifan Guan, Yifan Zhang, Peisong Wang, Jian Cheng

Organizations: 1The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · 3AiRiA · School of Computer Science and Engineering, Nanjing University of Science and Technology

Abstract

Full fine-tuning has achieved remarkable success on reasoning-intensive tasks, but it often requires substantial GPU resource consumption. Parameter-efficient fine-tuning (PEFT) provides a promising alternative for reducing this cost, yet we observe a substantial performance gap on reasoning-intensive tasks. In this paper, we present three key observations showing that this gap is closely associated with norm-related under-adaptation, where PEFT produces markedly smaller weight-update norms than full fine-tuning. Based on these observations, we propose Norm-Guided Parameter-Efficient Fine-Tuning (NormTune), a simple yet effective strategy that adaptively strengthens insufficiently updated layers according to their weight-update norms. Extensive experiments show that NormTune consistently enhances multiple PEFT methods across challenging reasoning benchmarks. Specifically, when trained on S1K-1.1, integrating NormTune into rank-8 LoRA, Bone, and MoRA yields substantial average accuracy improvements of 24.24%, 11.33%, and 16.21% , respectively, across AIME 2024, MATH 500, GPQA, and AIME 2025. Moreover, NormTune-enhanced rank-256 LoRA, Bone, and MoRA achieve performance comparable to S1 full fine-tuning, with respective average gains of 2.17%, 0.93%, and 3.00% over this baseline.

Figures & tables

Panel of Figure 1 — Figure 1: Performance across evaluation regimes and token-level loss under PEFT and full fine-tuning. Detailed results are reported in Appendix Tables 16 , 17 , and 11 .

Panel of Figure 2 — Figure 2: Layer-wise norm comparison among Full-FT, rank-256 PEFT methods, early-stage PEFT statistics, and NormTune+PEFT methods.

Panel of Figure 3 — Figure 3: Layer-wise norm comparison between full fine-tuning (Full-FT) and rank-8 PEFT methods on (a) commonsense reasoning, (b) GovReport, and (c) reasoning-intensive tasks.

Panel of Figure 4 — Figure 4: Gradient alignment analysis between token groups and layer-wise optimization directions. The left panel shows the cosine similarity between different gradients. The right panels visualize representative gradient directions and the L1 norms of the gradients for different LoRA parameters (LoRA A and B) across early and late layers after 30 training steps.

Model	Method	GPQA		AIME’24		AIME’25		Average
		Avg@4	Pass@4	Avg@16	Pass@16	Avg@16	Pass@16
OLMo-3-7B	Full-FT ∗	34.0	62.1	42.3	80.0	39.2	63.3	53.5
	AdaLoRA	42.6	66.2	22.1	43.3	22.7	50	41.2
	DoRA	30.6	57.5	33.3	73.3	30.8	56.7	47.0
	RSLoRA	42.6	70.2	22.7	56.6	21.7	53.3	44.5
	LoRA+	40.9	65.2	21.3	53.3	16.5	46.7	40.7

Table 3: Performance of Full-FT, PEFT, and NormTune-enhanced PEFT methods on OLMo-3-7B across GPQA, AIME’24, and AIME’25. ∗ denotes results reported in Kopiczko et al. (2026) .

Panel of Figure 5 — Figure 5: Results after training on S1K-1.1.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method	GPQA		AIME’24		AIME’25		Average
		Avg@4	Pass@4	Avg@16	Pass@16	Avg@16	Pass@16
Qwen3-4B	Full-FT ∗	39.3	68.7	19.2	43.3	18.8	40.0	38.2
	AdaLoRA	21.8	54	7.1	30	9.8	26.7	24.9
	DoRA	38.1	72.2	15	33.3	16.3	43.3	36.4
	RSLoRA	21.3	51.5	6.9	26.7	5.8	20	22.0
	LoRA+	27	62.1	12.5	36.7	8.1	30	29.4

Appendix

Table 7: Performance of Full-FT, PEFT, and NormTune-enhanced PEFT methods on Qwen3-4B across GPQA, AIME’24, and AIME’25. ∗ denotes results reported in Kopiczko et al. (2026) .

Method	Rank	ROUGE-1	ROUGE-2	ROUGE-L	ROUGE-Lsum	Average
Full-FT	-	55.11	27.39	28.95	28.95	35.10
LoRA	8	52.6	20.63	24.67	31.84	32.44
Bone	8	60.64	30.45	31.94	31.93	38.74
MoRA	8	58.8	30	31.29	31.08	37.79

Appendix

Table 17: ROUGE results (%) on the GovReport dataset.

Figure 6: Effect of the bucket number

n

in NormTune on GPQA after training on Qwen3-4B.

Figure 7: Gradient alignment across early and late layers, measured by $\mathrm{CosSim}(\nabla_{\theta}\mathcal{L}_{\mathrm{reasoning}},\nabla_{\theta}\mathcal{L}_{\mathrm{normal}})$ and $\mathrm{CosSim}(\nabla_{\theta}\mathcal{L}_{\mathrm{total}},\nabla_{\theta}\mathcal{L}_{\mathrm{normal}})$ .

Hyperparameter	GovReport				commonsense reasoning
Hyperparameter	Full-FT	LoRA	Bone	MoRA	Full-FT	LoRA	Bone	MoRA
Batch size	16	16	16	16	128	128	128	128
Optimizer	Adam	Adam	Adam	Adam	Adam	Adam	Adam	Adam
LR	1e-5	1e-5	1e-5	1e-5	3e-5	3e-5	3e-5	3e-5
Warmup Ratio	0.05	0.05	0.05	0.05	0.05	0.05	0.05	0.05
Epoch	5	5	5	5	3	3	3	3

Appendix

Table 18: Hyperparameter settings for GovReport and commonsense reasoning training configurations.

Hyperparameter	S1K-1.1	Z1-Code- Reasoning107K	Data-repetition	OLMo-3-7B
Batch size	16	128	1	1
Optimizer	Adam	Adam	8-bit Adam	8-bit Adam
$\eta_{\min}$	1e-5	1e-5	1e-5, 3e-5	2e-5
$\eta_{\max}$	1e-4 ,1e-3	2e-5, 1e-4	2e-5,3e-5, 1e-4	1e-4
$n$ buckets	16	16	16	16
Probing Steps	30	30	30	30

Appendix

Table 19: NormTune-specific configuration for different training benchmark. provides the optimal choice.

Configuration	Method	Batch Size	Optimizer	LR	Warmup Ratio	Epoch	Weight Decay	Rank	LoRA $\alpha$	MoRA Type	AdaLoRA_init_rank	Block Size
S1K-1.1	Full-FT	16	Adam	1e-5	0.05	5	1e-4	–	–	–	–	32768
	LoRA	16	Adam	1e-5	0.05	5	1e-4	8,256	16,512	–	–	32768
	Bone	16	Adam	1e-5	0.05	5	1e-4	8,256	–	–	–	32768
	MoRA	16	Adam	1e-5	0.05	5	1e-4	8,256	–	6,1	–	32768
	AdaLoRA	16	Adam	1e-5	0.05	5	1e-4	8,256	–	–	12,384	32768
	DoRA	16	Adam	1e-5	0.05	5	1e-4	8,256	–	–	–	32768

Appendix

Table 20: Hyperparameter settings for different PEFT training configurations.

Configuration	Method	Batch Size	Optimizer	LR	Warmup Ratio	Epoch	Weight Decay	Rank	LoRA $\alpha$	MoRA Type	AdaLoRA_init_rank	Block Size
S1K-1.1	Full-FT	16	Adam	1e-5	0.05	5	1e-4	–	–	–	–	32768
	LoRA	16	Adam	1e-5	0.05	5	1e-4	8,256	16,512	–	–	32768
	Bone	16	Adam	1e-5	0.05	5	1e-4	8,256	–	–	–	32768
	MoRA	16	Adam	1e-5	0.05	5	1e-4	8,256	–	6,1	–	32768
	AdaLoRA	16	Adam	1e-5	0.05	5	1e-4	8,256	–	–	12,384	32768
	DoRA	16	Adam	1e-5	0.05	5	1e-4	8,256	–	–	–	32768

Appendix

Table 21: Hyperparameter settings for TempBalance configurations.

Configuration	Evaluation suites	Metric / evaluator	Decoding	Max tokens	Completions per problem per run	Greedy evaluation runs ( $R$ )
S1K-1.1	AIME 2024, AIME 2025, MATH 500, GPQA	Mean greedy accuracy (GPT-4o-mini)	Greedy, $T=0$	32768	1	4: MATH 500, GPQA; 16: AIME 2024, AIME 2025
Z1-Code- Reasoning107K	LiveCodeBench, MATH 500, BigCodeBench-Hard, GPQA	Mean greedy accuracy (Qwen2.5-32B-Instruct)	Greedy, $T=0$	4096	1	4: MATH 500, GPQA; 16: LiveCodeBench, BigCodeBench-Hard
Data-repetition	GPQA, AIME 2024, AIME 2025	Avg@ $k$ and Pass@ $k$	Sampling, $T=0.6$ , top- $p=0.95$ , top- $k=20$	32768	4 (GPQA) 16 (AIME)	—
OLMo-3-7B	GPQA, AIME 2024, AIME 2025	Avg@ $k$ and Pass@ $k$	Sampling, $T=0.6$ , top- $p=0.95$ , top- $k=50$	32768	4 (GPQA) 16 (AIME)	—
GovReport	GovReport test split	ROUGE-1, ROUGE-2, ROUGE-L, ROUGE-Lsum	Sampling, $T=0.1$ , top- $p=0.9$	32768	1	—
Commonsense reasoning	BoolQ, PIQA, SIQA, WinoGrande, ARC-e/c, OBQA, HellaSwag	Accuracy	4-beam search; $T=0.1$ , top- $p=0.75$ , top- $k=40$ in the generation configuration	32	1	—

Appendix

Table 22: Evaluation settings for different training configurations. For S1K-1.1 and Z1-Code-Reasoning107K, mean greedy accuracy is averaged over repeated evaluation runs of the same checkpoint.

Fine-Tuning Language Models

See all 47 papers

Hierarchical Low-Rank Reparameterization for Parameter-Efficient Fine-Tuning
Oct 27, 2026Rong Huang, Yi Xiong, Yuan-Yuan Cheng +1LLM Fine-Tuning Parameter-Efficient Fine-Tuning
University of Science and Technology of China, Hefei, Anhui 230026, China
FoRA: Fisher-orthogonal Rank Adaptation for Parameter-Efficient Fine-Tuning
May 28, 2026Juneyoung Park, Seongbae Lee, Han-Sang Lee +6LLM Fine-Tuning Adapter Tuning
OptAI Inc · LG Uplus
ChainDoRA: Tensor-Train Factorized Weight-Decomposed Low-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning
Sep 8, 2026Ashfak Yeafi, Mehedi Hasan, Md Khairul IslamLLM Fine-Tuning Low-Rank Adaptation
Department of Electrical and Electronic Engineering, Khulna University of Engineering & Technology, Khulna, Bangladesh · Department of Computer Science and Engineering, Brac University, Dhaka, Bangladesh · Department of Mathematics and Computer Science, Hobart and William Smith Colleges, Geneva, NY, USA

Oct 27, 2026J/K move · Enter open · S save

Hierarchical Low-Rank Reparameterization for Parameter-Efficient Fine-Tuning

Rong Huang, Yi Xiong, Yuan-Yuan Cheng, Xiao-Ming Fu

University of Science and Technology of China, Hefei, Anhui 230026, China

LLM Fine-Tuning Parameter-Efficient Fine-Tuning Low-Rank Adaptation Low-Rank Compression

Open paper AskPDF arXiv

Parameter-efficient fine-tuning (PEFT) methods adapt large models by introducing a small number of trainable parameters. However, we find that these parameters still contain redundancy and can be further compressed while maintaining strong task performance. To this end, we propose a novel hierarchical low-rank reparameterization method. Its core idea is to reorganize the trainable parameters of PEFT methods into an approximately square matrix and then express it via a low-rank decomposition, which can be applied recursively. This forms a multi-level structure with factors compressed level by level, drastically reducing the number of trainable parameters. In our experiments, we combine this reparameterization with LoRA, DoRA, MoSA, and RaSA. We evaluate commonsense reasoning on LLaMA-3.1-8B and Qwen2.5-7B, with additional GSM8K math reasoning experiments on LLaMA-3.1-8B. Under comparable trainable-parameter budgets, our method combined with LoRA or DoRA achieves better performance. Furthermore,when the parameter count of the original methods is compressed by approximately an order of magnitude, our method can still maintain competitive task performance and even surpass the corresponding baselines in average accuracy across the eight commonsense reasoning benchmarks.

Abstract

Fine-Tuning Language Models

Explore similar work

Hierarchical Low-Rank Reparameterization for Parameter-Efficient Fine-Tuning

FoRA: Fisher-orthogonal Rank Adaptation for Parameter-Efficient Fine-Tuning

ChainDoRA: Tensor-Train Factorized Weight-Decomposed Low-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning