Hierarchical Low-Rank Reparameterization for Parameter-Efficient

Rong Huang; Yi Xiong; Yuan-Yuan Cheng; Xiao-Ming Fu

cs.CLOct 27, 2026

Hierarchical Low-Rank Reparameterization for Parameter-Efficient

Authors: Rong Huang, Yi Xiong, Yuan-Yuan Cheng, Xiao-Ming Fu

Organizations: University of Science and Technology of China, Hefei, Anhui 230026, China

Abstract

Parameter-efficient fine-tuning (PEFT) methods adapt large models by introducing a small number of trainable parameters. However, we find that these parameters still contain redundancy and can be further compressed while maintaining strong task performance. To this end, we propose a novel hierarchical low-rank reparameterization method. Its core idea is to reorganize the trainable parameters of PEFT methods into an approximately square matrix and then express it via a low-rank decomposition, which can be applied recursively. This forms a multi-level structure with factors compressed level by level, drastically reducing the number of trainable parameters. In our experiments, we combine this reparameterization with LoRA, DoRA, MoSA, and RaSA. We evaluate commonsense reasoning on LLaMA-3.1-8B and Qwen2.5-7B, with additional GSM8K math reasoning experiments on LLaMA-3.1-8B. Under comparable trainable-parameter budgets, our method combined with LoRA or DoRA achieves better performance. Furthermore,when the parameter count of the original methods is compressed by approximately an order of magnitude, our method can still maintain competitive task performance and even surpass the corresponding baselines in average accuracy across the eight commonsense reasoning benchmarks.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of HiLoRA (our method HiLoRR applied to LoRA). (a) Standard LoRA adds a rank- $r$ update $\Delta W=\frac{\alpha}{r}BA$ to the frozen weight $W_{0}$ . (b) HiLoRA reshapes each LoRA factor into a Near-Square matrix ( $B_{\text{reshape}}$ / $A_{\text{reshape}}$ ) and represents it by a second low-rank product $U_{B}V_{B}$ (resp. $U_{A}V_{A}$ ). The factor is then reshaped back into the reconstructed $B$ / $A$ (blue boxes), so the maximum rank of the matrix product $BA$ remains bounded by $r$ . HiLoRA directly parameterizes and trains only $U_{B},V_{B},U_{A},V_{A}$ , from which the factors ( $B$ / $A$ (blue boxes)) are reconstructed on the fly.

Method	Params(M)	Mem(GiB)	BoolQ	PIQA	SIQA	ARC-c	ARC-e	OBQA	HS	WG	Avg.	W-Avg.
LLaMA-3.1-8B
Low-parameter ( $\leq 6$ M)
LoRA-r2	3.54	11.149	62.17	86.40	75.59	80.46	91.67	82.80	90.26	74.11	80.43	83.13
DoRA-r2	4.33	11.160	62.17	86.45	75.38	81.40	91.54	84.00	90.40	74.27	80.70	83.25
SVFit	0.39	27.346	67.86	84.11	68.37	77.30	90.87	75.40	82.32	60.54	75.85	78.40
NoRA	5.24	24.675	62.17	86.51	75.59	82.00	92.68	82.40	91.20	76.80	81.17	83.88

Table 1: Accuracy (%) on eight commonsense reasoning benchmarks using LLaMA-3.1-8B and Qwen2.5-7B. Within each model, methods are grouped by trainable-parameter budget. Mem is training-phase peak PyTorch CUDA allocated memory in GiB. Rows shaded in gray indicate our methods.

Method	Params (M)	Mem (GiB)	Correct (/1,319)	GSM8K EM(%)
Base (no SFT)	0.00	–	390	29.57
Low-parameter ( $\leq 6$ M)
LoRA-r2	3.54	15.836	917	69.52
DoRA-r2	4.33	15.844	890	67.48
SVFit	0.39	31.447	711	53.90
NoRA	5.24	29.356	889	67.40

Table 2: Accuracy (%) on GSM8K after MetaMathQA-40K fine-tuning using LLaMA-3.1-8B. Our methods are shaded in gray.

Method	Params (M)	BoolQ	PIQA	SIQA	ARC-c	ARC-e	OBQA	HellaSwag	WinoGrande	Avg.	W-Avg.
LoRA	28.31	62.17	86.94	76.25	81.40	92.55	85.00	92.03	77.35	81.71	84.39
MoSA	28.31	70.64	85.47	77.79	82.51	90.87	84.40	90.92	81.85	83.06	85.27
HiMoSA	2.10	69.54	86.40	78.25	81.48	91.96	83.60	91.20	81.29	82.97	85.36
RaSA	28.32	69.60	86.18	79.73	81.06	92.55	84.20	92.72	81.14	83.40	86.21
HiRaSA	3.71	69.88	86.72	78.30	82.17	92.09	84.60	92.17	80.82	83.34	85.92

Table 3: Accuracy (%) on eight commonsense reasoning benchmarks when applying hierarchical low-rank reparameterization to other adapter families. All results use LLaMA-3.1-8B.

	Learning rate ( $\times 10^{-4}$ )
Metric	1	3	5	8	10	12	16	20	24	32
Avg. (%)	74.66	80.09	80.88	81.65	82.42	82.21	83.16	83.28	81.76	79.52
W-Avg. (%)	77.11	82.36	83.51	84.10	85.18	84.24	85.86	85.82	84.57	82.24

Table 4: Learning-rate ablation of low-parameter HiLoRA on LLaMA-3.1-8B across eight commonsense reasoning tasks. Best observed values are bold.

Variant	Params (M)	Avg.	W-Avg.	$\Delta$ W-Avg.
HiLoRA A+B	2.82	83.28	85.82	0.00
HiLoRA A-only	2.83	82.22	84.79	-1.03
HiLoRA B-only	2.82	81.92	84.55	-1.27

Table 5: Ablation study of HiLoRA variants on LLaMA-3.1-8B commonsense reasoning.

Low budget
Depth	Configuration	Params (M)	Avg.	W-Avg.
Single	$r{=}16,\,k_{1}{=}8$	1.409	80.45	83.17
Double	$r{=}16,\,k_{1}{=}16,\,k_{2}{=}16$	1.360	81.39	84.05
Medium budget
Single	$r{=}16,\,k_{1}{=}12$	2.114	82.07	84.84
Double	$r{=}16,\,k_{1}{=}16,\,k_{2}{=}24$	2.040	81.70	84.03

Table 6: HiLoRA depth–budget ablation on LLaMA-3.1-8B (Commonsense-15K).

Reshaping strategy	Params(M)	Avg.	W-Avg.	$\Delta$ W-Avg.
Near-Square	2.82	83.28	85.82	0.00
Random mean	2.82	80.95	83.65	-2.17
Elongated 1:4	2.82	81.02	83.40	-2.42
Elongated 1:16	2.93	81.04	83.55	-2.27

Table 7: Reshaping ablation on LLaMA-3.1-8B commonsense reasoning.

Configuration	Params (M)	Avg.	W-Avg.	$\Delta$ W-Avg.
$r{=}8,k_{A}{=}24,k_{B}{=}20$	2.86	82.01	84.30	-1.52
$r{=}16,k_{A}{=}16,k_{B}{=}16$	2.82	83.28	85.82	0.00
$r{=}24,k_{A}{=}14,k_{B}{=}12$	2.83	81.79	84.39	-1.43
$r{=}32,k_{A}{=}12,k_{B}{=}10$	2.86	81.94	84.33	-1.49
$r{=}64,k_{A}{=}8,k_{B}{=}8$	2.82	82.52	85.04	-0.78

Table 8: Ablation study of the outer rank

r

and hierarchical ranks

k_{A},k_{B}

for HiLoRA on LLaMA-3.1-8B commonsense reasoning.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 10 — Figure S1: Near-square reshaping and factor reconstruction in HiLoRA. (a) Element order is preserved while row boundaries change; numbers indicate element positions rather than weight values. (b) The factor is reconstructed as $X=\text{Back}(UV)$ , with only $U$ and $V$ directly optimized.

Hyperparameter	Commonsense	Math
Effective batch size	64	64
Micro-batch $\times$ accum.	$4\times 16$	$4\times 16$
Epochs	3	2
Max length	512	1024
Optimizer	AdamW	AdamW
HiLoRR $\alpha$	32	128

Appendix

Table S1: Common training hyperparameters. Target modules ( q,k,v,up,down_proj ) are shared across all methods.

Module	Factor	Shape	$d$	Reshape $(p{\times}q)$	$k$	std	FF	Ratio
q_proj	$B$ (out)	$4096{\times}16$	65,536	$256{\times}256$	16	65,536	8,192	$8.0\times$
q_proj	$A$ (in)	$16{\times}4096$	65,536	$256{\times}256$	16	65,536	8,192	$8.0\times$
k_proj	$B$ (out)	$1024{\times}16$	16,384	$128{\times}128$	16	16,384	4,096	$4.0\times$
k_proj	$A$ (in)	$16{\times}4096$	65,536	$256{\times}256$	16	65,536	8,192	$8.0\times$
v_proj	$B$ (out)	$1024{\times}16$	16,384	$128{\times}128$	16	16,384	4,096	$4.0\times$
v_proj	$A$ (in)	$16{\times}4096$	65,536	$256{\times}256$	16	65,536	8,192	$8.0\times$

Appendix

Table S2: HiLoRA reshaping configurations on LLaMA-3.1-8B (

r{=}16

k{=}16

, recursive double-sided variant). Per Transformer block, summed over

\{

q, k, v, up, down

\}

_proj.

d

: factor element count; std: the original PEFT formulation parameters; FF: reshaped parameters

k(p{+}q)

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 755 papers

ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition
Feb 24, 2026Xindian Ma, Rundong Kong, Peng Zhang +2LLM Fine-Tuning Low-Rank Adaptation
Tianjin University, Tianjin, China
ChainDoRA: Tensor-Train Factorized Weight-Decomposed Low-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning
Sep 8, 2026Ashfak Yeafi, Mehedi Hasan, Md Khairul IslamLLM Fine-Tuning Low-Rank Adaptation
Department of Electrical and Electronic Engineering, Khulna University of Engineering & Technology, Khulna, Bangladesh · Department of Computer Science and Engineering, Brac University, Dhaka, Bangladesh · Department of Mathematics and Computer Science, Hobart and William Smith Colleges, Geneva, NY, USA
Parameter-Efficient Fine-Tuning with Learnable Rank
Jun 3, 2026Arpit Garg, Simon Lucey, Hemanth SaratchandranNonlinear LoRA Low-Rank Adaptation
Australian Institute for Machine Learning Adelaide University

Feb 24, 2026J/K move · Enter open · S save

ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition

Xindian Ma, Rundong Kong, Peng Zhang, Ruoxiang Huang+1

Tianjin University, Tianjin, China

LLM Fine-Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

LoRA has become a universal Parameter-Efficient Fine-Tuning (PEFT) technique that equips Large Language Models (LLMs) to adapt quickly to new tasks. However, when these models are scaled up, even the latest LoRA variants still introduce considerable overhead in trainable parameters. Conversely, aggressively lowering the rank to curb this overhead markedly degrades performance in complex multi-task settings. We propose ID-LoRA, a novel PEFT framework that breaks the trade-off. Its core innovation lies in extracting and reusing clustered parameter groups from the pretrained weight matrix. These groups are then used to form multiple low-rank components, all of which share only a single initialized trainable low-rank matrix. This approach cuts the number of trainable parameters while keeping the model's capacity intact. We evaluate ID-LoRA on five diverse benchmarks: Mathematical Reasoning, Code Generation, MMLU, CommonsenseQA, and Safety Alignment. ID-LoRA outperforms both full fine-tuning and existing PEFT baselines (e.g., LoRA, DoRA, HydraLoRA) while using up to 46% fewer trainable parameters than the standard LoRA. In multi-task scenarios, it surpasses LoRA and its recent variants (e.g., DoRA and HydraLoRA) on both Code and MMLU tasks, yet requires only 54% of the trainable parameters demanded by the conventional LoRA.

Abstract

LLM Fine-Tuning

Explore similar work

ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition

ChainDoRA: Tensor-Train Factorized Weight-Decomposed Low-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning

Parameter-Efficient Fine-Tuning with Learnable Rank