IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoring

Xuan Cui; Huiyue Li; Run Zeng; Yunfei Zhao; Jinrui Qian; Wei Duan; Bo Liu; Zhanpeng Zhou

cs.LGMar 14, 2026

IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoring

Authors: Xuan Cui, Huiyue Li, Run Zeng, Yunfei Zhao, Jinrui Qian, Wei Duan, Bo Liu, Zhanpeng Zhou

Organizations: Chongqing Technology and Business University · Shanghai Jiao Tong University

Abstract

As large language models (LLMs) scale to billions of parameters, full-parameter fine-tuning becomes compute- and memory-prohibitive. Parameter-efficient fine-tuning (PEFT) mitigates this issue by updating only a small set of task-specific parameters while keeping the base model frozen. Among PEFT approaches, low-rank adaptation (LoRA) is widely adopted; however, it enforces a uniform rank across layers despite substantial variation in layer importance, motivating {layerwise} rank allocation. Recent adaptive-rank variants (e.g., AdaLoRA) allocate ranks based on importance scores, yet typically rely on instantaneous gradients that capture only local sensitivity, overlooking non-local, pathwise effects within the same layer, which yields unstable and biased scores. To address this limitation, we introduce IGU-LoRA, an adaptive-rank LoRA that (i) computes within-layer Integrated Gradients (IG) sensitivities and aggregates them into a layer-level score for rank allocation, and (ii) applies an uncertainty-aware scheme using exponential moving averages with deviation tracking to suppress noisy updates and calibrate rank selection. Theoretically, we prove an upper bound on the composite trapezoidal rule approximation error for parameter-space IG under a pathwise Hessian-Lipschitz condition, which informs the quadrature budget. Across diverse tasks and architectures, IGU-LoRA consistently outperforms strong PEFT baselines at matched parameter budgets, improving downstream accuracy and robustness. Ablations confirm the contributions of pathwise within-layer sensitivity estimates and uncertainty-aware selection to effective rank allocation. Our code is publicly available at https://github.com/withyou12/igulora.git

Figures & tables

Figure 1: Comparison of frameworks: left to right—(a) LoRA, (b) AdaLoRA, (c) the proposed IGU-LoRA. IGU-LoRA builds on LoRA and AdaLoRA, introducing integrated gradients (IG) to compute parameter importance scores. Please zoom in 300% for better clarity.

Panel of Figure 2 — Figure 2: Comparison of parameter importance scoring methods. (a) The simple gradient method fails in saturated regions, assigning near-zero importance. (b) Integrated gradients compute importance by integrating along the path from initial to final parameter values, capturing the actual total contribution.

Method	# Params	CoLA	SST-2	MRPC	QQP	STS-B	MNLI	QNLI	RTE	Avg.
Method	# Params	(mcc)	(acc)	(acc-f1)	(acc-f1)	(corr)	(acc)	(acc)	(acc)	Avg.
Full FT	355M	69.19	95.63	89.46	91.10	91.60	90.01	94.03	86.94	88.50
Housbly-Adapter	0.35M	67.80	94.38	89.75	89.41	91.08	90.28	93.52	84.36	87.57
P-tuning v2	0.31M	67.35	93.13	88.49	88.63	90.41	89.19	91.94	82.42	86.45
(IA) 3	0.33M	68.62	93.82	89.54	89.78	90.84	89.87	92.60	82.75	87.23
SSP	0.36M	69.89	94.96	90.08	90.14	91.37	90.42	94.16	84.88	88.24

Table 1: Performance comparison of fine-tuning methods on the GLUE task using RoBERTa-large. All results are reported as the median over 5 runs with different random seeds. Bold and Underline indicate the best and the second-best results. The metric for each task is explained in Appendix I.5 .

Method	# Params	BoolQ	ARC-e	ARC-c	GSM8K	AQuA	Avg.
Method	# Params	(acc)	(acc)	(acc)	(acc)	(acc)	Avg.
Full FT	494.0M	81.74	74.82	54.98	34.64	48.72	58.98
Housbly-Adapter	9.0M	78.36	71.04	53.26	28.67	42.85	54.84
LoRA	8.8M	78.94	72.78	54.38	31.42	45.33	56.57
AdaLoRA	8.9M	80.32	73.90	54.23	33.27	46.58	57.67
GoRA	8.8M	79.24	71.20	51.91	32.07	45.81	56.04

Table 2: Performance comparison of fine-tuning methods on the Mathematical and common-sense reasoning task using the Qwen-2.5-0.5B. All results are reported as the median over

5

runs with different random seeds. Bold and Underline indicate the best and the second-best results.

Table 5

Panel of Figure 3 — Figure 3: The impact of different hyperparameters $M,N,\beta_{1},\beta_{2}$ on performance when fine-tuning the Qwen2.5-0.5B model on the Boolq dataset. Please zoom in 300% for better clarity.

Panel of Figure 4 — Figure 4: The impact of different hyperparameters $M,N,\beta_{1},\beta_{2}$ on performance when fine-tuning the Qwen2.5-0.5B model on the GSM8K dataset. Please zoom in 300% for better clarity.

Panel of Figure 5 — Figure 5: Rank allocation by IGU-LoRA on the Qwen-2.5-0.5B backbone after fine-tuning for the BoolQ and GSM8K tasks. Please zoom in 300% for better clarity.

Panel of Figure 6 — Figure 6: Performances across different initial rank budgets. The $x$ -axis denotes the initial rank $r^{0}$ , while the $y$ -axis indicates the corresponding task performance. Please zoom in 300% for better clarity.

Llama-2-7B
Method	# Params	SST-2	RTE	QNLI	BoolQ	COPA	Avg.
Full FT	6738M	95.83	92.11	92.54	87.30	93.01	92.16
Adapter	21.2M	94.15	82.12	93.10	87.03	91.10	89.50
P-tuning v2	20.9M	93.42	79.62	92.64	84.73	90.30	88.14
SSP	40.0M	94.14	83.11	93.10	87.11	91.65	89.82
LoRA	20.0M	94.12	83.37	93.10	87.34	91.33	89.85

Table 5: PEFT methods comparison on different backbones. Left: GLUE accuracy (%) with Llama-2-7B. Right: BoolQ and GSM8K accuracy (%) with Llama-3-8B and DeepSeek-R1-Distill-Qwen-2.5-7B. Results are reported as the median over 5 random seeds. Bold and underline indicate the best and the second-best results.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	learning rate	batch size	Max. Sequence Length	# epochs	$\gamma$	$t_{i}$	$\Delta_{T}$	$t_{f}$
MNLI	$5\times 10^{-4}$	$128$	$512$	$25$	$0.1$	$500$	$20$	$10000$
RTE	$1\times 10^{-3}$	$32$	$512$	$25$	$0.1$	$300$	$5$	$2500$
QNLI	$5\times 10^{-4}$	$128$	$512$	$25$	$0.1$	$400$	$20$	$10000$
MRPC	$1\times 10^{-3}$	$32$	$512$	$25$	$0.1$	$300$	$5$	$2500$
QQP	$5\times 10^{-4}$	$128$	$512$	$25$	$0.1$	$500$	$20$	$10000$
SST-2	$1\times 10^{-3}$	$64$	$512$	$25$	$0.1$	$400$	$20$	$5000$

Appendix

Table 6: Hyperparameter setup of IGU-LoRA for training on different datasets.

Hyperparameter	$\epsilon$	$M$	$N$	$\beta_{1}$	$\beta_{2}$
Value	$1\times 10^{-6}$	$16$	$20$	$0.85$	$0.85$

Appendix

Table 7: Setting of the 5 hyperparameters (

\epsilon,M,N,\beta_{1},\beta_{2}

) in IGU-LoRA.

	Module Removed	Rank	Boolq
1	L3_K	10	81.15 (-1.30)
2	L10_V	10	81.12 (-1.33)
3	L3_K / L10_V	10 / 10	80.44 (-2.01)
4	L1_K	5	82.40 (-0.05)
5	L3_V	5	82.35 (-0.10)
6	L1_K / L3_V	5 / 5	82.30 (-0.15)

Appendix

Table 8: Ablation study on the impact of removing high-rank and low-rank modules from different layers on Qwen2.5-0.5B model performance. The numbers in parentheses indicate the performance drop compared to the model with no modules removed. The left table and the right table represent results on Boolq and GSM8K, respectively.

Method	Humanities	STEM	Social.	Other	Avg.
Full FT	49.91	41.70	57.53	57.02	51.54
LoRA	46.15	40.84	56.63	56.23	49.94
IGU-LoRA	47.33	41.71	58.12	57.10	51.07

Appendix

Table 9: The generalization performance of fine-tuning the Llama2-7B model on the MMLU benchmark using different methods, reporting the average results over 5 random seeds.

Method	VQAv2	GAQ	${\textbf{NVLR}^{2}}$	COCO Captioning	Avg.
Full FT	66.91	56.72	73.71	112.04	77.35
LoRA	64.32	54.10	71.25	109.56	74.31
DoRA	65.81	54.71	73.14	115.93	77.40
IGU-LoRA	65.78	55.32	73.42	115.36	77.47

Appendix

Table 10: Performance comparison of different fine-tuning methods on the VQA, GAQ,

\text{NVLR}^{2}

and COCO datasets using the VL-BART model. The results are averaged over 5 random seeds.

Figure 7: Empirical analysis of importance score statistics during fine-tuning. The plots show the changes in

y_{t}

, the mean and variance of

y_{t}

, the first-order lag autocorrelation, and

\mathsf{SNR}_{t}

across training iterations for representative module parameters.

Batch Size	Seed 1	Seed 2	Seed 3	Seed 4	Seed 5
2	82.46	82.47	82.45	82.46	82.45
4	82.45	82.46	82.44	82.45	82.44
8	82.44	82.45	82.43	82.44	82.43
16	82.45	82.46	82.44	82.45	82.44
32	82.40	82.41	82.39	82.40	82.39

Appendix

Table 11: Effect of Batch Size on BoolQ Accuracy across Different Random Seeds

Common-Sense reasoning tasks
Datasets	# train	# dev	# test	Type	Metrics
BoolQ	9427	-	3270	Common-Sense reasoning	Acc
ARC-e	2251	570	2376	Common-Sense reasoning	Acc
ARC-c	1119	299	1172	Common-Sense reasoning	Acc
COPA	400	100	500	Common-Sense reasoning	Acc
Mathematical reasoning tasks

Appendix

Table 12: Summary of the benchmark datasets.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 738 papers

IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning
Jul 24, 2026Wei Zhang, Xinwu Liu, Yihang ChengLLM Fine-Tuning Low-Rank Adaptation
Faculty of Artificial Intelligence in Education Laboratory for Artificial Intelligence and New Forms of Education Central China Normal University Wuhan, Hubei, China
Parameter-Efficient Fine-Tuning with Learnable Rank
Jun 3, 2026Arpit Garg, Simon Lucey, Hemanth SaratchandranNonlinear LoRA Low-Rank Adaptation
Australian Institute for Machine Learning Adelaide University
FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation
May 16, 2026Ramakrishnan SathyavageeswaranAdapter Tuning Low-Rank Adaptation
Intuit

Jul 24, 2026J/K move · Enter open · S save

IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning

Wei Zhang, Xinwu Liu, Yihang Cheng

Faculty of Artificial Intelligence in Education Laboratory for Artificial Intelligence and New Forms of Education Central China Normal University Wuhan, Hubei, China

LLM Fine-Tuning Low-Rank Adaptation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT) of LLMs whose effectiveness depends on rank allocation. Existing adaptive LoRA methods derive ranks from local gradient, activation, or matrix statistics collected before or during fine-tuning; training-time variants add overhead, and local signals reveal little about each module's structural role in information propagation, giving weak global grounding for scarce-capacity allocation. We propose IFCLoRA, a topology-aware method for pre-fine-tuning rank allocation and adapter initialization. Using a small calibration set, IFCLoRA performs intervention tracing on the frozen model and constructs a sparse task-conditioned interaction graph over LoRA target modules. From this graph it extracts a global information-flow topology prior and fuses it with each node's local gradient sensitivity to form a topology-dominant Information-Flow Centrality (IFC) score, measuring participation in task-conditioned multi-hop propagation. The IFC scores then serve as module-level routing signals for one-shot discrete rank allocation under a rank-budget constraint. Reusing response vectors from tracing, IFCLoRA constructs a function-preserving flow-response subspace initialization, giving adapters task-relevant output subspaces. Across all settings, IFCLoRA achieves higher mean scores than standard LoRA with comparable fine-tuning time and peak memory; it requires a one-time offline calibration stage. On GSM8K, IFCLoRA attains the highest mean accuracy among compared PEFT methods on both base models, exceeding standard LoRA by 4.75 percentage points on LLaMA-3.1-8B. Resulting rank allocations are non-uniform and vary across tasks and base models, suggesting that task-conditioned global information-flow topology can serve as a useful structural prior for rank allocation in low-budget PEFT.

Abstract

LLM Fine-Tuning

Explore similar work

IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning

Parameter-Efficient Fine-Tuning with Learnable Rank

FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation