Residual-Guided Adaptive Tucker Rank Allocation for Parameter-Efficient MCI Conversion Prediction

Guanghua He; Hancan Zhu; Gaohang Yu; An Zhang

cs.LGOct 19, 2026

Residual-Guided Adaptive Tucker Rank Allocation for Parameter-Efficient MCI Conversion Prediction

Authors: Guanghua He, Hancan Zhu, Gaohang Yu, An Zhang

Organizations: Department of Mathematics, Hangzhou Dianzi University, Hangzhou, 310018, China · School of Mathematics, Physics and Information, Shaoxing University, Shaoxing, 312000, China · Department of Mathematics, Zhejiang University of Science and Technology, Hangzhou, 310023, China

Abstract

Allocating limited trainable capacity across convolutional layers is a central challenge in parameter-efficient fine-tuning (PEFT) of pretrained three-dimensional (3D) networks. We propose Residual-Guided Adaptive Tucker Rank Allocation, which jointly selects input- and output-channel ranks under a fixed adapter-parameter budget while preserving the spatial kernel modes. To guide rank selection, we learn task-specific proxy updates from downstream training data and approximate them using candidate Tucker rank pairs. A perturbation analysis bounds the output discrepancy between the fixed proxy network and its Tucker approximation in terms of layerwise reconstruction residuals, motivating residual-based allocation. We normalize each squared residual by the corresponding proxy-update energy to compare relative approximation errors across layers. Dynamic programming minimizes their sum under a global budget constraint that accounts for both channel factors and the Tucker core. This yields a globally optimal allocation over the specified candidate rank pairs. We evaluate the method on the Alzheimer's Disease Neuroimaging Initiative (ADNI) cohort for predicting conversion from mild cognitive impairment to Alzheimer's disease within 36 months, using hippocampal segmentation pretraining. Across five random seeds, the proposed method achieves a mean area under the receiver operating characteristic curve (AUC) of 0.7706, compared with 0.7239 for fixed-ratio Tucker adaptation and 0.7430 for full fine-tuning. The total number of trainable parameters is limited to approximately 0.618 million, accounting for only 1.45% of the full model. Adaptive Tucker rank allocation thus provides an effective approach to parameter-efficient transfer learning in 3D medical imaging.

Figures & tables

Symbol	Meaning
$\mathcal{W}_{\ell},\Delta\mathcal{W}_{\ell}$	Pretrained kernel and trainable adapter update at layer $\ell$
$\Delta\mathcal{W}_{\ell}^{\mathrm{proxy}}$	Calibration-induced kernel update, held fixed during rank allocation
$\bm{r}_{\ell},\bm{r}_{\ell,q}$	Channel-rank pair at layer $\ell$ and its $q$ th candidate pair
$\bm{r},\mathcal{R}$	Network-wide rank configuration and Cartesian product of layerwise candidate sets
$\mathcal{H}_{\bm{s}}(\mathcal{X})$	Channel-mode HOSVD approximation of $\mathcal{X}$ at rank pair $\bm{s}$
$\mathcal{E}_{\ell}(\bm{r}_{\ell}),\mathcal{E}_{\ell,q}$	Proxy-update reconstruction residual tensor and its candidate-indexed form

Table 1: Main notation for residual-guided Tucker allocation.

Figure 1: Overview of residual-guided Tucker. Task-specific proxy updates provide normalized reconstruction residuals for joint input- and output-channel rank allocation under a fixed adapter-parameter budget. Final PEFT training starts from the original pretrained checkpoint with adapters initialized to produce zero updates at the selected ranks.

Characteristic	sMCI $n=305$	pMCI $n=264$	Total $n=569$
Age (years), mean $\pm$ SD a	$71.69\pm 7.63$	$73.93\pm 7.07$	$72.73\pm 7.45$
Female, $n$ (%)	120 (39.34)	105 (39.77)	225 (39.54)
Male, $n$ (%)	185 (60.66)	159 (60.23)	344 (60.46)

Table 2: Baseline characteristics of the ADNI cohort.

Method	Trainable (M)	AUC	ACC	F1	SEN	SPE
Full fine-tuning	$42.691$	$0.7430$ [-1pt] $(0.0099)$	$0.6684$ [-1pt] $(0.0218)$	$0.6100$ [-1pt] $(0.0229)$	$0.5585$ [-1pt] $(0.0413)$	$\mathbf{0.7639}$ [-1pt] $(0.0539)$
LoRA	$0.646$	$0.7301$ [-1pt] $(0.0243)$	$0.6579$ [-1pt] $(0.0206)$	$0.6194$ [-1pt] $(0.0403)$	$0.6038$ [-1pt] $(0.0789)$	$0.7049$ [-1pt] $(0.0544)$
AdaLoRA	$0.618$	$0.7218$ [-1pt] $(0.0102)$	$0.6632$ [-1pt] $(0.0338)$	$0.6368$ [-1pt] $(0.0457)$	$0.6377$ [-1pt] $(0.0631)$	$0.6852$ [-1pt] $(0.0356)$
PiSSA	$0.646$	$0.7218$ [-1pt] $(0.0186)$	$0.6579$ [-1pt] $(0.0317)$	$0.6022$ [-1pt] $(0.0309)$	$0.5585$ [-1pt] $(0.0605)$	$0.7443$ [-1pt] $(0.0849)$
PoLAR	$0.647$	$0.7203$ [-1pt] $(0.0156)$	$0.6579$ [-1pt] $(0.0240)$	$0.6001$ [-1pt] $(0.0570)$	$0.5623$ [-1pt] $(0.1135)$	$0.7410$ [-1pt] $(0.0700)$
Fixed-ratio Tucker	$0.618$	$0.7239$ [-1pt] $(0.0093)$	$0.6561$ [-1pt] $(0.0209)$	$0.6046$ [-1pt] $(0.0542)$	$0.5736$ [-1pt] $(0.0958)$	$0.7279$ [-1pt] $(0.0526)$

Table 3: Predictive performance of the compared methods. Performance values are means across five seeds, with sample SDs in parentheses below. Bold denotes the highest mean in each performance column. For AdaLoRA, the parameter count refers to the final compact allocation.

Figure 2: Mean test AUC versus trainable parameter count for the compared methods. The horizontal axis is broken to separate the PEFT methods from full fine-tuning; the two segments use different scales. M denotes millions of parameters. For AdaLoRA, the parameter count refers to the final compact allocation.

Configuration	Trainable (M)	AUC	ACC	F1	SEN	SPE
Head-only	$0.066$	$0.6820$ [-1pt] $(0.0107)$	$0.6351$ [-1pt] $(0.0229)$	$0.5520$ [-1pt] $(0.0128)$	$0.4830$ [-1pt] $(0.0103)$	$0.7672$ [-1pt] $(0.0484)$
Common only	$0.250$	$0.7250$ [-1pt] $(0.0133)$	$0.6579$ [-1pt] $(0.0164)$	$0.5903$ [-1pt] $(0.0290)$	$0.5321$ [-1pt] $(0.0488)$	$0.7672$ [-1pt] $(0.0425)$
Fixed-ratio Tucker + common	$0.618$	$0.7239$ [-1pt] $(0.0093)$	$0.6561$ [-1pt] $(0.0209)$	$0.6046$ [-1pt] $(0.0542)$	$0.5736$ [-1pt] $(0.0958)$	$0.7279$ [-1pt] $(0.0526)$
Residual-guided Tucker + common	$0.618$	$0.7706$ [-1pt] $(0.0381)$	$0.7035$ [-1pt] $(0.0474)$	$0.6763$ [-1pt] $(0.0694)$	$0.6755$ [-1pt] $(0.1079)$	$0.7279$ [-1pt] $(0.0587)$

Table 4: Contribution of the trainable components. Values are means across five seeds, with sample SDs in parentheses below. Common denotes the classification head, channel-changing shortcuts, and BatchNorm affine parameters.

Allocation score	AUC	ACC	F1	SEN	SPE
Absolute residual	$0.7524$ [-1pt] $(0.0253)$	$0.6895$ [-1pt] $(0.0395)$	$0.6480$ [-1pt] $(0.0647)$	$0.6226$ [-1pt] $(0.0962)$	$\mathbf{0.7475}$ [-1pt] $(0.0220)$
Normalized residual (proposed)	$\mathbf{0.7706}$ [-1pt] $(0.0381)$	$\mathbf{0.7035}$ [-1pt] $(0.0474)$	$\mathbf{0.6763}$ [-1pt] $(0.0694)$	$\mathbf{0.6755}$ [-1pt] $(0.1079)$	$0.7279$ [-1pt] $(0.0587)$

Table 5: Effect of residual normalization under a common adapter-budget ceiling. Values are means across five seeds, with sample SDs in parentheses below. Bold denotes the higher mean in each column.

Figure 3: Layerwise channel ranks and adapter parameter counts for residual-guided Tucker and fixed-ratio Tucker (seed 42).

Method	Adapter (M)	Trainable (M)	Trainable (%)	FLOPs (G)	Peak memory (GiB)	Test time (s)
Full fine-tuning	–	$42.691$	$100.00$	$313.801$	$2.703$	$4.955$
LoRA	$0.396$	$0.646$	$1.51$	$314.141$	$2.233$	$4.943$
AdaLoRA	$0.368$	$0.618$	$1.45$	$314.095$	$2.258$	$4.948$
PiSSA	$0.396$	$0.646$	$1.51$	$314.141$	$2.233$	$4.998$
PoLAR	$0.397$	$0.647$	$1.51$	$314.141$	$2.233$	$4.994$
Fixed-ratio Tucker	$0.368$	$0.618$	$1.45$	$316.543$	$2.235$	$5.134$

Table 6: Parameter and computational efficiency. FLOPs are measured for a cold forward pass, and memory denotes peak allocated memory during final training. Trainable parameter percentages are relative to full fine-tuning. Test-set processing time includes data loading and transfer. AdaLoRA is evaluated at its final compact allocation. Measurements exclude calibration and initialization.

Adaptive Rank Allocation

Adaptive rank allocation distributes low-rank model capacity across layers, modules, tasks, or inputs according to their estimated needs.

See all 30 papers

May 16, 2026

GLT-PEFT: Gated Lie-Tucker Parameter-Efficient Fine-Tuning for Alzheimer's Disease Diagnosis with Hippocampal Segmentation Pretraining

Parameter-efficient fine-tuning (PEFT) has emerged as a promising paradigm for adapting pretrained models under limited data conditions. However, most existing PEFT methods are designed for matrix-structured parameters and are not well suited for high-dimensional convolutional kernels in medical imaging models. Moreover, they typically rely on additive updates and lack mechanisms to preserve the geometric structure of pretrained parameters, while multiplicative (geometry-aware) updates are difficult to integrate within a unified framework. To address this issue, this paper proposes GLT-PEFT, a gated Lie-Tucker parameter-efficient fine-tuning framework for Alzheimer's disease (AD) diagnosis. The proposed approach transfers a hippocampal segmentation pretrained model to a downstream classification task. Tucker decomposition enables tensor-aware low-rank adaptation of 3D convolutional kernels, while Lie group-based transformations provide structure-preserving multiplicative updates. A gating mechanism further reconciles additive and multiplicative update forms, resulting in a unified and more stable fine-tuning strategy. Extensive experiments demonstrate that GLT-PEFT achieves effective cross-task transfer while significantly reducing trainable parameters, highlighting its effectiveness for efficient and robust adaptation in medical imaging models.

Guanghua He, Hancan Zhu, Gaohang Yu +1

Department of Mathematics, Hangzhou Dianzi University, Hangzhou, 310018, China · School of Mathematics, Physics and Information, Shaoxing2026 University, Shaoxing, 312000, China · Department of Mathematics, Zhejiang University of Science and Technology, Hangzhou, 310023, China

Medical Diagnosis Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Apr 30, 2026

Post-Optimization Adaptive Rank Allocation for LoRA

Exponential growth in the scale of modern foundation models has led to the widespread adoption of Low-Rank Adaptation (LoRA) as a parameter-efficient fine-tuning technique. However, standard LoRA implementations disregard the varying intrinsic dimensionality of model layers and enforce a uniform rank, leading to parameter redundancy. We propose Post-Optimization Adaptive Rank Allocation (PARA), a data-free compression method for LoRA that integrates seamlessly into existing fine-tuning pipelines. PARA leverages Singular Value Decomposition to prune LoRA ranks using a global threshold over singular values across all layers. This results in non-uniform rank allocation based on layer-wise spectral importance. As a post-hoc method, PARA circumvents the training modifications and resulting instabilities that dynamic architectures typically incur. We empirically demonstrate that PARA reduces parameter count by 75-90% while preserving the predictive performance of the original, uncompressed LoRA across multiple vision and language benchmarks. Code will be published upon acceptance.

Vishnuprasadh Kumaravelu, Sunil Gupta, P. K. Srijith

1Indian Institute of Technology Hyderabad · 2Deakin University

Low-Rank Adaptation Low-Rank Compression Low-Rank Approximation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning Neural Network Pruning

Ask PDF

Jul 5, 2026

RSLoRA: Training-free Rank Allocation for LoRA via Representational Sensitivity Probing

Low-Rank Adaptation (LoRA) has become a cornerstone of parameter-efficient fine-tuning (PEFT); however, the conventional practice of uniform rank assignment ignores the functional heterogeneity of neural layers. Existing rank allocation methods typically struggle with a trade-off between computational intensity and heuristic simplicity: training-based methods suffer from prohibitive overhead, while pre-allocation methods fail to capture the dynamic task-specific representation manifold. In this paper, we propose RSLoRA (Representational Sensitivity LoRA), a training-free and gradient-free rank allocator driven by activation-space geometry. We identify a "sensitivity regime shift" across layers, observing that static weight analysis and local gradients are insufficient to reflect how updates reshape a model's internal representations. To address this, RSLoRA introduces a virtual representational probing mechanism. By simulating adaptation through structured low-rank noise and measuring the resulting manifold displacement by using Effective Rank and Frechet Distance, we identify high-sensitivity modules that require higher rank capacity. Our framework effectively bridges the gap between expert-crafted heuristics and actual representational impact. Extensive evaluations demonstrate that RSLoRA consistently outperforms state-of-the-art allocators (e.g., AdaLoRA, GoRA) across mainstream benchmarks. By eliminating the need for iterative training-time adjustments and backward gradients, RSLoRA provides a highly efficient, robust, and representation-aware solution for large-scale model adaptation.

Jiaqi Liu, Haidong Kang, Qihui Zhao +1

School of Computer Science and Technology Dalian University of Technology · Hebei Key Laboratory of Marine Perception Network and Data Processing Northeastern University · School of Computer and Communication Engineering Northeastern University +1

Low-Rank Adaptation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning

Ask PDF

Abstract

Adaptive Rank Allocation

Explore similar work

GLT-PEFT: Gated Lie-Tucker Parameter-Efficient Fine-Tuning for Alzheimer's Disease Diagnosis with Hippocampal Segmentation Pretraining

Post-Optimization Adaptive Rank Allocation for LoRA

RSLoRA: Training-free Rank Allocation for LoRA via Representational Sensitivity Probing