Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptation

Xi Xiao; Chenrui Ma; Yunbei Zhang; Chen Liu; Zhuxuanzi Wang; Yanshu Li; Lin Zhao; Guosheng Hu

cs.CVMar 15, 2026

Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptation

Authors: Xi Xiao, Chenrui Ma, Yunbei Zhang, Chen Liu, Zhuxuanzi Wang, Yanshu Li, Lin Zhao, Guosheng Hu,

Organizations: University of Alabama at Birmingham · University of Virginia · Tulane University · Yale University ·

Abstract

Low-Rank Adaptation (LoRA) has become a cornerstone of parameter-efficient fine-tuning (PEFT). Yet, its efficacy is hampered by two fundamental limitations: semantic drift, by treating all update directions with equal importance, and structural incoherence, from adapting layers independently, resulting in suboptimal, uncoordinated updates. To remedy these, we propose StructLoRA, a framework that addresses both limitations through a principled, dual-component design: (1) an Information Bottleneck-guided filter that prunes task-irrelevant directions to mitigate semantic drift, and (2) a lightweight, training-only graph-based coordinator that enforces inter-layer consistency to resolve structural incoherence. Extensive experiments across large language model , vision language model, and vision model (including LLaMA, LLaVA, and ViT) demonstrate that StructLoRA consistently establishes a new state-of-the-art, outperforming not only vanilla LoRA but also advanced dynamic rank allocation and sparsity-based methods. Notably, the benefits are particularly pronounced in challenging low-rank and low-data regimes. Crucially, since our proposed modules operate only during training, StructLoRA enhances performance with zero additional inference cost, advancing the focus of PEFT -- from mere parameter compression to a more holistic optimization of information quality and structural integrity.

Figures & tables

Figure 1: Architectural comparison between LoRA and StructLoRA . The left illustrates the standard LoRA architecture with uniform low-rank updates, while the right shows our StructLoRA , which introduces an Information Bottleneck (IB) filter and a Graph-based Coordination mechanism . These modules selectively retain task-relevant update directions and align layer-wise updates through message passing. Both operate only during training and are removed at inference, preserving LoRA’s zero-latency efficiency.

Method	Type	Reasoning		Image Classification		Captioning	Visual QA
Method	Type	BoolQ $\uparrow$	PIQA $\uparrow$	CIFAR-100 $\uparrow$	ImageNet-1k $\uparrow$	COCO Caption $\uparrow$	VQAv2 $\uparrow$
Full Fine-tuning	—	82.6	85.3	85.9	78.8	123.5	76.2
Linear Probing	—	71.1	74.8	65.7	68.4	105.3	67.0
Adapter Houlsby et al. (2019)	Additive	79.2	82.5	81.3	75.7	116.5	73.0
Prefix-Tuning Li and Liang (2021)	Additive	79.6	82.8	81.7	76.1	117.0	73.3
LoRA Hu et al. (2022a)	Reparam.	79.1	82.4	81.5	76.2	116.2	73.5

Table 1: Main comparison of StructLoRA with baseline PEFT methods. We report primary metrics (Accuracy % for classification, reasoning and question answering; CIDEr for captioning) under a comparable budget of trainable parameters (

\sim

0.5–1%). See Appendix B for additional results.

Method	MNLI $\uparrow$	SST-2 $\uparrow$	MRPC $\uparrow$	CoLA $\uparrow$	QNLI $\uparrow$	QQP $\uparrow$	RTE $\uparrow$	STS-B $\uparrow$	Average $\uparrow$
LoRA	87.3	93.5	87.1	58.8	93.0	90.5	79.4	91.0	85.1
AdaLoRA	87.3	93.6	87.3	59.0	93.1	90.6	79.6	91.2	85.2
DyLoRA	87.2	93.7	87.3	59.0	93.0	90.6	79.6	91.2	85.2
Sensitivity-LoRA	87.6	94.6	87.7	60.2	93.6	90.7	81.8	91.3	86.0
StructLoRA (ours)	88.1	95.0	88.5	61.5	94.1	91.0	82.3	91.5	86.5

Table 2: Head-to-head comparison on the GLUE benchmark using RoBERTa-base. All baseline settings follow Zhang et al. (2025a) . StructLoRA consistently surpasses all dynamic rank allocation methods, demonstrating superior performance on standard natural language understanding (NLU) tasks.

Rank ( $r$ )	Trainable Params (%)	BoolQ $\uparrow$ (LLaMA-7B)		CIFAR-100 $\uparrow$ (ViT-B/16)		COCO Caption $\uparrow$ (LLaVA-1.5-7B)
Rank ( $r$ )	Trainable Params (%)	LoRA	StructLoRA	LoRA	StructLoRA	LoRA	StructLoRA
2	0.12	75.1	77.4 (+2.3)	78.3	80.1 (+1.8)	111.2	114.3 (+3.1)
4	0.24	77.6	79.9 (+2.3)	79.7	82.2 (+2.5)	113.8	117.0 (+3.2)
8	0.48	79.1	81.3 (+2.2)	81.5	84.1 (+2.6)	116.2	122.4 (+6.2)
16	0.95	80.3	81.7 (+1.4)	82.8	84.3 (+1.5)	118.1	123.6 (+5.5)
32	1.90	81.0	81.9 (+0.9)	83.4	84.5 (+1.1)	119.0	123.9 (+4.9)

Table 3: Performance under varying rank budgets. Accuracy (for BoolQ, CIFAR-100) and CIDEr (for COCO Caption) are reported. StructLoRA consistently outperforms LoRA, with the largest gains observed in low-rank (

r\leq 8

) settings, demonstrating superior parameter efficiency.

Dataset	Metric	Method	10% Data	25% Data	50% Data	100% Data
BoolQ (LLaMA-7B)	Accuracy $\uparrow$	LoRA	68.5	73.2	76.4	79.1
BoolQ (LLaMA-7B)	Accuracy $\uparrow$	StructLoRA	71.2 (+2.7)	76.3 (+3.1)	78.9 (+2.5)	81.3 (+2.2)
CIFAR-100 (ViT-B/16)	Accuracy $\uparrow$	LoRA	73.6	78.0	80.5	81.5
CIFAR-100 (ViT-B/16)	Accuracy $\uparrow$	StructLoRA	76.3 (+2.7)	80.5 (+2.5)	82.4 (+1.9)	84.1 (+2.6)
COCO Caption (LLaVA-1.5-7B)	CIDEr $\uparrow$	LoRA	100.2	108.3	114.0	116.2
COCO Caption (LLaVA-1.5-7B)	CIDEr $\uparrow$	StructLoRA	103.7 (+3.5)	112.4 (+4.1)	117.9 (+3.9)	122.4 (+6.2)

Table 4: Performance under limited supervision (few-shot learning). We evaluate with a fixed rank

r=8

while varying data availability. StructLoRA ’s advantage over LoRA grows as the amount of training data decreases, highlighting its superior data efficiency and robustness to overfitting.

Ablation Setting	BoolQ (LLaMA-7B)		CIFAR-100 (ViT-B/16)		COCO Caption (LLaVA-1.5-7B)
Ablation Setting	Accuracy (%) $\uparrow$	$\Delta$	Accuracy (%) $\uparrow$	$\Delta$	CIDEr $\uparrow$	$\Delta$
StructLoRA (Full)	81.3	- 0.0	84.1	- 0.0	122.4	- 0.0
w/o IB Filter	79.4	-1.9	81.9	-2.2	117.8	-4.6
w/o GNN Coordination	80.1	-1.2	82.6	-1.5	119.4	-3.0
w/o Both (i.e., Standard LoRA)	79.1	-2.2	81.5	-2.6	116.2	-6.2

Table 5: Module-wise ablation of StructLoRA . Removing either information bottleneck (IB) filter or the graph-based coordination network degrades performance, while removing both collapses StructLoRA to LoRA, demonstrating their synergistic contribution. See further analysis in Appendix C .

GNN Configuration	Accuracy (%) $\uparrow$
StructLoRA	81.3
Effect of Depth:
2-Layer GNN	80.4
3-Layer GNN	79.7
Effect of Graph Construction:
Adjacency Only	80.5

Table 6: Ablation on GNN design choices on the BoolQ dataset. The default StructLoRA employs a 1-layer GNN with a hybrid graph combining adjacency and similarity edges, achieving the highest accuracy.

Figure 2: Analysis of filtering strategies. We compare our IB-guided filter with two heuristics under the same keep ratio: Random Masking and Top- $k$ Norm (scored by $\|a_{j}\|_{2}\|b_{j}\|_{2}$ for each rank-one direction). Bars show mean performance across three random seeds.

Method	Training Time	Peak Memory
Method	/ Epoch	(GB)
LoRA	1.00 $\times$	16.8
StructLoRA	1.06 $\times$	17.5

Table 7: Training overhead analysis on LLaMA-7B with rank

r=8

. StructLoRA introduces minimal overhead compared to standard LoRA.

Figure 3: Visual attention comparison between LoRA and StructLoRA . The top row shows Grad-CAM Selvaraju et al. (2017) heatmaps from the baseline LoRA model, while the bottom row corresponds to StructLoRA . StructLoRA produces more concentrated and semantically aligned activation regions.

Figure 4: Layer-wise cosine similarity of updates. StructLoRA induces a coherent block-diagonal structure, while LoRA exhibits noisy and fragmented activation patterns. See Table 13 for more results.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method	Magpie-Pro			OpenPlatypus			Avg.
Model	Method	BLEU-4	ROUGE-1	ROUGE-L	BLEU-4	ROUGE-1	ROUGE-L	Avg.
Qwen2.5-7B	HAdapter	54.71	49.11	32.42	19.39	43.95	22.51	37.01
	PAdapter	54.83	49.15	32.24	19.42	44.03	22.54	37.04
	LoRA	55.03	48.82	32.42	19.72	43.83	22.53	37.06
	AdaLoRA	55.66	49.13	32.75	19.87	44.24	22.67	37.39
	DyLoRA	55.59	49.21	32.82	19.86	44.18	22.59	37.37

Appendix

Table 8: Evaluation results on Natural Language Generation (NLG) tasks across diverse, state-of-the-art model architectures. We compare StructLoRA with other PEFT baselines on two representative datasets, Magpie-Pro and OpenPlatypus. StructLoRA consistently establishes a new state-of-the-art, outperforming all other methods-including the strong Sensitivity-LoRA baseline-across all tested models. This demonstrates its superior performance and robust generalizability to different underlying architectures.

Method	Task Score (↑)	$\boldsymbol{\mathcal{E}}\times 10^{-2}$ (↓)	CosAdj (↑)	Extra Cost (train, %)
LoRA+Cos	82.3 / 79.4	3.98 / 4.22	0.46 / 0.42	+1.2
LoRA+Lap	83.0 / 80.1	3.72 / 3.95	0.49 / 0.45	+2.4
StructLoRA (GNN)	84.1 / 81.3	3.05 / 3.41	0.56 / 0.52	+5.3

Appendix

Table 9: Coordination alternatives under equal LoRA budget. Averaged over three runs on ViT-B/16 (CIFAR-100) and LLaMA-7B (BoolQ).

\mathcal{E}

and CosAdj follow the definitions above.

Task	Model	Context Length	Method	Performance	Metric	Trainable Params (%)	Peak Memory (GB)
BoolQ Reasoning	LLaMA-13B	512 tokens	LoRA	80.3	Accuracy	0.45	18.1
BoolQ Reasoning	LLaMA-13B	512 tokens	StructLoRA	81.7	Accuracy	0.45	18.8
COCO Captioning	LLaVA-1.5-7B	1024 tokens	LoRA	110.6	CIDEr	0.48	20.3
COCO Captioning	LLaVA-1.5-7B	1024 tokens	StructLoRA	114.9	CIDEr	0.48	20.7

Appendix

Table 10: StructLoRA ’s scalability with larger models and longer sequences. We evaluate on LLaMA-13B for reasoning and LLaVA-1.5-7B for long-context captioning. StructLoRA consistently outperforms LoRA while maintaining resource efficiency.

Task Domain	Model	Base PEFT Method	Enhancement Method	Metric	Base Performance	Enhanced Performance
Language	LLaMA-7B	QLoRA Dettmers et al. (2023)	StructLoRA	Accuracy (%)	80.0	81.6 (+1.6)
Language	LLaMA2-7B	VeRA Kopiczko et al. (2024)	StructLoRA	Accuracy (%)	82.6	83.9 (+1.3)
Vision	ViT-B/16	LoRA-FA Drost (2024)	StructLoRA	Accuracy (%)	81.8	83.6 (+1.8)
Multimodal	LLaVA-1.5-7B	AdapterFusion Pfeiffer et al. (2021)	StructLoRA	CIDEr	118.3	121.0 (+2.7)

Appendix

Table 11: Compatibility of StructLoRA with existing PEFT paradigms. StructLoRA is applied as an enhancement layer on top of strong PEFT baselines. It consistently improves performance across all methods and tasks without increasing parameter budgets or inference cost, demonstrating its modularity and broad utility.

Model	Method	Trainable Params (M)	Ratio (%)	Quantization	Activation Precision	Notes / Equivalent Setting
LLaMA-7B	LoRA	67.2	0.96	None	FP16	Rank $r=8$ , $\alpha=16$
	QLoRA	66.9	0.95	4-bit NF4	FP16	Same rank, quantized base weights
	DoRA	67.2	0.96	None	FP16	Direction-only adaptation
	AdaLoRA	67.4	0.97	None	FP16	Dynamic rank allocation (avg $r{=}8$ )
	Adapter	68.0	0.98	None	FP16	Bottleneck dim 32 (matches LoRA params)
	Prefix-Tuning	67.6	0.97	None	FP16	Prefix length 20 (equivalent budget)

Appendix

Table 12: Trainable parameter budget and configuration fairness across PEFT methods. All methods are aligned to an equivalent trainable-parameter ratio (0.5–1.0% of total parameters). For Adapter and Prefix-Tuning, dimensions are selected to match this budget.

Figure 5: Accuracy vs. Sequence. StructLoRA consistently outperforms LoRA across longer input sequences, showing stronger robustness under extended context lengths.

Figure 6: SVD Coverage vs. Rank. StructLoRA achieves broader singular-value coverage, particularly at low ranks, indicating improved compression and subspace expressiveness compared to LoRA.

Model / Layer Pair	LoRA	StructLoRA	Change ( $\uparrow$ )
Layers 1–2	0.31	0.57	+0.26
Layers 2–3	0.27	0.55	+0.28
Layers 3–4	0.34	0.63	+0.29
Layers 4–5	0.38	0.66	+0.28
Layers 5–6	0.41	0.69	+0.28
Layers 6–7	0.36	0.64	+0.28

Appendix

Table 13: Cosine similarity of update gradients between adjacent layers. Values are averaged across training iterations on SST-2 (BERT-base). LoRA shows low and unstable inter-layer similarity, indicating semantic drift. StructLoRA increases within-block coherence while preserving inter-block diversity.

Panel of Figure 7 — Figure 7: Instruction-tuning evaluation on MT-Bench. StructLoRA consistently yields higher GPT-4 evaluation scores than LoRA, generating more structured, informative, and contextually aligned responses across diverse question types.

Low-Rank Adaptation

Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.

See all 542 papers

Low-Rank Adaptation Redux for Large Models
Apr 23, 2026Bingcong Li, Yilang Zhang, Georgios B. GiannakisAdapter Tuning Low-Rank Adaptation
Dept. of CS, ETH Zürich, Switzerland · Dept. of ECE, U. of Minnesota, USA · Morgan Stanley, USA
TLoRA: Task-aware Low Rank Adaptation of Large Language Models
Apr 20, 2026Weicheng Lin, Yi Zhang, Jiawei Dang +1LLM Fine-Tuning Low-Rank Adaptation
College of Computer Science and Software Engineering, Shenzhen University, China
Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence
May 29, 2026Valérie Castin, Kimia Nadjahi, Pierre Ablin +1LLM Fine-Tuning Low-Rank Adaptation
1 ´Ecole Normale Sup´erieure PSL, Paris, France · 2CNRS · 3Apple, Paris, France.

Apr 23, 2026J/K move · Enter open · S save

Low-Rank Adaptation Redux for Large Models

Bingcong Li, Yilang Zhang, Georgios B. Giannakis

Dept. of CS, ETH Zürich, Switzerland · Dept. of ECE, U. of Minnesota, USA · Morgan Stanley, USA

Adapter Tuning Low-Rank Adaptation Low-Rank Approximation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Low-rank adaptation (LoRA) has emerged as the de facto standard for parameter-efficient fine-tuning (PEFT) of foundation models, enabling the adaptation of billion-parameter networks with minimal computational and memory overhead. Despite its empirical success and rapid proliferation of variants, it remains elusive which architectural choices, optimization techniques, and deployment constraints should guide practical method selection. This overview revisits LoRA through the lens of signal processing (SP), bridging modern adapter designs with classical low-rank modeling tools and inverse problems, as well as highlighting how SP principles can inform principled advances of fine-tuning approaches. Rather than providing a comprehensive enumeration and empirical comparisons of LoRA variants, emphasis is placed on the technical mechanisms underpinning these approaches to justify their effectiveness. These advances are categorized into three complementary axes: architectural design, efficient optimization, and pertinent applications. The first axis builds on singular value decomposition (SVD)-based factorization, rank-augmentation constructions, and cross-layer tensorization, while the second axis deals with initialization, alternating solvers, gauge-invariant optimization, and parameterization-aware methods. Beyond fine-tuning, emerging applications of LoRA are accounted across the entire lifecycle of large models, ranging from pre- and post-training to serving/deployment. Finally, open research directions are outlined at the confluence of SP and deep learning to catalyze a bidirectional frontier: classical SP tools provide a principled vocabulary for designing principled PEFT methods, while the unique challenges facing modern deep learning, especially the overwhelming scale and prohibitive overhead, also offer new research lines benefiting the SP community in return.

Abstract

Low-Rank Adaptation

Explore similar work

Low-Rank Adaptation Redux for Large Models

TLoRA: Task-aware Low Rank Adaptation of Large Language Models

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence