Parameter-Efficient Fine-Tuning (PEFT) has become a key strategy for adapting large language models, with recent advances in sparse tuning reducing overhead by selectively updating key parameters or subsets of data. Existing approaches generally focus on two distinct paradigms: layer-selective methods aiming to fine-tune critical layers to minimize computational load, and data-selective methods aiming to select effective training subsets to boost training. However, current methods typically overlook the fact that different data points contribute varying degrees to distinct model layers, and they often discard potentially valuable information from data perceived as of low quality. To address these limitations, we propose Gradient-aligned Sparse Tuning (GAST), an innovative method that simultaneously performs selective fine-tuning at both data and layer dimensions as integral components of a unified optimization strategy. GAST specifically targets redundancy in information by employing a layer-sparse strategy that adaptively selects the most impactful data points for each layer, providing a more comprehensive and sophisticated solution than approaches restricted to a single dimension. Experiments demonstrate that GAST consistently outperforms baseline methods, establishing a promising direction for future research in PEFT strategies.
Figures & tables
Figure 1: Difference among (a) layer-selective methods, (b) data-selective methods, and (c) our method. layer-selective methods generate a subset of all layers to be updated with all mini-batch data. Data-selective methods utilize partial mini-batch data to train all layers. Our GAST selects a different subset of data for each layer.
Figure 2: Overall of our proposed Gradient-aligned Sparse Tuning (GAST). During training, every mini-batch exhibits gradient conflicts both among data samples and across model layers. To mitigate these conflicts, GAST uses the gradient of the support set to decide which individual sample should be used to update each layer. This data-layer selection reduces gradient interference and thereby improves both convergence speed and generalization performance.
Method
Adaptiveness
Average
LoRA
-
74.7 (+0.0)
LISA
Layer
75.3 (+0.6)
AdaLoRA
Rank
76.2 (+1.5)
LoRA + RST
Layer
75.8 (+1.1)
LoRA + IST
Layer
76.5 (+1.8)
LoRA + GREATS
Data
76.3 (+1.6)
Table 1: Comparison with different adaptive methods.
Figure 3: Comparison of model convergence with loss curve.
Model
PEFT
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Avg.
ChatGPT
-
73.1
85.4
68.5
78.5
66.1
89.8
79.9
74.8
77.0
LLaMA 7B
Series
63.0
79.2
76.3
67.9
75.7
74.5
57.1
72.4
70.8
Series + IST
66.2
78.3
74.9
72.2
75.9
75.8
59.0
72.2
71.8
Series + GAST
68.0
79.1
77.8
79.8
78.5
77.4
62.0
74.8
74.7
Parallel
67.9
76.4
78.8
69.8
78.9
73.7
57.3
75.2
72.2
Parallel + IST
68.4
79.1
77.9
70.0
78.9
81.2
62.3
77.6
74.4
Table 2: Comparison across multiple LLMs using different PEFT approaches on eight commonsense reasoning benchmarks. Baseline results for GPT-J and LLaMA are taken from Hu et al. (2023) .
Method
GSM8K
AQuA
MAWPS
SVAMP
Avg.
ChatGPT
56.4
38.9
87.4
69.9
63.2
LoRA
61.0
26.4
91.6
74.4
63.4
LoRA + IST
62.8
31.5
89.9
76.3
64.7
LoRA + GAST
66.4
32.7
91.6
79.4
67.5
Table 3: Comparison with LLaMA3-8B on four math reasoning datasets.
Setting
Data-layer Selection
Average
LoRA
-
74.7
LoRA + GAST
Random Selection
76.4
LoRA + GAST
Top-k Selection
66.4
LoRA + GAST
Sampling-based Selection
77.5
Table 4: The effect of data-layer selection strategy.
Figure 4: Impact of data-layer sparsity in GAST.
Figure 5: (a) Visualization of the sampling probability of mini-batch data points across layers on two different iterations. A deeper red color indicates a higher probability. (b) Distribution of the number of layers each data point is trained on within a mini-batch.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
# Train
# Test
Answer
Commonsense
170K
-
-
BoolQ
9.4K
3,270
Yes/No
PIQA
16.1K
1,830
Option
SIQA
33.4K
1,954
Option
HellaSwag
39.9K
10,042
Option
WinoGrande
63.2K
1,267
Option
Appendix
Table 5: The statistics of datasets for evaluation. # Train and # Test denote the number of training and test samples respectively.
Figure 6: Additional experiment on support set.
Method
Memory Cost
Training Time
LoRA
43.4 GB
10 h
AdaLoRA
46.7 GB
13 h
IST
36.6 GB
10 h
GREATS
58.1 GB
21 h
GAST (memory-efficient)
51.5 GB
19 h
GAST (computation-efficient)
66.9 GB
11.5 h
Appendix
Table 6: Computational cost of GAST on llama-7B with weight memory of 14GB and batch size of 16.
Method
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-E
ARC-C
OBQA
AVG
LoRA
68.9
80.7
77.4
78.1
78.8
77.8
61.3
74.8
74.7
LISA*
–
–
–
–
–
–
–
–
75.3
AdaLoRA*
–
–
–
–
–
–
–
–
76.2
RST
68.3
82.3
78.1
76.6
80.1
79.9
63.3
77.8
75.8
IST
68.7
81.7
77.3
82.7
78.7
80.6
62.4
80.0
76.5
GREATS
69.6
81.1
78.0
81.0
79.8
79.5
63.8
77.6
76.3
Appendix
Table 7: Performance comparison on common-sense reasoning benchmarks. Methods marked with * report results from the original papers.
BoolQ
Please answer the following question with true or false, question: is house tax and property tax are same?
Answer format: true/false
the correct answer is true
PIQA
Please choose the correct solution to the question: Extend life of flowers in vase.
Solution1: Add small amount of coffee in vase.
Appendix
Table 8: Instructions format of commonsense reasoning task.
GSM8K
Q: The farm has 30 cows and the zoo has 20 sheep. The zoo has twice as many cows as the farm does. The farm has half as many sheep as the zoo does. How many animals do the farm and zoo have combined?
A: The farm has 30 cows and the zoo has 20 sheep. The zoo has twice as many cows as the farm, so the zoo has 30 cows. The farm has half as many sheep as the zoo, so the farm has 10 sheep. In total, the farm and zoo have 30 cows + 20 sheep + 10 sheep = 60 animals. The answer is 60.
Q: Rayna has 60 more dollars than Kassidy, and Kassidy has 3/4 times as much money as Aurelia. If Arelia has $120, how much would each have if they decided to add together their money and share equally among them?
A: Rayna has 60 more dollars than Kassidy. That means Kassidy has 60 less than Rayna. Kassidy has 3/4 times as much money as Aurelia. That means Aurelia has 4/3 times as much money as Kassidy. The answer is 120.
AQuA
Q: A rectangular piece of 150 sq m has a length which is 1m more than the 4 times the breadth. What is the perimeter of the piece? Answer Choices: (A) 60 m (B) 61 m (C) 62 m (D) 63 m (E) 64 m
Appendix
Table 9: Instructions format of arithmetic reasoning task.
Sparse fine-tuning adapts pretrained models by updating selected parameters, reducing training costs and limiting interference with existing capabilities.