Parameter-efficient fine-tuning (PEFT) aims to preserve adaptation quality while reducing training costs. We propose SCALAR, which achieves performance comparable to LoRA with substantially fewer trainable parameters, lower peak training memory, and higher throughput. SCALAR constructs a fixed input basis from task activations and assigns a random output dictionary to each input direction. Fine-tuning learns only layer-specific scalar coefficients, reducing gradient and optimizer-state storage. Grouped computation aggregates output atoms before applying the adapter to token activations, while dictionary sharing across compatible layers reduces fixed-basis storage. Experiments on commonsense reasoning, mathematical reasoning, code generation, and knowledge evaluation after instruction tuning demonstrate effective adaptation at small trainable budgets. On commonsense reasoning, SCALAR achieves accuracy comparable to LoRA with about 4% of its trainable parameters. System measurements on Qwen2.5-7B show 20% lower peak training memory and 6% higher throughput than LoRA. The combination of adaptation quality, a compact trainable state, and training efficiency makes SCALAR a promising approach to resource-constrained fine-tuning.
Figures & tables
Figure 1: Average accuracy on 8 commonsense reasoning tasks across trainable parameter budgets. SCALAR approaches LoRA-level accuracy with substantially fewer trainable parameters.
LoRA
LoRA-XS
VeRA
NOLA
RandLoRA
SCALAR
ΔW
BA
BMA
ΛbBΛdA
(∑i=1kBαiBi)(∑j=1kAβjAj)
∑j=1qBjΛjAΓj
∑r=1R∑h=1Hγr,hbr,har⊤
Trainable params
R(m+n)
R2
m+R
kA+kB
⌈d/R⌉(d+R)
RH
Table 1: Comparison of trainable parameters across methods. Let W∈Rm×n and d=min(m,n) . LoRA, LoRA-XS, VeRA, NOLA, and SCALAR use rank R . NOLA uses kA and kB random bases, while SCALAR uses H random output atoms per input direction.
Method
Params
BoolQ
PIQA
SIQA
HS
WG
ARC-E
ARC-C
OBQA
Avg.
LoRA
2,523,136
74.1
90.3
81.8
95.7
88.7
96.2
89.4
91.8
88.5
Learned-direction reweighting
196
73.9
90.1
81.8
96.0
89.0
96.4
89.9
92.4
88.7
Table 2: Accuracy of LoRA and learned-direction reweighting on 8 commonsense reasoning tasks. Learned-direction reweighting achieves comparable performance while optimizing fewer than 0.01% as many parameters as LoRA.
Model
Method
Params (%)
BoolQ
PIQA
SIQA
HS
WG
ARC-E
ARC-C
OBQA
Avg.
Qwen2.5-7B
FT
100.00
75.3
89.7
82.0
96.1
89.7
95.6
88.4
93.1
88.7 ±0.1
LoRA ( r=1 )
0.0331
74.3
90.3
81.9
95.8
88.7
96.1
89.2
91.7
88.5 ±0.1
LoRA ( r=8 )
0.2643
74.6
90.4
81.9
96.3
89.3
95.9
89.5
92.3
88.8 ±0.1
VeRA
0.0208
73.7
90.2
81.5
95.6
87.7
96.3
88.8
91.5
88.2 ±0.2
NOLA
0.0105
73.6
90.0
82.0
95.8
88.6
96.4
89.2
91.5
88.4 ±0.0
LoRA-XS
0.0105
73.4
89.8
82.4
95.6
87.2
96.5
88.5
90.9
88.0 ±0.2
Table 3: Commonsense reasoning accuracy of LLaMA3-8B and Qwen2.5-7B across 8 benchmarks. Results are averaged over three independent runs. The SCALAR configurations are highlighted.
Figure 2: Memory and throughput for fine-tuning Qwen2.5-7B on commonsense reasoning. We report peak GPU memory with a device batch size of 16 and token throughput on 2 NVIDIA H20 GPUs. SCALAR uses less memory and achieves higher throughput than the compared baselines.
Method
Params (%)
GSM8K
MATH
Avg.
HumanEval
MBPP
Avg.
FT
100.00
84.6
48.8
66.7
74.4
82.5
78.4
LoRA ( r=1 )
0.0331
84.8
48.3
66.6
74.4
81.5
78.0
LoRA ( r=8 )
0.2643
84.8
48.2
66.5
70.1
83.3
76.7
VeRA
0.0189
84.4
48.7
66.6
72.6
82.5
77.6
NOLA
0.0013
84.3
47.6
66.0
73.2
82.0
77.6
LoRA-XS
0.0014
83.0
47.1
65.1
72.6
81.0
76.8
Table 4: Comparison on mathematical and code generation benchmarks on Qwen2.5-7B. SCALAR achieves comparable performance on mathematical reasoning and code generation.
Random- A
Weight-SVD- A
Activation-SVD- A
Random- B
84.3
84.4
85.2
Weight-SVD- B
84.5
84.6
85.1
Table 5: Commonsense reasoning Acc. on Qwen2.5-3B with different initialization methods.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Gradient norms of the trainable adapter parameters when the input-side factor A uses activation-SVD or random initialization during Qwen2.5-7B fine-tuning on commonsense reasoning. The two input-side initializations are matched in scale. Activation-SVD initialization yields larger observed gradient norms over most of this run.
Fast SVD
Full SVD
Eigendecomposition time (s)
6.29
123.0
Speedup vs. full
19.6×
1×
Mean cosine cos equation 7
0.998
Affinity R1∑icos2θi
0.996
Mean principal angle (deg)
0.500
Appendix
Table 6: Fast (randomized) vs. full eigendecomposition for activation Gram C on Qwen2.5-7B ( R=16 ). Time is the summed top- R solve over all groups; subspace metrics compare Fast and full on the same Grams.
Model
K=128
K=256
K=512
Qwen2.5-7B
25,088 (0.0003%)
50,176 (0.0006%)
100,352 (0.0013%)
LLaMA3-8B
28,672 (0.0003%)
57,344 (0.0006%)
114,688 (0.0014%)
K=1024
K=2048
K=4096
Qwen2.5-7B
200,704 (0.0026%)
401,408 (0.0052%)
802,816 (0.0105%)
LLaMA3-8B
229,376 (0.0028%)
458,752 (0.0057%)
917,504 (0.0114%)
Appendix
Table 7: Trainable params (%) with different K on Qwen2.5-7B and LLaMA3-8B.
Hyperparameters
FT
LoRA
VeRA
NOLA
LoRA-XS
RandLoRA
SCALAR
Optimizer
AdamW
LR scheduler
Cosine
Precision
BF16
Batch size
32
Epochs
3
Warmup steps
100
Appendix
Table 8: Hyperparameters for fine-tuning LLaMA3-8B and Qwen2.5-7B on commonsense reasoning tasks.
Hyperparameters
FT
LoRA
VeRA
NOLA
LoRA-XS
RandLoRA
SCALAR
LR scheduler
Cosine
Precision
BF16
Batch size
256
Epochs
1
Warmup steps
100
Max sequence length
1024
Appendix
Table 9: Hyperparameters for fine-tuning Qwen2.5-7B on mathematical reasoning and code generation tasks.
Model
Method
Params (%)
BoolQ
PIQA
SIQA
HS
WG
ARC-E
ARC-C
OBQA
Avg.
Qwen2.5-7B
FT
100.00
75.3
89.7
82.0
96.1
89.7
95.6
88.4
93.1
88.7 ±0.1
LoRA ( r=1 )
0.0331
74.3
90.3
81.9
95.8
88.7
96.1
89.2
91.7
88.5 ±0.1
LoRA ( r=8 )
0.2643
74.6
90.4
81.9
96.3
89.3
95.9
89.5
92.3
88.8 ±0.1
VeRA
0.0208
73.7
90.2
81.5
95.6
87.7
96.3
88.8
91.5
88.2 ±0.2
VeRA
0.0235
73.6
90.0
81.8
95.8
88.0
96.1
89.5
91.2
88.2 ±0.1
NOLA
0.0026
73.2
89.3
82.5
95.8
88.0
96.5
89.0
91.2
88.2 ±0.1
Appendix
Table 10: Accuracy comparison (3 independent runs) of Qwen2.5-7B and LLaMA3-8B with various PEFT methods on eight commonsense reasoning benchmarks. This table supplements the main commonsense reasoning table by reporting the performance of each baseline under different trainable parameter budgets.
Figure 4: Optimization behavior of rank-one LoRA and learned-direction reweighting. The reweighting model freezes the directions learned by LoRA and re-optimizes one scalar coefficient for each adapted matrix. It optimizes 224 coefficients for LLaMA3-8B and 196 for Qwen2.5-7B, and follows a similar loss-reduction trajectory in these runs.
Model
Method
Params
Avg. Acc.
LLaMA3-8B
LoRA ( r=1 )
2,621,440
87.2
Direction reweighting
224
87.5
Qwen2.5-7B
LoRA ( r=1 )
2,523,136
88.5
Direction reweighting
196
88.7
Appendix
Table 11: Average accuracy of rank-one LoRA and learned-direction reweighting on eight commonsense reasoning tasks. The reweighting model uses fewer than 0.01% as many trainable parameters as rank-one LoRA and attains similar reported averages in these two settings.
Figure 5: Diagnostic training dynamics of LoRA on Qwen2.5-7B. We report the relative normalized captured activation-energy ratio ρt=ρt/ρ0 , where ρt is defined in equation 9 . Values above 1 indicate stronger activation alignment than the random-initialized LoRA input subspace. Most module groups in this run show higher normalized activation coverage, with variation across layers and training steps.
Method
Params (%)
Avg. Acc.
LoRA
0.0331
88.5
DoRA
0.0513
88.7
PiSSA
0.0331
88.5
EVA
0.0331
88.7
SCALAR
0.0105
88.9
Appendix
Table 12: Comparison with enhanced LoRA variants on Qwen2.5-7B. SCALAR achieves comparable reported accuracy with fewer trainable parameters than the evaluated LoRA variants.
Model
Method
Params (%)
STEM
Humanities
Social Sciences
Other
Overall
FT
100.00
27.37
27.04
24.54
25.97
26.33
LoRA
0.6481
27.85
26.82
24.67
26.65
26.54
SCALAR ( K=1024 )
0.0186
27.66
26.84
26.06
26.62
26.81
Gemma3-1B
SCALAR ( K=2048 )
0.0373
27.78
27.10
25.67
27.97
27.13
FT
100.00
48.33
51.94
65.68
64.34
56.89
LoRA
0.3770
48.02
51.82
64.97
64.15
56.57
Appendix
Table 13: Post-tuning knowledge performance on MMLU. Params denotes the percentage of trainable parameters. The best parameter-efficient result for each model is highlighted in bold.
Model
Method
Params (%)
BoolQ
PIQA
SIQA
HS
WG
ARC-E
ARC-C
OBQA
Avg.
LoRA ( r=1 )
0.0291
78.6
93.1
84.9
97.4
92.3
97.7
93.4
95.8
91.7
LoRA ( r=8 )
0.2324
77.8
93.7
83.8
97.6
92.3
97.9
93.4
96.8
91.7
SCALAR ( K=4096 )
0.0093
77.8
94.0
84.9
97.4
91.9
97.9
93.8
96.6
91.8
Qwen2.5-14B
SCALAR ( K=8192 )
0.0186
78.0
93.4
84.3
97.6
92.7
97.9
93.4
97.8
91.9
LoRA ( r=1 )
0.0255
80.0
94.7
84.0
97.7
92.7
98.7
95.6
97.4
92.6
LoRA ( r=8 )
0.2044
80.6
94.3
84.5
98.0
93.8
98.5
96.0
97.2
92.9
Appendix
Table 14: Commonsense reasoning results on larger Qwen2.5 models. Params denotes the percentage of trainable parameters. LoRA introduces r(m+n) trainable parameters for an m×n weight matrix, whereas SCALAR assigns K trainable scalar coefficients to each adapted module, independent of m and n .
Method
Iter./s
Mem. (GB)
Avg. Acc.
Grouped, equation 10 (K=1024,R=16)
0.75
49.1
88.5
Ungrouped, equation 11 (K=1024)
0.58
49.1
88.3
Appendix
Table 15: Ablation on the grouped update parameterization. Results are reported on the commonsense reasoning tasks using Qwen2.5-7B. The variants use the same coefficient budget and differ in the number of input directions, maximum update rank, and token-dependent computation.
Method
Iter./s
Mem. (GB)
Avg. Acc.
Shared B(K=1024)
0.75
49.1
88.5
Unshared B(K=1024)
0.75
51.6
88.5
Shared B(K=2048)
0.75
49.3
88.6
Unshared B(K=2048)
0.75
54.2
88.6
Shared B(K=4096)
0.74
49.7
88.9
Unshared B(K=4096)
0.74
59.4
88.8
Appendix
Table 16: Ablation on cross-layer sharing of B . Sharing the output-side bases maintains nearly identical accuracy and throughput while substantially reducing memory consumption, especially as K increases.
K=1024
K=2048
K=4096
R=8
88.4
88.6
88.7
R=16
88.5
88.6
88.9
R=32
88.5
88.6
88.8
R=64
88.4
88.5
88.7
Appendix
Table 17: Effect of the update rank R under fixed trainable-parameter budgets. We report the average accuracy on the commonsense reasoning benchmarks using Qwen2.5-7B. Performance remains stable across a broad range of ranks, with R=16 giving the strongest overall results.
Calibration size
Average accuracy
Calibration time (s)
16
88.3
73.483
32
88.5
108.18
64
88.4
147.67
128 (default)
88.5
237.44
256
88.5
373.28
512
88.4
927.95
Appendix
Table 18: Effect of the calibration-set size on downstream performance and calibration cost on the 7B model. Average accuracy is computed over the eight commonsense reasoning benchmarks. Calibration time denotes the one-time cost of collecting activation statistics and computing the input subspaces.
Output-atom initialization
K=1024
K=2048
Gaussian (default)
85.1
85.2
Rademacher
85.1
85.2
Kaiming
85.1
85.1
Gaussian, σ=0.05
85.0
85.2
Gaussian, σ=0.10
85.1
85.2
Gaussian, σ=0.20
85.0
85.0
Appendix
Table 19: Effect of the initialization distribution and element-wise scale of the fixed output atoms B . The upper block compares different initialization rules, while the lower block varies the Gaussian standard deviation σ .
Method
LoRA
LoRA-XS
VeRA
NOLA
RandLoRA
SCALAR
Compact adapter checkpoint
R(m+n)
R2+R(m+n)
m+R+seeds
kA+kB+seeds
Nscale+seeds
Rn+RH+seeds
Appendix
Table 20: Storage comparison for a single adapted matrix W∈Rm×n . Here, R is the adapter rank, H is the number of SCALAR output atoms per input direction, K=RH , and Nscale denotes the method-dependent number of stored scaling parameters for RandLoRA. Fixed random bases are assumed to be recoverable from seeds when applicable.
Storage type
Qwen2.5-3B
Qwen2.5-7B
Rank- 8 LoRA deployable adapter
29.93 MB
40.37 MB
SCALAR compact checkpoint ( R=8 )
14.45 MB
18.92 MB
SCALAR deployable adapter ( R=8 )
29.93 MB
40.37 MB
Rank- 16 LoRA deployable adapter
59.86 MB
80.74 MB
SCALAR compact checkpoint ( R=16 )
27.87 MB
37.04 MB
SCALAR deployable adapter ( R=16 )
59.86 MB
80.74 MB
Appendix
Table 21: Practical adapter storage on Qwen2.5-3B and Qwen2.5-7B under BF16 precision. We report LoRA and SCALAR with R∈{8,16} , while fixing K=RH=2048 for SCALAR . Storage is reported in MB with 1MB=106 bytes.
Large language models (LLMs) remain expensive to fine-tune because full-parameter updates require substantial memory, compute, and per-task storage. We study whether saliency signals originally developed for pruning can be reused to choose where a model should adapt. We propose Super, a sparse parameter-efficient fine-tuning (PEFT) method that fixes a small trainable support using a Wanda-style activation-weighted magnitude score [Sun et al., 2023] computed from a calibration pass. We then introduce Supra, a hybrid adapter that combines this sparse update with LoRA while preserving a matched trainable-parameter budget through a simple budget-splitting rule. In single-seed Math17K arithmetic experiments on Llama-3.2-1B and Meta-Llama-3-8B, the best Super/Supra variants achieve the highest average accuracy among the tested schedule-selected adapter configurations. We also include a PaFi-style magnitude-only support as a closest training-free sparse baseline and find that low-score supports under both magnitude and Wanda-style orderings can be effective. These results suggest that simple pruning-inspired orderings can provide useful fixed sparse supports for PEFT, especially when combined with low-rank adapters.
Parameter-efficient fine-tuning (PEFT) reduces the cost of adapting foundation models by focusing training on a small parameter subset. Complementary to this idea, we introduce RoSA (Rotational Sparse Adaptation), which narrows adaptation to a subset of layers at a time. RoSA freezes lower layers close to the input throughout training and rotates a trainable block over later layers, progressively increasing the number of frozen layers close to the input. This design reduces optimizer-state memory, shortens backpropagation, and even forward propagation if activations at the last frozen layer are cached. Because RoSA is orthogonal to the choice of trainable parameterization, it can be combined with PEFT methods or sparse optimizers within each active block. Experiments across multiple LLM architectures and tasks show that RoSA reduces peak memory while maintaining strong fine-tuning performance.
Muhammad Azeem Lodhi, Chao Zhou, Rebekka Burkholz
Saarland University, Saarbrücken, Germany · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
Parameter-efficient fine-tuning (PEFT) enables large language models (LLMs) to adapt to specialized tasks, but often at the cost of degrading general capabilities acquired during pretraining. Existing approaches primarily mitigate this trade-off through data replay or regularization, relying on additional data or explicit optimization constraints. We instead focus on a different question: where should adaptation be applied? We find that fine-tuning different Transformer layers produces different target-task gains and degrees of capability degradation, suggesting that not all layers are equally suitable for adaptation. To characterize this difference, we use layer-wise empirical Fisher information to measure target-task sensitivity. However, computing Fisher scores requires backward computation and becomes increasingly expensive for large models. We therefore introduce input--output cosine similarity as a lightweight, forward-only proxy for ranking layer sensitivity. Across models and tasks, layers with lower input--output similarity consistently exhibit higher empirical Fisher scores. Building on this observation, we propose Layer-Selective LoRA (LS-LoRA), which places trainable LoRA adapters only in layers with low input--output similarity. Experiments on mathematical reasoning and code generation show that LS-LoRA improves average target-task performance while retaining substantially more commonsense reasoning capability than standard all-layer LoRA, demonstrating that carefully choosing where to adapt can provide a simple and effective way to balance target-task adaptation and general capability retention.
Zhiqiang Pang, Zihong Sun, Qi Xie +3
School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, Shaanxi, China