When Does Subspace Direction Matter for LoRA? Regime Analysis of the Magnitude Principle in Few-Shot Adaptation
Organizations: Univ. of Delaware · Microsoft Research
Abstract
We show that the importance of a LoRA adapter's initial input-subspace direction is regime-dependent, and we characterize this dependence in theory and experiment. This refines the recent view that spectral initializers help through update magnitude alone. Direction matters most when labeled data are scarce relative to the representation dimension and the activation subspace used for initialization captures supervised task structure. In this regime, the adapter's row span remains sensitive to its starting point, making task-aligned initialization especially useful. A spiked-covariance perturbation analysis characterizes the finite-sample stability of the estimated subspace. A first-step analysis of LoRA training explains why the subspace's alignment with the task matters and motivates an alignment coefficient that closely tracks gains across classification tasks. We test this account with PivotLoRA, which initializes adapters from activation PCA at prediction-relevant positions and applies one output-preserving orthogonal pivot early in training. In few-shot BERT-base classification, PivotLoRA achieves the best average rank among 13 methods and improves over vanilla LoRA in 29 of 30 settings, even though it starts from a smaller input-factor initialization norm. Additional classification experiments extend the evaluation to 7B and 8B Llama decoders. Classification gains diminish with more labeled data, consistent with the proposed regime dependence. A targeted extension to completion-only reasoning also shows improvements on GSM8K and StrategyQA. These findings identify conditions under which initialization direction matters, with subspace-task alignment offering a low-cost diagnostic before training for classification.
Figures & tables
Appendix figures & tables47 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | AG News | SST-2 | CoLA |
| LoRA | .634 | .633 | .183 |
| PiSSA (our implementation) | .477 | .552 | .099 |
| PiSSA (reference PEFT) | .472 | .548 | .087 |
| LoRA | .636 | .639 | .201 |
| LoRAM (our implementation) | .476 | .597 | .141 |
| LoRAM (intended magnitude) | .495 | .580 | .106 |
| Method | 16 | 32 | 64 | 128 | 256 | PivotLoRA ahead |
| LoRA (as in Table 3 ) | .402 | .428 | .488 | .566 | .618 | 29/30 |
| LoRA (standard PEFT initialization) | .385 | .391 | .438 | .501 | .599 | 28/30 |
| LoRAM (as released) | .390 | .400 | .433 | .487 | .573 | 29/30 |
| LoRA-GA (as released) | .427 | .452 | .523 | .583 | .636 | 24/30 |
| LoRA-One (as released) | .409 | .417 | .462 | .575 | .621 | 28/30 |
| SRLoRA (as released) | .389 | .394 | .427 | .478 | .555 | 30/30 |
| Field | Value |
| Checkpoint | meta-llama/Llama-3.1-8B |
| Target modules | q_proj , v_proj |
| Adapted modules | 64 |
| Rank | |
| LoRA scaling | |
| Optimizer | AdamW |
| Method | Seed | Correct | Accuracy (%) | Parse rate (%) | Format (%) |
| Base model | – | 338 | 25.63 | 100.00 | 0.0 |
| LoRA | 42 | 376 | 28.51 | 100.00 | 98.0 |
| LoRA | 43 | 381 | 28.89 | 100.00 | 96.0 |
| LoRA | 44 | 378 | 28.66 | 100.00 | 95.8 |
| PivotLoRA | 42 | 421 | 31.92 | 100.00 | 98.2 |
| PivotLoRA | 43 | 476 | 36.09 | 100.00 | 97.6 |
| Method | Seed | Correct | Accuracy (%) | Parse rate (%) | Format (%) |
| Base model | – | 374 | 54.44 | 85.15 | 0.0 |
| LoRA | 42 | 439 | 63.90 | 98.11 | 95.2 |
| LoRA | 43 | 457 | 66.52 | 99.85 | 99.4 |
| LoRA | 44 | 426 | 62.01 | 99.42 | 96.7 |
| PivotLoRA | 42 | 476 | 69.29 | 99.85 | 99.6 |
| PivotLoRA | 43 | 457 | 66.52 | 99.71 | 99.4 |
| Method | 16 | 32 | 64 | 128 | 256 |
| LoRA | .524 .011 | .546 .037 | .652 .014 | .772 .006 | .844 .012 |
| DoRA | .528 .014 | .544 .036 | .647 .018 | .761 .010 | .841 .008 |
| AdaLoRA | .502 .019 | .500 .008 | .548 .016 | .580 .026 | .633 .019 |
| PiSSA | .503 .023 | .500 .008 | .548 .027 | .665 .058 | .823 .002 |
| LoRA-GA | .503 .018 | .500 .007 | .565 .033 | .664 .077 | .849 .004 |
| LoRA-One | .497 .015 | .501 .007 | .565 .032 | .711 .059 | .854 .010 |
| Method | 16 | 32 | 64 | 128 | 256 |
| LoRA | .081 .056 | .131 .047 | .163 .046 | .264 .072 | .407 .010 |
| DoRA | .085 .060 | .128 .041 | .171 .042 | .257 .070 | .393 .007 |
| AdaLoRA | .096 .055 | .081 .040 | .100 .029 | .152 .033 | .183 .023 |
| PiSSA | .090 .064 | .111 .040 | .111 .083 | .206 .059 | .338 .022 |
| LoRA-GA | .086 .061 | .097 .045 | .103 .078 | .214 .059 | .382 .003 |
| LoRA-One | .087 .060 | .089 .039 | .111 .062 | .240 .067 | .387 .007 |
| Method | 16 | 32 | 64 | 128 | 256 |
| LoRA | .350 .008 | .356 .006 | .366 .003 | .356 .008 | .392 .011 |
| DoRA | .349 .008 | .355 .006 | .366 .002 | .359 .008 | .392 .012 |
| AdaLoRA | .350 .004 | .354 .004 | .347 .013 | .352 .009 | .356 .013 |
| PiSSA | .352 .003 | .354 .002 | .349 .014 | .354 .009 | .383 .023 |
| LoRA-GA | .350 .003 | .356 .003 | .348 .013 | .351 .009 | .397 .014 |
| LoRA-One | .350 .004 | .355 .004 | .348 .013 | .357 .010 | .401 .012 |
| Method | 16 | 32 | 64 | 128 | 256 |
| LoRA | .519 .037 | .521 .040 | .555 .042 | .626 .061 | .662 .040 |
| DoRA | .517 .039 | .519 .040 | .556 .042 | .622 .055 | .663 .035 |
| AdaLoRA | .513 .029 | .514 .029 | .527 .023 | .537 .029 | .546 .024 |
| PiSSA | .510 .027 | .512 .041 | .528 .021 | .574 .035 | .616 .038 |
| LoRA-GA | .512 .027 | .514 .035 | .528 .024 | .567 .045 | .641 .043 |
| LoRA-One | .511 .028 | .516 .031 | .529 .023 | .574 .049 | .644 .045 |
| Method | 16 | 32 | 64 | 128 | 256 |
| LoRA | .508 .040 | .549 .036 | .551 .022 | .569 .015 | .549 .003 |
| DoRA | .511 .040 | .546 .024 | .555 .015 | .568 .018 | .540 .007 |
| AdaLoRA | .510 .035 | .526 .011 | .551 .022 | .540 .012 | .524 .014 |
| PiSSA | .505 .033 | .528 .032 | .539 .019 | .555 .025 | .528 .015 |
| LoRA-GA | .516 .036 | .525 .018 | .552 .016 | .554 .015 | .527 .014 |
| LoRA-One | .515 .037 | .528 .017 | .561 .021 | .561 .007 | .540 .013 |
| Method | 16 | 32 | 64 | 128 | 256 |
| LoRA | .431 .016 | .466 .050 | .640 .033 | .807 .006 | .855 .006 |
| DoRA | .432 .015 | .446 .041 | .619 .016 | .801 .010 | .857 .010 |
| AdaLoRA | .304 .027 | .308 .016 | .368 .038 | .438 .031 | .504 .028 |
| PiSSA | .338 .031 | .364 .007 | .479 .024 | .643 .036 | .795 .024 |
| LoRA-GA | .330 .022 | .354 .012 | .466 .038 | .679 .042 | .812 .030 |
| LoRA-One | .314 .025 | .338 .012 | .475 .033 | .696 .055 | .808 .032 |
| Method | 16 | 32 | 64 | 128 | 256 |
| LoRA | .602 .005 | .603 .010 | .551 .048 | .567 .039 | .601 .019 |
| DoRA | .601 .005 | .606 .007 | .553 .043 | .573 .032 | .596 .022 |
| AdaLoRA | .593 .018 | .603 .012 | .549 .030 | .542 .024 | .576 .017 |
| PiSSA | .597 .015 | .603 .012 | .549 .029 | .563 .030 | .591 .035 |
| LoRA-GA | .596 .015 | .602 .011 | .565 .006 | .562 .030 | .584 .017 |
| LoRA-One | .593 .018 | .602 .011 | .538 .032 | .549 .010 | .580 .017 |
| Method | Average rank ( ) | Wins vs. PivotLoRA |
| PivotLoRA (Ours) | 1.83 | N/A |
| TT-LDA | 3.13 | 5/30 |
| TT-PCA | 3.77 | 3/30 |
| LoRA | 5.23 | 1/30 |
| DoRA | 5.83 | 1/30 |
| SRLoRA | 6.93 | 1/30 |
| Method | SST-2 64 | SST-2 256 | AG News 64 | AG News 256 |
| LoRA | .708 .051 | .924 .007 | .748 .027 | .879 .004 |
| EVA | .713 .015 | .928 .016 | .745 .027 | .879 .004 |
| PiSSA | .635 .058 | .922 .010 | .700 .014 | .865 .010 |
| LoRAM | .763 .025 | .929 .014 | .742 .027 | .876 .003 |
| PivotLoRA | .769 .030 | .912 .020 | .784 .029 | .861 .020 |
| Method | SST-2 64 | SST-2 256 | AG News 64 | AG News 256 |
| LoRA | .704 .037 | .883 .030 | .759 .023 | .880 .009 |
| EVA | .643 .050 | .909 .017 | .743 .034 | .876 .009 |
| PiSSA | .622 .043 | .876 .030 | .715 .027 | .864 .010 |
| LoRAM | .662 .050 | .901 .012 | .748 .047 | .872 .010 |
| PivotLoRA | .709 .085 | .899 .012 | .799 .001 | .870 .030 |
| Method | ||
| LoRA | .554 .025 | .576 .020 |
| PiSSA | .527 .023 | .536 .021 |
| LoRAM | .534 .032 | .573 .014 |
| PivotLoRA | .602 .009 | .618 .004 |
| XSum | CNN/DailyMail | ||||
| Method | Source / recipe | 64 | 256 | 64 | 256 |
| LoRA | — | .185 | .184 | .212 | .227 |
| PivotLoRA | sequence mean / classification | .135 | .164 | .193 | .208 |
| PivotLoRA | response / classification | .147 | .168 | .198 | .210 |
| PivotLoRA | sequence mean / single rate | .188 | .191 | .209 | .228 |
| PivotLoRA | response / single rate | .191 | .192 | .211 | .229 |
| Method | SST-2 | CoLA | MNLI | QNLI | RTE | AG News |
| LoRA | .923 | .533 | .806 | .893 | .610 | .940 |
| PivotLoRA | .914 | .555 | .817 | .889 | .610 | .944 |
| +.022 | +.011 | .000 | +.004 |
| Method | SST-2 | CoLA | MNLI | QNLI | RTE | AG News |
| LoRA | .911 | .515 | .809 | .876 | .545 | .941 |
| PiSSA | .915 | .494 | .812 | .879 | .563 | .941 |
| PivotLoRA | .916 | .565 | .815 | .902 | .606 | .943 |
| Task | Full FT (110M) | LoRA (0.30M) | PivotLoRA (0.30M) |
| SST-2 | .700 .053 | .652 | .795 |
| CoLA | .286 .043 | .163 | .288 |
| AG News | .791 .042 | .640 | .834 |
| MNLI | .345 .007 | .366 | .367 |
| QNLI | .555 .004 | .555 | .594 |
| RTE | .520 .046 | .551 | .569 |
| Task | Fisher ratio | PivotLoRA gain (points) | |
| AG News | 0.843 | 0.14 | +19.4 |
| SST-2 | 0.667 | 0.02 | +14.3 |
| CoLA | 0.530 | 0.02 | +12.5 |
| QNLI | 0.458 | 0.02 | +3.9 |
| MNLI | 0.419 | 0.03 | +0.1 |
| BoolQ | 0.402 | 0.01 | +1.6 |
| Task | Original | Cross-split | PivotLoRA gain |
| AG News | 0.843 | 0.390 | +19.4 |
| SST-2 | 0.667 | 0.285 | +14.3 |
| CoLA | 0.530 | 0.259 | +12.5 |
| Task | Mean SD | Range | Subsets with | PivotLoRA gain (points) |
| AG News | 0.843 0.017 | 0.823–0.874 | 10/10 | +19.4 |
| SST-2 | 0.673 0.185 | 0.191–0.824 | 8/10 | +14.3 |
| QNLI | 0.440 0.079 | 0.360–0.623 | 1/10 | +3.9 |
| BoolQ | 0.457 0.137 | 0.306–0.749 | 1/10 | +1.6 |
| MNLI | 0.444 0.120 | 0.289–0.647 | 1/10 | +0.1 |
| RTE | 0.391 0.109 | 0.255–0.606 | 1/10 | +1.8 |
| Fisher’s discriminant ratio | |||||
| SST-2 | 0.07 | 0.04 | 0.02 | 0.01 | 0.01 |
| CoLA | 0.09 | 0.05 | 0.02 | 0.01 | 0.01 |
| AG News | 0.37 | 0.18 | 0.14 | 0.11 | 0.10 |
| MNLI | 0.16 | 0.07 | 0.03 | 0.02 | 0.01 |
| RTE | 0.07 | 0.03 | 0.01 | 0.01 | 0.00 |
| Setting | Mean gain | Range | ||
| BERT-base, | 768 | 12–48 | to | |
| BERT-base, | 768 | 3–6 | to | |
| BERT-base, full data | 768 | 0.002–0.31 | to | |
| Llama classification, | 4096 | 64 | to | |
| Llama classification, | 4096 | 16 | to | |
| Llama-3.1-8B reasoning | 4096 | 32 | to |
| Activation source | AG News | SST-2 |
| Target position ( [CLS] ) | .814 .015 | .778 .063 |
| All non-padding tokens | .845 .008 | .802 .048 |
| Target for query, all tokens for value | .835 .007 | .784 .024 |
| Target for query, attention-weighted for value | .813 .017 | .666 .045 |
| Paired gain of all-token | +.032 | +.023 |
| All-token seed-level wins | 3/3 | 2/3 |
| Task | LoRA | PivotLoRA | 95% CI | ||
| AG News | 16 | .397 .044 | .577 .035 | +.180 | [+.145, +.217] |
| AG News | 32 | .490 .016 | .721 .047 | +.231 | [+.191, +.265] |
| AG News | 64 | .650 .032 | .816 .022 | +.166 | [+.128, +.205] |
| SST-2 | 32 | .525 .026 | .615 .064 | +.090 | [+.056, +.125] |
| SST-2 | 64 | .655 .032 | .790 .022 | +.135 | [+.112, +.156] |
| CoLA | 64 | .148 .061 | .265 .095 | +.117 | [+.070, +.169] |
| Method | 16 | 32 | 64 | 128 | 256 |
| LoRA | .410 .029 | .477 .078 | .624 .054 | .774 .033 | .858 .010 |
| LoRA-DA | .317 .034 | .352 .015 | .493 .056 | .679 .066 | .814 .030 |
| Method | 16 | 32 | 64 | 128 | 256 |
| LoRA | .541 .017 | .527 .026 | .639 .063 | .777 .010 | .847 .007 |
| LoRA-DA | .528 .016 | .510 .020 | .598 .028 | .702 .056 | .841 .001 |
| 16 | 32 | 64 | 128 | 256 | |
| 1 | .570 | .601 | .762 | .818 | .854 |
| 2 | .568 | .630 | .779 | .821 | .854 |
| 3 (default) | .567 | .608 | .786 | .829 | .844 |
| 5 | .565 | .619 | .794 | .825 | .851 |
| 7 | .569 | .659 | .797 | .820 | .851 |
| 10 | .561 | .612 | .783 | .813 | .850 |
| 16 | 32 | 64 | 128 | 256 | |
| 1 | .127 | .183 | .285 | .403 | .412 |
| 2 | .126 | .191 | .296 | .378 | .400 |
| 3 (default) | .119 | .182 | .289 | .366 | .393 |
| 5 | .132 | .172 | .276 | .355 | .388 |
| 7 | .128 | .183 | .279 | .365 | .403 |
| 10 | .125 | .175 | .270 | .369 | .401 |
| 16 | 32 | 64 | 128 | 256 | |
| 1 | .589 | .708 | .799 | .857 | .877 |
| 2 | .593 | .717 | .802 | .861 | .878 |
| 3 (default) | .604 | .707 | .816 | .866 | .878 |
| 5 | .595 | .724 | .817 | .865 | .879 |
| 7 | .583 | .719 | .821 | .865 | .877 |
| 10 | .516 | .701 | .813 | .865 | .879 |
| Task | Mean | Max. | Mean span residual | Mean | ||
| AG News | 2.809 | |||||
| CoLA | 2.809 | |||||
| SST-2 | 2.809 |
| Method | |||||
| TT-PCA | .520 | .530 | .755 | .817 | .858 |
| QR-Pivot-NoReset | .552 | .623 | .795 | .829 | .858 |
| PivotLoRA | .541 | .628 | .784 | .828 | .847 |
| Method | |||||
| TT-PCA | .104 | .137 | .270 | .383 | .402 |
| QR-Pivot-NoReset | .117 | .172 | .301 | .381 | .398 |
| PivotLoRA | .117 | .174 | .304 | .381 | .393 |
| Method | |||||
| TT-PCA | .478 | .561 | .756 | .857 | .874 |
| QR-Pivot-NoReset | .589 | .689 | .811 | .859 | .878 |
| PivotLoRA | .597 | .703 | .811 | .857 | .878 |
| Variant | SST-2 | CoLA | AG News | Average | Wins |
| TT-PCA (no pivot) | .697 | .257 | .717 | .557 | – |
| Pivot, no rate change | .734 | .275 | .770 | .593 | 11/15 |
| Pivot, implemented policy | .728 | .276 | .772 | .592 | 12/15 |
| Pivot, uniform rate | .744 | .266 | .777 | .596 | 13/15 |
| Pivot, ramp then uniform rate | .723 | .265 | .766 | .585 | 12/15 |
| Method | Avg. anchor | Avg. best | Pivot anchor gap | Pivot best gap |
| LoRA | .522 | .622 | +.139 | +.045 |
| LoRA+ | .326 | .636 | +.335 | +.031 |
| PiSSA | .420 | .606 | +.241 | +.061 |
| LoRAM | .428 | .617 | +.233 | +.050 |
| EVA | .424 | .618 | +.237 | +.048 |
| TT-PCA | .643 | .655 | +.017 | +.012 |
| Method | AG News | SST-2 | CoLA |
| LoRA | .781 | .784 | .301 |
| LoRA+ | .833 | .757 | .318 |
| PiSSA | .754 | .733 | .330 |
| LoRAM | .732 | .789 | .330 |
| EVA | .742 | .786 | .327 |
| TT-PCA | .817 | .821 | .328 |
| Method | AG News | SST-2 | CoLA |
| LoRA | .640 .033 | .647 .040 | .188 .042 |
| TT-PCA | .751 .025 | .769 .038 | .268 .092 |
| Random+EigenLR | .444 .268 | .571 .031 | .028 .035 |
| TT-PCA+UniformLR | .752 .019 | .769 .035 | .276 .100 |
| TT-PCA+ShuffledLR | .754 .028 | .765 .028 | .281 .088 |
| SST-2, | SST-2, | AG News, | AG News, | |||||||||
| s42 | s43 | s44 | s42 | s43 | s44 | s42 | s43 | s44 | s42 | s43 | s44 | |
| LoRA | .711 | .757 | .655 | .931 | .917 | .922 | .724 | .743 | .777 | .877 | .884 | .876 |
| PiSSA | .683 | .571 | .650 | .911 | .927 | .930 | .706 | .709 | .684 | .867 | .874 | .854 |
| LoRAM | .765 | .787 | .737 | .927 | .944 | .915 | .719 | .771 | .737 | .878 | .877 | .872 |
| PivotLoRA | .802 | .742 | .764 | .927 | .921 | .889 | .809 | .791 | .753 | .876 | .869 | .839 |
| SST-2, | SST-2, | AG News, | AG News, | |||||||||
| s42 | s43 | s44 | s42 | s43 | s44 | s42 | s43 | s44 | s42 | s43 | s44 | |
| LoRA | .728 | .723 | .661 | .912 | .885 | .851 | .758 | .783 | .737 | .870 | .885 | .886 |
| PiSSA | .626 | .578 | .663 | .911 | .865 | .853 | .710 | .744 | .691 | .865 | .874 | .854 |
| LoRAM | .720 | .640 | .627 | .907 | .909 | .888 | .735 | .800 | .708 | .877 | .879 | .861 |
| PivotLoRA | .735 | .778 | .614 | .888 | .912 | .899 | .800 | .799 | .798 | .889 | .884 | .836 |
| Method | Initialization time | Additional persistent storage |
| LoRA | s | None |
| PivotLoRA | s plus s at pivot | None |
| PiSSA | s, layerwise SVD | Residual |
| LoRA-GA | s, gradient plus SVD | None |
| LoRA-One | s, gradient plus SVD | None |
| EVA | s, activation SVD | None |
| Method | Wall time (s) | Peak VRAM (MB) | Overhead vs. LoRA |
| LoRA | 69.4 | 1,266 | N/A |
| DoRA | 69.5 | 1,399 | +0.1 s (+0.2%) |
| AdaLoRA | 102.2 | 1,264 | +32.9 s (+47.4%) |
| PiSSA | 68.6 | 1,267 | s ( ) |
| LoRA-GA | 68.0 | 1,272 | s ( ) |
| LoRA-One | 69.0 | 1,272 | s ( ) |
| Method | Wall time (s) | Peak VRAM (GB) |
| LoRA | 14.6 | |
| PiSSA | 14.6 | |
| PivotLoRA | 14.6 |
Low-Rank Adaptation
Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.
See all 588 papers