Efficient LLM Adaptation with Forward-Only Passes
Organizations: Lund University, Sweden · Google DeepMind, USA
Abstract
Large language model (LLM) inference serving is undergoing rapid growth and large-scale deployment, motivating us to rethink how the inference process itself can be leveraged to enable efficient task-specific LLM adaptation. In this paper, we propose a forward-only approach for efficient LLM adaptation with forward-only passes in LLM inference serving. We exploit the angle concentration of activations induced by each singular value decomposition (SVD) component to measure its contribution-dispersed or concentrated-to the angle concentration of the hidden states. Based on this, our forward-only search then efficiently identifies the weight matrix with the highest dispersion that merits rank reduction. We then selectively remove the higher-order components and retain the lower-order components in SVD. Empirical results across diverse datasets demonstrate the competitive accuracy of our forward-only approach, while theoretical analysis shows lower peak memory and greater speedup than the gradient-based approach and greater speedup than the exhaustive search approach. The experiments further present the robustness and generalization of our forward-only approach to various LLMs with up to 57B parameters. The code is included in the supplementary material.
Figures & tables
| Method | Weight Memory | Dynamic Memory |
| Baseline | ||
| LASER | ||
| LASER-Grad | ||
| Ours |
| Dataset | Baseline | LASER (std) | LASER-Grad (std) | Ours (hierarchical) | ||||||
| Acc | Mem | Acc | Mem | Acc | Speedup | Mem | Acc | Speedup | Mem | |
| CounterFact | 13.1 | 11.3 GB | 24.0 | 11.5 GB | 24.4 | 1.99 | 21.8 GB | 24.0 | 47.1 | 11.5 GB |
| HotPotQA | 19.6 | 11.3 GB | 19.5 | 11.5 GB | 19.9 | 1.99 | 21.9 GB | 20.6 | 38.4 | 11.5 GB |
| FEVER | 50.2 | 11.3 GB | 56.2 | 11.5 GB | 56.0 | 1.99 | 21.9 GB | 58.4 | 37.4 | 11.5 GB |
| Bios Gender | 70.9 | 11.3 GB | 97.5 | 11.5 GB | 88.4 | 1.99 | 22.2 GB | 97.5 | 45.2 | 11.5 GB |
| Bios Profession | 75.6 | 11.3 GB | 82.1 | 11.5 GB | 80.5 | 1.99 | 22.2 GB | 79.1 | 40.7 | 11.5 GB |
| LASER (subset) | Ours | |||||||||
| Dataset | (std) | (hierarchical) | (subset) | |||||||
| Acc | Speedup | Mem | Acc | Speedup | Acc | Speedup | Acc | Speedup | Mem | |
| CounterFact | 17.7 | 131 | 489 MB | 19.2 | 2.4 | 17. 7 | 40.4 | 17.7 | 617 | 493 MB |
| HotPotQA | 5.9 | 29.2 | 489 MB | 6.3 | 4.8 | 6.3 | 32.9 | 5.5 | 137 | 496 MB |
| FEVER | 52.4 | 26.2 | 489 MB | 52.5 | 4.8 | 52.5 | 31.9 | 52.5 | 123 | 496 MB |
| Bios Gender | 92.2 | 79.3 | 495 MB | 93.4 | 4.8 | 93.4 | 38.7 | 93.2 | 372 | 514 MB |
| LLMs | Baseline | LASER | Ours | |
| Acc | Acc | Acc | Speedup | |
| LLaMA2-7B | 44.8 | 63.4 | 63.5 | 6.4 |
| DeepseekV1-7B | 55.1 | 61.4 | 61.8 | 29.6 |
| Mistral-7B | 54.8 | 61.4 | 61.4 | 9.1 |
| Qwen2-7B | 51.4 | 58.3 | 58.3 | 27.6 |
| Gemma-7B | 38.3 | 60.2 | 61.4 | 11.1 |
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Ours (subset) | Ours (std) | Ours (hierarchical) | Hit? |
| CounterFact | Yes | |||
| HotPotQA | No | |||
| FEVER | Yes | |||
| Bios Gender | Yes | |||
| Bios Profession | No | |||
| TruthfulQA | No |
| Dataset | Ours (std) | Ours (hierarchical) | ||
| Acc | Speedup | Acc | Speedup | |
| HotPotQA | 19.5 | 5.6 | 20.6 | 38.4 |
| Bios Profession | 79.8 | 5.6 | 79.1 | 40.7 |
| TruthfulQA | 55.7 | 5.6 | 55.7 | 28.4 |
| Dataset | LASER (subset) | LASER-Grad (subset) | Ours (std) | Ours (subset) | |||||||
| Acc | Speedup | Mem | Acc | Speedup | Mem | Acc | Speedup | Mem | Acc | Speedup | |
| CounterFact | 15.9 | 131 | 11.5 GB | 24.2 | 349 | 21.8 GB | 24.0 | 5.6 | 11.5 GB | 23.0 | 728 |
| HotPotQA | 19.5 | 29.2 | 11.5 GB | 19.7 | 77.5 | 21.9 GB | 19.5 | 5.6 | 11.5 GB | 19.6 | 162 |
| FEVER | 53.4 | 26.2 | 11.5 GB | 53.3 | 69.4 | 21.9 GB | 58.4 | 5.6 | 11.5 GB | 50.3 | 145 |
| Bios Gender | 97.6 | 79.3 | 11.5 GB | 88.4 | 210 | 22.2 GB | 97.5 | 5.6 | 11.5 GB | 97.9 | 439 |
| Bios Profession | 73.6 | 38.4 | 11.5 GB | 77.5 | 102 | 22.2 GB | 79.8 | 5.6 | 11.5 GB | 76.5 | 213 |
| Dataset | Ours (subset) | Ours (std) | Ours (hierarchical) | Hit? |
| CounterFact | No | |||
| HotPotQA | Yes | |||
| FEVER | Yes | |||
| Bios Gender | Yes | |||
| Bios Profession | Yes | |||
| TruthfulQA | No |
| Dataset | Baseline | LASER-Grad (std) | LASER (std) | Ours | |||||||
| Acc | Mem | Acc | Speedup | Mem | Acc | Mem | Acc | Acc | Speedup | Mem | |
| CounterFact | 17.3 | 479 MB | 19.3 | 0.86 | 814 MB | 19.3 | 489 MB | 19.2 | 19.2 | 2.4 | 493 MB |
| HotPotQA | 6.1 | 481 MB | 6.5 | 0.86 | 824 MB | 6.7 | 489 MB | 6.8 | 6.3 | 32.9 | 496 MB |
| FEVER | 50.0 | 481 MB | 52.7 | 0.86 | 822 MB | 52.3 | 489 MB | 51.9 | 52.5 | 31.9 | 496 MB |
| Bios Gender | 87.5 | 495 MB | 93.1 | 0.86 | 893 MB | 93.7 | 495 MB | 93.5 | 93.4 | 38.7 | 514 MB |
| Bios Profession | 64.5 | 495 MB | 75.1 | 0.86 | 891 MB | 72.5 | 495 MB | 71.6 | 71.9 | 34.8 | 514 MB |
| Size of | 100 | 10 | 5 | 1 |
| CounterFact | ✔ | ✔ | ✔ reordered | |
| HotPotQA | ✔ | ✔ | ✔ | |
| FEVER | ✔ reordered | ✔ | ✔ | |
| Bios Gender | ✘ ✔ ours | ✘ ✔ ours | ✘ ✔ ours | |
| Bios Profession | ✔ reordered | ✔ | ✘ ✔ ours | |
| TruthfulQA | ✔ reordered | ✔ reordered | ✔ reordered |
| Dataset | (100 from ) | Sequence Length | (20% of ) | |
| CounterFact | 100 | 17 | 13151 | 65757 |
| HotPotQA | 100 | 29 | 2923 | 14618 |
| FEVER | 100 | 27 | 2617 | 13086 |
| Bios Gender | 100 | 102 | 7928 | 39642 |
| Bios Profession | 100 | 100 | 3844 | 19223 |
| TruthfulQA | 100 | 36 | 1176 | 5882 |
| Setting | Max Sequence Length | ||||
| Roberta | 50265 | 768 | 12 | 12 | 512 |
| GPT-J | 50400 | 4096 | 16 | 28 | 2048 |
| Considering , and | Same as only ? |
| CounterFact | ✔ |
| HotPotQA | ✔ |
| FEVER | ✔ |
| Bios Gender | ✔ |
| Bios Profession | ✔ |
| TruthfulQA | ✔ |
| Dataset | Setting | Batch Size | Fine-Tuning Epoch | Learning Rate | Weight Decay |
| FEVER | Cold Start/Warm Start (Table 16 ) | 256 | 1 | 6e-5 | 0.01 |
| Fine-Tuning (Table 15 ) | 256 | 50 | 6e-5 | 0.01 | |
| Bios Gender | Cold Start/Warm Start (Table 16 ) | 256 | 1 | 2e-6 | 0.01 |
| Fine-Tuning (Table 15 ) | 256 | 10 | 2e-6 | 0.01 |
| Dataset | FEVER | Bios Gender | ||||||
| Fine-Tuning | Fine-Tuning | |||||||
| Final Candidate (50 epochs) | All | All | Ours (std) | Final Candidate (10 epochs) | All | All | Ours (std) | |
| Final Candidate (100 epochs) | Final Candidate (20 epochs) | |||||||
| Trainable Params | 1.89% | 22.7% | - | 1.89% | 22.7% | - | ||
| Acc | 74.6 0.24 | 76.5 0.16 | 76.5 0.16 | 52.5 | 94.2 0.02 | 95.6 0.02 | 95.6 0.01 | 93.4 |
| (75.2 0.14 ) | (95.1 0.01 ) | |||||||
| Dataset | FEVER | Bios Gender | ||||
| Fine-Tuning Final Candidate | Ours (std) | Fine-Tuning Final Candidate | Ours (std) | |||
| Cold Start | Warm Start | Cold Start | Warm Start | |||
| Acc | 50.00 0.00 | 53.35 0.17 | 52.5 | 88.58 0.03 | 93.50 0.11 | 93.4 |
| LLMs | LLaMA2-7B | Qwen2-7B-Instruct/Gemma-7B | DeepSeekV1-7B-Chat | Gemma4-E2B-it |
| Mistral-7B | Qwen2-57B-A14B-Instruct | |||
| 32 | 28 | 30 | 35 |
| LLMs | LASER (std) | Ours (std) | top- | last- |
| LLaMA2-7B | top- | last- | ||
| DeepseekV1-7B-Chat | top- | last- | ||
| Mistral-7B | top- | last- | ||
| Qwen2-7B-Instruct | top- {\color[rgb]{0,0,0}2} | last- | ||
| Gemma-7B | [\boldsymbol{W}_{out},3,{\color[rgb]{0,0,0}5}\%] | [\boldsymbol{W}_{in},{\color[rgb]{0,0,0}7},20\%] | top- {\color[rgb]{0,0,0}5} | last- {\color[rgb]{0,0,0}2}0 |
| Gemma4-E2B-it | top- | last- |
| Dataset | Ours (subset) | Ours (std) | Ours (hierarchical) | Hit? |
| CounterFact | Yes | |||
| HotPotQA | No | |||
| FEVER | No | |||
| Bios Gender | Yes | |||
| Bios Profession | Yes | |||
| TruthfulQA | No |
| Baseline | LASER | Ours | ||||||
| Dataset | (std) | (hierarchical) | (subset) | |||||
| Acc | Acc | Acc | Speedup | Acc | Speedup | Acc | Speedup | |
| CounterFact | 35.6 | 37.6 | 35.9 | 6.4 | 35.9 | 53.9 | 34.9 | 832 |
| HotPotQA | 16.5 | 17.2 | 19.7 | 6.4 | 20.5 | 43.9 | 17.5 | 185 |
| FEVER | 59.3 | 64.5 | 64.5 | 6.4 | 62.9 | 42.7 | 59.9 | 165 |
| Bios Gender | 75.5 | 88.4 | 91.5 | 6.4 | 91.5 | 51.7 | 91.5 | 502 |
| Baseline | Ours (with ) | Steering Prompts (transmute to ) | ||||
| 20% of | 40% of | 60% of | 80% of | |||
| 64.5 | 71.9 | 72.0 | 71.9 | 71.8 | 72.0 | 72.0 |
| Roberta | Our Approach | Laser-Grad | Our Approach | Laser-Grad |
| Dataset | FEVER | Bios Profession | ||
| Average Sequence Length | 27 | 100 | ||
| Memory Change | 130 MB | 343 MB | 235 MB | 1062 MB |
| Average Complexity | 3G FLOPs | 10G FLOPs | 13G FLOPs | 38G FLOPs |
| Average Execution Time | 1.00 second | 3.34 seconds | 1.31 seconds | 5.55 seconds |
| LLaMA2 on FEVER | Baseline | Ours (std) | ICL (1-shot) | ICL (2-shot) | ICL (4-shot) | ICL (8-shot) | ICL (16-shot) |
| Accuracy | 59.3 | 64.5 | 74.6 | 75.4 | 76.2 | 74.3 | 77.2 |
| Additional Tokens | 0 | 0 | 33 | 61 | 114 | 237 | 461 |
| Roberta | Bios Gender | Bios Profession | ||
| Final Candidate | Accuracy (%) | Final Candidate | Accuracy (%) | |
| Direct Inference | - | 87.5 | - | 64.5 |
| Activation Norms | 90.0 | 71.9 | ||
| Activation Variance | 91.8 | 67.2 | ||
| Cosine Dispersion Without SVD | 89.2 | 70.9 | ||
| Random Top-Layer Selection | 91.8 | 65.1 | ||