cs.AINov 7, 2026

Efficient LLM Adaptation with Forward-Only Passes

Authors: Baichuan Huang, Ananth Balashankar, Amir Aminifar

Organizations: Lund University, Sweden · Google DeepMind, USA

Abstract

Large language model (LLM) inference serving is undergoing rapid growth and large-scale deployment, motivating us to rethink how the inference process itself can be leveraged to enable efficient task-specific LLM adaptation. In this paper, we propose a forward-only approach for efficient LLM adaptation with forward-only passes in LLM inference serving. We exploit the angle concentration of activations induced by each singular value decomposition (SVD) component to measure its contribution-dispersed or concentrated-to the angle concentration of the hidden states. Based on this, our forward-only search then efficiently identifies the weight matrix with the highest dispersion that merits rank reduction. We then selectively remove the higher-order components and retain the lower-order components in SVD. Empirical results across diverse datasets demonstrate the competitive accuracy of our forward-only approach, while theoretical analysis shows lower peak memory and greater speedup than the gradient-based approach and greater speedup than the exhaustive search approach. The experiments further present the robustness and generalization of our forward-only approach to various LLMs with up to 57B parameters. The code is included in the supplementary material.

Figures & tables

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. PreFT: Prefill-only finetuning for efficient inference

    May 14, 2026Andrew Lanpouthakoun, Aryaman Arora, Zhengxuan Wu +4Fine-TuningLLM Serving

  2. Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs

    Jul 23, 2026Yidu Wu, Xiang Wang, Kejie Zhao +3LLM PruningLLM Inference Acceleration

  3. Strategic Over-Parameterization for Generalizable Low-Rank Adaptation

    May 15, 2026Jing Gao, Zhong-Yi Lu, Pan Zhang +1LLM Fine-TuningLow-Rank Adaptation