cs.LG · 2604.20313 Copy arXiv ID · Apr 22, 2026 Save Formalising the Logit Shift Induced by LoRA: A Technical Note Authors: Xiang Shi , Shuaizhi Cheng , Mingwei Li
Organizations: Imperial College London · Harbin Institute of Technology · KigLand Learning Lab
Abstract This technical note provides a first-order formalisation of the logit shift and fact-margin change induced by Low-Rank Adaptation (LoRA). Using a first-order Fréchet approximation around the base model trajectory, we show that the multi-layer LoRA effect can be decomposed into a linear summation of layerwise contributions and a higher-order remainder term representing inter-layer coupling.
Explore similar work Jun 2, 2026 · Zhengbao He, Ruiqi Ding, Zhehao Huang +3 Low-Rank Adaptation Large Language Model Compression
May 16, 2026 · cs.LG J/K move · Enter open · S save
Ramakrishnan Sathyavageeswaran
Low-rank adaptation (LoRA) assigns a uniform rank to every adapted weight matrix - a practical convenience that ignores a fundamental reality: different layers contribute unequally to task adaptation. We address this with a lightweight engineering solution: before fine-tuning begins, run eight calibration backward passes, compute the gradient variance of each LoRA-B matrix as a proxy for layer informativeness, and redistribute the rank budget proportionally. The resulting adapter is a standard LoRA with a per-layer rank pattern - no new parameters, no training overhead, no changes to serving infrastructure. We implement this via an efficient approximation of the empirical Fisher Information Matrix (eFIM) diagonal, restricted to LoRA adapter matrices only, which reduces memory cost by approximately 256x compared to full-model Fisher estimation. On GLUE with DeBERTa-v3-base, FIM-LoRA matches LoRA (88.6 vs. 88.7) at the same parameter budget, and on commonsense reasoning with LLaMA-3-8B reaches 68.5 vs. 68.7 for LoRA. The per-layer rank maps are interpretable: value projections and early-to-middle layers consistently receive higher rank, consistent with established findings on transformer layer roles.