Activation-Efficient Fine-Tuning via Sequence-Compressed Low-Rank Subspaces

Irene Tenison; Stella Ahn; Miriam Kim; Ebtisam Alshehri; Lalana Kagal

cs.LGApr 3, 2026

Activation-Efficient Fine-Tuning via Sequence-Compressed Low-Rank Subspaces

Authors: Irene Tenison, Stella Ahn, Miriam Kim, Ebtisam Alshehri, Lalana Kagal

Organizations: MIT CSAIL · Harvard SEAS

Abstract

Parameter-Efficient Fine-Tuning (PEFT) has become the standard approach for adapting large language models (LLMs). However, despite reducing trainable parameter counts, many PEFT methods remain memory-bound by token-level activation intermediates that scale linearly with sequence length, limiting their practicality for long-context and on-device adaptation. Unlike prior memory-efficient PEFT approaches that preserve token-level adapter activations, LARS (Low-memory Activation-Rank Subspace) reduces the sequence-length scaling of stored adapter activation intermediates from O(BSRL) to O(BRL) by performing adaptation within a sequence-level low-rank subspace. Through pooled feature extraction, low-rank subspace modulations, and residual projection, LARS reduces the overall memory footprint by 23% on GPUs and 47.8% on CPUs relative to LoRA at S = 1024, with relative savings increasing with context length. Experiments across reasoning, understanding, long-context, retrieval, and token-level tasks show that LARS maintains competitive performance relative to LoRA while substantially improving memory efficiency. Beyond GPUs, LARS also reduces average CPU power and energy consumption per optimisation step by 22.43% and 22.06%, respectively. These results demonstrate that activation-efficient adaptation provides a practical path toward resource-constrained and on-device fine-tuning of LLMs.

Figures & tables

Figure 1: Accuracy vs. peak memory (GB) for state-of-the-art PEFT methods. Bubble size represents the count of trainable parameters. Our analysis reveals a critical disconnect: trainable-parameter count is a poor proxy for actual memory footprint.

Figure 2: Peak memory scaling vs. sequence length. (Left) Without GC and with GC enabled (Right), LARS defines a more efficient Pareto frontier, enabling the fine-tuning of sequences longer than LoRA within the same hardware constraints.

Figure 3: Memory footprint breakdown of Llama 3.2 1B with LoRA and LARS across sequence lengths 512, 1024, and 5120. LARS reduces the sequence-length-dependent activation overhead of the adapter, yielding 12%, 23%, and 34% lower overall memory consumption, respectively. Adapter-specific weights, gradients, and optimizer states are comparatively small and are not visually prominent. Estimated adapter-specific activations are the adapter internals that are reduced by LARS. Adapter buffer includes activations of adapter outputs and this scales with $S$ . More details in Appendix A.2

Figure 4: Overview of LARS. Unlike LoRA, which performs token-level low-rank adaptation and retains sequence-dependent intermediates of size $\mathbb{R}^{B\times S\times R}$ during backpropagation, LARS compresses hidden states into a pooled sequence-level representation and performs adaptation entirely within a compact low-rank subspace $\mathbb{R}^{B\times R}$ . As a result LARS reduces adapter activation scaling from $\mathcal{O}(BSRL)$ in LoRA to $\mathcal{O}(BRL)$ .

Figure 5: Accuracy and Memory on Llama 1B and Qwen 7B on reasoning and understanding tasks across various datasets. While LARS consumes significantly lower memory than LoRA, IA3, and AdaLoRA, the performance is comparable on reasoning and understanding tasks.

Figure 6: Impact of Gating, Mixing, and Transformation components on Accuracy and Memory Usage (GB) across the BoolQ and Economics datasets.

	(%)	Llama-3.2-1B						Qwen2.5-7B-Instruct
	Trainable	Reasoning		Understanding		Long Context		Reasoning		Understanding		Long Context
	Params	Mem.( $\downarrow$ )	Acc.( $\uparrow$ )	Mem.( $\downarrow$ )	Acc.( $\uparrow$ )	Mem.( $\downarrow$ )	Acc.( $\uparrow$ )	Mem.( $\downarrow$ )	Acc.( $\uparrow$ )	Mem.( $\downarrow$ )	Acc.( $\uparrow$ )	Mem.( $\downarrow$ )	Acc.( $\uparrow$ )
LoRA	0.45	19.06	76.19	35.72	35.74	25.56	57.91	35.72	35.73	62.73	57	52.1	67.37
AdaLoRA	0.45	16.06	75.8	32.6	37.4	23.44	58.03	32.06	38.4	52.81	53.99	46.73	67.59
IA3	0.01	19.83	71.66	36.99	37.4	26.49	54.13	36.99	37.44	63.81	52.91	53	62.02
Prefix	0.03	13.47	39.23	25.49	9.05	22.30	24.62	25.49	9.03	45.22	9.17	38.89	25.95

Table 1: Average memory usage and accuracy of LARS and its learned pooling variant, LARS-LP on various reasoning, understanding, and long context tasks compared against baselines - LoRA, AdaLoRA, IA3, Prefix tuning, and Prompt Tuning. LARS consumes an average of

25\%

lower memory than the baselines across tasks and models while performing comparably to LoRA.

Figure 7: Memory consumption and accuracy of Qwen 7B on long context Quality and RACE datasets

	CoNLL(SL)		CoNLL(SD)	SQuAD		SAMSum
	acc	F1	Entity-F1	EM	F1	Rouge-1	Rouge-L
LARS	98.12	0.78	0.77	0.556	0.644	0.277	0.204
LoRA	96.11	0.73	0.82	0.558	0.646	0.302	0.230
IA3	96.89	0.75	0.73	0.414	0.495	0.294	0.220
AdaLoRA	96.68	0.76	0.75	0.556	0.642	0.293	0.224

Table 2: Additional experiments on representative token-level and generation tasks: sequence labeling (SL) and structured decoding (SD) on CoNLL, Extractive QA on SQuAD, and Summarization on SAMSum, to better understand the effect of sequences-compression in LARS.

Figure 8: Memory Usage (GB) and Accuracy of LARS and baselines with 4 bit and 8 bit quantization.

Figure 9: Throughput of LARS and other baselines during inference and fine-tuning across sequence lengths 256 and 1024.

	Accuracy (%)	Memory (GB)	Throughput (tok/sec)	Time (sec)	Power (W)	Energy (J)
S = 256
LARS	76.02	15.824	113.22	31.67	74.64	2238.16
LoRA	77.56	27.059	153.25	32.15	135.96	4231.88
IA3	67.5	18.937	101.78	27.8	89.5	2466.48
S = 512
LARS	74.58	15.926	111.88	30.65	140.81	4731.87

Table 3: Accuracy, memory consumption, throughput, average time per step, average power per step and average energy per step of LARS, LoRA and IA3 on AMD EPYC CPU.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Accuracy vs. peak training memory (GB) for state-of-the-art PEFT methods with CP. Even with checkpointing, the disconnect remains: trainable-parameter count is a poor proxy for actual memory footprint.

Figure 11: Accuracy vs. peak training memory (GB) for state-of-the-art PEFT methods without (left) with (right) CP. Unlike Figures 1 and 10 , these are based on static padding and the disconnect remains.

		Base Model Weights (GB)	Base Model Est. Activations (GB)	Adapter Est. Activations (GB)	Adapter Weights (MB)	Adapter Grad.(MB)	Adapter Opt. States (MB)	Adapter Buffer (GB)
	LARS	4.94	1.38	0.035	33.38	33.38	67.76	0.78
S=512	LoRA	4.94	1.38	1.39	22.58	22.58	45.16	0.83
	LARS	4.94	13.84	0.035	33.38	33.38	67.76	7.8
S=5120	LoRA	4.94	13.84	13.9	22.58	22.58	45.16	8.3

Appendix

Table 4: Memory decomposition of LoRA and LARS at

S=512

and

S=5120

. The adapter activations have been split into adapter-specific activations or internals and adapter buffer, which includes the activations of adapter outputs stored for backpropagation.

		Base Model Weights (GB)	Base Model Est. Activations (GB)	Adapter Est. Activations (GB)	Adapter Weights (MB)	Adapter Grad.(MB)	Adapter Opt. States (MB)
	LARS	4.94	0.08	0.05	33.38	33.38	67.76
S=512	LoRA	4.94	0.08	0.13	22.58	22.58	45.16
	LARS	4.94	0.78	0.48	33.38	33.38	67.76
S=5120	LoRA	4.94	0.78	1.33	22.58	22.58	45.16

Appendix

Table 5: Memory decomposition of LoRA and LARS at

S=512

and

S=5120

with gradient checkpointing. The reported estimated adapter activations include both adapter internals and buffers as they are significantly smaller in comparison to Table 4 due to gradient checkpointing.

Figure 12: Code snippets corresponding to the 3 stages in LARS - Sequence pooling, low-rank subspace modulations, and residual projection via broadcasting.

lr	1e-2	1e-3	1e-4	1e-5	1e-6	1e-2	1e-3	1e-4	1e-5	1e-6
	SIQA					Hellaswag
LARS	76.61	78.85	78.04	78.75	70.21	25	85.62	90.2	90.2	75.62
LoRA	42.94	76.25	78.77	79.2	75	30	70.11	90.45	90.21	75.62
IA3	71.25	73.75	74.92	48.75	45	85.62	85.77	85.26	65.2	45
AdaLoRA	71.25	78.5	76.61	63.75	25	30	90.21	85.85	75.2	25

Appendix

Table 6: Learning rate sweep for LARS, LoRA, AdaLoRA, and IA3 on SIQA and HellaSwag.

Figure 13: Comparison of accuracy and memory on long context tasks using QuALITY and Race datasets for models Llama 3.2 1B model.

Figure 14: Inference latency of LARS and other baselines.

Figure 15: Comparison of Memory Usage (GB) and Accuracy across different target modules for the LARS, LoRA, and IA3 fine-tuning methods.

Figure 16: Comparison of Memory Usage (GB) and Accuracy for the LARS, LoRA, and AdaLoRA methods across different ranks (r)

Figure 17: Inference and training throughput of LARS and baselines with increasing ranks.

Figure 18: Comparison of Memory Usage (GB) and Accuracy across increasing Data Sizes for the LARS, LoRA, and IA3 fine-tuning methods.

	LARS	LoRA	AdaLoRA	IA3
1024	99.2	99.2	99.2	99.2
16k	98.4	98.4	98.4	98.4
32k	72.6	73.2	72.6	72.6

Appendix

Table 7: Accuracy of Needle-In-A-Haystack (NIAH) style retrieval of a passkey.

	LoRA		LARS		LoRA +FA		LARS +FA		LoRA+ DropBP		LARS+ DropBP
	acc (%)	mem (GB)	acc (%)	mem (GB)	acc (%)	mem (GB)	acc (%)	mem (GB)	acc (%)	mem (GB)	acc (%)	mem (GB)
BoolQ	83	13.67	81.1	10.24	81.77	10.99	81.94	9.97	70.72	11.89	74.79	9.67
SIQA	77.28	20.55	76.74	14.45	77.17	14.52	77.04	10.58	73.44	12.34	73.95	11.85

Appendix

Table 8: Accuracy and memory consumption of LoRA and LARS in combination with DropBP Woo et al. (2024) and LoRA-FA Zhang et al. (2024) . These results reinstate theire complementary nature to the proposed method, LARS.

Figure 19: Impact of different model sizes on Accuracy and Memory Usage (GB) of LARS and baselines.

Figure 20: Accuracy and Memory Consumption of LARS and baselines with and without FlashAttention.

Memory-Efficient Fine-Tuning

Memory-efficient fine-tuning reduces the GPU, device, and communication resources needed to adapt large models while preserving task performance.

See all 61 papers

Techniques for Peak Memory Reduction for LoRA Fine-tuning of LLMs on Edge Devices
Jun 17, 2026Hassan Dbouk, Matthias Reisser, Prathamesh Mandke +2Memory-Efficient Fine-Tuning LLM Fine-Tuning
Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. · work done while at Qualcomm
Energy- and Memory-Efficient PEFT Methods for Personalized On-Device SLMs on Consumer GPUs
Aug 5, 2026Kuanysh Akhmetzhanov, Jurn-Gyu ParkFine-Tuning On-Device Language Model Inference
Department of Computer Science, Nazarbayev University, Astana 010000, Kazakhstan
RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning
Oct 5, 2026Muhammad Azeem Lodhi, Chao Zhou, Rebekka BurkholzMemory-Efficient Fine-Tuning LLM Fine-Tuning
Saarland University, Saarbrücken, Germany · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

Jun 17, 2026J/K move · Enter open · S save

Techniques for Peak Memory Reduction for LoRA Fine-tuning of LLMs on Edge Devices

Hassan Dbouk, Matthias Reisser, Prathamesh Mandke, Likhita Arun Navali+1

Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. · work done while at Qualcomm

Memory-Efficient Fine-Tuning LLM Fine-Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Fine-tuning of Large Language Models (LLMs) using Low-Rank Adaptation (LoRA) on an end-user's data offers personalized experiences while keeping data private, but faces severe memory constraints on consumer hardware. Peak memory during fine-tuning often exceeds device limits, especially for models with billions of parameters and long-context training data. This paper introduces a suite of complementary techniques to reduce memory footprint without sacrificing model quality: (1) base model quantization with on-the-fly dequantization, (2) memory-efficient checkpointing combining selective activation caching and disk offloading, (3) softmax approximation using semantically relevant token subsets, and (4) logits masking. Experiments on Llama-3.2 3B and Qwen-2.5 3B demonstrate up to

26\times

and

28\times

reduction in peak memory, enabling fine-tuning on resource-constrained devices.

Abstract

Memory-Efficient Fine-Tuning

Explore similar work

Techniques for Peak Memory Reduction for LoRA Fine-tuning of LLMs on Edge Devices

Energy- and Memory-Efficient PEFT Methods for Personalized On-Device SLMs on Consumer GPUs

RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning