Robust Parameter-Efficient LLM Adaptation on Analog Hardware

Jindan Li; Zhaoxian Wu; Tianyi Chen

cs.LGOct 4, 2026

Robust Parameter-Efficient LLM Adaptation on Analog Hardware

Authors: Jindan Li, Zhaoxian Wu, Tianyi Chen

Organizations: Cornell University

Abstract

Analog in-memory computing is a promising platform for on-device execution of large language models because it performs matrix--vector multiplications (MVMs) in memory and in parallel, reducing data movement. However, limited digital-to-analog converter precision, input noise, and finite conductance states can degrade model accuracy, while full-model retraining to address these effects can be costly. We develop an optimizer-agnostic, parameter-efficient adaptation method based on Low-Rank Adaptation (LoRA), keeping the pretrained weights stored on analog arrays fixed while training the LoRA weights to adapt to downstream tasks and hardware non-idealities. Reliable adaptation requires handling errors in both forward and backward MVMs and physical weight updates. We use input reshaping to reduce input-induced MVM errors and update accumulation to retain small updates before programming them to finite-state analog devices. Across Llama-3.2-1B-Instruct and Llama-3-8B with both Muon and AdamW, input reshaping improves analog LoRA fine-tuning under noisy MVM computation. Update accumulation separately preserves sub-threshold updates and substantially improves adaptation under finite-resolution programming, including configurations with as few as 20 conductance states. Additional experiments show consistent held-out negative log-likelihood improvements across noisy analog settings.

Figures & tables

Figure 1: Overview of analog LoRA fine-tuning. Pre-fine-tuning calibration provides channel statistics for input reshaping. For the frozen base path, function-preserving base-path rescaling reduces the input range. For the trainable LoRA path, outlier-channel splitting routes persistent outlier channels through the protected analog path and keeps the remaining channels on the noisy analog path. During weight programming, digital mini-batch aggregation combines sample gradients, and the update accumulator stores sub-threshold updates until they can be converted into programming pulses. Blue and green denote analog and digital computation, respectively.

Model	I/O	Optimizer	No reshaping + UA	Reshaping + UA	Reshaping w/o UA
Llama-3-8B	Perfect	Muon	46.84 / 37.89	47.84 / 39.26	35.51 / 26.17
Llama-3-8B	Perfect	AdamW	48.50 / 40.63	48.36 / 40.43	35.51 / 26.17
Llama-3-8B	7/0.02	Muon	24.49 / 18.36	43.55 / 35.35	28.73 / 19.92
Llama-3-8B	7/0.02	AdamW	31.51 / 24.22	43.86 / 35.94	30.32 / 20.90
Llama-3.2-1B	Perfect	Muon	30.88 / 23.24	30.10 / 24.22	10.61 / 0.00
Llama-3.2-1B	Perfect	AdamW	28.19 / 21.09	29.34 / 21.88	10.61 / 0.00

Table 1: Primary QA results. Each entry reports F1/EM (%). Reshaping denotes base-path rescaling + outlier-channel splitting. UA denotes update accumulation. I/O

b/s

denotes a

b

-bit input DAC with input-noise standard deviation

s

$N$	I/O setting	Muon [NLL $\downarrow$ ]		AdamW [NLL $\downarrow$ ]
		No reshaping	Reshaping	No reshaping	Reshaping
–	Digital	0.6823 $\pm$ 0.0003	–	0.7235 $\pm$ 0.0001	–
100	Perfect	0.7121 $\pm$ 0.0006	–	0.7548 $\pm$ 0.0013	–
20	Perfect	0.7896 $\pm$ 0.0025	–	0.8727 $\pm$ 0.0000	–
100	4/0.02	0.7356 $\pm$ 0.0009	0.7209 $\pm$ 0.0001	0.7852 $\pm$ 0.0005	0.7649 $\pm$ 0.0007
100	6/0.01	0.7154 $\pm$ 0.0004	0.7104 $\pm$ 0.0002	0.7569 $\pm$ 0.0007	0.7521 $\pm$ 0.0005

Table 2: Qwen/OpenThoughts held-out NLL. Mean

\pm

standard deviation over three seeds; lower is better. UA is enabled throughout.

Setting	Optimizer	No reshaping	Eval-only reshaping	Train+eval reshaping
No adaptation	–	$7.12\pm 0.07$	$9.14\pm 0.09$	–
Adaptation	AdamW	$18.20\pm 1.06$	$22.02\pm 0.26$	$\mathbf{23.57\pm 0.28}$
Adaptation	Muon	$19.52\pm 0.73$	$21.82\pm 0.67$	$\mathbf{24.41\pm 0.58}$

Table 3: Training-time versus evaluation-only reshaping. F1 (%) is reported as mean

\pm

sample standard deviation over three repeated noisy evaluations. Without adaptation, reshaping only uses base-path rescaling (BR). With adaptation, reshaping uses both BR and outlier-channel splitting.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	I/O	Optimizer	Validation NLL
			No reshaping + UA	Reshaping + UA	Reshaping w/o UA
Llama-3-8B	Perfect	Muon	1.8349	1.8250	–
Llama-3-8B	Perfect	AdamW	1.8728	1.8650	–
Llama-3-8B	7/0.02	Muon	2.3346	1.9617	4.5610
Llama-3-8B	7/0.02	AdamW	2.5149	2.1374	4.4824
Llama-3.2-1B	7/0.02	Muon	3.0193	2.9235	6.7572

Appendix

Table 4: Validation NLL for the primary QA experiments. Lower is better.

	Muon			AdamW
Configuration	F1	EM	NLL	F1	EM	NLL
No reshaping + UA	18.45	12.70	3.1689	26.89	20.31	3.0256
Reshaping + UA	39.96	31.84	2.3762	39.41	31.84	2.1780
Reshaping w/o UA	31.08	21.29	4.4745	30.26	20.70	4.5044

Appendix

Table 5: Llama-3-8B QA at

N=20

. F1 and EM are in %. Results are point estimates.

$N$	I/O setting	Muon [NLL $\downarrow$ ]		AdamW [NLL $\downarrow$ ]
		No reshaping	Reshaping	No reshaping	Reshaping
$10^{4}$	Perfect	0.6841 $\pm$ 0.0024	–	0.7001 $\pm$ 0.0008	–
100	Perfect	0.7164 $\pm$ 0.0012	–	0.7373 $\pm$ 0.0014	–
100	4/0.01	0.7355 $\pm$ 0.0021	0.7239 $\pm$ 0.0015	0.7564 $\pm$ 0.0025	0.7429 $\pm$ 0.0024
100	6/0.02	0.7302 $\pm$ 0.0021	0.7211 $\pm$ 0.0019	0.7488 $\pm$ 0.0024	0.7419 $\pm$ 0.0023
$10^{4}$	6/0.02	0.7048 $\pm$ 0.0015	0.6941 $\pm$ 0.0001	0.7151 $\pm$ 0.0023	0.7070 $\pm$ 0.0039

Appendix

Table 6: Llama-3-8B/OpenOrca response-token NLL. Mean

\pm

standard deviation over three seeds; lower is better. UA is enabled throughout.

$N$	I/O	Muon		AdamW
		No reshaping + UA	Reshaping + UA	No reshaping + UA	Reshaping + UA
$10^{4}$	Perfect	51.58 $\pm$ 1.4 / 44.33 $\pm$ 1.4	–	51.84 $\pm$ 0.7 / 44.53 $\pm$ 1.1	–
100	Perfect	50.67 $\pm$ 1.2 / 43.69 $\pm$ 0.5	–	49.95 $\pm$ 0.9 / 42.64 $\pm$ 0.6	–
100	7/0.02	46.84 $\pm$ 1.3 / 39.45 $\pm$ 1.3	50.81 $\pm$ 0.3 / 43.49 $\pm$ 0.8	45.00 $\pm$ 0.4 / 38.35 $\pm$ 0.9	49.06 $\pm$ 0.2 / 41.73 $\pm$ 0.5
100	6/0.01	49.76 $\pm$ 1.6 / 42.06 $\pm$ 1.4	50.72 $\pm$ 0.2 / 43.29 $\pm$ 0.2	48.42 $\pm$ 0.5 / 41.60 $\pm$ 0.6	48.85 $\pm$ 0.8 / 41.15 $\pm$ 0.2

Appendix

Table 7: Recovered linear-attention Llama-3-8B QA. Results at

N\in\{100,10^{4}\}

are mean

\pm

standard deviation over three seeds.

$N$	I/O	Muon		AdamW
		No reshaping + UA	Reshaping + UA	No reshaping + UA	Reshaping + UA
20	Perfect	49.01 / 42.38	–	50.00 / 43.16	–
20	7/0.02	40.42 / 33.40	48.08 / 41.41	45.85 / 38.09	49.64 / 42.19
20	6/0.01	47.54 / 41.21	48.77 / 41.80	48.20 / 40.62	47.95 / 41.02

Appendix

Table 8: Recovered linear-attention Llama-3-8B QA at

N=20

. Results are point estimates.

BR Off, OS Off	BR On, OS Off	BR On, OS On
24.49 (18.36)	42.01 (33.79)	43.55 (35.35)

Appendix

Table 9: Input-reshaping component ablation. Llama-3-8B QA with Muon under

N=100

and I/O setting 7/0.02. Results are point estimates of F1 (EM), in %. BR denotes function-preserving base-path rescaling and OS denotes outlier-channel splitting.

Figure 2: Calibration stability on Qwen/OpenThoughts. Each cell counts channels in a 64-step window’s top-8 set that are absent from the calibration set; zero indicates full overlap. Rows are monitored layers and columns are training windows.

Panel of Figure 3 — Figure 3: Input-reshaping quantization-noise diagnostics. Left: remaining analog channels’ input-noise variance ratio (Eq. ( 6 )). Right: base-path output-noise energy ratio, including weight rescaling (Eq. ( 7 )). Cells average per-token ratios within 200-step windows; values below one indicate compression.

Explore similar work

PLoRA: Efficient Concurrent LoRA Training for Large Language Models
Aug 4, 2025Minghao Yan, Zhuang Wang, Zhen Jia +2Training-Side Stage-Aware Low-Rank Adaptation Low-Rank Adaptation Framework
University of Wisconsin-Madison · Amazon Web Services
Signs Beat Floats: Low-Rank Double-Binary Adaptation for On-Device Fine-Tuning
May 22, 2026Yoshihiko Fujisawa, Yuma Ichikawa, Yudai Fujimoto +2Low-Rank Adaptation Adapters Large Language Model Quantization
Fujitsu Limited · Institute of Science Tokyo · RIKEN Center for AIP +1
Strategic Over-Parameterization for Generalizable Low-Rank Adaptation
May 15, 2026Jing Gao, Zhong-Yi Lu, Pan Zhang +1Training-Side Stage-Aware Low-Rank Adaptation Low-Rank Adaptation Adapters
School of Fundamental Physics and Mathematical Sciences, Hangzhou Institute for Advanced Study, UCAS, Hangzhou 310024, China · School of Physical Sciences, University of Chinese Academy of Sciences, No. 19A Yuquan Road, Beijing 100049, China · CAS Key Laboratory of Theoretical Physics, Institute of Theoretical Physics, Chinese Academy of Sciences, Beijing 100190, China +1

Aug 4, 2025, cs.LGJ/K move · Enter open · S save

PLoRA: Efficient Concurrent LoRA Training for Large Language Models

Minghao Yan, Zhuang Wang, Zhen Jia, Shivaram Venkataraman+1

University of Wisconsin-Madison · Amazon Web Services

Training-Side Stage-Aware Low-Rank Adaptation Low-Rank Adaptation Framework Low-Rank Adaptation Adapters

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) has gained popularity as a fine-tuning approach for Large Language Models (LLMs) due to its low resource requirements and good performance. While numerous studies have investigated ways to improve LoRA serving efficiency by serving multiple LoRAs concurrently, existing methods assume that a wide range of LoRA adapters are available for serving. In our work, we conduct extensive empirical studies to show that current LoRA training paradigms do not efficiently utilize hardware resources and incur high overhead to obtain a performant LoRA adapter. Leveraging these insights, we propose PLoRA, which automatically orchestrates concurrent LoRA fine-tuning jobs under given hardware and model constraints and develops performant kernels to improve training efficiency. Across a range of LLMs and LoRA configurations, PLoRA improves training throughput by up to 12.8x and reduces the overall fine-tuning makespan by up to 7.52x compared to existing approaches.

Abstract

Explore similar work

PLoRA: Efficient Concurrent LoRA Training for Large Language Models

Signs Beat Floats: Low-Rank Double-Binary Adaptation for On-Device Fine-Tuning

Strategic Over-Parameterization for Generalizable Low-Rank Adaptation