Enabling Retraining-Free Accurate Inference in Analog Compute in Memory Accelerators

George Karfakis; Puneet Gupta

cs.ARNov 5, 2026

Enabling Retraining-Free Accurate Inference in Analog Compute in Memory Accelerators

Authors: George Karfakis, Puneet Gupta

Organizations: Department of Electrical and Computer Engineering, University of California, Los Angeles, CA, USA.

Abstract

Analog compute-in-memory (CIM) accelerators compute matrix-vector products directly inside memory arrays, eliminating weight movement and promising order-of-magnitude gains in inference efficiency. However, converter and device errors make pretrained models unusable on these arrays, and hardware-aware retraining is costly. We enable retraining-free, accurate inference of pretrained models on simple analog arrays with low-cost, shared calibration. Our key observation is that analog error depends on how each layer is represented on the array, and that this representation can be changed offline without changing the computation. We center the inputs and apply Hadamard rotations designed for analog arrays, which keep shared reference errors spread and confine input mixing to each array. Across CNNs, vision transformers and LLMs up to 14B parameters on PCM and ReRAM, the method brings vision models and larger LLMs within a few points of floating-point accuracy and matches or approaches published hardware-aware training without training any weights. On ReRAM with 6-bit ADCs, it restores ViT-B/16 from 37.4% to 82.9% and Phi-4 from 27.0% to 72.2%, both within 2.2 points of floating-point accuracy.

Figures & tables

Observation	Array effect	Our fix	Sec.
Static input components	Static work picks up converter and device error on every input	Center inputs and restore digitally	3.1
Concentrated partial outputs	Peaks force a clip–quantization tradeoff under one ADC range	Regular output rotation to lower peaks	3.2
Concentrated weights	Outliers set the mapping scale and amplify conductance perturbations	Input and output rotations to reduce weight peaks	3.3
Constraint: shared/common-mode terms	Plain Hadamard can concentrate common-mode (e.g., reference) error	Regular Hadamard blocks ( $R\mathbf{1}=\mathbf{1}$ )	3.2
Constraint: tiled conversion	Cross-tile mixing changes per-tile quantized partial sums	Tile-local input rotations	3.4

Table 1: How the structure of the computation and the array’s constraints determine our isometric reparameterization.

Figure 1: An analog CIM array of width

w

and height

h

with its DAC and ADC (left), and a weight matrix split into tiles whose partial sums are added digitally before the bias (right).

Figure 2: Static input component across all analog layers of ViT-B/16, ResNet-50 and Qwen3-1.7B, with 4-bit ADCs and without rotations or device noise. Left: share of

\|x\|_{2}^{2}

in the per-channel mean. Right: local ADC error without and with centering. Bars are medians over layers.

Figure 3: Output rotation reduces average local ADC error across the FFN down-projections of ViT-B/16 and Qwen3-1.7B, with MSE-calibrated ranges (top) and ranges set from the largest output (bottom). Projections are untiled, with 8-bit DACs, one 4-bit ADC range per projection, no centering and no device noise. NMSE is averaged and Peak/RMS is the median over layers.

Figure 4: Regular Hadamard blocks are needed to avoid the large ADC error increase of plain blocks with random signs (DHP), which perform worse than no output rotation. Projections are untiled, with 8-bit DACs, one MSE-calibrated 4-bit ADC range per projection and no device noise.

Figure 5: Weight peak-to-RMS for no rotation, output only, input only, and both, across all 72 projections of ViT-B/16 and all 196 of Qwen3-1.7B. Output and input rotations each lower weight concentration on their own, but both are needed for the largest reduction. Faint lines are single projections, and medians are printed.

Figure 6: Local output error from device noise before and after both rotations, across the down projections of ViT-B/16 and Qwen3-1.7B, with PCM and ReRAM models at one day. Projections are untiled, with one weight mapping scale each and no converters or centering.

Figure 7: (a) Our method. Offline: calibrate the channel mean

\mu

, fold

c=W\mu+b

into the bias, build

R_{\mathrm{in}}

and

R_{\mathrm{out}}

, transform the weights to

\widetilde{W}=R_{\mathrm{out}}WR_{\mathrm{in}}^{\top}

, then program

\widetilde{W}

and calibrate the converters. Online: (1) center, (2) rotate the inputs within each tile, (3) compute in each array and digitize its partial output, (4) sum the digitized partial outputs, (5) undo the output rotation and apply the drift gain

g

(for PCM), (6) add

c

. (b) Placement of the rotations in a CNN bottleneck and a transformer block (ViT or LLM).

Table 9

Figure 8: ImageNet top-1 accuracy of ViTs and CNNs on PCM and ReRAM with N-bit ADCs, 1 day after programming.

Model	Device	Unmodified	HWA	Ours
ResNet-50	PCM	$\mathit{61.3\pm 3.1}$	74.7	$\mathbf{73.7\pm 0.1}$
ResNet-18	PCM	$\mathit{56.1\pm 0.8}$	68.0	$\mathbf{67.5\pm 0.1}$
ResNet-50	ReRAM	$\mathit{58.1\pm 1.9}$	72.4	$\mathbf{72.8\pm 0.1}$
ResNet-18	ReRAM	$\mathit{44.9\pm 3.3}$	65.1	$\mathbf{65.6\pm 0.2}$

Table 4: ResNet accuracy (%) alongside published HWA results Rasch et al. (2023) , with PCM or the ReRAM model of Wan et al. (2022) , 8-bit ADCs and one-day read age, with per-column weight mapping scales as in Rasch et al. (2023) . Unmodified and Ours report mean

\pm

SD over three seeds.

Figure 9: LLM mean accuracy over MMLU, HellaSwag, ARC-Easy and ARC-Challenge on PCM and ReRAM with 4-, 6- and 8-bit ADCs, one day after programming. Lines mark floating-point accuracy.

Model	Unmodified	AFM	Ours
Phi-3-mini	$\mathit{72.9\pm 0.9}$	79.7	$\mathbf{78.4\pm 0.5}$
Llama-3.2-1B	$\mathit{36.4\pm 1.6}$	45.5	$\mathbf{42.8\pm 1.5}$

Table 5: LLM mean accuracy (%) compared with the published AFM results Büchel et al. (2025) , in our reproduction of their setting, with their checkpoints, scoring and noise level (Appendix J ). Unmodified and Ours report mean

\pm

SD over ten seeds.

Figure 10: Accuracy as output rotations, input rotations and centering are added in that order, on PCM with 4-bit ADCs at one day and ReRAM with 8-bit ADCs at five days. Dashed lines mark floating-point accuracy.

Figure 11: Accuracy with per-array or per-column weight mapping scales and ADC ranges, one day after programming. NORA uses its native dynamic DAC.

Figure 12: Accuracy against read age with 6-bit ADCs. The arrays are programmed and calibrated once at 1 s and not refreshed.

Figure 13: Accuracy against the digital share of all operations on ReRAM with 8-bit ADCs at five days. Each curve is the frontier of the evaluated configurations, and dotted lines mark its peak accuracy. The unmodified models lie below the plotted range.

Figure 14: Accuracy with 4- and 6-bit ADCs and DACs on PCM at one day and ReRAM at five days: top-1 accuracy for ViT-B/16 and ResNet-50, and WikiText-2 Merity et al. (2017) perplexity for Qwen3-1.7B (lower is better). ViT-B/16 floating-point accuracy is 84.6%, ResNet-50 76.1%.

Figure 15: SRAM CIM accuracy with one weight mapping scale and one ADC range per tensor (open dots, as in PWS) or per array (filled dots, as in FWS). Dots at zero indicate accuracies below 2%.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Input mean $\mu$	Clean pass, per input channel
DAC range	Clean pass, per layer input
ADC range	Clean pass, per array, scaled offline for nominal drift
Drift gain $g(t)$	PCM only, probe readout at read time Joshi et al. (2020)
Reference column	ReRAM only, one per array, subtracted before the ADC
Weights	Programmed once at $t=0$ , never refreshed

Appendix

Table 6: What is calibrated, measured or refreshed.

Figure 16: Qwen3-1.7B first-token diagnostic. (a) Maximum absolute layer-2 projection inputs for the first and remaining tokens over four prompts. (b) WikiText NLL, the log perplexity, on PCM with 6-bit ADCs, 8-bit DACs and a one-day read age, showing the impact of executing the first token on the arrays.

Phi-3-mini
Method	Precision	$k$	MMLU	HellaSwag	ARC-C	ARC-E	Mean
Published results
FP	W16	–	69.4	83.5	84.6	91.1	82.1
Original	W16h	–	63.3	73.4	80.4	89.0	76.5
AFM	SI8-W16h-O8	–	65.1	80.6	83.2	89.8	79.7
LLM-QAT Liu et al. (2024)	SI8-W4h	–	60.1	74.8	77.9	87.2	75.0

Appendix

Table 7: AFM comparison: per-task accuracy (%) and the unweighted four-task mean.

k

multiplies the programming-noise standard deviation: 1.00 is nominal and 1.10 is adjusted to match the published noise-only baseline. Our SI8/O8 rows use one seed at

k=1.00

and ten at

k=1.10

, and noise-only rows use four Phi seeds and three/ten Llama seeds at

k=1.00/1.10

. Uncertainties are sample SDs. SI/DI denote static/dynamic inputs, and W16h/W4h denote the publication’s hardware-noisy weight formats.

Figure 17: Accuracy with one component removed from our method, on PCM with 4-bit ADCs at one day and ReRAM with 8-bit ADCs at five days. Vision models use 4,898 evaluation images and one seed.

Reference	Regular blocks	Plain blocks	Unmodified
Reference column	82.9	82.7	24.8
Fixed reference	82.7	0.0	0.1

Appendix

Table 8: ViT-B/16 top-1 accuracy (%) on CMO ReRAM with 8-bit ADCs five days after programming, with our method using regular or plain output blocks, on 4,898 evaluation images with one seed.

Model	Unmodified	Ours
ViT	5.7 / 21	0.66 / 2.8
ResNet	2.0 / 13	0.21 / 2.2
Qwen	7.9 / 22	0.92 / 2.4

Appendix

Table 9: Gain spread (%) of per-array compensation, PCM with 4-bit ADCs at one day: the relative difference between the gains it assigns to the arrays of one layer, as median / maximum over layers.

Figure 18: ViT-B/16 accuracy against the digital share of all analog and digital operations, on ReRAM with 8-bit ADCs at five days, with programming, read and drift noise scaled by

k

. The dashed line marks floating-point accuracy.

ADC	IR off	$1\times$	$1.5\times$	$2\times$
4	80.2/4.0	80.0/3.9	80.1/3.7	79.8/3.7
6	82.2/44.1	82.3/45.0	82.5/44.6	82.3/45.2
8	82.5/44.4	82.5/44.3	82.5/44.6	82.7/44.5

Appendix

Table 10: ViT-B/16 top-1 accuracy (%) on PCM at one day, as ours/unmodified, without IR drop and with IR drop at 1, 1.5 and 2 times the default wire resistance.

Model	Device	U	Q	O	FP
Llama-3.2-1B	PCM	24.6	26.1	41.0	50.1
	ReRAM	24.2	26.5	42.0
Qwen3-1.7B	PCM	23.9	31.3	47.2	59.0
	ReRAM	23.7	29.6	48.2
Phi-3-mini	PCM	25.7	27.3	60.2	68.4
	ReRAM	24.9	26.5	61.2

Appendix

Table 11: LLM mean accuracy (%) of the unmodified model (U), QuaRot (Q) and our method (O) with 4-bit ADCs, one day after programming. Phi models are scored on 512 questions per task for all three methods.

Device / configuration	1 s	1 h	1 day	5 days
PCM, pair + GDC	4.62 (3.32)	4.44 (2.55)	4.27 (2.06)	4.17 (1.86)
ReRAM 0.2% fixed ref.	5.67 (5.60)	4.41 (4.39)	4.06 (4.04)	3.91 (3.89)
ReRAM 2% fixed ref.	4.99 (4.40)	4.22 (3.95)	3.93 (3.74)	3.80 (3.63)
ReRAM 0.2% fixed ref. + GDC	5.69 (4.18)	4.48 (2.28)	4.17 (1.84)	4.04 (1.66)
ReRAM 2% fixed ref. + GDC	5.00 (4.07)	4.30 (2.38)	4.04 (1.92)	3.93 (1.73)
ReRAM 0.2% physical ref.	5.17 (5.14)	4.36 (4.32)	4.14 (4.00)	4.03 (3.86)

Appendix

Table 12: Effective precision

b_{\mathrm{eff}}

of continuous ViT-B/16 conductances versus read age. Parentheses give effective precision computed from the worst conditional RMS error over the conductance grid. GDC denotes global drift compensation.

Compute-in-Memory

Compute-in-memory (CIM) performs computation within or beside memory arrays to reduce data movement and accelerate AI workloads.

See all 59 papers

Robust Parameter-Efficient LLM Adaptation on Analog Hardware
Oct 4, 2026Jindan Li, Zhaoxian Wu, Tianyi ChenCompute-in-Memory LLM Fine-Tuning
Cornell University
Making Analog Training Scale: Co-Designing Mapping, Optimizer, and Converters
Sep 29, 2026Zhaoxian Wu, Tayfun Gokmen, Omobayode Fagbohungbe +2Mixed-Precision Training Analog Computing
Cornell Tech and Cornell University · IBM T. J. Watson Research Center · Sandia National Laboratories
Mitigating the Impact of Retention Loss on Inference Accuracy in 65 nm Single-Poly Floating-Gate Analog In-Memory Computing
Jul 27, 2026Mirko Brazzini, Giulio Filippeschi, Alessandro Catania +2Compute-in-Memory
Dipartimento di Ingegneria dell’Informazione, University of Pisa, Pisa, Italy · Quantavis SRL, Pisa, Italy

Oct 4, 2026J/K move · Enter open · S save

Robust Parameter-Efficient LLM Adaptation on Analog Hardware

Jindan Li, Zhaoxian Wu, Tianyi Chen

Cornell University

Compute-in-Memory LLM Fine-Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Analog in-memory computing is a promising platform for on-device execution of large language models because it performs matrix--vector multiplications (MVMs) in memory and in parallel, reducing data movement. However, limited digital-to-analog converter precision, input noise, and finite conductance states can degrade model accuracy, while full-model retraining to address these effects can be costly. We develop an optimizer-agnostic, parameter-efficient adaptation method based on Low-Rank Adaptation (LoRA), keeping the pretrained weights stored on analog arrays fixed while training the LoRA weights to adapt to downstream tasks and hardware non-idealities. Reliable adaptation requires handling errors in both forward and backward MVMs and physical weight updates. We use input reshaping to reduce input-induced MVM errors and update accumulation to retain small updates before programming them to finite-state analog devices. Across Llama-3.2-1B-Instruct and Llama-3-8B with both Muon and AdamW, input reshaping improves analog LoRA fine-tuning under noisy MVM computation. Update accumulation separately preserves sub-threshold updates and substantially improves adaptation under finite-resolution programming, including configurations with as few as 20 conductance states. Additional experiments show consistent held-out negative log-likelihood improvements across noisy analog settings.

Abstract

Compute-in-Memory

Explore similar work

Robust Parameter-Efficient LLM Adaptation on Analog Hardware

Making Analog Training Scale: Co-Designing Mapping, Optimizer, and Converters

Mitigating the Impact of Retention Loss on Inference Accuracy in 65 nm Single-Poly Floating-Gate Analog In-Memory Computing