Making Analog Training Scale: Co-Designing Mapping, Optimizer, and Converters

Zhaoxian Wu; Tayfun Gokmen; Omobayode Fagbohungbe; T. Patrick Xiao; Tianyi Chen

cs.LGSep 29, 2026

Making Analog Training Scale: Co-Designing Mapping, Optimizer, and Converters

Authors: Zhaoxian Wu, Tayfun Gokmen, Omobayode Fagbohungbe, T. Patrick Xiao, Tianyi Chen

Organizations: Cornell Tech and Cornell University · IBM T. J. Watson Research Center · Sandia National Laboratories

Abstract

Analog in-memory computing (AIMC) offers an alternative for model training by executing matrix operations directly where weights are stored. However, scaling AIMC to train modern deep models remains an open challenge due to severe hardware non-idealities, including physical weights with finite dynamic range and write granularity, analog-digital converters with finite resolution, and noisy and asymmetric updates. Guided by the insight that gradient accumulation is sensitive to precision and rounding errors, we adopt a mixed-precision training paradigm: executing forward and backward matrix multiplications in the analog domain while computing weight gradients in the digital domain. To enable scalable training, we present a holistic system-algorithm co-design that co-optimizes weight mapping to ensure well-conditioned physical and logical weight profiles, couples a preconditioned optimizer with threshold-triggered open-loop pulsing to stabilize training trajectories, and aligns converter dynamic ranges to suppress quantization errors. Evaluated via hardware-calibrated architectural simulations calibrated with electrochemical RAM measurements, our framework scales Transformer training up to $123\text{M}$ parameters with validation loss scaling as $L\propto N^{-0.231}$ , where $N$ is the parameter count, comparable to $L\propto N^{-0.238}$ for digital training.

Figures & tables

Figure 1: Scaling landscape of analog training, where each point is one work. Labels identify representative work.

Figure 2: Comparison of training partitions. Left: conventional digital mixed-precision training stores weights and executes the optimizer digitally, using reduced precision for selected arithmetic ( Micikevicius et al., 2018 ) . Middle: prior analog mixed-precision training ( Nandakumar et al., 2020 ) assigns MVMs to a DAC–crossbar–ADC path and uses a digital buffer to accumulate an SGD update before pulsed programming. Right: On top of the conventional analog mixed-precision design, our proposed architecture couples logical-to-conductance mapping with DAC/ADC range alignment (bound management, BM) at the MVM boundary, allowing modern logical-space optimizers such as Adam to stabilize the training process.

Figure 3: (a) Final validation loss of digital Shakespeare Transformer training with FP16 or FP8 weight, activation, or W-grad representations. (b) Cancellation factor

\kappa=\sum_{n}|g_{n}|/|\sum_{n}g_{n}|

of six projection types. (c) Relative gradient error versus cancellation strength when the accumulator is rounded after every addition under FP32, FP16, and FP8. (d) Single-step linear-layer update error at total batch size 128: green bars use mixed-precision (MP) digital-residual transfers at the indicated micro-batch sizes (#mbs), while the red bar uses direct rank updates. Bars and curves show four-seed means; whiskers and shaded bands span the observed seed min–max range.

Figure 4: Initialization maps and analog Transformer training on the Shakespeare dataset. (a, b) Signed conductance-deviation and logical-weight distributions for the fixed-logical-range control ( Nandakumar et al., 2020 ) , the native-AbsMax control ( Rasch et al., 2020 ) , and our proposed fixed-scale mapping. (c) Final validation loss across model depth for the same mapping-based controls.

Figure 5: Reconstruction distortion under Gaussian and Laplacian distributions. (a) and (b) show the effect of quantization precision. The gray bands mark precisions at which the max-aligned rail is within 10% of the optimal distortion. (c) and (d) show the effect of width.

Figure 6: Scaling and training trajectories for the five completed OpenWebText schedules following the Chinchilla rule, comparing matched digital and analog training on Li-ECRAM tiles. (a–c) Observed endpoints and preliminary power-law extrapolations fit

L=AX^{-\alpha}

against parameter count

N

, processed tokens, and the full-model algorithmic compute proxy, respectively. (d–e) Raw trajectories versus processed tokens and the shared compute proxy show the sample-efficiency and compute-normalized comparisons, respectively.

Scale	Params.	MpSGD	MpAdam	$+\mathrm{CA}$	$+\mathrm{init}$ (ours)	Digital Adam
Tiny	6.8M	10.7718	10.6035	6.8110	6.6075	6.3451
Small	16.0M	10.8545	10.5566	5.9143	5.6245	5.3033
Medium	29.9M	10.8293	10.7397	5.2073	4.8696	4.5499

Table 1: Ablation study from MpSGD to our recipe.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Work	Evidence	Task	Model	#analog wt.	Dataset/task	#tokens
Li et al. (2021)	H	F	Logic-gate net	$9$	Truth tables	$8$
Ando et al. (2025)	H	T	Binary classifier	$16$	MNIST	$5$ k
Chen et al. (2023a)	H	F	Bilayer MLP	$60$	Mushrooms	$81$ k
Ning et al. (2025)	H	F	Sparse NN	$900$	EMNIST L	$1.0$ k
Wang et al. (2019)	H	F	CNN/RNN	$1.0$ k	MNIST	$240$ k
Yao et al. (2020)	H	F	CNN	$2.8$ k	MNIST	$240$ k

Appendix

Table 2: Representative on-chip AIMC training studies. Evidence is H when the machine-learning task is executed with physical hardware and S for simulation only. Task is T for transfer learning from a pretrained model and F for training from scratch. Device characterization used to calibrate a simulation does not by itself qualify as H evidence. Studies reporting both forms of task-level evidence occupy separate H and S rows. The table includes only experiments in which all learned matrix-multiplication and convolution weights of the evaluated network are analog-mapped; experiments that retain any such network layers digitally and analog-map only selected layers are excluded. #analog wt. counts distinct logical weights mapped to analog arrays for the stated workload, whether trainable or fixed; digitally held non-matrix parameters, unused capacity, differential-device multiplicity, auxiliary/reference arrays, and replicated mappings are excluded. “Dataset/task” gives the largest qualifying machine-learning workload with a determinable analog-mapped weight count. A superscript

*

after a work indicates that its paper also reports a larger-scale experiment that leaves one or more learned linear layers digital; it is excluded to preserve a fair full-network analog-mapping comparison. EN denotes an equilibrium net.

Figure 7: Reduction of bound-management MVM retries through output-rail alignment.

Model	# blocks	Width	# heads	# parameters	Updates	Final validation step
Tiny	2	128	4	6.83M	278	273
Small	4	256	4	16.01M	652	640
Medium	6	384	6	29.92M	1,218	1,200
Large	8	512	8	50.91M	2,072	2,060
XLarge	12	768	12	123.55M	5,028	5,020

Appendix

Table 3: Decoder-only GPT model specifications and training schedules for the scaling-law simulations.

	Task accuracy $\uparrow$										Avg $\uparrow$
Model	HSw	PIQA	ARCe	ARCc	OBQA	Wino	CSQA	COPA	SciQ	WSC	Avg $\uparrow$
Digital Tiny	29.7	57.0	29.7	23.4	21.9	47.7	18.8	53.0	35.9	36.5	35.4
Digital Small	28.9	61.7	33.6	22.7	17.2	50.0	21.1	49.0	43.0	59.6	38.7
Digital Medium	27.3	58.6	33.6	18.8	21.9	49.2	27.3	52.0	48.4	60.6	39.8
Digital Large	26.6	60.2	33.6	21.1	18.0	50.8	25.0	54.0	39.8	63.5	39.3
Digital Xlarge	28.9	64.1	35.9	21.1	21.1	50.8	21.9	59.0	46.1	41.3	39.0

Appendix

Table 4: Zero-shot commonsense and multiple-choice reasoning screening accuracy (%, length-normalized conditional likelihood). Digital AdamW checkpoints appear first, followed by separately trained analog checkpoints. Each task uses 128 fixed-seed labelled examples where available, and otherwise its complete labelled evaluation split (COPA: 100 and WSC: 104 examples). HSw: HellaSwag; ARCe/ARCc: ARC-Easy/ARC-Challenge; OBQA: OpenBookQA; Wino: WinoGrande; CSQA: CommonsenseQA. Avg is the unweighted average across the ten displayed tasks.

	Task accuracy $\uparrow$												Avg $\uparrow$
Model	AG	Bool	CB	CoLA	MNLI	MRPC	QNLI	QQP	RTE	SST-2	WiC	WNLI	Avg $\uparrow$
Digital Tiny	18.8	39.8	41.1	46.1	35.9	39.1	47.7	51.6	52.3	58.6	49.2	56.3	44.7
Digital Small	21.1	42.2	41.1	43.0	38.3	35.9	46.1	39.8	52.3	57.0	48.4	62.0	43.9
Digital Medium	21.9	40.6	41.1	41.4	38.3	32.8	56.2	68.0	51.6	64.8	46.1	59.2	46.8
Digital Large	25.0	40.6	41.1	40.6	40.6	32.8	47.7	65.6	51.6	60.9	46.1	56.3	45.7
Digital Xlarge	22.7	56.2	41.1	39.1	38.3	33.6	58.6	60.2	52.3	57.0	46.1	63.4	47.4

Appendix

Table 5: Complementary zero-shot language-understanding screening accuracy (%, length-normalized conditional likelihood). Digital AdamW checkpoints appear first, followed by separately trained analog checkpoints. Each task uses 128 fixed-seed labelled examples where available, and otherwise its complete labelled evaluation split (CB: 56 and WNLI: 71 examples). Avg is the unweighted average across the 12 displayed tasks.

Preset device	$\mathrm{dw}_{\min}$	$\mathrm{up\_down}$	$w_{\max}$	$w_{\min}$	$\gamma_{\mathrm{up}}$	$\gamma_{\mathrm{down}}$	# states
ReRamES Gong et al. (2022)	0.00135	0.259359	1	-1	5	5	1481.5
ReRamSB Gong et al. (2022)	0.002	0	1.25	-0.75	–	–	1000
Capacitor Li et al. (2018b)	0.005	0	1	-1	0.05	0.05	400
Li-ECRAM Tang et al. (2018)	0.002	0	1.1724	-0.8276	0.1153	0.5085	1000
MO-ECRAM Kim et al. (2019b)	0.00028214	0	1.1714	-0.8286	0.4152	0.7342	7088.7
PCM Nandakumar et al. (2019)	0.01	0	2	0	2.5	2.5	200

Appendix

Table 6: AIHWKit SingleRPUConfig presets fitted to measured physical devices or arrays: primary update and conductance parameters. # states denotes the nominal ratio

(w_{\max}-w_{\min})/\mathrm{dw}_{\min}

. The citation in each row identifies the source measurement.

Preset device	$\mathrm{dw}_{\min,\mathrm{d2d}}$	$\mathrm{up\_down}_{\mathrm{d2d}}$	$w_{\max,\mathrm{d2d}}$	$w_{\min,\mathrm{d2d}}$	$\gamma_{\mathrm{up,d2d}}$	$\gamma_{\mathrm{down,d2d}}$	$\mathrm{dw}_{\min,\mathrm{std}}$	$\mathrm{write\_noise}_{\mathrm{std}}$
ReRamES Gong et al. (2022)	0.2	0.05	0.3	0.3	–	–	5	75
ReRamSB Gong et al. (2022)	0.3	0.01	0.24	0.4	–	–	3.75	56
Capacitor Li et al. (2018b)	0.1	0.06	0.07	0.07	0.01	0.01	0.3	0
Li-ECRAM Tang et al. (2018)	0.1	0.01	0.05	0.05	0.05	0.05	0.3	0
MO-ECRAM Kim et al. (2019b)	0.1	0.01	0.05	0.05	0.05	0.05	2	0
PCM Nandakumar et al. (2019)	0.2	0.05	0.1	0	–	–	0.6	0

Appendix

Table 7: AIHWKit SingleRPUConfig presets fitted to measured physical devices or arrays: device and cycle variation parameters. The citation in each row identifies the source measurement.

Figure 8: Response curves for all measured-device SingleRPUConfig presets in Tables 6 – 7 . Each panel shows five simulated traces under 1,000 potentiation pulses (pale blue) followed by 1,000 depression pulses (pale orange). The dashed line marks the reversal.

Device	L2	L4	L6	L8
Digital	2.2276	2.0239	1.9006	1.8422
SoftBounds	2.3302	2.0935	1.9733	1.8949
ReRamES	2.4486	2.3741	2.2386	2.1687
ReRamSB	2.4782	2.4455	2.3214	2.2486
Capacitor	2.3305	2.0934	1.9517	1.8699
Li-ECRAM	2.3139	2.0665	1.9424	1.8637

Appendix

Table 8: Final Shakespeare validation loss after 5,000 iterations for method S and the matched Digital baseline, across all general RPU device models. Lower is better; each cell is one seed.

Device	L2	L4	L6	L8
Digital	2.2276	2.0239	1.9006	1.8422
SoftBounds	2.4357	2.3168	2.2038	2.1422
ReRamES	2.5540	2.4946	2.4970	2.4409
ReRamSB	2.5309	2.5390	2.4911	2.4616
Capacitor	2.5324	2.5215	2.4885	2.5088
Li-ECRAM	2.4887	2.4689	2.3347	2.2591

Appendix

Table 9: Final Shakespeare validation loss after 5,000 iterations for method SFB and the matched Digital baseline, across all general RPU device models. Lower is better; each cell is one seed.

Panel of Figure 9 — Figure 9: Final Shakespeare validation perplexity after 5,000 updates for S under device variation, cycle variation, input noise and output noise, with one curve per transformer depth and all other settings held fixed. Device variation jointly changes the device variation of the conductance limits and minimum update step; cycle variation changes the cycle update-step standard deviation. Input- and output-noise values are matched between the forward and backward directions and are labelled as standard deviations relative to the simulator DAC and ADC bounds, respectively. The input-noise panel ends at 8% (the first evaluated level above 7.5%) so that the two noise panels share a common ordinate. Dashed vertical lines identify the corresponding AIHWKit default: 30% for device and cycle variation, zero input noise, and 0.5% of the ADC bound for output noise. The gray background marks the common prefix in which all four depths remain within 10% of their respective zero-perturbation final perplexities. Each condition uses one fixed data and device-construction seed, so the curves show observed sensitivity in this simulator configuration rather than uncertainty estimates.

Figure 10: Final Shakespeare validation loss after 5,000 iterations for method S under converter-resolution sweeps. Rows encode forward bits and columns backward bits; lower loss is better. The ADC panel sweeps the forward and backward output resolution while holding DAC at its default resolution, and the DAC panel analogously sweeps input resolution while holding ADC at its default resolution.

Bits	L2	L4	L6	L8
3	2.9552	2.9166	2.9166	2.8549
4	2.5548	2.5036	2.4457	2.3802
5	2.4658	2.4212	2.2929	2.2218
6	2.4379	2.3800	2.2233	2.1778
7	2.4362	2.3370	2.2036	2.1418
8	2.4212	2.3420	2.1939	2.1389

Appendix

Table 10: Final Shakespeare validation loss after 5,000 iterations for SFB with matched DAC and ADC resolution.

Pure-digital initialization controls
Profile	$L=2$	$L=4$	$L=6$	$L=8$
Digital	2.2276	2.0238	1.9006	1.8422
Digital-I	2.2355	2.0073	1.9018	1.8258
Original stack and factorization of S (analog)
MpSGD	3.4444	3.4605	3.4571	3.4403
Native-map Adam ( $\omega_{\mathrm{map}}=1$ )	2.4576	2.4279	2.3902	2.3472

Appendix

Table 11: Integrated profile ablation: validation loss after 5,000 updates (lower is better). All entries use seed 1337 and evaluations averaged over 200 held-out mini-batches. All analog profiles retain a digital positional embedding; the definitions and interventions are given in the surrounding text.

Optimizer	Profile	$L=2$	$L=4$	$L=6$	$L=8$
AdamW	Digital	2.2355	2.0073	1.9018	1.8258
	S	2.3422	2.0925	1.9696	1.8999
	SFB	2.4419	2.3393	2.2098	2.1432
Muon	Digital	2.0141	1.8948	1.7922	1.7252
	S	2.3053	2.0798	1.9650	1.8956
	SFB	2.3988	2.2972	2.1825	2.1384

Appendix

Table 12: Optimizer screen under matched digital, S, and SFB profiles: final Shakespeare validation loss after 5,000 updates (lower is better). Rows are grouped by optimizer. Digital uses the same width-stable initialization as S/SFB but no AIMC conversion. All cells use seed 1337 and evaluation over 200 held-out mini-batches.

Analog Computing

See all 24 papers

Dynamic Symmetric Point Tracking: Tackling Non-ideal Reference in Analog In-memory Training
Feb 24, 2026Quan Xiao, Jindan Li, Zhaoxian Wu +2Compute-in-Memory Analog Computing
Department of Electrical and Computer Engineering, Cornell University, New York, NY · IBM T. J. Watson Research Center, Yorktown Heights, NY · Rensselaer Polytechnic Institute, Troy, NY
Robust Parameter-Efficient LLM Adaptation on Analog Hardware
Oct 4, 2026Jindan Li, Zhaoxian Wu, Tianyi ChenCompute-in-Memory LLM Fine-Tuning
Cornell University
IMC-CLINIC: Coupled Loss-Informed Newton Iterations for Clipping in Analog In-Memory Computing
Sep 28, 2026Yung-Chin Chen, Chia-Yu Chen, Naveen VermaLLM Quantization Compute-in-Memory
Princeton University, NJ, USA · EnCharge AI, CA, USA

Feb 24, 2026J/K move · Enter open · S save

Dynamic Symmetric Point Tracking: Tackling Non-ideal Reference in Analog In-memory Training

Quan Xiao, Jindan Li, Zhaoxian Wu, Tayfun Gokmen+1

Department of Electrical and Computer Engineering, Cornell University, New York, NY · IBM T. J. Watson Research Center, Yorktown Heights, NY · Rensselaer Polytechnic Institute, Troy, NY

Compute-in-Memory Analog Computing

Open paper AskPDF arXiv

Analog in-memory computing (AIMC) performs computation directly within resistive crossbar arrays, offering an energy-efficient platform to scale large vision and language models. However, non-ideal analog device properties make the training on AIMC devices challenging. In particular, its update asymmetry can induce a systematic drift of weight updates towards a device-specific symmetric point (SP), which typically does not align with the optimum of the training objective. To mitigate this bias, most existing works assume the SP is known and pre-calibrate it to zero before training by setting the reference point as the SP. Nevertheless, calibrating AIMC devices requires costly pulse updates, and residual calibration error can directly degrade training performance. In this work, we present the first theoretical characterization of the pulse complexity of SP calibration and the resulting estimation error. We further propose a dynamic SP estimation method that tracks the SP during model training, and establishes its convergence guarantees. In addition, we develop an enhanced variant based on chopping and filtering techniques from digital signal processing. Numerical experiments demonstrate both the efficiency and effectiveness of the proposed method.

Abstract

Analog Computing

Explore similar work

Dynamic Symmetric Point Tracking: Tackling Non-ideal Reference in Analog In-memory Training

Robust Parameter-Efficient LLM Adaptation on Analog Hardware

IMC-CLINIC: Coupled Loss-Informed Newton Iterations for Clipping in Analog In-Memory Computing