Analog In-Memory Computing (IMC) is a promising paradigm for accelerating GEMM-based AI workloads by executing computations directly within the memory array in the analog domain. While IMC delivers substantial energy and throughput improvements, it can suffer from significant task accuracy degradation due to analog-to-digital converter (ADC) quantization noise, which occurs at the GEMM output and is thus different from weight/activation quantization, which occurs at the GEMM input. Mixed ADC resolution allows IMC modules to choose between high- and low-resolution ADC modes with different energy costs, offering a compelling accuracy-energy trade-off. However, selecting ADC resolutions across IMC units while respecting hardware granularity constraints remains challenging. In this paper, we propose SMART-AIMC, a Sensitivity-aware Mixed ADC Resolution framework with fine-grained Tunability for Analog In-Memory Computing. SMART-AIMC correlates the algorithmic task loss to hardware ADC quantization noise in the form of an ADC Channel Sensitivity (ACS) metric. Using this metric, SMART-AIMC systematically, and in a disciplined manner, allocates ADC resources to critical computations. Furthermore, we introduce a sensitivity-based channel reordering scheme that overcomes the hardware granularity constraints with no runtime overhead and approaches the optimal ADC quantization possible for the model. Evaluated across diverse workloads, SMART-AIMC unlocks a superior accuracy-energy Pareto frontier, providing (1) higher task accuracy at matched energy budgets and (2) fine-grained tuning across efficiency targets, from aggressive energy savings to near-digital inference accuracy. At the latter operating point, SMART-AIMC achieves 6.99× higher modeled system-level energy efficiency for linear projections than the INT8 digital baseline with negligible accuracy degradation.
Figures & tables
Figure 1: IMC architecture, signal swing reduction, and ADC resolution-energy trade-off.
Figure 2: Variations in ADC sensitivity. Sensitivity to ADC quantization varies significantly not only across decoder blocks, but also among individual channels, necessitating a systematic approach for optimal compute-to-ADC resolution mapping. The experiment setting is detailed in Appendix H .
Figure 3: Channel reordering for ADC allocation. Without reordering, high-sensitivity channels can be inadvertently mapped to low-resolution IMC units. Channel reordering strictly ensures that the high-sensitivity channels are mapped to high-resolution ADCs. The setting is detailed in Appendix H .
Figure 4: Sensitivity-aware channel reordering across a decoder layer. Output channels with similar sensitivity are grouped to reduce IMC unit-level mixed-resolution loss during ADC quantization.
Figure 5: IMC accelerator architecture. Static-weight linear projections are mapped to analog IMC, while attention matrix multiplications and vector operations are executed on digital engines.
Figure 6: Smart-AIMC ADC-resolution allocation. Smart-AIMC identifies distinct sensitivities across projections and blocks to optimize fine-grained ADC assignments under a budget of 20% 12-bit and 80% 9-bit ADCs.
Figure 7: Smart-AIMC fidelity–energy efficiency Pareto analysis. KL divergence and perplexity versus system-level energy-efficiency gain for linear projections over digital INT8 for LLaMA-2-7B on WikiText-2. Dashed lines show digital INT8 fidelity; lower fidelity metrics and higher gain are better. Smart-AIMC establishes a superior frontier over the heuristic and random baselines.
Method
LLaMA-2-7B
LLaMA-3.2-3B
Mistral-7B-v0.3
Qwen2-1.5B-Instruct
KL ( ↓ )
PPL ( ↓ )
Avg. Acc. ( ↑ )
TOPS/W ( ↑ )
KL ( ↓ )
PPL ( ↓ )
Avg. Acc. ( ↑ )
TOPS/W ( ↑ )
KL ( ↓ )
PPL ( ↓ )
Avg. Acc. ( ↑ )
TOPS/W ( ↑ )
KL ( ↓ )
PPL ( ↓ )
Avg. Acc. ( ↑ )
TOPS/W ( ↑ )
INT8 Baseline
0.0015
5.47
66.64
2.42
0.0024
7.82
64.99
2.41
0.0008
5.32
71.58
2.42
0.0123
10.12
60.76
2.40
Random Alloc.
1.0535
11.94
47.93
22.82
2.4698
73.01
40.63
22.58
0.5423
8.63
57.45
23.32
1.3563
32.22
42.80
21.91
Heuristic Alloc.
0.3510
7.33
59.24
22.82
0.7756
16.78
50.81
22.58
0.2946
6.91
64.22
23.32
0.9115
21.57
47.92
21.91
Smart-AIMC
0.2917
7.01
59.91
22.82
0.6762
14.92
52.37
22.58
0.2642
6.76
64.85
23.32
0.7791
19.39
47.72
21.91
Table 1: LLM quality and system-level energy efficiency for linear projections across four models under the matched O+V budget. Bold values denote the best quality among mixed-ADC allocation schemes.
Method
ViT-Small
ViT-Base
ViT-Large
KL ( ↓ )
Acc. ( ↑ )
KL ( ↓ )
Acc. ( ↑ )
KL ( ↓ )
Acc. ( ↑ )
INT8 Baseline
0.0011
79.80
0.0004
81.80
0.0006
83.40
O+V budget
Random Alloc.
2.1804
40.34
0.2754
78.18
6.5593
0.10
Heuristic Alloc.
2.2133
41.60
0.2169
79.90
6.5754
0.10
Smart-AIMC
0.5863
68.30
0.0964
80.90
6.1645
2.30
Table 2: ImageNet quality under the O+V and O+Down mixed-ADC budgets. Bold values denote the best mixed-ADC result for each model and budget.
Method
KL ( ↓ )
PPL ( ↓ )
Avg. Acc. ( ↑ )
TOPS/W ( ↑ )
INT8 Baseline
0.0015
5.47
66.64
2.42
Smart-AIMC
0.0224
5.57
65.84
16.93
Table 3: LLaMA-2-7B quality and system-level energy efficiency for linear projections at the higher-ADC-budget operating point.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: ADC quantization noise distributions. Both 9-bit and 12-bit ADC configurations exhibit near-uniform quantization noise, aligning with the statistical theory of quantization Widrow et al. (1996) .
Figure 9: Output channel independence. Both 9-bit and 12-bit ADC configurations exhibit near-zero correlation of ADC quantization noise between output channels.
Figure 10: Input IMC unit independence. Both 9-bit and 12-bit ADC configurations exhibit near-zero correlation of ADC quantization noise between input IMC units.
Table 4: 2-bit encoding examples for 0/1 and −1/1 representations.
Category
Operations and Associated Costs
IMC-Mapped Operations
Q, K, V, O, Gate, Up, and Down, including input/output precision conversion, weight accesses, partial-sum accumulation, and associated data movement.
Other Operations
Attention BMMs, RMSNorm, RoPE, softmax, SwiGLU, residual additions, and their associated costs.
Appendix
Table 5: Operator-based attribution of system-level energy for linear projections.
Figure 11: Hierarchical energy breakdown across ADC resolutions (9–12 bits). (a) Energy consumption of the IMC and non-IMC execution paths, with their respective operation counts. (b) Component-level breakdown of Eproj . (c) IMC macro energy breakdown, showing that ADC-related energy is the primary resolution-dependent component. All energies are reported per decoder block.
Permutations
Scope
Dimension
Sensitivity Metric
PA
global
dout
O + Down
PB
per-block †
dhead
V
PC
per-block †
dhead/2
Q + K
PD
per-block †
dout
Gate + Up
Appendix
Table 6: Permutation configurations for a RoPE-based SwiGLU LLM architecture.
Figure 12: Channel reordering is compatible with rotation methods, allowing optimizations on both input quantization and ADC quantization.
Method
ARC-c
ARC-e
BoolQ
HellaSwag
PIQA
OBQA
Winogrande
Avg.
LLaMA-2-7B
INT8 Baseline
45.05
73.74
79.48
76.13
78.78
44.0
69.30
66.64
Random Alloc.
27.05
45.16
59.63
53.45
65.89
29.40
54.93
47.93
Heuristic Alloc.
38.48
64.44
68.96
69.46
73.99
38.60
60.77
59.24
Smart-AIMC
39.76
65.82
70.12
69.64
73.72
38.20
62.12
59.91
LLaMA-3.2-3B
Appendix
Table 7: Full zero-shot accuracy (%) on seven common-sense reasoning tasks under the matched O+V energy budget. Bold values denote the best reported mixed-ADC allocation result for each model and task.
Method
LLaMA-2-7B
LLaMA-3.2-3B
Mistral-7B-v0.3
Qwen2-1.5B-Instruct
KL
PPL
Avg. Acc.
TOPS/W
KL
PPL
Avg. Acc.
TOPS/W
KL
PPL
Avg. Acc.
TOPS/W
KL
PPL
Avg. Acc.
TOPS/W
INT8 Baseline
0.0015
5.47
66.64
2.42
0.0024
7.82
64.99
2.41
0.0008
5.32
71.58
2.42
0.0123
10.12
60.76
2.40
Random Alloc.
0.5114
8.42
54.04
20.30
1.0914
21.69
48.40
19.90
0.3120
7.01
63.00
20.28
0.6594
18.54
49.56
18.92
Heuristic Alloc.
0.1531
6.26
63.17
20.30
0.3389
10.71
57.78
19.90
0.1397
6.04
67.75
20.20
0.4578
14.97
52.13
18.92
Smart-AIMC
0.1374
6.16
63.14
20.30
0.3048
10.54
57.17
19.90
0.1178
5.91
67.79
20.28
0.3032
12.99
53.87
18.92
Appendix
Table 8: Summary of LLM quality and system-level energy efficiency for linear projections under the matched O+Down+V energy budget, defined by assigning high-resolution ADCs to the attention output, down, and value projections in the heuristic baseline. KL denotes KL divergence, PPL denotes WikiText-2 perplexity, and average accuracy is the unweighted mean over seven common-sense reasoning tasks. Lower KL and PPL are better; higher average accuracy and TOPS/W are better. Bold values denote the best reported quality among mixed-ADC allocation schemes at the matched energy budget.
Allocation
Reordering
PPL ( ↓ )
Allocation pipelines
Heuristic
None
7.3283
Activation magnitude
Activation magnitude
13.5037
Hessian-only
Hessian-only
7.5437
ACS
ACS
7.0130
Reordering with ACS allocation
Appendix
Table 9: Sensitivity metric and channel-reordering ablations on LLaMA-2-7B under the O+V budget with 9/12-bit ADCs.
Configuration
Time (s)
GPU (GiB)
PPL ( ↓ )
A. Calibration sample count
8
9.9
31.4
7.0270
16
19.3
31.4
7.0113
32
37.9
31.4
7.0065
64 (default)
74.9
31.4
7.0138
128
150.0
31.4
7.0185
Appendix
Table 10: Calibration efficiency and robustness of Smart-AIMC on LLaMA-2-7B. GPU denotes peak allocated GPU memory.
ADC Bits
Smart-AIMC
ACS w/o Reord.
Heuristic
Random
9/12
7.0130
7.0445
7.3283
11.9438
9/11
7.1411
7.1773
7.4508
12.5052
9/10
7.7380
7.7748
8.0997
13.2116
8/12
309.8581
361.5514
634.2484
6,082.6392
8/11
333.8069
392.6672
689.6928
6,197.6001
8/10
476.3950
543.5512
892.2458
6,279.1348
Appendix
Table 11: Binary ADC resolution configurations on LLaMA-2-7B under the O+V allocation budget. The ACS w/o reordering variant uses ACS allocation without channel reordering. Lower PPL is better.
ADC Bits
PPL ( ↓ )
9/12
7.0130
9/10/12
5.8338
9/11/12
5.8462
Appendix
Table 12: Three-level ADC resolution configurations on LLaMA-2-7B at the same physical ADC-energy budget as the binary 9/12-bit O+V setting. Lower PPL is better.
Dimension
Random
Hessian-only
Smart-AIMC
Column sweep ( h=512 )
w=8
11.8732
7.5672
6.9997
w=16
11.9124
7.5616
7.0162
w=32
11.9438
7.5672
7.0130
w=64
12.2049
7.5709
7.0144
w=128
12.0034
7.5642
7.0233
Appendix
Table 13: Full WikiText-2 PPL on LLaMA-2-7B under different IMC macro dimensions using 9/12-bit ADCs and the O+V physical ADC-energy budget. NaN indicates numerical collapse. Lower PPL is better.
ADC Budget
Random
Heuristic
Smart-AIMC
O+V
11.9438
7.3283
7.0130
O+Down+V
8.1960
6.2600
6.1584
Appendix
Table 14: Full WikiText-2 PPL on LLaMA-2-7B under different physical ADC-energy budgets using 9/12-bit ADCs. Lower PPL is better.
Noise
Allocation Scheme
PPL ( ↓ )
Avg. Acc. ( ↑ )
0.1
Random Alloc.
14.80
44.01
Heuristic Alloc.
7.72
57.20
Smart-AIMC
7.31
58.26
0.2
Random Alloc.
42.00
39.25
Heuristic Alloc.
9.36
54.13
Smart-AIMC
8.47
54.26
Appendix
Table 15: Robustness to analog noise on LLaMA-2-7B under the O+V ADC budget. Noise magnitude is measured in ADC LSB RMS.
Model
License
Source
LLaMA-2-7B Touvron et al. (2023)
Llama 2 Community
[HuggingFace]
LLaMA-3.2-3B Meta AI (2024)
Llama 3.2 Community
[HuggingFace]
Mistral-7B-v0.3 Mistral AI (2024)
Apache 2.0
[HuggingFace]
Qwen2-1.5B-Instruct Yang et al. (2024)
Apache 2.0
[HuggingFace]
Appendix
Table 16: Documentation of pretrained language models used in this study.
Analog in-memory computing (AIMC) speeds up neural-network inference by doing the arithmetic directly inside a memory array, instead of shuttling weights back and forth between memory and a processor. This saves energy, but the physical devices that store the weights are imperfect: programming errors, electrical noise, limited-resolution converters, and outright broken cells all distort the computation, and every physical chip is distorted in its own way. A designer with several such chips available faces an uncomfortable choice: run all of them and combine the answers (safe, but wasteful of energy), or trust a single chip blindly (cheap, but with no guarantee on how often it is wrong). This paper introduces RACE-AIMC (Risk-Aware Certified Ensemble for AIMC), a framework that resolves this choice with statistics rather than guesswork. Offline, RACE-AIMC studies a pool of physical accelerators, picks the single best one for a given energy budget, and computes a mathematically exact upper bound on how often that accelerator will be wrong when it chooses to answer. Online, only that one accelerator is switched on; a lightweight check decides whether to accept its answer or defer to a fallback. In our simulations using a noisy weight mapping and multiple independent test runs, every certified bound stayed under a 10% error target (mean bound 7.83% +- 0.89%, with 70.88% +- 0.98% of inputs answered directly). The resulting system matches the accuracy of a clean digital baseline while cutting modeled energy use by 69.02% relative to always running every accelerator in the pool.
Analog in-memory computing (IMC) offers a promising path toward energy-efficient large language model (LLM) inference by executing matrix multiplications (MatMul) directly within memory arrays in the analog domain. Its efficiency, however, comes with an additional source of error: limited-precision analog-to-digital converters (ADCs) quantize accumulated analog partial sums, introducing output-side error distinct from conventional activation and weight quantization at the MatMul inputs. Clipping can mitigate both operand and ADC quantization errors, but the optimal clipping factors must jointly balance activation rounding and clipping, weight rounding and clipping, and ADC quantization. Existing clipping methods, designed for digital quantization, do not explicitly optimize these coupled sources of IMC error and often rely on costly search-based calibration. We introduce IMC-CLINIC (Coupled Loss-Informed Newton Iterations for Clipping), a clipping calibration framework based on an analytical surrogate for IMC MatMul output error. The surrogate jointly models operand quantization, accumulated clipping-induced bias, and ADC quantization, enabling efficient evaluation of its gradient and approximate curvature from a small calibration set. IMC-CLINIC jointly optimizes activation and weight clipping factors using a safeguarded Newton-type method. Across multiple models and datasets, it improves average zero-shot accuracy by 6.5-11.5 percentage points over the grid search baseline while reducing calibration time by factors of 10.0-12.1. Its analytical surrogate closely tracks empirical IMC output error, and its optimizer is certified within 1% of the global optimum under the loss objective across all projections on two representative models.
Yung-Chin Chen, Chia-Yu Chen, Naveen Verma
Princeton University, NJ, USA · EnCharge AI, CA, USA
Analog compute-in-memory (CIM) enables energy-efficient neural network inference, but device variation and read noise can severely degrade low-bit quantized models. Existing CIM-oriented quantization methods mainly minimize ideal quantization error, ignoring the hardware noise floor and thus causing inefficient precision allocation. We propose NANQ, a noise-aware mixed-precision non-uniform quantization framework for analog CIM. NANQ models magnitude-dependent weight noise from measured responses of an eFlash CIM array and converts the noise profile into an adaptive quantization density, assigning finer resolution to low-noise regions while avoiding ineffective precision in noise-dominated regions. It further assigns layer-wise bit-widths by identifying each layer's precision saturation point under hardware noise using a unified threshold. On-chip experiments on an eFlash CIM SoC show that, under 2-bit weight-magnitude quantization, NANQ improves vision-model accuracy by 8.05 percentage points and reduces language-model PPL by 54.7% on average over PowerQuant. Mixed-precision NANQ captures most of the gains obtainable from additional quantization resources with only 3.2-3.8 equivalent bits.
Yizhe Chen, Wenshuai Yao, Saiya Wang +6
School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China · School of Integrated Circuits, Peking University, Beijing, China · Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China