Robust Heterogeneous Analog-Digital Computing for Mixture-of-Experts Models with Theoretical Generalization Guarantees

Mohammed Nowaz Rabbani Chowdhury; Hsinyu Tsai; Geoffrey W. Burr; Kaoutar El Maghraoui; Liu Liu; Meng Wang

cs.LGMar 3, 2026

Robust Heterogeneous Analog-Digital Computing for Mixture-of-Experts Models with Theoretical Generalization Guarantees

Authors: Mohammed Nowaz Rabbani Chowdhury, Hsinyu Tsai, Geoffrey W. Burr, Kaoutar El Maghraoui, Liu Liu, Meng Wang

Organizations: Rensselaer Polytechnic Institute · IBM Research

Abstract

Sparse Mixture-of-Experts (MoE) models enable efficient scalability by activating only a small sub-set of experts per input, yet their massive parameter counts lead to substantial memory and energy inefficiency during inference. Analog in-memory computing (AIMC) offers a promising solution by eliminating frequent data movement between memory and compute units. However, mitigating hardware nonidealities of AIMC typically requires noise-aware retraining, which is infeasible for large MoE models. In this paper, we propose a retraining-free heterogeneous computation framework in which noise-sensitive experts, which are provably identifiable by their maximum neuron norm, are computed digitally while the majority of the experts are executed on AIMC hardware. We further assign densely activated modules, such as attention layers, to digital computation due to their high noise sensitivity despite comprising a small fraction of parameters. Extensive experiments on large MoE language models, including DeepSeekMoE and OLMoE, across multiple benchmark tasks validate the robustness of our approach in maintaining accuracy under analog nonidealities.

Figures & tables

Figure 1: a. In heterogeneous computing of MoE, the dense modules and noise-sensitive expert modules are computed in a digital accelerator while rest of the experts are computed in an analog accelerator. b. A schematic of digital and analog accelerator. c. The analog accelerator is comprised of non-volatile memory (NVM) tiles. Weights are programmed to the crossbar array of the tile.

Panel of Figure 2 — Figure 2: The heterogeneous computation strategy for MoE models.

Model	Noise	Modules	Tasks								Avg.
			PIQA	ARC-e	ARC-c	BoolQ	HellaS.	Wino.	MathQA	MMLU
DeepSeek- MoE	Digital (FP-16)	—	80.36	72.85	47.61	72.78	77.38	70.17	31.42	37.67	61.28
	DAC-ADC	Experts	79.16	73.40	47.44	72.57	76.78	69.77	31.22	37.92	61.03
	DAC-ADC	Experts+Dense	75.68	68.06	39.16	68.29	68.58	60.69	28.17	31.71	55.04
OLMoE	Digital (FP-16)	—	79.71	76.94	49.66	69.85	78.19	68.90	28.44	53.65	63.17
	DAC-ADC	Experts	79.49	76.43	49.49	69.02	77.34	67.72	28.98	51.22	62.46

Table 1: Accuracy (%) of MoE models with DAC–ADC noise added to the inputs and outputs of different modules.

Panel of Figure 3 — Figure 3: Effect of computing dense modules in analog. Accuracy degradation is significant when densely activated modules are executed on AIMC, despite their small parameter footprint.

Panel of Figure 4 — Figure 4: Performance of different digital expert selection methods in OLMoE

Panel of Figure 5 — Figure 5: Performance of different digital expert selection methods in DeepSeekMoE

Param.	Modules	Throughput	Energy Efficiency	Avg. Accuracy (%) vs. Prog. noise magnitude
in Digital	in Digital	(Tokens/s)	(Tokens/Watt $\cdot$ s)	1.0	1.5	2.5
100% (FP-16)	—	4220.07	10.55	63.17
0% (analog)	None	768.41	23949.07	61.05 $\pm$ 0.10	58.45 $\pm$ 0.15	49.74 $\pm$ 0.23
5.37% (het.gen)	Dense	49781.23	123.92	62.02 $\pm$ 0.07	60.73 $\pm$ 0.10	56.71 $\pm$ 0.15
17.01% (het.gen)	Dense + 12.5% experts	24924.12	62.20	62.38 $\pm$ 0.04	61.49 $\pm$ 0.06	58.60 $\pm$ 0.12
28.65% (het.gen)	Dense + 25.0% experts	14513.52	36.25	62.50 $\pm$ 0.04	61.82 $\pm$ 0.06	59.60 $\pm$ 0.08

Table 2: The throughput, energy efficiency, and accuracy of OLMoE evaluated for batch size 32.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

$\kappa$	10	18	25	30	35	40	45	50
PPL	10.98	7.59	7.10	7.00	6.97	6.99	7.02	7.07

Appendix

Table 3:

\kappa

vs. Perplexity (PPL) for OLMoE. Noise added only at the experts.

$\lambda$	0.75	0.9	1.0	1.125	1.25	1.5	1.75	2.0	2.25	2.5	2.75
PPL	7.15	7.10	7.10	7.11	7.13	7.19	7.26	7.32	7.42	7.48	7.57

Appendix

Table 4:

\lambda

vs. Perplexity (PPL) for OLMoE (

\kappa=35

). Noise added only at the experts.

$\kappa$	10	15	20	25	30	35	40	45
PPL	22.26	8.66	7.50	7.16	7.07	7.07	7.12	7.18

Appendix

Table 5:

\kappa

vs. Perplexity (PPL) for OLMoE. Noise added to both experts and dense modules.

$\lambda$	0.75	1.0	1.25	1.5
PPL	7.42	7.26	7.28	7.38

Appendix

Table 6:

\lambda

vs. Perplexity (PPL) for OLMoE (

\kappa=30

). Noise added to experts and dense modules.

$\kappa$	30	35	40	45	50	60
PPL	6.64	6.62	6.62	6.63	6.64	6.68

Appendix

Table 7:

\kappa

vs. Perplexity (PPL) for DeepSeekMoE. Noise added to experts.

$\lambda$	0.9	1.0	1.25	1.5	2.0	2.25	2.5	3.0	4.0
PPL	6.97	6.86	6.73	6.69	6.68	6.67	6.67	6.69	6.76

Appendix

Table 8:

\lambda

vs. Perplexity (PPL) for DeepSeekMoE (

\kappa=40

). Noise added to experts only.

$\kappa$	30	35	40	45	50
PPL	7.06	7.05	7.09	7.18	7.30

Appendix

Table 9:

\kappa

vs. Perplexity (PPL) for DeepSeekMoE. Noise added to experts and dense modules.

$\lambda$	1.0	1.25	1.5	1.75	2.0	2.25
PPL	7.75	7.52	7.51	7.58	7.70	7.85

Appendix

Table 10:

\lambda

vs. Perplexity (PPL) for DeepSeekMoE (

\kappa=35

). Noise added to experts and dense modules.

Panel of Figure 6 — Figure 6: Token visualization of the first MoE block of OLMoE. Expert 30, 50, and 14 correspond to lowest three $MaxNNorm$ neurons over the up-projection matrices. Similarly, Expert 38, 57, and 6 correspond to the highest $MaxNNorm$ neurons over the up-projection matrices.

Explore similar work

ROMER: Expert Replacement and Router Calibration for Robust MoE LLMs on Analog Compute-in-Memory Systems
May 12, 2026Wenyong Zhou, Yuannuo Feng, Yizhe Chen +6Compute-in-Memory Mixture-of-Experts Language Models
The Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong · The School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China
Speculating Experts Accelerates Inference for Mixture-of-Experts
Mar 9, 2026Vivan Madan, Prajwal Singhania, Abhinav Bhatele +2Mixture-of-Experts Inference LLM Inference Acceleration
University of Maryland · TogetherAI
Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study
Jun 19, 2026Alfarizy Alfarizy, Hung Truong Thanh Nguyen, René Richard +2LLM Inference Efficiency On-Device Language Model Inference
Analytics Everywhere Lab, University of New Brunswick, Fredericton, NB, Canada · TSAI Lab, University of New Brunswick, Fredericton, NB, Canada

May 12, 2026J/K move · Enter open · S save

ROMER: Expert Replacement and Router Calibration for Robust MoE LLMs on Analog Compute-in-Memory Systems

Wenyong Zhou, Yuannuo Feng, Yizhe Chen, Taiqiang Wu+5

The Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong · The School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China

Compute-in-Memory Mixture-of-Experts Language Models Mixture of Experts Mixture-of-Experts Inference Language Model Calibration Language Model Robustness

Open paper AskPDF arXiv

Large language models (LLMs) with mixture-of-experts (MoE) architectures achieve remarkable scalability by sparsely activating a subset of experts per token, yet their frequent expert switching creates memory bandwidth bottlenecks that compute-in-memory (CIM) architectures are well-suited to mitigate. However, analog CIM systems suffer from inherent hardware imperfections that perturb stored weights, and its negative impact on MoE-based LLMs in noisy CIM environments remains unexplored. In this work, we present the first systematic investigation of MoE-based LLMs under noise model calibrated with real chip measurements, revealing that hardware noise critically disrupts expert load balance and renders clean-trained routing decisions consistently suboptimal. Based on these findings, we propose ROMER, a post-training calibration framework that (1) replaces underactivated experts with high-frequency ones to restore load balance, and (2) recalibrates router logits via percentile-based normalization to stabilize routing under noise. Extensive experiments across multiple benchmarks demonstrate that ROMER achieves up to 58.6%, 58.8%, and 59.8% reduction in perplexity under real-chip noise conditions for DeepSeek-MoE, Qwen-MoE, and OLMoE, respectively, establishing its effectiveness and generalizability across diverse MoE architectures.

Abstract

Explore similar work

ROMER: Expert Replacement and Router Calibration for Robust MoE LLMs on Analog Compute-in-Memory Systems

Speculating Experts Accelerates Inference for Mixture-of-Experts

Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study