Mixture of Many Zero-Compute Experts: A High-Rate Quantization Theory Perspective

Yehuda Dar

cs.LGOct 3, 2025

Mixture of Many Zero-Compute Experts: A High-Rate Quantization Theory Perspective

Authors: Yehuda Dar

Organizations: Faculty of Computer and Information Science, Ben-Gurion University

Abstract

This paper uses classical high-rate quantization theory to provide new insights into mixture-of-experts (MoE) models for regression tasks. Our MoE is defined by a segmentation of the input space to regions, each with a single-parameter expert that acts as a constant predictor with zero-compute at inference. Motivated by high-rate quantization theory assumptions, we assume that the number of experts is sufficiently large to make their input-space regions very small. This lets us study the approximation error of our MoE model class: (i) for one-dimensional inputs, we formulate the test error and its minimizing segmentation and experts; (ii) for multidimensional inputs, we formulate an upper bound for the test error and study its minimization. Moreover, we consider the learning of the expert parameters from a training dataset, given an input-space segmentation, and formulate their statistical learning properties. This leads us to theoretically and empirically show how the tradeoff between approximation and estimation errors in MoE learning depends on the number of experts.

Figures & tables

Panel of Figure 1 — Figure 1: Examples for optimal segmentation, best predictor in $\mathcal{H}_{m,1}$ , and approximation error curves for a cosine $\beta$ and truncated Gaussian $p_{\mathrm{x}}$ . In (e)-(g), the red markers on the $x$ -axis denote the optimal segmentation points $\{a_{i}\}_{i=1}^{m}$ that were formed from the optimal segment density in (d).

Panel of Figure 2 — Figure 2: Examples for optimal segmentation, best predictor in $\mathcal{H}_{m,1}$ , and approximation error curves for a cosine with a constant segment $\beta$ and truncated Gaussian $p_{\mathrm{x}}$ . In (e)-(g), the red markers on the $x$ -axis denote the optimal segmentation points $\{a_{i}\}_{i=1}^{m}$ that were formed from the optimal segment density in (d).

Panel of Figure 3 — Figure 3: Examples for UBM-density-guided segmentation for a two-dimensional input space where $\beta$ is a two-dimensional separable cosine and $p_{\mathbf{x}}$ is a truncated isotropic Gaussian. In ( e )-( g ), the black lines denote the segmentation borders of the regions $\{A_{i}\}_{i=1}^{m}$ that were formed by Algorithm F using the UBM segment density in ( d ).

Panel of Figure 4 — Figure 4: Examples for UBM-density-guided segmentation for a two-dimensional input space where $\beta$ is a two-dimensional separable cosine with a constant area and $p_{\mathbf{x}}$ is a truncated isotropic Gaussian. In ( e )-( g ), the black lines denote the segmentation borders of the regions $\{A_{i}\}_{i=1}^{m}$ that were formed by Algorithm F using the UBM segment density in ( d ).

Panel of Figure 5 — Figure 5: Examples for UBM-density-guided segmentation for a two-dimensional input space where $\beta$ is a two-dimensional nonseparable cosine and $p_{\mathbf{x}}$ is a truncated isotropic Gaussian. In ( e )-( g ), the black lines denote the segmentation borders of the regions $\{A_{i}\}_{i=1}^{m}$ that were formed by Algorithm F using the UBM segment density in ( d ).

Panel of Figure 6 — Figure 6: Learning experts for uniform segmentation in one dimension . This experiment is for a cosine $\beta$ and truncated Gaussian $p_{\mathrm{x}}$ from Figs. 1 , 1 , respectively. Here, in (a)-(b), the red markers on the $x$ -axis denote the given uniform segmentation points $\{a_{i}=\frac{i}{m}\}_{i=1}^{m}$ ; the dotted magenta lines show the best predictor in $\mathcal{H}_{m,1}^{c}\left(\{a_{i}=\frac{i}{m}\}_{i=1}^{m}\right)$ ; the black lines show the predictor learned from a training data of 200 examples. (c) shows the empirical and theoretical approximation error curves of the best predictors in $\mathcal{H}_{m,1}^{c}\left(\{a_{i}=\frac{i}{m}\}_{i=1}^{m}\right)$ . (d) shows the empirical test error curves for the learned predictors for three sizes of training dataset.

Panel of Figure 7 — Figure 7: Learning experts for uniform segmentation in one dimension . This experiment is for a cosine with a constant segment $\beta$ and truncated Gaussian $p_{\mathrm{x}}$ from Figs. 2 , 2 , respectively. Here, in (a)-(b), the red markers on the $x$ -axis denote the given uniform segmentation points $\{a_{i}=\frac{i}{m}\}_{i=1}^{m}$ ; the dotted magenta lines show the best predictor in $\mathcal{H}_{m,1}^{c}\left(\{a_{i}=\frac{i}{m}\}_{i=1}^{m}\right)$ ; the black lines show the predictor learned from a training data of 200 examples. (c) shows the empirical and theoretical approximation error curves of the best predictors in $\mathcal{H}_{m,1}^{c}\left(\{a_{i}=\frac{i}{m}\}_{i=1}^{m}\right)$ . (d) shows the empirical test error curves for the learned predictors for three sizes of training dataset.

Panel of Figure 8 — Figure 8: Learning experts for uniform segmentation in two dimensions . This experiment is for a separable cosine $\beta$ and truncated Gaussian $p_{\mathbf{x}}$ from Figs. 3 , 3 , respectively. In the first three rows of subfigures: Each row of subfigures shows results for uniform segmentations with a different number of experts ( $m=100,484,1024$ ); the left subfigures show the true $\beta$ and the uniform segmentation (borders appear in black lines), the center subfigures show the optimal expert constants, the right subfigures show the learned expert constants from a training dataset of $n=3000$ examples. Subfigure ( j ) shows the empirical and theoretical approximation error curves for the uniform segmentations. Subfigure ( k ) shows the empirical test error vs. $m$ curves for three sizes of training dataset $n=1000,2000,3000$ .

Panel of Figure 9 — Figure 9: Learning experts for uniform segmentation in two dimensions . This experiment is for a cosine with a constant area $\beta$ and truncated Gaussian $p_{\mathbf{x}}$ from Figs. 4 , 4 , respectively. In the first three rows of subfigures: Each row of subfigures shows results for uniform segmentations with a different number of experts ( $m=100,484,1024$ ); the left subfigures show the true $\beta$ and the uniform segmentation (borders appear in black lines), the center subfigures show the optimal expert constants, the right subfigures show the learned expert constants from a training dataset of $n=3000$ examples. Subfigure ( j ) shows the empirical and theoretical approximation error curves for the uniform segmentations. Subfigure ( k ) shows the empirical test error vs. $m$ curves for three sizes of training dataset $n=1000,2000,3000$ .

Panel of Figure 10 — Figure 10: Learning experts for uniform segmentation in two dimensions . This experiment is for a nonseparable cosine and truncated Gaussian $p_{\mathbf{x}}$ from Figs. 5 , 5 , respectively. In the first three rows of subfigures: Each row of subfigures shows results for uniform segmentations with a different number of experts ( $m=100,484,1024$ ); the left subfigures show the true $\beta$ and the uniform segmentation (borders appear in black lines), the center subfigures show the optimal expert constants, the right subfigures show the learned expert constants from a training dataset of $n=3000$ examples. Subfigure ( j ) shows the empirical and theoretical approximation error curves for the uniform segmentations. Subfigure ( k ) shows the empirical test error vs. $m$ curves for three sizes of training dataset $n=1000,2000,3000$ .

Mixture-of-Experts Models

Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.

See all 454 papers

Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees
Apr 7, 2026Mohammed Nowaz Rabbani Chowdhury, Kaoutar El Maghraoui, Hsinyu Tsai +4Mixture-of-Experts Quantization Neural Network Quantization
Rensselaer Polytechnic Institute · IBM Research
Towards a Statistical Understanding of Mixture-of-Experts
Sep 3, 2026Siyuan He, Bokai Yang, Jie Hu +2Mixture of Experts Sparse Mixture-of-Experts
Qiuzhen College, Tsinghua University, Beijing, China. · KLATASDS-MOE, School of Statistics, East China Normal University, Shanghai, China. · Yau Mathematical Sciences Center, Tsinghua University, Beijing, China. +1
Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models
Sep 4, 2026Xing Chen, Hengshuai YaoMixture-of-Experts Language Models Mixture-of-Experts Inference

Apr 7, 2026J/K move · Enter open · S save

Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees

Mohammed Nowaz Rabbani Chowdhury, Kaoutar El Maghraoui, Hsinyu Tsai, Naigang Wang+3

Rensselaer Polytechnic Institute · IBM Research

Mixture-of-Experts Quantization Neural Network Quantization Post-Training Quantization Sparse Mixture-of-Experts

Open paper AskPDF arXiv

Sparse Mixture-of-Experts (MoE) allows scaling of language and vision models efficiently by activating only a small subset of experts per input. While this reduces computation, the large number of parameters still incurs substantial memory overhead during inference. Post-training quantization has been explored to address this issue. Because uniform quantization suffers from significant accuracy loss at low bit-widths, mixed-precision methods have been recently explored; however, they often require substantial computation for bit-width allocation and overlook the varying sensitivity of model performance to the quantization of different experts. We propose a theoretically grounded expert-wise mixed precision strategy that assigns bit-width to each expert primarily based on their change in routers l2 norm during training. Experts with smaller changes are shown to capture less frequent but critical features, and model performance is more sensitive to the quantization of these experts, thus requiring higher precision. Furthermore, to avoid allocating experts to lower precision that inject high quantization noise, experts with large maximum intra-neuron variance are also allocated higher precision. Experiments on large-scale MoE models, including Switch Transformer and Mixtral, show that our method achieves higher accuracy than existing approaches, while also reducing inference cost and incurring only negligible overhead for bit-width assignment.

Abstract

Mixture-of-Experts Models

Explore similar work

Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees

Towards a Statistical Understanding of Mixture-of-Experts

Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models