Mixture of Many Zero-Compute Experts: A High-Rate Quantization Theory Perspective

Yehuda Dar

cs.LGOct 3, 2025

Mixture of Many Zero-Compute Experts: A High-Rate Quantization Theory Perspective

Authors: Yehuda Dar

Organizations: Faculty of Computer and Information Science, Ben-Gurion University

Abstract

This paper uses classical high-rate quantization theory to provide new insights into mixture-of-experts (MoE) models for regression tasks. Our MoE is defined by a segmentation of the input space to regions, each with a single-parameter expert that acts as a constant predictor with zero-compute at inference. Motivated by high-rate quantization theory assumptions, we assume that the number of experts is sufficiently large to make their input-space regions very small. This lets us study the approximation error of our MoE model class: (i) for one-dimensional inputs, we formulate the test error and its minimizing segmentation and experts; (ii) for multidimensional inputs, we formulate an upper bound for the test error and study its minimization. Moreover, we consider the learning of the expert parameters from a training dataset, given an input-space segmentation, and formulate their statistical learning properties. This leads us to theoretically and empirically show how the tradeoff between approximation and estimation errors in MoE learning depends on the number of experts.

Figures & tables

Panel of Figure 1 — Figure 1: Examples for optimal segmentation, best predictor in $\mathcal{H}_{m,1}$ , and approximation error curves for a cosine $\beta$ and truncated Gaussian $p_{\mathrm{x}}$ . In (e)-(g), the red markers on the $x$ -axis denote the optimal segmentation points $\{a_{i}\}_{i=1}^{m}$ that were formed from the optimal segment density in (d).

Panel of Figure 2 — Figure 2: Examples for optimal segmentation, best predictor in $\mathcal{H}_{m,1}$ , and approximation error curves for a cosine with a constant segment $\beta$ and truncated Gaussian $p_{\mathrm{x}}$ . In (e)-(g), the red markers on the $x$ -axis denote the optimal segmentation points $\{a_{i}\}_{i=1}^{m}$ that were formed from the optimal segment density in (d).

Panel of Figure 3 — Figure 3: Examples for UBM-density-guided segmentation for a two-dimensional input space where $\beta$ is a two-dimensional separable cosine and $p_{\mathbf{x}}$ is a truncated isotropic Gaussian. In ( e )-( g ), the black lines denote the segmentation borders of the regions $\{A_{i}\}_{i=1}^{m}$ that were formed by Algorithm F using the UBM segment density in ( d ).

Panel of Figure 4 — Figure 4: Examples for UBM-density-guided segmentation for a two-dimensional input space where $\beta$ is a two-dimensional separable cosine with a constant area and $p_{\mathbf{x}}$ is a truncated isotropic Gaussian. In ( e )-( g ), the black lines denote the segmentation borders of the regions $\{A_{i}\}_{i=1}^{m}$ that were formed by Algorithm F using the UBM segment density in ( d ).

Panel of Figure 5 — Figure 5: Examples for UBM-density-guided segmentation for a two-dimensional input space where $\beta$ is a two-dimensional nonseparable cosine and $p_{\mathbf{x}}$ is a truncated isotropic Gaussian. In ( e )-( g ), the black lines denote the segmentation borders of the regions $\{A_{i}\}_{i=1}^{m}$ that were formed by Algorithm F using the UBM segment density in ( d ).

Panel of Figure 6 — Figure 6: Learning experts for uniform segmentation in one dimension . This experiment is for a cosine $\beta$ and truncated Gaussian $p_{\mathrm{x}}$ from Figs. 1 , 1 , respectively. Here, in (a)-(b), the red markers on the $x$ -axis denote the given uniform segmentation points $\{a_{i}=\frac{i}{m}\}_{i=1}^{m}$ ; the dotted magenta lines show the best predictor in $\mathcal{H}_{m,1}^{c}\left(\{a_{i}=\frac{i}{m}\}_{i=1}^{m}\right)$ ; the black lines show the predictor learned from a training data of 200 examples. (c) shows the empirical and theoretical approximation error curves of the best predictors in $\mathcal{H}_{m,1}^{c}\left(\{a_{i}=\frac{i}{m}\}_{i=1}^{m}\right)$ . (d) shows the empirical test error curves for the learned predictors for three sizes of training dataset.

Panel of Figure 7 — Figure 7: Learning experts for uniform segmentation in one dimension . This experiment is for a cosine with a constant segment $\beta$ and truncated Gaussian $p_{\mathrm{x}}$ from Figs. 2 , 2 , respectively. Here, in (a)-(b), the red markers on the $x$ -axis denote the given uniform segmentation points $\{a_{i}=\frac{i}{m}\}_{i=1}^{m}$ ; the dotted magenta lines show the best predictor in $\mathcal{H}_{m,1}^{c}\left(\{a_{i}=\frac{i}{m}\}_{i=1}^{m}\right)$ ; the black lines show the predictor learned from a training data of 200 examples. (c) shows the empirical and theoretical approximation error curves of the best predictors in $\mathcal{H}_{m,1}^{c}\left(\{a_{i}=\frac{i}{m}\}_{i=1}^{m}\right)$ . (d) shows the empirical test error curves for the learned predictors for three sizes of training dataset.

Panel of Figure 8 — Figure 8: Learning experts for uniform segmentation in two dimensions . This experiment is for a separable cosine $\beta$ and truncated Gaussian $p_{\mathbf{x}}$ from Figs. 3 , 3 , respectively. In the first three rows of subfigures: Each row of subfigures shows results for uniform segmentations with a different number of experts ( $m=100,484,1024$ ); the left subfigures show the true $\beta$ and the uniform segmentation (borders appear in black lines), the center subfigures show the optimal expert constants, the right subfigures show the learned expert constants from a training dataset of $n=3000$ examples. Subfigure ( j ) shows the empirical and theoretical approximation error curves for the uniform segmentations. Subfigure ( k ) shows the empirical test error vs. $m$ curves for three sizes of training dataset $n=1000,2000,3000$ .

Panel of Figure 9 — Figure 9: Learning experts for uniform segmentation in two dimensions . This experiment is for a cosine with a constant area $\beta$ and truncated Gaussian $p_{\mathbf{x}}$ from Figs. 4 , 4 , respectively. In the first three rows of subfigures: Each row of subfigures shows results for uniform segmentations with a different number of experts ( $m=100,484,1024$ ); the left subfigures show the true $\beta$ and the uniform segmentation (borders appear in black lines), the center subfigures show the optimal expert constants, the right subfigures show the learned expert constants from a training dataset of $n=3000$ examples. Subfigure ( j ) shows the empirical and theoretical approximation error curves for the uniform segmentations. Subfigure ( k ) shows the empirical test error vs. $m$ curves for three sizes of training dataset $n=1000,2000,3000$ .

Panel of Figure 10 — Figure 10: Learning experts for uniform segmentation in two dimensions . This experiment is for a nonseparable cosine and truncated Gaussian $p_{\mathbf{x}}$ from Figs. 5 , 5 , respectively. In the first three rows of subfigures: Each row of subfigures shows results for uniform segmentations with a different number of experts ( $m=100,484,1024$ ); the left subfigures show the true $\beta$ and the uniform segmentation (borders appear in black lines), the center subfigures show the optimal expert constants, the right subfigures show the learned expert constants from a training dataset of $n=3000$ examples. Subfigure ( j ) shows the empirical and theoretical approximation error curves for the uniform segmentations. Subfigure ( k ) shows the empirical test error vs. $m$ curves for three sizes of training dataset $n=1000,2000,3000$ .

Mixture-of-Experts Models

Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.

See all 454 papers

Apr 7, 2026

Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees

Sparse Mixture-of-Experts (MoE) allows scaling of language and vision models efficiently by activating only a small subset of experts per input. While this reduces computation, the large number of parameters still incurs substantial memory overhead during inference. Post-training quantization has been explored to address this issue. Because uniform quantization suffers from significant accuracy loss at low bit-widths, mixed-precision methods have been recently explored; however, they often require substantial computation for bit-width allocation and overlook the varying sensitivity of model performance to the quantization of different experts. We propose a theoretically grounded expert-wise mixed precision strategy that assigns bit-width to each expert primarily based on their change in routers l2 norm during training. Experts with smaller changes are shown to capture less frequent but critical features, and model performance is more sensitive to the quantization of these experts, thus requiring higher precision. Furthermore, to avoid allocating experts to lower precision that inject high quantization noise, experts with large maximum intra-neuron variance are also allocated higher precision. Experiments on large-scale MoE models, including Switch Transformer and Mixtral, show that our method achieves higher accuracy than existing approaches, while also reducing inference cost and incurring only negligible overhead for bit-width assignment.

Mohammed Nowaz Rabbani Chowdhury, Kaoutar El Maghraoui, Hsinyu Tsai +4

Rensselaer Polytechnic Institute · IBM Research

Mixture-of-Experts Quantization Neural Network Quantization Post-Training Quantization Sparse Mixture-of-Experts

Ask PDF

Sep 3, 2026

Towards a Statistical Understanding of Mixture-of-Experts

Mixture-of-experts (MoE) architectures increase model capacity by combining a collection of expert predictors through input-dependent routing, while often activating only a small subset of experts for each input. Despite their growing importance in modern large-scale models, the statistical roles of their design choices, especially routing, sparse activation, and shared experts, remain only partially understood, as existing theory has largely focused on parametric or correctly specified MoE models. In this paper, we view MoE as a form of localized aggregation and show how this localization reshapes the approximation-estimation-computation tradeoff. We derive oracle risk bounds for learning dense and sparse routing with evolving experts, separating approximation, expert-learning, and router-estimation errors, and characterize how sparse Top-K routing can retain the benefits of localized aggregation while controlling per-input computation. We also interpret gating through the geometry of input space, relating routing performance to regions of local expert advantage, and show how shared experts, as adopted in architectures such as DeepSeekMoE, can extract common predictive structure so that routed experts focus on residual local variation. Together, these results provide a unified statistical framework for understanding MoE through input-dependent expert aggregation, in which expert specialization and computational tradeoffs are governed by local predictive structure.

Siyuan He, Bokai Yang, Jie Hu +2

Qiuzhen College, Tsinghua University, Beijing, China. · KLATASDS-MOE, School of Statistics, East China Normal University, Shanghai, China. · Yau Mathematical Sciences Center, Tsinghua University, Beijing, China. +1

Mixture of Experts Sparse Mixture-of-Experts Mixture-of-Experts Models Statistical Learning Theory

Ask PDF

Sep 4, 2026

Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models

Modern fine-grained Mixture-of-Experts (MoE) models route each token to a small number of experts and renormalize their router probabilities. We show that this renormalization implicitly calibrates expert output gain to the training top-

k

: reducing

k

at inference changes not only which experts are used but also the strength of the expert branch. We separate these effects by activating the top

k_1

experts while normalizing by the probability mass of the top

k_2

experts, introducing one integer with no parameters, training, or measurable compute overhead. On Qwen3.6-35B-A3B, reducing from 8 to 4 experts causes a 4.65-point MMLU drop under standard renormalization but only 0.35 points with

k_2=16

, while halving routed-expert compute. The result replicates on the

11\times

larger Qwen3.5-397B-A17B, where reducing from 10 to 5 experts loses only 0.55 points with an appropriate reference set. Removing renormalization entirely is catastrophic, showing that preserving a suitable reference mass is crucial. We further find that perplexity and downstream accuracy favor different

k_2

, cautioning against selecting MoE compression settings using unlabeled text alone. Analyses also show that expert identity matters substantially more than expert weighting, while balanced and domain-specialized routing leaves limited room for expert pruning.

Xing Chen, Hengshuai Yao

Mixture-of-Experts Language Models Mixture-of-Experts Inference Mixture-of-Experts Pruning Parameter-Free Mixture-of-Experts Routing

Ask PDF

Abstract

Mixture-of-Experts Models

Explore similar work

Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees

Towards a Statistical Understanding of Mixture-of-Experts

Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models