Mixture-of-Experts under Finite-Rate Gating: Communication--Generalization Trade-offs

Ali Khalesi; Mohammad Reza Deylam Salehi

stat.MLFeb 16, 2026

Mixture-of-Experts under Finite-Rate Gating: Communication--Generalization Trade-offs

Authors: Ali Khalesi, Mohammad Reza Deylam Salehi

Organizations: Institut Polytechnique des Sciences Avancées (IPSA) and LINCS Lab, Paris, France · IEEE member, Nice, France

Abstract

Mixture-of-Experts (MoE) architectures decompose prediction tasks into specialized expert sub-networks selected by a gating mechanism. This letter adopts a communication-theoretic view of MoE gating, modeling the gate as a stochastic channel operating under a finite information rate. Within an information-theoretic learning framework, {we specialize a mutual-information generalization bound and develop a rate-distortion characterization $D(R_g)$ of finite-rate gating, where $R_g:=I(X; T)$ , yielding (under a standard empirical rate-distortion optimality condition) $\mathbb{E}[R(W)] \le D(R_g)+δ_m+\sqrt{(2/m)\, I(S; W)}$ . }The analysis yields capacity-aware limits for communication-constrained MoE systems, and numerical simulations on synthetic multi-expert models empirically confirm the predicted trade-offs between gating rate, expressivity, and generalization.

Figures & tables

Panel of Figure 1 — Fig. 1: MoE system viewed as a finite-rate stochastic communication link. The input feature vector $X$ is processed by a gating module implementing a channel $P(T\mid X;W_{\text{gate}})$ that maps $X$ to a routing index $T\in[n]$ under an information-rate constraint. The index selects an expert ${h_{g}(\cdot;W_{g})}_{g=1}^{n}$ , yielding the prediction $\hat{Y}=h_{T}(X;W_{T})$ . Here $I(X;T)$ is the effective communication rate to the expert bank, so the gate acts as a constrained link that limits how much information about $X$ reaches the experts, shaping expressivity and generalization.

Fig. 2: MoE simulation for Theorem 1 . The empirical generalization gap $\lvert\mathbb{E}[R-R_{S}]\rvert$ increases with the information term $\sqrt{2I(S;W)/m}$ and remains below the theoretical upper bound, illustrating the info-generalization trade-off at the algorithm level.

Fig. 3: BSC rate-distortion-generalization experiment for Theorem 2 . The empirical mean population risk $\mathbb{E}[R(W)]$ (dots) closely follows the rate-distortion curve $D(R_{g})$ , while the bound in ( 13 ) (triangles) remains safely above, confirming the theoretical trade-off between gating rate and prediction accuracy.

Mixture-of-Experts Models

Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.

See all 439 papers

Expert Routing for Communication-Efficient MoE via Finite Expert Banks
May 6, 2026Mohammad Reza Deylam Salehi, Ali KhalesiMixture of Experts Information-Theoretic Generalization Bounds
IEEE Graduate Member · Nice, France · Institut Polytechnique des Sciences Avancées (IPSA) +1
Sparsity Saves Compute, Not Samples: Oracle Inequalities for Dense and Sparse Mixture of Experts
Oct 10, 2026Sagar Ghosh, Huy Nguyen, Alessandro RinaldoSparse Mixture-of-Experts Mixture-of-Experts Models
The University of Texas at Austin
Towards a Statistical Understanding of Mixture-of-Experts
Sep 3, 2026Siyuan He, Bokai Yang, Jie Hu +2Mixture of Experts Sparse Mixture-of-Experts
Qiuzhen College, Tsinghua University, Beijing, China. · KLATASDS-MOE, School of Statistics, East China Normal University, Shanghai, China. · Yau Mathematical Sciences Center, Tsinghua University, Beijing, China. +1

May 6, 2026J/K move · Enter open · S save

Expert Routing for Communication-Efficient MoE via Finite Expert Banks

Mohammad Reza Deylam Salehi, Ali Khalesi

IEEE Graduate Member · Nice, France · Institut Polytechnique des Sciences Avancées (IPSA)+1

Mixture of Experts Information-Theoretic Generalization Bounds Communication-Efficient Mixture-of-Experts Expert Routing Mutual Information Estimation Mixture-of-Experts Models+1

Open paper AskPDF arXiv

Resource-efficient machine learning increasingly uses sparse Mixture-of-Experts (MoE) architectures, where the gate acts as both a learning component and a routing interface controlling computation, communication, and accuracy. Motivated by finite-rate interpretations of MoE gating, we treat the gate as a stochastic channel and use

I(X;T)

to quantify the routing information available to the selected expert. To make the associated information quantities tractable beyond synthetic examples, we develop a finite-bank MNIST construction using pretrained CNN experts and a discrete, data-dependent selection rule. Since the selected model belongs to a finite candidate set, the algorithmic mutual information

I(S;W)

admits a closed-form discrete-entropy estimator from the empirical posterior

q(W|S)

. Sweeping a data-dependence parameter

α

, we observe that

\widehat I(S;W)

monotonically tracks the generalization gap, while the Xu-Raginsky bound exhibits the expected looseness. We also compare with a uniform union-bound baseline and introduce an empirical estimator of

I(X;T)

together with a Blahut-Arimoto procedure for tracing an accuracy-rate curve over the expert bank. The proposed framework provides a practical tool for analyzing resource-aware MoE inference systems and for interpreting

I(X;T)

and

D(R_g)

as design proxies for efficient expert routing.

Abstract

Mixture-of-Experts Models

Explore similar work

Expert Routing for Communication-Efficient MoE via Finite Expert Banks

Sparsity Saves Compute, Not Samples: Oracle Inequalities for Dense and Sparse Mixture of Experts

Towards a Statistical Understanding of Mixture-of-Experts