Mixture of Channel Experts: Static Sparse Supports with Input-Adaptive Mixing for Pointwise Projections

Elian Iluk; Gil Ben-Artzi

cs.LGAug 24, 2026

Mixture of Channel Experts: Static Sparse Supports with Input-Adaptive Mixing for Pointwise Projections

Authors: Elian Iluk, Gil Ben-Artzi

Organizations: School of Computer Science, Ariel University, Israel

Abstract

Mixture-of-Experts (MoE) scales language models by routing each input through a small set of independently parameterized experts. We show that copying this design into convolutional networks fails for a structural reason: parallel convolutional experts that read the same input channels learn nearly identical filters. We therefore move the expert axis from operator duplication to channel selection. We introduce Mixture of Channel Experts (MoCE), a structured sparse channel-mixing layer, inspired by MoE, that replaces pointwise (1x1) channel-reduction projections. In MoCE, an expert is a single output channel with a learned sparse support of k << C input channels. The selected channels are combined by a softmax whose temperature is predicted per input, so each expert can move between mean-like and max-like aggregation. A residual expert summarizes the unselected channels, and a load-balancing loss keeps channel coverage complete. MoCE replaces a dense projection whose cost is quadratic in C with a mechanism whose relative cost scales as k/C, and the predicted savings hold in measured wall-clock time. Across ResNet backbones on ImageNet-1K and CIFAR-100, transfer learning, EfficientViT, and a strong modern training recipe, MoCE matches or exceeds dense baselines and prior channel-selection methods while reducing MACs by 16.7% and end-to-end latency.

Figures & tables

Figure 1: Parallel convolutional experts sharing an input (ResNet-50, CIFAR-100): pairwise kernel cosine similarity.

Figure 2: MoCE layer. $E\!-\!1$ routed experts over learned static supports, plus one residual aggregate.

	MACs (G)		Params (M)
Backbone	Dense	MoCE	Dense	MoCE (tr./dep.)
ResNet-50	4.112	3.426	25.56	25.55 / 21.26
ResNet-101	7.834	6.288	44.55	44.52 / 35.83
ResNet-152	11.559	9.156	60.19	60.15 / 47.84

Table 1: Cost and storage, accounted as in Sec. 3.5 : MACs are network totals with each replaced projection counted as

[(E{-}1)k+2C]HW

under the conservative bound

|\mathcal{U}|{=}C

. MoCE parameters are training/deployment counts of learned values.

Model	Params (M)	MACs (G)	Top-1 (%)
ResNet-50	25.56	4.112	75.98 $\pm$ 0.30
+ MoCE	25.55 / 21.26	3.426	76.71 $\pm$ 0.20
ResNet-101	44.55	7.834	77.21 $\pm$ 0.29
+ MoCE	44.52 / 35.83	6.288	77.54 $\pm$ 0.19
ResNet-152	60.19	11.559	77.78 $\pm$ 0.43
+ MoCE	60.15 / 47.84	9.156	78.24 $\pm$ 0.12

Table 2: ImageNet-1K; accuracies are mean

\pm

standard deviation over three seeds. MoCE parameters are training/deployment counts of learned values; deployment additionally stores packed support indices (

\approx 60

KB for ResNet-50).

Method	Params (M)	MACs (G)	Top-1 (%)
Dense	25.56	4.112	75.98 $\pm$ 0.30
SE	28.07	4.120	76.64 $\pm$ 0.19
CondConv	55.97	4.150	76.52 $\pm$ 0.24
Pick-or-Mix	25.56	3.178	76.25 $\pm$ 0.26
MoCE	25.55 / 21.26	3.426	76.71 $\pm$ 0.20

Table 3: Matched ResNet-50 comparison on ImageNet-1K. MoCE parameters are training/deployment counts of learned values.

	From scratch		Transfer
Model	MACs	C-100	MACs	C-10	C-100
ResNet-50	1.305	78.44 $\pm$ .32	4.11	97.52 $\pm$ .10	85.44 $\pm$ .20
+ MoCE	1.081	79.35 $\pm$ .35	3.43	97.76 $\pm$ .12	86.47 $\pm$ .27
ResNet-101	2.520	78.93 $\pm$ .32	7.83	97.80 $\pm$ .11	87.50 $\pm$ .11
+ MoCE	2.015	79.53 $\pm$ .19	6.29	98.06 $\pm$ .13	88.11 $\pm$ .10
ResNet-152	3.737	79.40 $\pm$ .27	11.56	97.88 $\pm$ .06	88.36 $\pm$ .16

Table 4: CIFAR-100 from scratch (left) and ImageNet

\rightarrow

CIFAR transfer (right); three seeds.

Model	Params (M)	MACs (G)	Top-1 (%)
EfficientViT-M5	12.13	0.526	75.56 $\pm$ 0.23
+ MoCE	12.11	0.410	75.95 $\pm$ 0.17
EfficientViT-M3	6.61	0.265	75.12 $\pm$ 0.10
+ MoCE	6.60	0.210	75.20 $\pm$ 0.05
EfficientViT-M2	3.99	0.204	74.61 $\pm$ 0.39
+ MoCE	3.98	0.167	74.75 $\pm$ 0.21

Table 5: MoCE in EfficientViT on CIFAR-100 (

k{=}64

, second feed-forward projection,

s{=}2

), three seeds. Parameter counts are training-time; the deployment packing reported for ResNet is not applied here.

Where input dependence is applied (CIFAR-100)
Setting	Top-1 (%)	Routing + gather
MoCE: adaptive mixing, static support	79.35	1.9 ms
Input-conditioned support (Eq. 11 )	79.38	13.5 ms
Static mixing ( $\tau\equiv 1$ , $L$ learnable)	78.16	1.9 ms
No residual aggregate	78.61	–
Support design (CIFAR-100)

Table 6: Ablations on ResNet-50; all entries are single runs. Components, support design, and temperature bounds on CIFAR-100; sparsity

k

and coverage weight

\lambda

on ImageNet-1K. Timings cover the inference-time routing and gather path only.

Scope	Dense	PiX	MoCE	MoCE/Dense
End-to-end	133.12	127.30	127.00	0.954
All replaced ops	19.63	15.82	15.60	0.795
Reduce 3.0	2.093	1.375	1.282	0.613
Reduce 4.0	2.116	0.802	0.701	0.331

Table 7: FP32 batch time (ms), batch

512

, ResNet-50 eval mode; medians after warm-up with device synchronization around each measurement. PiX and MoCE medians fall within their spread; IQRs, hardware, and library versions are in the supplement.

Figure 3: Routing in the first MoCE layer. Top left: top- $k$ overlap between experts. Top right: cosine similarity of routing-logit rows. Bottom: experts selecting each channel, against the uniform expectation.

Figure 4: Learned temperatures across experts, spanning concentrated, intermediate, and near-uniform regimes rather than collapsing to one mode.

Explore similar work

Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration
Aug 28, 2026Simeng Sun, Roger WaleffeMixture-of-Experts Language Models Mixture of Experts
NVIDIA
IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts
Sep 18, 2026Ran Cheng, Longfei Xu, Zheng Liu +2Mixture-of-Experts Models Mixture-of-Experts Inference
DreamX, Alibaba Group
Path-Constrained Mixture-of-Experts
Mar 18, 2026Zijin Gu, Tatiana Likhomanenko, Vimal Thilak +2Mixture-of-Experts Language Models Sparse Mixture-of-Experts
Apple · †Google

Aug 28, 2026J/K move · Enter open · S save

Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration

Simeng Sun, Roger Waleffe

NVIDIA

Mixture-of-Experts Language Models Mixture of Experts Efficient Language Model Training Communication-Efficient Distributed Training

Open paper AskPDF arXiv

When training Mixture-of-Experts (MoE) language models with expert parallelism, all-to-all token dispatch and combine collectives can consume a substantial fraction of end-to-end training time. In this work, we study communication-efficient MoE models (CE-MoE), in which we adopt a heterogeneous layer pattern that decouples token-mixing and channel-mixing depth. Compared to conventional models which interleave MoE layers after each token-mixing layer (e.g., attention, Mamba-2), CE-MoE models concentrate expert capacity in a select few routed MoE layers, while maintaining depth by adding additional token-mixing and dense-FFN layers. Across a scaling ladder from 2B to 31.5B total parameters, under matched total and activated parameters, CE-MoE models consistently reduce training cost while matching validation loss and downstream benchmarks with full-MoE baselines. At the 31.5B scale, CE-MoE uses 33.3% fewer GPU-hours while improving average downstream score and inference throughput.

Abstract

Explore similar work

Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration

IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts

Path-Constrained Mixture-of-Experts