Design and Behavior of Sparse Mixture-of-Experts Layers in CNN-based Semantic Segmentation

Svetlana Pavlitska; Haixi Fan; Konstantin Ditschuneit; J. Marius Zöllner

cs.CVApr 15, 2026

Design and Behavior of Sparse Mixture-of-Experts Layers in CNN-based Semantic Segmentation

Authors: Svetlana Pavlitska, Haixi Fan, Konstantin Ditschuneit, J. Marius Zöllner

Organizations: FZI Research Center for Information Technology · Karlsruhe Institute of Technology (KIT)

Abstract

Sparse mixture-of-experts (MoE) layers have been shown to substantially increase model capacity without a proportional increase in computational cost and are widely used in transformer architectures, where they typically replace feed-forward network blocks. In contrast, integrating sparse MoE layers into convolutional neural networks (CNNs) remains inconsistent, with most prior work focusing on fine-grained MoEs operating at the filter or channel levels. In this work, we investigate a coarser, patch-wise formulation of sparse MoE layers for semantic segmentation, where local regions are routed to a small subset of convolutional experts. Through experiments on the Cityscapes and BDD100K datasets using encoder-decoder and backbone-based CNNs, we conduct a design analysis to assess how architectural choices affect routing dynamics and expert specialization. Our results demonstrate consistent, architecture-dependent improvements (up to +3.9 mIoU) with little computational overhead, while revealing strong design sensitivity. Our work provides empirical insights into the design and internal dynamics of sparse MoE layers in CNN-based dense prediction. Our code is available at https://github.com/KASTEL-MobilityLab/moe-layers/.

Figures & tables

Figure 1: Overview of the studied PatchConvMoE layer within a CNN for semantic segmentation. A standard convolutional layer $\ell$ is replaced with a sparse MoE layer comprising convolutional layers as experts and a patch-wise gating network. The input feature map is split into a $g\times g$ grid of patches. Each patch is routed to a small subset of experts using top-k routing (here: a total of $n=8$ experts with top-2 routing, i.e., $k=2$ ). Expert outputs are linearly combined and reassembled into the output feature map.

Panel of Figure 2 — Figure 2: Gate architectures.

Architecture	Number of parameters		GFLOPs	Inf. time	mIoU in %
	total	active		per image, ms	Cityscapes	BDD100K
ENet
Baseline	351,660		8.72	12.04	51.80	52.27
PatchConvMoE, Conv-GAP gate	352,956 (+0.36%)	352,092 (+0.12%)	8.73 (+0.11%)	25.15 (+108.89%)	53.33 (+1.53)	53.43 (+1.16)
PatchConvMoE, 2Conv-GAP gate	353,548 (+0.54%)	352,684 (+0.29%)	8.74 (+0.23%)	25.20 (+109.30%)	55.70 (+3.90)	53.70 (+1.43)
PatchConvMoE, 3Conv-GAP gate	354,140 (+0.71%)	353,276 (+0.46%)	8.76 (+0.46%)	25.26 (+109.80%)	50.82	53.60 (+1.33)

Table 1: Performance of the baselines and models with one convolutional layer replaced with a PatchConvMoE layer in the decoder with 8 experts, top-2 routing,

\mathcal{L}_{entropy}

, and the 3

\times

3 grid for patchification. We highlight the best overall mIoU per architecture in bold and mIoU of models with a PatchConvMoE layer surpassing that of the baseline in green . Inference time evaluated on NVIDIA RTX2080Ti.

Patch size in px	ENet		ERFNet		U-Net		LR-ASPP		DeepLabv3+		PSPNet
	mIoU in %	Grid size	mIoU in %	Grid size	mIoU in %	Grid size	mIoU in %	Grid size	mIoU in %	Grid size	mIoU in %	Grid size
No patchification	31.37	-	16.25	-	57.95	-	55.12	-		-	64.47	-
Patch size 8 $\times$ 8 px	51.26	48 $\times$ 48	60.31	24 $\times$ 24	16.50	96 $\times$ 96	58.93	48 $\times$ 48	77.14	24 $\times$ 24	70.44	12 $\times$ 12
Patch size 16 $\times$ 16 px	51.69	24 $\times$ 24	60.24	12 $\times$ 12	18.35	48 $\times$ 48	60.68	24 $\times$ 24	77.41 (+0.15)	12 $\times$ 12	72.92	6 $\times$ 6
Patch size 32 $\times$ 32 px	52.97 (+1.17)	12 $\times$ 12	61.46 (+0.80)	6 $\times$ 6	20.09	24 $\times$ 24	61.36	12 $\times$ 12	77.49 (+0.23)	6 $\times$ 6	75.63 (+2.58)	3 $\times$ 3
Patch size 64 $\times$ 64 px	53.56 (+1.76)	6 $\times$ 6	61.98 (+1.32)	3 $\times$ 3	28.02	12 $\times$ 12	66.94 (+0.15)	6 $\times$ 6	77.61 (+0.35)	3 $\times$ 3	75.02 (+1.97)	1 $\times$ 1

Table 2: Image-based (i.e., no patchification) vs. patch-based routing and different patch sizes evaluated on Cityscapes . PatchConvMoE layers with 2Conv-GAP gate, other settings follow Table 1 .

Loss	ENet		ERFNet		U-Net		LR-ASPP		DeepLabv3+		PSPNet
	top-1	top-2	top-1	top-2	top-1	top-2	top-1	top-2	top-1	top-2	top-1	top-2
No loss	54.13 (+2.33)	55.72 (+3.92)	59.58	59.44	56.59	57.02	67.47(+0.68)	67.11 (+0.32)	78.05 (+0.79)	77.66 (+0.40)	70.21	70.35
$\mathcal{L}_{entropy}$	51.92 (+0.12)	55.70 (+3.90)	62.04 (+1.38)	61.98 (+1.32)	62.83	63.88	68.47 (+1.68)	68.63 (+1.84)	77.50 (+0.24)	77.61 (+0.35)	62.08	75.63 (+2.58)
$\mathcal{L}_{imp}$	53.76 (+1.96)	53.29 (+1.49)	59.05	62.12 (+1.46)	66.03 (+1.09)	67.21 (+2.27)	67.19 (+0.40)	67.35 (+0.56)	77.41 (+0.15)	77.55 (+0.29)	64.97	76.25 (+3.20)
$\mathcal{L}_{switch}$	53.75 (+1.95)	54.55 (+2.75)	59.36	60.86 (+0.20)	64.42	65.24 (+0.30)	67.23 (+0.44)	67.49 (+0.70)	77.30 (+0.04)	77.27 (+0.01)	66.99	75.46 (+2.41)

Table 3: mIoU in % for different balancing losses and parameter k for the PatchConvMoE layers with 2Conv-GAP gate and top-

k

routing evaluated on Cityscapes . We highlight the results for the best-performing balancing loss, other settings follow Table 1 .

ENet
Number of	Total number of	mIoU in %
experts	parameters	top-1	top-2	top-4	top-8
2	352,252	52.09 (+0.29)	53.86 (+2.06)	-	-
4	352,684	48.64	53.84 (+2.04)	51.97 (+0.17)	-
8	353,548	51.68	55.70 (+3.90)	50.98	54.16(+2.36)
16	355,276	52.07 (+0.27)	52.19 (+0.39)	52.05 (+0.25)	50.42

Table 4: Varying number of experts and parameter k for the top-

k

routing evaluated on Cityscapes . PatchConvMoE layers with 2Conv-GAP gate, other settings follow Table 1 .

Experts	Parameters	Routing	mIoU in %
8 independent experts	353,548	top-1	51.68
		top-2	55.70 (+3.90)
8 independent + 1 shared experts	353,692	top-1	51.45
		top-2	50.21
8 independent + 2 shared experts	353,836	top-1	54.89 (+3.09)
		top-2	50.14

Table 5: Independent vs. shared experts in the PatchConvMoE layer in ENet with 2Conv-GAP gate and patch size 128

\times

128 px evaluated on Cityscapes , other settings follow Table 1 .

Num MoE	MoE Layer position	Patch	Total number of	mIoU
layers		size, px	parameters	in %
ENet
1	3 $\times$ 3 conv layer in first encoder regular block ( regular1_1 )	64 $\times$ 64	369,532	53.56 (+1.76)
1	3 $\times$ 3 conv layer in bridge regular block ( regular3_4 )	32 $\times$ 32	419,068	54.80 (+3.00)
1	Last 3 $\times$ 3 conv layer in the decoder ( regular5_1 )	128 $\times$ 128	353,548	55.70 (+3.90)
2	regular3_4 and regular5_1	32 $\times$ 32,128 $\times$ 128	420,956	36.46

Table 6: Number and positions of MoE layers replacing convolutional layers evaluated on Cityscapes . We use architecture-specific layer names to facilitate reproducibility. PatchConvMoE layers with 2Conv-GAP gate, other settings follow Table 1 .

Model	$\mathcal{L}_{switch}$		$\mathcal{L}_{imp}$		$\mathcal{L}_{entropy}$
	NRE	TEC	NRE	TEC	NRE	TEC
ENet	0.9255	0.4394	0.5764	0.7425	0.8877	0.5024
ERFNet	0.9686	0.5183	0.8057	0.7790	0.8408	0.5563
U-Net	0.9887	0.4409	0.9880	0.4894	0.9874	0.5330
LR-ASPP	0.9951	0.2951	0.7701	0.6193	0.9468	0.4931
DeepLabv3+	0.9660	0.4379	0.9606	0.3888	0.8629	0.4724

Table 7: Routing collapse during inference for the PatchConvMoE models with 2Conv-GAP gate and 8 experts on the Cityscapes data: normalized routing entropy (NRE), where lower values mean routing collapse, and top expert concentration (TEC), where higher values mean routing collapse. Cases with the most stable routing are marked bold .

Panel of Figure 3 — Figure 3: Routing heatmaps during inference for models with one convolutional layer replaced with a PatchConvMoE layer in the decoder with 8 experts, 2Conv-GAP gate, top-2 routing, $\mathcal{L}_{entropy}$ , and the 3 $\times$ 3 grid for patchification evaluated on Cityscapes .

Panel of Figure 4 — Figure 4: Expert co-routing frequency during inference. The settings follow Figure 3 .

Explore similar work

When Does Sparse MoE Help in Vision? The Role of Backbone Compute Leverage in Sparse Routing
May 15, 2026Libo Sun, Po-wei Harn, Peixiong He +1Sparse Mixture-of-Experts Expert Routing
Department of Computer Science and Software Engineering Auburn University, Auburn, AL 36849, USA · Department of Information Management National Central University, Taoyuan 320317, Taiwan · Department of Computer Science and Software Engineering Auburn University
Path-Constrained Mixture-of-Experts
Mar 18, 2026Zijin Gu, Tatiana Likhomanenko, Vimal Thilak +2Mixture-of-Experts Language Models Sparse Mixture-of-Experts
Apple · †Google
Mixture of Channel Experts: Static Sparse Supports with Input-Adaptive Mixing for Pointwise Projections
Aug 24, 2026Elian Iluk, Gil Ben-ArtziSparse Mixture-of-Experts Mixture-of-Experts Models
School of Computer Science, Ariel University, Israel

May 15, 2026J/K move · Enter open · S save

When Does Sparse MoE Help in Vision? The Role of Backbone Compute Leverage in Sparse Routing

Libo Sun, Po-wei Harn, Peixiong He, Xiao Qin

Department of Computer Science and Software Engineering Auburn University, Auburn, AL 36849, USA · Department of Information Management National Central University, Taoyuan 320317, Taiwan · Department of Computer Science and Software Engineering Auburn University

Sparse Mixture-of-Experts Expert Routing Mixture-of-Experts Models Image Classification

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) networks promise favorable accuracy-compute trade-offs, yet practical vision deployments are hindered by expert collapse and limited end-to-end efficiency gains. We study when sparse top-

k

routing with hard capacity constraints helps in vision classification, evaluated under multi-seed protocols on four benchmarks (CIFAR-10/100, Tiny-ImageNet, ImageNet-1K). We observe a compute-leverage pattern: positive accuracy gaps require a substantial fraction

ρ

of total FLOPs to be routed; at ImageNet scale this is necessary but not sufficient, as multi-expert routing (

k \geq 2

) is additionally required. Two controlled experiments isolate these factors. A hidden-size sweep on CIFAR-10 yields both predicted sign reversals across standard and depthwise backbones, ruling out backbone family as the active variable. An ImageNet-1K ablation that varies only top-

k

-- holding architecture, initialization, and

ρ

fixed -- reverses the gap from positive to negative across all five seeds. A per-sample variant of Soft MoE that softmaxes over experts rather than the batch rescues CIFAR-100 above the dense baseline, identifying batch-axis dispatch as the dominant failure mode in per-sample CNN settings. Code and aggregate results: https://github.com/libophd/sparse-moe-vision-rho.

Abstract

Explore similar work

When Does Sparse MoE Help in Vision? The Role of Backbone Compute Leverage in Sparse Routing

Path-Constrained Mixture-of-Experts

Mixture of Channel Experts: Static Sparse Supports with Input-Adaptive Mixing for Pointwise Projections