IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts

Ran Cheng; Longfei Xu; Zheng Liu; Kaikui Liu; Xiangxiang Chu

cs.LGSep 18, 2026

IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts

Authors: Ran Cheng, Longfei Xu, Zheng Liu, Kaikui Liu, Xiangxiang Chu

Organizations: DreamX, Alibaba Group

Abstract

Mixture-of-Experts (MoE) scales capacity, but existing designs cannot set three quantities independently. For a single token, participation is how many experts contribute knowledge to its output, execution is how many are actually computed (compute cost), and materialization is how many expert-sized parameter sets must be built and stored (memory cost). Sparse routing keeps execution and materialization low, but shrinks participation: for each token, only a few experts contribute. Dense output-mixing restores full participation, but its execution grows with the number of experts. Parameter-merging keeps execution at one expert, but its materialization grows with the number of routing decisions. We propose IntBMoE, a block-conditioned MoE that decouples all three by pairing dense expert composition with sparse block execution. Its blocks come from a small learned codebook, one per entry. At each internal layer, a lightweight hypernetwork merges all expert bases in that layer's pool into one composed expert. Participation is full, because every composed expert draws on the entire pool. Execution stays sparse, because a router sends each token to only a few blocks. Materialization is bounded, because the codebook, not the input, fixes how many blocks exist. Dual-Path Residual Gating (DPRG) further couples two independently composed paths through multiplicative gating. Experiments on image classification show consistent gains over representative sparse and dense MoE baselines. Additional experiments on language modeling and sequential recommendation validate its generalization beyond vision. IntBMoE is fully deployed in AMap's generative recommendation system, serving hundreds of millions of users under a 60ms latency budget, with a 2.4% relative UVCTR gain in online A/B testing. Our code is available at https://github.com/AMAP-ML/DreamX-Rec/.

Figures & tables

Figure 1: Comparison of MoE design strategies. Filled circles indicate that a strategy satisfies the corresponding property, while hollow circles indicate otherwise.

Figure 2: Overview of IntBMoE. The left panel shows independent IntBMoE modules integrated into a multi-layer Transformer backbone. The right panel shows block-level expert parameter synthesis and token-level Top-

k

block routing and execution.

Method	ACC@1 $\uparrow$	ACC@5 $\uparrow$	Params (M)	Activated (M)	FLOPs (G)
Dense	0.6640	0.8769	3.938	3.938	1.453
Switch Transformer	0.7004	0.8937	24.070	3.951	1.458
DeepSeek-V3 MoE	0.7078	0.8969	24.004	6.306	2.382
ReMoE	0.6971	0.8946	24.070	4.574	1.458
V-MoE	0.7167	0.9030	24.070	5.135	1.920
Expert Choice	0.7170	0.9030	24.070	5.135	1.958

Table 1: Main results on ImageNet-1K.

\uparrow

indicates that higher values are better. Best results are highlighted with a blue background , and second-best results are shown in blue . Params denotes the total number of trainable parameters in the model; Activated Params denotes the number of trainable parameters activated as a single patch token passes through the entire model. FLOPs denotes the inference computational cost per image, measured with a batch size of 1.

Metric	Full	1-Layer	Fixed $\lambda$	w/o Gate	w/o Shared	w/o Filter	Softmax Coeff.
Top-1 $\uparrow$	0.7376	0.6878	0.7244	0.6804	0.7332	0.7372	0.7263
Top-5 $\uparrow$	0.9148	0.8910	0.9078	0.8831	0.9108	0.9132	0.9072

Table 2: Component ablations on ImageNet-1K.

Figure 3: Sensitivity of IntBMoE to architectural hyperparameters on ImageNet-1K. Each panel reports Top-1 accuracy while varying one hyperparameter; higher values are better. All panels use the same vertical scale. Orange points and gray horizontal lines mark the default configuration

(K,E,k,L)=(8,16,2,2)

and its accuracy, respectively.

Figure 4: Top-1 accuracy drop on ImageNet-1K after removing individual expert bases from one MoE layer at a time. The vertical axis uses a logarithmic scale.

Figure 5: Class-conditioned block routing for the same eight representative ImageNet-1K classes at Transformer layers 4 and 6. Each cell shows the token-assignment proportion for a class and block minus the uniform allocation of 12.5%, in percentage points. Positive (red) and negative (blue) values indicate above- and below-uniform use of a block, respectively. Each Top-

k

selection is counted as one token–block assignment.

Figure 6: Expert-composition coefficients learned by the eight blocks in Transformer layers 0, 2, 4, and 6 on ImageNet-1K, ordered from shallow to deep along the backbone. Rows represent blocks and columns represent expert bases. Each block’s coefficient vector is

\ell_{2}

-normalized only for visualization. Red and blue denote positive and negative coefficients, respectively.

Figure 7: Peak inference memory and inference FLOPs of cached and uncached IntBMoE as the number of experts

E

increases. Both quantities are measured for inference on a single image with a batch size of 1 using the ImageNet-1K model configuration. The horizontal axis uses a

\log_{2}

scale.

Method	MiniPile		IntTravel
Method	Loss $\downarrow$	PPL $\downarrow$	HR@1 $\uparrow$	HR@5 $\uparrow$	NDCG@5 $\uparrow$
Dense	2.8131	16.6621	0.6748	0.8659	0.7785
Switch Transformer	2.7156	15.1135	0.6798	0.8667	0.7812
DeepSeek-V3 MoE	2.7119	15.0580	0.6805	0.8674	0.7818
ReMoE	2.7346	15.4034	0.6833	0.8685	0.7836
DynMoE	2.7108	15.0406	0.6818	0.8677	0.7826

Table 3: Generalization results on MiniPile and IntTravel.

\uparrow

indicates that higher values are better, and

\downarrow

indicates that lower values are better. Best results per column are highlighted with a blue background , and second-best results are shown in blue .

Variant	MiniPile		IntTravel
Variant	Loss $\downarrow$	PPL $\downarrow$	HR@1 $\uparrow$	HR@5 $\uparrow$	NDCG@5 $\uparrow$
IntBMoE	2.6802	14.5878	0.6852	0.8692	0.7850
1-Layer	2.7174	15.1410	0.6824	0.8680	0.7828
Fixed $\lambda$	2.6938	14.7871	0.6844	0.8685	0.7827
w/o Gate	2.7660	15.8947	0.6820	0.8676	0.7828
w/o Shared	2.7281	15.3035	0.6846	0.8689	0.7844

Table 4: Component ablations on MiniPile and IntTravel.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Dataset	$H$	$d$	$n_{\mathrm{heads}}\!\times\!d_{\mathrm{head}}$	$d_{r}$	$E$	BS	Input	$\lambda_{0}$	Training
ImageNet-1K	8	192	$3\!\times\!64$	64	16	256	$224^{2}$ ( $p=16$ )	1	200 epochs
MiniPile	18	768	$12\!\times\!64$	64	8	8	1024	1	1 epoch
IntTravel	8	96	$1\!\times\!96$	64	32	64	126	0	610K steps

Appendix

Table 5: Dataset-specific model and training configurations.

Explore similar work

MoRE: Scaling mixture of experts with hardware-aware low-rank routing
Sep 28, 2026Honam Wong, Surbhi Goel, Enric Boix-AdseràParameter-Free Mixture-of-Experts Routing Low-Rank Matrix Decomposition
University of Pennsylvania · The Wharton School, University of Pennsylvania
OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale
Feb 5, 2026Jingze Shi, Zhangyang Peng, Yizhang Zhu +3Mixture-of-Experts Inference LLM Inference Acceleration
The Hong Kong University of Science and Technology (Guangzhou) · Beijing Academy of Artificial Intelligence
Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation
Aug 11, 2026Gongli Zhang, Zhulin Liu, C. L. Philip ChenMixture of Experts Mixture-of-Experts Inference
Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science and Engineering, South China University of Technology, Guangzhou 510006, China · Pazhou Lab, Guangzhou 510335, China · Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou 510641, China

Sep 28, 2026J/K move · Enter open · S save

MoRE: Scaling mixture of experts with hardware-aware low-rank routing

Honam Wong, Surbhi Goel, Enric Boix-Adserà

University of Pennsylvania · The Wharton School, University of Pennsylvania

Parameter-Free Mixture-of-Experts Routing Low-Rank Matrix Decomposition Mixture-of-Experts Inference LLM Inference Acceleration Mixture-of-Experts Models

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) layers are central to frontier language models, and recent architectures push toward more and smaller experts. In this regime, the standard linear router becomes a bottleneck: with

M

experts and hidden dimension

h

, its per-token cost

Θ(Mh)

dominates the MoE layer once

M

is large. We introduce MoRE (Mixture of Rank-reduced-routed Experts), which factorizes the router weight matrix at rank

r

and reduces the routing cost to

O((h + M)r)

. We prove that rank logarithmic in

M

suffices for routing expressivity when the number of active experts is fixed, and is necessary up to precision factors. We also prove that logarithmic rank preserves load balance in a Gaussian memorization model, and training on a synthetic phonebook task shows that low rank does not hurt memorization. At matched active FLOPs, the factorization allows a factor of

Θ(h/r)

more experts. To realize this gain in wall-clock time, we design a fused Triton kernel at inference that avoids expensive memory operations on HBM. Empirically, MoRE improves memorization on the phonebook task and performance on knowledge-intensive Q&A benchmarks after pretraining, while matching reasoning ability. Code available at https://github.com/Matheart/MoRE_code.

Abstract

Explore similar work

MoRE: Scaling mixture of experts with hardware-aware low-rank routing

OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale

Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation