IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts

Ran Cheng; Longfei Xu; Zheng Liu; Kaikui Liu; Xiangxiang Chu

cs.LGSep 18, 2026

IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts

Authors: Ran Cheng, Longfei Xu, Zheng Liu, Kaikui Liu, Xiangxiang Chu

Organizations: DreamX, Alibaba Group

Abstract

Mixture-of-Experts (MoE) scales capacity, but existing designs cannot set three quantities independently. For a single token, participation is how many experts contribute knowledge to its output, execution is how many are actually computed (compute cost), and materialization is how many expert-sized parameter sets must be built and stored (memory cost). Sparse routing keeps execution and materialization low, but shrinks participation: for each token, only a few experts contribute. Dense output-mixing restores full participation, but its execution grows with the number of experts. Parameter-merging keeps execution at one expert, but its materialization grows with the number of routing decisions. We propose IntBMoE, a block-conditioned MoE that decouples all three by pairing dense expert composition with sparse block execution. Its blocks come from a small learned codebook, one per entry. At each internal layer, a lightweight hypernetwork merges all expert bases in that layer's pool into one composed expert. Participation is full, because every composed expert draws on the entire pool. Execution stays sparse, because a router sends each token to only a few blocks. Materialization is bounded, because the codebook, not the input, fixes how many blocks exist. Dual-Path Residual Gating (DPRG) further couples two independently composed paths through multiplicative gating. Experiments on image classification show consistent gains over representative sparse and dense MoE baselines. Additional experiments on language modeling and sequential recommendation validate its generalization beyond vision. IntBMoE is fully deployed in AMap's generative recommendation system, serving hundreds of millions of users under a 60ms latency budget, with a 2.4% relative UVCTR gain in online A/B testing. Our code is available at https://github.com/AMAP-ML/DreamX-Rec/.

Figures & tables

Figure 1: Comparison of MoE design strategies. Filled circles indicate that a strategy satisfies the corresponding property, while hollow circles indicate otherwise.

Figure 2: Overview of IntBMoE. The left panel shows independent IntBMoE modules integrated into a multi-layer Transformer backbone. The right panel shows block-level expert parameter synthesis and token-level Top-

k

block routing and execution.

Method	ACC@1 $\uparrow$	ACC@5 $\uparrow$	Params (M)	Activated (M)	FLOPs (G)
Dense	0.6640	0.8769	3.938	3.938	1.453
Switch Transformer	0.7004	0.8937	24.070	3.951	1.458
DeepSeek-V3 MoE	0.7078	0.8969	24.004	6.306	2.382
ReMoE	0.6971	0.8946	24.070	4.574	1.458
V-MoE	0.7167	0.9030	24.070	5.135	1.920
Expert Choice	0.7170	0.9030	24.070	5.135	1.958

Table 1: Main results on ImageNet-1K.

\uparrow

indicates that higher values are better. Best results are highlighted with a blue background , and second-best results are shown in blue . Params denotes the total number of trainable parameters in the model; Activated Params denotes the number of trainable parameters activated as a single patch token passes through the entire model. FLOPs denotes the inference computational cost per image, measured with a batch size of 1.

Metric	Full	1-Layer	Fixed $\lambda$	w/o Gate	w/o Shared	w/o Filter	Softmax Coeff.
Top-1 $\uparrow$	0.7376	0.6878	0.7244	0.6804	0.7332	0.7372	0.7263
Top-5 $\uparrow$	0.9148	0.8910	0.9078	0.8831	0.9108	0.9132	0.9072

Table 2: Component ablations on ImageNet-1K.

Figure 3: Sensitivity of IntBMoE to architectural hyperparameters on ImageNet-1K. Each panel reports Top-1 accuracy while varying one hyperparameter; higher values are better. All panels use the same vertical scale. Orange points and gray horizontal lines mark the default configuration

(K,E,k,L)=(8,16,2,2)

and its accuracy, respectively.

Figure 4: Top-1 accuracy drop on ImageNet-1K after removing individual expert bases from one MoE layer at a time. The vertical axis uses a logarithmic scale.

Figure 5: Class-conditioned block routing for the same eight representative ImageNet-1K classes at Transformer layers 4 and 6. Each cell shows the token-assignment proportion for a class and block minus the uniform allocation of 12.5%, in percentage points. Positive (red) and negative (blue) values indicate above- and below-uniform use of a block, respectively. Each Top-

k

selection is counted as one token–block assignment.

Figure 6: Expert-composition coefficients learned by the eight blocks in Transformer layers 0, 2, 4, and 6 on ImageNet-1K, ordered from shallow to deep along the backbone. Rows represent blocks and columns represent expert bases. Each block’s coefficient vector is

\ell_{2}

-normalized only for visualization. Red and blue denote positive and negative coefficients, respectively.

Figure 7: Peak inference memory and inference FLOPs of cached and uncached IntBMoE as the number of experts

E

increases. Both quantities are measured for inference on a single image with a batch size of 1 using the ImageNet-1K model configuration. The horizontal axis uses a

\log_{2}

scale.

Method	MiniPile		IntTravel
Method	Loss $\downarrow$	PPL $\downarrow$	HR@1 $\uparrow$	HR@5 $\uparrow$	NDCG@5 $\uparrow$
Dense	2.8131	16.6621	0.6748	0.8659	0.7785
Switch Transformer	2.7156	15.1135	0.6798	0.8667	0.7812
DeepSeek-V3 MoE	2.7119	15.0580	0.6805	0.8674	0.7818
ReMoE	2.7346	15.4034	0.6833	0.8685	0.7836
DynMoE	2.7108	15.0406	0.6818	0.8677	0.7826

Table 3: Generalization results on MiniPile and IntTravel.

\uparrow

indicates that higher values are better, and

\downarrow

indicates that lower values are better. Best results per column are highlighted with a blue background , and second-best results are shown in blue .

Variant	MiniPile		IntTravel
Variant	Loss $\downarrow$	PPL $\downarrow$	HR@1 $\uparrow$	HR@5 $\uparrow$	NDCG@5 $\uparrow$
IntBMoE	2.6802	14.5878	0.6852	0.8692	0.7850
1-Layer	2.7174	15.1410	0.6824	0.8680	0.7828
Fixed $\lambda$	2.6938	14.7871	0.6844	0.8685	0.7827
w/o Gate	2.7660	15.8947	0.6820	0.8676	0.7828
w/o Shared	2.7281	15.3035	0.6846	0.8689	0.7844

Table 4: Component ablations on MiniPile and IntTravel.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Dataset	$H$	$d$	$n_{\mathrm{heads}}\!\times\!d_{\mathrm{head}}$	$d_{r}$	$E$	BS	Input	$\lambda_{0}$	Training
ImageNet-1K	8	192	$3\!\times\!64$	64	16	256	$224^{2}$ ( $p=16$ )	1	200 epochs
MiniPile	18	768	$12\!\times\!64$	64	8	8	1024	1	1 epoch
IntTravel	8	96	$1\!\times\!96$	64	32	64	126	0	610K steps

Appendix

Table 5: Dataset-specific model and training configurations.

Explore similar work

Sep 28, 2026

MoRE: Scaling mixture of experts with hardware-aware low-rank routing

Mixture-of-Experts (MoE) layers are central to frontier language models, and recent architectures push toward more and smaller experts. In this regime, the standard linear router becomes a bottleneck: with

M

experts and hidden dimension

h

, its per-token cost

Θ(Mh)

dominates the MoE layer once

M

is large. We introduce MoRE (Mixture of Rank-reduced-routed Experts), which factorizes the router weight matrix at rank

r

and reduces the routing cost to

O((h + M)r)

. We prove that rank logarithmic in

M

suffices for routing expressivity when the number of active experts is fixed, and is necessary up to precision factors. We also prove that logarithmic rank preserves load balance in a Gaussian memorization model, and training on a synthetic phonebook task shows that low rank does not hurt memorization. At matched active FLOPs, the factorization allows a factor of

Θ(h/r)

more experts. To realize this gain in wall-clock time, we design a fused Triton kernel at inference that avoids expensive memory operations on HBM. Empirically, MoRE improves memorization on the phonebook task and performance on knowledge-intensive Q&A benchmarks after pretraining, while matching reasoning ability. Code available at https://github.com/Matheart/MoRE_code.

Honam Wong, Surbhi Goel, Enric Boix-Adserà

University of Pennsylvania · The Wharton School, University of Pennsylvania

Parameter-Free Mixture-of-Experts Routing Low-Rank Matrix Decomposition Mixture-of-Experts Inference LLM Inference Acceleration Mixture-of-Experts Models

Ask PDF

Feb 5, 2026

OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale

Mixture-of-Experts (MoE) architectures are evolving towards finer granularity to improve parameter efficiency. However, existing MoE designs face an inherent trade-off between the granularity of expert specialization and hardware execution efficiency. We propose OmniMoE, a system-algorithm co-designed framework that pushes expert granularity to its logical extreme. OmniMoE introduces vector-level Atomic Experts, enabling scalable routing and execution within a single MoE layer, while retaining a shared dense MLP branch for general-purpose processing. Although this atomic design maximizes capacity, it poses severe challenges for routing complexity and memory access. To address these, OmniMoE adopts a system-algorithm co-design: (i) a Cartesian Product Router that decomposes the massive index space to reduce routing complexity from O(N) to O(sqrt(N)); and (ii) Expert-Centric Scheduling that inverts the execution order to turn scattered, memory-bound lookups into efficient dense matrix operations. Validated on seven benchmarks, OmniMoE (with 1.7B active parameters) achieves 50.9% zero-shot accuracy across seven benchmarks, outperforming coarse-grained (e.g., DeepSeekMoE) and fine-grained (e.g., PEER) baselines. Crucially, OmniMoE reduces inference latency from 73ms to 6.7ms (a 10.9-fold speedup) compared to PEER, demonstrating that massive-scale fine-grained MoE can be fast and accurate. Our code is open-sourced at https://github.com/flash-algo/omni-moe.

Jingze Shi, Zhangyang Peng, Yizhang Zhu +3

The Hong Kong University of Science and Technology (Guangzhou) · Beijing Academy of Artificial Intelligence

Mixture-of-Experts Inference LLM Inference Acceleration Expert Routing Mixture-of-Experts Models

Ask PDF

Aug 11, 2026

Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation

Mixture-of-experts (MoE) models have recently moved beyond routing a fixed number of complete experts. Shared-expert designs preserve reusable knowledge, fine-grained methods vary computation within experts, and dynamic routers adapt the number of active experts. Yet these decisions are usually made independently, overlooking a basic dependency: extracting reusable computation changes both what remains and how much expert capacity the remainder needs. We study this dependency by decomposing sparsely upcycled feed-forward experts into key-value channels. Co-activated experts align at a subset of value positions; removing these positions changes expert preference; and greater shared coverage is associated with lower residual expert demand. These observations lead to one principle: share first, then route what remains. We instantiate it in UniF-MoE, a unified framework for token-adaptive MoE computation. Each expert is partitioned into aligned blocks. A shared-demand score sets the shared block count and pathway weight, key prototypes select the shared content, and the complementary demand determines the residual expert count through cumulative routing mass. A Gram regularizer separates and normalizes router embeddings, promoting diverse routing directions, sparse expert overlap, and a simple routing geometry. Experiments on DomainBed and GLUE show that this unified design improves predictive performance over representative static and dynamic MoEs while reducing activated computation, inference latency, and memory. Code is available at https://github.com/existence0420/UniF-MoE.

Gongli Zhang, Zhulin Liu, C. L. Philip Chen

Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science and Engineering, South China University of Technology, Guangzhou 510006, China · Pazhou Lab, Guangzhou 510335, China · Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou 510641, China

Mixture of Experts Mixture-of-Experts Inference Expert Routing Mixture-of-Experts Models

Ask PDF