cs.LGOct 6, 2026

MASKerade: Token-Routed Mask Experts for Dense-to-MoE Upcycling

Authors: Mingyuan Zhang, Yue Bai, Zhongruo Wang, Yupin Huang, Yiyang Huang, Hailing Wang, Huimin Zeng, Yun Fu

Organizations: Department of Electrical and Computer Engineering, Northeastern University · Khoury College of Computer Science, Northeastern University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Sparsely activated Mixture-of-Experts (MoE) models increase model capacity without a proportional increase in per-token computation. Dense-to-MoE upcycling reuses pretrained dense models to construct such systems, commonly by copying feed-forward networks (FFNs) into independently trained experts. We introduce MASKerade, a dense-to-MoE training method that instead learns experts as sparse subnetworks of a frozen pretrained FFN. Each expert is defined by a learned binary mask, and a token-level router selects which masked FFNs to execute and combine. The router and mask scores are optimized jointly, while the underlying FFN weight values remain unchanged. This formulation supports neuron-structured, semi-structured, and unstructured experts within the same routing architecture. Our main configuration uses four 2:4 experts with top-2 routing, where two half-dense expert passes have the nominal FFN arithmetic of one dense pass, without requiring independent expert weight matrices. On five vision-language benchmarks with Qwen and Gemma backbones, this configuration achieves the highest performance among the compared baselines. Comparisons across mask granularities, routing interventions, and compute-matched controls distinguish the effects of learned connectivity from expert activation count. These results establish mask learning over frozen weights as a practical alternative for constructing token-routed MoE experts.

Figures & tables

Figure 1: Conventional MoE and MASKerade. Tokens select independent FFNs (left) or learned masks over one frozen FFN (right).
 MethodAct. Params.GQA ↑MME ↑POPE ↑SQA ↑TextVQA ↑
Qwen3-1.7BZero-shot ( Yang et al., 2025 )1.72B25.97962.083.5960.9332.38
Dense MLP1.72B58.41 ±\pm 0.351300.4 ±\pm 18.485.01 ±\pm 0.2461.16 ±\pm 0.3249.19 ±\pm 0.32
MoEfication ( Zhang et al., 2022 )1.46B53.60 ±\pm 0.201280.5 ±\pm 15.884.75 ±\pm 0.3360.11 ±\pm 0.9649.94 ±\pm 0.55
LLaVA-MoLE ( Chen et al., 2024a )1.91B58.05 ±\pm 0.381295.0 ±\pm 18.384.97 ±\pm 0.3660.24 ±\pm 0.0950.25 ±\pm 0.06
PESC ( Wu et al., 2024 )1.73B56.81 ±\pm 0.361309.1 ±\pm 11.384.31 ±\pm 0.2961.60 ±\pm 0.5749.95 ±\pm 0.25
LLaMA-MoE ( Zhu et al., 2024 )1.46B57.85 ±\pm 0.161233.3 ±\pm 17.585.45 ±\pm 0.3960.56 ±\pm 0.5850.90 ±\pm 0.58
Table 1: Main comparison with existing methods on small models. All methods start from the same Zero-shot checkpoint. We learn masks as experts over frozen FFN weights. Best and second-best results in each metric are shown in bold and underlined , respectively.
Figure 2: Initialization loss for 2:4 masks on Qwen3-1.7B.
 MethodAct. Params.GQA ↑MME ↑POPE ↑SQA ↑TextVQA ↑
Qwen3.6-27BZero-shot ( Yang et al., 2025 )26.90B61.391692.186.2392.7471.65
Dense MLP26.90B64.131716.086.9093.3873.41
MoEfication ( Zhang et al., 2022 )22.62B61.051691.286.1591.5572.00
LLaVA-MoLE ( Chen et al., 2024a )29.82B65.711702.586.4792.1774.08
PESC ( Wu et al., 2024 )26.94B65.611717.487.0891.1877.33
LLaMA-MoE ( Zhu et al., 2024 )22.62B64.331715.586.7893.4678.63
Table 2: Main comparison with existing methods on large models. All methods start from the same Zero-shot checkpoint. We learn masks as experts over frozen FFN weights. Best and second-best results in each metric are shown in bold and underlined , respectively.
 MethodGQAMMEPOPESQATextVQA
Qwen3-1.7BZero-shot25.97962.083.5960.9332.38
Dense MLP58.41 ±\pm 0.351300.4 ±\pm 18.485.01 ±\pm 0.2461.16 ±\pm 0.3249.19 ±\pm 0.32
Random 2:443.36 ±\pm 0.30871.8 ±\pm 21.382.15 ±\pm 0.1640.75 ±\pm 0.4228.22 ±\pm 0.48
Magnitude 2:451.54 ±\pm 0.261127.6 ±\pm 14.785.38 ±\pm 0.5358.34 ±\pm 0.8043.15 ±\pm 0.71
Wanda 2:4 ( Sun et al., 2024 )51.97 ±\pm 0.531148.2 ±\pm 23.883.96 ±\pm 0.4460.81 ±\pm 0.6444.01 ±\pm 0.27
Wanda Unstructured ( Sun et al., 2024 )52.24 ±\pm 0.481187.4 ±\pm 37.486.06 ±\pm 0.2763.01 ±\pm 0.5245.21 ±\pm 0.22
Table 3: Mask-based MoE comparison. Best and second-best results in each metric are shown in bold and underlined , respectively.
SettingGQA ↑MME ↑POPE ↑SQA ↑TextVQA ↑
Qwen3-1.7B     
Trained62.00 ±\pm 0.281332.4 ±\pm 11.986.49 ±\pm 0.1663.48 ±\pm 0.5752.50 ±\pm 0.37
Random (frozen during training)60.02 ±\pm 0.071307.7 ±\pm 21.085.96 ±\pm 0.4561.27 ±\pm 1.0550.97 ±\pm 0.28
Random (after training)59.30 ±\pm 0.391297.8 ±\pm 12.684.60 ±\pm 0.1160.32 ±\pm 0.6649.71 ±\pm 0.58
Label-permuted56.86 ±\pm 0.521270.0 ±\pm 12.283.82 ±\pm 0.6660.82 ±\pm 0.8449.63 ±\pm 0.61
Gemma3-1B     
Table 4: Router learning and inference-time interventions with four 2:4 experts and top-2 routing. Frozen-random routers remain fixed during mask learning.
Table 7
ConfigurationMasksCalculationAct. paramsGQA ↑MME ↑POPE ↑SQA ↑TextVQA ↑
MoEfication ( Zhang et al., 2022 )--1.46B53.60 ±\pm 0.201280.5 ±\pm 15.884.75 ±\pm 0.3360.11 ±\pm 0.9649.94 ±\pm 0.55
LLaMA-MoE ( Zhu et al., 2024 )--1.46B57.85 ±\pm 0.161233.3 ±\pm 17.585.45 ±\pm 0.3960.56 ±\pm 0.5850.90 ±\pm 0.58
DIVE ( Feng et al., 2025 )--1.46B57.87 ±\pm 0.161300.6 ±\pm 16.783.80 ±\pm 0.3460.84 ±\pm 0.7650.55 ±\pm 0.62
ToMoE ( Gao et al., 2026 )--1.46B58.54 ±\pm 0.221302.2 ±\pm 16.285.66 ±\pm 0.4061.61 ±\pm 0.5050.77 ±\pm 0.22
1 mask only2:450%×150\%\times 11.46B56.51 ±\pm 0.321258.1 ±\pm 15.584.50 ±\pm 0.2559.67 ±\pm 0.5649.13 ±\pm 0.68
[Ours] 4e/top-21:425%×225\%\times 21.46B61.40 ±\pm 0.321322.2 ±\pm 11.885.96 ±\pm 0.4661.79 ±\pm 0.5350.94 ±\pm 0.34
Table 7: Matched nominal activation budgets on Qwen3-1.7B. Calculation denotes retained density per expert multiplied by the number of activated experts.
Figure 3: Learned mask structure on Qwen3-1.7B: (a) 2:4 mask Jaccard overlap; (b)-(d) layer-wise sharing of retained connections.
Figure 4: Layer-local expert selection by training source on Qwen3-1.7B. Panels show layers 0, 14, 26, and an equal index-wise mean over all 14 converted layers. Expert IDs are not aligned across layers. 50% indicates balanced top-2 usage.
Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

SettingValue
Training data665K multimodal instruction mixture
Converted FFNsEven-indexed layers: Qwen3-1.7B, 14/28; Gemma3-1B, 13/26; Qwen3.6-27B, 32/64; Gemma4-31B, 30/60
Experts and routingFour experts with deterministic top-2 in training and inference using dropless dispatch
Default masksSemi-structured 2:4
Trainable LM parametersMask scores and routers
Pretrained LM weightsFrozen
Appendix
Table 9: Default MASKerade implementation details.
Figure 5: Random versus importance initialization across mask granularities on Qwen3-1.7B.
Figure 6: Random versus importance initialization across mask granularities on Gemma3-1B.
Table 14
ConfigurationMasksCalculationAct. paramsGQA ↑\uparrowMME ↑\uparrowPOPE ↑\uparrowSQA ↑\uparrowTextVQA ↑\uparrow
MoEfication ( Zhang et al., 2022 )--0.84B55.36 ±\pm 0.281212.9 ±\pm 13.580.92 ±\pm 0.3550.33 ±\pm 0.5540.68 ±\pm 0.58
LLaMA-MoE ( Zhu et al., 2024 )--0.84B57.67 ±\pm 0.331234.7 ±\pm 11.285.00 ±\pm 0.4252.71 ±\pm 0.6340.96 ±\pm 0.67
DIVE ( Feng et al., 2025 )--0.85B51.66 ±\pm 0.271129.3 ±\pm 13.282.80 ±\pm 0.2850.84 ±\pm 0.6341.79 ±\pm 0.58
ToMoE ( Gao et al., 2026 )--0.84B53.96 ±\pm 0.151123.0 ±\pm 17.981.37 ±\pm 0.4351.98 ±\pm 0.5240.63 ±\pm 0.32
1 mask only2:450%×150\%\times 10.84B58.30 ±\pm 0.301239.9 ±\pm 13.784.87 ±\pm 0.3852.59 ±\pm 0.6940.07 ±\pm 0.68
[Ours] 4e/top-21:425%×225\%\times 20.84B59.09 ±\pm 0.331248.7 ±\pm 15.585.11 ±\pm 0.3652.99 ±\pm 0.6142.47 ±\pm 0.64
Appendix
Table 12: Matched nominal activation budgets on Gemma3-1B. Calculation denotes retained density per expert times the number of activated experts. Only protocol-matched runs are included.
Figure 7: Learned mask structure on Gemma3-1B, averaged over three runs per family: (a) 2:4 Jaccard overlap; (b)-(d) layer-wise sharing of retained connections, normalized by dense weight positions.
Figure 8: Layer-local expert selection by training source on Gemma3-1B. Panels show layers 0, 12, 24, and the unaligned index-wise mean over all 13 converted layers. 50% denotes balanced top-2 usage.
Table 18
Table 19
LayerToken typeCosineDistancee0e_{0}e1e_{1}e2e_{2}e3e_{3}
Qwen3-1.7B
0Vision0.7000.39013.6013.7013.5013.80
0Prompt0.7800.3309.209.309.109.40
0Response0.7200.37011.6011.4011.8012.00
14Vision0.6500.42052.0053.0051.0054.00
14Prompt0.7300.37044.0045.0046.0044.50
Appendix
Table 19: Identical-input expert outputs by token type. Cosine and Distance are pairwise averages. Distance normalizes L2L_{2} differences by summed output norms, and eie_{i} denotes the mean output norm.
LayerSlotAlt.Mean Δ\Delta NLLMedianΔ\Delta NLL >0>0 (%)Δ\Delta AccC →\to WW →\to C
Qwen3-1.7B
000+0.025+0.00862.0-1.001.500.50
001+0.040+0.01267.0-1.502.000.50
010+0.015+0.00460.0-0.501.000.50
011+0.020+0.00663.0-1.001.500.50
1400+0.080+0.03573.0-3.004.001.00
Appendix
Table 20: Single-layer expert replacement with routing weights fixed. Slot and Alt. give zero-based router ranks among selected and unselected experts. Δ\Delta compares with original routing. C/W denote correct/incorrect answers. Accuracy changes and flips are in percentage points and percent, respectively.

Explore similar work

CardsList
Sep 28, 2026cs.LG

MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inference

Mixture-of-experts (MoE) language models often exceed the memory of a single GPU. Expert offloading keeps most experts in host memory and loads them on demand, so decoding speed depends on how many experts each token must fetch. Caching and prefetching reduce this cost only as far as the routing allows. Router-only fine-tuning can reshape the routing to reuse experts, but it keeps the experts frozen, so they cannot adapt to the tokens the new routing sends them. We propose MaskCoFT, a masked co-adaptive fine-tuning method that trains routers and experts together with the cross-entropy loss alone. During fine-tuning, a learnable binary mask restricts the Top-K routing of each layer to a subset of experts, and the experts adapt to the tokens redirected to them. At inference, the learned mask becomes a soft prior that re-ranks experts, so every expert remains selectable. We simulate a GPU cache of 4 experts per layer for Mixtral-8x7B and 12 for DeepSeek-V2-Lite. MaskCoFT cuts expert fetches per token by 23.7% and 10.1% relative to the base model. In real offloading system serving, it lowers the time per output token by up to 16.4% and 5.5%, respectively. Its average accuracy over nine benchmarks stays above the base model by 0.92 and 0.53 points.

Junfeng Wu, Zehao Fan, Hadjer Benmeziane +3

Department of Industrial and Systems Engineering, Rensselaer Polytechnic Institute · Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute · IBM Research

Mixture-Of-ExpertsParameter-Efficient Fine-Tuning MethodsModel Fine-TuningOffloadingDeepseekCross-Entropy Losses
AskPDF
May 14, 2026cs.AI

BEAM: Binary Expert Activation Masking for Dynamic Routing in MoE

Mixture-of-Experts (MoE) architectures enhance the efficiency of large language models by activating only a subset of experts per token. However, standard MoE employs a fixed Top-K routing strategy, leading to redundant computation and suboptimal inference latency. Existing acceleration methods either require costly retraining with architectural changes or suffer from severe performance drop at high sparsity due to train-inference mismatch. To address these limitations, we propose BEAM (Binary Expert Activation Masking), a novel method that learns token-adaptive expert selection via trainable binary masks. With a straight-through estimator and an auxiliary regularization loss, BEAM induces dynamic expert sparsity through end-to-end training while maintaining model capability. We further implement an efficient custom CUDA kernel for BEAM, ensuring seamless integration with the vLLM inference framework. Experiments show that BEAM retains over 98% of the original model's performance while reducing MoE layer FLOPs by up to 85%, achieving up to 2.5×\times faster decoding and 1.4×\times higher throughput, demonstrating its effectiveness as a practical, plug-and-play solution for efficient MoE inference.

Juntong Wu, Jialiang Cheng, Qishen Yin +5

Taobao & Tmall Group of Alibaba · Shenzhen Graduate School, Peking University

Mixture-Of-Expert InferenceMixture-Of-Experts
AskPDF
Jun 1, 2026cs.LG

DOT-MoE: Differentiable Optimal Transport for MoEfication

The scaling of Large Language Models (LLMs) has driven significant performance gains but created substantial challenges in inference efficiency. While Mixture of Experts (MoEs) architectures address this by decoupling model size from inference cost, training MoEs from scratch is often unstable and compute intensive. Conversion of pre-trained dense models into sparse MoEs has emerged as an alternative solution; however, existing methods typically rely on heuristic neuron clustering or random splitting to partition the Feed-Forward Network (FFN) into experts. In this work, we propose DOT-MoE, a novel framework that formulates the decomposition of dense layers as a Differentiable Optimal Transport (DOT) problem. Instead of static heuristics, we model neuron assignment as a balanced transport problem, utilizing differentiable Sinkhorn-Knopp iterations to enforce strict expert capacity constraints. Furthermore, we utilize Straight-Through Estimators (STE) to jointly learn the discrete neuron-to-expert assignment and the token-to-expert routing policy end-to-end. Extensive experiments across multiple architectures and benchmarks demonstrate that DOT-MoE significantly outperforms structured pruning, heuristic clustering, and random-split baselines, retaining 90% of the original dense model's performance while reducing active parameters by 50%.

Udbhav Bamba, Arnav Chavan, Aryamaan Thakur +2

Amazon

Localized Lora-MoeInference CostDifferentiable Optimal Transport
AskPDF