cs.AINov 4, 2026

SafeStart: Mitigating the Overlooked Safety Cost of Active Expert Reduction in MoE LLM Serving

Authors: Erfan Baghaei Potraghloo, Saeid Shokoufa, Seyedarmin Azizi, Arya Fayyazi, Souvik Kundu, Mehdi Kamal, Massoud Pedram

Organizations: University of Southern California, Los Angeles, CA, USA · Intel Labs, USA

Abstract

Mixture-of-Experts (MoE) language models send each token through a few of many expert subnetworks. Serving methods make these models faster by running even fewer experts per token, and they are validated on accuracy benchmarks. We show that these methods can weaken the model's refusal of harmful requests, and we introduce SafeStart, which largely restores refusal while keeping most of the speedup. In vLLM on one RTX A6000, keeping about two thirds of the selected experts speeds up offline decoding by 15 to 28% at batch size 8. We find that, at this setting, under a jailbreak that tells the model not to refuse, the share of harmful requests that the model answers rises by 18.4 percentage points on Qwen1.5-MoE, 9.5 on OLMoE, and 6.6 on DeepSeek-V2-Lite, the three of nine models where it rises significantly, while MMLU accuracy drops by less than two points. We discover that most of this increase arises while the model processes the prompt and generates its first few tokens. SafeStart therefore runs all experts for the prompt and the first eight generated tokens, and the reduced set of experts afterwards, through a fused kernel in vLLM. With experts removed, SafeStart keeps at least 85% of the offline decode speedup at batch sizes 8 and 32 and brings the change under this jailbreak to between -2.1 and +1.3 points for three expert-reduction methods on these three models. On 889 held-out requests, in an analysis fixed before generation, truncation raises harmful compliance by 5.8 to 21.4 points, and SafeStart leaves a rise of 0.6 to 2.6 points, non-inferior to the original model on two of the three models. Our results show that faster MoE serving need not cost safety: a serving system can keep most of the gain from running fewer experts by running all of them at the start of each response, and it should test refusal, not only accuracy, before it enables such a method.

Figures & tables

Appendix figures & tables43 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Frequency Is Not Sensitivity Identifying Safety-Sensitive Experts in Sparse MoE LLM

    Oct 2, 2026Md Nurul Absar Siddiky, Liuwan Zhu, Yingfei DongLLM SafetyMixture-of-Experts Language Models

  2. Decomposing Refusal Steering in Mixture-of-Experts Models

    Jun 2, 2026Anna C. Marbut, Daniel R. Olson, Travis J. WheelerMixture-of-Experts Language ModelsLanguage Model Steering

  3. RouteHijack: Routing-Aware Attack on Mixture-of-Experts LLMs

    May 1, 2026Zhiyuan Xu, Joseph Gardiner, Sana Belguith +1Mixture-of-Experts Language ModelsAdversarial Attacks on LLMs