Routing-Aware Safety Alignment for Mixture-of-Experts Models

Jiacheng Liang; Yuhui Wang; Tanqiu Jiang; Ting Wang

cs.LGFeb 4, 2026

Routing-Aware Safety Alignment for Mixture-of-Experts Models

Authors: Jiacheng Liang, Yuhui Wang, Tanqiu Jiang, Ting Wang

Organizations: Department of Computer Science Stony Brook University

Abstract

Mixture-of-Experts (MoE) language models introduce unique challenges for safety alignment due to their sparse routing mechanisms, which can enable degenerate optimization behaviors under standard full-parameter fine-tuning. In our preliminary experiments, we observe that naively applying full-parameter safety fine-tuning to MoE models can reduce attack success rates through routing or expert dominance effects, rather than by directly repairing Safety-Critical Experts. To address this challenge, we propose RASA, a routing-aware expert-level alignment framework that explicitly repairs Safety-Critical Experts while preventing routing-based bypasses. RASA identifies experts disproportionately activated by successful jailbreaks, selectively fine-tunes only these experts under fixed routing, and subsequently enforces routing consistency with safety-aligned contexts. Across two representative MoE architectures and a diverse set of jailbreak attacks, RASA achieves near-perfect robustness, strong cross-attack generalization, and substantially reduced over-refusal, while preserving general capabilities on benchmarks such as MMLU, GSM8K, and TruthfulQA. Our results suggest that robust MoE safety alignment benefits from targeted expert repair rather than global parameter updates, offering a practical and architecture-preserving alternative to prior approaches.

Figures & tables

Figure 1: Alignment shortcut in MoE safety training. Illustration of a failure mode where unsafe inputs may be routed to already-aligned experts, allowing safety loss to decrease without correcting Safety-Critical Experts, which remain bypassed and unaligned.

Figure 2: Overview of RASA: RASA alternates between selectively fine-tuning Safety-Critical Experts under fixed routing and optimizing router consistency using safe anchor distributions, preventing routing-based shortcut solutions.

Model	Attack Method	Harmlessness ( $\uparrow$ )			General ( $\uparrow$ )					Over-Refusal ( $\downarrow$ )
Model	Attack Method	Flip	Deepin	Pers	MMLU	GSM8K	TQA	MATH	GPQA	XsTest
O L M O E	Origin	0.47	0.10	0.24	0.38	0.62	0.73	0.27	0.25	0.17
	FlipAttack	1.00	0.18	0.20	0.39	0.66	0.74	0.29	0.28	0.12
	DeepInception	0.38	1.00	0.16	0.42	0.64	0.73	0.26	0.24	0.12
	Persuasion	0.39	0.58	1.00	0.38	0.64	0.71	0.25	0.26	0.24
	Mixed-Attack	1.00	1.00	0.98	0.42	0.71	0.72	0.28	0.27	0.24

Table 1: Model performance under different jailbreak attack scenarios. Comparison of harmlessness, general capability, and over-refusal for OLMoE and Qwen models under individual and mixed jailbreak attacks after targeted expert-level alignment. Harmlessness is judged by GPT-4o-mini; see Appendix D.4 for cross-judge validation with Claude Sonnet 4.6. Flip: FlipAttack, Deepin: DeepInception, Pers: Persuasion, TQA: TruthfulQA.

Training	SR-Flip	SR-Deepin	SR-Pers
Base	0.22	0.25	0.20
Flip	1.00	0.30	0.18
Deepin	0.44	0.99	0.22
Pers	0.55	0.54	0.96
Mixed	0.99	0.97	0.87

Table 2: Out-of-distribution generalization on StrongReject (OLMoE). Models trained on AdvBench intents, tested on disjoint StrongReject intents with different attack methods. SR-: StrongReject.

Attack	Model	Normal	Restored	$\Delta$
Flip	Origin	0.47	—	—
	Full-FT	0.99	0.61	-0.38
	RASA	1.00	0.94	-0.06
Deepin	Origin	0.10	—	—
	Full-FT	1.00	0.54	-0.46
	RASA	1.00	0.96	-0.04

Table 3: Routing restoration experiment across three attack types (OLMoE). Normal / Restored : harmlessness under the aligned model’s own routing vs. forcing the original model’s per-layer routing during prefill. Full-FT: full-parameter fine-tuning.

Attack	Model	KL	Mean Dev.	SCE Act.
Flip	Origin	—	—	0.0108
	Full-FT	0.0327	0.0038	0.0089
	RASA	0.0063	0.0015	0.0101
Deepin	Origin	—	—	0.0141
	Full-FT	0.0538	0.0047	0.0120
	RASA	0.0061	0.0021	0.0134

Table 4: Routing distribution shift on adversarial inputs (OLMoE). KL : KL divergence of the routing distribution relative to the original model. Mean Dev. : average per-expert activation frequency change. SCE Act. : activation rate of Safety-Critical Experts; lower values indicate adversarial inputs are routed away from unsafe experts.

Model	ASR
Base (unaligned)	0.85
Full-Param FT	0.60
RASA	0.12

Table 5: Attack success rate of the Routing-Evasion Attack (REA) on OLMoE. REA is a RASA-aware white-box adaptive attack that jointly optimizes for compliance and SCE avoidance.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
$\mathcal{M}(\Theta,\Phi)$	A Mixture-of-Experts (MoE) model with expert parameters $\Theta$ and router parameters $\Phi$ .
$L$	Number of MoE layers in the model.
$E$	Number of experts per layer.
$k$	Number of experts selected by the router per token (top- $k$ routing).
$G_{l}(\cdot;\Phi)$	Router function at layer l, parameterized by router parameters $\Phi$ .
$H_{l,e}(x)$	Binary indicator of whether expert $e$ in layer $l$ is selected by top-k routing for input $x$ .

Appendix

Table 6: Summary of Notation Used in the Methodology

Hyperparameter	Value
Training rounds	3
Expert learning rate	5e-5
Router learning rate	1e-4
Batch size	16
Max generation tokens	128
Threshold $\tau$ (top-k selection)	Top 10–15% per layer

Appendix

Table 7: Hyperparameters and training configuration

Category	Attack	Qwen	OLMoE
Direct Override	DevMode v2 ( Albert and Team, 2025 )	70.0	0.0
Obfuscation	FlipAttack ( Liu et al., 2024b )	93.3	71.5
Role-play Hijacking	AIM ( Albert and Team, 2025 )	80.0	0.0
Cognitive Manipulation	DeepInception ( Li et al., 2023 )	92.1	83.8
Adaptive / Feedback	PAIR ( Chao et al., 2024 )	85.0	5.0
Persuasion	Persuasion ( Zeng et al., 2024 )	78.6	49.6

Appendix

Table 8: Attack success rates (%) of representative jailbreak methods on Qwen and OLMoE. Bold rows are used in our main evaluation.

Model	Harmlessness	XsTest	Over-refusal
OLMoE	0.18 $\rightarrow$ 0.68		0.14
Qwen	0.06 $\rightarrow$ 0.76		0.12

Appendix

Table 9: RASA-FixedMT results on the MHJ dataset. Harmlessness before and after RASA alignment, with XsTest over-refusal.

Method	General (Avg)	Over-refusal	X-Teaming Harmlessness
Original (OLMoE)	0.45	0.17	0.0
RASA	0.46	0.17	0.20
RASA-MT	0.45	0.19	0.45–0.55

Appendix

Table 10: Results for multi-turn X-Teaming jailbreak defense on OLMoE.

Attack	GPT-4o-mini	Sonnet 4.6	Agree.
Flip	1.00	0.96	0.95
Deepin	1.00	0.95	0.93
Pers	0.98	0.93	0.92

Appendix

Table 11: Cross-judge validation on RASA-Mixed (OLMoE). Harmlessness rates from GPT-4o-mini and Claude Sonnet 4.6 (Sonnet 4.6) under the same rubric, with per-example agreement (Agree.).

	Flip	Deepin	Pers
Flip	1.00	0.38	0.31
Deepin	0.38	1.00	0.35
Pers	0.31	0.35	1.00
Random Baseline	${\approx}0.07$

Appendix

Table 12: Jaccard similarity of SCE sets identified under different attack types (OLMoE). Random baseline derived analytically under the hypergeometric model assuming no shared structure between attack types.

Figure 3: SCE distribution across layers on OLMoE. Number of Safety-Critical Experts (SCEs) selected per layer (out of 64 experts per layer, 16 layers total) for FlipAttack, DeepInception, and Persuasion training. We report the union of per-batch top-128 selections, i.e., the operational

\mathcal{E}_{\text{crit}}

used during fine-tuning; the total

|\mathcal{E}_{\text{crit}}|

ranges from 234 to 265. All three attacks place 86–96% of their SCEs in layers 0–3, with zero selections beyond layer 5, giving a nearly identical layer-wise profile across structurally distinct jailbreaks that supports the cross-attack generalization claim.

Figure 4: Aggregated-AAD top-

K

SCE distribution across layers on OLMoE (cross-check for Figure 3 ). SCEs are selected as the global top-128 of mean-pooled AAD across batches (rather than the union of per-batch top-128 used in Figure 3 ). Under this view, SCEs are distributed nearly uniformly across all 16 layers (4–12 per layer; uniform baseline

=8

), with L0–L3 slightly below baseline. The contrast with the sharply early-concentrated per-batch view indicates that early-layer AAD has high variance rather than high mean : per-batch maxima fall in L0–L3 even though mean AAD is comparable across all layers.

Model	Attack Method	Ablation	Harmlessness ( $\uparrow$ )			General ( $\uparrow$ )			Over-Refusal ( $\downarrow$ )
Model	Attack Method	Ablation	Flip	DeepIncep.	Pers	MMLU	GSM8K	TruthfulQA	XsTest
(a) Phase Balancing: Expert vs. Router Epochs ( $E_{\text{sce-ft}}:E_{\text{rt}}$ )
OLMoE	Flip	1:2	0.00	-0.04	+0.02	+0.03	-0.05	-0.05	0.00
	Deepin		+0.04	0.00	+0.02	+0.02	-0.01	+0.01	+0.04
	Pers		-0.07	-0.04	-0.14	+0.03	+0.06	-0.06	+0.05
QWEN	Flip		0.00	+0.40	+0.08	-0.03	-0.01	+0.04	-0.03

Appendix

Table 13: Ablation Study results (compared to default settings in Table 1 ).

Figure 5: Effect of expert selection and training configuration. Impact of (a) training rounds, (b) top-k safety-critical expert selection, and (c) adversarial data ratio on safety, general performance, and over-refusal across MoE architectures.

Model	Experts/L	Unsafe/L	Global Top- $k$
OLMoE	64	6–10 ( $\sim$ 10–15%)	96–160
Qwen	128	16–21 ( $\sim$ 12–16%)	768–1024

Appendix

Table 14: Layer-wise interpretation of top-k expert selection.

Explore similar work

Understanding Safety-Sensitive Expert Behavior in Mixture-of-Experts LLMs
May 28, 2026Zhibo Zhang, Yuxi Li, Zhen Ouyang +2Safety Alignment Mixture-Of-Experts Large Language Models
Huazhong University of Science and Technology, Wuhan, China · Nanyang Technological University, Singapore
RouteHijack: Routing-Aware Attack on Mixture-of-Experts LLMs
May 1, 2026Zhiyuan Xu, Joseph Gardiner, Sana Belguith +1Jailbreak Success Rates Mixture-Of-Experts
University of Bristol Bristol, United Kingdom
MESA: Improving MoE Safety Alignment via Decentralized Expertise
May 30, 2026Yitong Sun, Yao Huang, Teng Li +5Safety Alignment Large Language Model Alignment
Institute of Artificial Intelligence, State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, 100191, China · College of AI, Tsinghua University, Beijing, 100083, China · School of Computer Science, Fudan University, Shanghai, 200438, China +2

May 28, 2026, cs.CLJ/K move · Enter open · S save

Understanding Safety-Sensitive Expert Behavior in Mixture-of-Experts LLMs

Zhibo Zhang, Yuxi Li, Zhen Ouyang, Ling Shi+1

Huazhong University of Science and Technology, Wuhan, China · Nanyang Technological University, Singapore

Safety Alignment Mixture-Of-Experts Large Language Models Experts Agent Behavior Modeling

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) LLMs rely on sparse, router-driven expert activation, yet how safety alignment interacts with routed expert specialization remains underexplored. A common intuition is that safety behavior may be controlled by routing harmful requests to distinct refusal-oriented experts. In this work, we provide empirical evidence for a different picture: routing patterns in aligned MoE LLMs are largely topic-driven, while safety behavior can be altered with little change to the model's intrinsic routing path. Motivated by this observation, we present RASET (Router-Agnostic Safety-critical Expert Tuning), a red-teaming framework that probes safety enforcement that is localized in a small subset of experts while preserving the model's intrinsic routing behavior. RASET identifies safety-critical experts via a contrastive routing-sensitivity criterion and applies parameter-efficient tuning only to the selected experts, minimizing semantic disruption relative to router-steering interventions. These results reveal a distinct MoE safety risk, highlighting the need for expert-aware alignment mechanisms.

Abstract

Explore similar work

Understanding Safety-Sensitive Expert Behavior in Mixture-of-Experts LLMs

RouteHijack: Routing-Aware Attack on Mixture-of-Experts LLMs

MESA: Improving MoE Safety Alignment via Decentralized Expertise