cs.CLNov 17, 2026

What MoE Routing Controls: Causal Interventions on Expert Trajectories

Authors: Mukund Athmaram, Manoj Rao, Liang Ma

Organizations: Advanced Micro Devices Inc. (AMD) · Cornell University

Abstract

In mixture-of-experts (MoE) language models, a token is routed to a sequence of experts across layers. We investigate how forcing tokens onto chosen expert paths influences model behavior. We build a target path from the experts that a concept's or role's tokens most often use, and we add a fixed bias to each target expert's router logit during generation to steer the token toward this expert path. We study two kinds of targets. Format targets concern the form of the output, and concept targets concern its topic. In Qwen3-30B-A3B, we show that steering a format path changes the text the model writes. We also show that steering a concept path shifts the model's next-token probabilities toward the concept, even when the path is built in another language, yet this shift has little effect on the generated text. We investigate why formats reach the text but concepts do not by steering only part of the generation, and find evidence that a format, once started, reinforces itself. We show that the concepts religion and money steer most strongly, and that the effect is concentrated in the middle layers. We attempt to explain why the strength of the effect varies by concept through an analysis of embedding distinctiveness and word frequency.

Explore similar work

CardsList
  1. When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models

    May 8, 2026Youngsik Yoon, Siwei Wang, Wei Chen +1Mixture-of-Experts Language ModelsLLM Routing

  2. Geometric Routing Enables Causal Expert Control in Mixture of Experts

    Apr 15, 2026Ivan Ternovtsii, Yurii BilakMixture-of-Experts InterpretabilityMixture-of-Experts Expert Specialization

  3. Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs

    Apr 20, 2026Charles Ye, Bo Yuan, Lee SharkeyMixture-of-Experts InterpretabilityLLM Interpretability