cs.LGMar 11, 2026

Task-Conditioned Routing Signatures in Sparse Mixture-of-Experts Transformers

Authors: Mynampati Sri Ranganadha Avinash

Organizations: Independent Researcher | Asthra Labs

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Sparse Mixture-of-Experts (MoE) architectures enable efficient scaling of large language models through conditional computation, yet the routing mechanisms responsible for expert selection remain poorly understood. In this work, we introduce routing signatures, a vector representation summarizing expert activation patterns across layers for a given prompt, and use them to study whether MoE routing exhibits task-conditioned structure. Using OLMoE-1B-7B-0125-Instruct as an empirical testbed, we show that prompts from the same task category induce highly similar routing signatures, while prompts from different categories exhibit substantially lower similarity. Within-category routing similarity (0.8435 +/- 0.0879) significantly exceeds across-category similarity (0.6225 +/- 0.1687), corresponding to Cohen's d = 1.44. A logistic regression classifier trained solely on routing signatures achieves 92.5% +/- 6.1% cross-validated accuracy on four-way task classification. To ensure statistical validity, we introduce permutation and load-balancing baselines and show that the observed separation is not explained by sparsity or balancing constraints alone. We further analyze layer-wise signal strength and low-dimensional projections of routing signatures, finding that task structure becomes increasingly apparent in deeper layers. These results suggest that routing in sparse transformers is not merely a balancing mechanism, but a measurable task-sensitive component of conditional computation. We release MOE-XRAY, a lightweight toolkit for routing telemetry and analysis.

Figures & tables

CategoryCountDescription
Code20Programming tasks and algorithmic prompts
Math20Mathematical and symbolic reasoning prompts
Story20Creative writing and narrative prompts
Factual20Knowledge retrieval and question-answering prompts
Table 1: Prompt categories used in the experiments.
Figure 1: Routing signature similarity matrix across task categories. The diagonal entries correspond to within-category similarity and are consistently higher than off-diagonal entries.
Figure 2: Routing similarity compared to the load-balancing baseline. Empirical similarity follows the clean ordering Across << Load-Balance << Within.
Figure 3: Layer-wise task signal strength measured by Cohen’s dd . Task-conditioned separation grows stronger toward deeper layers.
Figure 4: PCA projection of routing signatures. Distinct clusters emerge for code, math, story, and factual prompts.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 5: t-SNE projection of routing signatures showing clear task-conditioned clusters.
More in this topic

Mixture-of-Experts Interpretability

See all 18 papers

Explore similar work

CardsList
Jun 12, 2026

A theoretical model for task routing in mixture-of-expert transformers

Mixture-of-experts (MoE) layers enable the scaling of transformer models while keeping the inference compute fixed. While task-expert specialization has been observed in empirical studies of frontier MoE transformer models, existing theoretical work analyzes this using continuous mixture models that cannot be used to model natural language effectively. An important open question is to theoretically explain task-expert specialization in transformer MoE models using discrete models of language. To address this, we represent structured knowledge via syntactic templates and finite key-value dictionaries, and prove formally that a single-layer MoE transformer can encode knowledge by using experts that specialize in the corresponding tasks. Our construction shows how queries are routed to unique, task-specific experts whose size depends solely on the intrinsic complexity of the given task (i.e. the combined size of its syntactic templates and factual dictionary). Our construction provides a theoretical support for empirical results on localized knowledge circuits in MoE models. We support our theoretical findings with experiments evaluating model performance under varying MoE loss functions.

Vinoth Nandakumar, Yongli Xiang, Yunzhi Yao +2

University of Sydney · Zhejiang University · Google Research

Transformer ExpressivityMixture of ExpertsLLM RoutingMixture-of-Experts ModelsMixture-of-Experts Expert Specialization
AskPDF
Mar 18, 2026

Path-Constrained Mixture-of-Experts

Sparse Mixture-of-Experts (MoE) architectures route each token through a subset of experts at each layer independently. We propose viewing MoE computation through the lens of expert paths -- the sequence of expert selections a token makes across all layers. This perspective reveals that, despite NLN^L possible paths for NN experts across LL layers, tokens in practice cluster into a small fraction of paths that align with linguistic function, yet the vast majority of paths remain unexplored, representing a statistical inefficiency. This motivates architectures that constrain the effective path space to amplify this natural concentration. As one instantiation, we introduce \pathmoe{}, which shares router parameters across blocks of consecutive layers. Analysis confirms that \pathmoe{} amplifies the emergent path structure: it produces more concentrated path clusters, better cross-layer consistency, and greater robustness to routing perturbations. Experiments on 0.9B and 16B parameter \pathmoe{} models demonstrate consistent improvements on perplexity and downstream tasks over independent routing, while eliminating the need for auxiliary losses. These results establish expert paths as a useful design axis for MoE architectures, complementary to existing work on independent routing mechanisms.

Zijin Gu, Tatiana Likhomanenko, Vimal Thilak +2

Apple · †Google

Mixture-of-Experts Language ModelsSparse Mixture-of-ExpertsExpert Routing
AskPDF
Jun 16, 2026

SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs

Sparse Mixture-of-Experts (MoE) architectures enable scaling LLM parameters under a fixed inference budget by activating only a small subset of experts via top-kk routing. While this preserves causality and suits autoregressive language models, the discrete top-kk operator is not differentiable, forcing a fixed number of active experts per input and resulting in inefficient use of computation. We propose SoftMoE, which replaces discrete routing with a truncated soft top-kk LapSum relaxation, allowing gradient-based optimization of expert routing. We further parameterize the mean number of active experts per layer and impose a global budget constraint, enabling the model to learn how to allocate expert capacity across layers. SoftMoE remains fully compatible with autoregressive modeling and achieves performance comparable to or better than sparse MoE on language modeling and downstream tasks, while activating significantly fewer experts. Notably, the learned allocation is highly non-uniform, with later layers activating more experts. The source code is publicly available†^\dagger.

Mikołaj Zasada, Łukasz Struski, Jacek Tabor +1

AGH University of Krakow, Poland · Faculty of Mathematics and Computer Science, Jagiellonian University, Poland · Jagiellonian University, Poland +1

Mixture-of-Experts Language ModelsSparse Mixture-of-ExpertsLLM Inference AccelerationLLM Routing
AskPDF