In Mixture-of-Experts language models, the router typically selects and weights experts based on the token's hidden state, utilizing limited contextual information. We propose Attention-Aware Routing (AAR), which augments the router with temporal and spectral features extracted from a sliding window of attention weights that represent a summary of the model's contextual state, disentangled from the hidden state. Keeping the base transformer entirely frozen, we train only the routing parameters, isolating routing as the sole variable. AAR improves GSM8K by +3.37 pp over a routing-only SFT baseline on OLMoE. Beyond performance, we show that routing and attention form a coupled circuit: routing changes at layer l propagate through the residual stream to amplify attention sinks at layer l+1, reshaping attention without any direct update to the attention mechanism itself. Further, AAR reduces long diverging generation, with incorrect answers getting shorter, while correct answers remain unchanged in length. Finally, AAR is strongly depth-sensitive: applying it indiscriminately across layers can degrade factual retrieval, whereas mathematical reasoning gains persist when it is introduced deeper in the network. This sensitivity exposes a retrieval--reasoning tension across depth and makes layer-selective AAR a controlled probe of the routing-relevant information carried by attention at different layers.
Figures & tables
Figure 1: Attention-Aware Routing. For the current token i , attention weights to the previous M tokens are averaged across heads and assembled into a sliding window w , then transformed into the augmented window w~=ϕ(w) . The attention router Al maps w~ to expert logits gattn , which are merged and gated with the standard router’s logits gstd=Wrh to produce the final routing decision over experts E1,…,EN .
Configuration
GSM8K
BBH
MMLU
MMLU-Math
HumanEval
IFEval
8-shot CoT acc.
0-shot acc.
5-shot acc.
5-shot acc.
pass@10
loose acc.
OLMoE-0125-SFT
51.33
36.80
53.96
31.01
41.46
56.19
+ routing-only SFT
51.99 ±0.76
36.17 ±0.45
53.91 ±0.01
30.45 ±0.56
41.46 ±0.80
56.56 ±1.09
+ AAR S , M=40
55.36 ±0.74
36.71 ±0.46
53.96 ±0.20
32.91 ±0.54
40.00 ±1.26
55.64 ±0.41
+ AAR S , M=20
54.89 ±1.02
36.30 ±0.42
53.93 ±0.10
33.48 ±0.49
40.98 ±1.80
56.67 ±1.36
+ AAR ST , M=20
54.57 ±1.10
36.33 ±0.61
53.78 ±0.14
32.29 ±0.64
40.85 ±1.14
56.34 ±1.88
Table 1: AAR compared against trained baselines. GSM8K consistently improves, with full AAR achieving the best results. S={9,…,15} for OLMoE. SQ={14,…,17} for Qwen1.5. Best in bold.
Method
MATH-500
OLMoE
Routing-only SFT
14.00 ± 1.83
AAR ( L9 – 15 )
16.64 ± 0.91
Qwen3.6-MoE
Routing-only SFT
78.47 ± 1.30
AAR ( L23,27,31 )
79.60 ± 1.51
Table 2: MATH-500 accuracy for AAR TF and routing-only SFT. OLMoE results are averaged over 5 seeds and Qwen3.6-MoE results over 3 seeds.
Figure 2: Normalized autocorrelation function (ACF) of the attention window signal wil(t) at layers 4, 9, and 12, for example window sizes ∈{20,64,128} . The ACF is averaged over query tokens and prompts from GSM8K (5-shot). The dashed vertical line marks lag =20 . Across all layers and window sizes, the ACF decays below 0.1 by lag ∼20 , revealing an intrinsic correlation length of 20 tokens in the attention signal. A window of M=20 is thus sufficient to capture most structured information.
Configuration
M
GSM8K
routing-only SFT
–
51.99 ±0.76
AAR T
10
51.99 ±1.13
20
54.10 ±0.52
128
53.74 ±0.91
512
52.96 ±0.76
Table 3: Window size ablation for AAR T , applied in all layers. Best in bold.
Figure 3: Top: Absolute weight to the attention sink, per layer (averaged across prompt’s tokens). Bottom: the absolute difference, isolated. The different routing decisions of layer 9, because of AAR, lead to different attention weights produced in the attention submodule, at layer 10.
Condition
Metric
Base
AAR-T
Rel. Δ
Both Correct
Mean
68.2
67.6
−0.8%
Median
54.0
54.0
0.0%
P99 (chars)
251.0
248.0
−1.2%
Both Wrong
Mean
163.7
144.4
−11.8%
Median
78.0
73.0
−6.4%
P99 (chars)
8513.0
7242.0
−14.9%
Table 4: Length analysis (tokens) on GSM8K averaged across seeds and context sizes M . Ramble Rate denotes responses exceeding 1,000 characters. AAR T leaves correct responses nearly unchanged while reducing the length of incorrect reasoning, especially in the tail (P99). The answer length reduction also persists for answers that get corrected by AAR T .
Figure 4: AAR T ( M=20 ) accuracy across MMLU subjects as a function of starting layer. Most subjects’ accuracy curves follow a bell shape.
MMLU Subject
ΔM =10
ΔM =20
ΔM =64
Math
+0.7
+0.6
+0.5
Comp. Sci.
+0.5
+0.4
−0.1
Biology
−4.5
−3.4
−3.8
History
−2.5
−2.4
−2.2
Health
−1.4
−1.8
−1.9
Psychology
−1.4
−1.7
−1.2
Table 5: MMLU subcategory Δ (absolute % accuracy difference from baseline, 5 seeds) when AAR T is applied to all layers of OLMoE. Reasoning-oriented subjects (top) are improved; degradation is concentrated in retrieval-heavy subjects (bottom).
Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.
Mixture-of-Experts (MoE) language models route each token to a small subset of experts, but whether the routes selected by a trained top-k router are good ones is rarely evaluated directly. Holding the model fixed, we compare each standard route against sampled equal-compute alternatives for the same token and score each by the next-token probability it assigns to the realized token in a verified reasoning trajectory. The result is sharply token-conditional: the standard router is well-aligned with route utility on confident tokens but uninformative on the fragile tokens that drive hard reasoning, where lower-loss equal-compute routes consistently exist inside the frozen model but are not selected. The same pattern holds across Qwen3-30B-A3B, GPT-OSS-20B, DeepSeek-V2-Lite, and OLMoE-1B-7B, and follows structurally from how standard top-k training evaluates routing decisions: the language modeling loss scores only the executed route, and load balancing depends only on aggregate routing statistics. A minimal router-only update to the final-layer router, leaving every expert and every other router frozen, is sufficient to shift pass@K on AIME 2024+2025 and HMMT 2025 for both Qwen3-30B-A3B and GPT-OSS-20B, suggesting that at least part of the failure reflects router-reachable misallocation rather than expert capacity alone.
Youngsik Yoon, Siwei Wang, Wei Chen +1
Department of Computer Science and Engineering, POSTECH, South Korea · Microsoft Research Asia, Beijing, China · Graduate School of Artificial Intelligence, POSTECH, South Korea
Mixture-of-Experts (MoE) layers increase model capacity by activating only a small subset of experts per token, and typically rely on a learned router to map hidden states to expert assignments. In this work, we ask whether a dedicated learned router is strictly necessary for MoE routing. We propose Self-Routing, a parameter-free routing mechanism that uses a designated subspace of the token hidden state directly as expert logits, eliminating the router projection entirely while leaving the rest of the MoE layer unchanged. We evaluate Self-Routing on language modeling across different expert counts and model scales, and on ImageNet-1K classification by comparing it against a standard learned router, random-routing baselines, and dense non-MoE baselines. Our results show that Self-Routing remains competitive with the learned-router baseline while removing all dedicated routing parameters, and yields more balanced expert utilization, with about 17% higher average normalized routing entropy and no explicit load-balancing loss. On ImageNet-1K with DeiT-S/16, Self-Routing also slightly improves over the corresponding learned-router MoE. These findings suggest that effective MoE routing can emerge from the hidden representation itself without requiring a separate learned router module.
Sparse Mixture-of-Experts (MoE) architectures route each token through a subset of experts at each layer independently. We propose viewing MoE computation through the lens of expert paths -- the sequence of expert selections a token makes across all layers. This perspective reveals that, despite NL possible paths for N experts across L layers, tokens in practice cluster into a small fraction of paths that align with linguistic function, yet the vast majority of paths remain unexplored, representing a statistical inefficiency. This motivates architectures that constrain the effective path space to amplify this natural concentration. As one instantiation, we introduce \pathmoe{}, which shares router parameters across blocks of consecutive layers. Analysis confirms that \pathmoe{} amplifies the emergent path structure: it produces more concentrated path clusters, better cross-layer consistency, and greater robustness to routing perturbations. Experiments on 0.9B and 16B parameter \pathmoe{} models demonstrate consistent improvements on perplexity and downstream tasks over independent routing, while eliminating the need for auxiliary losses. These results establish expert paths as a useful design axis for MoE architectures, complementary to existing work on independent routing mechanisms.