From Concentration to Differentiation and Back: Routing Effective Rank in MoE Reasoning Cohorts

Kang Chen; Sihan Zhao; Yixin Cao; Yu-Gang Jiang

cs.AISep 6, 2026

From Concentration to Differentiation and Back: Routing Effective Rank in MoE Reasoning Cohorts

Authors: Kang Chen, Sihan Zhao, Yixin Cao, Yu-Gang Jiang

Organizations: Fudan University · Shanghai Innovation Institute

Abstract

Test-time scaling produces cohorts of reasoning rollouts, yet there is no standard label-free account of how their internal computation reorganizes as inference unfolds. We introduce routing effective rank deff, the entropy-effective dimensionality of a cross-rollout graph built from MoE expert-routing similarity. Across ten MoE configurations and five math/science benchmarks, deff exhibits a reproducible low-high-low trajectory, with a prominent interior maximum in 98.5% of 3,105 model-question cohorts: routing similarity is concentrated early, maximally differentiated at intermediate budgets, and reconcentrated later, and the timing of this maximum varies systematically with architecture and reasoning effort. An exact decomposition separates cohort-wide common-mode mass from residual spectral dimensionality: common-mode reallocation accounts for about two thirds of the trajectory, while the residual spectrum contributes about one quarter and retains substantial variation beyond the common mode. The decomposition further localizes behavior: among non-unanimous cohorts, increases in common-mode concentration strongly predict same-answer recoverability, and higher reasoning effort delays the maximum by 2.59 octaves (doublings of the token budget) and consistently expands the high-rank period across all four tested architectures, locating the effort effect in timing and duration rather than peak amplitude. Correctness comparisons separate structural monitoring from answer selection, positioning routing effective rank as a decomposable, label-free diagnostic of cohort organization - a principled spectral lens on how MoE reasoning cohorts differentiate and reconcentrate over inference time.

Figures & tables

Figure 1: Routing effective rank follows a reproducible low–high–low trajectory whose peak timing is architecture- and effort-dependent. One panel per architecture (fixed expert count

E

; Qwen3-Next-80B,

E{=}512

, and the pooled curve are in Figure A1 ), one curve per reasoning-effort tier (medians). Triangles and whiskers mark the median and IQR of the peak budget, measured separately within each cohort ( Section 4.2 ). The maximum moves later with effort in all four architectures, whereas peak height is architecture-dependent. Panels share one

y

-scale, but absolute

d_{\mathrm{eff}}

levels depend on the routing representation and cohort size (

d_{\mathrm{eff}}\!\leq\!N

): compare timing and shape within an architecture, not levels across panels.

Figure 2: From routing traces to $d_{\mathrm{eff}}$ , on one real cohort (OSS-120B-High, AIME-24). (1–2) Each rollout’s routing trace ( $K{=}4$ of $E{=}128$ experts per token per layer) is summarized, over a trailing $16$ -token window at budget $b$ , by a gate-weight histogram $h_{i}$ —from router activations alone; answer annotations only locate the commit window and label alignment. (3) Weighted-Jaccard similarities ( Equation 1 ) form the cohort graph over $N{=}64$ rollouts. (4) Its normalised spectrum gives $d_{\mathrm{eff}}$ ( Equation 2 ), from $1$ to $N$ . (5) Read at every budget, $d_{\mathrm{eff}}$ traces the low–high–low trajectory ( $7.2\to 33.6\to 7.2$ ; the ends need $5$ – $7$ eigendirections for $80\%$ of spectral mass, the peak $37$ ): routing similarity is first concentrated, then distributed across more spectral directions, and finally reconcentrated—a direct geometric readout of cohort organization.

Figure 3: Spectral anatomy of routing effective rank. The identity $\log d_{\mathrm{eff}}=h(m)+(1-m)\log d_{\perp}$ is exact, with $m=p_{1}$ the common-mode mass and $d_{\perp}$ the residual dimensionality (evaluated term by term on a real eigenspectrum in Figure A4 ). (A) Aligned per-family medians: as $d_{\mathrm{eff}}$ rises, spectral mass drains out of the common mode; as it falls, mass returns. The residual spectrum contributes complementary motion on a smaller scale. (B) Over the full budget path the common-mode term carries $0.67$ of the absolute motion of $\log d_{\mathrm{eff}}$ , the residual term $0.24$ , and mixing $0.09$ (two-way clustered $95\%$ CIs); the remaining rows repeat the accounting on the early and late halves of the budget grid (details and controls in Appendix F ). (C) Residual dimensionality retains substantial variation beyond the common mode: a grouped cross-validated fit of $\log d_{\perp}$ on $m$ explains $37\%$ of out-of-sample variance, leaving $63\%$ as a distinct spectral axis for RQ3.

Figure 4: Common-mode concentration predicts same-answer recoverability; reasoning effort expands the high-rank regime. Top: across

28{,}622

adjacent-budget steps (clustered on

318

questions

\times

10

configurations), increases in common-mode mass make same-answer rollout pairs easier to recover from routing (the prevalence-corrected AP lift plotted here):

\beta_{\Delta m}=+0.522

[+0.430,+0.609]

, while the corresponding residual coefficient is

\beta_{\Delta r}=-0.047

[-0.092,-0.016]

; a within-question time shuffle collapses both toward zero. Bottom: higher effort delays the maximum, increases high-rank occupancy, widens the

d_{\mathrm{eff}}\!\geq 0.8d_{\max}

regime, and lowers integrated common-mode mass, with the same sign in all four architectures.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Family	$n$	mean	median	per-cohort [IQR]
OSS	$1863$	$256$	$192$	$768\,[256,2048]$
Qwen-Instruct	$636$	$768$	$768$	$1024\,[768,4096]$
Qwen-Thinking	$606$	$2048$	$2048$	$4096\,[2560,8192]$
pooled	$3105$	$512$	—	$1024$

Appendix

Table A1 : Three peak-budget calipers: argmax of the family mean curve, of the family median curve, and the median of per-cohort argmax positions [IQR]. The paper’s primary caliper is the per-cohort median (rightmost); the family-mean caliper is shown because aggregated curves are what a reader sees in a plot.

Figure A1 : Full four-architecture version of Figure 1 , including Qwen3-Next-80B (

E{=}512

). Same data, calipers, and conventions: triangles give the per-cohort argmax median with IQR whiskers; the inset is the pooled mixture curve. Qwen3-Next-80B repeats the low–high–low shape with a later clock and is the one architecture whose peak height inverts with effort (per-cohort median peak

d_{\mathrm{eff}}

: Instruct

37.2

vs. Thinking

36.7

), while its fixed-budget levels remain effort-monotone ( Table A2 ).

Figure A2 : The dataset sets the ordering of the effective-rank clock, nearly deterministically across configurations. Left: median peak-normalized

d_{\mathrm{eff}}

trajectories per dataset for one representative configuration per architecture (High/Thinking tier). Right: each configuration’s median peak budget per dataset (dots; all ten configurations) with cross-configuration medians (diamonds). The dataset ordering of peak timing is almost identical in every configuration (Kendall

W{=}0.946

, within-configuration permutation

p{<}10^{-4}

): GPQA peaks

1.5

–

2

octaves earlier than the math benchmarks (pooled median

\log_{2}t_{\max}

9.79

vs.

12.00

for HMMT’25), and HMMT’25 is latest with the widest high-rank period (

w_{0.8}

, octaves at

d_{\mathrm{eff}}{\geq}0.8\,d_{\max}

: median

3.64

vs.

2.38

)—consistent with the peak tracking each dataset’s length scale rather than a universal token count.

Configuration	$E$	mean $d_{\mathrm{eff}}$	median $d_{\mathrm{eff}}$ ( $n$ )
OSS-20B-Low	$32$	$3.62$	$3.19$ ( $318$ )
OSS-20B-Med	$32$	$8.27$	$5.99$ ( $318$ )
OSS-20B-High	$32$	$10.57$	$11.41$ ( $282$ )
OSS-120B-Low	$128$	$4.25$	$4.08$ ( $318$ )
OSS-120B-Med	$128$	$10.62$	$5.09$ ( $318$ )
OSS-120B-High	$128$	$17.85$	$19.13$ ( $309$ )

Appendix

Table A2 : Mean/median

d_{\mathrm{eff}}

b{=}4096

per configuration (weighted Jaccard). Monotone in reasoning effort within each architecture.

Family	at $b{=}4096$	at own peak	at commit
OSS	$9.1$	$15.7$	$4.3$
Qwen-Instruct	$18.8$	$25.1$	$9.2$
Qwen-Thinking	$24.6$	$26.1$	$6.6$

Appendix

Table A3 : Family mean

d_{\mathrm{eff}}

at a fixed budget, at each family’s own trajectory peak, and at commit (family-mean caliper; Appendix A ). The OSS -vs-reasoning separation holds at all three readouts, and the OSS effort tiers stay ordered at commit (

3.55/3.65/3.86

at 20B,

4.19/5.26/5.48

at 120B). The Instruct -vs- Thinking ordering, by contrast, is specific to the fixed mid-budget reading: it narrows at the peak-aligned readout ( Thinking peaks later) and reverses at commit (

9.2

vs.

6.6

), so fixed-budget readings conflate level with timing.

Aggregation	consensus	correctness
pooled-all	$+0.15$	$+0.14$
per-shard mean	$\mathbf{+0.46}\,[+.40,+.51]$	$\mathbf{+0.40}\,[+.34,+.47]$
per-family: OSS	$+0.23$	$+0.12$
per-family: Instruct	$+0.47$	$+0.44$
per-family: Thinking	$+0.27$	$+0.31$

Appendix

Table A4 :

\mathrm{Spearman}

(commit-graph

d_{\mathrm{eff}}

, difficulty). The consensus–correctness gap is clear for OSS ,

{\approx}0

for Instruct , and reverses for Thinking . Pooled-all is a single Spearman over the

3{,}104

cohorts with a defined majority share (one cohort produced no gradeable answer); per-shard mean averages the within-shard Spearman over the

50

model

\times

dataset shards (

2000\times

bootstrap CIs).

Figure A3 : The trajectory is a per-cohort fact, not an aggregation effect. Each of the $3{,}105$ rows is one (model, question) cohort, standardised within the row and sorted by the budget at which it peaks; the left stripe gives the architecture. The bright band runs diagonally—the sort constructs the diagonal; the band’s contrast, not its diagonality, carries the information—and the architecture stripe shows how the pooled peak combines architecture-specific clocks: OSS-20B cohorts concentrate at early peaks and Qwen3-Next-80B cohorts at late ones.

Stratum	b32	b128	b512	b2048	b8192	b32768	$\mathbf{W}_{C}$
easy ( $\geq 0.875$ )	$0.901$	$0.902$	$0.909$	$0.917$	$0.930$	$0.940$	$0.940$
mid ( $[0.60,0.875)$ )	$0.605$	$0.612$	$0.625$	$0.653$	$0.692$	$0.725$	$0.726$
hard ( $<0.60$ )	$0.317$	$0.321$	$0.337$	$0.394$	$0.433$	$0.480$	$0.484$
contested ( $<0.5$ )	$0.237$	$0.239$	$0.247$	$0.315$	$0.357$	$0.413$	$0.418$
all defined-AP	$0.611$	$0.614$	$0.627$	$0.657$	$0.688$	$0.717$	$0.719$

Appendix

Table A5 : Same-answer AP by budget and difficulty stratum (defined-AP cohorts). The column grid omits

b{=}4096

for space; the pooled defined-AP there is

0.673

(between

b{=}2048

’s

0.657

and

b{=}8192

’s

0.688

). The last column is the same-answer AP of the commit-window graph

\mathbf{W}_{C}

itself (

0.719

pooled); b32768 is the largest prefix budget. Absolute levels are not comparable across strata (each stratum has its own same-answer prevalence, hence its own random-AP baseline).

Figure A4 : The exact identity on one real, typical eigenspectrum. A normalised eigenspectrum (log scale) with the common-mode mass

m=p_{1}

highlighted; the annotation evaluates

\log d_{\mathrm{eff}}=h(m)+(1-m)\log d_{\perp}

term by term on this frame, and the two sides agree to the printed precision. The example is the committed per-shard spectrum whose

m

is closest to the median of those examples—typical rather than cherry-picked.

Figure A5 : The trajectory is a redistribution of spectral mass. Mean normalised eigenspectrum over

2{,}999

cohorts (band: IQR), at the first budget, at each cohort’s own peak (peak-aligned, since family peak budgets differ several-fold; Table A1 ), and at the last budget. The leading direction holds

p_{1}{=}0.689

early,

0.334

at the peak and

0.688

late, while the tail beyond rank

8

moves the opposite way (

0.084/0.238/0.096

). The low–high–low form in

d_{\mathrm{eff}}

is exactly this shift of mass out of and back into the leading eigendirections. Note that the two ends are spectrally alike, not merely equal in

d_{\mathrm{eff}}

p_{1}

differs by

0.001

between them. Cohorts with fewer than

32

graded rollouts are excluded so that every curve is averaged over the same eigenvalue ranks.

Segment	$n$	$\Delta\log d_{\mathrm{eff}}$	mixing $h$	common mode	residual
early half ( $32\!\to\!1024$ )	$2999$	$+1.136$	$+0.026$	$+0.631$ $[+.537,+.719]$	$+0.342$ $[+.271,+.429]$
late half ( $1024\!\to\!32768$ )	$2999$	$-0.920$	$+0.017$	$+0.824$ $[+.784,+.868]$	$+0.158$ $[+.117,+.189]$
late half, live only	$2256$	$-0.717$	$+0.005$	$+0.776$ $[+.707,+.845]$	$+0.219$ $[+.170,+.267]$
rise (peak-aligned)	$2931$	$+1.605$	$+0.020$	$+0.691$ $[+.611,+.772]$	$+0.289$ $[+.212,+.354]$
fall (peak-aligned)	$2931$	$-1.400$	$+0.012$	$+0.853$ $[+.822,+.885]$	$+0.135$ $[+.092,+.174]$
fall, live only	$2663$	$-1.075$	$-0.002$	$+0.836$ $[+.799,+.874]$	$+0.166$ $[+.112,+.216]$

Appendix

Table A6 : Attribution of trajectory motion by the midpoint split of Equation 3 : share of

|\Delta\log d_{\mathrm{eff}}|

carried by the mixing term

h(m)

, the common-mode term, and the residual term, per segment (two-way clustered

95\%

CIs; all common-mode and residual shares are positive in

\geq 9/10

configurations; mixing shares are n.s. with MDE 80

0.04

–

0.08

). “Live” restricts the segment to still-generating rollouts. Shares are pooled ratio-of-sums over each arm’s full cell set; the pre-registered H2 contrast of Appendix F is the same functional on the

2{,}604

cells matched across both halves (excluding

395

late-half segments whose

\Delta\log d_{\mathrm{eff}}

is exactly zero under frozen windows), where the early share is

0.317

—so the contrast is

+0.158

, not the difference of the displayed rows.

Subset	frames	CV $R^{2}$	Spearman
all	$47984$	$0.369$	$-0.639$
OSS	$29808$	$0.295$	$-0.530$
Qwen-Instruct	$9824$	$0.533$	$-0.764$
Qwen-Thinking	$8352$	$0.579$	$-0.767$
all, rising branch	$31109$	$0.545$	$-0.724$
all, falling branch	$18718$	$0.303$	$-0.345$

Appendix

Table A7 : Grouped cross-validated fit of

r=\log d_{\perp}

m

(folds grouped by question;

47{,}984

frames from the

2{,}999

complete cells). The residual axis is only partially predictable from the common mode.

Variant	$n$	med. $m$	med. $d_{\perp}$	med. $d_{\mathrm{eff}}$	prom.
real ( $N{=}64$ )	$204$	$0.450$	$38.9$	$14.4$	$97.5\%$
id. permutation	$204$	$0.082$	$62.9$	$59.5$	$3.4\%$
pseudo-cohorts ( $N{=}32$ )	$264$	$0.321$	$25.1$	$16.8$	$97.0\%$

Appendix

Table A8 : Nulls for the anatomy experiment, on its own

6

-shard pool and prominence gate (not comparable to the full-corpus gate; Appendix A ). The expert-identity permutation drives the residual spectrum toward its near-isotropic ceiling (

d_{\perp}\!\to\!N{-}1

); cross-problem pseudo-cohorts of unrelated rollouts retain the coarse arc.

Target	$n$	controls	${+}\mathrm{dens}$	${+}m$	${+}(m,r)$	${+}\log d_{\mathrm{eff}}$	$\Delta R^{2}_{r\mid m}$ [ $95\%$ ]	$\Delta R^{2}_{\mathrm{split}}$
$\log d_{\mathrm{ans}}$ (commit, P1)	$3{,}096$	$0.610$	$0.631$	$0.631$	$0.648$	$0.641$	$+0.018$ $[-.004,+.041]$	$+0.007$
$\log d_{\mathrm{ans}}$ (terminal, sens.)	$3{,}096$	$0.610$	$0.632$	$0.632$	$0.649$	$0.642$	$+0.017$ $[-.018,+.039]$	$+0.007$
AP lift (terminal, P2)	$1{,}906$	$0.216$	$0.260$	$0.264$	$0.267$	$0.268$	$+0.003$ $[-.051,+.023]$	$-0.002$
difficulty (secondary)	$3096$	$0.564$	$0.586$	$0.586$	$0.607$	$0.595$	$+0.022$ $[-.001,+.050]$	$+0.012$

Appendix

Table A9 : Semantic-audit ladders: out-of-fold

R^{2}

under GroupKFold by question.

\Delta R^{2}_{r\mid m}

carries two-way clustered

95\%

CIs; the Bonferroni

98.33\%

intervals for the two regression primaries are P1

[-0.011,+0.052]

and P2

[-0.102,+0.036]

—both span zero. Density and common-mode mass provide nearly identical predictive value on these endpoints (

\Delta R^{2}

-0.0005

on P1,

+0.0036

on P2), localizing the observable association to the global agreement channel; the spectral formulation contributes exact channel attribution and an independent residual coordinate rather than scalar predictive lift.

Architecture	$n$	$\Delta\log_{2}t_{\max}$	$\Delta\log d_{\max}$	$\Delta\mathrm{Occ}_{d}$	$\Delta w_{0.8}$	$\Delta\mathrm{AUC}_{m}$	$\Delta\mathrm{AUC}_{r}$
OSS-20B	$282$	$+3.86\,[3.66,4.03]$	$+.181\,[.141,.217]$	$+.107\,[.097,.117]$	$+1.47\,[1.28,1.63]$	$-.113\,[-.119,-.106]$	$-.144\,[-.195,-.093]$
OSS-120B	$309$	$+2.97\,[2.80,3.14]$	$+.234\,[.189,.285]$	$+.172\,[.161,.183]$	$+2.02\,[1.84,2.20]$	$-.156\,[-.163,-.150]$	$+.104\,[.064,.146]$
Qwen3-30B	$310$	$+1.70\,[1.56,1.83]$	$+.093\,[.058,.124]$	$+.040\,[.030,.052]$	$+.47\,[.29,.65]$	$-.038\,[-.045,-.031]$	$+.079\,[.040,.117]$
Qwen3-Next-80B	$296$	$+1.90\,[1.82,2.00]$	$-.093\,[-.130,-.052]$	$+.087\,[.078,.096]$	$+1.22\,[1.07,1.36]$	$-.041\,[-.048,-.033]$	$-.039\,[-.080,.006]$
Pooled	—	$+2.59\,[1.72,3.83]$	$+.104\,[-.079,.232]$	$+.102\,[.045,.168]$	$+1.29\,[.55,1.99]$	$-.087\,[-.154,-.036]$	$+.004\,[-.127,.111]$

Appendix

Table A10 : Effort localization, paired within architecture (higher minus lower tier, matched by question; question-clustered

95\%

CIs; pooled row clustered by question

\times

architecture,

4

architecture clusters).

\Delta\log d_{\max}

(peak height) is the registered primary endpoint of this contrast—it varies by architecture, with a pooled interval spanning zero; the remaining columns are secondary endpoints.

\mathrm{Occ}_{d}=\int d_{\mathrm{eff}}(\tau)/d_{\max}\,\mathrm{d}\tau

with

\tau

the normalized budget-grid index;

w_{0.8}

is the octave width of the contiguous

d_{\mathrm{eff}}\geq 0.8d_{\max}

regime. The consistent effects are later maxima, greater peak-normalized occupancy, a wider high-rank interval, and weaker integrated common-mode mass. Residual-spectrum area has no common shift.

Feature	cohort AUC [ $95\%$ ]	rollout AUC
vote margin	$0.833$ $[.790,.869]$	$0.860$
$d_{\mathrm{ans}}$ (neg.)	$0.821$ $[.777,.859]$	$0.856$
question prior (LOMO)	$0.909$ $[.872,.953]$	$0.856$
$m$ (terminal)	$0.542$ $[.472,.658]$	$0.548$
density (terminal)	$0.539$ $[.469,.660]$	$0.545$
$d_{\mathrm{eff}}$ (terminal)	$0.440$ $[.324,.509]$	$0.432$

Appendix

Table A11 : Correctness comparison: fit-free single-feature AUCs (cohorts with

\geq 10

graded rollouts). We retain the pre-registered one-sided reading and do not reinterpret the coefficient sign post hoc; answer-side baselines remain stronger than standalone routing features for correctness selection.

Figure A6 : Out-of-sample model ladders for the semantic endpoints. Grouped-CV $R^{2}$ (upper number) and its increment over controls (lower number). The predictive ladder localizes answer-space coupling to the global agreement channel: density and common-mode mass provide nearly identical predictive value on these targets, while the residual axis remains the distinct structural coordinate of RQ2 (increments bounded in Figure A8 ). The scalar $\log d_{\mathrm{eff}}$ lies between the agreement-only and split-spectrum models.

Figure A7 : Higher effort moves the effective-rank maximum later in every architecture. Peak-normalized trajectories for paired lower- and higher-effort configurations, with paired peak-delay intervals on the right. The timing effect is uniform; peak height is architecture-dependent.

Figure A8 : Residual terminal incremental associations are bounded on the tested targets. Incremental out-of-fold

R^{2}

from adding residual dimensionality after common-mode mass. Thick whiskers are

95\%

and thin whiskers Bonferroni

98.33\%

two-way clustered intervals. The intervals constrain independent residual contributions on the tested targets (

95\%

upper bounds:

+0.04

for answer effective count,

+0.02

for AP lift) while remaining compatible with small effects.

Figure A9 : Correctness selection favors answer-side statistics. Fit-free single-feature AUCs for cohort majority correctness (circles) and rollout-level correctness (diamonds); dotted line: chance; two-way clustered

95\%

CIs (

B{=}500

, uncorrected). Text-side baselines sit at

0.82

–

0.91

(the

0.91

question prior uses the other nine configurations’ outcomes), while the best routing feature reaches

0.542

, separating answer selection from the structural-monitoring role studied in the main paper.

$n$	terminal frame	peak frame
$2$	$0.517$ $[.430,.588]$	$0.200$ $[.158,.265]$
$4$	$0.738$ $[.654,.763]$	$0.411$ $[.315,.485]$
$8$	$0.850$ $[.788,.880]$	$0.611$ $[.493,.692]$
$16$	$0.926$ $[.889,.942]$	$0.791$ $[.721,.838]$
$32$	$0.971$ $[.957,.976]$	$0.916$ $[.890,.945]$

Appendix

Table A12 : Rank fidelity of subsampled

d_{\mathrm{eff}}

: median over

50

(model, dataset) shards of the Spearman correlation, across a shard’s cohorts, between

d_{\mathrm{eff}}

on a random

n

-rollout subcohort (

20

draws) and the full

64

-rollout graph; two-way (dataset

\times

model) clustered

95\%

CIs.

Figure A10 : Rank fidelity of subsampled

d_{\mathrm{eff}}

( Table A12 ): terminal-frame ordering is reliable from

n\!\approx\!8

(

\rho=0.85

), while peak-frame fidelity improves steadily from

0.20

n{=}2

0.92

n{=}32

, reflecting the greater spectral spread near the trajectory maximum. The dotted line at

0.91

is the fidelity of per-rollout density against a different (same-answer recoverability) target; it does not describe

d_{\mathrm{eff}}

Analysis	Retained	Unit	Filter
Pooled trajectory ( Figure 1 )	$3{,}105$	cohorts/cell	none (frozen windows retained; all $16$ frames finite)
Cohort corpus ( Section 4.1 )	$3{,}105$	cohorts	5 math/sci, 50 shards
Binary-Jaccard ablation ( Appendix J )	$3{,}105$	cohorts	same corpus, binary graph
Spectral anatomy ( Appendix F )	$49{,}549$ / $2{,}999$	frames / cells	$d_{\perp}$ defined ( $m{<}1$ ) / all $16$ frames
Audit P1 / difficulty ( Table A9 )	$3{,}096$	cohorts	$\geq 10$ graded rollouts
Audit P2, AP lift ( Table A9 )	$1{,}906$	cohorts	defined AP ( $\geq 2$ answers) and $\geq 10$ graded rollouts

Appendix

Table A13 : Sample accounting. All analyses draw from the same fixed activation cache; the retained count differs only by an explicit per-analysis filter.

Mixture-of-Experts Models

Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.

See all 462 papers

Sep 28, 2026

MoRE: Scaling mixture of experts with hardware-aware low-rank routing

Mixture-of-Experts (MoE) layers are central to frontier language models, and recent architectures push toward more and smaller experts. In this regime, the standard linear router becomes a bottleneck: with

M

experts and hidden dimension

h

, its per-token cost

Θ(Mh)

dominates the MoE layer once

M

is large. We introduce MoRE (Mixture of Rank-reduced-routed Experts), which factorizes the router weight matrix at rank

r

and reduces the routing cost to

O((h + M)r)

. We prove that rank logarithmic in

M

suffices for routing expressivity when the number of active experts is fixed, and is necessary up to precision factors. We also prove that logarithmic rank preserves load balance in a Gaussian memorization model, and training on a synthetic phonebook task shows that low rank does not hurt memorization. At matched active FLOPs, the factorization allows a factor of

Θ(h/r)

more experts. To realize this gain in wall-clock time, we design a fused Triton kernel at inference that avoids expensive memory operations on HBM. Empirically, MoRE improves memorization on the phonebook task and performance on knowledge-intensive Q&A benchmarks after pretraining, while matching reasoning ability. Code available at https://github.com/Matheart/MoRE_code.

Honam Wong, Surbhi Goel, Enric Boix-Adserà

University of Pennsylvania · The Wharton School, University of Pennsylvania

Parameter-Free Mixture-of-Experts Routing Low-Rank Matrix Decomposition Mixture-of-Experts Inference LLM Inference Acceleration Mixture-of-Experts Models

Ask PDF

Apr 8, 2026

MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale

Sparse Mixture-of-Experts (MoE) architectures are increasingly popular for frontier large language models (LLM) but they introduce training challenges due to routing complexity. Fully leveraging parameters of an MoE model requires all experts to be well-trained and to specialize in non-redundant ways. Assessing this, however, is complicated due to lack of established metrics and, importantly, many routing techniques exhibit similar performance at smaller sizes, which is often not reflective of their behavior at large scale. To address this challenge, we propose the MoE Routing Testbed, a setup that gives clearer visibility into routing dynamics at small scale while using realistic data. The testbed pairs a data mix with clearly distinguishable domains with a reference router that prescribes ideal routing based on these domains, providing a well-defined upper bound for comparison. This enables quantifiable measurement of expert specialization. To demonstrate the value of the testbed, we compare various MoE routing approaches and show that balancing scope is the crucial factor that allows specialization while maintaining high expert utilization. We confirm that this observation generalizes to models 35x larger.

Tobias Falke, Nicolas Anastassacos, Samson Tan +6

Amazon AGI

Mixture of Experts Sparse Mixture-of-Experts Expert Routing Expert Load Balancing

Ask PDF

Apr 15, 2026

Geometric Routing Enables Causal Expert Control in Mixture of Experts

Sparse Mixture-of-Experts (MoE) models scale parameters while fixing active computation per token, but the specialization of individual experts remains opaque. In a companion paper we showed that routing topology is quality-neutral: five structurally different configurations converge to statistically equivalent language modeling quality. Here we show that expert identity is nonetheless causally meaningful: individual rank-1 experts are monosemantic by construction, and cosine-similarity routing in a low-dimensional metric space makes their specialization directly inspectable. We present four lines of evidence. First, projecting expert output vectors through the unembedding matrix yields a Semantic Dictionary: 15% of experts are monosemantic specialists spanning 10 categories (temporal, geographic, cardinal, discourse, emotional, financial, military, scientific). Second, routing exhibits a frequency-to-syntax gradient: early layers separate tokens by word frequency, deeper layers by syntactic class (Zipf-confound controls, all

p < 0.001

). Third, causal interventions confirm these labels: steering toward a temporal expert's centroid increases P(temporal) by +321% (median across 44 prompts); suppressing a geographic expert drops P(geographic) by -23%; rewriting an expert's output vector halves target-category probability, and effects compose additively across layers. Fourth, the interventions are not unique to cosine routing: linear routers support comparable steering, but only cosine routing provides geometric transparency -- expert specialization is readable directly from the centroid matrix. MoE expert-level specialization is a first-class interpretability primitive: architecturally monosemantic, causally validated, and controllable at inference with zero overhead.

Ivan Ternovtsii, Yurii Bilak

Department of Software Systems, Uzhhorod National University, Narodna sq. 3, Uzhhorod, Ukraine, 88000 · HengeBytes

Mixture-of-Experts Interpretability Mixture-of-Experts Expert Specialization Mixture of Experts Mixture-of-Experts Models Neural Network Interpretability Representation Geometry in Language Models

Ask PDF

Abstract

Mixture-of-Experts Models

Explore similar work

MoRE: Scaling mixture of experts with hardware-aware low-rank routing

MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale

Geometric Routing Enables Causal Expert Control in Mixture of Experts