cs.AISep 6, 2026
SaveFrom Concentration to Differentiation and Back: Routing Effective Rank in MoE Reasoning Cohorts
Organizations: Fudan University · Shanghai Innovation Institute
Abstract
Test-time scaling produces cohorts of reasoning rollouts, yet there is no standard label-free account of how their internal computation reorganizes as inference unfolds. We introduce routing effective rank deff, the entropy-effective dimensionality of a cross-rollout graph built from MoE expert-routing similarity. Across ten MoE configurations and five math/science benchmarks, deff exhibits a reproducible low-high-low trajectory, with a prominent interior maximum in 98.5% of 3,105 model-question cohorts: routing similarity is concentrated early, maximally differentiated at intermediate budgets, and reconcentrated later, and the timing of this maximum varies systematically with architecture and reasoning effort. An exact decomposition separates cohort-wide common-mode mass from residual spectral dimensionality: common-mode reallocation accounts for about two thirds of the trajectory, while the residual spectrum contributes about one quarter and retains substantial variation beyond the common mode. The decomposition further localizes behavior: among non-unanimous cohorts, increases in common-mode concentration strongly predict same-answer recoverability, and higher reasoning effort delays the maximum by 2.59 octaves (doublings of the token budget) and consistently expands the high-rank period across all four tested architectures, locating the effort effect in timing and duration rather than peak amplitude. Correctness comparisons separate structural monitoring from answer selection, positioning routing effective rank as a decomposable, label-free diagnostic of cohort organization - a principled spectral lens on how MoE reasoning cohorts differentiate and reconcentrate over inference time.
Figures & tables
Figure 1: Routing effective rank follows a reproducible low–high–low trajectory whose peak timing is architecture- and effort-dependent. One panel per architecture (fixed expert count ; Qwen3-Next-80B, , and the pooled curve are in Figure A1 ), one curve per reasoning-effort tier (medians). Triangles and whiskers mark the median and IQR of the peak budget, measured separately within each cohort ( Section 4.2 ). The maximum moves later with effort in all four architectures, whereas peak height is architecture-dependent. Panels share one -scale, but absolute levels depend on the routing representation and cohort size ( ): compare timing and shape within an architecture, not levels across panels.
Figure 2: From routing traces to , on one real cohort (OSS-120B-High, AIME-24). (1–2) Each rollout’s routing trace ( of experts per token per layer) is summarized, over a trailing -token window at budget , by a gate-weight histogram —from router activations alone; answer annotations only locate the commit window and label alignment. (3) Weighted-Jaccard similarities ( Equation 1 ) form the cohort graph over rollouts. (4) Its normalised spectrum gives ( Equation 2 ), from to . (5) Read at every budget, traces the low–high–low trajectory ( ; the ends need – eigendirections for of spectral mass, the peak ): routing similarity is first concentrated, then distributed across more spectral directions, and finally reconcentrated—a direct geometric readout of cohort organization.
Figure 3: Spectral anatomy of routing effective rank. The identity is exact, with the common-mode mass and the residual dimensionality (evaluated term by term on a real eigenspectrum in Figure A4 ). (A) Aligned per-family medians: as rises, spectral mass drains out of the common mode; as it falls, mass returns. The residual spectrum contributes complementary motion on a smaller scale. (B) Over the full budget path the common-mode term carries of the absolute motion of , the residual term , and mixing (two-way clustered CIs); the remaining rows repeat the accounting on the early and late halves of the budget grid (details and controls in Appendix F ). (C) Residual dimensionality retains substantial variation beyond the common mode: a grouped cross-validated fit of on explains of out-of-sample variance, leaving as a distinct spectral axis for RQ3.
Figure 4: Common-mode concentration predicts same-answer recoverability; reasoning effort expands the high-rank regime. Top: across adjacent-budget steps (clustered on questions configurations), increases in common-mode mass make same-answer rollout pairs easier to recover from routing (the prevalence-corrected AP lift plotted here): , while the corresponding residual coefficient is ; a within-question time shuffle collapses both toward zero. Bottom: higher effort delays the maximum, increases high-rank occupancy, widens the regime, and lowers integrated common-mode mass, with the same sign in all four architectures.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Family | mean | median | per-cohort [IQR] | |
|---|---|---|---|---|
| OSS | ||||
| Qwen-Instruct | ||||
| Qwen-Thinking | ||||
| pooled | — |
Table A1 : Three peak-budget calipers: argmax of the family mean curve, of the family median curve, and the median of per-cohort argmax positions [IQR]. The paper’s primary caliper is the per-cohort median (rightmost); the family-mean caliper is shown because aggregated curves are what a reader sees in a plot.
Figure A1 : Full four-architecture version of Figure 1 , including Qwen3-Next-80B ( ). Same data, calipers, and conventions: triangles give the per-cohort argmax median with IQR whiskers; the inset is the pooled mixture curve. Qwen3-Next-80B repeats the low–high–low shape with a later clock and is the one architecture whose peak height inverts with effort (per-cohort median peak : Instruct vs. Thinking ), while its fixed-budget levels remain effort-monotone ( Table A2 ).
Figure A2 : The dataset sets the ordering of the effective-rank clock, nearly deterministically across configurations. Left: median peak-normalized trajectories per dataset for one representative configuration per architecture (High/Thinking tier). Right: each configuration’s median peak budget per dataset (dots; all ten configurations) with cross-configuration medians (diamonds). The dataset ordering of peak timing is almost identical in every configuration (Kendall , within-configuration permutation ): GPQA peaks – octaves earlier than the math benchmarks (pooled median vs. for HMMT’25), and HMMT’25 is latest with the widest high-rank period ( , octaves at : median vs. )—consistent with the peak tracking each dataset’s length scale rather than a universal token count.
| Configuration | mean | median ( ) | |
|---|---|---|---|
| OSS-20B-Low | ( ) | ||
| OSS-20B-Med | ( ) | ||
| OSS-20B-High | ( ) | ||
| OSS-120B-Low | ( ) | ||
| OSS-120B-Med | ( ) | ||
| OSS-120B-High | ( ) |
Table A2 : Mean/median at per configuration (weighted Jaccard). Monotone in reasoning effort within each architecture.
| Family | at | at own peak | at commit |
|---|---|---|---|
| OSS | |||
| Qwen-Instruct | |||
| Qwen-Thinking |
Table A3 : Family mean at a fixed budget, at each family’s own trajectory peak, and at commit (family-mean caliper; Appendix A ). The OSS -vs-reasoning separation holds at all three readouts, and the OSS effort tiers stay ordered at commit ( at 20B, at 120B). The Instruct -vs- Thinking ordering, by contrast, is specific to the fixed mid-budget reading: it narrows at the peak-aligned readout ( Thinking peaks later) and reverses at commit ( vs. ), so fixed-budget readings conflate level with timing.
| Aggregation | consensus | correctness |
|---|---|---|
| pooled-all | ||
| per-shard mean | ||
| per-family: OSS | ||
| per-family: Instruct | ||
| per-family: Thinking |
Table A4 : (commit-graph , difficulty). The consensus–correctness gap is clear for OSS , for Instruct , and reverses for Thinking . Pooled-all is a single Spearman over the cohorts with a defined majority share (one cohort produced no gradeable answer); per-shard mean averages the within-shard Spearman over the model dataset shards ( bootstrap CIs).
Figure A3 : The trajectory is a per-cohort fact, not an aggregation effect. Each of the rows is one (model, question) cohort, standardised within the row and sorted by the budget at which it peaks; the left stripe gives the architecture. The bright band runs diagonally—the sort constructs the diagonal; the band’s contrast, not its diagonality, carries the information—and the architecture stripe shows how the pooled peak combines architecture-specific clocks: OSS-20B cohorts concentrate at early peaks and Qwen3-Next-80B cohorts at late ones.
| Stratum | b32 | b128 | b512 | b2048 | b8192 | b32768 | |
|---|---|---|---|---|---|---|---|
| easy ( ) | |||||||
| mid ( ) | |||||||
| hard ( ) | |||||||
| contested ( ) | |||||||
| all defined-AP |
Table A5 : Same-answer AP by budget and difficulty stratum (defined-AP cohorts). The column grid omits for space; the pooled defined-AP there is (between ’s and ’s ). The last column is the same-answer AP of the commit-window graph itself ( pooled); b32768 is the largest prefix budget. Absolute levels are not comparable across strata (each stratum has its own same-answer prevalence, hence its own random-AP baseline).
Figure A4 : The exact identity on one real, typical eigenspectrum. A normalised eigenspectrum (log scale) with the common-mode mass highlighted; the annotation evaluates term by term on this frame, and the two sides agree to the printed precision. The example is the committed per-shard spectrum whose is closest to the median of those examples—typical rather than cherry-picked.
Figure A5 : The trajectory is a redistribution of spectral mass. Mean normalised eigenspectrum over cohorts (band: IQR), at the first budget, at each cohort’s own peak (peak-aligned, since family peak budgets differ several-fold; Table A1 ), and at the last budget. The leading direction holds early, at the peak and late, while the tail beyond rank moves the opposite way ( ). The low–high–low form in is exactly this shift of mass out of and back into the leading eigendirections. Note that the two ends are spectrally alike, not merely equal in : differs by between them. Cohorts with fewer than graded rollouts are excluded so that every curve is averaged over the same eigenvalue ranks.
| Segment | mixing | common mode | residual | ||
|---|---|---|---|---|---|
| early half ( ) | |||||
| late half ( ) | |||||
| late half, live only | |||||
| rise (peak-aligned) | |||||
| fall (peak-aligned) | |||||
| fall, live only |
Table A6 : Attribution of trajectory motion by the midpoint split of Equation 3 : share of carried by the mixing term , the common-mode term, and the residual term, per segment (two-way clustered CIs; all common-mode and residual shares are positive in configurations; mixing shares are n.s. with MDE 80 – ). “Live” restricts the segment to still-generating rollouts. Shares are pooled ratio-of-sums over each arm’s full cell set; the pre-registered H2 contrast of Appendix F is the same functional on the cells matched across both halves (excluding late-half segments whose is exactly zero under frozen windows), where the early share is —so the contrast is , not the difference of the displayed rows.
| Subset | frames | CV | Spearman |
|---|---|---|---|
| all | |||
| OSS | |||
| Qwen-Instruct | |||
| Qwen-Thinking | |||
| all, rising branch | |||
| all, falling branch |
Table A7 : Grouped cross-validated fit of on (folds grouped by question; frames from the complete cells). The residual axis is only partially predictable from the common mode.
| Variant | med. | med. | med. | prom. | |
|---|---|---|---|---|---|
| real ( ) | |||||
| id. permutation | |||||
| pseudo-cohorts ( ) |
Table A8 : Nulls for the anatomy experiment, on its own -shard pool and prominence gate (not comparable to the full-corpus gate; Appendix A ). The expert-identity permutation drives the residual spectrum toward its near-isotropic ceiling ( ); cross-problem pseudo-cohorts of unrelated rollouts retain the coarse arc.
| Target | controls | [ ] | ||||||
|---|---|---|---|---|---|---|---|---|
| (commit, P1) | ||||||||
| (terminal, sens.) | ||||||||
| AP lift (terminal, P2) | ||||||||
| difficulty (secondary) |
Table A9 : Semantic-audit ladders: out-of-fold under GroupKFold by question. carries two-way clustered CIs; the Bonferroni intervals for the two regression primaries are P1 and P2 —both span zero. Density and common-mode mass provide nearly identical predictive value on these endpoints ( : on P1, on P2), localizing the observable association to the global agreement channel; the spectral formulation contributes exact channel attribution and an independent residual coordinate rather than scalar predictive lift.
| Architecture | |||||||
|---|---|---|---|---|---|---|---|
| OSS-20B | |||||||
| OSS-120B | |||||||
| Qwen3-30B | |||||||
| Qwen3-Next-80B | |||||||
| Pooled | — |
Table A10 : Effort localization, paired within architecture (higher minus lower tier, matched by question; question-clustered CIs; pooled row clustered by question architecture, architecture clusters). (peak height) is the registered primary endpoint of this contrast—it varies by architecture, with a pooled interval spanning zero; the remaining columns are secondary endpoints. with the normalized budget-grid index; is the octave width of the contiguous regime. The consistent effects are later maxima, greater peak-normalized occupancy, a wider high-rank interval, and weaker integrated common-mode mass. Residual-spectrum area has no common shift.
| Feature | cohort AUC [ ] | rollout AUC |
|---|---|---|
| vote margin | ||
| (neg.) | ||
| question prior (LOMO) | ||
| (terminal) | ||
| density (terminal) | ||
| (terminal) |
Table A11 : Correctness comparison: fit-free single-feature AUCs (cohorts with graded rollouts). We retain the pre-registered one-sided reading and do not reinterpret the coefficient sign post hoc; answer-side baselines remain stronger than standalone routing features for correctness selection.
Figure A6 : Out-of-sample model ladders for the semantic endpoints. Grouped-CV (upper number) and its increment over controls (lower number). The predictive ladder localizes answer-space coupling to the global agreement channel: density and common-mode mass provide nearly identical predictive value on these targets, while the residual axis remains the distinct structural coordinate of RQ2 (increments bounded in Figure A8 ). The scalar lies between the agreement-only and split-spectrum models.
Figure A7 : Higher effort moves the effective-rank maximum later in every architecture. Peak-normalized trajectories for paired lower- and higher-effort configurations, with paired peak-delay intervals on the right. The timing effect is uniform; peak height is architecture-dependent.
Figure A8 : Residual terminal incremental associations are bounded on the tested targets. Incremental out-of-fold from adding residual dimensionality after common-mode mass. Thick whiskers are and thin whiskers Bonferroni two-way clustered intervals. The intervals constrain independent residual contributions on the tested targets ( upper bounds: for answer effective count, for AP lift) while remaining compatible with small effects.
Figure A9 : Correctness selection favors answer-side statistics. Fit-free single-feature AUCs for cohort majority correctness (circles) and rollout-level correctness (diamonds); dotted line: chance; two-way clustered CIs ( , uncorrected). Text-side baselines sit at – (the question prior uses the other nine configurations’ outcomes), while the best routing feature reaches , separating answer selection from the structural-monitoring role studied in the main paper.
| terminal frame | peak frame | |
|---|---|---|
Table A12 : Rank fidelity of subsampled : median over (model, dataset) shards of the Spearman correlation, across a shard’s cohorts, between on a random -rollout subcohort ( draws) and the full -rollout graph; two-way (dataset model) clustered CIs.
Figure A10 : Rank fidelity of subsampled ( Table A12 ): terminal-frame ordering is reliable from ( ), while peak-frame fidelity improves steadily from at to at , reflecting the greater spectral spread near the trajectory maximum. The dotted line at is the fidelity of per-rollout density against a different (same-answer recoverability) target; it does not describe .
| Analysis | Retained | Unit | Filter |
|---|---|---|---|
| Pooled trajectory ( Figure 1 ) | cohorts/cell | none (frozen windows retained; all frames finite) | |
| Cohort corpus ( Section 4.1 ) | cohorts | 5 math/sci, 50 shards | |
| Binary-Jaccard ablation ( Appendix J ) | cohorts | same corpus, binary graph | |
| Spectral anatomy ( Appendix F ) | / | frames / cells | defined ( ) / all frames |
| Audit P1 / difficulty ( Table A9 ) | cohorts | graded rollouts | |
| Audit P2, AP lift ( Table A9 ) | cohorts | defined AP ( answers) and graded rollouts |
Table A13 : Sample accounting. All analyses draw from the same fixed activation cache; the retained count differs only by an explicit per-analysis filter.
More in this topic
Mixture-of-Experts Models
Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.
See all 462 papers