Routing in Gradient Space: Balanced Usage Is Not Expert Specialization

Yuchen Li; Mingyu Du; Zongqi Fan; Nguyen H. Tran; Ken-Tye Yong

cs.LGSep 29, 2026

Routing in Gradient Space: Balanced Usage Is Not Expert Specialization

Authors: Yuchen Li, Mingyu Du, Zongqi Fan, Nguyen H. Tran, Ken-Tye Yong

Organizations: University of Sydney · University of New South Wales

Abstract

Sparse expert models can distribute traffic evenly while still grouping incompatible training signals within the same experts. We study routing as a gradient-partitioning problem and introduce gradient-aligned routing (GAR), whose load-normalized router objective rewards grouping observations with aligned gradients. On five multi-task text-classification mixtures, we compare GAR with task-loss-only routing, gradient-combination and gradient-conflict methods, and load-balancing losses. With a fully trainable RoBERTa backbone and classification-head experts, GAR has the highest aggregate validation accuracy, 1.07 percentage points above task-loss-only routing. With frozen DeBERTa and Qwen3-1.7B backbones and low-rank adapter experts, it again ranks first, 1.10 points above task-loss-only routing, with better-balanced expert load and higher gradient-mass purity, the share of each expert's gradient-norm mass from its dominant task; the load-balancing losses flatten load further but leave this purity near its task-loss-only level. Top-1 routing, trainable full-parameter feed-forward network (FFN) experts, and a larger backbone also show positive aggregate gains. The results distinguish expert-load balance from gradient-based routing organization and indicate the predictive value of gradient-informed routing in multi-task text classification.

Figures & tables

Panel of Figure 1 — Figure 1: Shared task pathway (solid) and added alignment pathway (dashed). Expert dispatch uses top-4 routing over eight experts. Detached group gradients and recomputed group probabilities form an auxiliary loss with no direct gradient to non-router parameters; task loss trains the predictive model. All resulting gradients subsequently undergo shared global clipping.

(a) Method	RoBERTa- head †	Head, top-1 †	DeBERTa- FFN †	Frozen- LoRA	Qwen3-8B	Single- task ‡
Baseline	0.7902	0.8039	0.8478	0.7593	0.7328	0.8820
CAGrad	0.7824	0.8066	0.8507	0.7625	0.7401	0.8799
GAR	0.8008	0.8145	0.8594	0.7702	0.7452	0.8913
$\Delta_{\text{Base}}$ (pp)	$+1.065$	$+1.056$	$+1.155$	$+1.095$	$+1.239$	$+0.929$
95% CI	$[0.697{,}1.433]$	$[0.831{,}1.282]$	$[0.730{,}1.579]$	$[0.661{,}1.529]$	$[0.514{,}1.965]$	$[0.207{,}1.651]$

Table 1: Endpoint comparisons over the five mixtures in Table 4 (five seeds). (a) Mean accuracy and paired GAR

-

Baseline gains (

\Delta_{\text{Base}}

, pp; 95% Student-

t

intervals across seeds).

\dagger

: trainable backbone; RoBERTa-head uses sequence-routed classification-head LoRA experts (three of seven methods shown); Head, top-1 uses straight-through one-hot gating (Appendix F.4 ); DeBERTa-FFN uses token-routed full-parameter FFN experts. Frozen-LoRA averages frozen DeBERTa and Qwen3-1.7B; Qwen3-8B is a frozen model-scale extension.

\ddagger

: DeBERTa single-task E4K2 controls average [CoLA, MRPC, RTE, SST-2, QQP], weighting frozen-LoRA and trainable-FFN adaptations equally. (b) Seven-method frozen LoRA-FFN two-backbone means. Bold marks the best accuracy in (a) and best metric values in (b). Full details are in Appendix F .

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Gradient-space routing intuition. The criterion groups gradient observations by signed associations. The expert clusters illustrate the grouping preference, rather than learned linear subspaces.

Analyzed property	Appendix subsection	Formal support
Idealized setting	Appendix C.1	Assumptions on manifold priors, gradient concentration, finite-window sampling, and the baseline dynamics.
Idealized baseline feedback	Appendix C.2	Proposition C.1 .
Load-normalized anti-amplification	Appendix C.3	Proposition C.2 .
Static directional tilt and coherence preference	Appendix C.4 and Appendix C.5	Proposition C.4 plus Lemmas C.3 , C.6 , and C.7 .
Collapsed assignments are not local maxima	Appendix C.6	Theorem C.8 for the leading-order alignment surrogate.

Appendix

Table 3: Map of load-scaling and routing-geometry analyses in Appendix C .

Mixture	Constituent classification datasets
Five tasks	QNLI, BoolQ, RTE, PAWS, WiC
Six tasks	QNLI, BoolQ, RTE, PAWS, ANLI, CB
Seven tasks (PAWS)	QNLI, BoolQ, RTE, PAWS, WiC, CoLA, SST-2
Seven tasks (MRPC)	QNLI, BoolQ, RTE, WiC, CoLA, SST-2, MRPC
Eight tasks	QNLI, BoolQ, RTE, PAWS, WiC, CoLA, SST-2, CB

Appendix

Table 4: The five task mixtures used throughout the experiments. Each named dataset supplies one supervised classification task. The two seven-task mixtures are distinguished by PAWS or MRPC; these descriptors always refer to the complete dataset sets below.

[QNLI, BoolQ, RTE, PAWS, WiC]
Method	Learning rate	Grad. clip	CAGrad $c$	GAR $\lambda$
Baseline	$1.1\times 10^{-4}$	1.2	–	–
CAGrad	$1.4\times 10^{-4}$	1.2	0.22	–
GAR	$1.4\times 10^{-4}$	1.2	–	$1.8\times 10^{-4}$
[QNLI, BoolQ, RTE, PAWS, ANLI, CB]
Baseline	$7.4\times 10^{-5}$	1.0	–	–

Appendix

Table 11: Selected Qwen3-8B configurations for the five–eight-task mixtures. Dashes indicate an inapplicable method coefficient.

Method	Learning rate	Grad. clip	Weight decay	Method-specific setting
[QNLI, BoolQ, RTE, PAWS, WiC] (5 tasks)
Baseline	$1.2\mathrm{e}{-4}$	$1.1$	$0.02$	–
CAGrad	$1.0\mathrm{e}{-4}$	$1.1$	$0.02$	$c=0.42$ ; inner lr $=0.1$
GAR	$1.0\mathrm{e}{-4}$	$1.1$	$0.02$	$\lambda_{\mathrm{align}}=10^{-3}$ ; load norm.
[QNLI, BoolQ, RTE, PAWS, ANLI, CB] (6 tasks)
Baseline	$1.0\mathrm{e}{-4}$	$0.8$	$0.02$	–

Appendix

Table 15: Final configurations for the trainable RoBERTa classification-head top-1 comparison. All five mixtures use E8K1 straight-through routing on the sequence-level router, rank-16 LoRA experts on the classification head (

\alpha=16

), BF16 forward/backward with FP32 router numerics, sequence length 256, per-task batch size 32, same-task groups of eight examples, warmup ratio 0.1, and 2,000 optimizer updates. Gradient clipping and weight decay are shared by the three methods within each mixture.

Task	Method	Learning rate	Grad. clip	Weight decay	Method-specific setting
CoLA	Baseline	$9.4\mathrm{e}{-6}$	$1.1$	$0$	–
	CAGrad	$9.4\mathrm{e}{-6}$	$1.1$	$0$	$c=0.31$ ; inner lr $=0.1$
	GAR	$9.4\mathrm{e}{-6}$	$1.1$	$0$	$\lambda_{\mathrm{align}}=1.7\mathrm{e}{-4}$ ; load norm.
MRPC	Baseline	$9.0\mathrm{e}{-5}$	$1.0$	$0.0044$	–
	CAGrad	$7.8\mathrm{e}{-5}$	$1.0$	$0.0044$	$c=0.91$ ; inner lr $=0.1$
	GAR	$5.5\mathrm{e}{-5}$	$1.0$	$0.0044$	$\lambda_{\mathrm{align}}=1\mathrm{e}{-3}$ ; load norm.

Appendix

Table 17: Final configurations for DeBERTa single-task LoRA results (SST-2 and QQP use 3 epochs; CoLA, MRPC, and RTE use 5 epochs).

Task	Method	Learning rate	Grad. clip	Weight decay	Method-specific setting
CoLA	Baseline	$2.0\mathrm{e}{-5}$	$1.0$	$0.017$	–
	CAGrad	$2.0\mathrm{e}{-5}$	$1.0$	$0.017$	$c=0.54$ ; inner lr $=0.1$
	GAR	$1.0\mathrm{e}{-5}$	$1.0$	$0.017$	$\lambda_{\mathrm{align}}=2.8\mathrm{e}{-5}$ ; load norm.
MRPC	Baseline	$1.1\mathrm{e}{-4}$	$1.0$	$0.011$	–
	CAGrad	$7.8\mathrm{e}{-5}$	$1.0$	$0.011$	$c=0.91$ ; inner lr $=0.1$
	GAR	$3.4\mathrm{e}{-5}$	$1.0$	$0.011$	$\lambda_{\mathrm{align}}=7.6\mathrm{e}{-4}$ ; load norm.

Appendix

Table 18: Final configurations for DeBERTa single-task FFN results (SST-2 and QQP use 3 epochs; CoLA, MRPC, and RTE use 5 epochs).

Mixture	LR	Clip	WD	Batch/group	Seq.
[QNLI, BoolQ, RTE, PAWS, WiC]	$1.0\mathrm{e}{-3}$	1.1	0.010	32/8	256
[QNLI, BoolQ, RTE, PAWS, ANLI, CB]	$1.0\mathrm{e}{-3}$	0.8	0.010	32/8	256
[QNLI, BoolQ, RTE, PAWS, WiC, CoLA, SST-2 ]	$2.0\mathrm{e}{-3}$	0.8	0.010	32/8	256
[QNLI, BoolQ, RTE, WiC, CoLA, SST-2 , MRPC]	$1.0\mathrm{e}{-3}$	0.8	0.010	32/8	256
[QNLI, BoolQ, RTE, PAWS, WiC, CoLA, SST-2 , CB]	$1.0\mathrm{e}{-3}$	0.8	0.010	32/8	256

Appendix

Table 19: Fixed RoBERTa LoRA-FFN configurations for the five-mixture E8K4 coefficient sweep. Each mixture reuses its selected GAR configuration, with 1,000 updates and

\lambda\in\{0,10^{-5},10^{-4},10^{-3},10^{-2}\}

. All rows use a frozen backbone, final-layer LoRA-FFN experts, token-level routing, and FP32 computation, rank 16, scaling 16, load normalization (

\varepsilon=10^{-8}

), and warmup ratio 0.1. LR denotes learning rate, Clip the gradient-clipping threshold, WD weight decay, and Seq. the maximum sequence length. Brackets list constituent tasks; batch/group is per task.

Mixture	LR	Clip	WD	Batch/group	Seq.
[QNLI, BoolQ, RTE, PAWS, WiC]	$1.0\mathrm{e}{-4}$	1.0	0.032	32/8	256
[QNLI, BoolQ, RTE, PAWS, ANLI, CB]	$1.2\mathrm{e}{-4}$	0.8	0.010	32/8	256
[QNLI, BoolQ, RTE, PAWS, WiC, CoLA, SST-2 ]	$7.0\mathrm{e}{-5}$	0.8	0.010	32/8	256
[QNLI, BoolQ, RTE, WiC, CoLA, SST-2 , MRPC]	$7.0\mathrm{e}{-5}$	1.1	0.020	32/8	256
[QNLI, BoolQ, RTE, PAWS, WiC, CoLA, SST-2 , CB]	$7.0\mathrm{e}{-5}$	1.0	0.010	32/8	256

Appendix

Table 20: Fixed RoBERTa classification-head LoRA configurations for the five-mixture E8K4 coefficient sweep. Each mixture reuses its selected GAR configuration, with 1,000 updates and

\lambda\in\{0,10^{-5},10^{-4},10^{-3},10^{-2}\}

. All rows use a trainable backbone, classification-head LoRA experts, sequence-level routing, and BF16 computation with FP32 router numerics, rank 16, scaling 16, load normalization (

\varepsilon=10^{-8}

Method	Objective	Auxiliary gradient path
Baseline	$\mathcal{L}_{\mathrm{task}}$	None.
STGC	$\mathcal{L}_{\mathrm{task}}+\beta_{\mathrm{STGC}}\sum_{m}v_{m}\mathcal{C}_{m}$	Through logits; conflict masks detached.
STGC+Load	$\mathcal{L}_{\mathrm{task}}+\beta_{\mathrm{STGC}}\sum_{m}v_{m}\mathcal{C}_{m}+\lambda_{\mathrm{load}}E\sum_{e}\mathrm{stopgrad}(f_{e}^{(1)})P_{e}$	Through logits; conflict masks and hard frequencies detached.
LoadPen	$\mathcal{L}_{\mathrm{task}}+\lambda_{\mathrm{load}}E\sum_{e}\mathrm{stopgrad}(f_{e}^{(1)})P_{e}$	Directly through the router; routing inputs detached.
SwitchAux	$\mathcal{L}_{\mathrm{task}}+\alpha_{\mathrm{switch}}\sum_{m}u_{m}E\sum_{e}\mathrm{stopgrad}(f_{m,e}^{(k)})P_{m,e}$	Directly through the router for frozen FFN; ordinary routing path for the trainable head.

Appendix

Table 21: Implemented scalar control objectives and their auxiliary gradient paths. Task-loss gradients follow the ordinary training path in every row. CAGrad is a gradient-combination control, described separately in the update semantics.

Single-task MRPC
Setting	Method	Step (s)	Wall (s)	Factor
Frozen LoRA	Baseline	0.0528	13.27	1.00 $\times$
Frozen LoRA	GAR	0.0593	14.77	1.12 $\times$
Trainable FFN	Baseline	$0.319{\pm}0.003$	$409.6{\pm}4.5$	1.00 $\times$
Trainable FFN	GAR	$0.329{\pm}0.002$	$422.5{\pm}2.0$	1.03 $\times$
Five-task mixture [QNLI, BoolQ, RTE, PAWS, WiC]

Appendix

Table 23: Runtime summary for DeBERTa runs, Baseline versus GAR. Single-task MRPC rows: the frozen-backbone LoRA rows report a one-epoch local probe with top-

k

gating, and the non-frozen FFN rows report the corresponding five-seed summaries; both use effective batch size 16 and two gradient observations per update. Five-task rows: frozen-backbone LoRA-FFN runs on [QNLI, BoolQ, RTE, PAWS, WiC] with E8K4 routing, per-task batch size 32, and same-task groups of eight (20 gradient observations per update), over 500 updates and five seeds; step time excludes validation and model loading, and wall time is the corresponding 500-update training time. All rows use the same data-loading and scheduling path as the final runs. Relative factors compare mean step time with the corresponding baseline; for the five-seed rows, step and wall time are reported as mean

\pm

population standard deviation.

[QNLI, BoolQ, RTE, PAWS, WiC]
Method	Acc	Seed std	LVar	Purity	Util.
Baseline	0.7268	0.0077	0.01229	0.5835	0.600
CAGrad	0.7360	0.0030	0.00966	0.6466	0.700
GAR	0.7326	0.0033	0.00790	0.6401	0.650
[QNLI, BoolQ, RTE, PAWS, ANLI, CB]
Baseline	0.6950	0.0113	0.05777	0.4679	0.375

Appendix

Table 30: Qwen3-8B five–six-task mixture endpoints. Acc is equal-task macro validation accuracy; seed std is its population standard deviation over five seeds. Routing metrics use the final diagnostic checkpoint; Purity is gradient-mass purity.

[QNLI, BoolQ, RTE, PAWS, WiC, CoLA, SST-2]
Method	Acc	Seed std	LVar	Purity	Util.
Baseline	0.7466	0.0088	0.01096	0.3761	0.625
CAGrad	0.7509	0.0075	0.01623	0.3934	0.600
GAR	0.7641	0.0059	0.00713	0.4391	0.775
[QNLI, BoolQ, RTE, WiC, CoLA, SST-2, MRPC]
Baseline	0.7475	0.0048	0.01020	0.5165	0.725

Appendix

Table 31: Qwen3-8B seven–eight-task mixture endpoints. Acc is equal-task macro validation accuracy; seed std is its population standard deviation over five seeds. Routing metrics use the final diagnostic checkpoint; Purity is gradient-mass purity.

Scope	Method	Acc	LVar	Purity	Util.
Five mixtures	Baseline	0.7328	0.02183	0.4633	0.560
	CAGrad	0.7401	0.01890	0.4789	0.595
	GAR	0.7452	0.00813	0.5233	0.690

Appendix

Table 32: Qwen3-8B aggregate endpoints. Every mixture and seed has equal weight across the five dataset mixtures, whose constituent tasks are listed in brackets in the preceding tables. These three-method means do not enter the two-backbone, seven-method frozen aggregate.

Scope	GAR $-$ Baseline	GAR $-$ CAGrad
Five mixtures	$+1.239$ [ $+0.514$ , $+1.965$ ]	$+0.505$ [ $+0.059$ , $+0.950$ ]

Appendix

Table 33: Paired Qwen3-8B accuracy gains (percentage points). Within each shared seed, first average the mixture endpoints and then subtract the comparator. Brackets are two-sided 95% Student-

t

intervals over the five paired seed differences (four degrees of freedom); no multiplicity adjustment is applied.

Scope	GAR $-$ Baseline	GAR $-$ CAGrad
RoBERTa	$+1.141$ [ $+0.760$ , $+1.523$ ]	$+1.078$ [ $+0.632$ , $+1.523$ ]
DeBERTa	$+1.176$ [ $+0.687$ , $+1.666$ ]	$+0.917$ [ $+0.350$ , $+1.485$ ]
Qwen3-1.7B	$+1.014$ [ $+0.260$ , $+1.768$ ]	$+0.638$ [ $-0.099$ , $+1.375$ ]
Two-backbone mean (DeBERTa, Qwen3-1.7B)	$+1.095$ [ $+0.661$ , $+1.529$ ]	$+0.778$ [ $+0.266$ , $+1.289$ ]
Three-backbone mean	$+1.111$ [ $+0.764$ , $+1.457$ ]	$+0.878$ [ $+0.596$ , $+1.160$ ]

Appendix

Table 34: Paired accuracy differences for the five mixtures (five to eight tasks), in percentage points. For each backbone and seed, we average the equal-task macro validation accuracy over the five matched dataset sets and then subtract the comparator’s result from GAR’s for the same seed. The last two rows first average the two (DeBERTa, Qwen3-1.7B) or three backbone-specific differences within each seed; the two-backbone mean is the frozen aggregate used in the main text. Brackets are two-sided 95% Student-

t

confidence intervals over the five paired seed differences (

n=5

, four degrees of freedom). No multiplicity correction is applied.

[QNLI, BoolQ, RTE, PAWS, WiC] (5 tasks)
Method	Acc	Seed std	LVar	Gradient-mass purity	Util.
Baseline	0.6294	0.0091	0.01398	0.4734	0.575
CAGrad	0.6288	0.0093	0.00937	0.4222	0.700
GAR	0.6367	0.0076	0.00848	0.4966	0.675
[QNLI, BoolQ, RTE, PAWS, ANLI, CB] (6 tasks)
Baseline	0.6375	0.0069	0.02184	0.4472	0.575

Appendix

Table 35: Frozen RoBERTa LoRA-FFN with top-1 routing (E8K1), trained for 2,000 optimizer updates on each of the five mixtures. Columns report equal-task macro validation accuracy, its population seed standard deviation, LVar, gradient-mass purity, and utilization at the final evaluation, with metrics defined as in Appendix E . Five seeds per cell. For the aggregate block, each metric is first averaged equally over the five mixtures within each seed; the displayed mean and accuracy standard deviation are then computed across the five seeds.

Five mixtures (five to eight tasks)
Quantity	GAR $-$ Baseline	GAR $-$ CAGrad
Accuracy (pp)	$+0.769$ [ $+0.595$ , $+0.944$ ]	$+0.577$ [ $-0.243$ , $+1.398$ ]
LVar	$-0.00450$ [ $-0.00604$ , $-0.00296$ ]	$-0.00518$ [ $-0.00869$ , $-0.00167$ ]
Gradient-mass purity	$+0.0386$ [ $+0.0281$ , $+0.0491$ ]	$+0.0287$ [ $+0.0065$ , $+0.0509$ ]
Utilization	$+0.0850$ [ $+0.0071$ , $+0.1629$ ]	$+0.0700$ [ $+0.0064$ , $+0.1336$ ]
Accuracy by task-count bin (pp)

Appendix

Table 36: Paired final-evaluation differences for frozen RoBERTa LoRA-FFN with top-1 routing (E8K1; 2,000 optimizer updates). For each seed, metrics are averaged equally over the five mixtures before subtracting the comparator for that seed. The lower block repeats the accuracy comparison within the five–six- and seven–eight-task groups. Accuracy is in percentage points; LVar, gradient-mass purity, and utilization are in their native units. Brackets are unadjusted two-sided 95% Student-

t

confidence intervals over the five paired seed differences (

n=5

, four degrees of freedom).

Metric	GAR $-$ Baseline	GAR $-$ CAGrad	GAR $-$ STGC
Accuracy (pp)	$+1.07$ [ $+0.70$ , $+1.43$ ]	$+1.84$ [ $+1.29$ , $+2.39$ ]	$+0.90$ [ $+0.35$ , $+1.46$ ]
LVar	$-0.0038$ [ $-0.0050$ , $-0.0026$ ]	$-0.0040$ [ $-0.0059$ , $-0.0020$ ]	$+0.0038$ [ $+0.0030$ , $+0.0047$ ]
Utilization	$+0.130$ [ $+0.041$ , $+0.219$ ]	$+0.145$ [ $+0.078$ , $+0.212$ ]	$-0.180$ [ $-0.231$ , $-0.129$ ]
Grad. purity	$+0.0111$ [ $-0.0156$ , $+0.0378$ ]	$-0.0085$ [ $-0.0492$ , $+0.0323$ ]	$+0.1025$ [ $+0.0823$ , $+0.1227$ ]
Struct. purity	$+0.0198$ [ $+0.0009$ , $+0.0387$ ]	$+0.0182$ [ $+0.0016$ , $+0.0349$ ]	$+0.0342$ [ $+0.0225$ , $+0.0459$ ]
NMI	$+0.0195$ [ $+0.0047$ , $+0.0343$ ]	$+0.0222$ [ $+0.0035$ , $+0.0409$ ]	$+0.0479$ [ $+0.0376$ , $+0.0583$ ]

Appendix

Table 39: Paired endpoint differences (GAR minus Baseline, CAGrad, STGC, LoadPen, SwitchAux, and STGC+Load) in the trainable RoBERTa classification-head setting on the five mixtures (five to eight tasks). Metrics are first averaged over mixtures within each seed; brackets are two-sided 95% Student-

t

intervals over five paired differences. Accuracy is in percentage points; routing entropy is normalized by

\log 8

. Lower LVar indicates more uniform load, lower inter-expert similarity indicates greater directional separation, and lower entropy indicates more concentrated routing. Entropy, inter-expert similarity, NMI, and ARI have no universally preferred direction. No multiplicity correction is applied. The lower block compares GAR with the three controls that include an explicit load-balancing term.

Figure 4: Routing trajectories for all seven methods in the trainable RoBERTa classification-head setting (E8K4), aggregated over the five mixtures. At each checkpoint, mixtures are averaged within each run before the five run-level means are averaged. Checkpoints are 200, 400,

\ldots

, 2,000 optimizer updates; bands are

\pm 1

standard error over the five run-level mixture means. Left to right: load variance, utilization, task–expert NMI, and routed-unit structure purity.

Figure 5: Complementary routing diagnostics for all seven methods in the trainable RoBERTa classification-head setting, using the same five-mixture and checkpoint aggregation as Figure 4 . Left to right: task–expert ARI, gradient-mass purity, intra-expert coherence, inter-expert similarity, and routing entropy normalized by

\log 8

. Bands are

\pm 1

standard error over the five run-level mixture means.

[QNLI, BoolQ, RTE, PAWS, WiC] (5 tasks)
Method	Acc	Seed std	LVar	Gradient-mass purity	Util.
Baseline	0.7864	0.0101	0.10806	0.1976	0.125
CAGrad	0.7897	0.0081	0.03731	0.5230	0.375
GAR	0.8038	0.0074	0.02377	0.5832	0.525
[QNLI, BoolQ, RTE, PAWS, ANLI, CB] (6 tasks)
Baseline	0.7818	0.0064	0.04267	0.2903	0.350

Appendix

Table 40: Trainable RoBERTa classification-head LoRA MoE with top-1 routing (E8K1), trained for 2,000 optimizer updates on each of the five mixtures. Columns report equal-task macro validation accuracy, its population seed standard deviation, LVar, gradient-mass purity, and utilization at the final evaluation, with metrics defined as in Appendix E . Five seeds per cell. For the aggregate block, each metric is first averaged equally over the five mixtures within each seed; the displayed mean and accuracy standard deviation are then computed across the five seeds.

Five mixtures (five to eight tasks)
Quantity	GAR $-$ Baseline	GAR $-$ CAGrad
Accuracy (pp)	$+1.056$ [ $+0.831$ , $+1.282$ ]	$+0.786$ [ $+0.300$ , $+1.272$ ]
LVar	$-0.05344$ [ $-0.07021$ , $-0.03667$ ]	$-0.01156$ [ $-0.02261$ , $-0.00051$ ]
Gradient-mass purity	$+0.3197$ [ $+0.2526$ , $+0.3868$ ]	$+0.0821$ [ $+0.0109$ , $+0.1532$ ]
Utilization	$+0.2500$ [ $+0.1543$ , $+0.3457$ ]	$+0.1100$ [ $+0.0629$ , $+0.1571$ ]
Accuracy by task-count bin (pp)

Appendix

Table 41: Paired final-evaluation differences for the trainable RoBERTa classification-head LoRA MoE with top-1 routing (E8K1; 2,000 optimizer updates). For each seed, metrics are averaged equally over the five mixtures before subtracting the comparator for that seed. The lower block repeats the accuracy comparison within the five–six- and seven–eight-task groups. Accuracy is in percentage points; LVar, gradient-mass purity, and utilization are in their native units. Brackets are unadjusted two-sided 95% Student-

t

confidence intervals over the five paired seed differences (

n=5

, four degrees of freedom).

[QNLI, BoolQ, RTE, PAWS, WiC] (5 tasks)
Method	Acc	Seed std	LVar	Purity	Util.
Baseline	0.8464	0.0027	0.00365	0.4015	0.825
CAGrad	0.8418	0.0059	0.00393	0.4208	0.825
GAR	0.8528	0.0043	0.00284	0.4102	0.900
[QNLI, BoolQ, RTE, PAWS, ANLI, CB] (6 tasks)
Baseline	0.8311	0.0110	0.00325	0.3537	0.875

Appendix

Table 42: Trainable DeBERTa full-parameter FFN multi-task results for Baseline, CAGrad, and GAR across the five mixtures. Columns are equal-task macro validation accuracy, the population seed standard deviation, LVar, gradient-mass purity, and utilization, each defined as in Appendix E and read at the final evaluation. All five mixtures use E8K4, matching the grouping of Appendix F.1 . Five seeds per cell.

Five mixtures (five to eight tasks)
Quantity	GAR $-$ Baseline	GAR $-$ CAGrad
Accuracy (pp)	$+1.155$ [ $+0.730$ , $+1.579$ ]	$+0.873$ [ $+0.562$ , $+1.183$ ]
LVar	$-0.00073$ [ $-0.00145$ , $-0.00001$ ]	$-0.00130$ [ $-0.00205$ , $-0.00056$ ]
Gradient-mass purity	$+0.0120$ [ $-0.0135$ , $+0.0375$ ]	$+0.0165$ [ $+0.0037$ , $+0.0294$ ]
Utilization	$+0.0700$ [ $+0.0295$ , $+0.1105$ ]	$+0.0900$ [ $+0.0381$ , $+0.1419$ ]
Accuracy by task-count bin (pp)

Appendix

Table 43: Paired endpoint differences in the trainable DeBERTa full-parameter FFN setting. The upper block covers the five mixtures (five to eight tasks): within each seed we average the quantity over those five mixtures and then subtract the comparator for the same seed. The lower block repeats the accuracy difference within the five–six- and seven–eight-task groups. Accuracy is in percentage points; LVar, gradient-mass purity, and utilization are in their native units. Brackets are two-sided 95% Student-

t

confidence intervals over the five paired seed differences (

n=5

, four degrees of freedom). No multiplicity correction is applied.

Adapt.	Method	Acc $\uparrow$	Seed std $\downarrow$	LVar $\downarrow$	Util. $\uparrow$
LoRA	Baseline	0.8783	0.0081	0.0300	0.690
	CAGrad	0.8802	0.0110	0.0324	0.720
	GAR	0.8896	0.0065	0.0274	0.710
FFN	Baseline	0.8857	0.0090	0.0700	0.530
	CAGrad	0.8797	0.0103	0.0847	0.460
	GAR	0.8929	0.0075	0.0653	0.540

Appendix

Table 44: DeBERTa single-task E4K2 endpoint summary: frozen-backbone LoRA-FFN versus full-parameter FFN experts with an unfrozen backbone. Each adaptation averages [CoLA, MRPC, RTE, SST-2, QQP]; Mean weights the two adaptations equally. Accuracy excludes MCC/F1. Seed std averages per-dataset population standard deviations; LVar and utilization average endpoint diagnostics. Purity is 1.0 in the reported single-task rows and is omitted.

Comparison	Mean difference (pp)	95% paired CI (pp)	Paired $t$ $p$
GAR $-$ Baseline	$+0.929$	$[+0.207,\,+1.651]$	$0.0234$
GAR $-$ CAGrad	$+1.133$	$[+0.420,\,+1.847]$	$0.0116$

Appendix

Table 47: Equal-adaptation paired accuracy uncertainty for the controlled DeBERTa single-task checks. Within each shared seed, we first average validation accuracy over CoLA, MRPC, RTE, SST-2, and QQP separately for LoRA and FFN, then average the two adaptation results equally, and finally subtract the matched comparator from GAR. Brackets are two-sided 95% Student-

t

confidence intervals over the five paired seed differences (

n=5

, four degrees of freedom);

p

values are from the corresponding two-sided paired

t

tests. No multiplicity correction or significance symbols are used.

Mixture	$\lambda=0$	$10^{-5}$	$10^{-4}$	$10^{-3}$	$10^{-2}$
[QNLI, BoolQ, RTE, PAWS, WiC]	$60.63\pm 1.18$	$61.53\pm 0.74$	$61.84\pm 1.07$	$62.18\pm 0.62$	$62.08\pm 0.67$
[QNLI, BoolQ, RTE, PAWS, ANLI, CB]	$64.54\pm 0.58$	$64.87\pm 0.59$	$65.22\pm 0.73$	$65.38\pm 0.69$	$64.89\pm 0.69$
[QNLI, BoolQ, RTE, PAWS, WiC, CoLA, SST-2 ]	$66.92\pm 0.70$	$67.17\pm 0.50$	$67.53\pm 0.43$	$67.63\pm 0.71$	$66.92\pm 0.61$
[QNLI, BoolQ, RTE, WiC, CoLA, SST-2 , MRPC]	$68.86\pm 0.58$	$68.90\pm 0.55$	$69.34\pm 0.54$	$69.23\pm 0.43$	$68.84\pm 0.78$
[QNLI, BoolQ, RTE, PAWS, WiC, CoLA, SST-2 , CB]	$71.29\pm 0.50$	$71.56\pm 0.65$	$71.42\pm 0.45$	$71.62\pm 0.45$	$70.97\pm 0.88$
Five-mixture mean	$66.45\pm 0.37$	$66.81\pm 0.37$	$67.07\pm 0.26$	$67.21\pm 0.36$	$66.74\pm 0.17$

Appendix

Table 48: RoBERTa LoRA-FFN coefficient sensitivity on the five E8K4 mixtures at 1,000 updates. Entries are accuracy (%) as mean

\pm

population standard deviation over five seeds. Tasks are equally weighted within each mixture. The five-mixture row averages mixtures within each seed before computing the mean and standard deviation. Fixed configurations are in Table 19 .

Task / metric	$\lambda=0$	$10^{-5}$	$10^{-4}$	$10^{-3}$	$10^{-2}$
[QNLI, BoolQ, RTE, PAWS, WiC] (5 tasks)
QNLI	77.04	78.54	79.02	79.84	78.89
BoolQ	61.36	61.85	62.13	61.90	61.41
RTE	54.15	55.74	55.88	57.18	56.61
PAWS	56.36	56.83	57.00	57.26	59.14
WiC	54.26	54.67	55.14	54.73	54.36

Appendix

Table 49: Per-task results and routing diagnostics for RoBERTa LoRA-FFN coefficient sensitivity on the five E8K4 mixtures. Task and mixture macro accuracies are five-seed means in percent; seed std is the population standard deviation of mixture macro accuracy in percentage points. LVar, gradient-mass purity, and utilization are five-seed means at 1,000 updates on their original scales. All 33 mixture–task combinations and five coefficients are reported. Mixture definitions and fixed configurations are in Table 19 .

Mixture	$\lambda=0$	$10^{-5}$	$10^{-4}$	$10^{-3}$	$10^{-2}$
[QNLI, BoolQ, RTE, PAWS, WiC]	$76.86\pm 2.16$	$78.03\pm 1.35$	$77.87\pm 1.36$	$78.91\pm 0.92$	$77.64\pm 1.46$
[QNLI, BoolQ, RTE, PAWS, ANLI, CB]	$76.39\pm 1.13$	$76.31\pm 0.88$	$76.77\pm 0.63$	$77.14\pm 0.81$	$77.05\pm 1.03$
[QNLI, BoolQ, RTE, PAWS, WiC, CoLA, SST-2 ]	$80.73\pm 0.60$	$80.82\pm 0.55$	$81.20\pm 0.83$	$81.01\pm 0.90$	$81.11\pm 0.73$
[QNLI, BoolQ, RTE, WiC, CoLA, SST-2 , MRPC]	$80.11\pm 0.14$	$80.31\pm 0.40$	$80.50\pm 0.36$	$80.90\pm 0.10$	$80.66\pm 0.36$
[QNLI, BoolQ, RTE, PAWS, WiC, CoLA, SST-2 , CB]	$80.76\pm 0.77$	$80.38\pm 0.76$	$80.96\pm 0.61$	$81.37\pm 0.94$	$80.84\pm 0.92$
Five-mixture mean	$78.97\pm 0.78$	$79.17\pm 0.57$	$79.46\pm 0.52$	$79.87\pm 0.58$	$79.46\pm 0.70$

Appendix

Table 50: RoBERTa classification-head LoRA coefficient sensitivity on the five E8K4 mixtures at 1,000 updates. Entries are accuracy (%) as mean

\pm

Task / metric	$\lambda=0$	$10^{-5}$	$10^{-4}$	$10^{-3}$	$10^{-2}$
[QNLI, BoolQ, RTE, PAWS, WiC] (5 tasks)
QNLI	86.12	87.99	88.10	88.15	87.93
BoolQ	72.06	72.51	71.91	73.02	71.86
RTE	71.05	72.71	72.85	73.57	72.64
PAWS	91.05	91.34	91.42	91.75	90.53
WiC	64.04	65.61	65.05	68.06	65.27

Appendix

Table 51: Per-task results and routing diagnostics for RoBERTa classification-head LoRA coefficient sensitivity on the five E8K4 mixtures. Task and mixture macro accuracies are five-seed means in percent; seed std is the population standard deviation of mixture macro accuracy in percentage points. LVar, gradient-mass purity, and utilization are five-seed means at 1,000 updates on their original scales. All 33 mixture–task combinations and five coefficients are reported. Mixture definitions and fixed configurations are in Table 20 .

Mixture	Acc $\uparrow$	Seed std $\downarrow$	LVar $\downarrow$	Purity $\uparrow$	Util. $\uparrow$
[QNLI, BoolQ, RTE, PAWS, WiC]	78.98	1.07	0.00561	0.5020	0.750
[QNLI, BoolQ, RTE, PAWS, ANLI, CB]	76.95	0.95	0.00615	0.5586	0.775
[QNLI, BoolQ, RTE, PAWS, WiC, CoLA, SST-2 ]	80.46	0.71	0.00605	0.3964	0.700
[QNLI, BoolQ, RTE, WiC, CoLA, SST-2 , MRPC]	80.17	0.56	0.00692	0.3761	0.750
[QNLI, BoolQ, RTE, PAWS, WiC, CoLA, SST-2 , CB]	80.41	0.42	0.00602	0.6280	0.725

Appendix

Table 52: Numerator-only RoBERTa classification-head LoRA results on the five E8K4 mixtures at 1,000 updates. Each row inherits the corresponding fixed

\lambda=10^{-3}

configuration in Table 20 and removes only the load denominator. Accuracy is in percent and its seed standard deviation in percentage points; Purity is gradient-mass purity; routing diagnostics are five-run means on their original scales. The

\lambda=0

and load-normalized results appear in Table 51 .

Panel of Figure 6 — Figure 6: Full training-flow comparison between standard MoE routing and gradient-aligned routing (GAR). The auxiliary branch detaches gradient observations and routing inputs, then recomputes group probabilities using the group-mean construction in Algorithm 1 . Gradients shown at the optimizer node specify the direct pre-clipping gradient inputs; task parameters include all trainable non-router parameters. The main settings dispatch with top-4 routing over eight experts; the top-1 extensions use the straight-through rule of Appendix D.4 .

Sparse Mixture-of-Experts

Sparse mixture-of-experts models route each input to a small subset of specialized networks to increase capacity while limiting computation.

See all 121 papers

Grassmannian Mixture-of-Experts: Concentration-Controlled Routing on Subspace Manifolds
Feb 19, 2026Ibne Farabi Shihab, Sanjeda Akter, Anuj SharmaExpert Routing Mixture-of-Experts Language Models
Department of Computer Science, Iowa State University · Department of Civil, Construction and Environmental Engineering, Iowa State University
Routing-Free Mixture-of-Experts
Apr 1, 2026Yilun Liu, Jinru Han, Sikuan Yan +2Mixture-of-Experts Models Expert Load Balancing
Ludwig Maximilian University of Munich · Munich Center for Machine Learning · University of California, Los Angeles
Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
Sep 29, 2026Yury Nahshan, Nati Daniel, Jacob Goldberger +1Mixture-of-Experts Language Models Parameter-Free Mixture-of-Experts Routing
Bar-Ilan University, Ramat-Gan, Israel · NVIDIA, Israel

Feb 19, 2026J/K move · Enter open · S save

Grassmannian Mixture-of-Experts: Concentration-Controlled Routing on Subspace Manifolds

Ibne Farabi Shihab, Sanjeda Akter, Anuj Sharma

Department of Computer Science, Iowa State University · Department of Civil, Construction and Environmental Engineering, Iowa State University

Expert Routing Mixture-of-Experts Language Models Sparse Mixture-of-Experts

Open paper AskPDF arXiv

Mixture-of-Experts models rely on learned routers to assign tokens to experts, yet standard softmax gating provides no principled mechanism to control the tradeoff between sparsity and utilization. We propose Grassmannian MoE (GrMoE), a routing framework that operates on the Grassmannian manifold of subspaces, where gating weights arise from the concentration parameters of Matrix Bingham distributions. This construction yields a single, interpretable knob -- the concentration matrix

Λ

-- that continuously controls routing entropy, replacing discrete top-

k

selection with a smooth, geometrically principled sparsity mechanism. We further develop an amortized variational inference procedure for posterior routing distributions, enabling uncertainty-aware expert assignment that naturally resists expert collapse. We formally prove tight bounds relating the Bingham concentration spectrum to routing entropy, expected top-

k

mass, and an exponential bound on expert collapse, establishing the first formal theory of concentration-controlled sparsity. On synthetic routing tasks, a 350M-parameter MoE language model with 8 experts, a 1.3B-parameter model with 16 experts, and a 2.7B-parameter model with 32 experts, GrMoE achieves 0% routing collapse across all seeds, comparable or better perplexity with 15--30% improved load balance, and a smooth monotonic relationship between concentration and effective sparsity that enables post-hoc sparsity tuning without retraining. Token-level analysis reveals that experts learn heterogeneous concentration values that correlate with linguistic specialization, providing interpretable routing behavior.

Abstract

Sparse Mixture-of-Experts

Explore similar work

Grassmannian Mixture-of-Experts: Concentration-Controlled Routing on Subspace Manifolds

Routing-Free Mixture-of-Experts

Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models