Sparse Mixture-of-Experts (MoE) is a powerful paradigm for scaling Neural Machine Translation (NMT) and managing language heterogeneity. However, practical limitations hinder its deployment on resource-constrained devices: the entire expert pool must remain in memory, and the per-token compute budget is statically fixed during training by rigid top-k routing. We introduce EDWARD-NMT (Elastic Dimensions With Adaptive Routing and Dual-consistency for NMT), a novel MoE encoder-decoder architecture designed for a train once, deploy anywhere paradigm that exposes a family of deployable sub-networks at inference time. EDWARD-NMT integrates three key innovations: (i) Matryoshka-style elastic dimensions to flexibly reduce the memory footprint; (ii) an adaptive sigmoid router with a learned, task-conditioned threshold, allowing each token to dynamically determine its active expert count; and (iii) a dual-consistency objective to maximize the performance of the extracted sub-networks. Extensive experiments on PhoMT (En-Vi), WMT-14 (En-De), multilingual OPUS-16, and a multi-domain De-En benchmark validate our approach. The full-capacity architecture matches or exceeds strong MoE baselines with improved parameter efficiency. Furthermore, the sub-networks seamlessly extracted from a single checkpoint consistently outperform comparably sized dense Transformers without any retraining.
Figures & tables
Figure 1: Overview of the EDWARD-NMT architecture. Left: The macro-level Transformer block integrating the Dynamic MoE layer. Center: The Adaptive Sigmoid Router computes a token-specific threshold ( τ ) conditioned on a translation direction embedding ( edir ) to dynamically allocate the sparse expert budget. Top Right: The internal structure of a Matryoshka Expert, illustrating the nested parameter slices used to extract varying capacities. Bottom Right: The Direction-Aware Shared Expert (DASE) applies FiLM modulation ( γdir,βdir ) to provide a stable, direction-aware backbone.
Variant
Dk
Total Params
Active Params
EDWARD-NMT-S
256
67−74 M
51−59 M
EDWARD-NMT-M
512
98−105 M
66−76 M
EDWARD-NMT-L
1024
161−168 M
97−109 M
EDWARD-NMT-XL
2048
287−294 M
158−175 M
Table 1: EDWARD-NMT sub-network variants extracted from a single trained model.
PhoMT
WMT-14
Method
Params
Active
En → Vi
Vi → En
Params
Active
En → De
De → En
Transformer Base
65M
65M
42.12
37.19
61M
61M
24.56
28.76
Transformer Big
213M
213M
42.94
37.83
210M
210M
27.53
31.48
mBART †
448M
448M
43.46
39.78
610M
610M
24.90
32.51
EDWARD-NMT-S
67M
51M
42.91
37.88
74M
59M
26.85
31.94
EDWARD-NMT-M
98M
66M
43.35
38.50
105M
76M
27.49
32.45
Table 2: Main results on PhoMT (En ↔ Vi) and WMT-14 (En ↔ De) measured in sacreBLEU. “Active” denotes average parameters activated by our sigmoid router. Best and second-best scores are bolded and underlined , respectively. † For mBART on WMT-14, we use a pretrained many-to-many model ( mbart-large-50-many-to-many-mmt ).
Multi-domain
OPUS-16
Method
Params
Active
Kor
IT
Med
Law
Sub
Avg
En → XX
XX → En
Transformer Base
61M
61M
20.53
40.01
51.25
53.72
25.29
38.16
26.92 †
30.67 †
Transformer Big
210M
210M
22.55
41.84
53.35
56.22
26.73
40.13
-
-
ST-MoE
360M
210M
21.73
41.00
52.01
55.69
26.18
39.32
29.68 †
33.89 †
GS-MoE
360M
260M
20.70
41.35
52.31
54.56
24.72
38.72
28.28 †
32.31 †
EDWARD-NMT-S
74M
57M
21.61
40.07
51.01
53.99
26.56
38.65
29.55
32.99
Table 3: Experimental results on the multi-domain (De ↔ En) averaged across both directions and multilingual OPUS-16 benchmarks (English to any En → XX and any to English XX → En). The “Params” and “Active” columns report footprints for the multi-domain setup. † Reported from Zhao et al. (2024) . Extended OPUS-16 comparisons are provided in Appendix B.2 .
WMT-14
Multi-domain
Method
Active
Avg
Active
Avg
Transformer Base
61M
80.78
61M
80.20
Transformer Big
210M
83.69
210M
81.62
mBART †
610M
83.56
-
-
ST-MoE
-
-
210M
80.95
GS-MoE
-
-
260M
78.20
Table 4: Semantic-level translation quality measured in COMET-22. Scores are averaged across all translation directions (and domains for the Multi-domain benchmark). Best and second-best scores are bolded and underlined . † mBART is a pretrained many-to-many model.
Method
PhoMT
WMT-14
MD
OPUS-16
EDWARD-NMT-S
1.44
1.64
1.37
1.34
EDWARD-NMT-M
1.45
1.66
1.37
1.35
EDWARD-NMT-L
1.47
1.69
1.38
1.37
EDWARD-NMT-XL
1.51
1.70
1.39
1.39
Table 5: Average active routed experts per token across the PhoMT, WMT-14, Multi-domain (MD) and OPUS-16 benchmarks (out of E=4 routed experts, excluding the fixed shared expert).
En ↔ De
Setting
XL
L
M
S
MoE ( k=2 )
40.35
39.64
39.21
38.24
+ ME
40.96
40.26
39.35
38.34
+ ME + Sigmoid
40.72
40.23
39.20
38.16
+ ME + DC
40.66
40.16
39.36
38.49
+ ME + DC + Sigmoid
40.75
40.28
39.56
38.65
Table 6: Ablation results on the Multi-domain benchmark, averaged over both translation directions. All settings use shared expert (DASE) .
En ↔ De
Setting
XL
L
M
S
No SE
39.75
39.40
38.84
38.13
Common SE
39.95
39.73
39.12
38.46
DASE (ours)
40.74
40.26
39.39
38.72
Table 7: Ablation on the shared expert (SE) routing strategies on the Multi-domain benchmark, averaged over both translation directions.
En ↔ De
Setting
XL
L
M
S
No loss
40.72
40.23
39.20
38.16
R-Drop
40.00
39.73
39.19
38.47
Sym. JSD
40.80
40.17
39.19
38.19
Asym. KL
40.76
40.14
39.50
38.59
Asym. KL (Max)
41.04
39.89
39.14
38.39
Table 8: Ablation on the Dual-Consistency loss (DC Loss) on the Multi-domain benchmark, averaged over both translation directions.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Lang
Train
Dev
Test
Vocab
PhoMT
En ↔ Vi
2,977k
19,151
19,151
20k SPM
WMT-14
En ↔ De
3,906k
3,000
3,003
32k BPE
MD
De ↔ En
1,284k
10,000
10,000
32k BPE
OPUS-16
16 lang
17,559k
30,000
30,000
32k SPM
Appendix
Table 9: Dataset sizes (sentence pairs) and shared vocabularies. SPM : SentencePiece. The “Train” column reports the final number of filtered sentence pairs used for model training. The multi-domain (MD) training set is the concatenation of five diverse domains (Koran, IT, Medical, Law, Subtitles) following Zhang et al. (2024) . The OPUS-16 dataset includes 16 languages following Zhao et al. (2024) .
Parameter
Value
Search Space
Optimiser
AdamW (β1=0.9,β2=0.98,ϵ=10−8)
–
Weight decay
0.01
–
Peak learning rate
10−2
{5×10−4,10−3,5×10−3,10−2}
Initial learning rate
10−6
–
Final learning rate
10−4
–
Warmup steps
1,000
{0,1000,2000,4000}
Appendix
Table 10: Full hyperparameter set and search space. Values marked with “–” were fixed based on hardware constraints or standard architectural practices. The same router and elasticity settings are used across all benchmarks.
Edge
Server Throughput (TPS)
Model
Params
Active
FLOPs (G)
TPS ( B=1 )
B=1
B=4
B=16
B=64
B=128
Transformer Base
61M
61M
7.98
117.7
2,354
5,867
13,078
19,694
21,755
Transformer Big
210M
210M
31.26
48.0
1,682
3,907
8,037
10,607
11,114
EDWARD-NMT-S
73M
57M
6.28
104.9
1,763
6,640
20,418
48,022
70,372
EDWARD-NMT-M
105M
72M
8.83
88.7
1,688
5,900
20,309
47,669
68,786
EDWARD-NMT-L
168M
102M
14.07
61.5
1,488
5,650
19,364
44,270
64,766
Appendix
Table 11: Inference efficiency across edge and server deployment regimes (64 output tokens). FLOPs (lower is better) are measured via native CPU execution. Edge throughput (Tokens Per Second, TPS, higher is better) is profiled using compiled LiteRT checkpoints on an Apple M4 CPU (4 threads, FP16) to simulate strict on-device constraints ( B=1 ). Server throughput is measured via JAX on a single NVIDIA H100 GPU across multiple batch sizes ( B ).
Models
En → XX
XX → En
Avg.
high
medium
low
Avg1.
high
medium
low
Avg2.
Dense
Transformer-base †
25.37
39.12
14.78
26.92
28.81
39.24
24.21
30.67
28.79
MoE
GS-MoE †
25.55
40.71
18.97
28.28
28.77
41.70
29.25
32.31
30.29
ST-MoE †
26.55
43.04
20.23
29.68
29.94
44.00
30.94
33.89
31.79
[1pt/2pt]
LS-MoE †
20.06
37.31
11.72
22.99
23.69
42.99
32.01
30.50
26.75
Hybrid-MoE †
24.56
31.70
13.38
24.23
29.59
41.74
27.81
32.48
28.35
Appendix
Table 12: Averaged BLEU scores on multilingual OPUS-16 dataset. The best and second best scores in each column are highlighted in bold and underlined. † Reported from Zhao et al. (2024) . ‡ Reported from Liang et al. (2025) .
L=8
L=6
Variant
Params
Active
Params
Active
S
74M
57M
59M
47M
M
105M
72M
83M
59M
L
169M
102M
131M
83M
XL
295M
161M
226M
128M
Appendix
Table 13: EDWARD-NMT variants with two settings of 8 layers ( L=8 ) and 6 layers ( L=6 ) on OPUS-16.
Dim
P.
Act.
Sp.
En → Tgt
Tgt → En
PhoMT (En ↔ Vi)
2048
287M
158M
1.44
43.79
39.13
1536
224M
128M
1.44
43.62
38.98
1024
161M
97M
1.45
43.61
38.93
768
130M
82M
1.46
43.48
38.58
512
98M
67M
1.47
43.35
38.50
Appendix
Table 14: Performance across dimensions on the PhoMT, WMT-14, and Multi-domain benchmarks. Column abbreviations denote total parameters ( P. ), average active parameters per token ( Act. ), and routing sparsity ( Sp. ) (active routed experts per token out of 4, excluding the shared expert). Multi-Domain scores are macro-averaged across five domains. The best and second best scores within each dataset are highlighted.
En ↔ De
Setting
XL
L
M
S
w/o ψ(Dk)
40.52
40.03
39.20
38.35
With ψ
40.74
40.26
39.39
38.72
Appendix
Table 15: Ablation of the Variance Preservation factor ψ(Dk) on the Multi-domain benchmark, averaged over both translation directions.
De → En
En → De
Method
Params
Active
Kor
IT
Med
Law
Sub
Avg
Kor
IT
Med
Law
Sub
Avg
Transformer Base
61M
61M
20.95
42.66
53.04
56.90
27.16
40.14
20.10
37.35
49.45
50.53
23.42
36.17
Transformer Big
210M
210M
22.06
44.99
55.37
59.36
28.75
42.11
23.03
38.68
51.32
53.08
24.70
38.16
ST-MoE
360M
210M
22.96
43.90
54.27
58.79
28.30
41.64
20.49
38.10
49.75
52.59
24.06
37.00
GS-MoE
360M
260M
20.36
44.79
54.23
57.51
25.99
40.58
21.03
37.91
50.38
51.60
23.44
36.87
EDWARD-NMT-S
74M
57M
22.27
42.67
53.13
57.08
28.76
40.78
20.94
37.47
48.89
50.89
24.36
36.51
Appendix
Table 16: Detailed multi-domain results on the De ↔ En benchmark with tokenized-BLEU.
WMT-14 chrF2 ( ↑ )
Method
Active
En → De
De → En
Transformer Base
61M
55.82
56.23
Transformer Big
210M
58.20
58.38
mBART †
610M
56.42
58.85
EDWARD-NMT-S
59M
57.52
58.43
EDWARD-NMT-M
76M
58.08
58.83
Appendix
Table 17: Character n -gram F-score (chrF2, higher is better) on the WMT-14 benchmark. Best and second-best scores are bolded and underlined , respectively. † mBART is a pretrained many-to-many model.
TER ( ↓ )
Method
Active
En → Vi
Vi → En
Transformer Base
65M
43.79
44.28
Transformer Big
213M
43.04
44.06
mBART †
448M
42.57
42.54
EDWARD-NMT-S
51M
42.75
43.27
EDWARD-NMT-M
66M
42.43
42.88
Appendix
Table 18: Translation Edit Rate (TER, lower is better) on the PhoMT benchmark. † mBART is a pretrained many-to-many model.
En ↔ Vi
Setting
XL
L
M
S
No loss
41.41
41.19
40.82
40.13
R-Drop
41.32
41.01
40.50
39.62
Asym. KL
41.31
41.13
40.65
40.08
DC Loss (ours)
41.46
41.27
40.93
40.40
Appendix
Table 19: Ablation on the Dual-Consistency loss (DC Loss) on PhoMT, averaged over both directions.
En ↔ Vi
Setting
XL
L
M
S
MoE ( k=2 )
40.72
40.71
40.56
39.86
+ ME
41.31
41.15
40.69
40.04
+ ME + Sigmoid
41.41
41.19
40.82
40.13
+ ME + DC
41.38
41.23
40.90
40.42
+ ME + DC + Sigmoid
41.46
41.27
40.93
40.40
Appendix
Table 20: Ablation on the contributions of each EDWARD-NMT component (PhoMT), averaged over both translation directions. ME, DC, Sigmoid stand for Matryoshka Experts, Dual-Consistency and Adaptive Sigmoid Routing respectively. All settings use DASE .
Target Sparsity ( ρtarget )
Direction
1.0
1.25
1.5
Avg.
40.40
40.66
40.73
Appendix
Table 21: Ablation on target density ( ρtarget ) for the XL variant on the multi-domain benchmark.
Min Threshold ( τmin )
Direction
0.01
0.05
0.10
Avg.
40.63
40.66
40.34
Appendix
Table 22: Ablation on the min routing threshold ( τmin ) for the XL variant on the multi-domain benchmark.
Router Jitter ( η )
Direction
0.01
0.05
0.10
Avg.
38.98
40.66
40.62
Appendix
Table 23: Ablation on the router jitter noise ( η ) for the XL variant on the multi-domain benchmark.
Figure 2: Average active experts per token during training.
Figure 3: Learnable threshold τ over the course of training.
Figure 4: Percentage of number of active experts per layer across different elastic dimensions on the multi-domain benchmark.
En ↔ De
Setting
XL
L
M
S
λdc=0.1
40.98
40.26
39.44
38.34
λdc=1.0
40.75
40.28
39.56
38.65
λdc=5.0
38.33
38.23
37.92
37.44
Appendix
Table 24: Ablation on the Dual-Consistency loss weight ( λdc ) on the multi-domain benchmark, averaged over both translation directions.
Method
Target Density ( ρ )
ρ=1.0
ρ=1.25
ρ=1.5
EDWARD-NMT-S
1.18
1.39
1.69
EDWARD-NMT-M
1.17
1.38
1.67
EDWARD-NMT-L
1.16
1.37
1.66
EDWARD-NMT-XL
1.15
1.37
1.65
Appendix
Table 25: Average active routed experts per token on the Multi-domain benchmark across different target density ( ρ ) settings. Values are averaged across both translation directions and all domains.
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.
Mixture-of-Experts (MoE) architectures enable efficient model scaling, yet expert routing behavior across underrepresented languages remains poorly understood. We analyze routing dynamics in two architecturally distinct MoE models -- a pure Transformer (Qwen3-30B-A3B) and a hybrid Mamba-Transformer (Nemotron-3-Nano-30B-A3B) -- using Hebrew as a morphologically rich, low-resource testbed. Both pre-trained models exhibit deep-layer routing collapse: usage entropy drops sharply in final layers and tokens concentrate on a narrow expert subset, a pattern largely absent for English. Continual pre-training (CPT) on balanced bilingual data substantially corrects this imbalance, increasing entropy and shifting routing toward shared, language-agnostic experts; supervised fine-tuning (SFT) alone achieves less complete correction. Extending the analysis to Japanese reveals quantitatively consistent collapse signatures, providing cross-linguistic evidence that the phenomenon is a systematic consequence of pre-training underrepresentation rather than any language-intrinsic property. Routing improvements correlate with consistent downstream benchmark gains, positioning routing entropy and expert specialization as principled diagnostics for multilingual capacity in MoE systems.
Mixture-of-Experts (MoE) models activate only a sparse subset of experts per token, yet consecutive tokens frequently activate different experts -- causing constant weight swapping between slow storage and fast memory on edge devices. Existing remedies are either system-level (caching heuristics) or post-hoc (router fine-tuning), leaving the root cause unchanged during pretraining. We propose StickyMoE, a differentiable routing consistency loss that penalises abrupt expert switches between adjacent tokens, encouraging the router to maintain the same expert assignment across semantically coherent spans. StickyMoE requires no architectural changes, adds a single hyperparameter lambda, and unlike post-hoc methods, allows expert representations and routing decisions to co-adapt from the first training step. Experiments on small-scale MoE language models show that StickyMoE reduces the expert switch rate by up to 60% with less than 4% perplexity degradation, Pareto-dominating post-hoc fine-tuning on the quality-locality frontier. Routing temporal locality is most efficiently instilled at training time.
We present Marco-MoE, a suite of fully open multilingual sparse Mixture-of-Experts (MoE) models. Marco-MoE features a highly sparse design in which only around 5% of the total parameters are activated per input token. This extreme sparsity, combined with upcycling from dense models, enables efficient pre-training on 5T tokens. Our models surpass similarly-sized competitors on English and multilingual benchmarks, achieving a best-in-class performance-to-compute ratio. We further post-train these models to create Marco-MoE-Instruct variants, which surpass the performance of competing models possessing 3--14× more activated parameters. Our analysis reveals that Marco-MoE learns structured expert activation patterns shared across related languages, while maintaining highly specialized utilization for linguistically isolated ones. We further show that Marco-MoE allows for scalable language expansion without the interference typical of dense models. To support the community, we disclose our full training datasets, recipes, and model weights.