EDWARD-NMT: Elastic Dimensions With Adaptive Routing and Dual-consistency for Neural Machine Translation

Edward Nguyen; Thai Tran; Kiet Anh Hoang; Duy Hoang; Duy Vo; Khanh Le; Vu Le

cs.CLOct 23, 2026

EDWARD-NMT: Elastic Dimensions With Adaptive Routing and Dual-consistency for Neural Machine Translation

Authors: Edward Nguyen, Thai Tran, Kiet Anh Hoang, Duy Hoang, Duy Vo, Khanh Le, Vu Le

Organizations: UNEY, Switzerland

Abstract

Sparse Mixture-of-Experts (MoE) is a powerful paradigm for scaling Neural Machine Translation (NMT) and managing language heterogeneity. However, practical limitations hinder its deployment on resource-constrained devices: the entire expert pool must remain in memory, and the per-token compute budget is statically fixed during training by rigid top-k routing. We introduce EDWARD-NMT (Elastic Dimensions With Adaptive Routing and Dual-consistency for NMT), a novel MoE encoder-decoder architecture designed for a train once, deploy anywhere paradigm that exposes a family of deployable sub-networks at inference time. EDWARD-NMT integrates three key innovations: (i) Matryoshka-style elastic dimensions to flexibly reduce the memory footprint; (ii) an adaptive sigmoid router with a learned, task-conditioned threshold, allowing each token to dynamically determine its active expert count; and (iii) a dual-consistency objective to maximize the performance of the extracted sub-networks. Extensive experiments on PhoMT (En-Vi), WMT-14 (En-De), multilingual OPUS-16, and a multi-domain De-En benchmark validate our approach. The full-capacity architecture matches or exceeds strong MoE baselines with improved parameter efficiency. Furthermore, the sub-networks seamlessly extracted from a single checkpoint consistently outperform comparably sized dense Transformers without any retraining.

Figures & tables

Figure 1: Overview of the EDWARD-NMT architecture. Left: The macro-level Transformer block integrating the Dynamic MoE layer. Center: The Adaptive Sigmoid Router computes a token-specific threshold ( $\tau$ ) conditioned on a translation direction embedding ( $e_{dir}$ ) to dynamically allocate the sparse expert budget. Top Right: The internal structure of a Matryoshka Expert, illustrating the nested parameter slices used to extract varying capacities. Bottom Right: The Direction-Aware Shared Expert (DASE) applies FiLM modulation ( $\gamma_{dir},\beta_{dir}$ ) to provide a stable, direction-aware backbone.

Variant	$\mathbf{D_{k}}$	Total Params	Active Params
EDWARD-NMT-S	$256$	$67-74$ M	$51-59$ M
EDWARD-NMT-M	$512$	$98-105$ M	$66-76$ M
EDWARD-NMT-L	$1024$	$161-168$ M	$97-109$ M
EDWARD-NMT-XL	$2048$	$287-294$ M	$158-175$ M

Table 1: EDWARD-NMT sub-network variants extracted from a single trained model.

	PhoMT				WMT-14
Method	Params	Active	En $\rightarrow$ Vi	Vi $\rightarrow$ En	Params	Active	En $\rightarrow$ De	De $\rightarrow$ En
Transformer Base	65M	65M	42.12	37.19	61M	61M	24.56	28.76
Transformer Big	213M	213M	42.94	37.83	210M	210M	27.53	31.48
mBART †	448M	448M	43.46	39.78	610M	610M	24.90	32.51
EDWARD-NMT-S	67M	51M	42.91	37.88	74M	59M	26.85	31.94
EDWARD-NMT-M	98M	66M	43.35	38.50	105M	76M	27.49	32.45

Table 2: Main results on PhoMT (En

\leftrightarrow

Vi) and WMT-14 (En

\leftrightarrow

De) measured in sacreBLEU. “Active” denotes average parameters activated by our sigmoid router. Best and second-best scores are bolded and underlined , respectively. † For mBART on WMT-14, we use a pretrained many-to-many model ( mbart-large-50-many-to-many-mmt ).

			Multi-domain						OPUS-16
Method	Params	Active	Kor	IT	Med	Law	Sub	Avg	En $\rightarrow$ XX	XX $\rightarrow$ En
Transformer Base	61M	61M	20.53	40.01	51.25	53.72	25.29	38.16	26.92 †	30.67 †
Transformer Big	210M	210M	22.55	41.84	53.35	56.22	26.73	40.13	-	-
ST-MoE	360M	210M	21.73	41.00	52.01	55.69	26.18	39.32	29.68 †	33.89 †
GS-MoE	360M	260M	20.70	41.35	52.31	54.56	24.72	38.72	28.28 †	32.31 †
EDWARD-NMT-S	74M	57M	21.61	40.07	51.01	53.99	26.56	38.65	29.55	32.99

Table 3: Experimental results on the multi-domain (De

\leftrightarrow

En) averaged across both directions and multilingual OPUS-16 benchmarks (English to any En

\rightarrow

XX and any to English XX

\rightarrow

En). The “Params” and “Active” columns report footprints for the multi-domain setup. † Reported from Zhao et al. (2024) . Extended OPUS-16 comparisons are provided in Appendix B.2 .

	WMT-14		Multi-domain
Method	Active	Avg	Active	Avg
Transformer Base	61M	80.78	61M	80.20
Transformer Big	210M	83.69	210M	81.62
mBART †	610M	83.56	-	-
ST-MoE	-	-	210M	80.95
GS-MoE	-	-	260M	78.20

Table 4: Semantic-level translation quality measured in COMET-22. Scores are averaged across all translation directions (and domains for the Multi-domain benchmark). Best and second-best scores are bolded and underlined . † mBART is a pretrained many-to-many model.

Method	PhoMT	WMT-14	MD	OPUS-16
EDWARD-NMT-S	1.44	1.64	1.37	1.34
EDWARD-NMT-M	1.45	1.66	1.37	1.35
EDWARD-NMT-L	1.47	1.69	1.38	1.37
EDWARD-NMT-XL	1.51	1.70	1.39	1.39

Table 5: Average active routed experts per token across the PhoMT, WMT-14, Multi-domain (MD) and OPUS-16 benchmarks (out of

E=4

routed experts, excluding the fixed shared expert).

	En $\leftrightarrow$ De
Setting	XL	L	M	S
MoE ( $k{=}2$ )	40.35	39.64	39.21	38.24
+ ME	40.96	40.26	39.35	38.34
+ ME + Sigmoid	40.72	40.23	39.20	38.16
+ ME + DC	40.66	40.16	39.36	38.49
+ ME + DC + Sigmoid	40.75	40.28	39.56	38.65

Table 6: Ablation results on the Multi-domain benchmark, averaged over both translation directions. All settings use shared expert (DASE) .

	En $\leftrightarrow$ De
Setting	XL	L	M	S
No SE	39.75	39.40	38.84	38.13
Common SE	39.95	39.73	39.12	38.46
DASE (ours)	40.74	40.26	39.39	38.72

Table 7: Ablation on the shared expert (SE) routing strategies on the Multi-domain benchmark, averaged over both translation directions.

	En $\leftrightarrow$ De
Setting	XL	L	M	S
No loss	40.72	40.23	39.20	38.16
R-Drop	40.00	39.73	39.19	38.47
Sym. JSD	40.80	40.17	39.19	38.19
Asym. KL	40.76	40.14	39.50	38.59
Asym. KL (Max)	41.04	39.89	39.14	38.39

Table 8: Ablation on the Dual-Consistency loss (DC Loss) on the Multi-domain benchmark, averaged over both translation directions.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Lang	Train	Dev	Test	Vocab
PhoMT	En $\leftrightarrow$ Vi	2,977k	19,151	19,151	20k SPM
WMT-14	En $\leftrightarrow$ De	3,906k	3,000	3,003	32k BPE
MD	De $\leftrightarrow$ En	1,284k	10,000	10,000	32k BPE
OPUS-16	16 lang	17,559k	30,000	30,000	32k SPM

Appendix

Table 9: Dataset sizes (sentence pairs) and shared vocabularies. SPM : SentencePiece. The “Train” column reports the final number of filtered sentence pairs used for model training. The multi-domain (MD) training set is the concatenation of five diverse domains (Koran, IT, Medical, Law, Subtitles) following Zhang et al. (2024) . The OPUS-16 dataset includes 16 languages following Zhao et al. (2024) .

Parameter	Value	Search Space
Optimiser	AdamW $(\beta_{1}{=}0.9,\beta_{2}{=}0.98,\epsilon{=}10^{-8})$	–
Weight decay	$0.01$	–
Peak learning rate	$10^{-2}$	$\{5{\times}10^{-4},10^{-3},5{\times}10^{-3},10^{-2}\}$
Initial learning rate	$10^{-6}$	–
Final learning rate	$10^{-4}$	–
Warmup steps	$1{,}000$	$\{0,1000,2000,4000\}$

Appendix

Table 10: Full hyperparameter set and search space. Values marked with “–” were fixed based on hardware constraints or standard architectural practices. The same router and elasticity settings are used across all benchmarks.

				Edge	Server Throughput (TPS)
Model	Params	Active	FLOPs (G)	TPS ( $B=1$ )	$B=1$	$B=4$	$B=16$	$B=64$	$B=128$
Transformer Base	61M	61M	7.98	117.7	2,354	5,867	13,078	19,694	21,755
Transformer Big	210M	210M	31.26	48.0	1,682	3,907	8,037	10,607	11,114
EDWARD-NMT-S	73M	57M	6.28	104.9	1,763	6,640	20,418	48,022	70,372
EDWARD-NMT-M	105M	72M	8.83	88.7	1,688	5,900	20,309	47,669	68,786
EDWARD-NMT-L	168M	102M	14.07	61.5	1,488	5,650	19,364	44,270	64,766

Appendix

Table 11: Inference efficiency across edge and server deployment regimes (64 output tokens). FLOPs (lower is better) are measured via native CPU execution. Edge throughput (Tokens Per Second, TPS, higher is better) is profiled using compiled LiteRT checkpoints on an Apple M4 CPU (4 threads, FP16) to simulate strict on-device constraints (

B=1

). Server throughput is measured via JAX on a single NVIDIA H100 GPU across multiple batch sizes (

B

Models		En $\rightarrow$ XX				XX $\rightarrow$ En				Avg.
Models		high	medium	low	Avg1.	high	medium	low	Avg2.	Avg.
Dense	Transformer-base †	25.37	39.12	14.78	26.92	28.81	39.24	24.21	30.67	28.79
MoE	GS-MoE †	25.55	40.71	18.97	28.28	28.77	41.70	29.25	32.31	30.29
MoE	ST-MoE †	26.55	43.04	20.23	29.68	29.94	44.00	30.94	33.89	31.79
[1pt/2pt]	LS-MoE †	20.06	37.31	11.72	22.99	23.69	42.99	32.01	30.50	26.75
	Hybrid-MoE †	24.56	31.70	13.38	24.23	29.59	41.74	27.81	32.48	28.35

Appendix

Table 12: Averaged BLEU scores on multilingual OPUS-16 dataset. The best and second best scores in each column are highlighted in bold and underlined. † Reported from Zhao et al. (2024) . ‡ Reported from Liang et al. (2025) .

	$L=8$		$L=6$
Variant	Params	Active	Params	Active
S	74M	57M	59M	47M
M	105M	72M	83M	59M
L	169M	102M	131M	83M
XL	295M	161M	226M	128M

Appendix

Table 13: EDWARD-NMT variants with two settings of 8 layers (

L=8

) and 6 layers (

L=6

) on OPUS-16.

Dim	P.	Act.	Sp.	En $\rightarrow$ Tgt	Tgt $\rightarrow$ En
PhoMT (En $\leftrightarrow$ Vi)
2048	287M	158M	1.44	43.79	39.13
1536	224M	128M	1.44	43.62	38.98
1024	161M	97M	1.45	43.61	38.93
768	130M	82M	1.46	43.48	38.58
512	98M	67M	1.47	43.35	38.50

Appendix

Table 14: Performance across dimensions on the PhoMT, WMT-14, and Multi-domain benchmarks. Column abbreviations denote total parameters ( P. ), average active parameters per token ( Act. ), and routing sparsity ( Sp. ) (active routed experts per token out of 4, excluding the shared expert). Multi-Domain scores are macro-averaged across five domains. The best and second best scores within each dataset are highlighted.

	En $\leftrightarrow$ De
Setting	XL	L	M	S
w/o $\psi(D_{k})$	40.52	40.03	39.20	38.35
With $\psi$	40.74	40.26	39.39	38.72

Appendix

Table 15: Ablation of the Variance Preservation factor

\psi(D_{k})

on the Multi-domain benchmark, averaged over both translation directions.

			De $\rightarrow$ En						En $\rightarrow$ De
Method	Params	Active	Kor	IT	Med	Law	Sub	Avg	Kor	IT	Med	Law	Sub	Avg
Transformer Base	61M	61M	20.95	42.66	53.04	56.90	27.16	40.14	20.10	37.35	49.45	50.53	23.42	36.17
Transformer Big	210M	210M	22.06	44.99	55.37	59.36	28.75	42.11	23.03	38.68	51.32	53.08	24.70	38.16
ST-MoE	360M	210M	22.96	43.90	54.27	58.79	28.30	41.64	20.49	38.10	49.75	52.59	24.06	37.00
GS-MoE	360M	260M	20.36	44.79	54.23	57.51	25.99	40.58	21.03	37.91	50.38	51.60	23.44	36.87
EDWARD-NMT-S	74M	57M	22.27	42.67	53.13	57.08	28.76	40.78	20.94	37.47	48.89	50.89	24.36	36.51

Appendix

Table 16: Detailed multi-domain results on the De

\leftrightarrow

En benchmark with tokenized-BLEU.

		WMT-14 chrF2 ( $\uparrow$ )
Method	Active	En $\rightarrow$ De	De $\rightarrow$ En
Transformer Base	61M	55.82	56.23
Transformer Big	210M	58.20	58.38
mBART †	610M	56.42	58.85
EDWARD-NMT-S	59M	57.52	58.43
EDWARD-NMT-M	76M	58.08	58.83

Appendix

Table 17: Character

n

-gram F-score (chrF2, higher is better) on the WMT-14 benchmark. Best and second-best scores are bolded and underlined , respectively. † mBART is a pretrained many-to-many model.

		TER ( $\downarrow$ )
Method	Active	En $\rightarrow$ Vi	Vi $\rightarrow$ En
Transformer Base	65M	43.79	44.28
Transformer Big	213M	43.04	44.06
mBART †	448M	42.57	42.54
EDWARD-NMT-S	51M	42.75	43.27
EDWARD-NMT-M	66M	42.43	42.88

Appendix

Table 18: Translation Edit Rate (TER, lower is better) on the PhoMT benchmark. † mBART is a pretrained many-to-many model.

	En $\leftrightarrow$ Vi
Setting	XL	L	M	S
No loss	41.41	41.19	40.82	40.13
R-Drop	41.32	41.01	40.50	39.62
Asym. KL	41.31	41.13	40.65	40.08
DC Loss (ours)	41.46	41.27	40.93	40.40

Appendix

Table 19: Ablation on the Dual-Consistency loss (DC Loss) on PhoMT, averaged over both directions.

	En $\leftrightarrow$ Vi
Setting	XL	L	M	S
MoE ( $k{=}2$ )	40.72	40.71	40.56	39.86
+ ME	41.31	41.15	40.69	40.04
+ ME + Sigmoid	41.41	41.19	40.82	40.13
+ ME + DC	41.38	41.23	40.90	40.42
+ ME + DC + Sigmoid	41.46	41.27	40.93	40.40

Appendix

Table 20: Ablation on the contributions of each EDWARD-NMT component (PhoMT), averaged over both translation directions. ME, DC, Sigmoid stand for Matryoshka Experts, Dual-Consistency and Adaptive Sigmoid Routing respectively. All settings use DASE .

	Target Sparsity ( $\rho_{\text{target}}$ )
Direction	$1.0$	$1.25$	$1.5$
Avg.	40.40	40.66	40.73

Appendix

Table 21: Ablation on target density (

\rho_{\text{target}}

) for the XL variant on the multi-domain benchmark.

	Min Threshold ( $\tau_{\min}$ )
Direction	$0.01$	$0.05$	$0.10$
Avg.	40.63	40.66	40.34

Appendix

Table 22: Ablation on the min routing threshold (

\tau_{\min}

) for the XL variant on the multi-domain benchmark.

	Router Jitter ( $\eta$ )
Direction	$0.01$	$0.05$	$0.10$
Avg.	38.98	40.66	40.62

Appendix

Table 23: Ablation on the router jitter noise (

\eta

) for the XL variant on the multi-domain benchmark.

Figure 2: Average active experts per token during training.

Figure 3: Learnable threshold $\tau$ over the course of training.

Panel of Figure 4 — Figure 4: Percentage of number of active experts per layer across different elastic dimensions on the multi-domain benchmark.

	En $\leftrightarrow$ De
Setting	XL	L	M	S
$\lambda_{dc}=0.1$	40.98	40.26	39.44	38.34
$\lambda_{dc}=1.0$	40.75	40.28	39.56	38.65
$\lambda_{dc}=5.0$	38.33	38.23	37.92	37.44

Appendix

Table 24: Ablation on the Dual-Consistency loss weight (

\lambda_{dc}

) on the multi-domain benchmark, averaged over both translation directions.

Method	Target Density ( $\rho$ )
Method	$\rho=1.0$	$\rho=1.25$	$\rho=1.5$
EDWARD-NMT-S	1.18	1.39	1.69
EDWARD-NMT-M	1.17	1.38	1.67
EDWARD-NMT-L	1.16	1.37	1.66
EDWARD-NMT-XL	1.15	1.37	1.65

Appendix

Table 25: Average active routed experts per token on the Multi-domain benchmark across different target density (

\rho

) settings. Values are averaged across both translation directions and all domains.

Machine Translation

Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.

See all 267 papers

Mixture of Experts for Low-Resource LLMs
May 17, 2026Ori Bar Joseph, Smadar Arvatz, Noam Kayzer +2Mixture-of-Experts Language Models Expert Routing
Sticky Routing: Training MoE Models for Memory-Efficient Inference
Jun 12, 2026Ali KayyamMixture-of-Experts Inference Memory-Efficient Inference
BrainChip Inc., Laguna Hills, CA
Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
Apr 28, 2026Fan Jiang, Yu Zhao, Chenyang Lyu +5Multilingual Language Models Mixture-of-Experts Language Models
Alibaba International Digital Commerce

May 17, 2026J/K move · Enter open · S save

Mixture of Experts for Low-Resource LLMs

Ori Bar Joseph, Smadar Arvatz, Noam Kayzer, Dan Revital+1

Mixture-of-Experts Language Models Expert Routing LLM Routing Low-Resource Language Processing

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) architectures enable efficient model scaling, yet expert routing behavior across underrepresented languages remains poorly understood. We analyze routing dynamics in two architecturally distinct MoE models -- a pure Transformer (Qwen3-30B-A3B) and a hybrid Mamba-Transformer (Nemotron-3-Nano-30B-A3B) -- using Hebrew as a morphologically rich, low-resource testbed. Both pre-trained models exhibit deep-layer routing collapse: usage entropy drops sharply in final layers and tokens concentrate on a narrow expert subset, a pattern largely absent for English. Continual pre-training (CPT) on balanced bilingual data substantially corrects this imbalance, increasing entropy and shifting routing toward shared, language-agnostic experts; supervised fine-tuning (SFT) alone achieves less complete correction. Extending the analysis to Japanese reveals quantitatively consistent collapse signatures, providing cross-linguistic evidence that the phenomenon is a systematic consequence of pre-training underrepresentation rather than any language-intrinsic property. Routing improvements correlate with consistent downstream benchmark gains, positioning routing entropy and expert specialization as principled diagnostics for multilingual capacity in MoE systems.

Abstract

Machine Translation

Explore similar work

Mixture of Experts for Low-Resource LLMs

Sticky Routing: Training MoE Models for Memory-Efficient Inference

Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling