Expert pruning compresses Mixture-of-Experts (MoE) language models by removing the experts that a calibration set marks as unimportant. Studies of high-resource languages largely find that English calibration transfers well to other languages. We test this for Lithuanian: we pruned four MoE models with REAP, removing 25% or 50% of the experts with English, Lithuanian or mixed calibration, and evaluated both languages on reading comprehension, translation and language modelling. English calibration does not transfer to Lithuanian: at 25% pruning, English comprehension fell by a median of 1.7 points but Lithuanian by 21.4. The transfer is asymmetric. Lithuanian calibration removed most of the Lithuanian loss and never significantly hurt English comprehension, which was on average 2.5 points better than under English calibration; only English language modeling got slightly worse. Routing suggests why: in the last layers, Lithuanian sends 55-72% of its expert selections to just eight experts, against 19-39% for English, and English calibration removes some of these experts. Compressed models should be calibrated, and evaluated, in every language they must serve.
Figures & tables
Figure 1: At 25% expert pruning, English calibration erodes Lithuanian far more than English. Black bars are the unpruned models and points their 25%-pruned versions. Unless a caption says otherwise, colour marks the calibration language in all figures (EN blue, LT orange, MIX green); circles and squares mark Wikipedia and Europarl calibration, and hollow and filled markers mark 25% and 50% pruning. Higher is better; the dotted line is chance on Belebele. English-calibrated models keep their English comprehension (except Gemma 4 with English Wikipedia calibration) but lose up to 67.6 points in Lithuanian, and they translate worse, especially into Lithuanian.
Work
Models
Compression
Calib. lang.
Languages
Gen.
Salim et al. (2026a)
gpt-oss-20b (MoE)
expert pruning
EN vs ZH
EN, 7 in MMLU-ProX
no
Salim et al. (2026b)
gpt-oss-20b (MoE)
expert pruning
yes
7 high-resource
partly
Martin et al. (2026)
gpt-oss-20b, Qwen3-30B (MoE)
expert pruning
translation data
4 + 3 unseen
yes
Chen et al. (2026)
Qwen3-30B (MoE)
expert masking
–
10
yes
Choenni and Titov (2025)
6 dense
Wanda pruning
multilingual
15
no
Chimoto et al. (2026)
2 dense
quantisation
yes
10
no
Table 1: How this study differs from the closest prior work (“Calib. lang.”: calibration language varied; “Gen.”: open generation evaluated).
Belebele (pp)
FLORES (chrF++)
Wikipedia (pp)
Calib.
Ratio
gap
sig. + / −
gap
sig. + / −
gap
sig. + / −
EN
25%
+23.5
8/0
+12.8
7/0
+50.0
8/0
50%
+30.6
6/0
+8.5
5/2
+116.7
8/0
LT
25%
+3.0
5/0
− 0.3
0/3
− 2.4
0/8
50%
+5.1
6/1
+1.2
2/3
− 8.1
0/8
MIX
25%
+5.6
5/0
+0.1
1/1
− 1.2
0/7
Table 3: Mean LT–EN gap over the four models and two domains (positive = Lithuanian loses more), and the number of the 8 configurations whose gap is significantly positive/negative.
Figure 2: LT–EN gap with 95% intervals for each model (columns) and calibration set (x axis). Rows: Belebele (pp), FLORES (chrF++) and Wikipedia (pp of relative NLL increase), with different y scales. Positive means that Lithuanian loses more; markers as in Figure 1 . English-calibrated gaps are large except for Gemma 4 and runs where both languages collapsed; LT and MIX gaps are small except for Gemma 4.
LT − EN
MIX − EN
MIX − LT
Evaluation
mean
B/W
mean
B/W
mean
B/W
Belebele EN (pp)
+2.5
6/0
+1.4
7/1
− 1.1
4/4
Belebele LT (pp)
+25.5
15/0
+22.5
14/1
− 3.0
0/7
FLORES LT → EN (chrF++)
+18.0
16/0
+16.5
16/0
− 1.5
3/2
FLORES EN → LT (chrF++)
+28.2
16/0
+27.2
16/0
− 1.0
1/8
Wikipedia EN (nats)
− 0.06
5/11
− 0.03
5/10
+0.03
12/3
Table 4: Effect of switching the calibration language: mean difference over 16 configurations (positive = the first is better; Wikipedia: NLL reduction in nats per token). B/W: how many are significantly better/worse.
Figure 3: Transfer between calibration languages is asymmetric. Degradation (positive = worse) of English (left) and Lithuanian (right) for each calibration language, at 25% (top) and 50% (bottom); one point per model and domain; black bars are medians. For FLORES, English is LT → EN. Every calibration language preserves English; only LT and MIX preserve Lithuanian.
Figure 4: Degradation at 0, 25 and 50% of experts pruned (positive = worse). Lines are medians over the six calibration sets and bands span their minimum to maximum; colour is the evaluated language (FLORES: output language). Losses grow moderately up to 25% and steeply to 50%; the worst Lithuanian cases are usually English-calibrated. Gemma 4 collapses at 50%.
Figure 5: Removed routing share : the share of each language’s routing that went to the experts pruning removed, measured on the unpruned model for English (blue) and Lithuanian (orange) Wikipedia text, at 25% (top) and 50% (bottom); circles and squares are Wikipedia and Europarl calibration; the dashed line is uniform routing. Under EN calibration Lithuanian loses more routing than English in every model; under LT calibration the order reverses.
Figure 6: Lithuanian relies on few experts at the end of the network, and EN calibration removes some of them. Top: share of routing handled by the 8 most-used experts in each MoE layer (unpruned; colour = evaluated language). Middle and bottom: removed routing share of each language at 25% under EN and LT calibration (mean of the two domains); dashed = uniform; the last 15% of layers are shaded. How much more Lithuanian routing EN calibration removes there depends on the model.
Figure 7: Lithuanian translation breaks beyond runaway generation. Left: mean EN → LT chrF++ drop (positive = worse) on all sentences, on the sentences the model finished (compared with the unpruned model on the same sentences) and with repetition loops collapsed; markers as in Figure 1 . Right: words of finished EN → LT translations that occur in no reference or unpruned output (% of all words), split into Lithuanian-looking forms (orange), words with non-Lithuanian letters (grey) and English or source words (blue). At 25% EN calibration mostly produces unattested Lithuanian-looking forms, at 50% mostly English.
Hypothesis
Decision rule
Verdict
H1: pruning degrades LT more than EN
gap significantly positive in most of the 16 configurations of a calibration language
Supported under EN calibration (14/16 Belebele, 12/16 FLORES, 16/16 Wikipedia); under LT/MIX only a small Belebele gap (11/16, 10/16)
H2: LT calibration reduces LT degradation
LT better on a Lithuanian evaluation in most of the 16 configurations and on average in all 4 models
Supported : 16/16 EN → LT, 16/16 Wikipedia, 15/16 Belebele; all four models
H3: LT/MIX preserve LT without hurting EN
English Belebele never significantly worse than under EN
Supported under LT for comprehension; 1/16 worse under MIX; English NLL rises slightly
H4: MIX more balanced than LT
MIX reduces the worse language’s Belebele loss in most of the 16 configurations
Not supported (3/16)
H5: the gap grows from 25% to 50%
change in gap significantly positive in most of 8 configurations
Only for Wikipedia under EN calibration (8/8); Belebele (3/8) and FLORES (3/8) capped by chance and English collapse
Table 5: Hypotheses, decision rules and verdicts. Counts are configurations with a significant effect.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Ratio
experts/layer
params (B)
params − %
size (GB)
size − %
prune time (min)
Qwen3-30B-A3B-Instruct-2507
25%
128 → 96
30.5 → 23.3
23.7
61.1 → 46.6
23.8
29–30
50%
128 → 64
30.5 → 16.0
47.5
61.1 → 32.1
47.5
27
Qwen3.6-35B-A3B
25%
256 → 192
34.7 → 26.6
23.2
71.9 → 55.8
22.4
51–52
50%
256 → 128
34.7 → 18.5
46.5
71.9 → 39.7
44.8
46–47
gemma-4-26B-A4B-it
25%
128 → 96
25.8 → 20.1
22.1
51.6 → 40.2
22.1
21
50%
128 → 64
25.8 → 14.4
44.3
51.6 → 28.8
44.3
20
Appendix
Table 6: Compression per model and ratio (means over the six calibration sets; prune time is the range). Parameter counts include the vision towers of Gemma 4 and Qwen3.6 and exclude the MTP heads; checkpoint sizes include the MTP heads, which are copied unpruned.
Figure 8: LT–EN gap for all 48 configurations. Panels from top: Belebele (pp), FLORES (chrF++), Wikipedia (pp); rows: model and ratio; columns: calibration set. Red = Lithuanian loses more, blue = English loses more; Wikipedia values rounded to integers. † = collapsed in both languages (Belebele accuracy ≤ 30%): a gap near 0 there means that both languages broke, not that they degraded equally. The EN-calibrated columns are red throughout.
Figure 9: LT vs EN calibration , paired by model, domain and ratio (rows). Columns: benchmarks; top row evaluated in English, bottom row in Lithuanian. Right of 0 = LT calibration better (Wikipedia NLL negated, nats/token); bars are 95% intervals; here fill = significance (filled = interval excludes 0), not ratio; colour = evaluated language. LT calibration is better for every Lithuanian evaluation except collapsed Gemma 4 Wikipedia, 50% ; English Belebele never gets significantly worse.
Factor
LT Belebele degr.
EN Belebele degr.
Belebele gap
FLORES gap
Wikipedia gap
calibration language
24
0
39
36
70
model
22
50
15
8
4
pruning ratio
21
19
1
0
4
calibration domain
0
1
1
1
0
Appendix
Table 7: Share of variance (%) across the 48 configurations explained by each design factor alone (one-way η2 ).
Model
Ratio
best by EN Belebele
best by LT Belebele
LT acc. lost (pp)
EN acc. cost (pp)
Qwen3-30B-A3B-Instruct-2507
25%
Europarl-LT
Europarl-MIX
0.1
0.3
50%
Europarl-MIX
Europarl-LT
1.6
0.4
Qwen3.6-35B-A3B
25%
Europarl-MIX
Europarl-LT
0.1
1.2
50%
Wikipedia-LT
Wikipedia-LT
0.0
0.0
gemma-4-26B-A4B-it
25%
Europarl-LT
Europarl-LT
0.0
0.0
50%
Europarl-LT
Europarl-MIX
0.2
13.0
Appendix
Table 8: Calibration set that maximises English or Lithuanian Belebele accuracy per model and ratio, the Lithuanian accuracy lost by choosing with English, and the English accuracy given up by choosing with Lithuanian.
Model
Calibration set
Belebele (pp)
FLORES (chrF++)
Wikipedia (pp)
Qwen3-30B
Wikipedia-EN
+40.8 [36.7, 44.8]
+4.7 [3.6, 5.8]
+68.0 [67.3, 68.7]
Wikipedia-LT
+8.3 [5.6, 11.3]
− 0.1 [ − 0.9, 0.6]
− 9.6 [ − 9.9, − 9.4]
Wikipedia-MIX
+14.4 [11.3, 17.7]
+0.6 [ − 0.1, 1.4]
− 5.7 [ − 6.0, − 5.4]
Europarl-EN
− 3.2 [ − 6.1, − 0.3]
− 5.1 [ − 5.9, − 4.1]
+19.6 [19.0, 20.2]
Europarl-LT
+4.3 [1.9, 6.9]
0.0 [ − 0.7, 0.7]
− 10.5 [ − 10.9, − 10.2]
Europarl-MIX
+6.8 [4.2, 9.3]
− 0.5 [ − 1.2, 0.3]
− 9.0 [ − 9.4, − 8.6]
Appendix
Table 9: Change of the LT–EN gap from 25% to 50% pruning (gap at 50% minus gap at 25%) with 95% paired bootstrap intervals (Belebele: items resampled jointly in both languages, 4,000 resamples; FLORES: sentences, 1,000 resamples; Wikipedia: documents per language, 4,000 resamples). Positive = the gap grows.
Unpruned routing, EN vs LT (range over 4 models)
On removed experts at 25% (%)
top-8 share
EN calib.
LT calib.
Depth
ΔH
JSD
ρ
EN
LT
EN
LT
EN
LT
first 15%
0.07–0.12
0.16–0.28
0.45–0.73
14–32
28–49
20.4
25.1
22.8
16.1
middle 35–65%
0.03–0.07
0.09–0.13
0.84–0.91
22–41
29–53
13.7
18.7
14.6
14.0
last 15%
0.14–0.22
0.33–0.49
0.21–0.69
19–39
55–72
10.3
20.5
14.8
8.1
Appendix
Table 10: Where in the network English and Lithuanian differ. Left: routing of the unpruned models, as ranges over the four models: EN − LT difference in normalised selection entropy ( ΔH ), routing divergence (JSD), rank correlation of expert usage ( ρ ), and share of routing handled by the 8 most-used experts (%). Right: share of each language’s routing on the experts removed at 25% (%, mean of 8 configurations). Depth bands: first 15%, middle 35–65% and last 15% of the MoE layers; the measures are defined in Appendix A .
Figure 10: Per-layer EN vs LT saliency rank correlation (saliency-only runs; solid = Wikipedia, dashed = Europarl). Agreement is lowest in the first and last layers.
Figure 11: Which calibration changes move the retained expert set? Jaccard of retained experts per kind of calibration pair, averaged per model (glyph = model); hollow = 25%, filled = 50%; black bars are means; dotted lines are random selection. Changing the domain moves the set at least as much as changing EN to LT in three of the four models.
Figure 12: Coverage vs robustness. Each language’s own routing share removed (x) against its own Belebele degradation (y; positive = worse), every configuration at 25%; colour = evaluated language; circle/square = Wikipedia/Europarl; grey rings mark Gemma 4, which is excluded from the fitted lines. Lithuanian accuracy falls steeply with the share of its routing removed, English accuracy barely moves.
Figure 13: Belebele accuracy vs answer-letter collapse for the 48 pruned runs, each evaluated in both languages (96 points; colour = evaluated language; circle/square = Wikipedia/Europarl; hollow/filled = 25%/50%), and the unpruned models ( × ). Left: share of the most-predicted letter; right: probability mass on A–D. Runs at chance accuracy put most answers on one letter and little probability on letters at all.
Mixture-of-Experts language models use input-dependent routing to activate a small subset of expert networks, increasing capacity without proportional compute.
Institute of Information Science, Academia Sinica · National Taiwan University of Science and Technology · AI Research Center, Hon Hai Research Institute