Word error rate (WER) is the dominant metric for automatic speech recognition, yet it cannot detect a systematic failure mode: models that produce fluent output in the wrong writing system. We define Script Fidelity Rate (SFR), the fraction of hypothesis characters in the target script block, computable without reference transcriptions, and report a systematic measurement of script collapse across ten languages spanning six writing systems and ten models (seven Whisper sizes, MMS-1B, SeamlessM4T-v2, and Gemma 4 E2B) on FLEURS test sets. Across 100 evaluated model-language pairs, 21 (21%; 95% Wilson CI: 14-30%) exhibit script collapse (SFR less than 10%): 20 involve Whisper and one involves Gemma 4 E2B on Urdu under a generic transcription prompt. In a ten-language Gemma 4 probe, script-aware prompting raises mean SFR from 71.2% to 97.7%, fixes Urdu collapse (6.5% to 97.0%), and recovers 5.9 chrF on downstream NLLB translation for the six languages whose baseline SFR is below 90%. We identify four collapse patterns: Latin phonetic substitution, Arabic substitution for Somali, Devanagari substitution for Bengali/Malayalam, and unique-script Latin collapse for Georgian.
Figures & tables
Language
Script
Rℓ (main range)
∣Uℓ∣
Pashto
Perso-Arabic
U+0600–06FF
12
Urdu
Perso-Arabic
U+0600–06FF
5
Arabic
Perso-Arabic
U+0600–06FF
0
Persian
Perso-Arabic
U+0600–06FF
4
Hindi
Devanagari
U+0900–097F
0
Bengali
Bengali
U+0980–09FF
0
Table 1: Script configurations for all ten target languages. Rℓ = primary Unicode block range(s); Uℓ = unique code points (non-empty for languages sharing a block with others).
Language
FLEURS code
Script
N
Role
Pashto
ps_af
Perso-Arabic
512
C
Urdu
ur_pk
Perso-Arabic
299
V
Arabic
ar_eg
Perso-Arabic
428
V
Persian
fa_ir
Perso-Arabic
871
V
Hindi
hi_in
Devanagari
418
C
Bengali
bn_in
Bengali
920
C
Table 2: Evaluation datasets. N = FLEURS test utterances. Role: C = collapse target, V = validation control, E = extension.
Figure 1: Script Fidelity Rate (%) for the 100 evaluated model–language pairs on FLEURS test sets. Green cells indicate high script fidelity; red cells indicate script collapse. The full numeric matrix appears in Appendix A .
Model family
Mean SFR
Median
Collapsed
Whisper
65.3 [54.6, 75.5]
98.3
20 / 70
MMS-1B
99.4 [99.2, 99.5]
99.3
0 / 10
SeamlessM4T
99.9 [99.8, 100.0]
100.0
0 / 10
Gemma 4
71.2 [50.2, 88.7]
82.9
1 / 10
All models
72.8 [64.5, 80.5]
99.1
21 / 100
Table 3: SFR distribution summary across 100 evaluated model–language pairs. Brackets give 95 % bootstrap confidence intervals for mean SFR, resampling model–language pairs within each family.
Language
Base SFR
Hint SFR
Δ WER
Δ chrF
Δ BLEU
Pashto
76.6
98.8
-12.6
4.4
1.1
Urdu
6.5
97.0
-75.3
5.0
3.9
Bengali
89.3
97.1
-5.0
0.7
0.8
Malayalam
63.5
98.0
-23.9
6.7
4.6
Tamil
70.1
96.4
-15.8
4.9
2.2
Georgian
19.5
93.0
-43.4
13.5
6.2
Table 4: Gemma 4 script-aware prompting and downstream MT recovery. Δ WER is script-hint minus baseline, so negative is better. Δ chrF and Δ BLEU are script-hint MT minus baseline MT against English FLEURS references.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Perso-Arabic
Indic + Tamil (collapse targets)
Latin
Georgian
Model
Pashto
Urdu
Arabic
Persian
Hindi
Bengali
Malayalam
Tamil
Somali
Georgian
Whisper tiny
7
100
100
100
1
0
0
100
99
0
Whisper base
43
100
100
100
5
0
0
99
0
0
Whisper small
97
100
99
100
100
0
0
99
0
16
Whisper medium
34
99
99
99
98
3
2
99
2
94
Whisper large-v2
71
100
99
99
96
1
0
99
0
41
Appendix
Table 5: Script Fidelity Rate (SFR, %) for all 100 evaluated model–language pairs on FLEURS test sets.
Figure 2: WER (%) vs SFR (%) for all 100 model–language pairs on FLEURS test sets, grouped by language. Red dashed line at SFR = 10 % (collapse boundary); green dashed line at SFR = 90 %. Points are coloured red (SFR < 10 %), orange (10–90 %), and green ( > 90 %).
SFR threshold
1 %
5 %
10 %
20 %
Collapsed pairs
15
18
21
24
Appendix
Table 6: Sensitivity of the collapse count to the SFR threshold.
Figure 3: Georgian SFR (%) and WER (%) by model. SFR bars are coloured by collapse zone (red < 10 %, orange 10–90 %, green > 90 %). WER is plotted on the right axis.
Language
Model family
SFR
Target
Substitute script
Subst.
Pattern 1: Latin phonetic substitution
Pashto
W-tiny/base
25
25
Latin
69 %
Pashto
W-small/med
66
64
Latin
6 %
Pashto
W-large-v2
71
71
Latin
29 %
Hindi
W-tiny/base
3
3
Latin
82 %
Bengali
W-tiny/base
0
0
Latin
92 %
Appendix
Table 7: Dominant output script for collapse-target languages, grouped by pattern. SFR = Script Fidelity Rate (%); Target = % utterances with target-script-dominant output; Subst. = % utterances with substitute script dominant. Rows omitted where SFR ≥ 90 %. “Other” covers mixed-script and unclassified blocks; rows may not sum to 100.
Language
Prompt
SFR
Low SFR
Top LID
Top LID %
Urdu
baseline
6.5
88.6
Hindi
91.0
Urdu
script hint
97.0
1.3
Urdu
98.3
Georgian
baseline
19.5
76.5
Estonian
27.3
Georgian
script hint
93.0
2.1
Estonian
83.0
Pashto
script hint
98.8
0.2
Persian
99.2
Somali
baseline
99.6
0.0
Somali
99.7
Appendix
Table 8: SFR+LID audit on saved Gemma 4 predictions. LID uses langdetect; top label is the most frequent detected language.