Is MT Ready for the Next Crisis or Pandemic?

Vipasha Bansal; William D. Lewis; Nicole Mihaylova; Elizabeth Brown; Benjamin Pong

cs.CLJan 15, 2026

Is MT Ready for the Next Crisis or Pandemic?

Authors: Vipasha Bansal, William D. Lewis, Nicole Mihaylova, Elizabeth Brown, Benjamin Pong

Organizations: University of Washington

Abstract

Communication in times of crisis is essential. However, there is often a mismatch between the language of governments, aid providers, doctors, and those to whom they are providing aid. It is reasonable to turn to commercial MT tools in these scenarios. But how effective are these tools for translating to and from low resource languages, particularly in the crisis or medical domain? In this study, we evaluate 5 commercial systems used for MT (Google Translate, Microsoft Translator, GPT, Gemini and Claude) against the TICO-19 dataset, which contains pandemic-related sentences from a large set of high-priority languages spoken by communities most likely to be adversely affected in the next pandemic. We assess the current degree of "readiness" for this next pandemic based on the usability of the output translations.

Figures & tables

BLEU Score	Interpretation
< 10	Almost useless
10 - 19	Hard to get the gist
20 - 29	Clear gist, major grammatical errors
30 - 40	Understandable to good translations
40 - 50	High quality translations
50 - 60	Very high quality, fluent translations

Table 1: BLEU Score Interpretation

Data Source	Domain	# Sentences
CMU	Conversational, medical	104
PubMed	Scientific, medical	676
Wikipedia	General	1,153
Wikivoyage	Travel	dev only
Wikinews	News	66
Wikisource	Announcements	99

Table 2: TICO-19 Test Data Statistics by Domain

Panel of Figure 1 — Figure 1: Graphs showing comparative 2026 CHRF++ scores across language pairs (E to X and X to E)

Language	EX/XE	Microsoft	Google	GPT-4o	GPT-5.4	GPT-5.4-mini	Gemini	Claude
es-la	EX	71.88	74.17	73.17	75.05	74.08	73.64	72.84
	XE	75.22	74.70	74.09	74.87	74.47	75.08	73.57
sw	EX	59.96	61.59	59.97	60.55	60.44	60.16	55.86
	XE	63.97	68.14	63.51	68.10	66.35	67.91	62.19
fa	EX	52.32	56.44	53.76	51.16	52.71	51.80	51.38
	XE	64.64	67.55	64.25	66.21	65.77	66.14	63.45

Table 3: 2026 CHRF++ Scores showing example comparisons between providers for Pivot, Important, and Priority languages. See Tables 15 – 22 for complete scores.

Language	EX/XE	Microsoft	Google	GPT-4o	GPT-5.4	GPT-5.4-mini	Gemini	Claude
id	EX	74.72	74.88	74.15	75.08	74.93	73.92	73.74
	XE	72.53	74.33	71.53	73.77	73.27	72.70	71.61
hi	EX	65.62	61.76	59.06	60.34	62.81	61.75	62.42
ha	EX	53.44	51.26	49.12	48.91	49.77	48.21	43.49
rw	EX	45.88	47.87	45.14	49.10	47.74	47.50	33.23
ckb	XE	56.53	57.09	41.84	49.45	46.52	60.66	53.37

Table 4: 2026 CHRF++ Scores showing notable exceptions to overall trends across languages and MT providers. See Tables 15 – 22 for complete scores.

E to X
Microsoft	hi , bn , mr , ta , am, ha , ku, ln , prs
Google	ar , ru , sw , zh , fa , ms , tl , ur , ckb, din, km , kr, lg , ne , om, ps , so , ti-er, ti-et, zu
GPT-5.4	es-la , fr , id , pt-br , my , rw
GPT-5.4 -mini	fr
X to E
Microsoft	es-LA , fr , pt-br , zh , ln

Table 5: Best System and Usability. Bold indicates a usable system, and underline a borderline one.

System		2023	2025	2026
Microsoft (XE)	km	53.70	58.69	60.67
	prs	57.44	57.45	63.11
Microsoft (EX)	ps	46.64	37.84	39.47
	ln	59.00	58.19	65.59
Google (XE)	ar	69.61	70.87	69.17
	so	57.53	37.13	59.39

Table 6: Notable Changes in CHRF++ Scores between 2023, 2025, and 2026 (red indicates drops is quality).

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

(1)	nrefs:1/case:mixed/eff:no/tok:char/smooth:exp/ version:2.3.1
(2)	nrefs:1/case:mixed/eff:no/tok:zh/smooth:exp/ version:2.3.1
(3)	nrefs:1/case:mixed/eff:yes/nc:6/nw:0/space:no/ version:2.3.1
(4)	nrefs:1/case:mixed/eff:no/tok:13a/smooth:exp/ version:2.3.1
(5)	nrefs:1/case:mixed/eff:yes/nc:6/nw:2/space:no/ version:2.3.1

Appendix

Table 7: Metric Signatures

	System	Error Type	Number
E to X
din	Claude	"Refusal"	943 (45%)
	Gemini	Blank Output	1154 (55%)
	GPT-4o	"Refusal"	498 (24%)
fuv	Claude	"Refusal"	83 (4%)
	Gemini	Blank Output	80 (4%)

Appendix

Table 8: LLM Translation Failures

GPT-4o	— English to Nigerian Fulfulde
Source	and can you tell me what other symptoms are you having along with this?
Reference	e footi a limta mandolji nyau feere je a don mari fahin na?
Output	Mido waawi wadugo dum, ko dum wadi dum didi dum wadi dum wadi dum wadi dum wadi dum
	wadi dum wadi dum.
Claude	—English to Nigerian Fulfulde
Source	These enzymes are expressed extensively in multiple organs, especially in the heart and liver, and are

Appendix

Table 9: Example string repetition errors in GPT-4o and Claude output when translating from English to Nigerian Fulfulde.

English to X
	Domain	Microsoft	Google	GPT-4o	GPT-5.4	GPT-5.4-mini	Gemini	Claude
es-LA	CM	32.62	38.80	33.03	45.16	41.44	36.18	34.5
	SM	53.90	59.04	55.45	59.10	56.87	57.42	54.61
	G	51.16	55.42	53.36	56.25	55.09	53.19	52.09
	N	56.19	54.85	59.51	60.63	59.55	58.17	58.67
	A	40.83	42.58	41.52	43.89	44.10	44.40	41.51

Appendix

Table 10: Domain BLEU Scores for Spanish (LA) and Farsi. Bold text indicates best system within each specific domain. Legend for domains: Conversational Medical (CM), Scientific Medical (SM), General (G), News (N), Announcements (A).

	GPT-4o	GPT-5.4	GPT-5.4-mini
ckb	427 (20%)	362 (17%)	431 (21%)
din	—	598 (28%)	707 (34%)
fuv	—	91 (4%)	558 (27%)
ha	112 (5%)	107 (5%)	111 (5%)
kr	406 (19%)	—	408 (19%)
lg	381 (18%)	383 (18%)	386 (18%)

Appendix

Table 11: GPT Content Filtering Rates (XE only)

	GPT-4o	GPT-5.4	GPT-5.4-mini
ckb	28.17/51.77	34.04/58.65	32.95/57.63
din	—	9.35/26.94	9.54/27.20
fuv	—	17.84/37.31	17.43/36.29
ha	34.13/56.36	38.49/60.43	37.00/58.86
kr	11.95/29.64	—	11.64/29.28
lg	31.97/53.59	34.72/56.82	32.58/54.97

Appendix

Table 12: BLEU/CHRF++ Scores for GPT Models with filtered sentences excluded (XE only).

Language	X to English
	Microsoft	Gemini	Margin of Error	% Margin of Error
Hindi	4.5298	4.7679	0.5988	14.97%
Dinka	—	1.8453	0.5988	14.97%
Lingala	3.5536	3.5774	0.2932	7.33%
Tigrinya (Eritrean)	3.0774	3.7262	0.2932	7.33%

Appendix

Table 13: 2026 XE average human evaluation scores and margin of error for all seven evaluated sets. See Appendix I for scale used in evaluation.

Language	% Critical Errors (score of 1)		% Usable Sentences (score of 3-5)
	Microsoft	Gemini	Microsoft	Gemini
Hindi	0.0/0.0/0.0/2.4	0.0/0.0/0.0/2.4	97.6/95.2/	100.0/100.0/
			100.0/95.2	100.0/97.6
Dinka	—	59.5/54.8/	—	19.1/19.1/
		52.4/38.1		19.1/33.3
Lingala	7.1	7.1	75.6	81.0

Appendix

Table 14: Proportion of sentences that were rated as having a critical error and the proportion that were considered usable by human evaluators for each language and system. For Dinka and Hindi, where each evaluator had the same set of sentences, the proportions for each evaluator’s responses are shown separately.

Figure 2: E to X BLEU x CHRF++ Scatter Plot for all TICO languages

Figure 3: X to E BLEU x CHRF++ Scatter Plot for all TICO languages

Language	English to X
	MS	Google	GPT-4o	GPT-5.4	GPT-5.4-mini	Gemini	Claude
——Pivot——
Arabic	28.71	33.24	31.35	32.37	31.68	30.75	28.36
Chinese	47.95	53.10	50.36	47.88	51.01	47.72	49.45
French	44.15	34.94	41.75	46.61	46.58	42.28	41.87
Hindi	46.12	38.88	36.70	37.57	40.63	40.01	40.24

Appendix

Table 15: 2026 EX BLEU scores. The best system for each language is in bold.

Language	English to X
	MS	Google	GPT-4o	GPT-5.4	GPT-5.4-mini	Gemini	Claude
——Pivot——
Arabic	55.08	59.23	57.05	59.32	57.86	57.27	55.64
Chinese	45.74	50.48	47.50	43.74	48.10	42.66	45.89
French	63.64	63.13	62.83	65.74	65.52	63.05	63.02
Hindi	65.62	61.76	59.06	60.34	62.81	61.75	62.42

Appendix

Table 16: 2026 EX CHRF++ scores. The best system for each language is in bold.

Language	X to English
	MS	Google	GPT-4o	GPT-5.4	GPT-5.4-mini	Gemini	Claude
——Pivot——
Arabic	44.91	45.36	42.88	44.62	44.42	46.57	40.40
Chinese	36.40	35.00	33.35	35.38	35.21	36.59	33.16
French	45.57	43.87	43.01	43.48	43.21	44.17	41.35
Hindi	53.28	58.67	46.22	53.37	53.62	53.69	49.70

Appendix

Table 17: 2026 XE BLEU scores. The best system for each language is in bold.

Language	X to English
	MS	Google	GPT-4o	GPT-5.4	GPT-5.4-mini	Gemini	Claude
——Pivot——
Arabic	68.06	69.17	66.88	69.12	68.78	69.19	66.51
Chinese	62.16	61.87	60.40	62.53	62.35	62.20	60.76
French	64.76	64.06	63.18	63.97	63.68	64.04	62.67
Hindi	73.92	76.79	69.39	74.52	74.29	74.13	72.26

Appendix

Table 18: 2026 XE CHRF++ scores. The best system for each language is in bold.

Language	English to X
	MS	Google	GPT-4o	GPT-5.4	GPT-5.4-mini	Gemini	Claude
——Pivot——
Arabic	0.8744	0.8800	0.8759	0.8830	0.8788	0.8797	0.8718
Chinese	0.8801	0.8882	0.8834	0.8710	0.8843	0.8716	0.8773
French	0.8772	0.8440	0.8692	0.8801	0.8791	0.8703	0.8701
Hindi	0.9038	0.8876	0.8809	0.8863	0.8982	0.8915	0.8956

Appendix

Table 19: 2026 EX BERTScores. The best system for each language is in bold.

Language	X to English
	MS	Google	GPT-4o	GPT-5.4	GPT-5.4-mini	Gemini	Claude
——Pivot——
Arabic	0.7486	0.7416	0.7406	0.7514	0.7506	0.7641	0.7285
Chinese	0.7148	0.6959	-0.5152	0.7098	0.7118	0.7160	0.6920
French	0.6865	0.6627	0.6702	0.6762	0.6744	0.6810	0.6700
Hindi	0.8113	0.8225	-0.7376	0.8118	0.8162	0.8206	0.7948

Appendix

Table 20: 2026 XE BERTScores (rescaled). The best system for each language is in bold.

Language	English to X
	MS	Google	GPT-4o	GPT-5.4	GPT-5.4-mini	Gemini	Claude
——Pivot——
Arabic	0.8485	0.8750	0.8683	0.8764	0.8688	0.8761	0.8649
Chinese	0.8749	0.8953	0.8840	0.8753	0.8862	0.8779	0.8804
French	0.8154	0.8242	0.8199	0.8245	0.8229	0.8239	0.8199
Hindi	0.8076	0.8203	0.8064	0.8061	0.8217	0.8177	0.8148

Appendix

Table 21: 2026 EX COMET. The best system for each language is in bold. Only supported languages are included.

Language	X to English
	MS	Google	GPT-4o	GPT-5.4	GPT-5.4-mini	Gemini	Claude
——Pivot——
Arabic	0.8725	0.8779	0.8765	0.8788	0.8784	0.8820	0.8716
Chinese	0.8751	0.8765	0.6173	0.8778	0.8781	0.8790	0.8728
French	0.8360	0.8353	0.8352	0.8370	0.8362	0.8367	0.8334
Hindi	0.9070	0.9150	0.4515	0.9121	0.9125	0.9135	0.9075

Appendix

Table 22: 2026 XE COMET. The best system for each language is in bold. Only supported languages are included.

	English to X				X to English
Language	Microsoft		Google		Microsoft		Google
	2023	2025	2023	2025	2023	2025	2023	2025
——Pivot——
Arabic	30.26	25.16	29.15	32.88	44.88	44.87	46.90	49.28
Portuguese (BR)	52.42	47.83	52.27	54.26	54.81	55.33	55.19	57.63
Chinese	52.89	50.24	54.29	54.27	36.46	36.48	37.71	39.56

Appendix

Table 23: BLEU Score Differences between 2023 and 2025 for Microsoft and Google

	English to X				X to English
Language	Microsoft		Google		Microsoft		Google
	2023	2025	2023	2025	2023	2025	2023	2025
——Pivot——
Arabic	57.73	56.41	56.70	59.18	67.74	67.73	69.61	70.87
Portuguese (BR)	72.79	71.12	72.48	74.04	75.43	75.88	75.45	76.79
Chinese	39.76	45.07	48.72	45.33	62.08	62.69	39.76	64.60

Appendix

Table 24: CHRF++ Score Differences between 2023 and 2025 for Microsoft and Google

	English to X				X to English
Language	Microsoft		Google		Microsoft		Google
	2023	2025	2023	2025	2023	2025	2023	2025
——Pivot——
Arabic	0.8841	0.8375	0.8791	0.8833	0.7661	0.7465	0.7701	0.7987
Portuguese (BR)	0.9363	0.8750	0.9339	0.9521	0.8434	0.8302	0.8248	0.8437
Chinese	0.8930	0.8619	0.8971	0.9450	0.7373	0.7234	0.7271	0.7400

Appendix

Table 25: BERTScore Differences between 2023 and 2025 for Microsoft and Google

	English to X				X to English
Language	Microsoft		Google		Microsoft		Google
	2023	2025	2023	2025	2023	2025	2023	2025
——Pivot——
Arabic	0.8450	0.8253	0.8486	0.8813	0.8710	0.8400	0.8777	0.8891
Portuguese (BR)	0.8978	0.8698	0.8976	0.8962	0.9007	0.8790	0.9002	0.8983
Chinese	0.8824	0.8652	0.8924	0.8910	0.8738	0.8090	0.8755	0.8752

Appendix

Table 26: COMET Score Differences between 2023 and 2025 for Microsoft and Google

Machine Translation

Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.

See all 267 papers

SIREN: Situation-aware Index for Response and Emergency ActioNability in Crisis Translation
Oct 12, 2026Maria Carmen Staiano, Lifeng Han, Johanna Monti +1Machine Translation MT Evaluation
University of Macerata · LUMC & Leiden University · University of Naples “L’Orientale”
Communicating with the Entire Population: Machine Translation of Emergency Alerts and Warnings
Oct 23, 2026Jen Wilson, Vipasha Bansal, William D. LewisMachine Translation Disaster Response
Computational Linguistics Program, University of Washington
CrisiLangs: A Multilingual Resource for Crisis Warning Generation
Oct 22, 2026Giacomo Gonella, Stefano Menini, Marco GueriniMultilingual Language Models Early Warning Detection
Fondazione Bruno Kessler, Italy · University of Trento, Italy

Oct 12, 2026J/K move · Enter open · S save

SIREN: Situation-aware Index for Response and Emergency ActioNability in Crisis Translation

Maria Carmen Staiano, Lifeng Han, Johanna Monti, Francesca Chiusaroli

University of Macerata · LUMC & Leiden University · University of Naples “L’Orientale”

Machine Translation MT Evaluation

Open paper AskPDF arXiv

Machine translation (MT) is increasingly utilized in multilingual crisis communication, yet conventional evaluation does not directly assess whether translated emergency messages preserve their actionability. We introduce SIREN (Situation-aware Index for Response and Emergency ActioNability), a human evaluation framework assessing whether translation choices compromise protective action, directive force, and urgency of the source text. We evaluate SIREN in two complementary settings. At the error level, we apply it to HOPE-annotated Italian--English translations produced by ChatGPT-4o and Google Translate: 31.63% and 48.73% of errors, respectively, compromise actionability, predominantly by affecting the protective action. At the message level, we evaluate English--French translations produced by TranslateGemma-4B and Google Translate, with actionability preserved in 85.34% and 92.67% of translations, respectively. TranslateGemma-4B failures predominantly weaken directive force, whereas Google Translate failures more often compromise protective actions and receive Major impact ratings. These findings show that translation systems can differ not only in how often actionability is lost, but also in the nature and functional impact of these failures. Alongside SIREN, we contribute an English--French crisis-translation resource with human actionability annotations and post-edited reference translations, supporting further research on function-oriented evaluation for crisis communication. Our experimental data is available at https://github.com/mcstaiano/SIREN.

Abstract

Machine Translation

Explore similar work

SIREN: Situation-aware Index for Response and Emergency ActioNability in Crisis Translation

Communicating with the Entire Population: Machine Translation of Emergency Alerts and Warnings

CrisiLangs: A Multilingual Resource for Crisis Warning Generation