It is essential that we communicate with the entirety of an affected population during a crisis, including those who do not speak a region's dominant language(s). Commercial Translation tools offer a tempting solution to broaden access and can provide a lifeline when no other resources are available. However, Emergency response managers often operate in the dark regarding the quality and adequacy of these automated outputs. In this paper, we conduct a nationwide survey of emergency managers to understand their current workflows, perspectives, and operational challenges. We then evaluate commercial translation tools against these needs, shedding light on their utility and offering practical guidance on how and when these tools can be safely used in emergency response.
Figures & tables
Figure 1: Human Translation and Translation Tools
Figure 2: Strategies of respondents to obtain translations.
Figure 3: Operational challenges faced by survey respondents.
Type of EM
ID
Juridisction Scope
Location
Local Emergency Manager
L1
County
West Coast
Public Health Coordinator
H1
County
West Coast
Public Information Officer
P1
City
East Coast
Advocate (Retired Researcher)
R1
State
West Coast
Table 1: Demographic information of quoted survey participants.
Figure 4: Count of Language Speakers in the NY/NJ Metropolitan area vs Nationally. Trend line represents the expected population if the percentages were identical in the NY/NJ Metropolitan area vs Nationally. U.S. Census Bureau (2024)
National
Atlanta
Chicago
Dallas
Los Angeles
Minneapolis
San Francisco
Seattle
Total
321,745,943
1.88%
2.77%
2.43%
3.82%
1.10%
1.37%
1.22%
Amharic, Somali, or other Afro-Asiatic languages
238,986
3.73%
3.56%
4.96%
2.26%
14.53%
1.84%
6.31%
Arabic
513,293
0.48%
4.20%
3.23%
5.14%
0.65%
1.86%
1.11%
Armenian
109,630
0.00%
0.54%
0.00%
83.24%
0.00%
0.40%
0.65%
Bengali
209,543
1.66%
1.74%
2.17%
3.35%
0.32%
0.17%
0.71%
Chinese
1,895,001
1.36%
2.76%
1.33%
14.23%
0.54%
10.76%
3.22%
Table 2: Language Variation of actual vs. expected representation for select Metropolitan areas in the United States U.S. Census Bureau (2024) . underrepresented = expected/2 overrepresented = expected*2 Languages spoken by fewer than 10,000 people are grouped with other languages by the U.S. Census Bureau. More information can be found here: https://usa.ipums.org/usa/volii/00language.shtml .
Language
Microsoft
Google
GPT-5.4- mini
Gemini 2026
Claude
Pivot
Arabic
7.34
7.42
7.66
8.34
7.67
Chinese
11.97
10.66
13.02
11.10
12.67
French
5.67
4.85
6.03
4.89
5.47
Hindi
13.16
9.28
10.33
8.34
12.24
Indonesian
1.98
3.33
2.97
3.03
2.44
Table 3: Differences between Overall CHRF++ Score and News+Announcements CHRF++. Blue indicates Overall score is higher than N+A, orange indicates Overall is lower than N+A. Google Translate results were unavailable for Dari. Khmer, Myanmar, and Chinese do not use whitespace to mark word boundaries and CHRF was used instead. Bansal et al. (2026)
Microsoft
Google
GPT-5.4-mini
Gemini 2026
Claude
Language
Overall
N+A
Overall
N+A
Overall
N+A
Overall
N+A
Overall
N+A
Pivot
Arabic
55.08
47.73
59.23
51.82
57.86
50.20
57.27
48.93
55.64
47.97
Chinese
45.74
33.77
50.48
39.82
48.10
35.08
42.66
31.56
45.89
33.21
French
63.64
57.97
63.13
58.28
65.52
59.49
63.05
58.16
63.02
57.55
Hindi
65.62
52.46
61.76
52.48
62.81
52.48
61.75
53.41
62.42
50.18
Table 4: Overall and News+Announcements (N+A) CHRF++ scores for English → Target Language across translation systems and languages. Highest scoring language result for Overall and N+A is in bold. Shading indicates relative score ranges within the evaluation: blue represents higher scores (>60), pale green represents mid-range scores (45-60), and orange represents lower scores (<45). CHRF++ does not have a universal safety or usability threshold, so these colors indicate relative performance only. Google Translate results were unavailable for Dari. Khmer, Myanmar, and Chinese do not use whitespace to mark word boundaries and CHRF was used instead. Bansal et al. (2026)
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.