Machine translation (MT) is increasingly utilized in multilingual crisis communication, yet conventional evaluation does not directly assess whether translated emergency messages preserve their actionability. We introduce SIREN (Situation-aware Index for Response and Emergency ActioNability), a human evaluation framework assessing whether translation choices compromise protective action, directive force, and urgency of the source text. We evaluate SIREN in two complementary settings. At the error level, we apply it to HOPE-annotated Italian--English translations produced by ChatGPT-4o and Google Translate: 31.63% and 48.73% of errors, respectively, compromise actionability, predominantly by affecting the protective action. At the message level, we evaluate English--French translations produced by TranslateGemma-4B and Google Translate, with actionability preserved in 85.34% and 92.67% of translations, respectively. TranslateGemma-4B failures predominantly weaken directive force, whereas Google Translate failures more often compromise protective actions and receive Major impact ratings. These findings show that translation systems can differ not only in how often actionability is lost, but also in the nature and functional impact of these failures. Alongside SIREN, we contribute an English--French crisis-translation resource with human actionability annotations and post-edited reference translations, supporting further research on function-oriented evaluation for crisis communication. Our experimental data is available at https://github.com/mcstaiano/SIREN.
Figures & tables
Figure 1: SIREN on Error Span Level.
Figure 2: SIREN on Full Message Level.
System
N
YES
NO
YES (%)
NO (%)
ChatGPT-4o
196
62
134
31.63
68.37
Google Translate
197
96
101
48.73
51.27
Table 1: SIREN-span results for the Italian–English experiment. A YES judgement indicates that the identified translation error compromises the actionability of the emergency message.
ChatGPT-4o ( n=62 )
Google Translate ( n=96 )
SIREN rationale
n
%
n
%
Directive force weakened
9
14.52
19
19.79
Directive force strengthened
4
6.45
0
0.00
Protective action compromised
47
75.81
70
72.92
Urgency marker omitted
2
3.23
7
7.29
Total
62
100.00
96
100.00
Table 2: Distribution of SIREN rationale categories among errors judged to compromise actionability ( Yes ) in the Italian–English experiment.
ChatGPT-4o ( n=62 )
Google MT ( n=96 )
Impact
Score
n
%
n
%
Minor
1
38
61.29
38
39.58
Medium
2
18
29.03
39
40.63
Major
3
6
9.68
19
19.79
Total
–
62
100.00
96
100.00
Table 3: Distribution of SIREN actionability-impact scores among errors judged to compromise actionability ( Yes ) in the Italian–English experiment. Scores 1, 2, and 3 correspond to Minor, Medium, and Major impact, respectively.
System
N
YES
NO
YES (%)
NO (%)
TranslateGemma-4B
191
163
28
85.34
14.66
Google Translate
191
177
14
92.67
7.33
Table 4: SIREN-msg results for the English–French experiment. A YES judgement indicates that the translation preserves the actionability of the emergency message.
TranslateGemma-4B ( n=28 )
Google Translate ( n=14 )
SIREN category
n
%
n
%
Directive force weakened
23
82.14
4
28.57
Directive force strengthened
0
0.00
0
0.00
Protective action compromised
4
14.29
9
64.29
Urgency marker omitted
1
3.57
1
7.14
Total
28
100.00
14
100.00
Table 5: Distribution of SIREN categories among translations for which actionability was not preserved (NO). Percentages are calculated within the NO cases of each system.
TranslateGemma-4B ( n=28 )
Google MT ( n=14 )
Impact
Score
n
%
n
%
Minor
1
15
53.57
4
28.57
Medium
2
10
35.71
2
14.29
Major
3
3
10.71
8
57.14
Total
–
28
100.00
14
100.00
Table 6: Distribution of actionability-impact scores among translations for which actionability was not preserved (NO). Scores 1, 2, and 3 correspond to Minor, Medium, and Major impact, respectively.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: SIREN-span annotation decision tree. Annotators first determine whether an existing translation error compromises actionability. For actionability-compromising errors, they identify the functional rationale and subsequently assign an actionability impact level ( Minor , Medium , or Major ).
System
Source
MT Output
SIREN Rationale
Impact
ChatGPT-4o
Non transitare lungo strade allagate: potrebbero esserci voragini, buche, tombini aperti o cavi elettrici tranciati.
Do not walk along flooded streets: there may be voids, holes, open manholes, or cut electrical wires.
Protective action compromised
Major
ChatGPT-4o
Prima di uscire di casa chiudi gas, acqua e luce e indossa le scarpe.
Before leaving your home, turn off the gas, water, and electricity, and put on sturdy shoes .
Directive force strengthened
Medium
Google Translate
Usa il telefono solo per reale necessità.
Use your phone only for real needs .
Directive force weakened
Medium
Google Translate
Informati sulle vie di allontanamento da seguire previste dalle pianificazioni comunale e nazionale.
Find out about the distancing routes to follow provided for by municipal and national plans.
Protective action compromised
Medium
Appendix
Table 7: Representative SIREN-span annotations from the Italian–English experiment. Boldface highlights translation choices relevant to the actionability judgement.
System
Source
MT Output
SIREN Rationale
Impact
TranslateGemma-4B
Inform yourself about your municipality’s Civil Protection plan.
Vous informer sur le plan de protection civile de votre commune.
Directive force weakened
Minor
TranslateGemma-4B
As a general indication, indoor sheltering may last up to 48 hours.
Pour information générale, la recherche d’abri en intérieur peut durer jusqu’à 48 heures.
Protective action compromised
Major
Google Translate
If you live in the intervention area, check that your home is safe and take all necessary steps to eliminate situations that can represent a danger in case of an earthquake.
Si vous habitez dans la zone d’intervention, vérifiez la sécurité de votre logement et prenez toutes les dispositions nécessaires pour éliminer les situations pouvant présenter un danger en cas de séisme.
Protective action compromised
Medium
Google Translate
Don’t use the car, it could become a trap.
N’utilisez pas votre voiture ; elle pourrait se transformer en piège .
Directive force weakened
Minor
Appendix
Table 8: Representative SIREN-msg annotations from the English–French experiment. Examples are balanced across TranslateGemma-4B and Google Translate. Boldface highlights translation choices relevant to the actionability judgement.
Batch
System
N
Agreement
κ
1
ChatGPT-4o
14
57.14%
−0.024
Google Translate
9
88.89%
0.769
2
ChatGPT-4o
30
90.00%
0.615
Google Translate
30
76.67%
0.516
3
ChatGPT-4o
35
85.71%
0.684
Google Translate
33
90.91%
0.812
Appendix
Table 9: Inter-annotator agreement across the three pilot annotation batches for the Italian–English and English–French experiments.
ChatGPT-4o
Google Translate
Code
HOPE error type
N
Yes (%)
N
Yes (%)
IMP
Impact
16
5 (31.25)
18
11 (61.11)
TRM
Terminology
40
12 (30.00)
33
17 (51.52)
UGR
Ungrammatical
4
1 (25.00)
12
5 (41.67)
MIS
Mistranslation
23
9 (39.13)
29
14 (48.28)
STL
Style
84
25 (29.76)
87
38 (43.68)
Appendix
Table 10: Actionability-compromising errors by HOPE error type in the Italian–English rollout. N indicates the total number of errors in each category. Yes indicates the number of errors judged to compromise actionability, with the percentage relative to N in parentheses.
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.
Emergency messaging such as extreme-weather reports and earthquake instructions can involve high stakes, to the extent that translation errors can lead to tragic consequences. The use of machine translation or generative artificial intelligence might therefore not be recommended. On the other hand, time savings in the initial translation can allow greater investments of resources in revision and authorization processes, as well as a wider range of target languages. An experiment with generative AI translations of an earthquake instruction text from English into Chinese and Spanish shows that use of discourse-specific prompts can considerably improve understandability and actionability, although the translations may still not be trusted by translators. Human revision is still required, not only to detect errors but also because of the ethical need for someone to take responsibility for any errors or delays in such messaging.
Nune Ayvazan, Anthony Pym, Yu Hao
Universitat Rovira i Virgili · University of Melbourne
It is essential that we communicate with the entirety of an affected population during a crisis, including those who do not speak a region's dominant language(s). Commercial Translation tools offer a tempting solution to broaden access and can provide a lifeline when no other resources are available. However, Emergency response managers often operate in the dark regarding the quality and adequacy of these automated outputs. In this paper, we conduct a nationwide survey of emergency managers to understand their current workflows, perspectives, and operational challenges. We then evaluate commercial translation tools against these needs, shedding light on their utility and offering practical guidance on how and when these tools can be safely used in emergency response.
Jen Wilson, Vipasha Bansal, William D. Lewis
Computational Linguistics Program, University of Washington
Communication in times of crisis is essential. However, there is often a mismatch between the language of governments, aid providers, doctors, and those to whom they are providing aid. It is reasonable to turn to commercial MT tools in these scenarios. But how effective are these tools for translating to and from low resource languages, particularly in the crisis or medical domain? In this study, we evaluate 5 commercial systems used for MT (Google Translate, Microsoft Translator, GPT, Gemini and Claude) against the TICO-19 dataset, which contains pandemic-related sentences from a large set of high-priority languages spoken by communities most likely to be adversely affected in the next pandemic. We assess the current degree of "readiness" for this next pandemic based on the usability of the output translations.
Vipasha Bansal, William D. Lewis, Nicole Mihaylova +2