Machine translation (MT) is increasingly utilized in multilingual crisis communication, yet conventional evaluation does not directly assess whether translated emergency messages preserve their actionability. We introduce SIREN (Situation-aware Index for Response and Emergency ActioNability), a human evaluation framework assessing whether translation choices compromise protective action, directive force, and urgency of the source text. We evaluate SIREN in two complementary settings. At the error level, we apply it to HOPE-annotated Italian--English translations produced by ChatGPT-4o and Google Translate: 31.63% and 48.73% of errors, respectively, compromise actionability, predominantly by affecting the protective action. At the message level, we evaluate English--French translations produced by TranslateGemma-4B and Google Translate, with actionability preserved in 85.34% and 92.67% of translations, respectively. TranslateGemma-4B failures predominantly weaken directive force, whereas Google Translate failures more often compromise protective actions and receive Major impact ratings. These findings show that translation systems can differ not only in how often actionability is lost, but also in the nature and functional impact of these failures. Alongside SIREN, we contribute an English--French crisis-translation resource with human actionability annotations and post-edited reference translations, supporting further research on function-oriented evaluation for crisis communication. Our experimental data is available at https://github.com/mcstaiano/SIREN.
Figures & tables
Figure 1: SIREN on Error Span Level.
Figure 2: SIREN on Full Message Level.
System
N
YES
NO
YES (%)
NO (%)
ChatGPT-4o
196
62
134
31.63
68.37
Google Translate
197
96
101
48.73
51.27
Table 1: SIREN-span results for the Italian–English experiment. A YES judgement indicates that the identified translation error compromises the actionability of the emergency message.
ChatGPT-4o ( n=62 )
Google Translate ( n=96 )
SIREN rationale
n
%
n
%
Directive force weakened
9
14.52
19
19.79
Directive force strengthened
4
6.45
0
0.00
Protective action compromised
47
75.81
70
72.92
Urgency marker omitted
2
3.23
7
7.29
Total
62
100.00
96
100.00
Table 2: Distribution of SIREN rationale categories among errors judged to compromise actionability ( Yes ) in the Italian–English experiment.
ChatGPT-4o ( n=62 )
Google MT ( n=96 )
Impact
Score
n
%
n
%
Minor
1
38
61.29
38
39.58
Medium
2
18
29.03
39
40.63
Major
3
6
9.68
19
19.79
Total
–
62
100.00
96
100.00
Table 3: Distribution of SIREN actionability-impact scores among errors judged to compromise actionability ( Yes ) in the Italian–English experiment. Scores 1, 2, and 3 correspond to Minor, Medium, and Major impact, respectively.
System
N
YES
NO
YES (%)
NO (%)
TranslateGemma-4B
191
163
28
85.34
14.66
Google Translate
191
177
14
92.67
7.33
Table 4: SIREN-msg results for the English–French experiment. A YES judgement indicates that the translation preserves the actionability of the emergency message.
TranslateGemma-4B ( n=28 )
Google Translate ( n=14 )
SIREN category
n
%
n
%
Directive force weakened
23
82.14
4
28.57
Directive force strengthened
0
0.00
0
0.00
Protective action compromised
4
14.29
9
64.29
Urgency marker omitted
1
3.57
1
7.14
Total
28
100.00
14
100.00
Table 5: Distribution of SIREN categories among translations for which actionability was not preserved (NO). Percentages are calculated within the NO cases of each system.
TranslateGemma-4B ( n=28 )
Google MT ( n=14 )
Impact
Score
n
%
n
%
Minor
1
15
53.57
4
28.57
Medium
2
10
35.71
2
14.29
Major
3
3
10.71
8
57.14
Total
–
28
100.00
14
100.00
Table 6: Distribution of actionability-impact scores among translations for which actionability was not preserved (NO). Scores 1, 2, and 3 correspond to Minor, Medium, and Major impact, respectively.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: SIREN-span annotation decision tree. Annotators first determine whether an existing translation error compromises actionability. For actionability-compromising errors, they identify the functional rationale and subsequently assign an actionability impact level ( Minor , Medium , or Major ).
System
Source
MT Output
SIREN Rationale
Impact
ChatGPT-4o
Non transitare lungo strade allagate: potrebbero esserci voragini, buche, tombini aperti o cavi elettrici tranciati.
Do not walk along flooded streets: there may be voids, holes, open manholes, or cut electrical wires.
Protective action compromised
Major
ChatGPT-4o
Prima di uscire di casa chiudi gas, acqua e luce e indossa le scarpe.
Before leaving your home, turn off the gas, water, and electricity, and put on sturdy shoes .
Directive force strengthened
Medium
Google Translate
Usa il telefono solo per reale necessità.
Use your phone only for real needs .
Directive force weakened
Medium
Google Translate
Informati sulle vie di allontanamento da seguire previste dalle pianificazioni comunale e nazionale.
Find out about the distancing routes to follow provided for by municipal and national plans.
Protective action compromised
Medium
Appendix
Table 7: Representative SIREN-span annotations from the Italian–English experiment. Boldface highlights translation choices relevant to the actionability judgement.
System
Source
MT Output
SIREN Rationale
Impact
TranslateGemma-4B
Inform yourself about your municipality’s Civil Protection plan.
Vous informer sur le plan de protection civile de votre commune.
Directive force weakened
Minor
TranslateGemma-4B
As a general indication, indoor sheltering may last up to 48 hours.
Pour information générale, la recherche d’abri en intérieur peut durer jusqu’à 48 heures.
Protective action compromised
Major
Google Translate
If you live in the intervention area, check that your home is safe and take all necessary steps to eliminate situations that can represent a danger in case of an earthquake.
Si vous habitez dans la zone d’intervention, vérifiez la sécurité de votre logement et prenez toutes les dispositions nécessaires pour éliminer les situations pouvant présenter un danger en cas de séisme.
Protective action compromised
Medium
Google Translate
Don’t use the car, it could become a trap.
N’utilisez pas votre voiture ; elle pourrait se transformer en piège .
Directive force weakened
Minor
Appendix
Table 8: Representative SIREN-msg annotations from the English–French experiment. Examples are balanced across TranslateGemma-4B and Google Translate. Boldface highlights translation choices relevant to the actionability judgement.
Batch
System
N
Agreement
κ
1
ChatGPT-4o
14
57.14%
−0.024
Google Translate
9
88.89%
0.769
2
ChatGPT-4o
30
90.00%
0.615
Google Translate
30
76.67%
0.516
3
ChatGPT-4o
35
85.71%
0.684
Google Translate
33
90.91%
0.812
Appendix
Table 9: Inter-annotator agreement across the three pilot annotation batches for the Italian–English and English–French experiments.
ChatGPT-4o
Google Translate
Code
HOPE error type
N
Yes (%)
N
Yes (%)
IMP
Impact
16
5 (31.25)
18
11 (61.11)
TRM
Terminology
40
12 (30.00)
33
17 (51.52)
UGR
Ungrammatical
4
1 (25.00)
12
5 (41.67)
MIS
Mistranslation
23
9 (39.13)
29
14 (48.28)
STL
Style
84
25 (29.76)
87
38 (43.68)
Appendix
Table 10: Actionability-compromising errors by HOPE error type in the Italian–English rollout. N indicates the total number of errors in each category. Yes indicates the number of errors judged to compromise actionability, with the percentage relative to N in parentheses.
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.