Communicating with the public during an emergency entails issuing warning messages promptly. In Europe, it often requires producing them in more than one language. Recent work has shown that language models can be trained to generate warning messages as a crisis unfolds, but the available data covers only English. To fill this gap, we present CrisiLangs, a multilingual extension of CrisiText (an English dataset of crisis events paired with warning messages written according to expert communication guidelines) to Italian, Bulgarian, German, and Greek. Expert native speakers translated the warning messages of 1,000 events drawn from 411 scenarios, for a total of 5,000 event-message pairs across five languages. We test the resource to fine-tune a multilingual model on all five languages jointly, with and without the emergency guidelines in the prompt, and compare it with a zero-shot baseline. The fine-tuned models outperform the baseline on ROUGE-L, BLEU and BERTScore in every language and configuration. In a human preference study, low decoding temperatures receive the most preferences in all five languages and in both configurations, while the effect of the guidelines varies by language.
Figures & tables
Figure 1: Workflow for the creation of CrisiLangs . Event descriptions and FEMA guidelines are machine-translated into each target language, while the warning messages are translated and adapted by a native speaker.
Figure 2: Example of event and warning message from CrisiLangs in the five languages. The English event and message are the originals from CrisiText . In the other four languages the messages are human translations produced by native speakers, while the events are machine-translated.
Category
Scenarios
Events
Human-caused
253
500
explosion
105
201
fire
76
153
attack
72
146
Natural
158
500
tornado
33
111
Table 1: Distribution of the event and message main categories over the 1,000 selected elements, grouped into human-caused and natural crises.
All
EN
IT
DE
EL
BG
306.9
268.6
316.4
324.1
331.1
294.4
Table 2: Mean warning message length in characters per language in CrisiLangs .
User
† {Based on the provided guidelines,} Create a warning message informing on the current happening, providing a suggestion, for the last line in the following chain of events (be short, max 300 characters). No other output other than the message.
† Guidelines: [GUIDELINES]
Chain of events: [EVENTS]
Figure 3: Prompt example. The instruction is always in English, while [EVENTS] is replaced by the chain of events of the instance, in its language, with the target event as its last line. The segments marked with † appear only in the +guidelines configuration, where [GUIDELINES] is replaced by the translated FEMA guidelines of the crisis categories of the event.
All
English
Italian
German
Greek
Bulgarian
T
R-L
B
BS
R-L
B
BS
R-L
B
BS
R-L
B
BS
R-L
B
BS
R-L
B
BS
Zero-shot
plain
1.0
.168
.037
.878
.209
.072
.890
.175
.041
.882
.185
.038
.884
.130
.019
.865
.143
.016
.871
0.8
.184
.043
.884
.232
.083
.893
.196
.045
.886
.202
.047
.888
.145
.021
.874
.148
.021
.878
0.6
.201
.048
.888
.234
.084
.895
.215
.049
.891
.218
.055
.892
.176
.032
.883
.166
.021
.883
0.4
.205
.051
.890
.232
.083
.894
.214
.053
.890
.226
.050
.894
.174
.036
.887
.181
.035
.887
0.2
.211
.056
.892
.240
.090
.894
.221
.056
.892
.233
.059
.896
.182
.039
.888
.182
.035
.889
Table 3: ROUGE-L F1 (R-L), BLEU (B) and BERTScore F1 (BS) on the test split, per configuration, temperature and language. Bold marks the best temperature within each block. Zero-shot uses the un-finetuned Llama-3.1-8B-Instruct with identical configuration.
Length
BLEU w/o BP
Gold
304
–
ZS, plain
173
.114
ZS, +guid.
205
.104
FT, plain
295
.177
FT, +guid.
299
.171
Table 4: Mean generation length in characters and BLEU without the brevity penalty (BP), at T=0.2 , over all five languages, for the zero-shot (ZS) and fine-tuned (FT) models. Per-language values are in Appendix B .
plain
+guidelines
0.6
0.4
0.2
N
0.6
0.4
0.2
N
English
8
15
27
50
12
8
30
50
Italian
6
15
29
50
7
15
27
49
German
12
13
24
49
7
17
26
50
Greek
11
2
37
50
4
22
24
50
Bulgarian
4
17
29
50
3
9
25
37
Table 5: Preference counts per configuration and language. N is the number of annotated events out of 50.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
All
EN
IT
BG
DE
EL
mean length (characters)
Gold
304
267
315
289
317
330
ZS, plain
173
188
163
163
182
168
ZS, +guid.
205
204
214
192
221
195
FT, plain
295
269
297
272
326
309
FT, +guid.
299
270
312
284
317
314
Appendix
Table 6: Per-language mean generation length in characters and BLEU without the brevity penalty, at T=0.2 , for the zero-shot (ZS) and fine-tuned (FT) models with the plain and +guidelines (+guid.) prompts. As in Table 3 , the comparison is intra-language. Languages are English (EN), Italian (IT), Bulgarian (BG), German (DE) and Greek (EL).
Translation task
The task consists in creating alert messages in your language based on the English alert messages we provide. We ask you to translate the messages and, when necessary, adapt them to sound more natural and appropriate as alert messages in your language.
Preference task
Your task is to:
– For each row, select your preferred message using the dropdown menu, based on grammaticality and consistency with the task.
– At the top of each sheet, indicate which configuration (column) you preferred overall, and briefly explain why.
– Let us know which of the two models (sheets) you preferred overall, and why.
Appendix
Figure 4: Instructions given to the annotators, reproduced verbatim. The translation instructions were provided together with the English source; the preference instructions accompanied one spreadsheet per training configuration.
Timely and reliable multilingual communication is critical during natural and human-induced disasters, but developing effective solutions for crisis communication is limited by the scarcity of curated parallel data. We propose a domain-adaptive pipeline that expands a small reference corpus, by retrieving and filtering data from general corpora. We use the resulting dataset to fine-tune a small language model for crisis-domain translation and then apply preference optimization to bias outputs toward CEFR A2-level English. Automatic and human evaluation shows that this approach improves readability, while maintaining strong adequacy. Our results indicate that simplified English, combined with domain adaptation, can function as a practical lingua franca for emergency communication when full multilingual coverage is not feasible.
Antonio Castaldo, Maria Carmen Staiano, Johanna Monti +2
University of Pisa · University of Naples “L’Orientale” · University of Macerata +1
Safety alignment in multilingual models is uneven: a model that reliably refuses a harmful request in English will often comply with the same request in a lower-resource language. We trace this gap mechanistically in sarvam, an Indic-multilingual mixture-of-experts reasoning model, and find it is not a failure to detect harm. Harm is encoded as an internal direction that is nearly language-invariant in mid-network (English-vs-Indic cosine ≈0.9 at L11), and steering that direction upstream causally controls refusal. But the detection direction is orthogonal to the change that actually writes the refusal, which is late and assembled over the course of generation rather than read off in a single forward pass. We attribute the write to a specific, localizable circuit, a mixture-of-experts writer held in check by an attention opposer and price every way of intervening on it: damping the opposer is cheap and effective, amplifying the writer is a cost wall, and surgical edits to the responsible heads do nothing. The circuit's organization, and the gradient method that exposes it, recur in a second, unrelated MoE model, while the lever's strength is architecture-specific. The result is a cost-measured map of where a multilingual safety repair can land, and what it costs
Language models have been adopted faster than evaluation practices that reliably assess their risks, and a growing share of use is affective rather than informational. Within that traffic users disclose suicide and self-harm (SSH) crises. Existing safety benchmarks score models primarily on refusal, but refusal is not the desired behavior here; the conversation requires a substantive supportive response. We introduce DistressBench, 718 English conversations with clinician-authored user turns spanning 23 SSH subcategories, each paired with a task-specific rubric of weighted atomic criteria across seven dimensions of crisis support and a clinician reference response. A model's score is the weighted average over the criteria its reply satisfies, reading directly as the fraction of required care delivered. The best of 25 frontier models satisfies 88.4% of weighted criteria and the median model 71.4%, against 98.7% for clinician references scored on the same criteria. Pooled across all 25 models, in 35.3% of conversations where a model satisfied every criterion for recognizing the crisis, it failed to route the user toward human help. The shortfall concentrates in the dimensions requiring the model to act rather than withhold. Contrasting each dimension's median model against its best separates capability limits from choices: de-escalation reaches 74.6% in the strongest model against a median of 46.0%, and safety disclaimers 84.6% against 8.3%. Multi-turn conversations score below single-turn ones (68.5% against 74.2%). We release the corpus, rubrics, judge calibration labels, and a companion annex of 20 long-horizon red-team transcripts. These results argue for evaluating crisis safety on the care a model delivers rather than on what it declines to say, and for treating de-escalation and resource routing as deployment requirements.