Design and evaluation of an agentic workflow for crisis-related synthetic tweet datasets

Roben Delos Reyes; Timothy Douglas; Asanobu Kitamoto

cs.CLMar 13, 2026

Design and evaluation of an agentic workflow for crisis-related synthetic tweet datasets

Authors: Roben Delos Reyes, Timothy Douglas, Asanobu Kitamoto

Organizations: School of Computing and Information Systems, The University of Melbourne, Parkville, Victoria, Australia. · National Institute of Informatics, Tokyo, Japan. · Department of Computer Science, University College London, London, United Kingdom.

Abstract

Twitter (now X) has become an important source of social media data for situational awareness during crises. Crisis informatics research has widely used tweets from Twitter to develop and evaluate artificial intelligence (AI) systems for various crisis-relevant tasks, such as extracting locations and estimating damage levels from tweets to support damage assessment. However, recent changes in Twitter's data access policies have made it increasingly difficult to curate real-world tweet datasets related to crises. Moreover, existing curated tweet datasets are limited to past crisis events in specific contexts and are costly to annotate at scale. These limitations constrain the development and evaluation of AI systems used in crisis informatics. To address these limitations, we introduce an agentic workflow for generating crisis-related synthetic tweet datasets. The workflow iteratively generates synthetic tweets conditioned on prespecified target characteristics, evaluates them using predefined compliance checks, and incorporates structured feedback to refine them in subsequent iterations. As a case study, we apply the workflow to generate synthetic tweet datasets relevant to post-earthquake damage assessment. We show that the workflow can generate synthetic tweets that capture their target labels for location and damage level. We further demonstrate that the resulting synthetic tweet datasets can be used to evaluate AI systems on damage assessment tasks like geolocalization and damage level prediction. Our results indicate that the workflow offers a flexible and scalable alternative to real-world tweet data curation, enabling the systematic generation of synthetic social media data across diverse crisis events, societal contexts, and crisis informatics applications.

Figures & tables

Figure 1: Schematic overview of the agentic workflow for generating crisis-related synthetic tweet datasets. The workflow includes three agents: (1) a tweet generator

g

that generates synthetic tweets, (2) a compliance evaluator

e

that evaluates those tweets, and (3) a feedback augmenter

a

that provides feedback to the tweet generator

g

based on the evaluations of the compliance evaluator

e

. The workflow iterates for

n

rounds, with synthetic tweets that gets accepted by the compliance evaluator

e

at each round added to the synthetic tweet dataset

\mathcal{D}_{syn}

Compliance result	Compliance message
Incorrect location ( $c_{loc}=0$ )	Location “{location_text}” not found in tweet
Incorrect damage level ( $c_{dmg}=0$ )	Damage mismatch: expected “{target_damage_level}”, predicted “{kNN_damage_level}”
Lack diversity ( $c_{div}=0$ )	Too similar to accepted corpus (Self-BLEU={bleu_score} $>$ {bleu_threshold})

Table 1: Compliance results and their corresponding messages, which are included in the feedback text

f

provided by the feedback augmenter

a

Earthquake event	ShakeMap intensity	DYFI response	Fatality count	Economic loss	Damage index
2014 Napa, California	VIII	VIII	Green	Red	1.95
2014 Iquique, Chile	VIII	IX	Yellow	Yellow	2.12
2015 Nepal	IX	VIII	Red	Red	2.89
2019 Ridgecrest, California	IX	VIII	Green	Yellow	1.34
2021 Fukushima, Japan	VII	IX	Yellow	Orange	1.47
2021 Haiti	IX	IX	Red	Red	2.95

Table 2: Characteristics of the six real-world earthquake events considered in the experiments, spanning different years and geographic locations. Impacts of the earthquake events are described using ShakeMap intensities, Did You Feel It (DYFI) survey responses, estimated fatalities and economic losses, and Twitter-derived damage indices based on a four-level damage scale, as summarised in [ 9 ] .

Figure 2: Percentage of synthetic tweets that passed compliance checks for location correctness, damage level correctness, and textual diversity. In the initial generation attempt, at least 30% of synthetic tweets passed each compliance check. This percentage improved with every additional feedback round.

Figure 3: Distribution of synthetic tweets by the number of compliance checks passed across the six earthquake events. The percentage of accepted synthetic tweets increases with more feedback rounds.

Figure 4: Percentage of tweets that passed the compliance checks when using different temperature $\tau$ values. Higher temperature values generated more diverse synthetic tweets, resulting in more synthetic tweets that passed the textual diversity check and were eventually accepted and added to the synthetic tweet dataset. Results shown are based on the 2014 Napa, California earthquake event.

Figure 5: Tweet length distributions and relative hashtag frequencies in the real and synthetic tweet datasets. The synthetic tweets reflect common characteristics of real tweets, such as character limits and the use of common hashtags during crises (e.g., #earthquake, #napa, #iquique, #nepal, #ridgecrest, #fukushima, #haiti).

Event	Size	Unique	0	1	2	3
		Location	Damage level
2014 Napa	6,819	789 (12)	3,841 (56)	2,386 (35)	592 (9)	0 (0)
2014 Iquique	2,646	446 (17)	2,238 (85)	229 (9)	179 (7)	0 (0)
2015 Nepal	31,709	3,476 (11)	9,581 (30)	7,156 (23)	12,114 (38)	2,858 (9)
2019 Ridgecrest	58,495	5,325 (9)	50,859 (87)	6,600 (11)	1,019 (2)	17 ( $<$ 1)
2021 Fukushima	2,547	340 (13)	1,612 (63)	807 (32)	128 (5)	0 (0)

Table 3: Characteristics of the synthetic tweet datasets, including dataset size, unique location counts, and frequency distribution of damage levels. The datasets comprise synthetic tweets produced by the workflow with three feedback rounds (

n=3

) and a temperature

\tau=1.4

Earthquake event	2014 Napa	2014 Iquique	2015 Nepal	2019 Ridgecrest	2021 Fukushima	2021 Haiti
Geolocalization
Percentage of geolocated tweets
Real	94.41	94.33	95.69	91.36	99.14	97.36
Synthetic	95.76	72.56	89.83	85.71	93.95	85.44
Accuracy relative to geolocated tweets
Real	64.59	32.85	73.63	76.48	75.37	74.21

Table 4: Evaluation of pretrained AI systems on geolocalization and damage level prediction tasks using the real and synthetic tweet datasets. The synthetic tweet datasets enabled the AI systems to identify locations and predict damage levels similarly to the real tweet datasets. AI system performance on the two tasks, measured by percentage of valid outputs and accuracy of these outputs relative to their target labels, was comparable when using the real and synthetic tweet datasets.

round	$c_{loc}$	$c_{dmg}$	$c_{div}$	feedback text $f$
0	0	1	1	Generated tweet: Just saw a tremor downtown. Looks like someone dropped something HUGE. #SanFrancisco; Location ‘San Francisco’ not found in tweet
1	1	0	1	Generated tweet: Downtown San Francisco is crumbling after the quake! Massive structural damage reported, with officials saying damage is minimal. #SanFrancisco #Earthquake; Damage mismatch: expected ‘0’, predicted ‘1’
2	1	0	1	Generated tweet: Downtown San Francisco is showing significant cracks and deformation. Emergency services are responding to the situation. #SanFrancisco #Earthquake; Damage mismatch: expected ‘0’, predicted ‘1’
3	1	0	0	Generated tweet: Downtown San Francisco is cracking with damage. Authorities confirm structural damage, with significant visible cracks. #SanFrancisco #Earthquake; Damage level: 1; Damage mismatch: expected ‘0’, predicted ‘1’; Too similar to accepted corpus (Self-BLEU=47.9 $>$ 40.0)

Table S1: Examples of the feedback text

f

, which includes corresponding compliance messages based on the compliance results for location correctness (

c_{loc}

), damage level correctness (

c_{dmg}

), and textual diversity (

c_{div}

). These synthetic tweets were generated using the gemma-3-1b-it model with temperature

\tau=1.4

, given the target location

y_{loc}=San

Francisco

and target damage level

y_{dmg}=0

Figure S1: Distribution of synthetic tweets by the number of compliance checks passed when using different LLMs (Google’s gemma-3-1b-it model, Qwen’s Qwen-3-0.6B model, and Meta’s Llama-3.2-1B-Instruct model) as the tweet generator $g$ . Results shown are based on the 2014 Napa, California earthquake event.

Figure S2: Percentage of tweets that passed the compliance checks when using different temperature $\tau$ values across the six earthquake events.

Synthetic Data Generation

Synthetic data generation creates artificial examples, simulations, and datasets to train, test, or protect machine-learning systems.

See all 623 papers

Model in Distress: Sentiment Analysis on French Synthetic Social Media
Apr 20, 2026Pierre-Carl Langlais, Pavel Chizhov, Yannick Detrois +3Synthetic Data Augmentation Synthetic Data Generation
PleIAs, Paris, France · CAIRO, THWS, Würzburg, Germany · EPFL, Lausanne, Switzerland +1
NodeSyn: Socially Aligned Synthetic Data for AI Evaluation
May 14, 2026Qazi Mamunur Rashid, Xuan Yang, Zhengzhe Yang +5Language Model Generation Evaluation Language Model Safety Evaluation
Google Research, USA · Google, USA · Google DeepMind, USA
Reasoning-Driven Synthetic Data Generation and Evaluation
Mar 31, 2026Tim R. Davidson, Benoit Seguin, Enrico Bacis +2Synthetic Data Generation Synthetic Data Evaluation
EPFL · Google · Google Deepmind

Apr 20, 2026J/K move · Enter open · S save

Model in Distress: Sentiment Analysis on French Synthetic Social Media

Pierre-Carl Langlais, Pavel Chizhov, Yannick Detrois, Carlos Rosas-Hinostroza+2

PleIAs, Paris, France · CAIRO, THWS, Würzburg, Germany · EPFL, Lausanne, Switzerland+1

Synthetic Data Augmentation Synthetic Data Generation Sentiment Analysis Synthetic Data Generation for Language Models Emotion Recognition

Open paper AskPDF arXiv

Automated analysis of customer feedback on social media is hindered by three challenges: the high cost of annotated training data, the scarcity of evaluation sets, especially in multilingual settings, and privacy concerns that prevent data sharing and reproducibility. We address these issues by developing a generalizable synthetic data generation pipeline applied to a case study on customer distress detection in French public transportation. Our approach utilizes backtranslation with fine-tuned models to generate 1.7 million synthetic tweets from a small seed corpus, complemented by synthetic reasoning traces. We train 600M-parameter reasoners with English and French reasoning that achieve 77-79% accuracy on human-annotated evaluation data, matching or exceeding SOTA proprietary LLMs and specialized encoders. Beyond reducing annotation costs, our pipeline preserves privacy by eliminating the exposure of sensitive user data. Our methodology can be adopted for other use cases and languages.

Abstract

Synthetic Data Generation

Explore similar work

Model in Distress: Sentiment Analysis on French Synthetic Social Media

NodeSyn: Socially Aligned Synthetic Data for AI Evaluation

Reasoning-Driven Synthetic Data Generation and Evaluation