Organizations: School of Computing and Information Systems, The University of Melbourne, Parkville, Victoria, Australia. · National Institute of Informatics, Tokyo, Japan. · Department of Computer Science, University College London, London, United Kingdom.
Twitter (now X) has become an important source of social media data for situational awareness during crises. Crisis informatics research has widely used tweets from Twitter to develop and evaluate artificial intelligence (AI) systems for various crisis-relevant tasks, such as extracting locations and estimating damage levels from tweets to support damage assessment. However, recent changes in Twitter's data access policies have made it increasingly difficult to curate real-world tweet datasets related to crises. Moreover, existing curated tweet datasets are limited to past crisis events in specific contexts and are costly to annotate at scale. These limitations constrain the development and evaluation of AI systems used in crisis informatics. To address these limitations, we introduce an agentic workflow for generating crisis-related synthetic tweet datasets. The workflow iteratively generates synthetic tweets conditioned on prespecified target characteristics, evaluates them using predefined compliance checks, and incorporates structured feedback to refine them in subsequent iterations. As a case study, we apply the workflow to generate synthetic tweet datasets relevant to post-earthquake damage assessment. We show that the workflow can generate synthetic tweets that capture their target labels for location and damage level. We further demonstrate that the resulting synthetic tweet datasets can be used to evaluate AI systems on damage assessment tasks like geolocalization and damage level prediction. Our results indicate that the workflow offers a flexible and scalable alternative to real-world tweet data curation, enabling the systematic generation of synthetic social media data across diverse crisis events, societal contexts, and crisis informatics applications.
Figures & tables
Figure 1: Schematic overview of the agentic workflow for generating crisis-related synthetic tweet datasets. The workflow includes three agents: (1) a tweet generator g that generates synthetic tweets, (2) a compliance evaluator e that evaluates those tweets, and (3) a feedback augmenter a that provides feedback to the tweet generator g based on the evaluations of the compliance evaluator e . The workflow iterates for n rounds, with synthetic tweets that gets accepted by the compliance evaluator e at each round added to the synthetic tweet dataset Dsyn .
Too similar to accepted corpus (Self-BLEU={bleu_score} > {bleu_threshold})
Table 1: Compliance results and their corresponding messages, which are included in the feedback text f provided by the feedback augmenter a .
Earthquake event
ShakeMap intensity
DYFI response
Fatality count
Economic loss
Damage index
2014 Napa, California
VIII
VIII
Green
Red
1.95
2014 Iquique, Chile
VIII
IX
Yellow
Yellow
2.12
2015 Nepal
IX
VIII
Red
Red
2.89
2019 Ridgecrest, California
IX
VIII
Green
Yellow
1.34
2021 Fukushima, Japan
VII
IX
Yellow
Orange
1.47
2021 Haiti
IX
IX
Red
Red
2.95
Table 2: Characteristics of the six real-world earthquake events considered in the experiments, spanning different years and geographic locations. Impacts of the earthquake events are described using ShakeMap intensities, Did You Feel It (DYFI) survey responses, estimated fatalities and economic losses, and Twitter-derived damage indices based on a four-level damage scale, as summarised in [ 9 ] .
Figure 2: Percentage of synthetic tweets that passed compliance checks for location correctness, damage level correctness, and textual diversity. In the initial generation attempt, at least 30% of synthetic tweets passed each compliance check. This percentage improved with every additional feedback round.
Figure 3: Distribution of synthetic tweets by the number of compliance checks passed across the six earthquake events. The percentage of accepted synthetic tweets increases with more feedback rounds.
Figure 4: Percentage of tweets that passed the compliance checks when using different temperature τ values. Higher temperature values generated more diverse synthetic tweets, resulting in more synthetic tweets that passed the textual diversity check and were eventually accepted and added to the synthetic tweet dataset. Results shown are based on the 2014 Napa, California earthquake event.
Figure 5: Tweet length distributions and relative hashtag frequencies in the real and synthetic tweet datasets. The synthetic tweets reflect common characteristics of real tweets, such as character limits and the use of common hashtags during crises (e.g., #earthquake, #napa, #iquique, #nepal, #ridgecrest, #fukushima, #haiti).
Location
Damage level
Event
Size
Unique
0
1
2
3
2014 Napa
6,819
789 (12)
3,841 (56)
2,386 (35)
592 (9)
0 (0)
2014 Iquique
2,646
446 (17)
2,238 (85)
229 (9)
179 (7)
0 (0)
2015 Nepal
31,709
3,476 (11)
9,581 (30)
7,156 (23)
12,114 (38)
2,858 (9)
2019 Ridgecrest
58,495
5,325 (9)
50,859 (87)
6,600 (11)
1,019 (2)
17 ( < 1)
2021 Fukushima
2,547
340 (13)
1,612 (63)
807 (32)
128 (5)
0 (0)
Table 3: Characteristics of the synthetic tweet datasets, including dataset size, unique location counts, and frequency distribution of damage levels. The datasets comprise synthetic tweets produced by the workflow with three feedback rounds ( n=3 ) and a temperature τ=1.4 .
Earthquake event
2014 Napa
2014 Iquique
2015 Nepal
2019 Ridgecrest
2021 Fukushima
2021 Haiti
Geolocalization
Percentage of geolocated tweets
Real
94.41
94.33
95.69
91.36
99.14
97.36
Synthetic
95.76
72.56
89.83
85.71
93.95
85.44
Accuracy relative to geolocated tweets
Real
64.59
32.85
73.63
76.48
75.37
74.21
Table 4: Evaluation of pretrained AI systems on geolocalization and damage level prediction tasks using the real and synthetic tweet datasets. The synthetic tweet datasets enabled the AI systems to identify locations and predict damage levels similarly to the real tweet datasets. AI system performance on the two tasks, measured by percentage of valid outputs and accuracy of these outputs relative to their target labels, was comparable when using the real and synthetic tweet datasets.
round
cloc
cdmg
cdiv
feedback text f
0
0
1
1
Generated tweet: Just saw a tremor downtown. Looks like someone dropped something HUGE. #SanFrancisco; Location ‘San Francisco’ not found in tweet
1
1
0
1
Generated tweet: Downtown San Francisco is crumbling after the quake! Massive structural damage reported, with officials saying damage is minimal. #SanFrancisco #Earthquake; Damage mismatch: expected ‘0’, predicted ‘1’
2
1
0
1
Generated tweet: Downtown San Francisco is showing significant cracks and deformation. Emergency services are responding to the situation. #SanFrancisco #Earthquake; Damage mismatch: expected ‘0’, predicted ‘1’
3
1
0
0
Generated tweet: Downtown San Francisco is cracking with damage. Authorities confirm structural damage, with significant visible cracks. #SanFrancisco #Earthquake; Damage level: 1; Damage mismatch: expected ‘0’, predicted ‘1’; Too similar to accepted corpus (Self-BLEU=47.9 > 40.0)
Table S1: Examples of the feedback text f , which includes corresponding compliance messages based on the compliance results for location correctness ( cloc ), damage level correctness ( cdmg ), and textual diversity ( cdiv ). These synthetic tweets were generated using the gemma-3-1b-it model with temperature τ=1.4 , given the target location yloc=SanFrancisco and target damage level ydmg=0 .
Figure S1: Distribution of synthetic tweets by the number of compliance checks passed when using different LLMs (Google’s gemma-3-1b-it model, Qwen’s Qwen-3-0.6B model, and Meta’s Llama-3.2-1B-Instruct model) as the tweet generator g . Results shown are based on the 2014 Napa, California earthquake event.
Figure S2: Percentage of tweets that passed the compliance checks when using different temperature τ values across the six earthquake events.
Automated analysis of customer feedback on social media is hindered by three challenges: the high cost of annotated training data, the scarcity of evaluation sets, especially in multilingual settings, and privacy concerns that prevent data sharing and reproducibility. We address these issues by developing a generalizable synthetic data generation pipeline applied to a case study on customer distress detection in French public transportation. Our approach utilizes backtranslation with fine-tuned models to generate 1.7 million synthetic tweets from a small seed corpus, complemented by synthetic reasoning traces. We train 600M-parameter reasoners with English and French reasoning that achieve 77-79% accuracy on human-annotated evaluation data, matching or exceeding SOTA proprietary LLMs and specialized encoders. Beyond reducing annotation costs, our pipeline preserves privacy by eliminating the exposure of sensitive user data. Our methodology can be adopted for other use cases and languages.
Pierre-Carl Langlais, Pavel Chizhov, Yannick Detrois +3
Recent advancements in generative AI facilitate large-scale synthetic data generation for model evaluation. However, without targeted approaches, these datasets often lack the sociotechnical nuance required for sensitive domains. We introduce NodeSyn, a methodology that pairs a fine-tuned taxonomy generator (TaG) with credible published sources and extracted sensitive attributes to generate queries reflecting authentic societal contexts and nuanced harms. Evaluated across ten open-weight and proprietary LLMs in Medical Advice and Self-Harm using human-validated autoraters, NodeSyn ranked first on 8 of 10 models in each domain. NodeSyn datasets induced average failure rates 3.1 times higher than generic synthetic baselines, 2.4 times higher than structured synthetic benchmarks, and 1.6 times higher than authentic human-authored queries. Ablation studies confirm that, controlling for structural confounds, our granular Level 3 taxonomic expansion significantly drives these failure rates over Level 2 across 9 of 10 models (+29.2% relative lift in Medical Advice and +24.0% in Self-Harm). Variance attribution analysis reveals that failures are driven not only by what topics are queried (taxonomic nodes and lexical specificity), but also by their sociotechnical grounding (affected demographics, occupations, geographies, and querying personas). We embed this method into a research prototype, and open-source (https://github.com/google-research/nodesynth) the code and datasets to enable scalable, high-stakes model evaluation.
Qazi Mamunur Rashid, Xuan Yang, Zhengzhe Yang +5
Google Research, USA · Google, USA · Google DeepMind, USA
Although many AI applications of interest require specialized multi-modal models, relevant data to train such models is inherently scarce or inaccessible. Filling these gaps with human annotators is prohibitively expensive, error-prone, and time-consuming, leading model builders to increasingly consider synthetic data as a scalable alternative. However, existing synthetic data generation methods often rely on manual prompts, evolutionary algorithms, or extensive seed data from the target distribution - limiting their scalability, explainability, and control. In this paper, we introduce Simula: a novel reasoning-driven framework for data generation and evaluation. It employs a seedless, agentic approach to generate synthetic datasets at scale, allowing users to define desired dataset characteristics through an explainable and controllable process that enables fine-grained resource allocation. We show the efficacy of our approach on a variety of datasets, rigorously testing both intrinsic and downstream properties. Our work (1) offers guidelines for synthetic data mechanism design, (2) provides insights into generating and evaluating synthetic data at scale, and (3) unlocks new opportunities for developing and deploying AI in domains where data scarcity or privacy concerns are paramount.