Reasoning-Driven Synthetic Data Generation and Evaluation

Tim R. Davidson; Benoit Seguin; Enrico Bacis; Cesar Ilharco; Hamza Harkous

cs.AIMar 31, 2026

Reasoning-Driven Synthetic Data Generation and Evaluation

Authors: Tim R. Davidson, Benoit Seguin, Enrico Bacis, Cesar Ilharco, Hamza Harkous

Organizations: EPFL · Google · Google Deepmind

Abstract

Although many AI applications of interest require specialized multi-modal models, relevant data to train such models is inherently scarce or inaccessible. Filling these gaps with human annotators is prohibitively expensive, error-prone, and time-consuming, leading model builders to increasingly consider synthetic data as a scalable alternative. However, existing synthetic data generation methods often rely on manual prompts, evolutionary algorithms, or extensive seed data from the target distribution - limiting their scalability, explainability, and control. In this paper, we introduce Simula: a novel reasoning-driven framework for data generation and evaluation. It employs a seedless, agentic approach to generate synthetic datasets at scale, allowing users to define desired dataset characteristics through an explainable and controllable process that enables fine-grained resource allocation. We show the efficacy of our approach on a variety of datasets, rigorously testing both intrinsic and downstream properties. Our work (1) offers guidelines for synthetic data mechanism design, (2) provides insights into generating and evaluating synthetic data at scale, and (3) unlocks new opportunities for developing and deploying AI in domains where data scarcity or privacy concerns are paramount.

Figures & tables

Figure 1: Synthetic Coverage Examples. The outer squares of (a-b-c) represent a semantic factor of interest, e.g., “ cat type .” (a) Characterizes “random” sampling behavior, with no notion of a global coverage space. This often results in samples clustered around semantic modes and misses edge cases. The grid-like structures in (b-c) represent discrete semantic spaces defined by a taxonomy’s leaf nodes at increasing levels of granularity. For example, the first level could represent “ cat type ” broken down into “ domestic, big wild cats, small wild cats , and feral ”, whereas a square at a lower level might represent a specific cat breed like the “ British shorthair .” (b) Represents perfect global planning at increasing granularity; and (c) shows global planning with progressive coverage loss, e.g., missing the branch “ big wild cats ” entirely (bottom left) or missing specific breeds (bottom right).

Figure 2: Schematic of Simula Framework. Given user instructions $y$ and/or a data sample $\mathcal{S}$ , we first (a) determine factors of interest $f_{i}$ , which (b) are expanded into taxonomies $\mathcal{T}_{i}$ . Next, (c) nodes of $\mathcal{T}_{i}$ are sampled to obtain mixes, and (d) turned into “meta prompts”. A user-defined fraction, $c$ , of meta prompts is “complexified.” (e) Finally meta prompts are used to generate data proposals by a Generator , and iteratively refined using a Critic step.

	Taxonomy Depth	Meta Prompting	Critic Steps
Baseline	1	$\times$	$\times$
Local	1	✓	$\times$
Global	>1	$\times$	$\times$
Local + Global	>1	✓	$\times$
Local + Global + Critique (i.e., Full System)	>1	✓	✓

Table 1: System Versions. We evaluate the efficacy and additive properties of different system components.

Panel of Figure 3 — Figure 3: Double-Critic Rejection Sampling on MATH. (a) We establish the theoretical lift in the controlled setting, noting that rejection costs increase with task complexity. (b) Critic capabilities transport to the empirical setting but lose some effectiveness. (c) Calibrated Elo scores show model-human alignment. Stratified by complexity level, rejected samples are consistently assigned higher model complexity.

Method	Grounded		Conceptual
Method	Completeness	Soundness	Completeness	Soundness	Novelty	Coverage
Simula	0.74	0.75	0.78	0.97	0.94	1.72
0-Shot	0.52	0.70	0.50	0.97	0.32	0.83

Table 2: Taxonomy evaluation. Average metrics across taxonomies. Novelty and total coverage are only reported for Conceptual taxonomies, since we assume that grounded taxonomies should have no room for further extensions.

Panel of Figure 4 — Figure 4: Intrinsic Diversity Metrics. We display dataset-wide (top) and nearest-neighbors (middle) embedding-based diversity, and taxonomic coverage (bottom). We note that Global diversification is crucial for increasing dataset-wide diversity, and that Local and Global diversification generally have an additive effect. We further note that while real data can be more or less diverse according to embedding-based metrics, it almost always covers less of the target domain than Simula variants on a taxonomy basis.

Figure 5: Complexity Elo Distribution of Synthetic and Real Data. We display density plots of the complexity Elo rankings for the various system versions on four datasets. We note that synthetic data can cover the entire complexity range of all datasets and that Local and Global components are generally additive, i.e., they account for different types of complexity.

Figure 6: Downstream Performance on Different Datasets. We report mean accuracy with 95% CI for different system versions. We note that while the contribution of different components to performance depends on the dataset and data size, the full Simula system always reaches the best result.

Figure 7: Downstream Impact of Data Complexity on Mean Accuracy. For GSM8k and CTI MCQ, higher complexity data improves performance and scaling. Conversely, for LEXam, only the Low Complexity split shows improvement, suggesting that complex data can be detrimental when the teacher model is weak.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Topic	Method	Completeness	Soundness	Novelty	Coverage
Conceptual	Online Harmful Content	Simula	0.749	0.980	0.865	1.614
	Online Harmful Content	0-shot	0.588	0.957	0.412	1.000
	Logical Fallacies	Simula	0.726	0.919	1.679	2.405
	Logical Fallacies	0-shot	0.458	0.966	0.193	0.651
	Risks of Language Models	Simula	0.867	1.000	0.267	1.134
	Risks of Language Models	0-shot	0.467	1.000	0.367	0.834

Appendix

Table 3: Performance metrics for different taxonomies.

Panel of Figure 8 — Figure 8: Comparison of Online Harmful Content Taxonomy (Simula vs. Expert).

Panel of Figure 9 — Figure 9: Comparison of Chemical Element Taxonomy (Simula vs. Expert).

Complexity	Critic	$\mu_{\text{gen}}$	Elo	$\|\mathcal{D}\|$	$\mu_{\text{gen}}$	Elo	$\|\mathcal{D}\|$	$\mu_{\text{gen}}$	Elo	$\|\mathcal{D}\|$
		English (High)			Korean (Mid)			Nepali (Low)
Level 1	$\times$	0.93 $\pm 0.07$	290 $\pm 8$	14	0.86 $\pm 0.07$	306 $\pm 7$	29	0.76 $\pm 0.07$	308 $\pm 7$	41
Level 1	✓	0.98 $\pm 0.01$	303 $\pm 2$	364	0.97 $\pm 0.01$	301 $\pm 2$	349	0.97 $\pm 0.01$	304 $\pm 2$	337
Level 2	$\times$	0.52 $\pm 0.07$	427 $\pm 7$	46	0.64 $\pm 0.06$	428 $\pm 5$	53	0.58 $\pm 0.06$	431 $\pm 6$	60
Level 2	✓	0.94 $\pm 0.01$	427 $\pm 2$	578	0.92 $\pm 0.01$	426 $\pm 2$	571	0.93 $\pm 0.01$	425 $\pm 2$	564
Level 3	$\times$	0.67 $\pm 0.10$	473 $\pm 5$	24	0.54 $\pm 0.10$	471 $\pm 5$	26	0.41 $\pm 0.09$	473 $\pm 4$	34

Appendix

Table 4: Critic Rejection Sampling on Multi-Lingual MMLU Questions. We evaluate our critic-rejection sampling method for MMLU questions on Mathematics, Physics, and Computer Science. We use the subject education level (elementary, high-school, and college) as the ground-truth Complexity. We display the realized change in accuracy,

\mu_{\text{gen}}

of following critic rejections. Also shown are the average Elo complexity score and the size of the rejected and accepted subsets.

Figure 10: Complexity score ablation MATH test set. The MATH dataset comes with ground truth complexity levels ranging from 1-5. We group by the ground truth complexity level and plot the model-assigned complexity scores for each group. We compare using average raw scores (Score) against using Elo rankings (Elo) and ablate the batch size ( BS ) of question scored simultaneously and the number of times each question is scored ( N ). All results are scaled per (BS, N)-grouping to lie between 0 and 100. For

BS=1

, no pairwise rankings are done, so Elo rankings do not apply.

		English (High)			Arabic (High)			Dutch (High)			Korean (Mid)			Nepali (Low)
BS	N	NMI	ARI	AUC	NMI	ARI	AUC	NMI	ARI	AUC	NMI	ARI	AUC	NMI	ARI	AUC
1	1	0.27	0.23	0.61	0.32	0.27	0.60	0.32	0.28	0.59	0.32	0.27	0.59	0.32	0.27	0.56
1	5	0.27	0.24	0.62	0.33	0.28	0.61	0.34	0.30	0.59	0.34	0.29	0.59	0.33	0.29	0.56
5	1	0.36	0.31	0.63	0.38	0.32	0.60	0.37	0.31	0.59	0.38	0.33	0.59	0.37	0.32	0.57
5	5	0.42	0.36	0.64	0.42	0.35	0.62	0.41	0.35	0.59	0.40	0.34	0.60	0.41	0.35	0.57
10	1	0.38	0.33	0.64	0.39	0.34	0.61	0.37	0.32	0.60	0.40	0.35	0.61	0.39	0.33	0.57

Appendix

Table 5: Multi-lingual MMLU Complexity Scoring. We use exam questions for the topics Mathematics, Physics, and Computer Science, on education levels elementary (Mathematics only), high-school, and college. Taking the education level as our ground-truth complexity level (1-3), we run KMeans on the complexity scores generated by the model and compute the Normalized Mutual Information ( NMI ) and the Adjusted Rand Index ( ARI ). Finally, we compute the Area Under the Curve ( AUC ) of using the complexity scores as an estimator for the model’s generative performance. We ablate the batch size ( BS ) of exam questions scored simultaneously and the number of times each exam question is scored ( N ).

Dataset	Learning Rates	LoRA Rank	Batch Size
CTI RCM	{0.005, 0.01}	8	16
CTI MCQ	{0.005, 0.01}	8	16
GSM8K	{0.005, 0.01}	16	16
LEXam	{0.005, 0.01}	16	16
MMLU	{0.005, 0.01}	16	16

Appendix

Table 6: Final hyperparameters used for each dataset. The final learning rate was chosen from the specified set based on validation performance for each training data size.

Figure 11: Downstream Performance on Global MMLU English and Korean. We report mean accuracy with 95% CI for these additional datasets. We continue to see that the full Simula system version with the critic, provides the best performance compared to the other versions.

Figure 12: Downstream Performance using Muon Optimizer. We report mean accuracy with 95% CI for the CTI MCQ dataset for student models fine-tuned using the Muon optimizer. We continue to see that the full Simula system version with the critic, provides the best performance compared to the other versions.

Figure 13: Downstream Performance using Qwen3-Next-80B Instruct. We report mean accuracy with 95% CI for the CTI MCQ dataset for data generated using the Qwen-3 80B instruction-tuned model. We continue to see that the full Simula system version with the critic, provides the best performance compared to the other versions.

Synthetic Data Generation

Synthetic data generation creates artificial examples, simulations, and datasets to train, test, or protect machine-learning systems.

See all 623 papers

NodeSyn: Socially Aligned Synthetic Data for AI Evaluation
May 14, 2026Qazi Mamunur Rashid, Xuan Yang, Zhengzhe Yang +5Language Model Generation Evaluation Language Model Safety Evaluation
Google Research, USA · Google, USA · Google DeepMind, USA
Autodata: An agentic data scientist to create high quality synthetic data
Jun 24, 2026Ilia Kulikov, Chenxi Whitehouse, Tianhao Wu +12Synthetic Data Generation LLM Agent Self-Improvement
FAIR at Meta
Memisis: Orchestrating and Evaluating Synthetic Data for Tabular Health Datasets
May 18, 2026Nitish Nagesh, Pengbao Zhou, Atchuth Naveen Chilaparasetti +8Healthcare Synthetic Tabular Data Generation
University of California, Irvine, CA, USA · PathAI, USA

May 14, 2026J/K move · Enter open · S save

NodeSyn: Socially Aligned Synthetic Data for AI Evaluation

Qazi Mamunur Rashid, Xuan Yang, Zhengzhe Yang, Yanzhou Pan+4

Google Research, USA · Google, USA · Google DeepMind, USA

Language Model Generation Evaluation Language Model Safety Evaluation LLM Auditing Synthetic Data Generation Synthetic Benchmark Generation Synthetic Data Generation for Language Models+1

Open paper AskPDF arXiv

Recent advancements in generative AI facilitate large-scale synthetic data generation for model evaluation. However, without targeted approaches, these datasets often lack the sociotechnical nuance required for sensitive domains. We introduce NodeSyn, a methodology that pairs a fine-tuned taxonomy generator (TaG) with credible published sources and extracted sensitive attributes to generate queries reflecting authentic societal contexts and nuanced harms. Evaluated across ten open-weight and proprietary LLMs in Medical Advice and Self-Harm using human-validated autoraters, NodeSyn ranked first on 8 of 10 models in each domain. NodeSyn datasets induced average failure rates 3.1 times higher than generic synthetic baselines, 2.4 times higher than structured synthetic benchmarks, and 1.6 times higher than authentic human-authored queries. Ablation studies confirm that, controlling for structural confounds, our granular Level 3 taxonomic expansion significantly drives these failure rates over Level 2 across 9 of 10 models (+29.2% relative lift in Medical Advice and +24.0% in Self-Harm). Variance attribution analysis reveals that failures are driven not only by what topics are queried (taxonomic nodes and lexical specificity), but also by their sociotechnical grounding (affected demographics, occupations, geographies, and querying personas). We embed this method into a research prototype, and open-source (https://github.com/google-research/nodesynth) the code and datasets to enable scalable, high-stakes model evaluation.

Abstract

Synthetic Data Generation

Explore similar work

NodeSyn: Socially Aligned Synthetic Data for AI Evaluation

Autodata: An agentic data scientist to create high quality synthetic data

Memisis: Orchestrating and Evaluating Synthetic Data for Tabular Health Datasets