Invent a Dataset: Measuring dataset generation abilities with zero seed

Shivalika Singh; Andrija Djurisic; Gbemileke Onilude; Sudip Roy; Sara Hooker

cs.LGOct 1, 2026

Invent a Dataset: Measuring dataset generation abilities with zero seed

Authors: Shivalika Singh, Andrija Djurisic, Gbemileke Onilude, Sudip Roy, Sara Hooker

Organizations: Adaption

Abstract

Building datasets remains one of the most manual and brittle parts of AI development. In this technical report, we focus on the most extreme but also most prevalent setting real world practitioners face: a zero data regime. Here, practitioners don't have any data for the capability they want to learn. We introduce Invent-A-Dataset which is a prompt based system to go from dataset description to realistic and large scale post-training datasets. We evaluate Invent-A-Dataset against five frontier model APIs including Anthropic, Google, Open AI, DeepSeek, Zai. Across eight task types and dataset sizes up to 20K samples, Invent-A-Dataset significantly outperforms with both the highest quality (17% relative gains) while simultaneously producing the most diverse samples (19% relative gains). Its diversity advantage widens with scale of training dataset size (from parity at 200 samples to 37% relative gains at 20K samples). This translates into considerable downstream training gains, resulting in far more performant post-trained models. Invent-A-Dataset fine-tune consistently ranks higher compared to other generator fine-tunes across different post-trained model architectures.

Figures & tables

Figure 1 : Diversity vs. quality trade-off. Quality (x-axis) versus diversity (y-axis), measured by DCScore with $\tau=0.05$ , for four external models and the Invent API. Both scores are averaged over eight 5K-sample datasets in unconstrained setting. Higher is better on both axes; the upper right is optimal. The Invent API attains high quality and high diversity simultaneously.

Dataset	Unconstrained Query	Constrained Query
African QA	Dataset containing question-and-answer pairs covering diverse topics related to African research, literature, history, and current events.	Dataset containing multi-turn conversations between a teacher and a student on an exam , covering diverse topics related to African research, literature, history, and current events. Prompts must use OpenAI format while responses should be plain text.
Legal QA	Dataset containing question–answer pairs focusing on legal advice, factual explanations, and comparative analysis across diverse topics such as copyright law, government types, and liability.	Dataset containing question–answer pairs focusing on legal advice, factual explanations, and comparative analysis across diverse topics such as copyright law, government types, and liability. Questions should offer four choices; answers are limited to A, B, C, or D with no additional text.
Serbian Car Ads	A dataset containing a collection of used car advertisements in Serbian, featuring detailed descriptions of vehicle conditions, service histories, and equipment packages.	A dataset containing a collection of summaries of used car advertisements in Serbian, featuring detailed descriptions of vehicle conditions, service histories, and equipment packages. Each prompt is an ad summary, and each response is a JSON object containing vehicle_description , service_history , and equipment_packages .

Table 2 : The single request queries used to generate datasets. Queries are chosen to represent a range of 1) task types, 2) languages and 3) domains. We evaluate on both unconstrained and constrained versions of the query, where the constrained query requires additional requirements such as 1) tone constraints, 2) json format, 3) multiple choice output restrictions. Differences in the constrained query are highlighted .

API	Prompt	Completion
Claude Opus 5	“How is hypertension diagnosed, and what blood pressure numbers matter?”	“Hypertension is diagnosed from properly measured, repeated blood pressure readings rather than a single office value. The patient should be seated quietly for five minutes, back supported, feet flat, […]”
Gemini 3.1 Pro	“What class of anticoagulant is Dabigatran?”	“Dabigatran is a Direct Thrombin Inhibitor (DTI). It reversibly binds to the active site of thrombin (Factor IIa), preventing the conversion of fibrinogen to fibrin. It was the first DOAC approved as an […]”
DeepSeek V4 Pro	“What is the mechanism of action of digoxin?”	“Digoxin inhibits the ATPase pump in cardiac myocytes, raising intracellular sodium and, via the sodium-calcium exchanger, intracellular calcium […]”
GPT 5.6 Sol	“What are warning signs of a heart attack?”	“Warning signs include pressure, squeezing, heaviness, or pain in the chest, sometimes spreading to an arm, shoulder, jaw, back, or upper abdomen. Shortness of breath, sweating, nausea, dizziness, or unusual fatigue may also occur, and symptoms can be atypical in women, older adults, and people with diabetes. […]”
GLM 5.3	“What are the diagnostic criteria for hypertension?”	“Hypertension is diagnosed based on repeated elevated office blood pressure measurements. Under the 2017 ACC/AHA guidelines, stage 1 is systolic 130–139 or diastolic 80–89 mmHg, and stage 2 is 140/90 mmHg or higher. Out-of-office ambulatory or home monitoring is recommended to confirm the diagnosis. […]”
Invent API	“A 62-year-old male presents with progressive shortness of breath, bilateral inspiratory crackles, and a dry cough. He has a 30-year history of asbestos exposure. Discuss the differential diagnosis between idiopathic pulmonary fibrosis and asbestosis.”	“ Differential Diagnosis: IPF vs. Asbestosis. The presentation of progressive dyspnea, dry cough and bilateral ‘Velcro’ crackles with a 30-year asbestos history is a classic diagnostic challenge. Both are chronic, progressive fibrosing interstitial lung diseases sharing a Usual Interstitial Pneumonia (UIP) pattern. […]”

Table 3 : Example prompt–completion pairs generated by each API for the Medical QA dataset, whose query was: “Dataset of medical question-answer pairs covering diagnoses, treatments, drug explanations, and healthcare roles. Each entry includes a prompt and a detailed, expert-generated response”

Figure 2 : Diversity across different generation strategies. We compare strategies for generating a dataset from a single query request (Medical QA) across three models: Gemini 3.1 Pro, GPT-5.6 Sol, and GLM-5.3. Strategies differ in how many samples are produced per request: one sample at a time ( one-by-one ), fixed batches of 100, or as many samples as fit within the model’s maximum output-token limit ( batch_size max ). The y-axis shows DCScore ( $\tau=0.1$ ), where higher is better. Diversity generally increases with batch size, and generating samples one by one leads to near-complete diversity collapse ( $\leq 0.03$ ) for every model.

Panel of Figure 3 — Figure 3 : Diversity vs. Quality in Multilingual Setting. Diversity–quality trade-off at $5{,}000$ samples for non-English datasets in the constrained setting. In both settings the Invent API occupies the upper-right region, leading on diversity throughout and on quality for the six additional languages.

Figure 4 : Diversity under constrained vs. unconstrained settings per generator , measured with DCScore ( $\tau=0.1$ ) and and averaged across 8 datasets of 20000 samples each. The Invent API is the most robust: its diversity is essentially unchanged ( $0.0428$ in both settings), while every baseline drops by $15$ – $22\%$ . As a result, under constraints the Invent API has the highest diversity of any pipeline ( $0.0428$ vs. $0.0271$ for the next best, GLM-5.3).

Panel of Figure 5 — Figure 5 : Left: Quality with scale. Average dataset quality for GLM-5.3 and the Invent API at $200$ , $2{,}000$ , and $5{,}000$ and $20{,}000$ samples. The Invent API leads at every scale and remains stable as the dataset grows. Right: Diversity with scale. Average diversity (DCScore, $\tau=0.1$ ) for all generators at $2{,}000$ , and $5{,}000$ and $20{,}000$ samples measured on $2{,}000$ samples randomly drawn from each dataset. Per-sample diversity declines with scale for most baseline generators, whereas Invent API’s diversity remains high (and even increases), so its relative advantage grows with scale.

Panel of Figure 6 — Figure 6 : Left: Average dataset prompt and completion quality at the $5{,}000$ -sample scale. The Invent API is the highest-quality dataset generator, ahead of Claude Opus 5 and GLM-5.3. Right: Relevance per generator. Average sample relevance at the $5{,}000$ -sample scale. All pipelines produce on-specification data ( $7.1$ – $8.9$ ); Claude Opus 5 and GLM-5.3 lead, with the Invent API competitive.

	NGD ${}_{\text{prompt}}\uparrow$			CR ${}_{\text{prompt}}\downarrow$			Dup ${}_{\text{prompt}}$ (%) $\downarrow$
Data Variant	2K	5K	20K	2K	5K	20K	2K	5K	20K
Invent API	1.75	1.82	1.87	3.67	3.51	3.47	0.0	0.0	0.0
GLM-5.3	1.57	1.70	1.69	4.70	3.95	3.96	6.2	6.8	8.9
Gemini-3.1-pro	1.53	1.62	1.66	4.59	4.22	4.08	7.5	6.8	6.5
GPT-5.6 Sol	1.60	1.65	1.59	4.33	4.33	4.39	9.6	10.4	12.4
Claude Opus 5	1.55	1.54	1.48	4.27	4.25	4.28	8.5	7.9	8.4

Table 4: Lexical prompt diversity and duplication in the unconstrained setting, averaged over the eight datasets. Each dataset size (2K/5K/20K) is measured on a fixed

2{,}000

prompt random sample, so columns are directly comparable. n-gram diversity (NGD):

\uparrow

implies more diverse. Compression Ration (CR):

\downarrow

implies less redundant; Dup is the percentage of exact-duplicate prompts (

\downarrow

is better). Best data variant is indicated in bold for each column.

Panel of Figure 7 — Figure 7 : Top-1 rank share in downstream ranking evaluation. Percentage of held-out medical test prompts on which each fine-tune is ranked first, with all six responses to a prompt judged together in a single call. Within each panel, the x-axis lists models fine-tuned on different synthetic data variants, plus the untuned base model (grey) as a control. Left: Llama-3.3-70B. The Invent API fine-tune is the clear winner, ranking first on $54\%$ of prompts, more than double the untuned base at $25\%$ . Right: Gemma-4-31B-it. The Invent API fine-tune again leads at $41\%$ , well above the base at $21\%$ . In both cases, the models fine-tuned on data from other generators (GLM-5.3, Claude Opus 5, GPT-5.6 Sol, DeepSeek-V4-Pro) trail the base by a wide margin.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8 : Per constrained dataset prompt quality, completion quality, and dataset relevance scores for each generator across dataset sizes $N\in\{200,2000,5000,20000\}$ .

Panel of Figure 9 — Figure 9 : Per unconstrained dataset prompt quality, completion quality, and dataset relevance scores for each generator across dataset sizes $N\in\{200,2000,5000,20000\}$ .

Dataset	Unconstrained Query	Constrained Query
Customer Support Banking	Dataset of customer service responses guiding users through credit card activation, blocking, and mortgage inquiries.	Dataset of customer service conversations guiding frustrated users through credit card activation, blocking, and mortgage inquiries. The customer service agent is always welcoming and polite.
Hindi News	Dataset containing Hindi news articles sourced from various Indian news websites, paired with their corresponding headlines and summaries.	Dataset containing Hindi news articles sourced from various Indian news websites about sports, paired with their corresponding summaries. The prompt is always the article headline, and the completion is the article summary.
Customer Support General	A dataset of customer support inquiries and corresponding responses, covering issues such as product information, technical troubleshooting, billing disputes, and service returns.
Medical QA	Dataset of medical question–answer pairs covering diagnoses, treatments, drug explanations, and healthcare roles. Each entry includes a prompt and a detailed, expert-generated response.
Hotel Reviews	A collection of customer reviews describing hotel stays, focusing on service, room quality, location, and amenities. The dataset should include both positive and negative experiences.

Appendix

Table 5 : The single request queries used to generate datasets. Queries are choosen to represent a range of 1) task types, 2) languages and 3) domains. We evaluate on both unconstrained and constrained versions of the query, where the constrained query requires additional requirements such as 1) tone constraints, 2) json format, 3) multiple choice output restrictions.

Model	Not in language	%	Detected languages
claude-opus-5	7/2000	0.35	hi 1993, mr 7
deepseek-v4-pro-0813	145/2000	7.25	hi 1855, mr 144, ne 1
gemini-3.1-pro	77/2000	3.85	hi 1923, mr 75, ne 2
gpt-5.6-sol	50/2000	2.50	hi 1950, mr 50
glm-5.3	49/2000	2.45	hi 1951, en 38, mr 11
invent-api	3/2000	0.15	hi 1997, mr 2, lt 1

Appendix

Table 6 : Output language consistency on hindi_news (2k samples drawn from 20k). Not in language counts all outputs detected as any language other than Hindi, including Marathi and Nepali. Language codes: hi = Hindi, mr = Marathi, ne = Nepali, en = English, lt = Lithuanian.

	NGD ${}_{\text{prompt}}\uparrow$			CR ${}_{\text{prompt}}\downarrow$			Dup ${}_{\text{prompt}}$ (%) $\downarrow$
Model	2K	5K	20K	2K	5K	20K	2K	5K	20K
Invent API	1.69	1.69	1.72	3.55	3.53	3.50	0.1	0.1	0.1
GLM-5.3	1.65	1.80	1.71	4.59	3.75	3.83	0.9	1.2	3.9
Gemini-3.1-pro	1.34	1.39	1.53	5.18	4.75	4.03	2.2	2.3	1.8
GPT-5.6 Sol	1.56	1.57	1.53	4.16	4.22	4.29	2.2	3.7	5.0
Claude Opus 5	1.61	1.56	1.46	4.00	4.02	4.09	2.2	3.9	4.3

Appendix

Table 7: Lexical diversity and duplication in the constrained setting, averaged over the eight datasets (computed for each dataset size on a fixed

2{,}000

prompt random sample). n-gram diversity, NGD (

\uparrow

is better), Compression Ratio, CR (

\downarrow

is better), Duplicate Rate (Dup %,

\downarrow

is better). The best per column is highlighted in bold.

Figure 10 : Top-1 rank share for Qwen3.5-9B fine-tunes. Percentage of held-out medical prompts on which each model ranks first (all six responses judged together per prompt); grey is the untuned base. Qwen3.5-9B has an unusually strong base that edges the Invent API fine-tune on both first-place rate ( $44\%$ vs. $38\%$ ) and mean rank ( $2.17$ vs. $2.28$ ). Invent API fine-tuned variant nonetheless remains far ahead of all variants corresponding to other generators.

Figure 11 : Diversity vs. quality on 200-sample datasets. Each point shows the average quality score (x-axis) and average diversity score, measured by DCScore ( $\tau=0.1$ ) (y-axis), across 8 dataset in unconstrained setting.

Figure 12 : Diversity vs. quality on 2000-sample datasets. Each point shows the average quality score (x-axis) and average diversity score, measured by DCScore ( $\tau=0.1$ ) (y-axis), across 8 dataset in unconstrained setting.

Figure 13 : Diversity vs. quality on 5000-sample datasets. Each point shows the average quality score (x-axis) and average diversity score, measured by DCScore ( $\tau=0.1$ ) (y-axis), across 8 dataset in unconstrained setting.

Figure 14 : Diversity vs. quality on 20000-sample datasets. Each point shows the average quality score (x-axis) and average diversity score, measured by DCScore ( $\tau=0.1$ ) (y-axis), across 8 dataset in unconstrained setting.

Explore similar work

Jun 24, 2026cs.AI

Autodata: An agentic data scientist to create high quality synthetic data

We introduce Autodata, a general method that enables AI agents to act as data scientists who build high quality training and evaluation data. We show how to train (meta-optimize) such a data scientist agent, so that it learns to create even stronger data. We describe the overall formulation, and a specific practical implementation, Agentic Self-Instruct. We conduct experiments on computer science research tasks, legal reasoning tasks and reasoning with mathematical objects, where we obtain improved results compared to classical synthetic dataset creation methods. Further, meta-optimizing the data scientist agent itself delivers an even larger performance uplift. Agentic data creation provides a way to convert increased inference compute into higher quality model training. Overall, we believe this direction has the potential to change the way we build AI data.

Ilia Kulikov, Chenxi Whitehouse, Tianhao Wu +12

FAIR at Meta

Data Science Agents Synthetic Data Artificial Intelligence Scientists Prompt Optimization

Ask PDF

Apr 20, 2026cs.AI

Adversarial Arena: Crowdsourcing Data Generation through Interactive Competition

Post-training Large Language Models requires diverse, high-quality data which is rare and costly to obtain, especially in low resource domains and for multi-turn conversations. Common solutions are crowdsourcing or synthetic generation, but both often yield low-quality or low-diversity data. We introduce Adversarial Arena for building high quality conversational datasets by framing data generation as an adversarial task: attackers create prompts, and defenders generate responses. This interactive competition between multiple teams naturally produces diverse and complex data. We validated this approach by conducting a competition with 10 academic teams from top US and European universities, each building attacker or defender bots. The competition, focused on safety alignment of LLMs in cybersecurity, generated 19,683 multi-turn conversations. Fine-tuning an open-source model on this dataset produced an 18.47% improvement in secure code generation on CyberSecEval-Instruct and 29.42% improvement on CyberSecEval-MITRE.

Prasoon Goyal, Sattvik Sahai, Michael Johnston +14

Amazon Nova Responsible AI

Large Language Model Generation Adversarial Robustness Conversational Datasets Data Generation LLM Defense Mechanisms Competition

Ask PDF

May 31, 2026cs.AI

ANDES: Agent Native Data Evolving Synthesis Tool for Autonomous Instruction Alignment

AI agents are increasingly being tasked with automating AI research itself, particularly the critical post-training phase that transforms base LLMs into aligned assistants. However, recent evaluations reveal that even frontier agents struggle to perform this task. While the success of post-training fundamentally relies on acquiring high-quality data, relying on agents to autonomously curate targeted training datasets from the open web introduces severe challenges. Executing the long-horizon tasks of searching, filtering, and balancing data within noisy web environments frequently overwhelms an agent's limited context, ultimately leading to degraded dataset quality and suboptimal downstream training performance. To bridge this gap, we introduce Andes (Agent Native Data Evolving Synthesis), a framework that reimagines data generation as a plug-and-play \emph{agent skill}. Rather than forcing agents to devise complex data-gathering strategies from scratch, \textsc{Andes} provides an intelligent abstraction layer. By leveraging a self-evolving World Tree routing mechanism and actionable diagnostic reports, it allows trainer agents to dynamically steer data synthesis through an interactive, closed-loop interface. We demonstrate that under strict compute constraints, equipping foundationally weaker agents with Andes improves automated alignment, securing state-of-the-art performance on PostTrainBench and robust cross-task generalization. Our project is available at https://github.com/zzy1127/ANDES.

Zhengyang Zhao, Shengjie Ye, Lu Ma +3

Peking University · Sichuan University, Chengdu

Data Science Agents Synthetic Data Downstream Fine-Tuning Artificial Intelligence Research Instruction

Ask PDF