cs.LGOct 1, 2026
SaveInvent a Dataset: Measuring dataset generation abilities with zero seed
Organizations: Adaption
Abstract
Building datasets remains one of the most manual and brittle parts of AI development. In this technical report, we focus on the most extreme but also most prevalent setting real world practitioners face: a zero data regime. Here, practitioners don't have any data for the capability they want to learn. We introduce Invent-A-Dataset which is a prompt based system to go from dataset description to realistic and large scale post-training datasets. We evaluate Invent-A-Dataset against five frontier model APIs including Anthropic, Google, Open AI, DeepSeek, Zai. Across eight task types and dataset sizes up to 20K samples, Invent-A-Dataset significantly outperforms with both the highest quality (17% relative gains) while simultaneously producing the most diverse samples (19% relative gains). Its diversity advantage widens with scale of training dataset size (from parity at 200 samples to 37% relative gains at 20K samples). This translates into considerable downstream training gains, resulting in far more performant post-trained models. Invent-A-Dataset fine-tune consistently ranks higher compared to other generator fine-tunes across different post-trained model architectures.
Figures & tables
Figure 1 : Diversity vs. quality trade-off. Quality (x-axis) versus diversity (y-axis), measured by DCScore with , for four external models and the Invent API. Both scores are averaged over eight 5K-sample datasets in unconstrained setting. Higher is better on both axes; the upper right is optimal. The Invent API attains high quality and high diversity simultaneously.
| Dataset | Unconstrained Query | Constrained Query |
| African QA | Dataset containing question-and-answer pairs covering diverse topics related to African research, literature, history, and current events. | Dataset containing multi-turn conversations between a teacher and a student on an exam , covering diverse topics related to African research, literature, history, and current events. Prompts must use OpenAI format while responses should be plain text. |
| Legal QA | Dataset containing question–answer pairs focusing on legal advice, factual explanations, and comparative analysis across diverse topics such as copyright law, government types, and liability. | Dataset containing question–answer pairs focusing on legal advice, factual explanations, and comparative analysis across diverse topics such as copyright law, government types, and liability. Questions should offer four choices; answers are limited to A, B, C, or D with no additional text. |
| Serbian Car Ads | A dataset containing a collection of used car advertisements in Serbian, featuring detailed descriptions of vehicle conditions, service histories, and equipment packages. | A dataset containing a collection of summaries of used car advertisements in Serbian, featuring detailed descriptions of vehicle conditions, service histories, and equipment packages. Each prompt is an ad summary, and each response is a JSON object containing vehicle_description , service_history , and equipment_packages . |
Table 2 : The single request queries used to generate datasets. Queries are chosen to represent a range of 1) task types, 2) languages and 3) domains. We evaluate on both unconstrained and constrained versions of the query, where the constrained query requires additional requirements such as 1) tone constraints, 2) json format, 3) multiple choice output restrictions. Differences in the constrained query are highlighted .
| API | Prompt | Completion |
| Claude Opus 5 | “How is hypertension diagnosed, and what blood pressure numbers matter?” | “Hypertension is diagnosed from properly measured, repeated blood pressure readings rather than a single office value. The patient should be seated quietly for five minutes, back supported, feet flat, […]” |
| Gemini 3.1 Pro | “What class of anticoagulant is Dabigatran?” | “Dabigatran is a Direct Thrombin Inhibitor (DTI). It reversibly binds to the active site of thrombin (Factor IIa), preventing the conversion of fibrinogen to fibrin. It was the first DOAC approved as an […]” |
| DeepSeek V4 Pro | “What is the mechanism of action of digoxin?” | “Digoxin inhibits the ATPase pump in cardiac myocytes, raising intracellular sodium and, via the sodium-calcium exchanger, intracellular calcium […]” |
| GPT 5.6 Sol | “What are warning signs of a heart attack?” | “Warning signs include pressure, squeezing, heaviness, or pain in the chest, sometimes spreading to an arm, shoulder, jaw, back, or upper abdomen. Shortness of breath, sweating, nausea, dizziness, or unusual fatigue may also occur, and symptoms can be atypical in women, older adults, and people with diabetes. […]” |
| GLM 5.3 | “What are the diagnostic criteria for hypertension?” | “Hypertension is diagnosed based on repeated elevated office blood pressure measurements. Under the 2017 ACC/AHA guidelines, stage 1 is systolic 130–139 or diastolic 80–89 mmHg, and stage 2 is 140/90 mmHg or higher. Out-of-office ambulatory or home monitoring is recommended to confirm the diagnosis. […]” |
| Invent API | “A 62-year-old male presents with progressive shortness of breath, bilateral inspiratory crackles, and a dry cough. He has a 30-year history of asbestos exposure. Discuss the differential diagnosis between idiopathic pulmonary fibrosis and asbestosis.” | “ Differential Diagnosis: IPF vs. Asbestosis. The presentation of progressive dyspnea, dry cough and bilateral ‘Velcro’ crackles with a 30-year asbestos history is a classic diagnostic challenge. Both are chronic, progressive fibrosing interstitial lung diseases sharing a Usual Interstitial Pneumonia (UIP) pattern. […]” |
Table 3 : Example prompt–completion pairs generated by each API for the Medical QA dataset, whose query was: “Dataset of medical question-answer pairs covering diagnoses, treatments, drug explanations, and healthcare roles. Each entry includes a prompt and a detailed, expert-generated response”
Figure 2 : Diversity across different generation strategies. We compare strategies for generating a dataset from a single query request (Medical QA) across three models: Gemini 3.1 Pro, GPT-5.6 Sol, and GLM-5.3. Strategies differ in how many samples are produced per request: one sample at a time ( one-by-one ), fixed batches of 100, or as many samples as fit within the model’s maximum output-token limit ( batch_size max ). The y-axis shows DCScore ( ), where higher is better. Diversity generally increases with batch size, and generating samples one by one leads to near-complete diversity collapse ( ) for every model.
Figure 3 : Diversity vs. Quality in Multilingual Setting. Diversity–quality trade-off at samples for non-English datasets in the constrained setting. In both settings the Invent API occupies the upper-right region, leading on diversity throughout and on quality for the six additional languages.
Figure 4 : Diversity under constrained vs. unconstrained settings per generator , measured with DCScore ( ) and and averaged across 8 datasets of 20000 samples each. The Invent API is the most robust: its diversity is essentially unchanged ( in both settings), while every baseline drops by – . As a result, under constraints the Invent API has the highest diversity of any pipeline ( vs. for the next best, GLM-5.3).
Figure 5 : Left: Quality with scale. Average dataset quality for GLM-5.3 and the Invent API at , , and and samples. The Invent API leads at every scale and remains stable as the dataset grows. Right: Diversity with scale. Average diversity (DCScore, ) for all generators at , and and samples measured on samples randomly drawn from each dataset. Per-sample diversity declines with scale for most baseline generators, whereas Invent API’s diversity remains high (and even increases), so its relative advantage grows with scale.
Figure 6 : Left: Average dataset prompt and completion quality at the -sample scale. The Invent API is the highest-quality dataset generator, ahead of Claude Opus 5 and GLM-5.3. Right: Relevance per generator. Average sample relevance at the -sample scale. All pipelines produce on-specification data ( – ); Claude Opus 5 and GLM-5.3 lead, with the Invent API competitive.
| NGD | CR | Dup (%) | |||||||
| Data Variant | 2K | 5K | 20K | 2K | 5K | 20K | 2K | 5K | 20K |
| Invent API | 1.75 | 1.82 | 1.87 | 3.67 | 3.51 | 3.47 | 0.0 | 0.0 | 0.0 |
| GLM-5.3 | 1.57 | 1.70 | 1.69 | 4.70 | 3.95 | 3.96 | 6.2 | 6.8 | 8.9 |
| Gemini-3.1-pro | 1.53 | 1.62 | 1.66 | 4.59 | 4.22 | 4.08 | 7.5 | 6.8 | 6.5 |
| GPT-5.6 Sol | 1.60 | 1.65 | 1.59 | 4.33 | 4.33 | 4.39 | 9.6 | 10.4 | 12.4 |
| Claude Opus 5 | 1.55 | 1.54 | 1.48 | 4.27 | 4.25 | 4.28 | 8.5 | 7.9 | 8.4 |
Table 4: Lexical prompt diversity and duplication in the unconstrained setting, averaged over the eight datasets. Each dataset size (2K/5K/20K) is measured on a fixed prompt random sample, so columns are directly comparable. n-gram diversity (NGD): implies more diverse. Compression Ration (CR): implies less redundant; Dup is the percentage of exact-duplicate prompts ( is better). Best data variant is indicated in bold for each column.
Figure 7 : Top-1 rank share in downstream ranking evaluation. Percentage of held-out medical test prompts on which each fine-tune is ranked first, with all six responses to a prompt judged together in a single call. Within each panel, the x-axis lists models fine-tuned on different synthetic data variants, plus the untuned base model (grey) as a control. Left: Llama-3.3-70B. The Invent API fine-tune is the clear winner, ranking first on of prompts, more than double the untuned base at . Right: Gemma-4-31B-it. The Invent API fine-tune again leads at , well above the base at . In both cases, the models fine-tuned on data from other generators (GLM-5.3, Claude Opus 5, GPT-5.6 Sol, DeepSeek-V4-Pro) trail the base by a wide margin.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8 : Per constrained dataset prompt quality, completion quality, and dataset relevance scores for each generator across dataset sizes .
Figure 9 : Per unconstrained dataset prompt quality, completion quality, and dataset relevance scores for each generator across dataset sizes .
| Dataset | Unconstrained Query | Constrained Query |
| Customer Support Banking | Dataset of customer service responses guiding users through credit card activation, blocking, and mortgage inquiries. | Dataset of customer service conversations guiding frustrated users through credit card activation, blocking, and mortgage inquiries. The customer service agent is always welcoming and polite. |
| Hindi News | Dataset containing Hindi news articles sourced from various Indian news websites, paired with their corresponding headlines and summaries. | Dataset containing Hindi news articles sourced from various Indian news websites about sports, paired with their corresponding summaries. The prompt is always the article headline, and the completion is the article summary. |
| Customer Support General | A dataset of customer support inquiries and corresponding responses, covering issues such as product information, technical troubleshooting, billing disputes, and service returns. | |
| Medical QA | Dataset of medical question–answer pairs covering diagnoses, treatments, drug explanations, and healthcare roles. Each entry includes a prompt and a detailed, expert-generated response. | |
| Hotel Reviews | A collection of customer reviews describing hotel stays, focusing on service, room quality, location, and amenities. The dataset should include both positive and negative experiences. |
Table 5 : The single request queries used to generate datasets. Queries are choosen to represent a range of 1) task types, 2) languages and 3) domains. We evaluate on both unconstrained and constrained versions of the query, where the constrained query requires additional requirements such as 1) tone constraints, 2) json format, 3) multiple choice output restrictions.
| Model | Not in language | % | Detected languages |
| claude-opus-5 | 7/2000 | 0.35 | hi 1993, mr 7 |
| deepseek-v4-pro-0813 | 145/2000 | 7.25 | hi 1855, mr 144, ne 1 |
| gemini-3.1-pro | 77/2000 | 3.85 | hi 1923, mr 75, ne 2 |
| gpt-5.6-sol | 50/2000 | 2.50 | hi 1950, mr 50 |
| glm-5.3 | 49/2000 | 2.45 | hi 1951, en 38, mr 11 |
| invent-api | 3/2000 | 0.15 | hi 1997, mr 2, lt 1 |
Table 6 : Output language consistency on hindi_news (2k samples drawn from 20k). Not in language counts all outputs detected as any language other than Hindi, including Marathi and Nepali. Language codes: hi = Hindi, mr = Marathi, ne = Nepali, en = English, lt = Lithuanian.
| NGD | CR | Dup (%) | |||||||
| Model | 2K | 5K | 20K | 2K | 5K | 20K | 2K | 5K | 20K |
| Invent API | 1.69 | 1.69 | 1.72 | 3.55 | 3.53 | 3.50 | 0.1 | 0.1 | 0.1 |
| GLM-5.3 | 1.65 | 1.80 | 1.71 | 4.59 | 3.75 | 3.83 | 0.9 | 1.2 | 3.9 |
| Gemini-3.1-pro | 1.34 | 1.39 | 1.53 | 5.18 | 4.75 | 4.03 | 2.2 | 2.3 | 1.8 |
| GPT-5.6 Sol | 1.56 | 1.57 | 1.53 | 4.16 | 4.22 | 4.29 | 2.2 | 3.7 | 5.0 |
| Claude Opus 5 | 1.61 | 1.56 | 1.46 | 4.00 | 4.02 | 4.09 | 2.2 | 3.9 | 4.3 |
Table 7: Lexical diversity and duplication in the constrained setting, averaged over the eight datasets (computed for each dataset size on a fixed prompt random sample). n-gram diversity, NGD ( is better), Compression Ratio, CR ( is better), Duplicate Rate (Dup %, is better). The best per column is highlighted in bold.
Figure 10 : Top-1 rank share for Qwen3.5-9B fine-tunes. Percentage of held-out medical prompts on which each model ranks first (all six responses judged together per prompt); grey is the untuned base. Qwen3.5-9B has an unusually strong base that edges the Invent API fine-tune on both first-place rate ( vs. ) and mean rank ( vs. ). Invent API fine-tuned variant nonetheless remains far ahead of all variants corresponding to other generators.
Figure 11 : Diversity vs. quality on 200-sample datasets. Each point shows the average quality score (x-axis) and average diversity score, measured by DCScore ( ) (y-axis), across 8 dataset in unconstrained setting.
Figure 12 : Diversity vs. quality on 2000-sample datasets. Each point shows the average quality score (x-axis) and average diversity score, measured by DCScore ( ) (y-axis), across 8 dataset in unconstrained setting.
Figure 13 : Diversity vs. quality on 5000-sample datasets. Each point shows the average quality score (x-axis) and average diversity score, measured by DCScore ( ) (y-axis), across 8 dataset in unconstrained setting.
Figure 14 : Diversity vs. quality on 20000-sample datasets. Each point shows the average quality score (x-axis) and average diversity score, measured by DCScore ( ) (y-axis), across 8 dataset in unconstrained setting.