Synthetic Data Generation for Language Models

Overview

This field studies how to create artificial text, examples, and interaction traces that teach language models new skills or provide evaluation data when human-authored examples are scarce. Common approaches include prompting teacher models, generating tasks from structured sources such as knowledge graphs and documents, building executable simulators, and using agents to create tool-use or reasoning trajectories. Filtering, symbolic or execution-based verification, and feedback from student-model performance are used to improve quality and tailor data to particular models, domains, and difficulty levels. A central tension is whether synthetic data can scale useful training without losing diversity, factual grounding, or resemblance to real human data; gains vary by task, and recursive generation can reinforce errors or cause collapse.

Latest papers

CardsList

305 papers13 papers in the last four weeks, down 69% on the four weeks before. 0.1% of all new papers.

  1. Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillation

    Oct 13, 2025Hengyuan Zhang, Shiping Yang, Xiao Liang +8Synthetic Data GenerationLanguage Model Distillation

  2. In-Context Bias Propagation in LLM-Based Tabular Data Generation

    Jun 11, 2025Pol G. Recasens, Alberto Gutierrez, Jordi Torres +4Synthetic Data GenerationSynthetic Tabular Data Generation

  3. RECAST: Expanding the Boundaries of LLMs' Complex Instruction Following with Multi-Constraint Data

    May 25, 2025Zhengkang Guo, Wenhao Liu, Mingchen Xie +13LLM Fine-TuningSynthetic Data Generation

  4. ShIOEnv: A Command Evaluation Environment for Grammar-Constrained Synthesis and Execution Behavior Modeling

    May 23, 2025Jarrod Ragsdale, Rajendra BoppanaSynthetic Data Generation for Language ModelsCybersecurity

  5. Beyond Empathy: Integrating Diagnostic and Therapeutic Reasoning with Large Language Models for Mental Health Counseling

    May 21, 2025He Hu, Juzheng Si, Qianning Wang +5Mental Health CounselingStructured Reasoning in Language Models