Synthetic Data Generation for Language Models

Ask about this topic

Overview

This field studies how to create artificial text, examples, and interaction traces that teach language models new skills or provide evaluation data when human-authored examples are scarce. Common approaches include prompting teacher models, generating tasks from structured sources such as knowledge graphs and documents, building executable simulators, and using agents to create tool-use or reasoning trajectories. Filtering, symbolic or execution-based verification, and feedback from student-model performance are used to improve quality and tailor data to particular models, domains, and difficulty levels. A central tension is whether synthetic data can scale useful training without losing diversity, factual grounding, or resemblance to real human data; gains vary by task, and recursive generation can reinforce errors or cause collapse.

Latest papers

CardsList

294 papers36 papers in the last four weeks, up 20% on the four weeks before. 0.2% of all new papers.

  1. Steerable Instruction Following Coding Data Synthesis with Actor-Parametric Schema Co-Evolution

    Feb 27, 2026Tinglin Huang, Bo Chen, Xiao Zhang +2Synthetic Data Generation for Language ModelsCode Generation

    Yale University · ByteDance Seed

  2. MMKG-RDS: Reasoning Data Synthesis via Deep Mining of Multimodal Knowledge Graphs

    Feb 27, 2026Lun Zhan, Feng Xiong, Huanyong Liu +2Synthetic Data GenerationSynthetic Data Generation for Language Models

    AI Research Institute, Qihoo 360

  3. LLM-Driven Multi-Turn Task-Oriented Dialogue Synthesis for Realistic Reasoning

    Feb 27, 2026Yu Zhu, Kai YangSynthetic Benchmark GenerationLLM Reasoning

    Department of Computer Science and Technology, Tongji University

  4. MAPLE: Metadata Augmented Private Language Evolution

    Feb 26, 2026Eli Chien, Yuzheng Hu, Ryan McKenna +3Privacy-Preserving Language ModelsDifferentially Private Synthetic Data Generation

    National Taiwan University · University of Illinois Urbana-Champaign · Google Research +1

  5. On Data Engineering for Scaling LLM Terminal Capabilities

    Feb 24, 2026Renjie Pi, Grace Lam, Mohammad Shoeybi +3Terminal AgentsLLM Agent Training

  6. ReSyn: Autonomously Scaling Synthetic Environments for Reasoning Models

    Feb 23, 2026Andre He, Nathaniel Weir, Kaj Bostrom +4Synthetic Data Generation for Language ModelsRL for Language Model Reasoning

    Carnegie Mellon University · Amazon Web Services

  7. Adaptive Problem Generation via Symbolic Representations

    Feb 22, 2026Teresa Yeo, Myeongho Jeon, Dulaj Weerakoon +4Mathematical Problem GenerationSynthetic Data Generation for Language Models

    Singapore-MIT Alliance for Research and Technology · Korea Advanced Institute of Science and Technology · Massachusetts Institute of Technology +1

  8. Next Reply Prediction X Dataset: Linguistic Discrepancies in Naively Generated Content

    Feb 22, 2026Simon Münker, Nils Schwager, Kai Kugler +2Language Model Generation EvaluationText Generation

    Trier University, Computational Linguistics · University of Oxford, Sociology

  9. DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuning

    Feb 20, 2026Fangyuan Xu, Sihao Chen, Zinan Lin +13Differentially Private Synthetic Data GenerationSynthetic Data Generation for Language Models

    New York University · Microsoft · University of Southern California

  10. Agentic Adversarial QA for Improving Domain-Specific LLMs

    Feb 20, 2026Vincent Grari, Ciprian Tomoiaga, Sylvain Lamprier +2Domain Adaptation for LLMsSynthetic Data Generation for Language Models

    AXA Group Operations · TRAIL, Sorbonne Université, Paris, France · LERIA, Université d’Angers, France +2

  11. Label-Consistent Data Generation for Aspect-Based Sentiment Analysis Using LLM Agents

    Feb 18, 2026Mohammad H. A. Monfared, Lucie Flek, Akbar KarimiSynthetic Data AugmentationLarge Language Model-Based Data Augmentation

    Bonn-Aachen International Center for Information Technology, University of Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany

  12. Generating Realistic, Protocol-Compliant Maritime Radio Dialogues using Self-Instruct and Low-Rank Adaptation

    Feb 16, 2026Gürsel Akdeniz, Emin Cagatay NakilciogluSynthetic Data Generation for Language Models

    Fraunhofer Center for Maritime Logistics and Services (CML), Hamburg, Germany

  13. Synthetic Interaction Data for Scalable Personalization in Large Language Models

    Feb 12, 2026Yuchen Ma, Yue Huang, Wenjie Wang +3RL for Language ModelsLLM Personalization

    Munich Center for Machine Learning & LMU Munich · University of Notre Dame · Sichuan University

  14. FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

    Jan 29, 2026Ajay Patel, Colin Raffel, Chris Callison-BurchLanguage Model PretrainingSynthetic Data Pretraining

    University of Pennsylvania · University of Toronto · Vector Institute +1

  15. CALM-IT: Enhancing Synthetic Motivational Interviewing Dialogue Quality with Conversational Dynamics

    Jan 15, 2026Viet Cuong Nguyen, Nhi Yen Nguyen, Kristin A. Candan +5Large Language Model-Based Role-Play SimulationSynthetic Data Generation for Language Models

    Georgia Institute of Technology · Northwell Health · Columbia University

  16. Edit Policy-Aligned Text Simplification

    Dec 6, 2025Xuanxin Wu, Yuki Arase, Masaaki NagataText SimplificationControllable Text Generation

    The University of Osaka · Institute of Science Tokyo · NTT Inc.

  17. Learning to Pose Problems: Reasoning-Driven and Solver-Adaptive Data Synthesis

    Nov 13, 2025Yongxian Wei, Yilin Zhao, Li Shen +7Synthetic Data GenerationSynthetic Data Generation for Language Models

    Tsinghua University · Tencent · Sun Yat-sen University +1

  18. LuxIT: A Luxembourgish Instruction Tuning Dataset from Monolingual Seed Data

    Oct 28, 2025Julian Valline, Cedric Lothritz, Siwen Guo +1LLM Fine-TuningSynthetic Data Generation

    Luxembourg Institute of Science and Technology 5 Av. des Hauts-Fourneaux L-4362 Esch-sur-Alzette

  19. Reclaiming Low-Quality Instruction Data with Rule-Guided LLM Reconstruction

    Oct 27, 2025Zile Yang, Ling Li, Na Di +5Instruction-Tuning Data SelectionSupervised Fine-Tuning

    The Hong Kong University of Science and Technology (Guangzhou) · University of California, Santa Cruz · Hong Kong Baptist University

  20. Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence

    Oct 18, 2025Bingji Yi, Qiyuan Liu, Yuwei Cheng +1Synthetic Data PretrainingModel Collapse

    Independent Researcher. Work done while visiting UChicago CS. · Work done while visiting UChicago CS. · Department of Statistics, University of Chicago. +1

  21. Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillation

    Oct 13, 2025Hengyuan Zhang, Shiping Yang, Xiao Liang +8Synthetic Data GenerationLanguage Model Distillation

    The University of Hong Kong · Simon Fraser University · University of California, Los Angeles +3

  22. In-Context Bias Propagation in LLM-Based Tabular Data Generation

    Jun 11, 2025Pol G. Recasens, Alberto Gutierrez, Jordi Torres +4Synthetic Data GenerationSynthetic Tabular Data Generation

    Barcelona Supercomputing Center (BSC), Spain · Universitat Politècnica de Catalunya (UPC), Spain · IBM Research Ireland

  23. RECAST: Expanding the Boundaries of LLMs' Complex Instruction Following with Multi-Constraint Data

    May 25, 2025Zhengkang Guo, Wenhao Liu, Mingchen Xie +13LLM Fine-TuningSynthetic Data Generation

    College of Computer Science and Artificial Intelligence, Fudan University · Xiaohongshu Inc.

  24. ShIOEnv: A Command Evaluation Environment for Grammar-Constrained Synthesis and Execution Behavior Modeling

    May 23, 2025Jarrod Ragsdale, Rajendra BoppanaSynthetic Data Generation for Language ModelsCybersecurity

    Department of Computer Science, University of Texas at San Antonio, San Antonio, USA.