Synthetic Data Generation for Language Models

Overview

This field studies how to create artificial text, examples, and interaction traces that teach language models new skills or provide evaluation data when human-authored examples are scarce. Common approaches include prompting teacher models, generating tasks from structured sources such as knowledge graphs and documents, building executable simulators, and using agents to create tool-use or reasoning trajectories. Filtering, symbolic or execution-based verification, and feedback from student-model performance are used to improve quality and tailor data to particular models, domains, and difficulty levels. A central tension is whether synthetic data can scale useful training without losing diversity, factual grounding, or resemblance to real human data; gains vary by task, and recursive generation can reinforce errors or cause collapse.

Latest papers

CardsList

305 papers13 papers in the last four weeks, down 69% on the four weeks before. 0.1% of all new papers.

  1. Learning from Mutants: Teaching Small Language Models to Synthesise Loop Invariants

    Nov 14, 2026Daragh King, Vasileios KoutavasLLM-Based Program SynthesisProgram Synthesis

  2. BharatSynthQA: A Context-Grounded Question Answering Dataset on Indian Temples and Religious Heritage

    Nov 9, 2026Riya Naik, Shivam Sharma, Swati AgarwalQuestion AnsweringEvidence-Grounded QA

  3. Toward LLM Self-Play Under Data Scarcity: Effectiveness, Generalizability, and Trustworthiness

    Nov 6, 2026Zhiyin Yu, Yuchen Mou, Bingzhe Wu +3Self-Play for Language ModelsLanguage Model Self-Improvement

  4. Mass Producing Training Data with Surprising Side Effects

    Nov 4, 2026Yaowen Ye, Jacob Steinhardt, Sarah Schwettmann +1LLM TrainingEmergent Misalignment in Language Models

  5. VCoT-SQL: Enhancing Text-to-SQL Generation through Dual Chain-of-Thought Validation Training

    Oct 31, 2026Rui Zhang, Yujia Su, Ruixuan Li +2Text-to-SQLExecution-Guided Text-to-SQL

  6. From Weeks to Hours: Fast and Principled SFT Data Generation for Reasoning LLMs

    Oct 31, 2026Hongyi Henry Jin, Wenhan Yang, Meysam Ghaffari +2Training Data SelectionSynthetic Data Generation for Language Models

  7. DiRAFT: Privacy-Aware Financial Transcript Synthesis from Aggregate Discourse Priors

    Oct 29, 2026Yunbai Zhang, Yunyi Gao, Han-Tai ShiaoSynthetic Data Generation for Language ModelsText Generation

  8. MemSteer: Turning User Histories into Supervision for Agent Memory Optimization

    Oct 29, 2026Jie Huang, Cuiling Lan, Qize Yu +2LLM Agent MemoryAgent Memory Management

  9. Text Generation in Low-Resource Languages Using Human-Curated Vocabularies

    Oct 27, 2026Michael Christlambert Sinanta, Muhammad Dehan Al Kautsar, Amanda Nurul Izzah +4Text GenerationSynthetic Data Generation for Language Models

  10. Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints

    Oct 22, 2026Nicholas Roberts, Ahmed Ahmed, Suhas Kotha +4Synthetic Data PretrainingLanguage Model Scaling Laws

  11. Benchmarking Large Language Models for Detecting Satirical Code-Mixed Sentences

    Oct 21, 2026Răzvan-Alexandru Smădu, Ioana Buhnila, Daniela Tanase +3Code-SwitchingText Classification

  12. Ask to Distill: Learning to Synthesize Data for On-Policy Distillation

    Oct 20, 2026Jiashuo Sun, Hao Zhang, Xueqiang Xu +9Language Model DistillationOn-Policy Distillation

  13. Guided Diffusion for Trustworthy Synthetic Text Generation

    Oct 19, 2026Qi Pang, Hanshen Xiao, Andre Manoel +4Synthetic Data Generation for Language ModelsDifferentially Private Synthetic Data Generation

  14. TIRE: Top-Down Formula Instantiation with Reliable Execution for Physics Problem Synthesis

    Oct 18, 2026Xiaoyu Wu, Zhaoxin Yu, Xinglin Xiao +1Synthetic Data Generation for Language ModelsScientific Reasoning in Language Models

  15. Exploring The Empirical Privacy Landscape of Synthetic Data

    Oct 15, 2026Zihao Zhao, Avi Schwarzschild, Casey Meehan +3Privacy Leakage in Language ModelsPrivacy-Preserving Language Models

  16. GRADS: Graph-Regulated Agentic Data Synthesis with Controllable Difficulty

    Oct 15, 2026Gaocheng Yang, Ruijie Tao, Kai Ye +3Synthetic Data Generation for LLM AgentsSynthetic Data Generation for Language Models

  17. Instruction Mining From Images: Constructing a Synthetic Dataset for Multimodal Learning ILSP DCVLR Competition Submission

    Oct 14, 2026Eleni E. Kasoura, Dimitrios Damianos, Georgios Paraskevopoulos +1Visual Instruction TuningSynthetic Data Generation for VLMs

  18. CodeWalk: Generating Coding Benchmarks by Walking a Problem Graph

    Oct 11, 2026Jiaxin Fang, Runyuan He, Arnav Gupta +3Synthetic Benchmark GenerationBenchmark Design

  19. Probable but Not Valid: A Psychometric Audit of Decision Models as Synthetic Survey Respondents

    Oct 10, 2026Mantas Lukauskas, Viktorija ŠarkauskaitėSynthetic Data GenerationSynthetic Data Evaluation

  20. Terrarium: Growing Personal Assistants via Scalable Synthesis of App Worlds

    Oct 10, 2026Mengzhou Wu, Yuzhe Guo, Jiawen Tao +9LLM Agent TrainingSynthetic Data Generation for Language Models

  21. MADS: Multi-Agent Dialog Simulation for Robust Conversational Agents

    Oct 8, 2026Arkajyoti Chakraborty, Adam Zernik, Ishika Kulkarni +6Task-Oriented Dialogue SystemsAdversarial Robustness of Language Models

  22. MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis

    Oct 8, 2026Ximing Lu, Aviral Kumar, Jan Kautz +2Reinforcement Learning with Verifiable RewardsSynthetic Data Generation for Language Models

  23. UniData: Universal Multimodal Instruction Generation Pipeline

    Oct 8, 2026Jiaqi Tang, Yi-Feng Wu, Yuting Zhang +12Synthetic Data Generation for Language ModelsMultimodal Generation

  24. SynCo: Data Synthesis Co-Training for Self-Evolving LLMs via Multi-Agent Reinforcement Learning

    Oct 8, 2026Wei Yang, Shawn Li, Yuehan Qin +8Self-Evolving AgentsSynthetic Data Generation for Language Models