Synthetic Data Generation for Language Models

Follow this topic Ask about this topic

This field studies how to create artificial text, examples, and interaction traces that teach language models new skills or provide evaluation data when human-authored examples are scarce. Common approaches include prompting teacher models, generating tasks from structured sources such as knowledge graphs and documents, building executable simulators, and using agents to create tool-use or reasoning trajectories. Filtering, symbolic or execution-based verification, and feedback from student-model performance are used to improve quality and tailor data to particular models, domains, and difficulty levels. A central tension is whether synthetic data can scale useful training without losing diversity, factual grounding, or resemblance to real human data; gains vary by task, and recursive generation can reinforce errors or cause collapse.

Latest papers

294 papers36 papers in the last four weeks, up 20% on the four weeks before. 0.2% of all new papers.

Dynamic Context Evolution for Scalable Synthetic Data Generation
Apr 8, 2026Ryan Lingo, Rajeev ChhajerRepetition in Language Generation Output Diversity in Language Models
Honda Research Institute, USA, Inc.
iTAG: Inverse Design for Natural Text Generation with Accurate Causal Graph Annotations
Apr 8, 2026Wenshuo Wang, Boyu Cao, Nan Zhuang +1Synthetic Data Generation for Language Models Inverse Design
School of Future Technology, South China University of Technology, Guangzhou, China · China Mobile GBA Innovation Institute, Guangzhou, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
WRAP++: Web discoveRy Amplified Pretraining
Apr 8, 2026Jiang Zhou, Yunhao Wang, Xing Wu +2Knowledge Augmentation for Language Models Synthetic Data Pretraining
Data-Driven Function Calling Improvements in Large Language Model for Online Financial QA
Apr 7, 2026Xing Tang, Hao Chen, Shiwei Li +7LLM Tool Use Financial QA
Shenzhen Technology University Shenzhen, China · FiT, Tencent Shenzhen, China · Huazhong University of Science and Technology Wuhan, China +3
$π^2$ : Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models
Apr 6, 2026Quyet V. Do, Thinh Pham, Nguyen Nguyen +3Long-Context Reasoning in Language Models LLM Reasoning
Virginia Tech, VA 24061, USA
ToolWeave: Structured Synthesis of Complex Multi-Turn Tool-Calling Dialogues
Apr 3, 2026Dinesh Khandelwal, Gnana Prakash Punnavajhala, GPS Bhargav +4Synthetic Data Generation for LLM Agents Synthetic Data Generation for Language Models
IBM Research · IIIT Hyderabad
From Theory to Practice: Code Generation Using LLMs for CAPEC and CWE Frameworks
Apr 2, 2026Murtuza Shahzad, Joseph Wilson, Ibrahim Al Azher +2Software Security Synthetic Data Generation for Language Models
Department of Computer Science, Northern Illinois University, DeKalb, Illinois USA
ORBIT: Scalable and Verifiable Data Generation for Search Agents on a Tight Budget
Apr 1, 2026Nandan Thakur, Zijian Chen, Xueguang Ma +1Synthetic Data Generation for Language Models LLM Agent Training
David R. Cheriton School of Computer Science University of Waterloo
Translation Asymmetry in LLMs as a Data Augmentation Factor: A Case Study for 6 Romansh Language Varieties
Mar 26, 2026Jannis Vamvas, Ignacio Pérez Prat, Angela Heldstab +3Machine Translation Low-Resource MT
University of Zurich · Lia Rumantscha
Knowledge-Guided Retrieval-Augmented Generation for Zero-Shot Psychiatric Data: Privacy Preserving Synthetic Data Generation
Mar 26, 2026Adam Jakobsen, Sushant Gautam, Hugo Lewi Hammer +4Synthetic Clinical Data Generation Synthetic Tabular Data Generation
∗SimulaMet, Norway · †Oslo Metropolitan University, Norway · ‡Uppsala University, Sweden
AnyDoc: Enhancing Document Generation via Large-Scale HTML/CSS Data Synthesis and Height-Aware Reinforcement Optimization
Mar 26, 2026Jiawei Lin, Wanrong Zhu, Vlad I Morariu +1Text Generation Synthetic Data Generation for Language Models
Xi’an Jiaotong University · Adobe Research
Synthetic Rewriting as a Quality Multiplier: Evidence from Portuguese Continued Pretraining
Mar 25, 2026Thales Sales Almeida, Rodrigo Nogueira, Hélio PedriniSynthetic Data Generation for Language Models Continued Pretraining
Institute of Computing (IC) University of Campinas (UNICAMP) · Maritaca AI · School of Electrical and Computer Engineering (FEEC) University of Campinas (UNICAMP)
A Deep Dive into Scaling RL for Code Generation with Synthetic Data and Curricula
Mar 25, 2026Cansu Sancaktar, David Zhang, Gabriel Synnaeve +1RL for Code Generation Synthetic Data Augmentation
University of Tübingen · Max Planck Institute for Intelligent Systems · Meta FAIR
Generating High Quality Synthetic Data for Dutch Medical Conversations
Mar 25, 2026Cecilia Kuan, Aditya Kamlesh Parikh, Henk van den HeuvelSynthetic Data Generation for Language Models Text Generation
Center for Language Studies, Radboud University, Nijmegen, Netherlands
MedAidDialog: A Multilingual Multi-Turn Medical Dialogue Dataset for Accessible Healthcare
Mar 25, 2026Shubham Kumar Nigam, Suparnojit Sarkar, Piyush PatelMedical LLMs Text Generation
University of Birmingham, Dubai, United Arab Emirates · Heritage Institute of Technology, Kolkata, India · Madan Mohan Malaviya University of Technology, India
FinToolSyn: A forward synthesis Framework for Financial Tool-Use Dialogue Data with Dynamic Tool Retrieval
Mar 25, 2026Caishuang Huang, Yang Qiao, Rongyu Zhang +8LLM Tool Use Synthetic Data Generation for Language Models
College of Computer Science and Artificial Intelligence, Fudan University · FiT, Tencent · Zhejiang University
Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAG
Mar 24, 2026Seungju Han, Konwoo Kim, Chanwoo Park +5Knowledge Augmentation for Language Models Synthetic Data Augmentation
Stanford University · MIT · University of Washington
How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data
Mar 23, 2026Zixian Huang, Kaichen Yang, Xu Huang +6Synthetic Data Generation for Language Models Supervised Fine-Tuning
Shanghai AI Laboratory · Dalian University of Technology · Nanjing University
SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injection
Mar 23, 2026Kexian Tang, Jiani Wang, Shaowen Wang +1Knowledge Augmentation for Language Models Synthetic Data Generation for Language Models
Institute for Interdisciplinary Information Sciences, Tsinghua University
Probing How Scalable Table Data Enhances General Long-Context Reasoning
Mar 23, 2026Huaibing Xie, Guoliang Zhao, Yang Liu +8Long-Context Reasoning in Language Models Synthetic Data Generation for Language Models
Large Language Model Department, Tencent · Xi’an Jiaotong University, Xi’an, China · Fudan University, Shanghai, China
DRTriton: Large-Scale Synthetic Data Driven Reinforcement Learning for Triton Kernel Generation
Mar 23, 2026Siqi Guo, Ming Lin, Tianbao YangGPU Kernel Generation GPU Kernel Optimization
Texas A&M University · Oracle
ORACLE: Optimizing Reasoning Abilities of Large Language Models via Constraint-Led Synthetic Data Elicitation
Mar 22, 2026Zhuojie Yang, Wentao Wan, Keze WangSynthetic Data Generation for Language Models LLM Reasoning
School of Computer Science and Engineering, Sun Yat-sen University
Learning to Disprove: Formal Counterexample Generation with Large Language Models
Mar 19, 2026Zenan Li, Zhaoyu Li, Kaiyu Yang +2LLM Mathematical Reasoning Synthetic Data Generation for Language Models
ETH Zurich · University of Toronto · MiroMind +1
HumorGen: Cognitive Synergy for Humor Generation in Large Language Models via Persona-Based Distillation
Mar 19, 2026Edward Ajayi, Prasenjit MitraHumor Explanation Generation Text Generation
Carnegie Mellon University Africa
Interplay: Training Independent Simulators for Reference-Free Conversational Recommendation
Mar 19, 2026Jerome Ramos, Feng Xia, Xi Wang +4Recommender Systems User Simulation
University College London, London, UK · University of Sheffield, Sheffield, UK · Missouri University of Science and Technology, Rolla, USA
Data-efficient pre-training by scaling synthetic megadocs
Mar 19, 2026Konwoo Kim, Suhas Kotha, Yejin Choi +3Language Model Pretraining Synthetic Data Pretraining
Stanford University
Synthetic Data Generation for Training Diversified Commonsense Reasoning Models
Mar 18, 2026Tianhui Zhang, Bei Peng, Danushka BollegalaSynthetic Data Generation for Language Models Synthetic Data Generation for LLM Agents
University of Liverpool · University of Sheffield · Amazon
Argument Reconstruction as Supervision for Critical Thinking in LLMs
Mar 18, 2026Hyun Ryu, Gyouk Chu, Gregor Betz +3Synthetic Data Generation for Language Models
Carnegie Mellon University · KAIST · KIT
Continually self-improving AI
Mar 18, 2026Zitong YangLanguage Model Self-Improvement Self-Training for Language Models
Department of Statistics, Stanford University
TharuChat: Bootstrapping Large Language Models for a Low-Resource Language via Synthetic Data and Human Validation
Mar 17, 2026Prajwal Panth, Agniva MaitiLow-Resource Language Processing Synthetic Data Generation for Language Models
School of Computer Engineering, KIIT Deemed to be University, Bhubaneswar, India

Apr 8, 2026J/K move · Enter open · S save

Dynamic Context Evolution for Scalable Synthetic Data Generation

Ryan Lingo, Rajeev Chhajer

Honda Research Institute, USA, Inc.

Repetition in Language Generation Output Diversity in Language Models LLM Prompting Synthetic Data Generation for Language Models

Open paper AskPDF arXiv

Large language models produce repetitive output when prompted independently across many batches, a phenomenon we term cross-batch mode collapse: the progressive loss of output diversity when a language model is prompted repeatedly without access to its prior generations. Practitioners have long mitigated this with ad hoc deduplication and seed rotation, but no principled framework exists. We introduce Dynamic Context Evolution (DCE), comprising three mechanisms: (1) verbalized tail sampling (the model labels each idea with a guess about how obvious it is, and obvious ideas are discarded), which filters high-probability candidates via model self-assessment; (2) semantic memory, which maintains a persistent embedding index to reject near-duplicates across batches; and (3) adaptive prompt evolution, which reconstructs the generation prompt each batch using memory state and rotating diversity strategies. In experiments across three domains (sustainable packaging concepts, educational exam questions, and creative writing prompts) and two model families (gpt-5-mini and claude-haiku-4-5), a component ablation across 2-3 random seeds per method shows that DCE achieves 0.0 +/- 0.0% collapse versus 5.6 +/- 2.0% for naive prompting, while producing 17-18 HDBSCAN clusters per seed versus naive's volatile 2-17, indicating reliably richer conceptual structure. These results are validated with an independent embedding model (all-MiniLM-L6-v2) and hold across sensitivity sweeps of the VTS threshold tau and dedup threshold delta. Deduplication and prompt evolution are individually insufficient but jointly effective, at approximately $0.50 per 1,000 candidates using only standard API calls, with no fine-tuning or custom architectures required.