Synthetic Data Pretraining

Ask about this topic

Overview

The field studies how models can learn useful representations, behaviors, or task-solving strategies from generated examples before being adapted to real tasks. Common approaches include physics-based simulation and procedural generation, synthetic text or code derived from source material, and prior-fitted models trained on generated tasks such as tables, graphs, and time series. Results show that synthetic pretraining can improve data efficiency and transfer, especially in specialized or low-data settings, but gains depend on the generator, curriculum, and match between training tasks and deployment. A central tension is how to scale synthetic data without teaching misleading patterns, losing diversity, or mistaking strong synthetic-benchmark performance for real-world generalization; many settings still benefit from real-data fine-tuning or careful verification.

Latest papers

CardsList

122 papers27 papers in the last four weeks, up 50% on the four weeks before. 0.2% of all new papers.

  1. Rethinking Pre-pretraining

    Oct 29, 2026Alexander Stern, Tim Rocktäschel, Acyr Locatelli +1Language Model PretrainingSynthetic Data Pretraining

    MIT · AI Centre, UCL

  2. FDSML: Formula-Driven Supervised Metric Learning with Parameter-Aware Triplet Loss

    Oct 29, 2026Hina Otake, Shinei Arakawa, Keitaro Tanaka +3Metric LearningSynthetic Data Pretraining

    Waseda University · Waseda Research Institute for Science and Engineering · National Institute of Advanced Industrial Science and Technology +1

  3. Learning a Prior-Data Fitted Network for Graph Foundation Models via Structural Causal Model Priors

    Oct 28, 2026Ziwei Zhang, Tianyin Liao, Chunyu Hu +7Graph Foundation ModelsPrior-Data Fitted Networks

    Beihang University Haidian District, Beijing, China · Nankai University Binhai New Area, Tianjin, China · Tsinghua University Haidian District, Beijing, China

  4. Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints

    Oct 22, 2026Nicholas Roberts, Ahmed Ahmed, Suhas Kotha +4Synthetic Data PretrainingLanguage Model Scaling Laws

    Princeton University · Stanford University · Jane Street

  5. SyRA-CS: Synthetic-to-Real Adaptation for Code-Switched ASR with Meta-Learning

    Oct 22, 2026Fadel Thior, Ahmer Tabassum, Haania Siddiqui +2Code-Switching Speech RecognitionSynthetic-to-Real Domain Adaptation

    Mohamed bin Zayed University of Artificially Intelligence

  6. Learning Robot Behavior Priors from Non-Robotic Synthetic Data

    Oct 22, 2026Mateusz Odrowaz-Sypniewski, Michael Amir, Jasmine Bayrooti +2Synthetic Data PretrainingData-Efficient Robot Learning

    Department of Computer Science and Technology, University of Cambridge, UK

  7. Is spectral predictability enough? Testing Ω-matching for synthetic pretraining under distribution shift

    Oct 22, 2026Edric Castel C. Hao, Juan Raphael S. Seña, Neon S. De JesusSynthetic Data PretrainingTime Series Forecasting

    Analog Devices, Inc.

  8. Two Is Enough: Binary Classification Priors Suffice for Multi-Class In-Context Tabular Prediction

    Oct 20, 2026Zhixiong Cheng, Shiyuan Han, Jingjing Suo +1Tabular Foundation ModelsTabular ICL

    Bairong Inc.AI, Beijing, China

  9. Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs

    Oct 19, 2026Abhinav Khattar, Aishwarya Chhabra, Ameya Velingker +24Language ModelingLanguage Model Pretraining

    NVIDIA · AI4Bharat

  10. PRECOCIAL: Pretraining with Code Curriculums via Iterative Search for Accelerated Learning

    Oct 18, 2026Edward Gu, Ali Payani, Kevin EllisSynthetic Data PretrainingSynthetic Data Generation for Language Models

    Cornell University · Cisco Research

  11. Path to Prior-Data Fitted Relational Models via Learned Latent Multi-table Representations

    Oct 16, 2026Mohammad Sadeq Abolhasani, Viswanath GanapathyRelational Foundation ModelsSynthetic Data Pretraining

    SAP Labs

  12. What Do Foundation Models for Market Microstructure Learn? A Controlled Study with Synthetic Limit Order Books

    Oct 14, 2026Fernando AceroLimit Order Book ModelingTime-Series Foundation Models

    J.P. Morgan QTR AI Research, United Kingdom

  13. AFCS: Atomic-Fact-Conditioned Synthesis for Factual Knowledge Pretraining

    Oct 14, 2026Yunhao Wang, Feng Zhang, Tinghao Yu +2Synthetic Data Generation for Language ModelsSynthetic Data Pretraining

    Hunyuan Team, Tencent

  14. Synthetic reaction data from transformation rules for graph-to-sequence pretraining in biodegradation pathway prediction

    Oct 14, 2026Irsyaad Rijwan, Liam Brydon, Aidan Jared +1Chemical Reaction PredictionSynthetic Data Pretraining

    School of Computer Science, University of Auckland, Auckland, New Zealand. · enviPath Limited, Auckland, New Zealand.

  15. Scaling Cheap Synthetic Pretraining Improves Indoor Radio-Map Prediction

    Oct 13, 2026Khoren Petrosyan, Artashes Mkrtchyan, Rafayel Mkrtchyan +2Radio Propagation ModelingSynthetic Data Pretraining

    YerevaNN, Armenia · Yerevan State University, Armenia · Institute of Informatics and Telematics, National Research Council, Italy

  16. Improving Neural Symbolic Regression with a Diverse Prior and Post-training Alignment

    Oct 13, 2026David Otte, Jörg K. H. Franke, Frank HutterSymbolic RegressionSynthetic Data Pretraining

    LMU Munich · Munich Center for Machine Learning (MCML) · University of Freiburg +3

  17. Is Real-World Training Data Necessary for Generalist Graph Anomaly Detection?

    Oct 8, 2026Yujing Liu, Yixin Liu, Yue Tan +4Anomaly DetectionGraph Foundation Models

    School of Information and Communication Technology, Griffith University, Australia · School of Computer Science and Technology, Tongji University, China

  18. Timer-M1: A Multivariate Time Series Foundation Model via Learning Primitives

    Oct 8, 2026Haoran Zhang, Haixuan Liu, Xingjian Su +5Multivariate Time Series ForecastingTime-Series Foundation Models

    School of Software, BNRist, Tsinghua University · ByteDance, Beijing, China

  19. Conditional Transfer from Controlled Pretraining Mixtures to Code

    Oct 8, 2026Ohad RubinSynthetic Data PretrainingData Mixture Optimization

  20. MotherTree: Meta-learning on synthetic data improves decision tree training

    Oct 7, 2026Ziyuan Wang, Fredrik D. JohanssonDecision Tree LearningMeta-Learning

    Chalmers University of Technology University of Gothenburg

  21. HuC-VideoMAE: Human-Centric Video Masked Autoencoding from synthetic data

    Oct 6, 2026Ricardo Pizarro, Roberto Valle, José M. Buenaposada +2Synthetic Data PretrainingMasked Autoencoders

    Universidad de Alcal´a, Alcal´a de Henares, Spain · Universidad Polit´ecnica de Madrid, Madrid, Spain · Universidad Rey Juan Carlos, M´ostoles, Spain

  22. TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversity

    Oct 6, 2026Zijian Li, Xiangchen Song, Gongxu Luo +7Synthetic Data PretrainingTabular Foundation Models

    Carnegie Mellon University · Mohamed bin Zayed University of Artificial Intelligence · Guangdong University of Technology +1

  23. A General Pipeline for Dense Illuminant Estimation via Physically Based Synthetic Data

    Oct 5, 2026Luca Cogo, Gianmarco Corti, Simone Bianco +1Synthetic Data PretrainingComputational Imaging

    Department of Informatics, Systems and Communication, University of Milano – Bicocca, Milan 20126, Italy

  24. Geometry Meets Physics: Data-Efficient Pre-Training for Unstructured Neural PDE Solvers

    Oct 2, 2026Luis Medrano-Navarro, Giacomo Baldan, Qiang Liu +4Synthetic Data PretrainingNeural PDE Solvers

    Technical University of Munich · University of Stuttgart

  25. When Does Synthetic Relational Data Teach Models to Use Relations? Tracing Predictive Structure from Pretraining Data to Model Behavior

    Oct 2, 2026Shivam Dubey, Mohamed Bouadi, Nassim Bouarour +3Relational Foundation ModelsRelational Reasoning

    Lexsi Labs

  26. Diffusion-Based Synthetic Data Pretraining for Enhancing Activity Recognition

    Oct 1, 2026E. Riveros, D. Vega-Oliveros, A. Soriano-Vargas +1Human Activity RecognitionSynthetic Data Pretraining

    Institute of Computing, State University of Campinas, Campinas, Brazil · Institute of Computing State University of Campinas Unicamp Campinas, Brazil · Institute of Science and Technology, Federal University of Sao Paulo, Sao Jose dos Campos, Brazil +3

  27. Effective Synthetic Data Curation Requires Group-Level Signals

    Sep 30, 2026Cathy Jiao, Chenyan XiongSynthetic Data PretrainingTraining Data Curation

    Language Technologies Institute, Carnegie Mellon University

  28. How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text

    Sep 30, 2026Jenna Russell, Ben Glickenhaus, Katherine Thai +4Language Model PretrainingSynthetic Data Pretraining

    University of Maryland · Pangram Labs

  29. Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior

    Sep 30, 2026Atsuki Yamaguchi, Tatsuro Inaba, Joel Niklaus +3Language Model PretrainingSynthetic Data Pretraining

    University of Sheffield, UK · Mohamed bin Zayed University of Artificial Intelligence, UAE · Hugging Face +3

  30. It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs

    Sep 29, 2026Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois +7Language Model PretrainingSynthetic Data Pretraining

    PleIAs · Sorbonne Center for Artificial Intelligence · Sciences Po Médialab +6