Synthetic Data Pretraining

Ask about this topic

Overview

The field studies how models can learn useful representations, behaviors, or task-solving strategies from generated examples before being adapted to real tasks. Common approaches include physics-based simulation and procedural generation, synthetic text or code derived from source material, and prior-fitted models trained on generated tasks such as tables, graphs, and time series. Results show that synthetic pretraining can improve data efficiency and transfer, especially in specialized or low-data settings, but gains depend on the generator, curriculum, and match between training tasks and deployment. A central tension is how to scale synthetic data without teaching misleading patterns, losing diversity, or mistaking strong synthetic-benchmark performance for real-world generalization; many settings still benefit from real-data fine-tuning or careful verification.

Latest papers

CardsList

122 papers27 papers in the last four weeks, up 50% on the four weeks before. 0.2% of all new papers.

  1. Mind the Gap? A Distributional Comparison of Real and Synthetic Priors for Tabular Foundation Models

    May 7, 2026Alex O. Davies, Telmo de Menezes e Silva Filho, Nirav AjmeriSynthetic Data PretrainingSynthetic-to-Real Domain Adaptation

    School of Geographical Sciences University of Bristol, UK · School of Engineering Mathematics and Technology University of Bristol, UK · School of Computer Science University of Bristol, UK

  2. Distribution-Free Pretraining of Classification Losses via Evolutionary Dynamics

    May 5, 2026Meng Xiang, Yan PeiSynthetic Data PretrainingImage Classification

    School of Computer Science and Engineering University of Aizu Aizuwakamatsu, Japan

  3. Learning Posterior Predictive Distributions for Node Classification from Synthetic Graph Priors

    Apr 21, 2026Jeongwhan Choi, Jongwoo Kim, Woosung Kang +1Synthetic Data PretrainingOOD Generalization

    KAIST Daejeon, Republic of Korea

  4. How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data

    Apr 15, 2026Joel Niklaus, Atsuki Yamaguchi, Michal Štefánik +9Synthetic Data PretrainingSynthetic Data Generation for Language Models

    Hugging Face · University of Sheffield, United Kingdom · National Institute of Informatics, Japan

  5. Epidemiology of Model Collapse: Modeling Synthetic Data Contamination via Bilayer SIR Dynamics

    Apr 14, 2026Xiangyu WangModel CollapseData Contamination in Language Models

  6. SynFlow: Scaling Up LiDAR Scene Flow Estimation with Synthetic Data

    Apr 10, 2026Qingwen Zhang, Xiaomeng Zhu, Chenhan Jiang +1Synthetic Data PretrainingSynthetic Data Generation

    KTH Royal Institute of Technology, Stockholm, Sweden · Hong Kong University of Science and Technology, Hong Kong, China

  7. WRAP++: Web discoveRy Amplified Pretraining

    Apr 8, 2026Jiang Zhou, Yunhao Wang, Xing Wu +2Knowledge Augmentation for Language ModelsSynthetic Data Pretraining

  8. Dynamic Linear Coregionalization for Realistic Synthetic Multivariate Time Series

    Apr 6, 2026Annita Vapsi, Penghang Liu, Saheed Obitayo +8Synthetic Data GenerationTime Series Generation

    JPMorganChase

  9. Survival In-Context: Amortized Bayesian Survival Analysis via Prior-Fitted Networks

    Mar 31, 2026Dmitrii Seletkov, Paul Hager, Georgios Kaissis +3Survival AnalysisPrior-Data Fitted Networks

    Institute of Diagnostic and Interventional Radiology, Technical University of Munich, Germany · Chair for AI in Healthcare and Medicine, Technical University of Munich, Germany · Hasso Plattner Institute for Digital Engineering, University of Potsdam, Germany +3

  10. The Surprising Effectiveness of Noise Pretraining for Implicit Neural Representations

    Mar 30, 2026Kushal Vyas, Alper Kayabasi, Daniel Kim +3Implicit Neural RepresentationsNeural Network Initialization

    Rice University · University of California, Riverside

  11. Comparing Natural and Synthetic Structured Data: A Study of the Passive Verb Alternation in French and Italian

    Mar 26, 2026Giuseppe Samo, Paola MerloSynthetic Data EvaluationSynthetic Data Pretraining

    IDIAP Research Institute · University of Geneva

  12. CORA: Generalizable coronary artery disease assessment and risk stratification from coronary CT angiography using pathology-centric representation learning

    Mar 25, 2026Jinkui Hao, Gorkem Durak, Halil Ertugrul Aktas +4Synthetic Data PretrainingCoronary Angiography

    Department of Radiology, Northwestern University, Chicago, IL, USA · Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA · Department of Cardiology, Northwestern University, Chicago, IL, USA +1

  13. Data-efficient pre-training by scaling synthetic megadocs

    Mar 19, 2026Konwoo Kim, Suhas Kotha, Yejin Choi +3Language Model PretrainingSynthetic Data Pretraining

    Stanford University

  14. Continually self-improving AI

    Mar 18, 2026Zitong YangLanguage Model Self-ImprovementSelf-Training for Language Models

    Department of Statistics, Stanford University

  15. Training Language Models via Neural Cellular Automata

    Mar 9, 2026Dan Lee, Seungwook Han, Akarsh Kumar +1Language Model PretrainingSynthetic Data Pretraining

    Independent Contributor · MIT · Improbable AI Lab

  16. Engineering Regression Without Real-Data Training: Domain Adaptation for Tabular Foundation Models Using Multi-Dataset Embeddings

    Mar 5, 2026Lyle Regenwetter, Rosen Yu, Cyril Picard +1Synthetic Data PretrainingSynthetic-to-Real Domain Adaptation

    Massachusetts Institute of Technology

  17. Relational In-Context Learning via Synthetic Pre-training with Structural Prior

    Mar 4, 2026Yanbo Wang, Jiaxuan You, Chuan Shi +1Relational ICLRelational Foundation Models

    Institute for Artificial Intelligence, Peking University · University of Illinois at Urbana-Champaign · Institute of Computing Technology, Beijing University of Post and Telecommunication +1

  18. Reasoning Core: A Scalable Procedural Data Generation Suite for Symbolic Pre-training and Post-Training

    Mar 2, 2026Valentin Lacombe, Valentin Quesnel, Damien SileoSynthetic Data Generation for Language ModelsSynthetic Data Pretraining

    Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 - CRIStAL, F-59000 Lille, France

  19. Fake It Right: Injecting Anatomical Logic into Synthetic Supervised Pre-training for Medical Segmentation

    Mar 1, 2026Jiaqi Tang, Mengyan Zheng, Shu Zhang +2Medical Image SegmentationSynthetic Data Pretraining

    Peking University · Beihang University · Deepwise Ltd.

  20. GeoPT: Scaling Physics Simulation via Lifted Geometric Pre-Training

    Feb 23, 2026Haixu Wu, Minghao Guo, Zongyi Li +4Physics-Based SimulationNeural Surrogate Modeling

    MIT CSAIL · Tsinghua University

  21. MantisV2: Closing the Zero-Shot Gap in Time Series Classification with Synthetic Data and Test-Time Strategies

    Feb 19, 2026Vasilii Feofanov, Songkang Wen, Jianfeng Zhang +2Time Series ClassificationTime-Series Foundation Models

    Huawei Noah’s Ark Lab

  22. Universal priors: solving empirical Bayes via Bayesian inference and pretraining

    Feb 16, 2026Nick Cannella, Anzo Teh, Yanjun Han +1Synthetic Data PretrainingEmpirical Bayes

    Courant Institute of Mathematical Sciences, NYU, New York City, NY · Department of EECS, MIT, Cambridge, MA · Courant Institute of Mathematical Sciences and Center for Data Science, NYU, New York City, NY

  23. TabICLv2: A better, faster, scalable, and open tabular foundation model

    Feb 11, 2026Jingang Qu, David Holzmüller, Gaël Varoquaux +1Synthetic Data PretrainingTabular Foundation Models

    SODA Team, INRIA Saclay, Palaiseau, France · Probabl, France

  24. PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models

    Feb 3, 2026Vignesh Kothapalli, Rishabh Ranjan, Valter Hudovernik +4Relational Foundation ModelsSynthetic Data Pretraining

    1Stanford University · 2Kumo AI · 3SAP Labs LLC

  25. FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

    Jan 29, 2026Ajay Patel, Colin Raffel, Chris Callison-BurchLanguage Model PretrainingSynthetic Data Pretraining

    University of Pennsylvania · University of Toronto · Vector Institute +1

  26. ECGFlowCMR: Pretraining with ECG-Generated Cine CMR Helps Cardiac Disease Classification and Phenotype Prediction

    Jan 28, 2026Xiaocheng Fang, Zhengyao Ding, Guangkun Nie +9Synthetic Data PretrainingMedical Diagnosis

    State Key Laboratory of General Artificial Intelligence, Peking University · School of Intelligence Science and Technology, Peking University, Beijing, China · Polytechnic Institute of Zhejiang University, Zhejiang University, Hangzhou, China +4

  27. Long-Range Distillation: Distilling 10,000 Years of Simulated Climate into Long Timestep AI Weather Models

    Dec 28, 2025Scott A. Martin, Noah Brenowitz, Dale Durran +1Weather ForecastingSpatiotemporal Forecasting

    NVIDIA Research, Santa Clara, CA, USA · School of Oceanography, University of Washington, Seattle, WA, USA · Department of Atmospheric & Climate Science, University of Washington, Seattle, WA, USA

  28. Towards Scalable Meta-Learning of near-optimal Interpretable Models via Synthetic Model Generations

    Nov 6, 2025Kyaw Hpone Myint, Zhe Wu, Alexandre G. R. Day +1Decision Tree LearningMeta-Learning

    Capital One

  29. Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence

    Oct 18, 2025Bingji Yi, Qiyuan Liu, Yuwei Cheng +1Synthetic Data PretrainingModel Collapse

    Independent Researcher. Work done while visiting UChicago CS. · Work done while visiting UChicago CS. · Department of Statistics, University of Chicago. +1

  30. GraphPFN: A Prior-Data Fitted Graph Foundation Model

    Sep 25, 2025Dmitry Eremeev, Oleg Platonov, Gleb Bazhenov +2Synthetic Data PretrainingIn-Context Learning

    1HSE University · 2Yandex Research