Synthetic Data Pretraining

Overview

The field studies how models can learn useful representations, behaviors, or task-solving strategies from generated examples before being adapted to real tasks. Common approaches include physics-based simulation and procedural generation, synthetic text or code derived from source material, and prior-fitted models trained on generated tasks such as tables, graphs, and time series. Results show that synthetic pretraining can improve data efficiency and transfer, especially in specialized or low-data settings, but gains depend on the generator, curriculum, and match between training tasks and deployment. A central tension is how to scale synthetic data without teaching misleading patterns, losing diversity, or mistaking strong synthetic-benchmark performance for real-world generalization; many settings still benefit from real-data fine-tuning or careful verification.

Latest papers

CardsList

130 papers15 papers in the last four weeks, down 37% on the four weeks before. 0.1% of all new papers.

  1. Synthetic Crystal Data Generation for Diffusion Pretraining in Material Property Prediction

    Nov 17, 2026Aamod Varma, Shuyi Jia, Victor FungSynthetic Data PretrainingMaterials Property Prediction

  2. Does synthetic pre-training pay off on a microcontroller? Accuracy, latency and energy across two MCUs

    Nov 15, 2026Achintya YadavTiny MLSynthetic Data Pretraining

  3. Measurement-Calibrated Synthetic Pretraining for Rapid Adaptation of UAV Channel Predictors

    Nov 12, 2026Qiuheng Zhou, Wei Jiang, Donglin Wang +1Channel State PredictionWireless Communications

  4. Shaping Useful Noise: Energy Distributions Predict Visual Pretraining Quality

    Nov 9, 2026Ching Lam Choi, Antonio Torralba, Phillip Isola +1Synthetic Data PretrainingSynthetic Data Evaluation

  5. SynForecast: Controlled Synthetic Time Series for Forecasting Research

    Nov 2, 2026Olivier Sprangers, Cristian Challu, Jan Rathfelder +2Time Series GenerationSynthetic Data Pretraining

  6. Rethinking Pre-pretraining

    Oct 29, 2026Alexander Stern, Tim Rocktäschel, Acyr Locatelli +1Language Model PretrainingSynthetic Data Pretraining

  7. FDSML: Formula-Driven Supervised Metric Learning with Parameter-Aware Triplet Loss

    Oct 29, 2026Hina Otake, Shinei Arakawa, Keitaro Tanaka +3Metric LearningSynthetic Data Pretraining

  8. Learning a Prior-Data Fitted Network for Graph Foundation Models via Structural Causal Model Priors

    Oct 28, 2026Ziwei Zhang, Tianyin Liao, Chunyu Hu +7Graph Foundation ModelsPrior-Data Fitted Networks

  9. Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints

    Oct 22, 2026Nicholas Roberts, Ahmed Ahmed, Suhas Kotha +4Synthetic Data PretrainingLanguage Model Scaling Laws

  10. SyRA-CS: Synthetic-to-Real Adaptation for Code-Switched ASR with Meta-Learning

    Oct 22, 2026Fadel Thior, Ahmer Tabassum, Haania Siddiqui +2Code-Switching Speech RecognitionSynthetic-to-Real Domain Adaptation

  11. Learning Robot Behavior Priors from Non-Robotic Synthetic Data

    Oct 22, 2026Mateusz Odrowaz-Sypniewski, Michael Amir, Jasmine Bayrooti +2Synthetic Data PretrainingData-Efficient Robot Learning

  12. Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs

    Oct 19, 2026Abhinav Khattar, Aishwarya Chhabra, Ameya Velingker +24Language ModelingLanguage Model Pretraining

  13. Synthetic reaction data from transformation rules for graph-to-sequence pretraining in biodegradation pathway prediction

    Oct 14, 2026Irsyaad Rijwan, Liam Brydon, Aidan Jared +1Chemical Reaction PredictionSynthetic Data Pretraining

  14. Scaling Cheap Synthetic Pretraining Improves Indoor Radio-Map Prediction

    Oct 13, 2026Khoren Petrosyan, Artashes Mkrtchyan, Rafayel Mkrtchyan +2Radio Propagation ModelingSynthetic Data Pretraining

  15. Improving Neural Symbolic Regression with a Diverse Prior and Post-training Alignment

    Oct 13, 2026David Otte, Jörg K. H. Franke, Frank HutterSymbolic RegressionSynthetic Data Pretraining

  16. Is Real-World Training Data Necessary for Generalist Graph Anomaly Detection?

    Oct 8, 2026Yujing Liu, Yixin Liu, Yue Tan +4Anomaly DetectionGraph Foundation Models

  17. Timer-M1: A Multivariate Time Series Foundation Model via Learning Primitives

    Oct 8, 2026Haoran Zhang, Haixuan Liu, Xingjian Su +5Multivariate Time Series ForecastingTime-Series Foundation Models

  18. MotherTree: Meta-learning on synthetic data improves decision tree training

    Oct 7, 2026Ziyuan Wang, Fredrik D. JohanssonDecision Tree LearningMeta-Learning

  19. HuC-VideoMAE: Human-Centric Video Masked Autoencoding from synthetic data

    Oct 6, 2026Ricardo Pizarro, Roberto Valle, José M. Buenaposada +2Synthetic Data PretrainingMasked Autoencoders

  20. TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversity

    Oct 6, 2026Zijian Li, Xiangchen Song, Gongxu Luo +7Synthetic Data PretrainingTabular Foundation Models

  21. A General Pipeline for Dense Illuminant Estimation via Physically Based Synthetic Data

    Oct 5, 2026Luca Cogo, Gianmarco Corti, Simone Bianco +1Synthetic Data PretrainingComputational Imaging