Synthetic Data Pretraining

Ask about this topic

Momentum

17 papers in the last four weeks, up 240% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 82

All topics
CardsList

  1. Is Real-World Training Data Necessary for Generalist Graph Anomaly Detection?

    Oct 8, 2026Yujing Liu, Yixin Liu, Yue Tan +4Anomaly DetectionGraph Foundation Models

    School of Information and Communication Technology, Griffith University, Australia · School of Computer Science and Technology, Tongji University, China

  2. Timer-M1: A Multivariate Time Series Foundation Model via Learning Primitives

    Oct 8, 2026Haoran Zhang, Haixuan Liu, Xingjian Su +5Multivariate Time Series ForecastingTime-Series Foundation Models

    School of Software, BNRist, Tsinghua University · ByteDance, Beijing, China

  3. Conditional Transfer from Controlled Pretraining Mixtures to Code

    Oct 8, 2026Ohad RubinSynthetic Data PretrainingData Mixture Optimization

  4. MotherTree: Meta-learning on synthetic data improves decision tree training

    Oct 7, 2026Ziyuan Wang, Fredrik D. JohanssonDecision Tree LearningMeta-Learning

    Chalmers University of Technology University of Gothenburg

  5. HuC-VideoMAE: Human-Centric Video Masked Autoencoding from synthetic data

    Oct 6, 2026Ricardo Pizarro, Roberto Valle, José M. Buenaposada +2Synthetic Data PretrainingMasked Autoencoders

    Universidad de Alcal´a, Alcal´a de Henares, Spain · Universidad Polit´ecnica de Madrid, Madrid, Spain · Universidad Rey Juan Carlos, M´ostoles, Spain

  6. TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversity

    Oct 6, 2026Zijian Li, Xiangchen Song, Gongxu Luo +7Synthetic Data PretrainingTabular Foundation Models

    Carnegie Mellon University · Mohamed bin Zayed University of Artificial Intelligence · Guangdong University of Technology +1

  7. A General Pipeline for Dense Illuminant Estimation via Physically Based Synthetic Data

    Oct 5, 2026Luca Cogo, Gianmarco Corti, Simone Bianco +1Synthetic Data PretrainingComputational Imaging

    Department of Informatics, Systems and Communication, University of Milano – Bicocca, Milan 20126, Italy

  8. Geometry Meets Physics: Data-Efficient Pre-Training for Unstructured Neural PDE Solvers

    Oct 2, 2026Luis Medrano-Navarro, Giacomo Baldan, Qiang Liu +4Synthetic Data PretrainingNeural PDE Solvers

    Technical University of Munich · University of Stuttgart

  9. Effective Synthetic Data Curation Requires Group-Level Signals

    Sep 30, 2026Cathy Jiao, Chenyan XiongSynthetic Data PretrainingTraining Data Curation

    Language Technologies Institute, Carnegie Mellon University

  10. How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text

    Sep 30, 2026Jenna Russell, Ben Glickenhaus, Katherine Thai +4Language Model PretrainingSynthetic Data Pretraining

    University of Maryland · Pangram Labs

  11. Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior

    Sep 30, 2026Atsuki Yamaguchi, Tatsuro Inaba, Joel Niklaus +3Language Model PretrainingSynthetic Data Pretraining

    University of Sheffield, UK · Mohamed bin Zayed University of Artificial Intelligence, UAE · Hugging Face +3

  12. It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs

    Sep 29, 2026Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois +7Language Model PretrainingSynthetic Data Pretraining

    PleIAs · Sorbonne Center for Artificial Intelligence · Sciences Po Médialab +6

  13. Self-Play Pretraining with Zero Data

    Sep 24, 2026Aditya Cowsik, Kfir Dolev, Michael Y. Li +4Synthetic Data PretrainingSynthetic Data Generation for Language Models

    Independent Researcher · Tel Aviv University · Stanford University +1

  14. TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling

    Sep 22, 2026Julien Knafou, Luc Mottin, Anaïs Mottaz +2Synthetic Data PretrainingLanguage Modeling

    HES-SO, Geneva, Switzerland · SIB, Swiss Institute of Bioinformatics, Geneva, Switzerland

  15. Learning to Fluctuate: Statistical Foundations for Causal Tabular Pretraining

    Sep 22, 2026Zhiheng ZhangCausal Foundation ModelsSynthetic Data Pretraining

    School of Statistics and Data Science Shanghai University of Finance and Economics

  16. QVAC Genesis III: A Large-Scale, High-Quality Open Synthetic STEM Corpus for Efficient Language Model Pre-Training

    Sep 17, 2026Davide Vitabile, N. Ranjan, Akshay Nambiar +2Language Model PretrainingSynthetic Data Pretraining

    Tether Data, S.A. de C.V. d.b.a. Tether AI Research

  17. Preventing Model Collapse: A Fisher-Rao Perspective on the Dynamics of Training with Synthetic Data

    Sep 16, 2026Matteo Marchi, João Pedro Silvestre, Bahman Gharesifard +1Synthetic Data PretrainingModel Collapse

    Electrical and Computer Engineering Department, University of California at Los Angeles, Los Angeles, CA 90095 USA · Department of Mathematics and Statistics at Queens University, Kingston, ON, Canada

  18. MiST: Mid-Training LLMs for Cybersecurity

    Sep 16, 2026Oded Ovadia, Elad Ben Zaken, Elad Guttman +1Synthetic Data PretrainingDomain-Adaptive Pretraining

    Dream

  19. Procedural Pretraining for Molecular Property Prediction

    Sep 15, 2026Moritz Friedemann, Zachary Shinnick, Philip Torr +1Synthetic Data PretrainingSelf-Supervised Pre-Training

    University of Oxford · Adelaide University · Slater Labs

  20. Measuring Annotation Efficiency for Handwritten Devanagari Recognition: Sample-Complexity Curves for Four Pretraining Regimes

    Sep 15, 2026Manglesh Kumar Pandey, Sumit Kumar BanshalSynthetic Data PretrainingOptical Character Recognition

    Alliance School of Liberal Arts and Sciences, Alliance University Bengaluru, India · Department of Computer Science and Engineering, Alliance University Bengaluru, India

  21. Rethinking Procedural Audio Pre-training: Source Scaling and Objective Adaptation

    Sep 14, 2026Jiajun Peng, Fengrui Liu, Xinyu Liu +1Synthetic Data PretrainingAudio Representation Learning

    School of Data Science, University of Science and Technology of China, Hefei 230026, China · School of Computer Science and Technology, East China Normal University, Shanghai 200062, China · School of Psychology, Shanghai Jiao Tong University, Shanghai 200030, China

  22. Physics-Guided Synthetic High-Frequency Ultrasound Generation for Skin Layer Segmentation

    Sep 14, 2026Junkyung ju, Kyungho Yoon, Minwoo ShinSynthetic Data PretrainingUltrasound Image Segmentation

    Department of Software, Yonsei University (Mirae Campus), Wonju, 26493, Republic of Korea · School of Mathematics and Computing, Yonsei University, Seoul, 03722, Republic of Korea

  23. Learning with Synthetic Data via SGD in High-Dimensional Linear Regression

    Sep 10, 2026Jichu li, Difan ZouSynthetic Data PretrainingModel Collapse

    Renmin University of China · The University of Hong Kong

  24. Distillation of Synthetic Data for Time Series Foundation Models

    Sep 9, 2026Niloy Biswas, Noureddine El KarouiSynthetic Data PretrainingTime-Series Foundation Models

    Meta AI

  25. Xiaomi-TabLDM: A Tabular Foundation Model Technical Report

    Sep 3, 2026Xiaomi-TabLDM Team, :, Penghui Wang +10Synthetic Data PretrainingTabular Foundation Models

    Xiaomi

  26. Synthetic Semantic Supervision for Contrastive Code Representation Learning in Small Transformers: An Empirical Study

    Sep 3, 2026Kenneth Paulsen, Florian Tambon, Mike Papadakis +1Synthetic Data PretrainingContrastive Learning

    University of Luxembourg (SnT) · School of Computing, KAIST

  27. Synthetic Worlds for Temporal Evaluation and Knowledge Updating in LLMs

    Aug 31, 2026Jonathan Zheng, Zirui Shao, Alan Ritter +1Synthetic Data PretrainingKnowledge Editing

    Georgia Institute of Technology · Zhejiang University

  28. REER-PT: Reverse-Engineered Reasoning for Perplexity-Guided Pre-training Data Augmentation

    Aug 31, 2026Haoran Que, Jiajun Shi, Ting Huang +7Language Model PretrainingSynthetic Data Pretraining

    Peking University · ByteDance Seed · Work done at ByteDance Seed +1

  29. Synthetic Persona Pretraining: Alignment from Token Zero

    Aug 13, 2026Julian Minder, Viktor Moskvoretskii, Raghav Singhal +12Synthetic Data PretrainingLLM Alignment

    EPFL · SJTU · Northeastern University +8

  30. Patient Pose Assessment Using a CT-Based Framework for Synthetic Data Generation

    Aug 6, 2026Manuel Laufer, Dominik Mairhöfer, Malte Sieren +7Synthetic Data PretrainingMedical Imaging

    Institute for Neuro- and Bioinformatics, University of Lübeck, Germany · University Medical Center Schleswig-Holstein, Lübeck, Germany · IMAGE Information Systems Europe GmbH, Rostock, Germany +1