Synthetic Data Pretraining

Follow this topic Ask about this topic

Momentum

17 papers in the last four weeks, up 240% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 82

All topics

Not All Synthetic Data Is Yours to Learn From
May 29, 2026Sina Alemohammad, Li Chen, Richard G. Baraniuk +1Synthetic Data Pretraining Memorization in Language Models
ECE Department The University of Texas at Austin · Apple · ECE Department Rice University
Shaping the Prior: How Synthetic Task Distributions Determine Tabular Foundation Model Quality
May 18, 2026Mohamed Bouadi, Nassim Bouarour, Varun Kulkarni +3Synthetic Data Pretraining Distribution Shift Robustness
Lexsi Labs
Generating Pretraining Tokens from Organic Data for Data-Bound Scaling
May 18, 2026Zichun Yu, Chenyan XiongLanguage Model Pretraining Synthetic Data Pretraining
Language Technologies Institute, Carnegie Mellon University · Xlue
Language Acquisition Device in Large Language Models
May 16, 2026Masato Mita, Taiga Someya, Ryo Yoshida +1Language Model Pretraining Synthetic Data Pretraining
The University of Tokyo
A Data Efficiency Study of Synthetic Fog for Object Detection Using the Clear2Fog Pipeline
May 12, 2026Mohamed Ahmed Mohamed, Xiaowei HuangSynthetic Data Pretraining Synthetic-to-Real Domain Adaptation
School of Computer Science and Informatics, University of Liverpool, Liverpool, L69 3DR, UK
Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data
May 11, 2026Xu Guo, Runyu Peng, Jian Tong +4Language Model Pretraining Synthetic Data Pretraining
Shanghai AI Laboratory · Fudan University · Shanghai Innovation Institute
An Information-Theoretic Criterion for Efficient Data Synthesis
May 11, 2026Hanyu Li, Zhengqi Sun, Xiaotie DengSynthetic Data Pretraining Synthetic Data Generation
CFCS, School of Computer Science, Peking University, Beijing, China · Department of Information Management, Peking University, Beijing, China
Curated Synthetic Data Doesn't Have to Collapse: A Theoretical Study of Generative Retraining with Pluralistic Preferences
May 8, 2026Ali Falahati, Mohammad Mohammadi Amiri, Kate Larson +1Synthetic Data Pretraining Model Collapse
University of Waterloo · 2Rensselaer Polytechnic Institute.
Mind the Gap? A Distributional Comparison of Real and Synthetic Priors for Tabular Foundation Models
May 7, 2026Alex O. Davies, Telmo de Menezes e Silva Filho, Nirav AjmeriSynthetic Data Pretraining Synthetic-to-Real Domain Adaptation
School of Geographical Sciences University of Bristol, UK · School of Engineering Mathematics and Technology University of Bristol, UK · School of Computer Science University of Bristol, UK
Distribution-Free Pretraining of Classification Losses via Evolutionary Dynamics
May 5, 2026Meng Xiang, Yan PeiSynthetic Data Pretraining Image Classification
School of Computer Science and Engineering University of Aizu Aizuwakamatsu, Japan
Learning Posterior Predictive Distributions for Node Classification from Synthetic Graph Priors
Apr 21, 2026Jeongwhan Choi, Jongwoo Kim, Woosung Kang +1Synthetic Data Pretraining OOD Generalization
KAIST Daejeon, Republic of Korea
CORA: Generalizable coronary artery disease assessment and risk stratification from coronary CT angiography using pathology-centric representation learning
Mar 25, 2026Jinkui Hao, Gorkem Durak, Halil Ertugrul Aktas +4Synthetic Data Pretraining Coronary Angiography
Department of Radiology, Northwestern University, Chicago, IL, USA · Department of Biomedical Engineering, Northwestern University, Evanston, IL, USA · Department of Cardiology, Northwestern University, Chicago, IL, USA +1
Universal priors: solving empirical Bayes via Bayesian inference and pretraining
Feb 16, 2026Nick Cannella, Anzo Teh, Yanjun Han +1Synthetic Data Pretraining Empirical Bayes
Courant Institute of Mathematical Sciences, NYU, New York City, NY · Department of EECS, MIT, Cambridge, MA · Courant Institute of Mathematical Sciences and Center for Data Science, NYU, New York City, NY
TabICLv2: A better, faster, scalable, and open tabular foundation model
Feb 11, 2026Jingang Qu, David Holzmüller, Gaël Varoquaux +1Synthetic Data Pretraining Tabular Foundation Models
SODA Team, INRIA Saclay, Palaiseau, France · Probabl, France
PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models
Feb 3, 2026Vignesh Kothapalli, Rishabh Ranjan, Valter Hudovernik +4Relational Foundation Models Synthetic Data Pretraining
1Stanford University · 2Kumo AI · 3SAP Labs LLC
FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale
Jan 29, 2026Ajay Patel, Colin Raffel, Chris Callison-BurchLanguage Model Pretraining Synthetic Data Pretraining
University of Pennsylvania · University of Toronto · Vector Institute +1
ECGFlowCMR: Pretraining with ECG-Generated Cine CMR Helps Cardiac Disease Classification and Phenotype Prediction
Jan 28, 2026Xiaocheng Fang, Zhengyao Ding, Guangkun Nie +9Synthetic Data Pretraining Medical Diagnosis
State Key Laboratory of General Artificial Intelligence, Peking University · School of Intelligence Science and Technology, Peking University, Beijing, China · Polytechnic Institute of Zhejiang University, Zhejiang University, Hangzhou, China +4
Towards Scalable Meta-Learning of near-optimal Interpretable Models via Synthetic Model Generations
Nov 6, 2025Kyaw Hpone Myint, Zhe Wu, Alexandre G. R. Day +1Decision Tree Learning Meta-Learning
Capital One
Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence
Oct 18, 2025Bingji Yi, Qiyuan Liu, Yuwei Cheng +1Synthetic Data Pretraining Model Collapse
Independent Researcher. Work done while visiting UChicago CS. · Work done while visiting UChicago CS. · Department of Statistics, University of Chicago. +1
GraphPFN: A Prior-Data Fitted Graph Foundation Model
Sep 25, 2025Dmitry Eremeev, Oleg Platonov, Gleb Bazhenov +2Synthetic Data Pretraining In-Context Learning
1HSE University · 2Yandex Research
AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists
Jun 9, 2025Yifei Li, Hanane Nour Moussa, Ziru Chen +16Synthetic Data Pretraining Synthetic Data Generation
Department of Computer Science and Engineering · The Ohio State University · Department of Chemistry +6
AneumoBench: A Source-Linked Benchmark for Synthetic-Geometry Transfer in Aneurysm CFD
May 19, 2025Xigui Li, Yuanye Zhou, Feiyang Xiao +16Neural Surrogate Modeling Synthetic Data Pretraining
Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China · Shanghai Academy of Artificial Intelligence for Science, Shanghai, China · Human Phenome Institute, Fudan University, Shanghai, China +1

May 29, 2026, cs.CLJ/K move · Enter open · S save

Not All Synthetic Data Is Yours to Learn From

Sina Alemohammad, Li Chen, Richard G. Baraniuk, Zhangyang Wang

ECE Department The University of Texas at Austin · Apple · ECE Department Rice University

Synthetic Data Pretraining Memorization in Language Models Self-Training for Language Models Language Model Self-Improvement

Open paper AskPDF arXiv

Can a language model improve from plain text sampled from itself, with no prompts, no teacher, no verifier, and no reward model? Yes, but only when the synthetic corpus is compatible with the student, a relational property of the source-student pair rather than an intrinsic property of the data. We call this the latent capability resurfacing hypothesis: weak self-training can amplify capabilities already present in the pretrained model, but only under this compatibility condition. We study this in the minimal setting of prompt-free unconditional self-training, where base language models are fine-tuned on text generated from the BOS token alone, with no task specification or external supervision. We report three findings. First, synthetic utility is relational rather than intrinsic: self-generated data is the most effective source, same-lineage transfer outperforms stronger but differently trained sources, and cross-family transfer is substantially weaker. Second, common intrinsic proxies fail: neither benchmark-level semantic similarity nor average per-token likelihood under the student predicts which corpora help. Third, this regime produces a surprising byproduct. In controlled Pythia experiments, capability and verbatim memorization decouple: benchmark utility is preserved or improved while held-out exact-match extraction drops by over 95 percent, with no forget set, privacy objective, or targeted unlearning. Together, these results suggest that prompt-free self-training works by amplifying what the student already knows, not by importing structure from the data. They also reveal a regime in which capability and verbatim memorization can be separated without any explicit unlearning objective.

Momentum

Related topics

Latest papers 82