Synthetic Data Pretraining

Follow this topic Ask about this topic

Momentum

17 papers in the last four weeks, up 240% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 82

All topics

The Fairness Collapse Phenomenon: Bias Amplification in Language Models Trained on Synthetic Data
Aug 4, 2026Irina Proskurina, Antoine Gourru, Julien VelcinSynthetic Data Pretraining Self-Training for Language Models
Laboratoire Hubert Curien, UMR CNRS 5516, Saint-Étienne, France · Université Claude Bernard Lyon 1, Université Lumière Lyon 2, ERIC · École Centrale de Lyon, LIRIS, CNRS UMR 5205
PluRel-to-RDB-PFN: Schema-Guided Synthetic Relational Pretraining
Jul 31, 2026Mohammad Sadeq Abolhasani, Viswanath GanapathyRelational Foundation Models Synthetic Data Pretraining
SAP Labs, LLC., Palo Alto 94304, United States.
Curriculum Matters: Data-Efficient Relational PFN Pretraining with Synthetic Data
Jul 31, 2026Mohammad Sadeq Abolhasani, Viswanath GanapathySynthetic Data Pretraining Synthetic Data Generation
SAP Labs, LLC. Palo Alto, CA, United States
Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training
Jul 30, 2026Jiawen Tao, Miao Peng, Yaoming Li +7Language Model Pretraining Synthetic Data Pretraining
Hunyuan Team, Tencent · Peking University · The Hong Kong University of Science and Technology (Guangzhou)
PerturbPFN: Probing the Limits of Synthetic Priors in Drug Perturbation Modelling
Jul 26, 2026Yuche Gao, José Miguel Hernández-Lobato, Siyuan GuoSynthetic Data Pretraining Structural Causal Models
University of Cambridge, Cambridge, United Kingdom · Prior Labs, Freiburg, Germany
Patient-Agnostic Synthetic Pretraining for Efficient Patient-Specific Intraoperative 2D/3D Registration
Jul 25, 2026Minheng Chen, Youyong KongSynthetic Data Pretraining Medical Image Registration
Department of Computer Science and Engineering, University of Texas at Arlington, Arlington, 76013, TX, USA. · School of Computer Science and Engineering, Southeast University, Nanjing, 210096, Jiangsu, China.
mmSimPrior: Learning Simulation Priors for Data-Efficient and Generalizable Real-World Radar-based Human Motion Reconstruction
Jul 25, 2026Cheng Guo, Qiming Cao, Shengkai Xu +4Synthetic Data Pretraining Sim-to-Real Transfer
University of North Carolina at Charlotte, Charlotte, NC, USA · Purdue University, West Lafayette, IN, USA
SynPre-FL: Synthetic data-driven pretraining integrated Federated Learning training framework
Jul 21, 2026Akarsh K Nair, Muhammad Arifur Rahman, Nicholas Shopland +8Synthetic Data Pretraining Clinical Outcome Prediction
Department of Computer Science, Nottingham Trent University, Nottingham, United Kingdom · Division of Epidemiology and Public Health, University of Nottingham, Nottingham, United Kingdom · Information and Computer Science Department, King Fahd University of Petroleum and Minerals, Dhahran, Saudi Arabia +1
Demographically-Conditioned Synthetic Medical Images for Bias Mitigation and Bias Detection in Disease Classifiers
Jul 16, 2026Mahmoud Ibrahim, Bart Elen, Chang Sun +2Synthetic Data Pretraining Algorithmic Fairness
Institute of Data Science, Faculty of Science and Engineering, Maastricht University, Maastricht, The Netherlands · Department of Advanced Computing Sciences, Faculty of Science and Engineering, Maastricht University, Maastricht, The Netherlands · VITO, Belgium
TimEE: End-to-end Time Series Classification via In-Context Learning
Jul 8, 2026Jaris Küken, Shi Bin Hoo, Martin Mráz +2Synthetic Data Pretraining Time Series Classification
University of Freiburg · Zuse School ELIZA Darmstadt · Prior Labs +1
RMISC: A Large-scale Real-world Multivariate Corpus for Time Series Foundation Models
Jul 7, 2026Qian Sun, Yong-Ming Tian, Jia-Wei Huang +2Synthetic Data Pretraining Multivariate Time Series Forecasting
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · School of Intelligent Science and Technology, Nanjing University, Suzhou, China · Nanjing University – Siemens Joint Research Center on Industrial AI, Suzhou, China +1
Mitigating Domain Shift in Conditioned Floor Plan Generation: Synthetic Pre-training for Data-Efficient Adaptation
Jul 7, 2026Matthieu Ospici, Arnaud Gueze, Luc Bourrat +1Synthetic Data Pretraining Synthetic-to-Real Domain Adaptation
Homiwoo, 8 rue la Boétie, 75008 Paris, France · École Polytechnique, Route de Saclay, 91128 Palaiseau, France
MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languages
Jul 1, 2026Maximilian Idahl, Jörg Tiedemann, Sampo Pyysalo +19Language Model Pretraining Synthetic Data Pretraining
ellamind · Leibniz University Hannover · University of Helsinki +14
Scaling LLM Knowledge Boundaries via Distribution-Optimized Synthesis
Jun 22, 2026Songze Li, Yarong Lan, Zhongpu Bo +16Synthetic Data Pretraining Knowledge Augmentation for Language Models
Zhejiang University · ZJU-Ant Group Joint Lab of Knowledge Graph · Ant Group
A large-scale foundation model enables simulation-to-real adaptation for nuclear magnetic resonance-based molecular structure analysis
Jun 18, 2026Chen Yang, Zheng Fang, Hanyu Sun +7Synthetic Data Pretraining Synthetic-to-Real Domain Adaptation
The Hong Kong University of Science and Technology (Guangzhou) · Peking Union Medical College and Chinese Academy of Medical Sciences · Xiamen University +3
Towards Engineering Scaling Laws with Pretraining Data Composition
Jun 18, 2026Jan-Lucas Uslu, Kevin Greif, Daniel Whiteson +1Synthetic Data Pretraining High-Energy Physics
Department of Applied Physics Stanford University Stanford, CA 94305, USA · Department of Physics and Astronomy University of California, Irvine Irvine, CA 92697, USA · Department of Particle Physics and Astrophysics SLAC National Accelerator Laboratory Stanford University Menlo Park, CA 94025, USA
Reinforcement Learning Foundation Models Should Already Be A Thing
Jun 17, 2026Abdelrahman Zighem, Jill-Jênn VieSynthetic Data Pretraining Reinforcement Learning
Ecole normale supérieure de Paris, PSL University, Paris, France · Soda team, Inria Saclay, Palaiseau, France.
When Sample Selection Bias Precipitates Model Collapse
Jun 11, 2026Xinbao Qiao, Xianglong Du, Wei Liu +4Synthetic Data Pretraining Model Collapse
National University of Singapore (Chongqing) Research Institute · Chongqing Key Laboratory of Trusted Perception and Interaction Technology for Intelligent and Connected Vehicles, Chongqing, China · The Chinese University of Hong Kong +4
From Physics to Representation: Audio Learning with Synthetic Pre-training via Procedural Generation
Jun 11, 2026Fengrui Liu, Ruiyang Huang, Qijian Zheng +2Synthetic Data Pretraining Audio Representation Learning
East China Normal University · Southeast University · Fudan University +1
Boosting ECG Classification Performance by Pre-training with Synthesized Data
Jun 9, 2026Naoki Nonaka, Jun SeitaSynthetic Data Pretraining Synthetic-to-Real Domain Adaptation
Advanced Data Science Project, RIKEN Information R&D and Strategy Headquarters
CITRAS-FM: Tiny Time Series Foundation Model for Covariate-Informed Zero-Shot Forecasting
Jun 9, 2026Yosuke Yamaguchi, Issei Suemitsu, Yuki Kajihara +1Synthetic Data Pretraining Zero-Shot Time Series Forecasting
Research & Development Group, Hitachi Ltd., Tokyo, Japan
CodeAlchemy: Synthetic Code Rewriting at Scale
Jun 8, 2026Ankit Gupta, Aditya Prasad, Rameswar PandaSynthetic Data Pretraining Synthetic Data Generation
MIT-IBM Watson AI Lab, IBM Research
In-Context Learning for Latent Space Bayesian Optimization
Jun 8, 2026Tuan A. Vu, Harri Lähdesmäki, Julien MartinelliSynthetic Data Pretraining Molecular Optimization
Aalto University
Active Flow Expansion for Out-of-Distribution Discovery: from Theory to Molecules
Jun 7, 2026Riccardo De Santi, Bruce Lee, Cristian Perez Jensen +6Synthetic Data Pretraining Normalizing Flows
ETH Zurich ETH AI Center · University of Pennsylvania · Caltech FutureHouse
Mix, Don't Pick: Why Synthetic Corpus Composition Matters for Time Series Foundation Model Pretraining
Jun 6, 2026Aaryan Nagpal, Debdeep Sanyal, Murari Mandal +2Synthetic Data Pretraining Time Series Forecasting
Birla AI Labs, Mumbai, India · KIIT, Bhubaneswar, India · BITS Pilani, Pilani, India
Trio: Learning Time-Series Forecasting with Temporal-Spatial-Sample Attention and Structural Causal Priors
Jun 5, 2026Tao Chen, Yexu Zhou, Zhi Gong +9Synthetic Data Pretraining Multivariate Time Series Forecasting
SUPCON Technology Co., Ltd. Hangzhou, China · Hangzhou Dianzi University, HangZhou, 310018, China · Xidian University, Xi’an 710071, Shaanxi, China.
In-Context Multiple Instance Learning
Jun 4, 2026Alexander Möllers, Marvin Sextro, Julius Hense +2Synthetic Data Pretraining Few-Shot Learning
Berlin Institute for the Foundations of Learning and Data, Berlin, Germany · Machine Learning Group, Technische Universität Berlin, Berlin, Germany · Aignostics, Berlin, Germany +3
OpenRFM: Dissecting Relational In-Context Learning
Jun 3, 2026Zhikai Chen, Junyu Yin, Jialiang Gu +5Relational Foundation Models Synthetic Data Pretraining
Michigan State University · George Mason University · Georgia Institute of Technology +1
KletterMix: Climbing Toward High-Quality German Pretraining Data
Jun 2, 2026Maurice Kraus, Ruben Härle, Sebastian Sztwiertnia +4Language Model Pretraining Synthetic Data Pretraining
AI & ML Group, TU Darmstadt · Lab1141 · Fraunhofer IAIS +3
TabCausal: Pretraining Across Causal Environments for Tabular Causal Discovery
May 29, 2026Zi-Rong Li, Si-Yang Liu, Tian-Zuo Wang +1Causal Foundation Models Synthetic Data Pretraining
Nanjing University

Aug 4, 2026, cs.CLJ/K move · Enter open · S save

The Fairness Collapse Phenomenon: Bias Amplification in Language Models Trained on Synthetic Data

Irina Proskurina, Antoine Gourru, Julien Velcin

Laboratoire Hubert Curien, UMR CNRS 5516, Saint-Étienne, France · Université Claude Bernard Lyon 1, Université Lumière Lyon 2, ERIC · École Centrale de Lyon, LIRIS, CNRS UMR 5205

Synthetic Data Pretraining Self-Training for Language Models Social Bias in Language Models Language Model Bias Evaluation Linguistic Bias in Language Models

Open paper AskPDF arXiv

Generative models trained on artificially generated data have been shown to exhibit model collapse, resulting in significant performance degradation. As synthetic content increasingly contaminates the training corpora of language models, this raises critical concerns about the use of open data in continued pretraining. Although previous work has demonstrated model collapse in language models, it remains unclear whether exposure to synthetic data amplifies or attenuates the social biases already present in pretrained models. Because language models are known to reproduce and amplify demographic stereotypes, recursive training on self-generated data may create a self-reinforcing feedback loop in which biased associations become progressively stronger across generations. We call this hypothesized phenomenon fairness collapse. In this work, we construct controlled training regimes in which models are repeatedly trained on synthetic data using the Bias in Bios dataset. Across experiments, we observe a consistent and concerning pattern: fairness degradation emerges before substantial degradation is reflected by standard language-modeling metrics. This result highlights a critical risk associated with synthetic data contamination in language model training: bias can increase silently before strong indicators of model collapse become apparent.

Momentum

Related topics

Latest papers 82