Synthetic Data Pretraining

Follow this topic Ask about this topic

Momentum

17 papers in the last four weeks, up 240% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 82

All topics

Is Real-World Training Data Necessary for Generalist Graph Anomaly Detection?
Oct 8, 2026Yujing Liu, Yixin Liu, Yue Tan +4Anomaly Detection Graph Foundation Models
School of Information and Communication Technology, Griffith University, Australia · School of Computer Science and Technology, Tongji University, China
Timer-M1: A Multivariate Time Series Foundation Model via Learning Primitives
Oct 8, 2026Haoran Zhang, Haixuan Liu, Xingjian Su +5Multivariate Time Series Forecasting Time-Series Foundation Models
School of Software, BNRist, Tsinghua University · ByteDance, Beijing, China
Conditional Transfer from Controlled Pretraining Mixtures to Code
Oct 8, 2026Ohad RubinSynthetic Data Pretraining Data Mixture Optimization
MotherTree: Meta-learning on synthetic data improves decision tree training
Oct 7, 2026Ziyuan Wang, Fredrik D. JohanssonDecision Tree Learning Meta-Learning
Chalmers University of Technology University of Gothenburg
HuC-VideoMAE: Human-Centric Video Masked Autoencoding from synthetic data
Oct 6, 2026Ricardo Pizarro, Roberto Valle, José M. Buenaposada +2Synthetic Data Pretraining Masked Autoencoders
Universidad de Alcal´a, Alcal´a de Henares, Spain · Universidad Polit´ecnica de Madrid, Madrid, Spain · Universidad Rey Juan Carlos, M´ostoles, Spain
TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversity
Oct 6, 2026Zijian Li, Xiangchen Song, Gongxu Luo +7Synthetic Data Pretraining Tabular Foundation Models
Carnegie Mellon University · Mohamed bin Zayed University of Artificial Intelligence · Guangdong University of Technology +1
A General Pipeline for Dense Illuminant Estimation via Physically Based Synthetic Data
Oct 5, 2026Luca Cogo, Gianmarco Corti, Simone Bianco +1Synthetic Data Pretraining Computational Imaging
Department of Informatics, Systems and Communication, University of Milano – Bicocca, Milan 20126, Italy
Geometry Meets Physics: Data-Efficient Pre-Training for Unstructured Neural PDE Solvers
Oct 2, 2026Luis Medrano-Navarro, Giacomo Baldan, Qiang Liu +4Synthetic Data Pretraining Neural PDE Solvers
Technical University of Munich · University of Stuttgart
Effective Synthetic Data Curation Requires Group-Level Signals
Sep 30, 2026Cathy Jiao, Chenyan XiongSynthetic Data Pretraining Training Data Curation
Language Technologies Institute, Carnegie Mellon University
How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text
Sep 30, 2026Jenna Russell, Ben Glickenhaus, Katherine Thai +4Language Model Pretraining Synthetic Data Pretraining
University of Maryland · Pangram Labs
Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior
Sep 30, 2026Atsuki Yamaguchi, Tatsuro Inaba, Joel Niklaus +3Language Model Pretraining Synthetic Data Pretraining
University of Sheffield, UK · Mohamed bin Zayed University of Artificial Intelligence, UAE · Hugging Face +3
It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs
Sep 29, 2026Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois +7Language Model Pretraining Synthetic Data Pretraining
PleIAs · Sorbonne Center for Artificial Intelligence · Sciences Po Médialab +6
Self-Play Pretraining with Zero Data
Sep 24, 2026Aditya Cowsik, Kfir Dolev, Michael Y. Li +4Synthetic Data Pretraining Synthetic Data Generation for Language Models
Independent Researcher · Tel Aviv University · Stanford University +1
TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling
Sep 22, 2026Julien Knafou, Luc Mottin, Anaïs Mottaz +2Synthetic Data Pretraining Language Modeling
HES-SO, Geneva, Switzerland · SIB, Swiss Institute of Bioinformatics, Geneva, Switzerland
Learning to Fluctuate: Statistical Foundations for Causal Tabular Pretraining
Sep 22, 2026Zhiheng ZhangCausal Foundation Models Synthetic Data Pretraining
School of Statistics and Data Science Shanghai University of Finance and Economics
QVAC Genesis III: A Large-Scale, High-Quality Open Synthetic STEM Corpus for Efficient Language Model Pre-Training
Sep 17, 2026Davide Vitabile, N. Ranjan, Akshay Nambiar +2Language Model Pretraining Synthetic Data Pretraining
Tether Data, S.A. de C.V. d.b.a. Tether AI Research
Preventing Model Collapse: A Fisher-Rao Perspective on the Dynamics of Training with Synthetic Data
Sep 16, 2026Matteo Marchi, João Pedro Silvestre, Bahman Gharesifard +1Synthetic Data Pretraining Model Collapse
Electrical and Computer Engineering Department, University of California at Los Angeles, Los Angeles, CA 90095 USA · Department of Mathematics and Statistics at Queens University, Kingston, ON, Canada
MiST: Mid-Training LLMs for Cybersecurity
Sep 16, 2026Oded Ovadia, Elad Ben Zaken, Elad Guttman +1Synthetic Data Pretraining Domain-Adaptive Pretraining
Dream
Procedural Pretraining for Molecular Property Prediction
Sep 15, 2026Moritz Friedemann, Zachary Shinnick, Philip Torr +1Synthetic Data Pretraining Self-Supervised Pre-Training
University of Oxford · Adelaide University · Slater Labs
Measuring Annotation Efficiency for Handwritten Devanagari Recognition: Sample-Complexity Curves for Four Pretraining Regimes
Sep 15, 2026Manglesh Kumar Pandey, Sumit Kumar BanshalSynthetic Data Pretraining Optical Character Recognition
Alliance School of Liberal Arts and Sciences, Alliance University Bengaluru, India · Department of Computer Science and Engineering, Alliance University Bengaluru, India
Rethinking Procedural Audio Pre-training: Source Scaling and Objective Adaptation
Sep 14, 2026Jiajun Peng, Fengrui Liu, Xinyu Liu +1Synthetic Data Pretraining Audio Representation Learning
School of Data Science, University of Science and Technology of China, Hefei 230026, China · School of Computer Science and Technology, East China Normal University, Shanghai 200062, China · School of Psychology, Shanghai Jiao Tong University, Shanghai 200030, China
Physics-Guided Synthetic High-Frequency Ultrasound Generation for Skin Layer Segmentation
Sep 14, 2026Junkyung ju, Kyungho Yoon, Minwoo ShinSynthetic Data Pretraining Ultrasound Image Segmentation
Department of Software, Yonsei University (Mirae Campus), Wonju, 26493, Republic of Korea · School of Mathematics and Computing, Yonsei University, Seoul, 03722, Republic of Korea
Learning with Synthetic Data via SGD in High-Dimensional Linear Regression
Sep 10, 2026Jichu li, Difan ZouSynthetic Data Pretraining Model Collapse
Renmin University of China · The University of Hong Kong
Distillation of Synthetic Data for Time Series Foundation Models
Sep 9, 2026Niloy Biswas, Noureddine El KarouiSynthetic Data Pretraining Time-Series Foundation Models
Meta AI
Xiaomi-TabLDM: A Tabular Foundation Model Technical Report
Sep 3, 2026Xiaomi-TabLDM Team, :, Penghui Wang +10Synthetic Data Pretraining Tabular Foundation Models
Xiaomi
Synthetic Semantic Supervision for Contrastive Code Representation Learning in Small Transformers: An Empirical Study
Sep 3, 2026Kenneth Paulsen, Florian Tambon, Mike Papadakis +1Synthetic Data Pretraining Contrastive Learning
University of Luxembourg (SnT) · School of Computing, KAIST
Synthetic Worlds for Temporal Evaluation and Knowledge Updating in LLMs
Aug 31, 2026Jonathan Zheng, Zirui Shao, Alan Ritter +1Synthetic Data Pretraining Knowledge Editing
Georgia Institute of Technology · Zhejiang University
REER-PT: Reverse-Engineered Reasoning for Perplexity-Guided Pre-training Data Augmentation
Aug 31, 2026Haoran Que, Jiajun Shi, Ting Huang +7Language Model Pretraining Synthetic Data Pretraining
Peking University · ByteDance Seed · Work done at ByteDance Seed +1
Synthetic Persona Pretraining: Alignment from Token Zero
Aug 13, 2026Julian Minder, Viktor Moskvoretskii, Raghav Singhal +12Synthetic Data Pretraining LLM Alignment
EPFL · SJTU · Northeastern University +8
Patient Pose Assessment Using a CT-Based Framework for Synthetic Data Generation
Aug 6, 2026Manuel Laufer, Dominik Mairhöfer, Malte Sieren +7Synthetic Data Pretraining Medical Imaging
Institute for Neuro- and Bioinformatics, University of Lübeck, Germany · University Medical Center Schleswig-Holstein, Lübeck, Germany · IMAGE Information Systems Europe GmbH, Rostock, Germany +1

Oct 8, 2026, cs.LGJ/K move · Enter open · S save

Is Real-World Training Data Necessary for Generalist Graph Anomaly Detection?

Yujing Liu, Yixin Liu, Yue Tan, Xiaofeng Cao+3

School of Information and Communication Technology, Griffith University, Australia · School of Computer Science and Technology, Tongji University, China

Anomaly Detection Graph Foundation Models Synthetic Data Pretraining Synthetic Anomaly Generation

Open paper AskPDF arXiv

Generalist graph anomaly detection (GAD) aims to build a foundation model that detects anomalies on arbitrary unseen graphs without retraining or fine-tuning. Sufficient data are essential for foundation model training, yet generalist GAD still faces a data shortage, as real-world anomalous graphs are scarce and costly to collect and annotate. To fill this gap, we propose AG-FORGE, an Anomalous Graph generation Forge for automatic synthesis of anomalous graphs, exploring the feasibility of synthetic data-driven training for generalist GAD. Empirically, we find that synthetic data can achieve performance comparable to real-world training, but fail to push the performance boundary further due to the limited capacity of existing methods. To further unlock model capacity as training data scale up, we develop TS-GGAD, a Topology-Semantic coordinated Generalist GAD that captures complementary topological and semantic anomaly evidence, together with a curriculum learning strategy tailored to large-scale synthetic training. Extensive experiments on 14 real-world datasets demonstrate that TS-GGAD, trained on data generated by AG-FORGE, significantly outperforms state-of-the-art methods.

Momentum

Related topics

Latest papers 82