Statistical Learning Theory

Follow this topic Ask about this topic

Statistical learning theory asks when a learner can infer rules that work beyond its training data, and quantifies sample complexity, prediction error, and limits on what can be learned. Classical tools include VC and Littlestone dimensions, covering numbers, Rademacher complexity, stability, PAC-Bayes bounds, and minimax analysis; these provide guarantees for empirical risk minimization, online learning, kernels, and robust or private learning. Current work also studies high-dimensional and overparameterized regression, including interpolation and double descent, neural networks and Transformers, and learning under distribution shift, dependence, partial labels, or adaptive data. An enduring tension is that statistically optimal rates may be computationally unattainable, while guarantees for modern architectures often rely on idealized assumptions and may not explain practical optimization.

Latest papers

584 papers79 papers in the last four weeks, up 58% on the four weeks before. 0.5% of all new papers.

Statistical Analysis of Synthetic Data with Verification: A Linear Regression Framework
Oct 29, 2026Heejune Sheen, Harrison H. ZhouSynthetic Data Selection Self-Training
Department of Statistics and Data Science Yale University New Haven, CT 06511 USA
Representation Complexity Along Learning Trajectories: Trading Width for Precision and Time
Oct 28, 2026Ryan Singh, Miguel De Llanza, Christopher L. BuckleyOverparameterized Neural Network Neural Network Training Dynamics
Department of Engineering and Informatics, University of Sussex
New Multiple Descent Phenomena Induced by Heterogeneous Data
Oct 27, 2026Mark-Anthony Moisescu-Pareja, Elvis DohmatobDouble Descent Statistical Learning Theory
Concordia University · Mila – Quebec AI Institute
The Structure Gap: When More Data Cannot Replace Structured Reasoning
Oct 27, 2026Kheer Sagar Patel, Mojtaba SoltanalianStatistical Learning Theory Sample Complexity
University of Illinois Chicago
When Does Data Value Reduce to Class Balance? A Coverage View of Per-Point Data Valuation
Oct 26, 2026Yingjie Bai, Weiming ZhiData Selection Training Data Selection
School of Computer Science, The University of Sydney
Internal Discrepancy for Contrastive Learning: Generalization Bounds and Downstream Behaviors
Oct 23, 2026Jialiang Wang, Qing Ma, Xiong Zhou +3Contrastive Learning Statistical Learning Theory
Harbin Institute of Technology, Harbin, China · City University of Hong Kong, Hong Kong, China
Optimal Multiclass Online Regret with Arbitrary Label Spaces
Oct 23, 2026Sajad Ashkezari, Shai Ben-DavidMulticlass Classification Statistical Learning Theory
University of Waterloo
Sharp Bounds on the Winner's Bias under Independent and Nonnegatively Correlated Noise
Oct 22, 2026Ivan Viakhirev, Kirill Borodin, Grach MkrtchianCross-Validation Selection Bias
lab260 (lab260.ru) · BitmanagerAI (bitmanager.ai)
Combinatorial Characterizations of the Learnability of Conditional Distributions
Oct 22, 2026Mihir Dhanakshirur, Ambuj TewariPAC Learning Statistical Learning Theory
University of Michigan
Efficient and Optimal Learning of Discrete Distributions with Person-Level Privacy
Oct 22, 2026Gautam Kamath, Mahbod Majid, Ankur Moitra +2Robust Statistics Differential Privacy
New York University · Massachusetts Institute of Technology · University of Texas at Austin and University of Waterloo +1
Nonparametric autoregressive estimation of probability distributions with parallel gradient boosting
Oct 22, 2026Rémy ChapelleAutoregressive Modeling Statistical Learning Theory
CB, EVDG, CESP
Deep Learning for Random Objects in Metric Spaces: Beyond Manifold Assumptions
Oct 22, 2026Marcos Matabuena, Xiaokai Luo, Oscar Hernan Madrid PadillaNonparametric Regression Distributional Regression
Mohamed bin Zayed University of Artificial Intelligence · University of Notre Dame · Department of Statistics at University of California, Los Angeles
Towards Faithful Data Selection: Statistical Limits and Scalable Algorithms
Oct 21, 2026Ahmed Mehdi Inane, Mahdi Haghifam, Gintare Karolina Dziugaite +1Training Data Selection Statistical Learning Theory
Université de Montréal, Mila · Toyota Technological Institute at Chicago · Google DeepMind +1
Actually, All Learnable Distribution Classes Are Privately Learnable
Oct 20, 2026Mohammad Afzali, Hassan Ashtiani, Alex Bie +4Differential Privacy Statistical Learning Theory
University of British Columbia. · McMaster University and Vector Institute. · Google Research. +4
A Reverse-Gold Theorem: Positive-Data Identification by Substrate-Bounded Contrastive Learners
Oct 20, 2026Rakesh SenguptaAutomata Learning Statistical Learning Theory
School of Interwoven Arts and Sciences Krea University Sri City, AP 517646, India
Understanding Data Augmentation under Label Noise
Oct 20, 2026Hong Zheng, Fei TengData Augmentation Learning with Noisy Labels
School of Computing and Artificial Intelligence Southwest Jiaotong University Chengdu, 611756, China · School of Computing and Artificial Intelligence and Engineering Research Center of Sustainable Urban Intelligent Transportation Southwest Jiaotong University and Ministry of Education Chengdu, 611756, China
Gaussian Approximation of Besov Functions for Logistic Classification on Manifolds
Oct 19, 2026Guanhang Lei, Lei Shi, Xiaoming YuanClassification Statistical Learning Theory
Department of Mathematics, The University of Hong Kong, Hong Kong, China · School of Mathematical Sciences, Shanghai Key Laboratory for Contemporary Applied Mathematics, Center for Applied Mathematics, Fudan University, Shanghai, China
Statistical Learning Theory for Direct Preference Optimization
Oct 19, 2026Joel Wills, Yuanfan Li, Yiming Ying +1Direct Preference Optimization Pairwise Preference Learning
School of Mathematics and Statistics, The University of Sydney
A Distribution-Free Separation Between Learned and Fixed Features
Oct 18, 2026Daniel BarzilaiTwo-Layer Neural Networks Statistical Learning Theory
Weizmann Institute of Science
The Optimal Sample Complexity of Pointwise Replicable Agnostic Learning
Oct 18, 2026Guangjian ZhangAgnostic Learning Sample Complexity
Strictly Replicable Realizable Learning: Unknown Marginals, Finite-Class Interpolation, and the Cost of Shared Randomness
Oct 18, 2026Guangjian ZhangSample Complexity Statistical Learning Theory
Scalable and Adaptive Distribution Regression via Random Features
Oct 17, 2026Yinsong Wang, Xiao Liu, Yu DingDistributional Regression Random Feature Methods
School of Mechanical, Industrial and Material Engineering, University of Nevada Reno, Reno 89557, USA. · H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta 30332, USA.
Transformers as Adaptive Kernel Learners across Tasks for In-Context Learning
Oct 17, 2026Zhaiming Shen, Alexander Cloninger, Rongjie Lai +1Transformer Attention In-Context Learning
Department of Mathematics and Statistics, Utah State University, Logan, UT 84322 · Department of Mathematics, Purdue University, West Lafayette, IN 47907 · School of Mathematics, Georgia Institute of Technology, Atlanta, GA 30332
Margin Certificates for Voting Classifiers under Mixing Dependence
Oct 16, 2026Adria Binte Habib, Ibne Farabi ShihabStatistical Learning Theory Generalization Bounds
Department of Computer Science and Engineering, BRAC University, Bangladesh · Department of Computer Science, Iowa State University, USA
Deterministic Bounds on Accuracy and Stability of Metric-Space Regression and Classification
Oct 15, 2026Nina Maria Gottschling, Jakob Gawlikowski, Michele CaprioStatistical Learning Theory Generalization Bounds
Computing and Computational Sciences, Oak Ridge National Laboratory, Oak Ridge, Tennessee. · German Aerospace Center (DLR), Remote Sensing Technology Institute, Wessling, Germany. · Department of Computer Science, University of Warwick, Coventry, United Kingdom.
Two Sample Sizes in Test-Time LLM Ensembling: Buffered ERM and Minimax Theory
Oct 15, 2026Zhi Zhang, Arash A. AminiLanguage Model Ensembles Test-Time Scaling
Department of Statistics and Data Science, UCLA
Hard Way, The Only Way: Provable Merits of Training on Hardest Examples
Oct 15, 2026Elvis DohmatobHigh-Dimensional Asymptotics Dataset Pruning
Concordia University · Mila – Quebec AI Institute · FAIR – Meta
Sign Betting for Misspecified Regression: Variance Adaptivity and Optimal Approximation
Oct 14, 2026Yinan Li, Han Han, Kwang-Sung JunStatistical Learning Theory Model Misspecification
Department of Computer Science, University of Arizona · GSAI/CSE, POSTECH
Optimal Explicit and Implicit Regularization Strengths in High-Dimensional Continual Linear Regression
Oct 14, 2026Gilad Karpel, Daniel SoudryHigh-Dimensional Linear Regression Regularization-Based Continual Learning
Department of Data and Decisions Sciences, Technion · Department of Electrical and Computer Engineering, Technion
Posterior Frequencies and the Cost of Not Fitting: A Correction to a Memorization Lower Bound
Oct 14, 2026Zhenisbek Assylbekov, Rustem TakhanovStatistical Learning Theory
Department of Mathematical Sciences, Purdue University Fort Wayne · School of Computing and Artificial Intelligence, Nazarbayev University

Oct 29, 2026J/K move · Enter open · S save

Statistical Analysis of Synthetic Data with Verification: A Linear Regression Framework

Heejune Sheen, Harrison H. Zhou

Department of Statistics and Data Science Yale University New Haven, CT 06511 USA

Synthetic Data Selection Self-Training Statistical Learning Theory Synthetic Data Generation

Open paper AskPDF arXiv

Synthetic data are widely used for training modern AI systems. However, their statistical understanding remains limited. It is well recognized in statistics that repeated self-training on synthetic data can amplify errors inherited from pre-trained models, leading to model collapse. Verification plays a crucial role in selecting high-quality synthetic data to help prevent model collapse and improve model performance. We study verified self-training in a linear regression framework, where a pre-trained estimator generates synthetic data with injected generation noise and a verifier selects samples according to a verification threshold. Post-training on the accepted samples updates either all coordinates (full fine-tuning) or only an active subset (partial fine-tuning). Our analysis reveals three potentially counterintuitive effects. We establish that tighter verification does not always lead to lower prediction risk. We also find that a suitable level of injected generation noise can reduce prediction risk, while excessive noise can increase it. Moreover, full and partial fine-tuning can respond differently to the same verification rule, since pre-training error in the frozen coordinates affects both sample selection and post-training risk. We analyze these effects by deriving finite sample risk bounds for four separate regimes determined by the relative scales of the verification threshold, generation noise, and pre-training error. The bounds precisely characterize how verification affects prediction risk across these regimes. In particular, we quantify how the generation noise, the effective accepted sample size, and the covariance of the accepted covariates affect prediction risk. The analysis extends to iterative verified full fine-tuning, where sufficiently tight verification reduces the initial error contribution at a geometric rate.