High-Dimensional Linear Regression

Follow this topic Ask about this topic

Latest papers

16 papers4 papers in the last four weeks, up 33% on the four weeks before. 0.0% of all new papers.

Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning
Oct 16, 2026Yalda Shabanzadeh, Diyuan Wu, Marco MondelliTransfer Learning Ridge Regression
Institute of Science and Technology Austria (ISTA).
Optimal Explicit and Implicit Regularization Strengths in High-Dimensional Continual Linear Regression
Oct 14, 2026Gilad Karpel, Daniel SoudryHigh-Dimensional Linear Regression Regularization-Based Continual Learning
Department of Data and Decisions Sciences, Technion · Department of Electrical and Computer Engineering, Technion
When do data mixtures improve scaling laws? Insights from high-dimensional regression
Sep 29, 2026Diyuan Wu, Lehan Chen, Theodor Misiakiewicz +1High-Dimensional Linear Regression Data Mixture Optimization
Institute of Science and Technology Austria (ISTA). · Department of Applied and Computational Mathematics, Yale University. · Department of Statistics and Data Science, Yale University.
Benign Overfitting for General Norms and Distributions
Sep 27, 2026Daniel Barzilai, Ohad ShamirBenign Overfitting Statistical Learning Theory
Weizmann Institute of Science · University of Toronto & Vector Institute
A Bayesian Vertical Federated Learning Framework for Multivariate Reduced-Rank High-Dimensional Regression
Sep 18, 2026Brigham Halverson, Sharmistha Guha, Jessica Bernard +1High-Dimensional Linear Regression Federated Learning
Department of Statistics Texas A&M University · Department of Psychological & Brain Sciences Texas A&M University
Structured Features Overfit Where Random Features Grok
Sep 14, 2026Chon-Fai Kam, Miloud Bessafi, Frederic CadetNeural Network Generalization Ridge Regression
University Paris City & University of Reunion, Paris, France; Dipartimento di Fisica e Chimica · Emilio Segr`e, Universit`a degli Studi di Palermo, Italy · EnergyLab, University of Reunion, Saint-Denis, France +2
Learning with Synthetic Data via SGD in High-Dimensional Linear Regression
Sep 9, 2026Jichu li, Difan ZouSynthetic Data Pretraining High-Dimensional Linear Regression
Renmin University of China · The University of Hong Kong
High-dimensional ridgeless least squares interpolation under spiked covariance structures
Aug 7, 2026Zhijun Liu, Dandan JiangDouble Descent Benign Overfitting
College of Sciences, Northeastern University, China. · School of Mathematics and Statistics, Xi’an Jiao Tong University, China.
The Fourth Quadrant: A Stylized View of Benign Misfitting
Aug 2, 2026Gireeja Ranade, Anant SahaiBenign Overfitting Linear Regression
University of California, Berkeley
The Zero Pattern of a Design Matrix Drives Multiple Descent in Over-parameterized Regression
Jul 27, 2026Kevin Han Huang, Haoyu Ye, Somak Laha +1Double Descent Linear Regression
University of Warwick · Harvard University
De-floored Principal Component Regression: When Rank Selection Alone Is Insufficient for Prediction
Jul 18, 2026Peng ZhaoPrincipal Component Analysis High-Dimensional Linear Regression
Department of Applied Economics and Statistics, University of Delaware
What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regression
May 31, 2026Wendao Wu, Fangqing Zhang, Haihan Zhang +1High-Dimensional Linear Regression Weak-to-Strong Generalization
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University, China · School of Mathematical Sciences, Peking University, China
High dimensional theory of two-phase optimizers
Mar 27, 2026Atish AgarwalaMomentum Methods Distributed Optimization
Google DeepMind
How Does the ReLU Activation Affect the Implicit Bias of Gradient Descent on High-dimensional Neural Network Regression?
Mar 5, 2026Kuo-Wei Lai, Guanghui Wang, Molei Tao +1High-Dimensional Linear Regression Random Feature Methods
School of Electrical and Computer Engineering, Georgia Institute of Technology · College of Computing, Georgia Institute of Technology · School of Mathematics, Georgia Institute of Technology +1
Scaling Laws for Precision in High-Dimensional Linear Regression
Feb 22, 2026Dechen Zhang, Xuan Tang, Yingyu Liang +1High-Dimensional Linear Regression Scaling Laws
Institute of Data Science, The University of Hong Kong · School of Computing & Data Science, The University of Hong Kong · Institute of Data Science and School of Computing & Data Science, The University of Hong Kong +1
Efficient Cross-Validation for Sparse Linear Regression
Jun 26, 2023Ryan Cory-Wright, Andrés GómezModel Selection Mixed-Integer Linear Programming
Department of Analytics, Marketing and Operations, Imperial Business School, London, UK · Department of Industrial and Systems Engineering, Viterbi School of Engineering, University of Southern California, CA

Oct 16, 2026J/K move · Enter open · S save

Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning

Yalda Shabanzadeh, Diyuan Wu, Marco Mondelli

Institute of Science and Technology Austria (ISTA).

Transfer Learning Ridge Regression High-Dimensional Linear Regression Minimax Estimation

Open paper AskPDF arXiv

Pretrained models are often reused across downstream tasks, motivating the question of whether this transfer learning procedure can provide a substantial performance improvement, namely, in terms of the scaling law. We address this question in the prototypical setting of high-dimensional regression, where (i) a pretrained source model is trained via ridge regression, and (ii) a fine-tuned target model is regularized via the

L_2

distance from the source estimator. First, we establish a deterministic equivalent for the risk of this two-step procedure under both model and covariance shift. Next, by specializing the deterministic equivalent to a shared regression function and commuting power-law covariance spectra, we identify a regime where the resulting scaling law achieves the minimax-optimal exponent. Notably, in this regime, the optimal source regularization depends on source data only, indicating its universality across all targets. Furthermore, we show that the same optimal rate is achieved by performing the fine-tuning step via one-pass stochastic gradient descent. We support our theoretical analysis via experiments on both synthetic settings and language models.