Linear Regression

Follow this topic Ask about this topic

Momentum

6 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 48

Oct 6, 2026cond-mat.dis-nn

Asymptotic Analysis of Empirical Risk Minimization on Entry-wise i.i.d. Heavy-Tailed Data

Many real-world datasets exhibit unusually large values far more frequently than predicted by Gaussian models. Heavy-tailed distributions capture this behavior, yet evaluating learning performance under them remains challenging because rare, large feature entries retain non-vanishing effects even in high dimensions. Even in the canonical setting of empirical risk minimization for linear regression with entry-wise i.i.d. symmetric

α

-stable data, a precise asymptotic characterization of prediction has been lacking. In this work, we introduce a functional order parameter that describes the random effective problem associated with each coefficient. Using the replica method, we fully characterize the generalization error in the proportional high-dimensional limit where the sample size and feature dimension diverge at a fixed ratio. Additionally, this analysis establishes a heavy-tail universality law, scaling laws relating typical errors to prediction reliability, and the Bayes-optimal prediction error. In addition to characterizing the effects of extreme entries on the learning process, our method applies broadly to other systems with persistent local heterogeneity.

Kaito Takanami, Takashi Takahashi, Yoshiyuki Kabashima

Graduate School of Science, The University of Tokyo · Institute for Physics of Intelligence, The University of Tokyo · Trans-Scale Quantum Science Institute, The University of Tokyo

Statistical Physics of Learning Linear Regression Heavy-Tailed Distributions Empirical Risk Minimization

Oct 5, 2026cs.LG

LinearPFN: Amortized Variable Selection for Linear Models with Interactions

Spike-and-slab regression is a standard Bayesian formulation of variable selection: it returns a posterior distribution over which candidate effects are active rather than a single selected subset, so that every candidate effect carries an inclusion probability. Its cost grows exponentially with the number of candidate effects, so the posterior can be enumerated exactly only when the number of predictors is small. Beyond that reach, the posterior has to be approximated, typically by Markov chain Monte Carlo over the model space, which requires a fresh run for every dataset and, within a fixed budget of steps, may fail to converge. We present LinearPFN, a prior-data fitted transformer network that amortizes spike-and-slab inference for linear models with main effects and pairwise interactions. The network is pretrained once on synthetic datasets, drawn from an explicitly specified prior, and a single forward pass over a new dataset returns posterior inclusion probabilities, posterior-mean coefficients and posterior predictive distributions with no per-dataset fitting. The prior is conjugate by design, so that the posterior for each fixed set of active effects has a closed form, and wherever the exact posterior is still computable by enumeration we verify the network's outputs against it. On real predictor matrices from published social-science datasets, with outcomes drawn from the prior so that the true active set is known, LinearPFN attains a higher per-dataset selection AUC and a higher F1 under the median probability model rule than five classical baselines. The lead holds when the coefficients, the interactions or the noise depart from the prior. Code: https://github.com/schiekiera/LinearPFN. Trained model: https://huggingface.co/schiekiera/LinearPFN.

Louis Schiekiera, Max Zimmer, Christophe Roux +3

Institute of Psychology, Humboldt-Universität zu Berlin, Berlin, Germany · Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany · Department for AI in Society, Science, and Technology, Zuse Institute Berlin, Germany +1

Model Selection Prior-Data Fitted Networks Linear Regression Feature Selection

Sep 30, 2026stat.ML

Distributionally robust linear regression through the lens of adversarial training

Distributionally robust optimization (DRO) studies parameter estimation under uncertainty in the underlying probability distribution and has emerged as a principled framework for analyzing robustness and generalization. In particular, Wasserstein DRO, with distributional uncertainty induced by the Wasserstein distance, generalizes several popular regularizers. This paper studies Wasserstein DRO linear regression, unifying square-root Lasso and adversarial linear regression as important special cases. We prove that many properties of these two special cases carry over to this general method. In particular, we show (i) deterministic and non-asymptotic in-sample error bounds

O(n^{-1/2})

in general and

O(n^{-1})

under design matrix and sparsity conditions; (ii) insensitivity to the noise level, also known as the pivotal property; and (iii) solution equivalences for small and large ambiguity sets. The key proof step is to recast the method into a quadratic form, mimicking adversarial linear regression. We also show that the method can be solved efficiently, and we validate our findings through numerical simulations.

Elis Stefansson, David Vävinggren, Antônio H. Ribeiro

Uppsala University, Sweden · Science for Life Laboratory, Sweden

Robust Regression Distributionally Robust Optimization Wasserstein Distributionally Robust Optimization Linear Regression Robust Estimation

Sep 30, 2026stat.ML

Principal Component Regression Dominates all Monotone Spectral Filters for Linear Regression

We compare the instance-wise, finite-sample risks of monotone spectral filters for linear regression, a broad class of estimators including principal component regression (PCR), gradient descent (GD), and ridge regression. We show that PCR dominates all monotone spectral filters: compared to any such filter, the risk of optimally tuned PCR is no bigger by a constant factor for all problems. Furthermore, the dominance is strong if the filter is separated from step functions (e.g., GD and ridge): there exist problem instances for which the risk of PCR is smaller by a polynomial factor in sample size dependence. Our comparison results show that PCR is optimal and thus admissible among monotone filters, significantly extending Wu et al. (2026)'s result that GD strongly dominates ridge. From a technical perspective, we establish new upper and lower bounds for general spectral filters, which are instance-wise sharp when specialized to ridge or GD, recovering or improving the best-known bounds.

Juno Kim, Hengyu Fu, Peter Bartlett +2

University of California, Berkeley · Google DeepMind

Spectral Regularization Linear Regression Statistical Learning Theory

Sep 28, 2026stat.ML

Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networks

Adapting pretrained models to downstream tasks with limited data has become a central paradigm in modern deep learning. Yet, despite its widespread practical success, how fine-tuning leverages information from pretraining remains poorly understood theoretically. We study fine-tuning from pretrained weights through the lens of sparse linear regression and two-layer diagonal linear networks. In our setting, pretraining provides information through the support (and signs) of the initialization predictor, which may contain coordinates relevant to the downstream task. We show how pretrained information reshapes the implicit bias and training dynamics, and can thereby reduce the sample complexity of recovering the target parameters and support. In particular, for a clean initialization with correctly inherited signs, we show that the required sample size is comparable to that of a weighted Lasso estimator that explicitly exploits the pretrained support through a suitably chosen regularizer. Our results thus show how information encoded in pretrained weights can be implicitly exploited by gradient-based fine-tuning, reducing the amount of data needed to recover a downstream task.

Alexandre Declèves, Etienne Boursier, Nicolas Flammarion

TML, EPFL, Lausanne, Switzerland · INRIA, LMO, Université Paris-Saclay, Orsay, France

Fine-Tuning Deep Linear Networks Linear Regression Data-Efficient Learning

Sep 20, 2026stat.ML

Sparse Regression Distilled from a Single Robust Fit

Robust linear fits can resist response contamination yet remain too dense or unstable for useful global explanations. We propose penalized distillation, which fits a smoothly clipped absolute deviation (SCAD) estimator to a robust initial estimator's empirical fitted surface along a safeguarded coordinate-descent path and evaluates candidate states separately for fidelity, parsimony, perturbation stability, and held-out prediction. The new results attach to the states the algorithm actually computes. Conditional on a fixed uncontaminated design, deterministic bounds transfer response-replacement boundedness from the initial fit to every retained path state. Turning to fixed dimension, we characterize the oracle-support branch by its empirical-Gram projection and influence function, give conditions for covariance-weighted least-squares approximation equivalence, and establish a path-conditional generalized information criterion. By contrast, at large dimension-to-sample ratios the full-coordinate robust fit collapses without warning, and screening restores the construction. Under a sure-screening framework, the robustness bound and the support and selection guarantees transfer to the screened fit. Simulations separate robustness transfer from support recovery, efficiency, and computation across the dimension-to-sample ratio, with p up to 240, and the signal density, which isolates what the sparse stage adds once the screen over-selects. In a duplicate-grouped superconductivity study, the distilled estimator remains predictively stable under prespecified training-response shifts but retains 66.8--68.8 of 81 slopes. Stronger sparsification reduces the model to 12.6--14.0 slopes only at visible fidelity and prediction cost. Distillation therefore preserves predictive stability on these data without substantiating a compact coordinate-level explanation.

Wooyoung Shin, Seunghwan Park

Department of Information Statistics, Kangwon National University, 1, Kangwondaehak-gil, Chuncheon-si, Gangwon-do, 24341, Republic of Korea.

Robust Regression Linear Regression Robust Estimation

Sep 10, 2026stat.ML

Learning with Synthetic Data via SGD in High-Dimensional Linear Regression

Synthetic data has become a promising way to scale model training beyond limited human-generated data but it may also induce strong model collapse (Dohmatob et al., 2024), where any fixed fraction of synthetic data prevents model performance from improving under data scaling, leaving a non-vanishing excess risk floor. In this paper, we study how synthetic data affects the generalization of one-pass SGD in high-dimensional linear regression with model shift. We establish finite-sample risk bounds for mixed and two-stage training, separating standard bias and variance from source-mismatch effects, namely fluctuation and persistent drift under mixing and filtered initialization bias under two-stage. These bounds reveal a sharp contrast: mixed training induces strong model collapse, while two-stage training avoids the floor by using synthetic data only in the first stage, showing that collapse is not inevitable under a simple data curriculum. Under a random sketch model, we further obtain scaling laws for both protocols, with tight results for mixed training in the optimization-saturated regime. These laws show that larger models may amplify synthetic-induced degradation under mixing, and quantify how high-quality synthetic pretraining may reduce bias in two-stage training. Finally, we establish an exact finite-sample necessary-and-sufficient condition for two-stage training to strictly outperform real-only training under the same real-data budget and identical real-stage updates. Overall, our results highlight that synthetic data is neither inherently harmful nor beneficial; its effect depends critically on both its quality and the training protocol used to incorporate it.

Jichu li, Difan Zou

Renmin University of China · The University of Hong Kong

Synthetic Data Pretraining Model Collapse Linear Regression

Sep 8, 2026stat.ML

Optimal estimation for Functional Linear Regression with Noisy Discretized Data

In this paper, we consider the scalar-on-function linear regression model under a realistic sampling scheme in which the functional covariates are observed on a regular grid and contaminated by additive noise. We propose a two-step estimation procedure: first, the underlying curves are reconstructed from the discrete noisy observations using a Fourier-based projection method; second, the slope function is estimated by a penalized least-squares criterion over finite-dimensional trigonometric spaces, with data-driven selection of the model dimension. We establish oracle-type inequalities for the prediction error, both with respect to the reconstructed curves and to the true latent curves. Under regularity assumptions on the slope function and polynomial decay of the eigenvalues of the covariate, we derive convergence rates for the prediction error and show that our estimator attains the minimax rate when the number of grid points is sufficiently large. Finally, the proposed method is illustrated on simulated data and on a real meteorological dataset.

Sixtine Sphabmixay

Université Paris Cité, CNRS, MAP5, F-75006 Paris, France

Nonparametric Regression Minimax Estimation Linear Regression

Aug 31, 2026cs.LG

Exact Recovery Thresholds for Weighted Data Selection in Vector-Valued Linear Regression

We resolve the threshold part of Question 4 of the COLT 2025 open problem "Data Selection for Regression Tasks" of Hanneke, Moran, Shlimovich and Yehudayoff. In vector-valued linear regression with square loss

\ell_{(x,y)}(W)=|Wx-y|_2^2

, where

x\in\mathbb{R}^d

,

y\in\mathbb{R}^m

and the learner is the empirical risk minimizer of minimal Frobenius norm, we prove that the minimal budget of weighted examples that recovers the full-data loss on every finite dataset is exactly

n^*(d,m)=(m+1)d

. We further determine two more values of the weighted selection profile

F_w(d,m,n)

: at the near-threshold budget,

F_w(d,m,(m+1)d-1)=1+\frac{1}{dm^2}

, and at the spanning budget,

F_w(d,m,d)=d+1

for every

m

, while

F_w(d,m,n)=\infty

for

n<d

. For the smallest open intermediate cell

(d,m)=(2,2)

we prove

F_w(2,2,3)\in[13/8,15/8]

and

F_w(2,2,4)\in[5/4,3/2]

, reduce the conjectured exact values

13/8

and

5/4

to a finite moment problem on the circle with at most seven atoms, and establish strong structural evidence for the conjecture. The upper-bound techniques (a fixed-basis conic compression lemma, a determinant-facet rigidity theorem for maximal certificates, and sharp sparsification lemmas for zero-mean weighted point systems) are of independent interest. As a byproduct we correct an erroneous claim circulating in a recent unrefereed preprint, exhibiting an explicit dataset with

m=2

on which no weighted selection of

2d

points recovers the optimal loss. All results are new only for

m\ge 2

; the scalar case

m=1

is due to Hanneke et al.

Guangjian Zhang

Data Selection Training Data Selection Linear Regression Coreset Selection

Aug 27, 2026cs.LG

Contact Geometry and Covariance Deficits in Volume-Sampled Least Squares

We classify when ordinary fixed-size volume sampling followed by unweighted least squares attains its sharp coefficient-covariance ceiling on a fixed design. For a real whitened design without coloops and a fixed positive-loss residual, the contact space is unchanged at every strict-interior sample size. Its possible nonzero values form a finite orthogonal family: each maximal parallel class of normalized Naimark-complement rows determines a deletion nullspace of dimension one less than the class size. A single residual attains an entire query precisely when the query range lies in one class space. The proof starts from two-sided Loewner comparison of every normalized covariance deficit with an explicit leave-one-out operator, using supported omission moments and reverse deletion. Residual augmentation provides resolvent and second-moment upper bounds, while complement geometry yields query-specific margins, angular concentration, local alignment, and a multi-output energy obstruction. Exact families give closed-form margins and covariances, exhibit support-boundary jumps, and approach the ceiling despite a uniformly positive geometric margin. Finally, the same moment identities give upper and lower bounds on expected fixed-query squared-loss excess. The subset draw is the only randomness; all support and endpoint restrictions are explicit.

Kihun Rhee, Hanjoon Byun, Junpyo Seo

Randomized Sketching Linear Regression Statistical Learning Theory

Aug 19, 2026math.ST

Algorithms for adaptive and heteroskedastic linear regression at the computational threshold

We study finite-sample linear regression in the presence of varied and unknown label noise, focusing on the heteroskedastic and adaptive linear regression models. Heteroskedastic linear regression models settings where the labels are of varying quality. We receive

n

pairs

(X_i,Y_i)

with labels

Y_i=X_i^\topβ+\varepsilon_i

, where

\varepsilon_i\sim N(0,σ_i^2)

and the variances are unknown to the estimator. One natural measurement of the difficulty of this problem is the number of samples

m

for which

σ_i^2\le1

(larger

m

is easier). We obtain a polynomial-time estimator with rate

\tilde{O}((nd^3/m^4)^{1/6})

when

m\gg d^{3/4}n^{1/4}

, as well as nearly-matching lower bounds. For

d=O(1)

, our estimator achieves error

o(1)

when

m\gg n^{1/4}

, whereas

L_1

regression and other traditional approaches require

m\gg n^{1/2}

. In adaptive linear regression, the errors are drawn i.i.d. from an unknown distribution

p

, and our goal is to design a generic estimator that performs nearly as well as the best custom estimator that knows

p

. We introduce a (computationally inefficient) adaptive estimator that, so long as

p

is a mixture of

k

symmetric log-concave densities, achieves error comparable with the optimal estimator that knows

p

and has

\tildeΘ(n/k)

samples. For

k=1

, we show that

L_q

regression (with data-dependent

q

) gives a polynomial-time estimator. Finally, to study the computational limits of both problems, we introduce the planted linear regression problem, where

X_i\sim N(0,I_d)

,

m

unknown samples are noiseless, and the rest have error

\varepsilon_i\sim N(0,1)

. We conjecture that recovering

β

up to error

\ll\sqrt{d/n}

(or exactly) may have an information-computation gap between

m=d+1

and

m\sim d^{3/4}n^{1/4}

, as is suggested by our near-matching polynomial-time estimator and statistical query (SQ) lower bound.

Spencer Compton, Tselil Schramm

Stanford University

Minimax Estimation Heteroscedastic Regression Linear Regression

Aug 18, 2026stat.ML

Feature Priming in Online Linear Regression: Sparse-Regret Lower Bounds and Tight Coordinatewise Rates

In high-dimensional online prediction, sparse comparators motivate regret bounds that depend on sparsity rather than ambient dimension. Feature priming seeks such adaptation by reweighting features using past data and refitting a minimum-norm predictor. At COLT 2023, Warmuth and Amid posed the open problem of whether the univariate, Pearson, or multivariate priming rules admit competitive online regret guarantees. Under the natural past-only Moore--Penrose protocol, we establish sparse-regret lower bounds that refute the corresponding sparse-logarithmic guarantee. The key obstruction is cheap nuisance interpolation, which permits exact interpolation of the history while assigning insufficient weight to the truly predictive coordinate. An exact target-mass identity and a two-sign argument convert this obstruction into clipped prediction loss. Hadamard constructions yield

Ω(\min\{T,\sqrt d\})

clipped regret for each of the three unit-power rules against a zero-loss one-sparse comparator. For every fixed power

α\ge1

, one shared paired construction further yields linear regret simultaneously for all three powered rules and selectors among them in sufficiently high dimension. A rank upper bound is tight for powered univariate priming, even with Euclidean-unit inputs, and for unit-power Pearson priming with coordinatewise bounded inputs and target-preserving totalization. A separate algebraic construction gives

Ω(\min\{T,d^{1/4}\})

regret for unit-power multivariate priming under Euclidean-unit inputs. The univariate lower bound persists under any nonnegative second-stage ridge schedule, while a paired ridge construction yields linear lower bounds for all three powered rules. Exploratory diagnostics on frozen language-model activations are consistent with the same qualitative mechanism. The exact multivariate frontier remains open.

Huibo Xu, Shi Fu, Qixin Zhang +1

Nanyang Technological University, Singapore

Linear Regression Statistical Learning Theory

Aug 2, 2026cs.LG

The Fourth Quadrant: A Stylized View of Benign Misfitting

Training error is what we can observe on a training set; test error is the quantity we actually care about. We study linear regression with squared-error in a deterministic

(d+1)

-dimensional single-spike model. Each stylized training vector has the same informative spike coordinate, of amplitude

\sqrtγ

with

γ>1

. The remaining directions are nuisance, and the nuisance components of distinct training vectors all have equal norm and are mutually orthogonal. The training labels are all

1

. Fresh test points are drawn from

\vec{x}_{\rm test} \sim \mathcal{N}(\vec{0},\operatorname{diag}(γ,1,\ldots,1))

, with the noise-free test labels being the normalized spike coordinate

x_{\rm test}[1]/\sqrtγ

. We focus on linear predictors in the span of the training vectors, the class naturally reached by zero-initialized linear gradient methods. We exhibit a range of training-set sizes

n

in which every span predictor that generalizes well must fit the training data \emph{worse} than the zero predictor. We call this regime \emph{benign misfitting}, or the fourth quadrant. The best span predictor begins to generalize when

n\gg d/γ^2

, while interpolation does not generalize until the later threshold

n\gg d/γ

. In the window

d/γ^2 \ll n \ll d/γ

, useful prediction within the linear span lies beyond interpolation: predictions on the training points overshoot the labels. We show that one-pass stochastic gradient descent (SGD), with a large constant learning rate, reaches small test error throughout this window---matching the best span predictor up to a logarithmic factor. We also verify directly that it indeed has \emph{large} empirical training error (despite the descent premise in its name). Finally, we show that the unavoidable nuisance component responsible for the training misfit also controls the predictor's adversarial sensitivity.

Gireeja Ranade, Anant Sahai

University of California, Berkeley

Benign Overfitting Linear Regression Statistical Learning Theory

Jul 27, 2026math.ST

The Zero Pattern of a Design Matrix Drives Multiple Descent in Over-parameterized Regression

Over-parameterized linear regression has been widely studied over the last decade. However, most existing works assume that the covariates are independent and that their covariance matrices are non-degenerate. In this paper, we relax both assumptions and derive deterministic equivalents for the prediction risk in a vanishing-ridge regime. We show that degeneracy of the covariance matrices and dependence can lead to multiple descent, and characterize where the corresponding peaks can occur. Our proofs use a novel graph representation of the variance profile. We show that maximum matchings and the Dulmage--Mendelsohn decomposition of the associated bipartite graph identify the configurations at which the variance becomes singular.

Kevin Han Huang, Haoyu Ye, Somak Laha +1

University of Warwick · Harvard University

Double Descent Linear Regression

Jul 24, 2026cs.LG

Beyond Negative-Ridge Endpoints: Mixed-Sign Spectral Regularization via Negative-Shifted Gradient Descent

In overparameterized linear regression, many weak spectral directions act like a ridge penalty on the signal-bearing spectrum; negative ridge is the natural correction, pushing filters above one. The stable negative-ridge endpoint, however, is structurally limited: its pole must stay below the smallest nonzero empirical eigenvalue, and it anti-shrinks smaller eigenvalues more than larger ones. Early-stopped negative-shifted gradient descent escapes this constraint. Its filter is smooth at the would-be pole and mixed-sign-capable: above-ridgeless directions form a leading prefix, with lower directions shrunk or exposure-controlled while stopping sets the crossover. In a Gaussian spike-plus-flat model we discover a Marchenko-Pastur barrier: the shift that cancels the implicit penalty lies a bulk width above the smallest empirical eigenvalue, and the stopped path improves on every admissible endpoint by a polynomial factor in risk under explicit conditions. Our main theorem permits a general high-effective-rank tail: its trace sets the implicit floor, its squared spectrum controls exposure, and the floor-critical path recovers all head scales at once, beyond positive shrinkage and, once scales separate, every uniform rescaling of ridgeless. Handling the noncontractive shifted dynamics is the central technical challenge; localized Duhamel integrals control them. A finite-grid hold-out inequality transfers the separations to the validation-selected algorithm.

Peng Zhao

Department of Applied Economics and Statistics · University of Delaware

Spectral Regularization Early Stopping Gradient Descent Linear Regression Ridge Regression Implicit Regularization

Jul 18, 2026math.ST

De-floored Principal Component Regression: When Rank Selection Alone Is Insufficient for Prediction

Principal component regression (PCR) regularizes high-dimensional prediction by choosing a spectral cutoff, but rank selection cannot correct systematic inflation of the retained empirical eigenvalues. We study clean Gaussian random designs in which the aggregate covariance tail creates a nearly scalar sample-space floor comparable to the predictive head scale. De-floored principal component regression (dPCR) retains the cutoff and subtracts an estimated floor from the retained denominators. We prove an ordinary-PCR prediction-risk lower bound uniform over all ranks and a high-probability dPCR upper bound. When the floor is sharp and inexpensive to remove in population prediction risk, the conditional risk of dPCR is asymptotically negligible relative to that of the best ordinary PCR rank. An exact risk decomposition explains the separation: denominator inflation is governed by first spectral mass, whereas the clean prediction cost of correction is governed by squared spectral mass. A same-sample trimmed-mean floor estimate attains the oracle dPCR upper-bound rate at a prespecified rank, and the separation persists under approximate predictive alignment when the tail prediction-energy fraction vanishes. Separate pointwise fixed-aspect formulas show that the risk-optimal positive scalar correction improves rank-

1

PCR, whereas mean-floor subtraction is generally not optimal for a broad Marchenko--Pastur bulk.

Peng Zhao

Department of Applied Economics and Statistics, University of Delaware

Principal Component Analysis Linear Regression

Jul 17, 2026cs.LG

In-context learning of closed form solution to simple linear regression task using transformer with linear self-attention

In-context learning is a remarkable property of transformers and has recently received a lot of interest. In many studies of in-context learning, it has been shown that transformers are capable of implementing solver for linear and non-linear regression problems, in which the most of them implement gradient descent algorithm. However, it is still unclear whether those implementations have actually been acquired through training. In this paper, we construct a transformer with linear self-attention, which in-context learns the least squares estimate in a simple regression task. The point here is that the closed form (analytical) solution is approximately obtained by using layer normalization rather than an approximate solution based on gradient descent algorithm. Then, we show an experimental example, in which our implementation is mainly used in the transformer trained with l1 regularization when the target output is the least squares estimate.

Katsuyuki Hagiwara

Faculty of Education, Mie University · Kurima-Machiya-cho, Tsu, 514-8507, Japan

In-Context Learning Linear Regression

Jul 13, 2026econ.EM

Partial Identification with Multiple Nonlinear Measurements of a Latent Regressor

We study linear regression when the regressor is latent and observed only through multiple noisy measurements, each a smooth but possibly nonlinear function of the latent variable. The problem is acute in the measurement of occupational exposure to artificial intelligence, where competing scores yield downstream estimates that differ by a factor of eleven. A regression on any single measurement recovers a source-specific coefficient rather than the structural one. We fix the latent scale by requiring the consensus measurement function to be linear and bound the remaining curvature heterogeneity across sources relative to slope. Under this bound, the structural coefficient lies in a closed-form interval centered at a symmetric cross-source estimator. The interval is invariant to unknown source loadings, and its half-width is second order in the curvature bound and sharp to the same order. With at least four measurements, the bound is estimable from the joint distribution of the sources through a split-instrument auxiliary regression, and Imbens-Manski confidence intervals with the Stoye critical value attain uniform coverage over the curvature class, including at the point-identified boundary. The application matches six exposure measures to an American Community Survey panel of 8.88 million person-year observations for 2015 to 2024. The post-2022 employment coefficient changes sign between the language-model measures and the Webb patent-text measure, and an ex ante factor-analytic rule separates the Webb measure as a distinct construct. The five retained sources yield a loading-invariant consensus coefficient of -0.239, with a partial-identification half-width of 1.23 percent of the point estimate, or 1.88 percent at the one-sided 95 percent upper bound on the curvature. We read the application as measurement reconciliation rather than as a causal estimate of AI displacement.

Burhan Ogut, Michelle Yin

American Institutes for Research. · Northwestern University, School of Education and Social Policy.

Partial Identification Linear Regression

Jul 8, 2026cs.LG

Distributed Sketching on Data Partitions for OLS Regression

This paper studies distributed sketching for ordinary least squares (OLS) regression, an approach that distributes small sketches of a large data set over multiple machines to separately construct OLS estimators and average them. Unlike prior studies that consider sketching on the whole data set, we consider sketching on partitioned subsets to further reduce computational cost. Under the fixed design setting, we characterize the exact excess loss of the averaged OLS estimator. Results show that this loss is comparable to the established loss for sketching on the whole data set when the divergence among subset covariances is small.

Luyuan Yang, Brayden Garner, Shayan Shafaei +1

a School of Computer Science, University of Oklahoma, U.S.

Randomized Sketching Linear Regression

Jul 7, 2026cs.LG

The Approximation Ratio for the Risk of Myopic Bayesian Active Learning for Linear Regression

Active learning studies the fundamental question: what data should we choose to observe? The greedy algorithm in optimal experiment design is a common heuristic and also equivalent to myopic Bayesian active learning for linear regression, the common framework where long-term planning is replaced with the one-step optimal choice. In this work, we prove a first-of-its-kind approximation ratio for the greedy algorithm's risk that is tight up to an absolute constant. The approximation ratio is linear in the maximum initial leverage score (MILS), a newly identified quantity fundamental to the greedy algorithm's performance. Finally, we illustrate the results with simple numerical simulations.

Stephen Mussmann

School of Computer Science Georgia Institute of Technology

Active Learning Linear Regression Bayesian Experimental Design Approximation Algorithms

Jun 30, 2026cs.LG

Distributionally Robust Linear Regression With Block Lewis Weights

We present an algorithm for the group distributionally robust (GDR) least squares problem. Given

m

groups, a parameter vector in

\mathbb{R}^d

, and stacked design matrices and responses

\mathbf{A}

and

\mathbf{b}

, our algorithm obtains a

(1+\varepsilon)

-multiplicative optimal solution using

\widetilde{O}(\min\{\mathsf{rank}(\mathbf{A}),m\}^{1/3}\varepsilon^{-2/3})

linear-system-solves of matrices of the form

\mathbf{A}^{\top}\mathbf{B}\mathbf{A}

for block-diagonal

\mathbf{B}

. Our technical methods follow from a recent geometric construction, block Lewis weights, that relates the empirical GDR problem to a carefully chosen least squares problem and an application of accelerated proximal methods. Our algorithm improves over known interior point methods for moderate accuracy regimes and matches the state-of-the-art guarantees for the special case of

\ell_{\infty}

regression. We also give algorithms that smoothly interpolate between minimizing the average least squares loss and the distributionally robust loss.

Naren Sarayu Manoj, Kumar Kshitij Patel

TTIC · Institute for Foundations of Data Science, Yale University

Robust Regression Distributionally Robust Optimization Linear Regression Robust Estimation

Jun 22, 2026stat.ME

Target-Aware Linear Regression Under Distribution Shift

Distribution shift between training and deployment is a pervasive challenge for modern AI systems. In many cases, the target marginals of covariates and response are known or specified through population-level observations, boundary conditions, properties of simulator configurations, or alignment-time distributional constraints. Such knowledge may provide valuable side information for regression estimation. We study this problem in the multivariate linear regression setting with a stable conditional mean

E[Y\mid X]

across source and target, and identify the hybrid-loss estimator, which jointly incorporates both target marginals, as a benchmark target-aware estimator. Its direct computation, however, requires solving a coupled nonlinear optimization that is expensive at scale. Our main contribution is to develop and evaluate two computationally tractable alternatives: a constrained moment-matching estimator and a two-stage estimator that augments ordinary least squares with a calibration step. For all three estimators, we derive and compare closed-form asymptotic mean squared errors, yielding conditions under which the tractable alternatives match or closely approximate the hybrid benchmark, and regimes in which they do not. Monte Carlo experiments across three controlled shift regimes validate the theoretical results, investigate the accuracy-runtime tradeoffs among the three estimators, and translate into guidance on estimator choice. In particular, the two-stage estimator nearly matches the hybrid benchmark in the high signal-to-noise regime at essentially no additional cost, providing theoretical grounding for empirical observations in nonlinear settings.

Zhewen Hou, Tian Zheng

Department of Statistics Columbia University New York, NY

Distribution Shift Robustness Distribution Matching Linear Regression

Jun 20, 2026cs.LG

Alternate loss functions and regression models that achieve robustness to outliers by modulating the learning rate

Most real-world datasets used for training supervised learning models are contaminated with noisy data and outliers leading to large prediction errors. This paper proposes a new approach for achieving robustness where the learning rate is modulated by a factor that is sensitive to outliers. In this approach a reduction of the learning rate is shown to be achieved by using alternate loss functions that are infinitely differentiable, strictly convex or quasiconvex and more closely approximate the absolute error than Huber and log-cosh losses. A comparison of the performance of regression models trained with different loss functions on a wide variety of benchmarks and datasets is presented to demonstrate the superior performance of the Square Root Loss (SRL) and Smooth Mean Absolute Error (SMAE) losses proposed in this paper. Two new robust linear regression models are presented. Highly vectorized robust parameter update formulae that take advantage of modern GPUs for both stochastic and batch gradient descent are presented.

Mathew Mithra Noel, Arindam Banerjee, Yug D. Oswal +2

aSchool of Electrical Engineering, Vellore Institute of Technology, Vellore, 632 014, Tamil Nadu, India · bErnst and Young GDS, Godrej Waterside, Ring Rd, DP Block, Sector V, Bidhannagar, Kolkata, 700 091, West Bengal, India · cSchool of Computer Science and Engineering, Vellore Institute of Technology, Vellore, 632 014, Tamil Nadu, India

Robust Regression Robust Loss Functions Linear Regression Robust Estimation

Jun 16, 2026cs.LG

Measurement noise limits the advantage of nonlinear models over linear models in biomedical prediction

On biomedical tabular data, flexible models such as deep networks, gradient-boosted trees, and kernel methods are repeatedly matched or beaten by linear and logistic regression given the same features. The usual reaction is to treat this as a model-side shortfall, to be fixed with more data, a better architecture, or tuning, on the assumption that the nonlinear structure is there and the model has failed to capture it. We argue that these fixes cannot help when the binding limit is the measurement rather than the model, as it frequently is in biomedicine. Additive noise blurs the population-optimal predictor, and because blurring removes a function's fine, rapidly varying detail before its broad shape, it erases nonlinear structure faster than linear structure. A degree-

k

interaction is attenuated by the

k

-th power of feature reliability, while the linear part is attenuated only once. At the reliabilities typical of biomedical measurement, the nonlinear advantage can vanish even when the underlying biology is strongly nonlinear, and what the noise removes cannot be recovered by a larger cohort or a more flexible model, only by better measurement. The nonlinearity is hidden, not absent, and a tie between linear and flexible models is not by itself a verdict on the biology. These pieces are classical, drawn from measurement-error statistics, psychometrics, and Gaussian analysis, and we assemble them into an exact excess-risk identity. Measurement reliability is one of three conditions, alongside sample size and feature representation, that must align for a flexible model to help, and together they leave only a narrow window that most biomedical tasks fall outside. Across 140 UK Biobank tasks, the gap between flexible and linear models, where it exists, carries the predicted noise signature, and the three conditions can be separated by intervention but not by a benchmark alone.

Marc-Andre Schulz, Kerstin Ritter

Hertie Institute for AI in Brain Health, University of Tübingen, Tübingen, Germany · Tübingen AI Center, University of Tübingen, Tübingen, Germany · Department of Psychiatry and Neurosciences, Charité – Universitätsmedizin Berlin, Berlin, Germany +2

Linear Regression Clinical Prediction

Jun 14, 2026cond-mat.dis-nn

The limits of interpretability in multiple linear regression

Interpreting machine-learning models has attracted increasing attention, particularly in the physical sciences, where one often seeks to understand the underlying mechanisms rather than merely make predictions. Multiple linear regression is often regarded as an interpretable alternative to more complex models, such as deep neural networks, because its predictions are expressed as explicit weighted sums of input features. However, when input features are strongly correlated, namely in the presence of multicollinearity, the learned weights can exhibit large dataset-to-dataset fluctuations and oscillatory behavior across physically similar features, making their interpretation difficult or even impossible. Although the instability of the weights under multicollinearity is well known in statistics, its consequences for physical interpretation, in particular its connection to oscillatory weights across physically similar features, have not been systematically clarified. Here, we theoretically discuss the mechanism behind this loss of interpretability by analyzing the eigenmodes of the feature correlation matrix. We show that small-eigenvalue modes associated with multicollinearity amplify fluctuations in the weights and generate oscillatory patterns that do not necessarily reflect meaningful contributions. We test this theoretical picture numerically on physics datasets and show that Ridge regularization suppresses these unstable modes, although the resulting weights must still be interpreted with caution. We further confirm the generality of our findings beyond physics by analyzing a diverse collection of publicly available datasets. Our results clarify why, in the presence of multicollinearity, physical interpretation can remain difficult even for linear regression models.

Anand Sharma, Chen Liu, Daniele Coslovich +1

Indian Institute of Science Education and Research, Dr. Homi Bhabha Road, Pashan, Pune 411008, India · Innovation and Research Division, Ge-Room Inc., 93160 Noisy le Grand, France · Dipartimento di Fisica, Universit`a di Trieste, Strada Costiera 11, 34151, Trieste, Italy∗ +1

Interpretable ML Linear Regression

Jun 12, 2026stat.ML

Geometric Domain Adaptation via Optimal Transport for Linear Regression in R^2

Optimal Transport has become recently a powerful method for domain adaptation by aligning source and target distributions. We study a supervised domain adaptation problem where source and target domains are related by a rotation or a translation or a homothety in

\mathbb{R}^2

. We prove that the optimal transport map recovers the underlying map when using a

p-

norm cost with

p \geq 2

. Based on this insight, we develop a method combining

K-

means and optimal transport to estimate the underlying map, enabling adaptation of linear regression models when target data is scarce. Simulations demonstrate improved performance over baseline methods. Rather than relying on highly expressive deep learning architectures, we focus on classical machine learning models to emphasize interpretability and theoretical insight. This perspective allows us to explicitly characterize the role of optimal transport in recovering geometric transformations such as rotations, translations, and homotheties. Our contributions include a theoretical result linking optimal transport and rotations, translations and homothecies in

\mathbb{R}^2

, and a practical method for adaptation in linear regression offering both conceptual clarity and applied value in domain adaptation tasks in this space.

Brian Britos, Mathias Bourel

IESTA, Facultad de Ciencias Económicas y de Administración, Universidad de la República, Montevideo, Uruguay · IRL-2030, Instituto Franco-Uruguayo de Matemática e Interacciones (IFUMI)

Domain Adaptation Linear Regression Optimal Transport

Jun 9, 2026math.ST

Bidirectional Random Projections

This paper analyzes bidirectional random projections for ordinary least squares (OLS) regression under the fixed design setting. Let

(X,Y) \in \mathbb{R}^{n \times p} \times \mathbb{R}^n

be a sample and

R \in \mathbb{R}^{n_1 \times n}, W \in \mathbb{R}^{p \times p_1}

be two properly distributed random projections. We develop an expected excess loss bound for the OLS estimator built on

(WXR, WY)

. Compared to an established bound for OLS estimator built on

(XR, Y)

, the gap is approximately

O\left( p_1 + C \frac{1}{p_1} \right)

, where

C

scales with

n_1/n

and can be negative for small

n_1/n

. Its implications are confirmed by numerical results on real-world data.

Chao Lan, Luyuan Yang

a School of Computer Science, University of Oklahoma, U.S.

Randomized Sketching Linear Regression Generalization Bounds

May 28, 2026cs.LG

Ridge Regression from Poisson Resetting: A Renewal Perspective on Spectral Regularization

We connect stochastic resetting from non-equilibrium statistical physics with ridge regularization in statistical learning. For linear gradient flow, resetting to the origin at rate

r

produces stationary mean

(X^\top X+rI)^{-1}X^\top y

, exactly the ridge estimator with penalty

λ=r

. This uses the known Laplace-transform relationship between ridge regression and exponential-time averaging of gradient flow, with the exponential time now interpreted as the stationary age associated with Poisson resetting. We then extend this identity to general renewal reset laws: the exponential reset time distribution is the unique renewal law whose stationary mean reproduces scalar ridge in every eigendirection as an exact filter identity for every positive curvature, while non-exponential renewal laws generate alternative spectral filters. At the fluctuation level, we study a separate additive Ornstein-Uhlenbeck extension with constant diffusion, interpreted as a stylized SGD approximation. In this setting, the equality holds only at the level of the mean, since the reset process has a nonzero stationary covariance from accumulated OU noise and reset-timing variance, whereas deterministic ridge is a fixed estimator with the same center. Stylized experiments compare the deterministic renewal-induced filters directly and illustrate when filters induced by non-exponential reset-time laws can differ predictively from ridge. The results for the stationary mean and the induced spectral filters are established for continuous-time gradient flow with isotropic resetting on quadratic objectives; the covariance and risk formulas additionally assume additive noise with state-independent covariance.

Petar Jolakoski

Research Center for Computer Science and Information Technologies, Macedonian Academy of Sciences and Arts, Bul. Krste Misirkov 2, 1000 Skopje, Macedonia

Spectral Regularization Linear Regression Ridge Regression

May 28, 2026stat.ML

Joint Model and Data Sparsification via the Marginal Likelihood

Sparse recovery in linear systems underpins applications from signal processing to high-dimensional regression. Sparse Bayesian Learning, grounded in the principle of automatic relevance determination (ARD), offers a practical Bayesian mechanism for feature sparsity via marginal likelihood optimization. Yet, its reliance on a homoscedastic noise model renders it sensitive to data contaminations such as outliers or misspecified noise, harming model fit and predictions. Instead, we propose jointly learning individual feature and sample relevancies, enabling simultaneous model and data sparsification via a single Bayesian objective. This symmetric pruning of model and data offers a natural extension that preserves conjugacy, admits closed-form updates for standard optimization procedures, and aligns with perspectives from robust regression and influence functions. Empirical results across diverse regression tasks affirm that a joint ARD approach consistently yields both sparse and robust prediction models.

Alexander Timans, Thomas Möllenhoff, Christian A. Naesseth +2

1UvA-Bosch Delta Lab, University of Amsterdam · 2RIKEN Center for AI Project, Tokyo, Japan · Department of Computer Science, Johns Hopkins University.

Dataset Pruning Robust Regression Sparse Recovery Bayesian Inference Linear Regression

May 27, 2026cs.LG

Optimal ridge regularization revisited

We consider

L^2

-regularized linear (ridge) regression over a finite data sample

X

with bounded covariance and linear prediction targets

y

with additive isotropic noise of finite variance. We present an iterative procedure to compute the optimal regularization strength numerically from the generative parameters in the fixed-

X

setting and prove its convergence at limited noise levels. Our experimental evaluation over synthetic data shows that the proposed procedure combined with sample-based parameter estimates attains near-optimal random-

X

generalization across a wide range of sample sizes, aspect ratios, and noise levels, at an added computational cost equivalent to one preliminary ridge regression in the underparameterized regime and two in the overparameterized case.

Jack Timmermans, Sergio A. Alvarez

Department of Computer Science, Boston College, Chestnut Hill, MA 02467 USA

Linear Regression Statistical Learning Theory Ridge Regression Hyperparameter Optimization