Kernel Regression

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 29

Sep 16, 2026stat.ML

Fast Learning Rates for Physics-Informed Kernel Methods

In physics-informed machine learning, a target function

u^*

is learned from noisy value observations

y_i=u^*(x_i)+ \varepsilon_i

, together with differential information, given either by noisy observations

d_j=(Du^*)(z_j)+ξ_j

or by a known physical constraint

Du^*=v

. We consider the setting where

D

is a linear differential operator and analyze a physics-informed kernel estimator

\hat u

combining

n

value observations and

m

differential observations. In this context, we ask how much can differential information improve predictions, and how does this improvement depend quantitatively on

n

,

m

, and

D

. We prove finite-sample bounds, supported by numerical simulations, revealing a two-regime structure for the prediction error. When

m

is limited, the rate depends jointly on

n

and

m

; when

m

exceeds a problem-dependent threshold, the rate saturates and matches the oracle rate obtained when the perfect constraint

D \hat u = Du^*

is imposed. Examples are discussed for Sobolev spaces which are reproducing kernel Hilbert spaces and include partial Laplacian constraints on the torus and gradient observations on bounded domains. These examples illustrate the range of possible learning rate improvements --- from the standard nonparametric

n^{-1/4}

to the parametric rate

n^{-1/2}

. Finally, we derive physically consistent rates in a stronger norm that jointly controls the errors in

\hat u

and

D\hat u

.

Luc Brogat-Motte, Joachim Bona-Pellissier, Giacomo Meanti +1

SAIL Unit, Istituto Italiano di Tecnologia, Genoa, Italy · MaLGa Center, DIBRIS, Università degli Studi di Genova, Genoa, Italy

Kernel Regression Physics-Informed ML Statistical Learning Theory

Sep 15, 2026cs.LG

Information Geometric Self-Organization at the Edge of Stability in High-Capacity Kernel Associative Memories

High-capacity associative memories based on Kernel Logistic Regression (KLR) exhibit exceptional storage capabilities and robustness. Previous empirical studies identified a hyperparameter regime, the "Ridge of Optimization," where attractor stability is maximized. However, the geometric nature of this regime and the optimization dynamics required to reach it have remained unclear. In this paper, we investigate the static geometry of the parameter space and the learning trajectory of Gradient Descent (GD) in KLR-trained Hopfield networks. Using the eigenvalue spectrum of the Hessian, we reveal that the Ridge corresponds to a phase boundary located adjacent to a rank-1 spectral collapse, acting as a geometric singularity where the principal curvature is massively amplified. Furthermore, we demonstrate that the learning dynamics exhibit a transient self-stabilizing behavior driven by the Edge of Stability (EoS) phenomenon. Rather than seeking flat regions, the network parameters are driven toward a state where the local curvature dynamically equilibrates near the stability limit dictated by the learning rate, allowing the optimization to survive the initial instability. We provide analytical derivations for both the rank-1 asymptotic collapse and the dynamic feedback loop governing this equilibration. These findings suggest that optimal, high-capacity memory representations are not formed in flat minima, but are dynamically sculpted at the highly curved boundaries of geometric singularities.

Akira Tamamori

Faculty of Information Science, Aichi Institute of Technology, Aichi, 470-0392, Japan

Information Geometry Edge of Stability Kernel Regression Hopfield Networks Neural Network Training Dynamics Associative Memory

Sep 15, 2026cs.LG

A Weighted Kernel Method for Approximation that Adapts to Learned Multivariable Structure

Approximating the input-output behavior of a multivariable black-box function from limited data is challenging when blind to the importance of its inputs and their interactions. We introduce total sensitivity kernels (TSKs), a method based on families of weighted ANOVA kernels that learn and adapt to this multivariable structure. TSKs parameterize the weights on each multivariable component of the target function by factors for each input. We propose learning these factors directly from function evaluations by selecting the reproducing kernel Hilbert space (RKHS) in which the target function has minimum norm. Under suitable conditions, we show that this norm-minimization problem admits a unique solution, and we establish consistency of a finite-data formulation based on minimum-norm interpolation. The learned TSK factors characterize the participation of individual inputs across interactions and main effects, providing a kernel-dependent notion of input sensitivity related to total Sobol indices. Numerical experiments demonstrate that adapting the kernel to learned multivariable structure can substantially improve approximation accuracy over a standard product kernel.

John E. Darges, Laura Weidensager

Department of Mathematics, Emory University · Department of Mathematics, Simon Fraser University

Kernel Regression Kernel Methods Statistical Learning Theory

Sep 14, 2026stat.ML

Learning under Target Shift: Optimal Density Ratio Estimation and Importance-Weighted Regression

We study density ratio estimation and importance-weighted regression under target shift with continuous outputs. Under target shift, the conditional distribution of the inputs given the outputs remains invariant across the training and test distributions, while the output marginal distribution may change. Although this problem has been extensively studied for discrete outputs, the continuous setting is substantially less understood: the importance weights are determined by an unknown density ratio function, for which existing estimation methods lack explicit finite-sample convergence rates. We propose a spectral regularization method in a reproducing kernel Hilbert space (RKHS) for estimating the continuous density ratio from labeled training samples and unlabeled test inputs. Under a source condition with regularity parameter

ι>0

, we establish high-probability finite-sample guarantees and show that the estimator achieves the capacity-independent minimax-optimal RKHS-norm rate

O(n_η^{-ι/(2ι+2)})

. We then incorporate the estimated density ratio into importance-weighted regression and characterize the propagation of density-ratio estimation error to the final predictor. When sufficiently many samples are available for density ratio estimation, the resulting regression estimator attains the minimax-optimal rates of standard kernel regression. These results establish a finite-sample theory for continuous density ratio estimation and importance-weighted learning under target shift.

Ren-Rui Liu, Zheng-Chu Guo

School of Mathematical Sciences, Zhejiang University, Hangzhou 310058, China

Kernel Regression Distribution Shift Density Ratio Estimation

Sep 10, 2026stat.ML

Generalization Analysis of Distributed Kernel-based Robust Gradient Descent Algorithms

In this paper, we investigate the generalization performance of distributed gradient descent algorithms in a reproducing kernel Hilbert space under a robust loss function

l_σ

. By exploiting the spectral characterization of gradient descent together with the intrinsic properties of robust loss functions, we establish optimal learning rates for the distributed kernel-based robust gradient descent (DKRGD) algorithm with an appropriately chosen scale parameter

σ

. The proposed parameter choice of

σ

simultaneously alleviates the saturation phenomenon and guarantees statistical robustness. A key technical contribution is a novel error analysis that provides substantially sharper bounds for products of operators, thereby significantly relaxing existing restrictions on the maximum number of local machines while retaining optimal learning rates. Finally, we develop a communication-efficient strategy that further improves the convergence performance of DKRGD.

Jun-Yi Meng, Zheng-Chu Guo, Yuan Mao

School of Mathematical Sciences, Zhejiang University, Hangzhou 310058, P. R. China · College of Informatics, Huazhong Agricultural University, Wuhan 430070, P. R. China

Kernel Regression Communication-Efficient Distributed Training Robust Loss Functions Distributed Training Statistical Learning Theory

Jul 30, 2026stat.ML

The Noise Premium in Adversarial Training for Kernel Regression

Adversarial training can improve the robustness of predictive models to bounded perturbations, often at the cost of statistical efficiency. We study this trade-off in kernel regression over a reproducing kernel Hilbert space (RKHS). It is shown that, under squared loss, adversarial training in RKHS introduces a term involving the product of the function norm with the mean absolute value of the response noise, which we call the \textit{noise premium}. Our analysis shows that the noise premium makes the prediction error of adversarial training converge strictly more slowly than the nonparametric minimax benchmark even after balancing approximation and estimation errors. Moreover, for a fixed perturbation budget, once the budget exceeds a certain threshold, the solution to adversarial training collapses to the zero function. To mitigate these effects of the noise premium, we propose noise-debiased adversarial training. The resulting noise-debiased estimator can attain the minimax optimal rate up to a logarithmic factor for the prediction error, raises the collapse threshold, and admits an explicit bound on the increase in adversarial loss. Numerical experiments on synthetic and real data support the theoretical findings and validate the effectiveness of the proposed noise-debiased method.

Yiling Xie, Xiaoming Huo

Department of Decision Analytics and Operations, City University of Hong Kong · H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology

Adversarial Training Kernel Regression Nonparametric Regression Minimax Estimation Statistical Learning Theory

Jul 19, 2026stat.ML

Kernel Regression with Tensor Trains and Hadamard Overparameterization

Kernel regression with tensor trains and Hadamard overparameterization (KReTTaH) is introduced as a training-data-free, interpretable, and nonparametric framework for multi-way data imputation. The imputation problem is reformulated as regression in reproducing kernel Hilbert spaces (RKHS), where the tensor regression coefficients are explicitly constrained to lie on fixed-rank tensor-train (TT) manifolds and structured via Hadamard overparameterization to promote sparsity and high representational efficiency. Rather than relying on costly cross-validation, KReTTaH jointly optimizes the TT coefficient tensors and the kernel covariance matrices within a Riemannian product-manifold framework -- the former on fixed-rank TT manifolds, the latter on the manifold of positive-definite matrices -- thereby enabling automated kernel-hyperparameter selection. Numerical tests on two challenging applications -- imputation of high-dimensional functional magnetic resonance imaging (fMRI) data and recovery of missing edge flows in dynamic graphs -- demonstrate that KReTTaH consistently outperforms state-of-the-art tensor-, Bayesian-, and neural-network-based baselines in terms of modeling accuracy.

Duc Thien Nguyen, Konstantinos Slavakis, Eleftherios Kofidis +1

Institute of Science Tokyo, Dept. Information and Communications Engineering, Yokohama, Japan · University of Piraeus, Dept. Statistics and Insurance Science, Greece · Florida Atlantic University, Center for Connected Autonomy and Artificial Intelligence & Dept. Electrical Engineering and Computer Science, USA

Incomplete Data Imputation Kernel Regression

Jul 7, 2026stat.ML

A Function-Space Dichotomy for Compositional Learning: Exponential Sub-Optimality of the Neural Tangent Kernel

A persistent empirical observation is that trained neural networks outperform their neural tangent kernel (NTK) limit on tasks with compositional structure, yet a quantitative account of

\textbf{when}

and

\textbf{by how much}

has been lacking. Working on the unit circle, we give such an account through a dichotomy between two complexity measures of the target: its

\textbf{Fourier complexity}

, which controls NTK kernel regression, and its

\textbf{architectural complexity}

, which controls learning over depth-

L

, width-

w

ReLU networks with the variation norm of the weights bounded by

R

. We first characterize the minimax rate of the architecture class

\mathcal{C}_{L,w,R}

, pinning it down up to a single factor of

L

: between

Ω(Lw^2R^2/n)

and

\tilde{O}(L^2w^2R^2/n)

. We then show the NTK estimator sits

\textbf{exponentially}

above this floor whenever the two complexities decouple: for the depth-

L

iterated sawtooth, NTK regression needs

Ω(4^L)

samples while the minimax floor is polynomial in

L

. Numerical experiments confirm the theoretical claims: on bandlimited smooth targets, the NTK is competitive or better, while on the hypercube sparse-parity model, a standard two-layer network beats the NTK by four to six orders of magnitude in test error. The gap is thus a function-space property, a mismatch between the kernel's smoothness bias and the target's compositional structure, rather than a generic kernel-versus-network phenomenon.

Arkaprabha Ganguli, Emil Constantinescu

Mathematics & Computer Science Division, Argonne National Laboratory, Lemont, IL, USA

Neural Network Generalization Kernel Regression ReLU Neural Networks Neural Network Approximation Theory Neural Tangent Kernel

Jul 7, 2026cs.SD

Learning-based Physics-Constrained Neural Kernel for Sound Field Estimation With Source-Position-Dependent Directional Weighting

A learning-based physics-constrained neural kernel for sound field estimation is proposed. Sound field estimation aims to estimate the spatial distribution of an acoustic field from a discrete set of microphone measurements, which have a wide range of applications. Among existing sound field estimation methods, kernel-regression-based methods offer a flexible and principled framework for incorporating physical constraints and allow inference through linear operation. It is also possible to adapt the kernel function to the target acoustic environment by representing the directional weighting function as an implicit neural representation (INR) and optimizing hyperparameters using measurements. However, the kernel function is generally optimized for single snapshot measurements of the microphones, which can lead to strong overfitting and poor generalization. We propose a source-position-dependent INR for the directional weighting function, enabling the kernel function to capture common directional patterns and to generalize to unseen source positions in the target acoustic environment. Experimental results indicate that our proposed method outperforms the snapshot-based method by estimating a directional weighting function that matches the directivity of the target sound field.

Mattia Marella, Shoichi Koyama

National Institute of Informatics, Tokyo, Japan · University of Ferrara, Ferrara, Italy

Kernel Regression Implicit Neural Representations Kernel Methods Physics-Informed ML Spatial Audio

Jun 30, 2026cs.LG

Learning dynamical systems from noisy data with Weak-form Kernel Ridge Regression

Accurate prediction of complex dynamical systems from noisy measurements remains a significant challenge in scientific computing. Kernel ridge regression learning strategies are often effective when applied to clean data, but have limited success with noisy data. Recent work has observed that a weak formulation can act to filter noisy data, and different learning strategies have achieved increased noise robustness with a weak-form framework. In this manuscript, we give an overview of the filtering mechanism behind the weak formulation and provide a bias-variance error decomposition. Using these insights, we combine a weak formulation with a kernel learning strategy to propose Weak-form Kernel Ridge Regression (WKRR) for learning dynamical systems. The proposed framework is simple to implement, effective for both clean and noisy data, and outperforms several baseline methods. We demonstrate the performance of WKRR on chaotic benchmark systems in up to 64 dimensions, as well as 15,000-dimensional real-world fluid data.

Max Kreider, John Harlim, Daning Huang

Department of Mathematics The Pennsylvania State University, University Park, PA 16802, USA · Department of Mathematics, Institute for Computational and Data Sciences The Pennsylvania State University, University Park, PA 16802, USA · Department of Aerospace Engineering The Pennsylvania State University, University Park, PA 16802, USA

Kernel Ridge Regression Kernel Regression Nonlinear System Identification Chaotic Dynamical Systems Scientific ML

Jun 4, 2026cs.LG

TorchKM: A GPU-Oriented Library for Kernel Learning and Model Selection

TorchKM is an open-source library for kernel machines, including support vector machines, kernel logistic regression, and kernel quantile regression, with GPU acceleration. The library features a scikit-learn-style API and is designed to exploit GPU-friendly linear algebra, accelerating the full training and model-selection pipeline through intelligent reuse of matrix operations. Benchmarks show competitive predictive performance with substantial speedups over standard baselines. The efficiency and programmable design also make TorchKM a kernel-learning component for AI-driven workflows. Code and documentation are available at https://github.com/YikaiZhang95/torchkm, and the package can be easily installed via PyPI.

Yikai Zhang, Gaoxiang Jia, Jie Ding +1

Department of Statistics and Actuarial Science, University of Iowa, Iowa City, IA 52242, USA · Individual Researcher · AIScientists, Inc. (MorphMind), 245 Main Street, Cambridge, MA 02142, USA

Model Selection Kernel Regression GPU Acceleration Support Vector Machines Kernel Methods

May 25, 2026cs.LG

Conditional KRR: Injecting Unpenalized Features into Kernel Methods with Applications to Kernel Thresholding

Conditionally positive definite (CPD) kernels are defined with respect to a function class

\mathcal{F}

. It is well known that such a kernel

K

is associated with its native space (defined analogously to an RKHS), which in turn gives rise to a learning method -- called conditional kernel ridge regression (conditional KRR) due to its analogy with KRR -- where the estimated regression function is penalized by the square of its native space norm. This method is of interest because it can be viewed as classical linear regression, with features specified by

\mathcal{F}

, followed by the application of standard KRR to the residual (unexplained) component of the target variable. Methods of this type have recently attracted increasing attention. We study the statistical properties of this method by reducing its behavior to that of KRR with another fixed kernel, called the residual kernel. Our main theoretical result shows that such a reduction is indeed possible, at the cost of an additional term in the expected test risk, bounded by

\mathcal{O}(1/\sqrt{N})

, where

N

is the sample size and the hidden constant depends on the class

\mathcal{F}

and the input distribution. This reduction enables us to analyze conditional KRR in the case where

K

is positive definite and

\mathcal{F}

is given by the first

k

principal eigenfunctions in the Mercer decomposition of

K

. We also consider the setting where

\mathcal{F}

consists of

k

random features from a random feature representation of

K

. It turns out that these two settings are closely related. Both our theoretical analysis and experiments confirm that conditional KRR outperforms standard KRR in these cases whenever the

\mathcal{F}

-component of the regression function is more pronounced than the residual part.

Rustem Takhanov, Zhenisbek Assylbekov

Department of Mathematics, Nazarbayev University, Astana, Kazakhstan · Nazarbayev University Research Administration, Astana, Kazakhstan · Purdue University Fort Wayne, Indiana, USA.

Kernel Ridge Regression Kernel Regression Random Feature Methods Kernel Methods Statistical Learning Theory Ridge Regression

May 14, 2026stat.ML

Average Gradient Outer Product in kernel regression provably recovers the central subspace for multi-index models

We study a prototypical situation when a learned predictor can discover useful low-dimensional structure in data, while using fewer samples than are needed for accurate prediction. Specifically, we consider the problem of recovering a multi-index polynomial

f^*(x)=h(Ux)

, with

U\in\mathbb{R}^{r\times d}

and

r\ll d

, from finitely many data/label pairs. Importantly, the target function depends on input

x

only through the projection onto an unknown

r

-dimensional central subspace. The algorithm we analyze is appealingly simple: fit kernel ridge regression (KRR) to the data and compute the Average Gradient Outer Product (AGOP) from the fitted predictor. Our main results show that under reasonable assumptions the top

r

-dimensional eigenspace of AGOP provably recovers the central subspace, even in regimes when the prediction error remains large. Specifically, if the target function

f^*

has degree

p^*

, it is known that

n\asymp d^{p^*}

samples are necessary for KRR to achieve accurate prediction. In contrast, we show that if a low degree

p

component of

f^*

already carries all relevant directions for prediction, subspace recovery occurs in the much lower sample regime

n\asymp d^{p+δ}

for any

δ\in(0,1)

. Our results thus demonstrate a separation between prediction and representation, and provide an explanation for why iterative kernel methods such as Recursive Feature Machines (RFM) can be sample-efficient in practice.

Libin Zhu, Damek Davis, Dmitriy Drusvyatskiy +1

Department of Mathematics, University of Washington, Seattle, WA 98195 · Wharton Department of Statistics and Data Science, University of Pennsylvania, Philadelphia, PA 19104, USA · Department of Mathematics, U. Washington, Seattle, WA 98195 +1

Kernel Ridge Regression Representation Learning Kernel Regression Inverse Dimensionality Reduction Dimensionality Reduction Multi-Index Models

May 14, 2026cs.LG

AQKA: Active Quantum Kernel Acquisition Under a Shot Budget

Estimating an

N \times N

quantum kernel from circuit fidelities requires

Θ(N^2 S)

measurement shots, the dominant bottleneck for deployment on near-term hardware. Existing budget-saving methods (Nyström-QKE, ShoFaR, kernel-target alignment) sub-sample \emph{which} entries to measure but allocate shots \emph{uniformly} within their chosen subset, ignoring how much each entry drives the downstream classifier. We close this gap with two contributions. \textbf{First, a complete regime decomposition} for shot-budgeted quantum kernel learning: a principled menu of when each allocator wins. Our method, \emph{AQKA}, dominates the budget-limited regime (

B \lesssim 16 n_{\mathrm{pairs}}

) on sparse-sensitivity KRR, with the gap \emph{growing} from

+8

to

+25

pts over uniform as

N

scales

225{\to}1000

and reaching

+26

--

32

pts on an \texttt{ibm_pittsburgh} (156-qubit Heron) hardware kernel; Nyström-QKE wins at saturating budgets on planted-sparse via low-rank reconstruction; ShoFaR is competitive only at extreme low budgets. \textbf{Second, a closed-form pair-level acquisition theory}:

s_{ij}^{\star} \propto |g_{ij}|\sqrt{K_{ij}(1-K_{ij})}

with explicit gradient

g_{ij}

for KRR (Lemma1, $|β_iα_j+β_jα_i|\sqrt{K_{ij}(1-K_{ij})}$ ) and SVM via the envelope theorem ( $|η_i^*η_j^*|\sqrt{K_{ij}(1-K_{ij})}$ ); a \emph{corrected} sparsity-aware Cauchy--Schwarz rate $ρ\le 2m/N$ matching empirics (vs.\ the naive $m^2/N^2$ ); an explicit-constant plug-in regret bound (Theorem2); and a tighter SVM ceiling

ρ^{\mathrm{SVM}} \le m_{\mathrm{sv}}^2/N^2

. We close with the first multi-seed live online adaptive shot allocation on quantum hardware:

+17.0 \pm 4.8

pts at

N{=}20

on \texttt{ibm_aachen} (

3.5σ

, 5 seeds), with the advantage holding at

N{=}30

at higher budget on \texttt{ibm_berlin} (

+14.0 \pm 8.5

pts, 5 seeds).

Jian Xu, Chao Li, Delu Zeng +2

1RIKEN iTHEMS · 2RIKEN AIP · 3South China University of Technology +1

Kernel Regression Active Learning Quantum Kernel Methods Kernel Methods

May 14, 2026stat.ML

Large Dimensional Kernel Ridge Regression: Extending to Product Kernels

Recent studies have reported

\textit{saturation effects}

and

\textit{multiple descent behavior}

in large dimensional kernel ridge regression (KRR). However, these findings are predominantly derived under restrictive settings, such as inner product kernels on sphere or strong eigenfunction assumptions like hypercontractivity. Whether such behaviors hold for other kernels remains an open question. In this paper, we establish a broad, new family of large dimensional kernels and derive the corresponding convergence rates of the generalization error. As a result, we recover key phenomena previously associated with inner product kernels on sphere, including:

i)

the

\textit{minimax optimality}

when the source condition

s\le 1

;

ii)

the

\textit{saturation effect}

when

s>1

;

iii)

a

\textit{periodic plateau phenomenon}

in the convergence rate and a

\textit {multiple-descent behavior}

with respect to the sample size

n

.

Yang Zhou, Yicheng Li, Yuqian Cheng +1

Department of Statistics and Data Science Tsinghua University Beijing, 100084, China · Department of Mathematical Science Tsinghua University Beijing, 100084, China

Double Descent Kernel Ridge Regression Kernel Regression Kernel Methods Ridge Regression Generalization Bounds

May 10, 2026cs.LG

On Uniform Error Bounds for Kernel Regression under Non-Gaussian Noise

Providing non-conservative uncertainty quantification for function estimates derived from noisy observations remains a fundamental challenge in statistical machine learning, particularly for applications in safety-critical domains. In this work, we propose novel non-asymptotic probabilistic uniform error bounds for kernel-based regression. Compared to related bounds in the literature that are restricted to (conditionally) independent sub-Gaussian noise, our bounds allow to consider a broad class of non-Gaussian distributions, such as sub-Gaussian, bounded, sub-exponential, and variance/moment-bounded noise. Moreover, our results apply to correlated and uncorrelated noise. We compare our proposed error bounds with existing results in terms of the induced uncertainty region and their performance in safe control, demonstrating the tightness of the proposed bounds.

Johannes Teutsch, Oleksii Molodchyk, Marion Leibold +2

Chair of Automatic Control Engineering, Department of Computer Engineering, Technical University of Munich, 80333 Munich, Germany · Institute of Control Systems, Hamburg University of Technology, 21073 Hamburg, Germany · Department of Electrical and Computer Engineering, National University of Singapore, 117583 Singapore

Kernel Regression Uncertainty Quantification Nonparametric Regression Convergence Analysis Kernel Methods Statistical Learning Theory

May 8, 2026cs.LG

Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression

Mechanistic accounts of in-context learning (ICL) have identified iterative algorithms for linear regression and related linear prediction tasks, often using linear or ReLU attention variants. For nonlinear ICL, prior work has related softmax and kernelized attention to functional-gradient-type dynamics, but it remains unclear whether a standard transformer with softmax attention can implement a convergent solver with an end-to-end prediction-error guarantee. In this paper, we study in-context kernel ridge regression (KRR) with Gaussian kernels and show that a standard softmax-attention transformer can approximate the KRR predictor during its forward pass by implementing preconditioned Richardson iteration on the associated kernel linear system. Under bounded-data assumptions, we construct a single-head transformer with

O(\log(1/ε))

blocks and MLP width

O(\sqrt{N/ε})

that achieves

ε

-accurate prediction for prompts of length

N

. Our construction reveals a functional decomposition within the transformer architecture: softmax attention produces a row-normalized Gaussian-kernel operator needed for cross-token interactions, while ReLU MLP layers act locally to approximate the intra-token scalar arithmetic required by the update. Empirically, we train GPT-2-style transformers on Gaussian-process regression tasks to further test the preconditioned Richardson interpretation. Through linear probing, we compare the transformer's layer-wise predictions with the step-wise outputs of classical KRR solvers and find that its error profiles align most consistently with preconditioned Richardson iteration. Ablation studies further support this interpretation. Together, our theory and experiments identify preconditioned Richardson iteration as a concrete mechanism that softmax-attention transformers can realize for nonlinear in-context Gaussian-kernel regression.

Mingsong Yan, Dongyang Li, Charles Kulick +1

Department of Mathematics, University of California, Santa Barbara, CA

Softmax Attention Transformer Interpretability Kernel Ridge Regression Kernel Regression Self-Attention In-Context Learning+2

May 8, 2026stat.ML

Characterizing and Correcting Effective Target Shift in Online Learning

Online learning from a stream of data is a defining feature of intelligence, yet modern machine learning systems often struggle in this setting, especially under distributional shift. To understand its basic properties, we study the relationship between online and offline learning in the context of kernel regression. We derive a closed-form expression for the function learned by online kernel regression, revealing that online kernel regression is equivalent to offline regression with shifted, inaccurate target outputs. Conversely, we show that by compensating for this effective shift in the teaching signal through target correction, online kernel-based learning can provably learn the same predictor as its offline counterpart. We derive both a closed-form expression for this target correction and an iterative form that can be applied sequentially. Applying this framework to image classification tasks on CIFAR-10 and CORe50, we show that online stochastic gradient descent with iteratively corrected targets outperforms learning with the true targets in continual learning settings. This work therefore provides a basic framework for analyzing and improving online learning in non-stationary environments.

Ziyan Li, Naoki Hiratani

Department of Physics Washington University in St. Louis St. Louis, USA · Department of Neuroscience Washington University in St. Louis St. Louis, USA

Label Shift Kernel Regression Distribution Shift Continual Learning Kernel Methods Online Learning

May 7, 2026math.ST

Optimal Confidence Band for Kernel Gradient Flow Estimator

In this paper, we investigate the supremum-norm generalization error and the uniform inference for a specific class of kernel regression methods, namely the kernel gradient flows. Under the widely adopted capacity-source condition framework in the kernel regression literature, we first establish convergence rates for the supremum norm generalization error of both continuous and discrete kernel gradient flows under the source condition

s>α_0

, where

α_0\in(0,1)

denotes the embedding index of the kernel function. Moreover, we show that these rates match the minimax optimal rates. Building on this result, we then construct simultaneous confidence bands for both continuous and discrete kernel gradient flows. Notably, the widths of the proposed confidence bands are also optimal, in the sense that their shrinkage rates are greater than, while can be arbitrarily close to, the minimax optimal rates.

Yuqian Cheng, Zhuo Chen, Qian Lin

Department of Mathematical Sciences Tsinghua University Beijing, 100084, China · Department of Statistics and Data Science2026 Tsinghua University Beijing, 100084, ChinaMay

Kernel Regression Kernel Methods Generalization Bounds

May 4, 2026cs.LG

Differentiable Kernel Ridge Regression for Deep Learning Pipelines

Deep neural networks dominate modern machine learning, while alternative function approximators remain comparatively underexplored at scale. In this work, we revisit kernel methods as drop-in components for standard deep learning pipelines. We introduce \emph{Sparse Kernels} (SKs), a differentiable, localized, and lazy variant of kernel ridge regression (KRR) that defers training to inference time and reduces to the solution of small local systems. We integrate SKs into PyTorch as modular layers that preserve end-to-end trainability, and we show that they expose three distinct sets of parameters -- feature representations, target values, and evaluation points -- each of which can be fixed or learned. This decomposition broadens the design space available to practitioners, enabling, in particular, training-free transfer, nonlinear probing, and hybrid kernel-neural models. Across convolutional networks, vision transformers, and reinforcement learning, SK-based modules serve two complementary roles: in some settings, they match the performance of trained neural readouts with substantially less training; in others, they augment existing models and improve their performance when used as additional components. Our results suggest that kernel methods, once made scalable and differentiable, can be readily integrated with deep learning rather than treated as a separate paradigm.

Jean-Marc Mercier, Gabriele Santin

1MPG-Partners, Paris, France · Department of Environmental Sciences, Informatics and Statistics, Ca’ Foscari University of Venice, Italy

Kernel Ridge Regression Kernel Regression Kernel Methods Ridge Regression

Apr 24, 2026cs.LG

Complex Stochastic Gradient Descent and Directional Bias in Reproducing Kernel Hilbert Spaces

Stochastic Gradient Descent (SGD) is a known stochastic iterative method popular for large-scale convex optimization problems due to its simple implementation and scalability. Some objectives, such as those found in complex-valued neural networks, benefit from updates like in SGD and Gradient Descent (GD) with a newly defined ``gradient'' that allows for complex parameters. This complex variant of the SGD/GD methods has already been proposed, but convergence guarantees without analyticity constraints have not yet been provided. We propose a variant of SGD (complex SGD) that allows for complex parameters, and we provide convergence guarantees under assumptions that parallel those from the real setting. Notably, these results extend to GD as well, and with the same set of assumptions, we confirm that some directional bias results extend from the real to the complex setting for kernel regression problems. We provide empirical results demonstrating the efficacy of the complex SGD in kernel regression problems utilizing complex reproducing kernel Hilbert spaces. In particular, we demonstrate we may recover superoscillation functions and Blaschke products from the Fock Space and Hardy Space, respectively, as the optimal functions for a particular choice of a loss function.

Natanael Alpay, Emeric Battaglia

Kernel Regression Reproducing Kernel Hilbert Spaces Kernel Methods Stochastic Optimization Convergence Stochastic Gradient Descent

Apr 24, 2026stat.ML

Pack only the essentials: Adaptive dictionary learning for kernel ridge regression

One of the major limits of kernel ridge regression (KRR) is that storing and manipulating the kernel matrix K_n for n samples requires O(n^2) space, which rapidly becomes unfeasible for large n. Nystrom approximations reduce the space complexity to O(nm) by sampling m columns from K_n. Uniform sampling preserves KRR accuracy (up to epsilon) only when m is proportional to the maximum degree of freedom of K_n, which may require O(n) columns for datasets with high coherence. Sampling columns according to their ridge leverage scores (RLS) gives accurate Nystrom approximations with m proportional to the effective dimension, but computing exact RLS also requires O(n^2) space. (Calandriello et al. 2016) propose INK-Estimate, an algorithm that processes the dataset incrementally and updates RLS, effective dimension, and Nystrom approximations on-the-fly. Its space complexity scales with the effective dimension but introduces a dependency on the largest eigenvalue of K_n, which in the worst case is O(n). In this paper we introduce SQUEAK, a new algorithm that builds on INK-Estimate but uses unnormalized RLS. As a consequence, the algorithm is simpler, does not need to estimate the effective dimension for normalization, and achieves a space complexity that is only a constant factor worse than exact RLS sampling.

Daniele Calandriello, Alessandro Lazaric, Michal Valko

SequeL team, INRIA Lille - Nord Europe, France

Kernel Ridge Regression Kernel Regression Importance Sampling Low-Rank Approximation Ridge Regression

Apr 22, 2026cs.LG

Analysis of Nystrom method with sequential ridge leverage scores

Large-scale kernel ridge regression (KRR) is limited by the need to store a large kernel matrix K_t. To avoid storing the entire matrix K_t, Nystrom methods subsample a subset of columns of the kernel matrix, and efficiently find an approximate KRR solution on the reconstructed matrix. The chosen subsampling distribution in turn affects the statistical and computational tradeoffs. For KRR problems, recent works show that a sampling distribution proportional to the ridge leverage scores (RLSs) provides strong reconstruction guarantees for the approximation. While exact RLSs are as difficult to compute as a KRR solution, we may be able to approximate them well enough. In this paper, we study KRR problems in a sequential setting and introduce the INK-ESTIMATE algorithm, that incrementally computes the RLSs estimates. INK-ESTIMATE maintains a small sketch of K_t, that at each step is used to compute an intermediate estimate of the RLSs. First, our sketch update does not require access to previously seen columns, and therefore a single pass over the kernel matrix is sufficient. Second, the algorithm requires a fixed, small space budget to run dependent only on the effective dimension of the kernel matrix. Finally, our sketch provides strong approximation guarantees on the distance between the true kernel matrix and its approximation, and on the statistical risk of the approximate KRR solution at any time, because all our guarantees hold at any intermediate step.

Daniele Calandriello, Alessandro Lazaric, Michal Valko

SequeL team INRIA Lille - Nord Europe

Kernel Ridge Regression Kernel Regression Randomized Sketching Spectral Methods Ridge Regression Streaming Algorithms

Feb 6, 2026stat.ML

Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay

We study optimal learning rate (LR) schedules under the functional scaling law (FSL) framework (Li et al., 2025), which decomposes training dynamics into signal learning and noise forgetting. In power-law kernel regression, these two components are governed by a source exponent

s>0

and a capacity exponent

q>1

, respectively, with smaller

s

corresponding to harder tasks. For a fixed training horizon

N

, we characterize the schedules that minimize the final-step loss under a stability constraint and reveal a sharp phase transition. In the easy-task regime

s>1-1/q

, the optimal schedule follows power decay from the beginning of training; in the hard-task regime

s<1-1/q

, it becomes warmup-stable-decay (WSD)-like (Hu et al., 2024), staying at the largest admissible LR for most of training before a final decay. In both regimes, the decay exponent is

2q-1

: task difficulty determines when to decay, while model capacity determines how to decay. Beyond the exact optimum, we study fractional schedules, whose shape is defined over relative training progress. We show that precise tuning of the decay shape is often unnecessary: a broad class of profiles attains the optimal convergence rate, while overly slow terminal decay leads to schedule-induced capacity saturation. Finally, for one-pass SGD in kernel regression, FSL-motivated power-decay schedules achieve optimal last-iterate rates. Experiments support the theoretical predictions and the task-dependent transition between early and delayed decay.

Binghui Li, Zilin Wang, Fengling Chen +3

Center for Machine Learning Research, Peking University · School of Mathematical Sciences, Peking University · AI for Science Institute, Beijing

Learning Rate Scheduling Kernel Regression Stochastic Gradient Descent

Feb 2, 2026cs.LG

Universal Redundancies in Time Series Foundation Models

Time Series Foundation Models (TSFMs) leverage extensive pretraining to accurately predict unseen time series during inference, without the need for task-specific fine-tuning. Through large-scale evaluations on standard benchmarks, we find that leading transformer-based TSFMs exhibit redundant components in their intermediate layers. We introduce a set of tools for mechanistic interpretability of TSFMs, including ablations of specific components and direct logit attribution on the residual stream. Our findings are consistent across several leading TSFMs with diverse architectures, and across a diverse set of real-world and synthetic time-series datasets. We discover that all models in our study are robust to ablations of entire layers. Furthermore, we develop a theoretical framework framing transformers as kernel regressors, motivating a purely intrinsic strategy for ablating heads based on the stable rank of the per-head projection matrices. Using this approach, we uncover the specific heads responsible for degenerate phenomena widely observed in TSFMs, such as parroting of motifs from the context and seasonality bias. Our study sheds light on the universal properties of this emerging class of architectures for continuous-time sequence modeling.

Anthony Bao, Venkata Hasith Vattikuti, Jeffrey Lai +1

ECE Department, UT Austin · Department of Physics, UT Austin · Oden Institute, UT Austin

Transformer Interpretability Kernel Regression Mechanistic Interpretability Time-Series Foundation Models Neural Network Pruning

Jan 29, 2026cs.LG

Mechanistic Evidence for Spectral Structures in Prior-Data Fitted Networks

Prior-Data Fitted Networks (PFNs) perform approximate Bayesian inference in a single forward pass, and tabular foundation models (TFMs) built on them are now widely used. To understand what networks infer internally, recent mechanistic studies of TFMs locate where predictions form, but treat these models as tabular predictors rather than as PFNs. It therefore remains unknown whether PFNs represent the spectral content of their context, the quantity that specifies a stationary kernel, and whether this content can be read out as an explicit kernel. We answer both questions. First, across seven PFNs, including four pretrained TFMs and a model trained only on a decision-tree prior, a linear probe on the residual stream recovers the frequency of the context with

R^2 \geq 0.95

. This structure is led by a single principal direction. Second, activation and subspace patching show that the network uses the structure through a low-dimensional subspace, where a few spectral directions move predictions far more than random ones. This holds even for the decision-tree model, so a spectral training prior is not required. On real datasets with up to 499 features, 64 of the 192 directions of TabPFN, chosen without labels, carry 85 to 95% of the causal effect of the context in all but one pair. Third, we introduce a Filter Bank Decoder that turns frozen PFN representations into an explicit stationary kernel through Bochner's theorem. Without any test-time optimization, the decoded kernel supports Gaussian process regression competitive with deep kernel learning and random Fourier features at about

200\times

lower cost. PFN latents therefore hold spectral structure that is causally used and recoverable as a portable kernel.

Kaustubh Sharma, Srijan Tiwari, Ojasva Nema +1

Department of Electrical Engineering, Indian Institute of Technology Roorkee (IIT Roorkee), Uttarakhand, India · Department of Metallurgical and Materials Engineering, Indian Institute of Technology Roorkee (IIT Roorkee), Uttarakhand, India

Prior-Data Fitted Networks Neural Network Interpretability Frequency-Domain Feature Learning Kernel Methods Mechanistic Interpretability Kernel Regression

Oct 16, 2025cs.LG

Predicting kernel regression learning curves from only raw data statistics

We study kernel regression with common rotation-invariant kernels on real datasets including CIFAR-5m, SVHN, and ImageNet. We give a theoretical framework that predicts learning curves (test risk vs. sample size) from only two measurements: the empirical data covariance matrix and an empirical polynomial decomposition of the target function

f_*

. The key new idea is an analytical approximation of a kernel's eigenvalues and eigenfunctions with respect to an anisotropic data distribution. The eigenfunctions resemble Hermite polynomials of the data, so we call this approximation the Hermite eigenstructure ansatz (HEA). We prove the HEA for Gaussian data, but we find that real image data is often "Gaussian enough" for the HEA to hold well in practice, enabling us to predict learning curves by applying prior results relating kernel eigenstructure to test risk. Extending beyond kernel regression, we empirically find that MLPs in the feature-learning regime learn Hermite polynomials in the order predicted by the HEA. Our HEA framework is a proof of concept that an end-to-end theory of learning which maps dataset structure all the way to model performance is possible for nontrivial learning algorithms on real datasets.

Dhruva Karkada, Joseph Turnbull, Yuxi Liu +1

UC Berkeley · Imbue

Kernel Regression Kernel Methods Statistical Learning Theory

Oct 2, 2025stat.ML

Learning Multi-Index Models with Hyper-Kernel Ridge Regression

Deep neural networks excel in high-dimensional problems, outperforming models such as kernel methods, which suffer from the curse of dimensionality. However, the theoretical foundations of this success remain poorly understood. We follow the idea that the compositional structure of the learning task is the key factor determining when deep networks outperform other approaches. Taking a step towards formalizing this idea, we consider a simple compositional model, namely the multi-index model (MIM). In this context, we introduce and study hyper-kernel ridge regression (HKRR), an approach blending neural networks and kernel methods. Our main contribution is a sample complexity result demonstrating that HKRR can adaptively learn MIM, overcoming the curse of dimensionality. Further, we exploit the kernel nature of the estimator to develop ad hoc optimization approaches. Indeed, we contrast alternating minimization and alternating gradient methods both theoretically and numerically. These numerical results complement and reinforce our theoretical findings.

Shuo Huang, Hippolyte Labarrière, Ernesto De Vito +2

Istituto Italiano di Tecnologia, Genoa, Italy · MaLGa - DIBRIS - Universit`a di Genova, Genoa, Italy · MaLGa - DIMA - Universit`a di Genova, Genoa, Italy +2

Kernel Ridge Regression Kernel Regression Statistical Learning Theory Multi-Index Models

Sep 17, 2025cs.LG

A Compositional Kernel Model for Feature Learning

We study a compositional variant of kernel ridge regression in which the predictor is applied to a coordinate-wise reweighting of the inputs. Formulated as a variational problem, this model provides a tractable setting for studying feature learning in compositional architectures. From the perspective of variable selection, we show how relevant variables are recovered while noise variables are eliminated. We prove that both global minimizers and stationary points discard noise coordinates when the noise variables are Gaussian distributed. A central finding is that

\ell_1

-type kernels, such as the Laplace kernel, succeed in recovering features contributing to nonlinear effects at stationary points, whereas Gaussian kernels recover only linear ones.

Feng Ruan, Keli Liu, Michael Jordan

*Department of Statistics and Data Science, Northwestern University. · *Company E · *Department of Statistics and Computer Science, University of California, Berkeley.

Kernel Ridge Regression Kernel Regression Kernel Methods Feature Selection