Training-point consistency of kernel ridge regression with varying correlation scales
Authors: Weiyan Mu, Shifeng Xiong, Yuming Shen
Organizations: School of Science, Beijing University of Civil Engineering and Architecture, Beijing, China · SKLMS, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing, China · School of Information Science and Technology, Guangdong University of Foreign Studies, Guangzhou, China
The scale parameter in kernel ridge regression plays a pivotal role in modeling the correlation structure of data. In modern applications, sampling density and the underlying correlation range often change simultaneously, making a fixed-scale assumption highly restrictive. However, establishing asymptotic theories under varying scale parameters is fundamentally challenging. To address this gap, this paper develops a unified theoretical framework for kernel ridge regression under varying scale parameters. We establish that, on uniform tensor grids with product exponential kernels, the consistency of the training-point risk holds if and only if the effective local sample size diverges to infinity. This sharp criterion universally covers all regimes along arbitrary deterministic scale sequences, seamlessly unifying the long-range, fixed-scale, and short-range. Furthermore, we provide an explicit, signal-independent construction of a deterministic regularization penalty that guarantees this consistency. Numerical experiments illustrate the criterion and the corresponding penalty across all the three regimes.
Figures & tables
Figure 1: Training-point risk under four correlation regimes in one and two dimensions.
Recent studies have reported saturation effects and multiple descent behavior in large dimensional kernel ridge regression (KRR). However, these findings are predominantly derived under restrictive settings, such as inner product kernels on sphere or strong eigenfunction assumptions like hypercontractivity. Whether such behaviors hold for other kernels remains an open question. In this paper, we establish a broad, new family of large dimensional kernels and derive the corresponding convergence rates of the generalization error. As a result, we recover key phenomena previously associated with inner product kernels on sphere, including: i) the minimax optimality when the source condition s≤1; ii) the saturation effect when s>1; iii) a periodic plateau phenomenon in the convergence rate and a multiple-descent behavior with respect to the sample size n.
Yang Zhou, Yicheng Li, Yuqian Cheng +1
Department of Statistics and Data Science Tsinghua University Beijing, 100084, China · Department of Mathematical Science Tsinghua University Beijing, 100084, China
Random feature methods provide a scalable approximation to kernel ridge regression (KRR), but the regularization parameter that yields the oracle learning rate depends on unknown smoothness and capacity parameters. In this work, we propose a neighboring early-stopping rule for adaptive regularization in KRR with random features (KRR-RF). The method uses a grid that is uniform in inverse regularization and compares only adjacent estimators, reducing the number of discrepancy comparisons relative to standard all-pairs Lepskii-type procedures. Both the neighboring discrepancy and its empirical complexity term can be computed directly in the random feature space, without constructing the exact kernel Gram matrix. We establish a high-probability comparison bound for neighboring KRR-RF estimators and show that, under standard source and capacity conditions together with suitable grid and random feature budget conditions, the selected estimator attains the oracle polynomial learning rate up to logarithmic factors. The result allows the regularization parameter to be selected without prior knowledge of the source and capacity exponents and covers both well-specified and partially misspecified regimes. Our analysis is based on an empirical random feature effective dimension that connects the observable stopping threshold with the population complexity of the random feature model. Simulation and real-data experiments illustrate the prediction performance and computational behavior of the proposed method in comparison with standard tuning procedures.
Caixing Wang, Zhibo Chen, Yue Wang
School of Statistics and Data Science, Southeast University · Faculty of Business for Science and Technology, School of Management, University of Science and Technology of China
Classical Gaussian processes and Kriging models are commonly based on stationary kernels, whereby correlations between observations depend exclusively on the relative distance between scattered data. While this assumption ensures analytical tractability, it limits the ability of Gaussian processes to represent heterogeneous correlation structures. In this work, we investigate variably scaled kernels as an effective tool for constructing non-stationary Gaussian processes by explicitly modifying the correlation structure of the data. Through a scaling function, variably scaled kernels alter the correlations between data and enable the modeling of targets exhibiting abrupt changes or discontinuities. We analyse the resulting predictive uncertainty via the variably scaled kernel power function and clarify the relationship between variably scaled kernels-based constructions and classical non-stationary kernels. Numerical experiments demonstrate that variably scaled kernels-based Gaussian processes yield improved reconstruction accuracy and provide uncertainty estimates that reflect the underlying structure of the data
Gianluca Audone, Francesco Marchetti, Emma Perracchione +1
Faculty of Engineering and Physical Sciences, University of Southampton, United Kindom · Dipartimento di Matematica “Tullio Levi-Civita”, Universit`a di Padova, Italy · Dipartimento di Scienze Matematiche Giuseppe Luigi Lagrange, Politecnico di Torino, Italy +1