Overview

Kernel methods represent data through similarity functions, allowing linear algorithms in feature spaces to model nonlinear patterns without explicitly constructing those spaces. Common approaches include support vector machines and kernel ridge regression for prediction, Gaussian processes for uncertainty, and kernel mean embeddings and maximum mean discrepancy for comparing distributions and testing dependence. Researchers also use kernels to learn dynamical systems and operators, and scale computations with random features, Nyström approximations, and structured kernels. A central tension is balancing expressive, well-chosen kernels and reliable guarantees against the computation and sensitivity to kernel or bandwidth selection that can limit practical use.

Latest papers

253 papers25 papers in the last four weeks, down 22% on the four weeks before. 0.2% of all new papers.

May 10, 2025

Out-of-Sample Embedding with Proximity Data: Projection versus Restricted Reconstruction

The problem of using proximity (similarity or dissimilarity) data for the purpose of "adding a point to a vector diagram" was first studied by J.C. Gower in 1968. Since then, a number of methods -- mostly kernel methods -- have been proposed for solving what has come to be called the problem of out-of-sample embedding. We survey the various kernel methods that we have encountered and show that each can be derived from one or the other of two competing strategies: projection or restricted reconstruction. Projection can be analogized to a well-known formula for adding a point to a principal component analysis. Restricted reconstruction poses a different challenge: how to best approximate redoing the entire multivariate analysis while holding fixed the vector diagram that was previously obtained. This strategy results in a nonlinear optimization problem that can be simplified to a unidimensional search. Various circumstances may warrant either projection or restricted reconstruction.
Mar 21, 2025

Benign Overfitting with Quantum Kernels

Kernel methods compare inputs through feature maps. Quantum kernels follow the same principle: input data are encoded into quantum states, which define quantum feature representations in Hilbert spaces. Kernel values are then obtained by estimating inner products between these states using suitable quantum circuit measurements. As a result, quantum kernels may be intractable to compute classically while remaining efficiently computable on quantum hardware, potentially leading to a quantum advantage. However, designing effective quantum kernels remains a major challenge. Many quantum kernels, such as the fidelity kernel, suffer from exponential concentration. This results in near-identity kernel matrices that fail to capture meaningful data correlations and lead to overfitting and poor generalization. In this paper, we propose a novel strategy for constructing quantum kernels that achieve good generalization performance, drawing inspiration from benign overfitting in classical machine learning. We introduce the concept of Local-Global quantum kernels, which combine two components: a local quantum kernel based on measurements of small subsystems, and a global quantum kernel derived from full-system measurements. To support the effectiveness of the proposed construction, we show theoretically and empirically that Local-Global quantum kernels exhibit benign overfitting.
Mar 2, 2025

Data-Efficient Kernel Methods for Learning Differential Equations and Their Solution Operators: Algorithms and Error Analysis

We introduce a novel kernel-based framework for learning differential equations and their solution maps. The framework is efficient in its data requirements in terms of the number of observed states and the amount of data available per state. Our formulation also allows us to employ efficient Gauss-Newton-type optimization algorithms in the training procedure to improve accuracy and speed of the training procedure. We also support our algorithm with rigorous theoretical guarantees in the form of quantitative worst-case error bounds for the learned equation that can be extended to the subsequent solution operators under smoothness assumptions. Numerical benchmarks demonstrate significant improvements in computational complexity and robustness while achieving one to two orders of magnitude improvements in terms of accuracy compared to certain state-of-the-art algorithms.
Feb 24, 2025

A Mechanistic Interpretation of Prescribed-Parameter Convolutional Kernels

Prescribed-parameter convolutional transforms achieve strong time series classification performance using fixed kernels, simple pooling operations, and a linear classifier. Their success raises a mechanistic question: which properties of the representation make this construction effective? We study the proportion of positive values (PPV), a pooling operator shared by ROCKET, MiniROCKET, and MultiROCKET. We establish exact invariance to circular translations under periodic boundaries and quantify departures from invariance under finite boundaries. A complementary analysis bounds the sensitivity of a smooth PPV surrogate. We then define the population pairwise geometry induced by a distribution of convolutional parameters and show how a finite random kernel bank concentrates around that geometry. This separates the role of the parameter distribution from the number of sampled kernels. Experiments on the UCR classification collection examine effective dimensionality, distance correlation with the input, and classification accuracy using PPV-only, unit-energy ROCKET kernels. Increasing kernel count produces diminishing representational gains. Increasing kernel length generally raises effective dimensionality and distance correlation, while the best test performance occurs at intermediate lengths and training accuracy remains close to perfect. Together, these results support an interpretation in which nonlinear pooling creates a richer but structured representation that can favour linear separation, while further representational richness need not improve generalisation.
Feb 23, 2025

Optimal Kernel Learning for Gaussian Process Models with High-Dimensional Input

Gaussian process (GP) regression is a popular surrogate modeling tool for computer simulations in engineering and scientific domains. However, it often struggles with high computational costs and low prediction accuracy when the simulation involves too many input variables. For some simulation models, the outputs may only be significantly influenced by a small subset of the input variables, referred to as the active variables''. We propose an optimal kernel learning approach to identify these active variables, thereby overcoming GP model limitations and enhancing system understanding. Our method approximates the original GP model's covariance function through a convex combination of kernel functions, each utilizing low-dimensional subsets of input variables. Inspired by the Fedorov-Wynn algorithm from optimal design literature, we develop an optimal kernel learning algorithm to determine this approximation. We incorporate the effect heredity principle, a concept borrowed from the field of design and analysis of experiments'', to facilitate efficient computation. Through several examples, we demonstrate that the proposed method provides competitive prediction and variable selection, with favorable computational performance in the examples considered. It is an effective solution for interpreting the surrogate GP regression and simplifying the complex underlying system.
Feb 16, 2025

Physics-Informed Support Vector Kernels via Green-Function Analogies and Jackson-Chebyshev Spectral Design

Kernel selection for regression of physical observables is often heuristic. We investigate a physics-informed strategy in which functional forms and spectral structures associated with Green's functions motivate kernel selection without requiring an exact identification between a machine-learning kernel and a physical propagator. The principal construction is a Jackson-damped Chebyshev kernel inspired by the kernel polynomial method (KPM); its explicit feature map yields a positive-semidefinite Gram matrix by construction and provides an inspectable spectral prior for structured observables. We evaluate standard and custom SVR models on copper-conductivity proxies, local Dirac-like band dispersion, quartic-oscillator energy levels, photonic-crystal transmission, and Fibonacci-chain transmission using repeated nested validation, learning curves, random-forest and multilayer-perceptron baselines, and low-rank Nyström tests where relevant. The framework is intended for finite-data regression of precomputed observables while boundary conditions remain part of the physical model that generates those observables.
Jan 18, 2025

Fixed-Gaussian Spectral Algorithms: Minimax Optimal Rates for Misspecified Learning and Transfer

The principal objective of this work is twofold within nonparametric regression settings: (1) to establish the minimax optimal convergence rates for fixed-bandwidth Gaussian kernel spectral algorithms when the true regression function resides in a Sobolev space, and (2) to apply Gaussian spectral algorithms for achieving robust and adaptive transfer learning under concept shift. While minimax optimality of misspecified spectral algorithms has been established, existing guarantees are typically restricted to the non-saturation regime. We demonstrate that the infinite smoothness of fixed-bandwidth Gaussian kernels provides universal robustness to model misspecification by showing that this kernel choice enables any spectral algorithm to attain minimax optimal rates, provided the regularization parameter decays exponentially. This result effectively decouples optimality from the algorithm's inherent qualification. Building on this, we then advocate Gaussian spectral algorithms as powerful components in a learning framework for robust and adaptive transfer. Specifically, we derive the adaptive convergence rate of the excess risk for this framework and show that the rates are optimal up to logarithmic factors. Our results also reveal the impact of the magnitude of the concept shift and the sample size on the generalization error.
Oct 29, 2024

Conditional Moment Kernels: Joint Modeling of Expected Returns and Covariances in Unbalanced Panels

We propose the conditional moment kernel (CMK), a joint model of the conditional mean and covariance matrix of asset returns given their characteristics. A single kernel function of the characteristics gives the means and second moments of all returns. The implied covariance matrices are positive semidefinite for any number and composition of assets, so the model applies directly to unbalanced panels. Its estimation is a convex program whose size does not depend on the number of assets. A floor at the scale of the idiosyncratic noise keeps the estimated systematic covariance away from zero where the data cannot resolve it, and a trace penalty selects the number of factors. A closed form, the CMK estimator, approximates the solution. For fixed features and hyper-parameters, we prove identification, consistency and finite-sample bounds. In out-of-sample tests on US stocks from 1972 to 2021, the CMK estimator fits the conditional moments significantly better than instrumented principal component analysis (IPCA) with the same characteristics. Almost all of the gain is in the covariance matrix, and nonlinear functions of the characteristics add to it. The implied unconditionally mean--variance efficient portfolios earn annualized Sharpe ratios of 4.0 to 5.1 before costs, close to those of IPCA and declining over time.
Oct 14, 2024

Which Spaces can be Embedded in LpL_p-type Reproducing Kernel Banach Space? A Characterization via Metric Entropy

In this paper, we establish a novel connection between the metric entropy growth and the embeddability of function spaces into reproducing kernel Hilbert/Banach spaces. Metric entropy characterizes the information complexity of function spaces and has implications for their approximability and learnability. Classical results show that embedding a function space into a reproducing kernel Hilbert space (RKHS) implies a bound on its metric entropy growth. Surprisingly, we prove a converse: a bound on the metric entropy growth of a function space allows its embedding to a Lp−L_p-type Reproducing Kernel Banach Space (RKBS). This shows that the Lp−{L}_p-type RKBS provides a broad modeling framework for learnable function classes with controlled metric entropies. Our results shed new light on the power and limitations of kernel methods for learning complex function spaces.
Sep 13, 2024

Improved Finite-Particle Convergence Rates for Stein Variational Gradient Descent

We provide finite-particle convergence rates for the Stein Variational Gradient Descent (SVGD) algorithm in the Kernelized Stein Discrepancy (KSD\mathsf{KSD}) and Wasserstein-2 metrics. Our key insight is that the time derivative of the relative entropy between the joint density of NN particle locations and the NN-fold product target measure, starting from a regular initial distribution, splits into a dominant negative part' proportional to $N$ times the expected $\mathsf{KSD}^2$ and a smaller positive part'. This observation leads to KSD\mathsf{KSD} rates of order 1/N1/\sqrt{N}, in both continuous and discrete time, providing a near optimal (in the sense of matching the corresponding i.i.d. rates) double exponential improvement over the recent result by Shi and Mackey (2024). Under mild assumptions on the kernel and potential, these bounds also grow polynomially in the dimension dd. By adding a bilinear component to the kernel, the above approach is used to further obtain Wasserstein-2 convergence in continuous time. For the case of `bilinear + Matérn' kernels, we derive Wasserstein-2 rates that exhibit a curse-of-dimensionality similar to the i.i.d. setting. We also obtain marginal convergence and long-time propagation of chaos results for the time-averaged particle laws.
May 12, 2024

Nonparametric Control Koopman Operators

This paper presents a novel Koopman composition operator representation framework for control systems in reproducing kernel Hilbert spaces (RKHSs) that is free of explicit dictionary or input parametrizations. By establishing fundamental equivalences between different model representations, we are able to close the gap of control system operator learning and infinite-dimensional regression, enabling various empirical estimators and the connection to the well-understood learning theory in RKHSs under one unified framework. Consequently, our proposed framework allows for arbitrarily accurate finite-rank approximations in infinite-dimensional spaces and leads to finite-dimensional predictors without a priori restrictions to a finite span of functions or inputs. To enable applications to high-dimensional control systems, we improve the scalability of our proposed control Koopman operator estimates by utilizing sketching techniques. Numerical experiments demonstrate superior prediction accuracy compared to bilinear EDMD, especially in high dimensions. Finally, we show that our learned models are readily interfaced with linear-parameter-varying techniques for model predictive control.
Oct 18, 2023

Consistent Distributed Ranking of Generative Models via Kernel Distances

Ranking generative models based on the fidelity and diversity of their outputs is required to identify the best generator in a group of candidate generative AI models. To rank a group of models in a conventional centralized setting, a standard score is commonly evaluated for each involved model. The selection and design of reference-based evaluation scores have been extensively studied in centralized settings, where the reference samples are drawn from a single probability distribution. However, in practical scenarios including distributed learning contexts, reference samples are distributed across multiple clients, each potentially with a heterogeneous data distribution. In this work, we investigate the ranking of generative models in such distributed settings with heterogeneous data distributions across clients. We focus on the widely used family of kernel distance (KD) evaluation metrics. We prove that, for every kernel function, ranking models by the averaged KD scores of individual clients yields the same ordering as a centralized KD evaluation using the combined reference data from all the clients. We further extend our analysis to other popular metrics, including the Fréchet Distance (FD), for which the individual client scores could be insufficient for accurate model ranking. We present the numerical results of several experiments on standard image datasets and generative models to validate our theoretical findings regarding distributed ranking using various evaluation scores.
Dec 23, 2018

Distribution-Free Uncertainty Quantification for Kernel Methods by Gradient Perturbations

We propose a data-driven approach to quantify the uncertainty of models constructed by kernel methods. Our approach minimizes the needed distributional assumptions, hence, instead of working with, for example, Gaussian processes or exponential families, it only requires knowledge about some mild regularity of the measurement noise, such as it is being symmetric or exchangeable. We show, by building on recent results from finite-sample system identification, that by perturbing the residuals in the gradient of the objective function, information can be extracted about the amount of uncertainty our model has. Particularly, we provide an algorithm to build exact, non-asymptotically guaranteed, distribution-free confidence regions for ideal, noise-free representations of the function we try to estimate. For the typical convex quadratic problems and symmetric noises, the regions are star convex centered around a given nominal estimate, and have efficient ellipsoidal outer approximations. Finally, we illustrate the ideas on typical kernel methods, such as LS-SVC, KRR, ε\varepsilon-SVR and kernelized LASSO.