Reproducing Kernel Hilbert Spaces

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Jul 6Week of Sep 21

Latest papers 29

Sep 8, 2026math.ST

MiNCE: Nonparametric, Strongly Consistent Confidence Envelopes for Band-Limited Functions and their Smoothed Spectra

Minimum-norm confidence envelope strategies offer a nonparametric approach to constructing nonasymptotic, simultaneous confidence regions for band-limited functions, exploiting the theory of Reproducing Kernel Hilbert Spaces (RKHS). While the finite-sample coverage guarantees of these envelopes have been established, their consistency has not been analyzed so far. In this paper, we study this construction, here termed the Minimum-Norm Confidence Envelope (MiNCE) framework, and establish the strong uniform consistency of the resulting bands, both for noise-free and noisy observation models, under mild assumptions on the measurement noises. We further extend this formulation to the frequency domain, deriving nonasymptotic, simultaneous, strongly uniformly consistent confidence bands for the smoothed spectra. Numerical experiments in nonparametric regression and spectral estimation empirically confirm our theoretical results, illustrating the contraction of the confidence envelopes toward the target function as the sample size increases.

Balázs Csanád Csáji, Bálint Horváth

HUN-REN SZTAKI: Institute for Computer Science and Control, Budapest, Hungary · Institute of Mathematics, Eötvös Loránd University (ELTE), Budapest, Hungary

Confidence Intervals Exponential Family Reproducing Kernel Hilbert Spaces Spectral Filtering

Sep 7, 2026cs.LG

Revisiting Thinning Methods for Kernel Learning Problems

Kernel methods are widely used because of their strong theoretical guarantees and empirical performance. However, their high computational cost limits their applicability to large-scale datasets. To address this shortcoming, several approaches use Maximum Mean Discrepancy to construct representative subsets that preserve the properties of the full dataset in a Reproducing Kernel Hilbert Space. We introduce Backward Kernel Herding, an algorithm that addresses this problem by iteratively removing points from the dataset, achieving results comparable to current state-of-the-art approaches while accelerating the subsampling process in realistic scenarios where the reduced size is less than half of the dataset. Moreover, we overcome a limitation of Kernel Thinning by proposing an extension that enables the construction of subsets of arbitrary size rather that restricting to successive halvings. Finally, we conduct an extensive experimental comparison focusing on the most relevant kernel learning procedures: Gaussian Processes and Kernel Support Vector Machines. The results show that Backward Kernel Herding consistently achieves competitive performance with the most favorable training-time efficiency, while the proposed Flexible Kernel Thinning frequently achieves the best predictive performance. These gains become especially pronounced for moderate compression ratios, highlighting the benefits of incorporating supervised information into the thinning process. In terms of memory consumption, Flexible Kernel Thinning is also competitive, whereas Backward Kernel Herding remains an alternative when computational efficiency is the primary objective. Overall, no single method dominates across all scenarios, underscoring the importance of selecting the reduction strategy according to the desired trade-off between predictive performance, training cost, and memory requirements.

Blanca Cano-Camarero, Yago R. Aguado-Carrillo-de-Albornoz, Ángela Fernández-Pascual +1

Departamento de Ingeniería Informática, Universidad Autónoma de Madrid, Madrid, Spain

Kernel Method Reproducing Kernel Hilbert Spaces Backward Passes

Jul 31, 2026stat.ML

Simple-regret rates and minimax optimality of fixed-prior expected improvement in Matérn and squared-exponential RKHSs

We study expected improvement (EI) for minimizing a deterministic function

f

in the RKHS

\mathcal H_k

of a continuous positive-semidefinite kernel

k

on a nonempty compact set

\mathcal X\subset\mathbb R^d

. Function values are observed exactly, and EI is computed from a fixed zero-mean Gaussian-process model with covariance

σ^2k

,

σ>0

. A weak-EI policy queries a point whose EI is at least a fixed positive fraction of its maximum. We introduce a notion of sequential separation radius relating ranked selected-point innovation norms to Kolmogorov widths, drawing on greedy approximation. Standard power-function estimates from scattered-data approximation and a finite-budget regret argument yield the rates. After

N

post-initial queries, every weak-EI policy has simple regret

O(N^{-ν/d})

for isotropic Matérn kernels of smoothness

ν>0

and

O(\exp[-c_1\min\{N,N^{1/d}\log(eN)\}])

for the isotropic squared-exponential kernel, with

c_1>0

. For

d=1

, the sharper bound

O(\exp[-c_2N\log(eN)])

holds for exact EI, with

c_2>0

. These bounds are uniform over each fixed RKHS ball. If

\mathcal X

has nonempty interior and

B>0

, the exact EI policy is minimax-rate optimal over the RKHS ball of radius

B

for Matérn kernels, even among randomized strategies whose final recommendation need not be a query point. For the squared-exponential kernel, it is minimax-rate optimal up to constants in the exponent among deterministic methods whose final recommendation may be any point of

\mathcal X

.

Emmanuel Vazquez, Sébastien Petit

Université Paris-Saclay, CNRS, CentraleSupélec, Laboratoire des Signaux et Systèmes, 91190 Gif-sur-Yvette, France · Laboratoire national de métrologie et d’essais (LNE), 1 rue Gaston Boissier, 75724 Paris Cedex 15, France

Reproducing Kernel Hilbert Spaces Minimax Rate Kernel Method Minimax Objective Functions Improvement

Jul 29, 2026stat.ML

PIKS: Universal Physics-Informed Kernel Methods

Physics-informed machine learning incorporates physical principles --often expressed via differential operators-- into data-driven models. While physics-informed neural networks (PINNs) dominate empirical applications, the complexity of neural network architectures and optimization landscapes hinders the development of a corresponding learning theory. In turn, kernel methods offer an appealing alternative with closed-form solutions and analytical tractability, yet existing guarantees primarily cover the well-specified setting where the target belongs to the native Reproducing Kernel Hilbert Space (RKHS). This imposes unrealistic regularity assumptions that physical targets often fail to satisfy. In this paper, we introduce and analyze Physics-Informed Kernel methodS (PIKS). We establish the universal consistency of PIKS for linear differential constraints, proving that for universal kernels (such as Gaussian or Matérn), the estimator asymptotically learns the target while satisfying physical constraints. We further derive finite-sample bounds under suitable source conditions. Our analysis is based on extending classical operator-theoretic analysis of kernel methods to physics-informed machine learning. Numerical experiments demonstrate that PIKS can be competitive with PINNs and traditional finite element methods.

Joachim Bona-Pellissier, Giacomo Meanti, Matteo Santacesaria +1

MaLGa Center, DIBRIS, Università degli Studi di Genova, Genoa, Italy · MaLGa Center, DIMA, Università degli Studi di Genova, Genoa, Italy · Istituto Italiano di Tecnologia, Genoa, Italy

Physics-Informed Learning Reproducing Kernel Hilbert Spaces Kernel Method Inequality Constraints

Jun 19, 2026stat.ML

Subsampling for supervised learning in reproducing kernel Hilbert spaces

In the era of big data, subsampling became a common practice in statistical learning. By selecting a subgroup of individuals based on which the learner is trained, subsampling aims at reducing the computational cost and time of the estimation step, and ideally leads to a decrease of its energy consumption and carbon footprint. This work focuses on a nonparametric setting, in which the hypotheses set lies in a reproducing kernel Hilbert space, and the estimator is a minimizer of an empirical risk reweighted à la Horvitz-Thompson. By studying the asymptotic properties of this estimator, we reveal an optimal subsampling scheme (regarding the trace of the covariance operator) and show that it can be used via plug-in. A numerical study on synthetic and real-world datasets shows the practicability and the benefit of the proposed approach.

Eyal Vayness, Maxime Sangnier

Sorbonne Universit´e, LPSM, Paris, France

Reproducing Kernel Hilbert Spaces Kernel Hilbert Spaces Subsampling Exponential Family Covariance Semi-Supervised Learning

Jun 15, 2026cs.LG

Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games with Average Reward

We study inverse reinforcement learning for discrete-time, infinite-horizon mean-field games (MFGs) under an average-reward criterion. Expert demonstrations are assumed to arise from a stationary mean-field equilibrium under an unknown reward, and the goal is to recover a policy explaining the observed behaviour via the maximum causal entropy principle. We formulate the inverse problem by enforcing consistency with the expert mean-field term and long-run feature expectations, treating two reward classes within a unified occupation-measure framework. For finite-dimensional linear rewards, we give a convex dual reformulation with an explicit log-partition objective, and prove smoothness and curvature properties justifying constant-step-size gradient descent. For infinite-dimensional RKHS rewards, we develop a Lagrangian relaxation whose inner-maximising policy is characterised by a soft Bellman equation. The main obstacle is the absence of a discount-factor contraction. We resolve this by introducing a minorisation-based sub-stochastic kernel that yields a strict contraction of the soft Bellman operator. We establish Fréchet differentiability and Lipschitz smoothness of the log-likelihood score, leading to a gradient ascent algorithm with convergence guarantees. Two numerical examples, a malware-spread MFG and an RKHS-based consumer-choice model, show that the recovered policies closely match expert behaviour.

Şevket Kaan Alkır, Naci Saldı, Berkay Anahtarcı +1

Department of Mathematics Bilkent University 06800 Ankara, Turkey · Department of Mathematical Engineering Özyeğin University 34794 Istanbul, Turkey

Mean Field Games Entropy Regularized Reinforcement Learning Offline Reinforcement Learning Bellman Equation Reproducing Kernel Hilbert Spaces Mean

Jun 7, 2026cs.SE

Structuring agentic AI for HPC code modernization

Modernization of legacy scientific codes is often necessary to keep up with the ever-evolving changes in the compute resource ecosystem. Parallelization and migration from poorly supported software ecosystems are two of the most time-consuming activities in the research software engineering field. This paper presents our experience in the successful, two-phase AI-assisted modernization of NMAP-RKPM, a roughly 60,000-line, 3D explicit solid mechanics physics engine based on the Reproducing Kernel Particle Method (RKPM). We converted this single-threaded, Fortran based MPI application into a OpenMP-parallel C++ based MPI tool in the span of a few months. While Large Language Model (LLM) based tools on their own proved inadequate, we developed a highly structured "hand-holding" agentic AI methodology, like providing manually created examples, ensuring continuous buildability and limiting session scope, that was instead highly effective. The paper provides both the AI-assisted steps that were successful and the problems that we had to overcome, alongside the reasoning behind the chosen path.

Anthony Marinov, Igor Sfiligoi

San Diego Supercomputer Center University of California, San Diego La Jolla, California, United States

High-Performance Computing Cloud Reproducing Kernel Hilbert Spaces

May 28, 2026cs.GR

FreeForm: Reduced-Order Deformable Simulation from Particle-Based Skinning Eigenmodes

We present a novel formulation for mesh-free, reduced-order simulation of deformable hyperelastic objects. Existing work in reduced-order elastodynamic simulation represents the input geometry by either meshes, which can be difficult to obtain due to challenges in scanning and triangulating complex shapes, or by neural fields that require per-shape optimization. We propose to adopt a Reproducing Kernel Particle Method (RKPM) representation, which enables the construction of reduced-order skinning weights by solving a generalized eigensystem on the Hessian matrix of the elastic energy. We demonstrate that this formulation not only leads to a 40x training speedup compared with the per-shape optimization of neural fields, but also achieves lower simulation error when evaluated against the converged results of finite element method. We show our simulation results on a wide variety of objects in different representations including meshes and Gaussian splats, as well as the application of our method in the downstream task of robot simulation.

Donglai Xiang, Vismay Modi, Rishit Dagli +5

NVIDIA · 1NVIDIA · University of Toronto

Finite Element Method Reduced-Order Models Deformable Linear Objects Reproducing Kernel Hilbert Spaces Skeleton

May 17, 2026stat.ME

Controlling False Discovery in Arbitrarily Structured Hypothesis Spaces via Reproducing Kernels

Large-scale hypothesis testing is central to modern science, where controlling the False Discovery Rate (FDR) has become the standard approach to managing false positives across many simultaneous tests. Hypotheses rarely exist in isolation; they often exhibit structure through proximity, connectivity, or hierarchy. This structure represents both a challenge and an opportunity: while classical methods treat these dependencies as obstacles requiring conservative correction, leveraging them can substantially increase discovery power. Here, we reframe structured FDR control as a regularized learning problem. By optimizing within a suitable Reproducing Kernel Hilbert Space (RKHS), we introduce a framework that unifies continuous domains, graphs, and hierarchies under a single algorithm through kernel choice alone. This formulation enables smooth solutions in place of the piecewise-constant fits of prior methods, principled likelihood-based hyperparameter selection rather than heuristic tuning, and inference at unobserved locations which in turn supports sample-efficient experimental design. Building on this estimator, we provide two decision rules which we prove to control the FDR. We validate our method on two sources: spatial locations derived from high-dimensional real-world datasets, and a differential gene expression task utilizing protein-protein interaction graphs.

Binyamin Perets, Shie Mannor

Technion – Israel Institute of Technology · NVIDIA

False Discovery Rate Reproducing Kernel Hilbert Spaces Hypothesis Class Two-Sample Testing Exponential Family Regularization+1

May 13, 2026stat.ME

Wahkon: A Statistically Principled Deep RKHS Superposition Network

Deep learning excels at prediction but often lacks finite-sample guarantees and calibrated uncertainty; RKHS (Reproducing Kernel Hilbert Space)-based methods provide those guarantees but struggle to adapt in high dimensions. We propose Wahkon, a deep RKHS superposition network that unifies Kolmogorov's superposition principle with RKHS regularization in the smoothing-spline tradition of Wahba. This yields a finite-dimensional deep representer theorem that makes training tractable and provides explicit layerwise complexity control. We show the penalized estimator is exactly the MAP (maximum a posteriori) estimate under a hierarchical Gaussian-process prior, extending the spline/GP duality to deep compositions. Using metric-entropy arguments, we establish minimax-optimal convergence rates under mild smoothness and clarify how depth and width trade off with regularity. Empirically, Wahkon outperforms multilayer perceptrons, Neural Tangent Kernels, and Kolmogorov--Arnold Networks across simulation benchmarks and a single-cell CITE-seq study. By unifying Kolmogorov's superposition principle with RKHS regularization, Wahkon delivers accuracy, interpretability, and statistical rigor in a single framework.

Yongkai Chen, Wenxuan Zhong, Ping Ma

Department of Statistics, Harvard University · Department of Statistics, University of Georgia

Reproducing Kernel Hilbert Spaces Kolmogorov-Arnold Networks Statistical Inference

May 12, 2026cs.LG

DriftXpress: Faster Drifting Models via Projected RKHS Fields

Drifting Models have emerged as a new paradigm for one-step generative modeling, achieving strong image quality without iterative inference. The premise is to replace the iterative denoising process in diffusion models with a single evaluation of a generator. However, this creates a different trade-off: drifting reduces inference cost by moving much of the computation into training. We introduce DriftXpress, an accelerated formulation of drifting models based on projected RKHS fields. DriftXpress approximates the drifting kernel in a low-rank feature space. This preserves the attraction-repulsion structure of the original drifting field while reducing the cost of field evaluation. Across image-generation benchmarks, DriftXpress achieves comparable FID to standard drifting while reducing wall-clock training cost. These results show that the training-inference trade-off of drifting models can be pushed further without giving up their one-step inference advantage.

Ali Falahati, Elliot Creager, Gautam Kamath +1

Cheriton School of Computer Science, University of Waterloo and Vector Institute. · Department of Electrical and Computer Engineering, University of Waterloo and Vector Institute. · A*STAR Centre for Frontier AI Research, Singapore.

Diffusion Models Generative Models Iterative Denoising Methods Reproducing Kernel Hilbert Spaces Image Quality Assessment

May 11, 2026cs.LG

Nearly-Optimal Algorithm for Adversarial Kernelized Bandits

This paper studies kernelized bandits (also known as Gaussian process bandits) in an adversarial environment, where the reward functions in a known reproducing kernel Hilbert space (RKHS) may be adversarially chosen at each round. We show that the exponential-weight algorithm achieves

\tilde{O}(\sqrt{T γ_T})

adversarial regret, where

T

and

γ_T

denote the number of total rounds and the maximum information gain, respectively. For squared exponential (SE) and

ν

-Matérn kernels, we also show algorithm-independent lower bounds that guarantee the optimality of our algorithm up to polylogarithmic factors. Furthermore, we present a computationally efficient variant of our algorithm using Nyström approximation while maintaining nearly optimal regret guarantees.

Shogo Iwazaki

LY Corporation Tokyo, Japan

Bandits Bounded Adversary Reproducing Kernel Hilbert Spaces Weight Decay

May 10, 2026cs.LG

Bayesian Optimization with Structured Measurements: A Vector-Valued RKHS Framework

Bayesian optimization (BO) is an efficient framework for optimizing expensive black-box functions. However, it is typically formulated as learning an end-to-end mapping from inputs to scalar objectives, thereby discarding the potentially rich information whenever a structured system output is available. In this work, we study Bayesian optimization over a vector-valued operator with structured measurements, where each measurement observes multidimensional or functional outputs, e.g., trajectories or spatial fields, rather than a single scalar value. The objective is then defined as a linear functional of these measurements. This allows each observation to reveal substantially richer information about the underlying system compared to scalar observations. Assuming the unknown operator lies in a vector-valued reproducing kernel Hilbert space (RKHS), we derive high-probability concentration bounds for the kernel ridge regression (KRR) estimator directly in the measurement space, characterizing uncertainty in a general Hilbert space. Building on these results, we propose an algorithm based on the upper confidence bound (UCB) acquisition function with regret guarantees under mild assumptions, recovering sublinear rates for common kernels. Empirically, we demonstrate that leveraging structured measurements leads to improved sample efficiency by enabling efficient transfer of information across objectives and adaptation to time-varying settings.

Wenbin Wang, Colin N. Jones

Automatic Control Laboratory, EPFL

Bayesian Optimization Reproducing Kernel Hilbert Spaces Kernel Hilbert Spaces Concentration Bounds

May 9, 2026cs.LG

The Pokémon Theorem and other Fairness Impossibility Results

Fairness impossibility results often look like distinct scalar incompatibility statements. We show that several share one RKHS geometry: fairness criteria are linear constraints on conditional mean embeddings, and unequal base rates make the law of total expectation overdetermine those constraints. This view yields four results. The Kleinberg--Mullainathan--Raghavan dichotomy needs only group-conditional unbiasedness, not full calibration. The \emph{Pokémon theorem} shows that a distinct group pair satisfying any finite collection of linear mean-fairness criteria leaves a residual violation witnessed by the MMD, decaying at the Kolmogorov

m

-width rate under spectral regularity. The same tools prove an impossibility for fair feature learning: parity and class-conditional separation in representation space force class collapse under unequal base rates. The approximate relaxations yield signal and error frontiers, allowing a trade-off between real-world estimators and fairness goals. Experiments on standard fairness benchmarks are consistent with our bounds.

Daniel Matsui Smola, Alex Smola

Department of Computer Science University of Washington Seattle, WA · Boson AI Santa Clara, CA

Algorithmic Fairness Impossibility Conditional Independence Compatibility Reproducing Kernel Hilbert Spaces

May 8, 2026eess.SY

A Behavioral Framework for Data-Driven Modeling of Nonlinear Systems in Vector-Valued Reproducing Kernel Hilbert Spaces

We generalize Jan Willems' behavioral approach to a class of discrete-time nonlinear systems in a vector-valued reproducing kernel Hilbert space (RKHS). Apart from linear time-invariant systems, this class covers nonlinear systems modeled by Volterra series and their autoregressive variants, as well as systems admitting Hammerstein-type state-space realizations. We apply the proposed framework to the problem of data-driven modeling of such systems, i.e., when simulation or control objectives for an unknown system are carried out without an explicit system identification step. To that end, we link the behavioral approach to two data-driven modeling methods in a vector-valued RKHS: (1) minimum-norm interpolation and (2) subspace identification.

Boya Hou, Maxim Raginsky

Carl R. Woese Institute for Genomic Biology and the Coordinated Science Laboratory, University of Illinois Urbana-Champaign, Urbana, IL, 61801 USA · Department of Electrical and Computer Engineering and the Coordinated Science Laboratory, University of Illinois Urbana-Champaign, Urbana, IL, 61801 USA

System Identification Reproducing Kernel Hilbert Spaces Nonlinear Dynamics Kernel Hilbert Spaces State Space Models Data-Driven

May 7, 2026math.NA

Convex-Geometric Error Bounds for Positive-Weight Kernel Quadrature

Kernel quadrature can exploit RKHS spectral structure and outperform Monte Carlo on smooth integrands, but optimized quadrature weights are generally signed and may be numerically unstable. We study whether spectral acceleration remains possible when the weights are constrained to be positive, i.e., simplex weights. In the exact-target fixed-pool setting, an evaluated i.i.d. candidate pool of size

N

is already available and the task is to reweight it so as to approximate the kernel mean embedding. We show that this positive reweighting problem is governed not by the equal-weight empirical average, but by the random convex hull generated by the pool. Our main geometric result shows that the mean of a bounded

d

-dimensional random vector can be approximated by a convex combination of

N

i.i.d. samples at accuracy

O(d/N)

with high probability, sharper than equal-weight averaging in the fixed-dimensional regime. We transfer this

d

-dimensional convex-hull approximation to full RKHS worst-case error through an augmented Mercer-truncation argument. The resulting positive-weight KQ bounds consist of a spectral tail term and a finite-sample convex-hull term, yielding Monte-Carlo-beating rates in favorable spectral regimes, including near-

O(1/N)

rates up to logarithmic factors under exponential spectral decay. We also provide a constructive Frank--Wolfe algorithm that operates directly on the pool atoms, maintains simplex weights, and admits an explicit optimization-error bound.

Satoshi Hayakawa

The University of Tokyo

Reproducing Kernel Hilbert Spaces Bayesian Quadrature Concentration Bounds Convex Sets Spectral Norm Upper Bounds

May 5, 2026cs.CV

A Robust Unsupervised Domain Adaptation Framework for Medical Image Classification Using RKHS-MMD

Labeling medical images is a major bottleneck in the field of medical imaging, as it requires domain-specific expertise, and it gets further complicated due to variability across different medical centers and different imaging devices. Such heterogeneity introduces domain shifts and modality discrepancies, which limits the generalization of trained models. To address this important challenge, we propose an unsupervised domain adaptation framework that combines transfer learning with a Reproducing Kernel Hilbert Space based Maximum Mean Discrepancy loss for the alignment of source and target domains. By jointly optimizing classification and RKHS-MMD losses, the methodology enhances generalization to unannotated medical datasets while diminishing reliance on manual annotation. Experimental evaluations presented on two chest X-ray datasets, which are obtained from different medical centers, show outstanding improvements over models trained without adaptation. Furthermore, we perform a comparative study to see that RKHS-MMD performs better than the standard Maximum Mean Discrepancy in reducing modality gap, emphasizing its effectiveness for medical image classification and also its strong capability in advanced AI-driven medical diagnostics.

Sapna Sachan, Rakesh Kumar Sanodiya, Amulya Kumar Mahto

Mehta Family School of Data Science and Artificial Intelligence, Indian Institute of Technology Guwahati, India · Indian Institute of Information Technology, Design and Manufacturing,2026 Jabalpur, India

Medical Image Classification Domain Adaptation Maximum Mean Discrepancy Modality Gap Reproducing Kernel Hilbert Spaces

May 5, 2026cs.LG

A Universal Reproducing Kernel Hilbert Space from Polynomial Alignment and IMQ Distance

We introduce the Yat kernel

k_{b,\varepsilon}(\mathbf{w},\mathbf{x})=\frac{(\mathbf{w}^\top\mathbf{x}+b)^2}{\|\mathbf{x}-\mathbf{w}\|^2+\varepsilon},\qquad b\ge 0,\ \varepsilon>0,

a rational hidden-unit primitive whose units are Mercer sections over a shared input/weight space. For

b\ge 0

the kernel is PSD; for

b>0

it dominates a scaled inverse-multiquadric (IMQ) in the Loewner order, yielding fixed-kernel universality, characteristicness, and strict positive definiteness on every compact domain. The polynomial numerator opens nonradial alignment channels absent from finite IMQ expansions, witnessed by the directional far-field trace

T_\infty g_\varepsilon(\cdot;\mathbf{w},b)(\mathbf{u})=(\mathbf{u}^\top\mathbf{w})^2

. Algebraically, a second finite difference in the bias recovers any IMQ atom from three positive-bias Yat atoms exactly, sharp at three atoms in every dimension at exact pointwise equality. A trained shared-

(b,\varepsilon)

Yat layer is therefore a finite learned-center expansion in a fixed universal characteristic RKHS, with closed-form norm

\boldsymbolα^\top\mathbf{K}\boldsymbolα

and explicit diagonal

(\|\mathbf{x}\|^2+b)^2/\varepsilon

driving a Rademacher generalization bound.

Taha Bouhsine

Azetta AI

Reproducing Kernel Hilbert Spaces Fourier Generalization Bounds Distance Value Alignment

Apr 27, 2026cs.LG

Dynamic Regret for Online Regression in RKHS via Discounted VAW and Subspace Approximation

We study online regression with the square loss in a reproducing kernel Hilbert space under a dynamic regret criterion. The learner is compared with a time-varying comparator sequence, and the bounds depend on its path length in the RKHS norm. The proposed method transfers the finite-dimensional discounted Vovk--Azoury--Warmuth approach of Jacobsen & Cutkosky (2024) to the RKHS setting by means of finite-dimensional subspace approximations. For a fixed subspace, we run a VAW-based ensemble of discounted VAW forecasters over a geometric grid of discount factors. The additional approximation error is controlled by the uniform projection error of kernel sections. We then introduce a general orthogonal truncation method: starting from a feature expansion of the kernel, we construct the associated RKHS by introducing an inner product that makes the feature functions orthonormal, and then use the spans of the first basis functions as finite-dimensional approximation spaces. The resulting subspace reduction is applied to several approximation schemes. Explicit feature expansions yield fast-regime bounds for Gaussian and analytic dot-product kernels. Mercer truncations provide a spectral approximation method and lead to dynamic regret bounds in fast and slow regimes, depending on the eigenvalue decay. Finally, we study subspaces spanned by kernel sections and apply this construction to Matérn kernels.

Dmitry B. Rokhlin, Georgiy A. Karapetyants

Institute of Mathematics, Mechanics and Computer Sciences of the Southern Federal University

Reproducing Kernel Hilbert Spaces Kernel Method Subspace

Apr 27, 2026cs.LG

Generalising maximum mean discrepancy: kernelised functional Bregman divergences

Bregman divergences play a pivotal role in statistics, machine learning and computational information geometry. Particularly in the context of machine learning, they are central to clustering, exponential families, parameter estimation and optimisation, among other things. Despite this, the full toolkit of Hilbert spaces and in particular reproducing kernel Hilbert spaces have not been systematically developed and applied to functional Bregman divergences, where points are functions rather than finite-dimensional parameter vectors. While other types of functional Bregman divergences have been studied, these are typically in a Banach space rather than more directly aligned with kernel methods and Hilbert-space geometry commonly used in machine learning. We consider functional Bregman divergences on a Hilbert space, where the self-dual pairing and Riesz representer afford us particularly convenient calculus. Further specialising Bregman generators as a composition involving a kernel mean embedding makes such divergences easy to estimate. We discuss applications in clustering, universal estimation, robust estimation and generative modelling, and contrast our approach with other types of Bregman divergences.

Russell Tsuchida, Frank Nielsen

Department of Data Science and AI, Monash University, Australia. · Sony Computer Science Laboratories, Inc. Tokyo, Japan.

Bregman Divergences Reproducing Kernel Hilbert Spaces Maximum Mean Discrepancy Divergence

Apr 25, 2026stat.ML

Explicit integral representations and quantitative bounds for two-layer ReLU networks

An approach to construct explicit integral representations for two-layer ReLU networks is presented, which provides relatively simple representations for any multivariate polynomial. Quantitative bounds are provided for a particular, sharpened ReLU integral representation, which involves a harmonic extension and a projection. The bounds demonstrate that functions can be approximated with

L^{2}(\mathcal{D})

errors that do not depend explicitly on dimension or degree, but rather the coefficients of their monomial expansions and the distribution

\mathcal{D}

. We also present a connection to the RKHS of the exponential kernel

K(x,y)=\exp\left(\left\langle x,y\right\rangle \right)

, and a very simple integral representation involving additionally multiplication via a fixed function which has better quantitative bounds.

Anthony Lee

School of Mathematics, University of Bristol

Rectified Linear Unit Networks Rectified Linear Unit Reproducing Kernel Hilbert Spaces Neural Network Representations Polynomials Integral+2

Apr 24, 2026cs.LG

Complex Stochastic Gradient Descent and Directional Bias in Reproducing Kernel Hilbert Spaces

Stochastic Gradient Descent (SGD) is a known stochastic iterative method popular for large-scale convex optimization problems due to its simple implementation and scalability. Some objectives, such as those found in complex-valued neural networks, benefit from updates like in SGD and Gradient Descent (GD) with a newly defined ``gradient'' that allows for complex parameters. This complex variant of the SGD/GD methods has already been proposed, but convergence guarantees without analyticity constraints have not yet been provided. We propose a variant of SGD (complex SGD) that allows for complex parameters, and we provide convergence guarantees under assumptions that parallel those from the real setting. Notably, these results extend to GD as well, and with the same set of assumptions, we confirm that some directional bias results extend from the real to the complex setting for kernel regression problems. We provide empirical results demonstrating the efficacy of the complex SGD in kernel regression problems utilizing complex reproducing kernel Hilbert spaces. In particular, we demonstrate we may recover superoscillation functions and Blaschke products from the Fock Space and Hardy Space, respectively, as the optimal functions for a particular choice of a loss function.

Natanael Alpay, Emeric Battaglia

Reproducing Kernel Hilbert Spaces Stochastic Gradient Descent Kernel Ridge Regression Gradient

Apr 16, 2026stat.ML

Structural interpretability in SVMs with truncated orthogonal polynomial kernels

We study post-training interpretability for Support Vector Machines (SVMs) built from truncated orthogonal polynomial kernels. Since the associated reproducing kernel Hilbert space is finite-dimensional and admits an explicit tensor-product orthonormal basis, the fitted decision function can be expanded exactly in intrinsic RKHS coordinates. This leads to Orthogonal Representation Contribution Analysis (ORCA), a diagnostic framework based on normalized Orthogonal Kernel Contribution (OKC) indices. These indices quantify how the squared RKHS norm of the classifier is distributed across interaction orders, total polynomial degrees, marginal coordinate effects, and pairwise contributions. The methodology is fully post-training and requires neither surrogate models nor retraining. We illustrate its diagnostic value on a synthetic double-spiral problem and on a real five-dimensional echocardiogram dataset. The results show that the proposed indices reveal structural aspects of model complexity that are not captured by predictive accuracy alone.

Víctor Soto-Larrosa, Nuria Torrado, Edmundo J. Huertas

Dpto. de Biociencias, Facultad de Ciencias Biomédicas y de la Salud, Universidad Europea de Madrid, C/ Tajo, s/n, 28670 Villaviciosa de Odón, Madrid, Spain. · Dpto. de Matemáticas, Facultad de Ciencias, Universidad Autónoma de Madrid, C/ Francisco Tomás y Valiente, 7, 28049 Madrid, Spain. · Instituto de Ciencias Matemáticas (ICMAT), Campus de Cantoblanco UAM, C/ Nicolás Cabrera, 13–15, 28049 Madrid, Spain. +1

Support Vector Machines Kernel Method Reproducing Kernel Hilbert Spaces Interpretability Principal Component Analysis Echocardiography

Mar 17, 2026cs.LG

Optimal uncertainty bounds for multivariate kernel regression under bounded noise: A Gaussian process-based dual function

Non-conservative uncertainty bounds are essential for making reliable predictions about latent functions from noisy data, and thus, a key enabler for safe learning-based control. In this domain, kernel methods such as Gaussian process regression are established techniques, thanks to their inherent uncertainty quantification mechanism. Still, existing bounds either pose strong assumptions on the underlying noise distribution, are conservative, do not directly apply in the multi-output case, or are difficult to integrate into downstream tasks. This paper addresses these limitations by presenting a tight, deterministic bound for multi-output functions in Reproducing Kernel Hilbert Spaces (RKHSs) subject to bounded noise. It is obtained through an unconstrained, duality-based formulation, which shares the same structure as classic Gaussian process confidence bounds, and can thus be straightforwardly integrated into downstream optimization pipelines. We show that the proposed bound generalizes existing results and illustrate its application using an example inspired by quadrotor dynamics learning.

Amon Lahr, Anna Scampicchio, Johannes Köhler +1

Institute for Dynamical Systems and Control, ETH Zurich, Zurich, Switzerland

Gaussian Process Reproducing Kernel Hilbert Spaces Kernel Method Quadrotor

Sep 14, 2025stat.ML

A Kernel-based Stochastic Approximation Framework for Nonlinear Operator Learning

We develop a stochastic approximation framework for learning nonlinear operators between infinite-dimensional spaces utilizing general Mercer operator-valued kernels. Our framework encompasses two key classes: (i) operator-valued kernels whose associated integral operators are compact and hence admit discrete spectral decompositions, and (ii) separable kernels of the form

K(x,x')=k(x,x')T

, where

k

is a scalar-valued kernel and

T

is a positive operator on the output space. This broad setting induces expressive vector-valued reproducing kernel Hilbert spaces (RKHSs) that generalize the classical

K=kI

paradigm, thereby enabling rich structural modeling with rigorous theoretical guarantees. To address target operators lying outside the RKHS, we introduce vector-valued interpolation spaces to precisely quantify misspecification error. Within this framework, we establish non-asymptotic convergence rates for prediction, estimation, and misspecification errors in the online and finite-horizon settings. Importantly, the framework also accommodates a range of operator learning settings, from Fredholm integral operators to encoder--decoder architectures. Numerical experiments on the two-dimensional Navier--Stokes equations illustrate the proposed approach.

Jia-Qi Yang, Lei Shi

School of Mathematical Sciences and Shanghai Key Laboratory for Contemporary Applied Mathematics, Fudan University, Shanghai 200433, China.

Probabilistic Operator Learning Nonlinear Operators Reproducing Kernel Hilbert Spaces Kernel Hilbert Spaces Kernel Method Stochastic Approximation+1

Jun 20, 2025stat.ML

Gaussian Processes and Reproducing Kernel Hilbert Spaces: Connections and Equivalences

This monograph studies the relations between two approaches using positive definite kernels: probabilistic methods using Gaussian processes, and non-probabilistic methods using reproducing kernel Hilbert spaces (RKHS). They are widely studied and used in machine learning, statistics, and numerical analysis. We study connections and equivalences for fundamental topics such as regression, interpolation, numerical integration, distributional discrepancies, and statistical dependence, as well as sample path properties of Gaussian processes. A unifying perspective for these equivalences is established, based on the equivalence between the Gaussian Hilbert space and the RKHS. The monograph serves as a basis to bridge many other methods based on Gaussian processes and reproducing kernels, which are developed in parallel by the two research communities.

Motonobu Kanagawa, Philipp Hennig, Dino Sejdinovic +1

EURECOM · University of Tübingen, Tübingen AI Center Max Planck Institute for Intelligent Systems · Adelaide University Australian Institute for Machine Learning +1

Reproducing Kernel Hilbert Spaces Kernel Hilbert Spaces Gaussian Process Kernel Method Equivalence

Oct 14, 2024math.NA

Which Spaces can be Embedded in $L_p$ -type Reproducing Kernel Banach Space? A Characterization via Metric Entropy

In this paper, we establish a novel connection between the metric entropy growth and the embeddability of function spaces into reproducing kernel Hilbert/Banach spaces. Metric entropy characterizes the information complexity of function spaces and has implications for their approximability and learnability. Classical results show that embedding a function space into a reproducing kernel Hilbert space (RKHS) implies a bound on its metric entropy growth. Surprisingly, we prove a \textbf{converse}: a bound on the metric entropy growth of a function space allows its embedding to a

L_p-

type Reproducing Kernel Banach Space (RKBS). This shows that the

{L}_p-

type RKBS provides a broad modeling framework for learnable function classes with controlled metric entropies. Our results shed new light on the power and limitations of kernel methods for learning complex function spaces.

Yiping Lu, Daozhe Lin, Qiang Du

Industrial Engineering & Management Sciences, Northwestern University · Department of Applied Physics & Applied Mathematics, Columbia University

Reproducing Kernel Hilbert Spaces Learnability Approximation Algorithms

Sep 27, 2024cs.CV

DynaWeightPnP: Toward global real-time 3D-2D solver in PnP without correspondences

This paper addresses a special Perspective-n-Point (PnP) problem: estimating the optimal pose to align 3D and 2D shapes in real-time without correspondences, termed as correspondence-free PnP. While several studies have focused on 3D and 2D shape registration, achieving both real-time and accurate performance remains challenging. This study specifically targets the 3D-2D geometric shape registration tasks, applying the recently developed Reproducing Kernel Hilbert Space (RKHS) to address the "big-to-small" issue. An iterative reweighted least squares method is employed to solve the RKHS-based formulation efficiently. Moreover, our work identifies a unique and interesting observability issue in correspondence-free PnP: the numerical ambiguity between rotation and translation. To address this, we proposed DynaWeightPnP, introducing a dynamic weighting sub-problem and an alternative searching algorithm designed to enhance pose estimation and alignment accuracy. Experiments were conducted on a typical case, that is, a 3D-2D vascular centerline registration task within Endovascular Image-Guided Interventions (EIGIs). Results demonstrated that the proposed algorithm achieves registration processing rates of 60 Hz (without post-refinement) and 31 Hz (with post-refinement) on modern single-core CPUs, with competitive accuracy comparable to existing methods. These results underscore the suitability of DynaWeightPnP for future robot navigation tasks like EIGIs.

Jingwei Song, Maani Ghaffari

United Imaging Research Institute of Intelligent Imaging, Beijing 100144, China, Beijing, 100144, China · University of Michigan, Ann Arbor, 48109, MI, USA

Image Registration Reproducing Kernel Hilbert Spaces Dynawm Perspective Solvers

May 12, 2024eess.SY

Nonparametric Control Koopman Operators

This paper presents a novel Koopman composition operator representation framework for control systems in reproducing kernel Hilbert spaces (RKHSs) that is free of explicit dictionary or input parametrizations. By establishing fundamental equivalences between different model representations, we are able to close the gap of control system operator learning and infinite-dimensional regression, enabling various empirical estimators and the connection to the well-understood learning theory in RKHSs under one unified framework. Consequently, our proposed framework allows for arbitrarily accurate finite-rank approximations in infinite-dimensional spaces and leads to finite-dimensional predictors without a priori restrictions to a finite span of functions or inputs. To enable applications to high-dimensional control systems, we improve the scalability of our proposed control Koopman operator estimates by utilizing sketching techniques. Numerical experiments demonstrate superior prediction accuracy compared to bilinear EDMD, especially in high dimensions. Finally, we show that our learned models are readily interfaced with linear-parameter-varying techniques for model predictive control.

Petar Bevanda, Bas Driessen, Lucian Cristian Iacob +3

Chair of Information-oriented Control, TU München, Germany · Control Systems Group, TU/e, Eindhoven, The Netherlands · Systems and Control Laboratory, HUNREN Institute for Computer Science and Control, Budapest, Hungary +2

Koopman Operator Reproducing Kernel Hilbert Spaces Probabilistic Operator Learning Kernel Hilbert Spaces Learning-Based Control Robust Control+1