Kernel Methods

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 125% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 168

May 4, 2026stat.ML

Measuring Differences between Conditional Distributions using Kernel Embeddings

Comparing conditional distributions is a fundamental challenge in statistics and machine learning, with applications across a wide range of domains. While proposed methods for measuring discrepancies using kernel embeddings of distributions in a reproducing kernel Hilbert space (RKHS) provide powerful non-parametric techniques, the existing literature remains fragmented and lacks a unified theoretical treatment. This paper addresses this gap by establishing a coherent framework for studying kernel-based methods to measure divergence between conditional distributions through what we refer to as conditional maximum mean discrepancy (CMMD). The CMMD consists of a family of metrics which we call levels, with three special cases each using a different type of RKHS embedding: CMMD

_0

(conditional mean operators), CMMD

_1

(conditional mean embeddings), and CMMD

_2

(joint mean embeddings). We additionally introduce a general level

s

CMMD, clarifying the required assumptions, and establishing mathematical connections between the levels through the lens of operator-based smoothing. In addition to reviewing previously proposed estimators, we introduce a novel doubly robust estimator for the CMMD that maintains consistency provided at least one of the underlying models is correctly specified. We provide numerical experiments demonstrating that the CMMD effectively captures complex conditional dependencies for statistical testing.

Peter Moskvichev, Siu Lun Chau, Dino Sejdinovic

School of Mathematical Sciences, Adelaide University · College of Computing and Data Science, Nanyang Technological University

Kernel Mean Embeddings Maximum Mean Discrepancy Kernel Methods

May 3, 2026stat.ML

A Semi-Supervised Kernel Two-Sample Test

We consider the problem of two-sample testing in a semi-supervised setting with abundant unlabeled covariate data. Standard two-sample tests neglect covariate information, which has the potential to significantly boost performance. However, incorporating covariates potentially breaks the exchangeability assumption under the null, which further complicates a calibration procedure. To address these issues, we propose a semi-supervised method that produces a test statistic with asymptotic normality, while effectively integrating additional information from covariates. Our test is straightforward to calibrate due to the asymptotic normality under the null and achieves asymptotic power that is often much higher than existing kernel tests without covariates. Furthermore, we formally show that the proposed method is consistent in power against fixed and local alternatives. Simulations confirm the practical and theoretical strengths of our approach.

Gyumin Lee, Shubhanshu Shekhar, Ilmun Kim

Dept. of Statistics Pennsylvania State University · Dept. of EECS University of Michigan · Dept. of Mathematical Sciences KAIST

Two-Sample Testing Kernel Methods Semi-Supervised Learning Statistical Hypothesis Testing

May 3, 2026cs.LG

Complex Diffusion Maps with $ω$ -Parameterized Kernels Revealing Inherent Harmonic Representations

In this paper, we propose Complex Diffusion Maps (CDM), a novel diffusion mapping framework that aims to reveal the dominant complex harmonics of high-dimensional data. Inspired by the local Gaussian kernel relevant to the heat equation and the nonlocal Schrödinger kernel relevant to the Schrödinger equation, we propose a unified family of

ω

-parameterized complex-valued kernels for the trade-off between local and nonlocal connections. We establish the theoretical foundation based on the operator spectrum theory, where the corresponding diffusion operator, diffusion distance, and complex harmonic maps are well-defined. An optimization-based interpretation of the maps is also developed, aiming to preserve angular structure in the complex diffusion space rather than relying solely on real-valued magnitude. We extensively evaluate CDM on both synthetic and real-world datasets. The complex-valued kernel amplifies differences among easily confusable samples, improving discriminative power over both linear and nonlinear methods based on real-valued kernels. CDM remains robust in high-noise settings, yielding a clearer eigengap that enhances spectral separation. For resting-state fMRI data, CDM captures more strongly correlated and nonlocal spatiotemporal dynamics. Without task-specific tuning, CDM achieves competitive performance on a public EEG sleep dataset, while maintaining high computational efficiency compared with both traditional machine learning and deep neural network approaches, highlighting its generality and practical value.

Tongzhen Dang, Weiyang Ding, Michael K. Ng

Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China. · Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China; Key Laboratory of Computational Neuroscience and Brain-Inspired Intelligence (Fudan University), Ministry of Education, China. · Department of Mathematics, Hong Kong Baptist University, Hong Kong, China.

Manifold Learning Kernel Methods Spectral Methods

May 2, 2026cs.DS

New Bounds for Kernel Sums via Fast Spherical Embeddings

We study query time bounds for the fundamental problem of estimating the kernel mean

\frac1{|X|}\sum_{x\in X}\mathbf{k}(x,y)

of a query

y

in a finite dataset

X\subset\mathbb{R}^d

up to a prescribed additive error

\varepsilon

. The best known bounds for the Gaussian kernel are

O(d/\varepsilon^2)

,

\widetilde O(d+1/\varepsilon^4)

, and

\widetilde O(d+Δ^2/\varepsilon^2)

, where

Δ

is the diameter of a region containing the points. We prove the new bound

\tilde O(d+\varepsilonΔ^2+1/\varepsilon^3)

, which improves over the previous ones in regimes with small error

\varepsilon

and intermediate diameter

Δ

. At the center of our proof is a new fast spherical embedding theorem in the sense introduced by Bartal, Recht and Schulman (2011), which limits the embedded data diameter while preserving local Euclidean distances and avoiding ``distance collapse'' at larger scales. This fast embedding theorem may be of independent interest.

Tal Wagner

Tel Aviv University

May 1, 2026cs.LG

Kernel Affine Hull Machines as Compute-Efficient Encoders for Frozen Semantic Spaces

Transformer-based semantic encoders are effective for retrieval, but in many deployments the recurring bottleneck is online query encoding rather than offline corpus indexing. This paper studies whether, once a strong teacher representation space and corpus index are fixed, repeated neural query encoding can be replaced by a substantially lighter and analytically explicit estimator. We formulate fixed-teacher lexical-to-semantic encoding as a conditional-mean estimation problem in which the target semantic vector is represented as a noisy mixture of semantic prototypes weighted by posterior cluster probabilities. Kernel Affine Hull Machine (KAHM) geometry is used to estimate these posterior weights from inexpensive lexical features in an explicitly identified RKHS hypothesis space, and the semantic prototypes are refined by normalized least-mean-squares updates from noisy teacher embeddings. This yields a backpropagation-free query-side encoder together with an end-to-end error decomposition into posterior-approximation, finite-sample/generalization, and teacher-noise terms. We instantiate the approach on a controlled Austrian-law retrieval benchmark with 5,000 test queries, 84 candidate laws, and 10,762 aligned retrieval units, using law-specific encoders into a frozen Mixedbread embedding space. Among evaluation-matched learned adapters, KAHM achieves the strongest teacher-space reconstruction and the best rank-sensitive retrieval performance at all evaluated cutoffs. At k=20, it obtains MRR@20 = 0.504, Hit@20 = 0.694, and Top-1 Accuracy = 0.411, while reducing online per-query time by 8.53 relative to direct transformer query encoding in the reported CPU setting. The results support KAHMs as compute-efficient encoders for supervised fixed-representation deployment regimes.

Mohit Kumar, Somayeh Kargaran, Bernhard A. Moser +1

University of Rostock, Germany · Software Competence Center Hagenberg GmbH, Austria Hagenberg, Austria

Legal IR Kernel Methods Text Embedding Models Dense Retrieval

May 1, 2026cs.LG

Deep Kernel Learning for Stratifying Glaucoma Trajectories

Effectively stratifying patient risk in chronic diseases like glaucoma is a major clinical challenge. Clinicians need tools to identify patients at high risk of progression from sparse and irregularly-sampled electronic health records (EHRs). We propose a novel deep kernel learning (DKL) architecture that leverages a Gaussian Process (GP) backend. The GP's kernel is defined by a transformer-based feature extractor applied to clinical-BERT embeddings to model glaucoma patient trajectories from multimodal EHR data. Our method successfully identifies three clinically distinct patient subgroups. Crucially, the model learns to decouple disease progression from current severity, identifying a high-risk group with a worsening trajectory despite having better average visual acuity than a second, stably poor group. This reveals that the model learns to identify progression risk rather than just the current disease state. This ability to stratify patients based on their risk trajectory progression offers a powerful tool for clinical decision support, enabling targeted interventions for high-risk individuals and improving the management of glaucoma care.

Bruce Rushing, Angela Danquah, Alireza Namazi +2

Research Computing, University of Virginia · School of Data Science, University of Virginia · School of Medicine, University of Virginia

Clinical Outcome Prediction Kernel Methods Longitudinal EHR Modeling Clinical Risk Prediction Gaussian Process Regression

Apr 29, 2026stat.ML

Laplace Approximation for Bayesian Tensor Network Kernel Machines

Uncertainty estimation is essential for robust decision-making in the presence of ambiguous or out-of-distribution inputs. Gaussian Processes (GPs) are classical kernel-based models that offer principled uncertainty quantification and perform well on small- to medium-scale datasets. Alternatively, formulating the weight space learning problem under tensor network assumptions yields scalable tensor network kernel machines. However, these assumptions break Gaussianity, complicating standard probabilistic inference. This raises a fundamental question: how can tensor network kernel machines provide principled uncertainty estimates? We propose a novel Bayesian Tensor Network Kernel Machine (LA-TNKM) that employs a (linearized) Laplace approximation for Bayesian inference. A comprehensive set of numerical experiments shows that the proposed method consistently matches or surpasses Gaussian Processes and Bayesian Neural Networks (BNNs) across diverse UCI regression benchmarks, highlighting both its effectiveness and practical relevance.

Albert Saiapin, Kim Batselier

1Delft Center for Systems and Control, Delft University of Technology

Uncertainty Quantification Bayesian Inference Kernel Methods

Apr 28, 2026math.DS

Dictionary learning for Kernel EDMD

Studying nonlinear dynamical systems through their state space behavior can be challenging, and one possible alternative is to analyze them via their associated Koopman operator. This turns the nonlinear problem into a linear, infinite-dimensional one. To approximate the operator in finite dimensions, extended dynamic mode decomposition (EDMD) is a commonly used algorithm. It requires a finite list of functionals and a set of snapshots from the system to compute an approximation of the operator and its corresponding spectrum. Instead of choosing the list of functionals directly, it can be implicitly defined via kernels, a method known as kernel extended dynamic mode decomposition (kEDMD). However, one still needs to define the kernel and choose its parameter values. In this paper, we aim to streamline this process by extending dictionary learning for EDMD to kernel learning in kEDMD. By simplifying kEDMD we show how to perform gradient-based optimization over the learnable kernel parameters, and demonstrate that this method leads to useful kernels for the original kEDMD. The focus of our work is a method that takes a weighted list of kernels with randomly initialized values as input and outputs a list of kernels and parameter values suitable for approximating the Koopman operator of the underlying system. We demonstrate that unimportant kernels can be removed from the list by analyzing the weights in the weighted sum. We evaluate the method across several experiments, including the Duffing oscillator and the Kuramoto-Sivashinsky PDE, showcasing the method's different strengths.

Erik Lien Bolager, Boumediene Hamzi, Houman Owhadi +2

School of Computation, Information and Technology, Munich Data Science Institute, Munich Center for Machine Learning, Technical University of Munich, Germany · Department of Computing and Mathematical Sciences, Caltech, USA · The Alan Turing Institute, UK +1

Koopman Operator Learning Nonlinear System Identification Kernel Methods Dictionary Learning

Apr 27, 2026cs.LG

Dynamic Regret for Online Regression in RKHS via Discounted VAW and Subspace Approximation

We study online regression with the square loss in a reproducing kernel Hilbert space under a dynamic regret criterion. The learner is compared with a time-varying comparator sequence, and the bounds depend on its path length in the RKHS norm. The proposed method transfers the finite-dimensional discounted Vovk--Azoury--Warmuth approach of Jacobsen & Cutkosky (2024) to the RKHS setting by means of finite-dimensional subspace approximations. For a fixed subspace, we run a VAW-based ensemble of discounted VAW forecasters over a geometric grid of discount factors. The additional approximation error is controlled by the uniform projection error of kernel sections. We then introduce a general orthogonal truncation method: starting from a feature expansion of the kernel, we construct the associated RKHS by introducing an inner product that makes the feature functions orthonormal, and then use the spans of the first basis functions as finite-dimensional approximation spaces. The resulting subspace reduction is applied to several approximation schemes. Explicit feature expansions yield fast-regime bounds for Gaussian and analytic dot-product kernels. Mercer truncations provide a spectral approximation method and lead to dynamic regret bounds in fast and slow regimes, depending on the eigenvalue decay. Finally, we study subspaces spanned by kernel sections and apply this construction to Matérn kernels.

Dmitry B. Rokhlin, Georgiy A. Karapetyants

Institute of Mathematics, Mechanics and Computer Sciences of the Southern Federal University

Sequential Decision Making Reproducing Kernel Hilbert Spaces Kernel Methods

Apr 27, 2026stat.ML

Identifiability and Stability of Generative Drifting with Companion-Elliptic Kernel Families

This paper studies the identifiability and stability of drifting fields in the framework of Generative Modeling via Drifting. The motivating question is whether a zero-drift equilibrium identifies the target distribution and whether an approximately vanishing drift implies weak distributional convergence. Since the original drifting model employs the Laplace kernel by default, we first analyze why Gaussian score-based arguments fail to apply. This analysis motivates the introduction of companion-elliptic kernel families, which are characterized by a companion potential satisfying an elliptic closure relation. We show that this class naturally contains the Laplace kernel and consists precisely of Gaussian and Matérn kernels with smoothness parameter

ν>0

. Within this class, we establish field identifiability for arbitrary Borel probability measures on

R^d

: if the drifting field between two such measures vanishes identically, then they must coincide. For stability, we demonstrate that convergence of the field alone does not guarantee weak convergence, since mass may escape to infinity while remaining invisible to the field. Although tightness directly removes this obstruction and restores weak stability, we prove that, even without tightness, every

C_0

-vague cluster point lies exactly on the defect ray

\{cp:0\le c\le1\}

. Consequently, a single scalar

C_0

observable suffices to detect the missing mass and recover weak convergence.

HakGeun Lee, Hyonho Chun

Department of Mathematical Sciences, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea.

Generative Modeling Convergence Analysis Kernel Methods Drifting Models

Apr 27, 2026cs.LG

Generalising maximum mean discrepancy: kernelised functional Bregman divergences

Bregman divergences play a pivotal role in statistics, machine learning and computational information geometry. Particularly in the context of machine learning, they are central to clustering, exponential families, parameter estimation and optimisation, among other things. Despite this, the full toolkit of Hilbert spaces and in particular reproducing kernel Hilbert spaces have not been systematically developed and applied to functional Bregman divergences, where points are functions rather than finite-dimensional parameter vectors. While other types of functional Bregman divergences have been studied, these are typically in a Banach space rather than more directly aligned with kernel methods and Hilbert-space geometry commonly used in machine learning. We consider functional Bregman divergences on a Hilbert space, where the self-dual pairing and Riesz representer afford us particularly convenient calculus. Further specialising Bregman generators as a composition involving a kernel mean embedding makes such divergences easy to estimate. We discuss applications in clustering, universal estimation, robust estimation and generative modelling, and contrast our approach with other types of Bregman divergences.

Russell Tsuchida, Frank Nielsen

Department of Data Science and AI, Monash University, Australia. · Sony Computer Science Laboratories, Inc. Tokyo, Japan.

Kernel Mean Embeddings Functional Bregman Divergences Maximum Mean Discrepancy Kernel Methods

Apr 26, 2026cs.CE

Partition-of-Unity Gaussian Kolmogorov-Arnold Networks

Gaussian basis functions provide an efficient and flexible alternative to spline activations in KANs. In this work, we introduce the partition-of-unity Gaussian KAN (PU-GKAN), a Shepard-type normalized Gaussian KAN in which the Gaussian basis values on each edge are divided by their local sum over fixed centers. This produces a partition-of-unity feature map with trainable coefficients, while preserving the standard edge-based KAN structure. The normalized construction gives exact constant reproduction at the edge level and admits an explicit finite-feature kernel interpretation. We formulate both the standard Gaussian KAN (GKAN) and PU-GKAN from a finite-feature and additive-kernel viewpoint, making the induced layer kernels and empirical feature matrices explicit. Using the first-layer feature matrix as the reference object, we adopt a practical scale-selection interval for

ε

, with the lower endpoint determined by adjacent-center overlap and the upper endpoint determined by a conservative conditioning threshold. Numerical experiments show that PU-GKAN reduces sensitivity to

ε

, improves validation accuracy for most smooth and moderately non-smooth targets, and gives more stable training behavior. The benefit persists across sample-size and center-number sweeps, higher-dimensional architectures, Matérn RBF bases, and physics-informed examples involving Helmholtz and wave equations. These results indicate that Shepard-type partition-of-unity normalization is a simple and effective stabilization mechanism for RBF-based KANs.

Amir Noorizadegan

Department of Mathematics, Hong Kong Baptist University, Hong Kong SAR, China

Kolmogorov-Arnold Networks Kernel Methods Radial Basis Function Networks

Apr 24, 2026cs.LG

Complex Stochastic Gradient Descent and Directional Bias in Reproducing Kernel Hilbert Spaces

Stochastic Gradient Descent (SGD) is a known stochastic iterative method popular for large-scale convex optimization problems due to its simple implementation and scalability. Some objectives, such as those found in complex-valued neural networks, benefit from updates like in SGD and Gradient Descent (GD) with a newly defined ``gradient'' that allows for complex parameters. This complex variant of the SGD/GD methods has already been proposed, but convergence guarantees without analyticity constraints have not yet been provided. We propose a variant of SGD (complex SGD) that allows for complex parameters, and we provide convergence guarantees under assumptions that parallel those from the real setting. Notably, these results extend to GD as well, and with the same set of assumptions, we confirm that some directional bias results extend from the real to the complex setting for kernel regression problems. We provide empirical results demonstrating the efficacy of the complex SGD in kernel regression problems utilizing complex reproducing kernel Hilbert spaces. In particular, we demonstrate we may recover superoscillation functions and Blaschke products from the Fock Space and Hardy Space, respectively, as the optimal functions for a particular choice of a loss function.

Natanael Alpay, Emeric Battaglia

Kernel Regression Reproducing Kernel Hilbert Spaces Kernel Methods Stochastic Optimization Convergence Stochastic Gradient Descent

Apr 23, 2026stat.ML

A Kernel Nonconformity Score for Multivariate Conformal Prediction

Multivariate conformal prediction requires nonconformity scores that compress residual vectors into scalars while preserving certain implicit geometric structure of the residual distribution. We introduce a Multivariate Kernel Score (MKS) that produces prediction regions that explicitly adapt to this geometry. We show that the proposed score resembles the Gaussian process posterior variance, unifying Bayesian uncertainty quantification with the coverage guarantees of frequentist-type. Moreover, the MKS can be decomposed into an anisotropic Maximum Mean Discrepancy (MMD) that interpolates between kernel density estimation and covariance-weighted distance. We prove finite-sample coverage guarantees and establish convergence rates that depend on the effective rank of the kernel-based covariance operator rather than the ambient dimension, enabling dimension-free adaptation. On regression tasks, the MKS reduces the volume of prediction region significantly, compared to ellipsoidal baselines while maintaining nominal coverage, with larger gains at higher dimensions and tighter coverage levels.

Louis Meyer, Wenkai Xu

University of Warwick

Conformal Prediction Kernel Methods

Apr 23, 2026cs.CV

Deep kernel video approximation for unsupervised action segmentation

This work focuses on per-video unsupervised action segmentation, which is of interest to applications where storing large datasets is either not possible, or nor permitted. We propose to segment videos by learning in deep kernel space, to approximate the underlying frame distribution, as closely as possible. To define this closeness metric between the original video distribution and its approximation, we rely on maximum mean discrepancy (MMD) which is a geometry-preserving metric in distribution space, and thus gives more reliable estimates. Moreover, unlike the commonly used optimal transport metric, MMD is both easier to optimize, and faster. We choose to use neural tangent kernels (NTKs) to define the kernel space where MMD operates, because of their improved descriptive power as opposed to fixed kernels. And, also, because NTKs sidestep the trivial solution, when jointly learning the inputs (video approximation) and the kernel function. Finally, we show competitive results when compared to state-of-the-art per-video methods, on six standard benchmarks. Additionally, our method has higher F1 scores than prior agglomerative work, when the number of segments is unknown.

Silvia L. Pintea, Jouke Dijkstra

Department of Intellgient Systems (DIS), Tilburg University, Tilburg, NL · Division of Image Processing, LUMC, Leiden, NL

Temporal Action Segmentation Kernel Methods

Apr 20, 2026quant-ph

Benchmarking Quantum Kernel Support Vector Machines Against Classical Baselines on Tabular Data: A Rigorous Empirical Study with Hardware Validation

Quantum kernel methods have been proposed as a promising approach for leveraging near-term quantum computers for supervised learning, yet rigorous benchmarks against strong classical baselines remain scarce. We present a comprehensive empirical study of quantum kernel support vector machines (QSVMs) across nine binary classification datasets, four quantum feature maps, three classical kernels, and multiple noise models, totalling 970 experiments with strict nested cross-validation. Our analysis spans four phases: (i) statistical significance testing, revealing that none of 29 pairwise quantum-classical comparisons reach significance at

α= 0.05

; (ii) learning curve analysis over six training fractions, showing steeper quantum slopes on six of eight datasets that nonetheless fail to close the gap to the best classical baseline; (iii) hardware validation on IBM ibm_fez (Heron r2), demonstrating kernel fidelity

r \geq 0.976

across six experiments; and (iv) seed sensitivity analysis confirming reproducibility (mean CV 1.4%). A Kruskal-Wallis factorial analysis reveals that dataset choice dominates performance variance (

\varepsilon^2 = 0.73

), while kernel type accounts for only 9%. Spectral analysis offers a mechanistic explanation: current quantum feature maps produce eigenspectra that are either too flat or too concentrated, missing the intermediate profile of the best classical kernel, the radial basis function (RBF). Quantum kernel training (QKT) via kernel-target alignment yields the single competitive result -- balanced accuracy 0.968 on breast cancer -- but with ~2,000x computational overhead. Our findings provide actionable guidelines for quantum kernel research. The complete benchmark suite is publicly available to facilitate reproduction and extension.

Siavash Kakavand, Christoph Strohmeyer, Michael Schlotter

SHARE at FAU, Schaeffler Technologies AG & Co. KG, Herzogenaurach 91074, Bavaria, Germany

Quantum Feature Maps Quantum Kernel Methods Support Vector Machines Tabular ML Kernel Methods

Apr 17, 2026cs.LG

UniCon: Unified Framework for Efficient Contrastive Alignment via Kernels

Contrastive objectives power state-of-the-art multimodal models, but their training remains slow, relying on long stochastic optimization. We propose a Unified Framework for Efficient Contrastive Alignment via Kernels (UniCon), which spans linear and nonlinear encoders as well as one-to-one and many-to-many alignments. At its core, UniCon introduces the contrastive similarity weight matrix

S(γ)

, which enables closed-form global solutions that provably replace minibatch back-propagation with exact updates. Through the lens of reproducing kernel Hilbert spaces (RKHS), UniCon provides a kernelized perspective that unifies contrastive alignment and reveals its connection to spectral methods. To validate the theory, we conduct experiments on synthetic, unimodal, multimodal, and zero-shot tasks, demonstrating that UniCon achieves substantial efficiency gains while preserving generality and strong empirical performance.

Hangke Sui, Yuqing Wang, Minh N Do

Department of Electrical & Computer Engineering, The Grainger College of Engineering, UIUC · Coordinated Science Laboratory, University of Illinois Urbana-Champaign (UIUC) · Siebel School of Computing and Data Science, The Grainger College of Engineering, UIUC +1

Contrastive Learning Cross-Modal Alignment Kernel Methods Multimodal Contrastive Learning

Apr 16, 2026stat.ML

Structural interpretability in SVMs with truncated orthogonal polynomial kernels

We study post-training interpretability for Support Vector Machines (SVMs) built from truncated orthogonal polynomial kernels. Since the associated reproducing kernel Hilbert space is finite-dimensional and admits an explicit tensor-product orthonormal basis, the fitted decision function can be expanded exactly in intrinsic RKHS coordinates. This leads to Orthogonal Representation Contribution Analysis (ORCA), a diagnostic framework based on normalized Orthogonal Kernel Contribution (OKC) indices. These indices quantify how the squared RKHS norm of the classifier is distributed across interaction orders, total polynomial degrees, marginal coordinate effects, and pairwise contributions. The methodology is fully post-training and requires neither surrogate models nor retraining. We illustrate its diagnostic value on a synthetic double-spiral problem and on a real five-dimensional echocardiogram dataset. The results show that the proposed indices reveal structural aspects of model complexity that are not captured by predictive accuracy alone.

Víctor Soto-Larrosa, Nuria Torrado, Edmundo J. Huertas

Dpto. de Biociencias, Facultad de Ciencias Biomédicas y de la Salud, Universidad Europea de Madrid, C/ Tajo, s/n, 28670 Villaviciosa de Odón, Madrid, Spain. · Dpto. de Matemáticas, Facultad de Ciencias, Universidad Autónoma de Madrid, C/ Francisco Tomás y Valiente, 7, 28049 Madrid, Spain. · Instituto de Ciencias Matemáticas (ICMAT), Campus de Cantoblanco UAM, C/ Nicolás Cabrera, 13–15, 28049 Madrid, Spain. +1

Interpretable ML Support Vector Machines Kernel Methods

Apr 16, 2026cs.CV

KVNN: Learnable Multi-Kernel Volterra Neural Networks

Higher-order learning is fundamentally rooted in exploiting compositional features. It clearly hinges on enriching the representation by more elaborate interactions of the data which, in turn, tends to increase the model complexity of conventional large-scale deep learning models. In this paper, a kernelized Volterra Neural Network (kVNN) is proposed. The key to the achieved efficiency lies in using a learnable multi-kernel representation, where different interaction orders are modeled by distinct polynomial-kernel components with compact, learnable centers, yielding an order-adaptive parameterization. Features are learned by the composition of layers, each of which consists of parallel branches of different polynomial orders, enabling kVNN filters to directly replace standard convolutional kernels within existing architectures. The theoretical results are substantiated by experiments on two representative tasks: video action recognition and image denoising. The results demonstrate favorable performance-efficiency trade-offs: kVNN consistently yields reduced model (parameters) and computational (GFLOPs) complexity with competitive and often improved performance. These results are maintained even when trained from scratch without large-scale pretraining. In summary, we substantiate that structured kernelized higher-order layers offer a practical path to balancing expressivity and computational cost in modern deep networks.

Haoyu Yun, Hamid Krim, Yufang Bao

Department of Electrical and Computer Engineering, North Carolina State University, Raleigh, NC, USA · Department of Mathematics and Computer Science, Fayetteville State University, Fayetteville, NC, USA

Efficient Neural Network Inference Kernel Methods Convolutional Neural Networks

Mar 31, 2026stat.ML

Breaking Data Symmetry is Needed For Generalization in Feature Learning Kernels

Grokking occurs when a model achieves high training accuracy but generalization to unseen test points happens long after that. This phenomenon was initially observed on a class of algebraic problems, such as learning modular arithmetic (Power et al., 2022). We study grokking on algebraic tasks in a class of feature learning kernels via the Recursive Feature Machine (RFM) algorithm (Radhakrishnan et al., 2024), which iteratively updates feature matrices through the Average Gradient Outer Product (AGOP) of an estimator in order to learn task-relevant features. Our main experimental finding is that generalization occurs only when a certain symmetry in the training set is broken. Furthermore, we empirically show that RFM generalizes by recovering the underlying invariance group action inherent in the data. We find that the learned feature matrices encode specific elements of the invariance group, explaining the dependence of generalization on symmetry.

Marcel Tomàs Bernal, Neil Rohit Mallinar, Mikhail Belkin

CFIS UPC · UC San Diego

Symmetry Breaking Kernel Methods Invariant Representation Learning Grokking

Mar 17, 2026cs.LG

Optimal uncertainty bounds for multivariate kernel regression under bounded noise: A Gaussian process-based dual function

Non-conservative uncertainty bounds are essential for making reliable predictions about latent functions from noisy data, and thus, a key enabler for safe learning-based control. In this domain, kernel methods such as Gaussian process regression are established techniques, thanks to their inherent uncertainty quantification mechanism. Still, existing bounds either pose strong assumptions on the underlying noise distribution, are conservative, do not directly apply in the multi-output case, or are difficult to integrate into downstream tasks. This paper addresses these limitations by presenting a tight, deterministic bound for multi-output functions in Reproducing Kernel Hilbert Spaces (RKHSs) subject to bounded noise. It is obtained through an unconstrained, duality-based formulation, which shares the same structure as classic Gaussian process confidence bounds, and can thus be straightforwardly integrated into downstream optimization pipelines. We show that the proposed bound generalizes existing results and illustrate its application using an example inspired by quadrotor dynamics learning.

Amon Lahr, Anna Scampicchio, Johannes Köhler +1

Institute for Dynamical Systems and Control, ETH Zurich, Zurich, Switzerland

Kernel Methods Gaussian Process Regression

Mar 7, 2026cs.LG

Margin in Abstract Spaces

Margin-based learning, exemplified by linear and kernel methods, is one of the few classical settings where generalization guarantees are independent of the number of parameters. This makes it a central case study in modern highly over-parameterized learning. We ask what minimal mathematical structure underlies this phenomenon. We begin with a simple margin-based problem in arbitrary metric spaces: concepts are defined by a center point and classify points according to whether their distance lies below

r

or above

R

. We show that whenever

R>3r

, this class is learnable in \emph{any} metric space. Thus, sufficiently large margins make learnability rely only on the triangle inequality, without any linear or analytic structure being necessary. Our first main result extends this phenomenon to concepts defined by bounded linear combinations of distance functions, and reveals a sharp threshold: there exists a universal constant such that whenever the margin is larger than this constant, the class is learnable in every metric space, while below it there exist metric spaces where it is not learnable at all. We then ask whether margin-based learnability can always be explained via an embedding into a linear space -- that is, reduced to linear classification in some Banach space through a kernel-type construction. We answer this negatively by demonstrating a margin learnable class that cannot be embedded into any Banach space in which linear classification with margins is learnable.

Yair Ashlagi, Roi Livni, Shay Moran +1

School of Electrical and Computer Engineering, Tel Aviv University. · Departments of Mathematics, Computer Science, and Data and Decision Sciences, Technion, and Google Research. · Departments of Mathematics, Technion.

Kernel Methods Statistical Learning Theory

Feb 26, 2026stat.ML

Regular Fourier Features for Nonstationary Gaussian Processes

Simulating a Gaussian process requires sampling from a high-dimensional Gaussian distribution, which scales cubically with the number of sample locations. Spectral methods address this challenge by exploiting the Fourier representation and treating the spectral density as a probability distribution suitable for Monte Carlo approximation. Although this probabilistic interpretation is valid for stationary processes, it is overly restrictive for the nonstationary case, where spectral densities are generally not probability measures. To avoid this limitation, we propose regular Fourier features for harmonizable processes with one-dimensional inputs. Our method discretizes the spectral representation directly, preserving the correlation structure among spectral weights without requiring probability assumptions. Assuming finite spectral support, this yields an efficient low-rank approximation that is positive semi-definite by construction and consistent under mild regularity conditions. When the spectral density is unknown, the framework also extends to kernel learning from data, which we explore as a proof of concept. We demonstrate the approximation on locally stationary and harmonizable mixture kernels, the latter with a complex-valued spectral density. As a feasibility study, we then apply the kernel-learning extension to real and synthetic data, where it matches competitive baselines.

Arsalan Jawaid, Abdullah Karatas, Jörg Seewig

Institute of Measurement and Sensor Technology, University of Kaiserslautern-Landau, Kaiserslautern, Germany · Independent Researcher, Mannheim, Germany

Fourier Feature Embeddings Gaussian Processes Low-Rank Approximation Kernel Methods Spectral Methods

Feb 23, 2026cs.LG

Generative Modeling via Kernelized Stochastic Interpolants

We develop a kernel method for generative modeling within the stochastic interpolant framework, replacing neural network training with linear systems. The drift of the generative SDE is

\hat b_t(x) = \nablaφ(x)^\topη_t

, where

η_t \in \mathbb{R}^P

solves a

P\times P

system computable from data, with

P

independent of the data dimension

d

. Since estimates are inexact, the diffusion coefficient

D_t

affects sample quality; the optimal

D_t^*

from Girsanov diverges at

t=0

, but this poses no difficulty and we develop an integrator that handles it seamlessly. The framework accommodates diverse feature maps: scattering transforms, pretrained generative models, etc, enabling generation and model combination without neural network training. We demonstrate the approach on financial time series, turbulence, and image generation.

Florentin Coeurdoux, Etienne Lempereur, Nathanaël Cuvelle-Magar +2

Capital Fund Management, Paris, France · Département d’informatique, ENS, Université PSL, Paris, France · Collège de France, Paris, France +2

Stochastic Differential Equations Stochastic Interpolants Generative Modeling Kernel Methods

Jan 29, 2026cs.LG

Mechanistic Evidence for Spectral Structures in Prior-Data Fitted Networks

Prior-Data Fitted Networks (PFNs) perform approximate Bayesian inference in a single forward pass, and tabular foundation models (TFMs) built on them are now widely used. To understand what networks infer internally, recent mechanistic studies of TFMs locate where predictions form, but treat these models as tabular predictors rather than as PFNs. It therefore remains unknown whether PFNs represent the spectral content of their context, the quantity that specifies a stationary kernel, and whether this content can be read out as an explicit kernel. We answer both questions. First, across seven PFNs, including four pretrained TFMs and a model trained only on a decision-tree prior, a linear probe on the residual stream recovers the frequency of the context with

R^2 \geq 0.95

. This structure is led by a single principal direction. Second, activation and subspace patching show that the network uses the structure through a low-dimensional subspace, where a few spectral directions move predictions far more than random ones. This holds even for the decision-tree model, so a spectral training prior is not required. On real datasets with up to 499 features, 64 of the 192 directions of TabPFN, chosen without labels, carry 85 to 95% of the causal effect of the context in all but one pair. Third, we introduce a Filter Bank Decoder that turns frozen PFN representations into an explicit stationary kernel through Bochner's theorem. Without any test-time optimization, the decoded kernel supports Gaussian process regression competitive with deep kernel learning and random Fourier features at about

200\times

lower cost. PFN latents therefore hold spectral structure that is causally used and recoverable as a portable kernel.

Kaustubh Sharma, Srijan Tiwari, Ojasva Nema +1

Department of Electrical Engineering, Indian Institute of Technology Roorkee (IIT Roorkee), Uttarakhand, India · Department of Metallurgical and Materials Engineering, Indian Institute of Technology Roorkee (IIT Roorkee), Uttarakhand, India

Prior-Data Fitted Networks Neural Network Interpretability Frequency-Domain Feature Learning Kernel Methods Mechanistic Interpretability Kernel Regression

Jan 20, 2026stat.ML

Finite-Sample Unbiased Variance of MMD under Unbalanced Sampling: Exact Estimation and Quasi-Linear Computation

Accurately and efficiently estimating the variance of the Maximum Mean Discrepancy (MMD) remains challenging, particularly for unbalanced sample sizes. In this paper, we derive a finite-sample unbiased estimator of the MMD variance. To overcome the traditional

\mathcal{O}(N^2)

computational bottleneck, we develop a recursive prefix-suffix accumulation scheme for the Laplace kernel, reducing the computational complexity to

\mathcal{O}(N \log N)

while requiring

\mathcal{O}(N)

memory. Experimental results verify the theoretical exactness and numerical stability of the proposed estimator and demonstrate its scalability on large datasets. Furthermore, the method proves effective for monitoring distributional convergence during the training of Time-series Generative Adversarial Networks (TimeGAN).

Shijie Zhong, Yikun Yang, Da Gong +1

School of Power and Energy, Northwestern Polytechnical University, 1 Dongxiang Road, Xi’an, 710129, Shanxi, China. · Advanced Power Research Institute, Northwestern Polytechnical University, 619 Jicui Road, Chengdu, 610299, Sichuan, China. · Taihang Laboratory, Chengdu, 610200, Sichuan, China. +1

Maximum Mean Discrepancy Kernel Methods

Jan 7, 2026cs.LG

Minimum distance classification for nonlinear dynamical systems

We address the problem of classifying trajectories or sequences generated by nonlinear dynamical systems, where each class corresponds to a distinct dynamical system. We propose Dynafit, a kernel-based method that learns a distance metric between training data and the underlying dynamics. New observations are assigned to the class whose dynamics best fit the observations according to the learned metric.The learning algorithm approximates the Koopman operator, which globally linearizes the dynamics in a (potentially infinite-dimensional) feature space associated with a kernel function. The distance metric is computed in the feature space independently of its dimensionality by exploiting the kernel trick commonly used in machine learning. The kernel function can be tailored to incorporate prior knowledge of the dynamics when available. We consider a classical test example, the logistic map as a discrete dynamical system, and derive analytically the kernel function from the polynomial Koopman basis that exactly linearizes the dynamics. Dynafit is applicable to a wide range of classification tasks involving nonlinear dynamical systems and sensors. We illustrate its effectiveness through three examples: chaos detection in the logistic map, recognition of handwritten dynamical patterns, and classification of visual dynamic textures.

Dominique Martinez

Aix-Marseille University, CNRS, ISM, 13009 Marseille, France

Dynamical Systems Time Series Classification Classification Kernel Methods

Dec 23, 2025stat.ML

Computationally efficient goodness-of-fit tests through kernelized Stein discrepancy

Models with intractable normalizing constants are widely used in statistics and machine learning. Assessing the adequacy of such models poses significant challenges: obtaining samples from the fitted model often requires sophisticated sampling algorithms. Moreover, model fitting sometimes requires iterative numerical optimization, making bootstrap procedures that require repeated refitting computationally expensive. In this paper, we leverage the kernel-based testing framework to develop a general semiparametric goodness-of-fit test based on the kernelized Stein discrepancy. We establish the consistency and the asymptotic null distribution of the test statistic under general nuisance estimation. To produce a level-

α

test, we propose a novel influence-adjusted wild bootstrap that requires neither refitting the model nor sampling from it. We prove the consistency of the proposed bootstrap test procedure under the null and the alternative, and characterize its limiting power under contiguous local alternatives. Across simulations ranging from classical normality testing to models with intractable likelihoods, the proposed test delivers competitive or superior power at a computational cost orders of magnitude lower than that of existing approaches. We illustrate the method by assessing the adequacy of a protein signaling network model for reverse-phase protein array data from lung adenocarcinoma tumors. As a complementary insight, we show that the SKSD test can be regarded as a nonparametric score test under exponentially tilted models, connecting score-based and distance-based goodness-of-fit testing.

Zhihan Huang, Ziang Niu

Department of Statistics and Data Science University of Pennsylvania

Kernel Methods Statistical Hypothesis Testing

Nov 5, 2025cs.LG

Sparse, self-organizing ensembles of local kernels detect rare statistical anomalies

Modern artificial intelligence has revolutionized how we extract representations from scientific data, yet the statistical properties of these representations remain poorly controlled, causing misspecified anomaly detection methods to falter. The hardest anomalies to detect are the rare, weakly separable ones hiding within the nominal distribution-a regime that grows in importance as models mature and easily separable signals are exhausted. We identify structural desiderata for detection in this regime under minimal prior information: sparsity, to enforce parsimony; locality, to preserve geometric sensitivity; and competition, to promote efficient allocation of model capacity. These principles define a class of self-organizing local kernels that adaptively partition the representation space around regions of statistical imbalance. As an instantiation, we introduce SparKer, a sparse ensemble of Gaussian kernels trained in a semi-supervised Neyman-Pearson framework to locally model the likelihood ratio between a sample that may contain anomalies and an anomaly-free reference. We provide theoretical insights into the mechanisms driving detection and self-organization, and demonstrate the approach on realistic high-dimensional problems in scientific discovery, open-world novelty detection, intrusion detection, and generative-model validation. Ensembles of only a handful of kernels identify statistically significant anomalies in representation spaces of thousands of dimensions while remaining sensitive across regimes, underscoring the interpretability, efficiency, and scalability of the approach.

Gaia Grosso, Sai Sumedh R. Hindupur, Thomas Fel +3

NSF AI Institute for Artificial Intelligence and Fundamental Interactions, Cambridge, 02139, MA · Laboratory for Nuclear Science, Massachusetts Institute of Technology, Cambridge, 02139, MA · School of Engineering and Applied Sciences, Harvard University, Allston, 02134, MA +1

Kernel Methods Statistical Learning Theory Anomaly Detection

Oct 17, 2025cs.LG

Theoretical Refinement of CLIP by Utilizing Linear Structure of Optimal Similarity

In this study, we propose an enhancement to the similarity computation mechanism in multimodal contrastive pretraining frameworks such as CLIP. Prior theoretical research has demonstrated that the optimal similarity metrics between paired modalities should correspond to the pointwise mutual information (PMI) between the two modalities. However, the current implementations of CLIP and its variants fail to fully utilize the underlying linear structure of PMI. We therefore propose KME-CLIP, which leverages this structure through the inner product in a reproducing kernel Hilbert space (RKHS). We theoretically prove that, under our assumptions, the KME-CLIP similarity can bring the contrastive loss arbitrarily close to its optimal value, which is attained by PMI, as the size of the point set grows, and we empirically evaluate KME-CLIP against CLIP and its kernel-based variants across several retrieval and classification tasks.

Naoki Yoshida, Satoshi Hayakawa, Yuhta Takida +3

The University of Tokyo · Sony Group Corporation · Sony AI

Cross-Modal Learning Kernel Mean Embeddings Multimodal Pretraining Cross-Modal Retrieval Multimodal Classification Kernel Methods+2