Atom-Averaged Features

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 6Week of Sep 21

Latest papers 17

Sep 30, 2026cs.LG

Dimension-Free Rank Lifting from Random Hyperplane Arrangements

We study the width required for a randomly initialized hidden layer of a neural network to achieve rank lifting. Namely, given a dataset

X \in \mathbb{R}^{m \times d}

of

m

,

d

-dimensional input vectors separated by an angle of at least

θ

, we consider the random feature matrix

σ(XR)

, where

R

is standard Gaussian. For positively homogeneous nonpolynomial activations, which include sign, Heaviside, ReLU, and ReLU powers among others, we prove that

n \gtrsim \frac{1}θ\max\left\{m,\log\left(\frac{1}δ\right)\right\}

neurons suffice for

σ(XR)

to have full row rank

m

with probability at least

1-δ

. This dimension-free bound exponentially improves the previous general-dimensional guarantee for sign features (Drago et al., 2026) and is essentially tight. The proof shows that one random feature column escapes every proper subspace of

\mathbb{R}^m

with probability

Ω(θ)

, using a coupling of nearby Gaussian directions and a local crossing of the induced hyperplane arrangement. We also study stable rank lifting, where the goal is to establish a quantitative analogue of exact rank lifting, i.e., a lower bound on the smallest eigenvalue of the empirical feature Gram matrix in high-probability. Our analysis unifies and generalizes stable rank guarantees for all

q

-homogeneous non-polynomial activations following prior work in Panigrahi et al. (2020) and Song (2026). In particular, we combine a diagonally dominant Taylor tail of the population kernel with truncation and matrix concentration, to show that for positively homogeneous nonpolynomial activations, stable rank lifting is achieved at width

n \gtrsim C^q \frac{m}{θ^{2q+1}} \log^{2q+\frac{1}{2}}\left(\frac{m}θ\right) \log\left(\frac{m}δ\right),

where

q

is the degree of the activation and

C > 0

is some universal constant.

Luca Becchetti, Matteo Russo, Ruben Skorupinski

Sapienza University of Rome, Italy · EPFL, Switzerland

Atom-Averaged Features Low-Rank Structure Intrinsic Dimensionality Activation Functions Ranking

Sep 14, 2026cs.CR

Implementing a White-Box Undetectable Backdoor for Random Fourier Features

Goldwasser et al. showed that undetectable backdoors can be planted in machine learning models trained with the Random Fourier Features (RFF) algorithm, under a hardness assumption tied to the Continuous Learning With Errors (CLWE) problem. Under standard cryptographic assumptions, even a full white-box audit of a model's weights cannot detect this class of backdoor. The construction is stated in terms of cryptographic reductions and probabilistic lemmas, without a reference implementation, and relies on secondary machinery such as the Sparse Gaussian Pancakes distribution and a homogeneous CLWE conditional density. Its realizability in ordinary numerical code is not obvious from the paper alone. This paper implements the white-box CLWE-RFF backdoor construction end to end using only numpy and scipy, to test whether this threat is realizable with commodity scientific-computing tools or requires specialized cryptographic infrastructure. We give two samplers for the core

GP_d(b_k)

distribution. The first is a rejection-sampling proxy. The second is an exact closed-form sampler derived from the homogeneous CLWE density and verified against its own analytic form. Using this implementation, we run statistical indistinguishability tests, covering both weight-space and functional black-box comparisons. We find no evidence of detectable difference between backdoored and clean models across a range of sparsity ratios

ρ= d_{\text{sparse}}/D

. We report which parts of the construction were straightforward to realize, which required derivation not spelled out in the paper. We also highlight which parts we did not attempt to reproduce, including the underlying lattice hardness reduction. We see this work as a contribution to understanding the practical realizability of the Goldwasser white-box CLWE core, not as a new theoretical result.

Michael Collins, Jada Cumberland, Brianne Dunn +3

Laboratory for Advanced Cybersecurity Research Laurel, MD 20707 · School of Cybersecurity Old Dominion University Norfolk, VA 23259 · Old Dominion University Norfolk, VA 23259 +1

Atom-Averaged Features Black Box Obfuscation Fourier Replay-Based Continual Learning

Sep 9, 2026quant-ph

A Quantum-Inspired Dequantization Method for Diagonally Weighted Matrix Functions: Application to Learning with Optimized Random Features

Quantum-inspired classical algorithms have dequantized several quantum machine learning routines by replacing quantum linear-algebra subroutines with classical counterparts. However, the sampler based on quantum singular value transformation (QSVT) for learning with optimized random features is not covered by existing dequantization frameworks, because the matrix to be inverted is not itself available through sampling access. In this work, we develop a classical algorithm to address this type of quantum-advantage candidate. Our method samples heavy indices, reduces the transformation to a small principal block, and outputs a sparse classical representation with operator-norm guarantees. Applying this method dequantizes the sampler for optimized random features, giving a classical sampler with prescribed accuracy and polynomially related runtime. These results show that the factorization underlying a quantum block encoding can itself provide sufficient classical structure even when sampling-and-query access to the composite matrix is unavailable.

Natsuto Isogai, Mio Murao, Hayata Yamasaki

Department of Physics, Graduate School of Science, The University of Tokyo, Tokyo, Japan · Department of Computer Science, Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan

Quantum Computational Advantage Atom-Averaged Features Singular Value Decomposition Quantum Error Correction Matrix Markovian Sampling

Sep 3, 2026math.NA

Spectral Convergence of Random Feature Method in Multiple Dimensions

We first prove spectral convergence of the random feature method (RFM) for multidimensional targets in Sobolev, Gevrey, ultra-analytic, and bandlimited classes. The analysis establishes general high-probability approximation estimates in the interpolation scale generated by a kernel integral operator. On a single event determined only by the sampled features, one random space approximates every target in a prescribed source ball; moreover, for each target, a single coefficient vector defines an approximant that attains spectral accuracy simultaneously in all admissible error norms. For both regularity-adapted frequency distributions and uniform distributions on growing frequency windows, the resulting rates range from super-exponential to algebraic, depending on the regularity of the target. Second, we establish abstract error estimates for strong- and weak-form RFM discretizations, thereby converting the preceding approximation bounds into convergence estimates for multidimensional second-order elliptic boundary value and eigenvalue problems. Finally, for random feature matrices (RFMtxs), we prove super-exponential singular-value decay with Fourier features and exponential decay with

\tanh

features, together with corresponding condition-number lower bounds. The analysis identifies a common mechanism: the same spectral approximation that yields high accuracy also drives severe ill-conditioning.

Pingbing Ming, Hao Yu

Atom-Averaged Features Spectral Representation Method Regularity Singular Value Decomposition Convergence

Aug 4, 2026cs.LG

Random features for Grassmannian kernel approximation with bounded rank-one projections

We propose a family of random feature maps for scalable kernel machines on low-dimensional subspaces, ie on the Grassmannian manifold. Such representations are useful when data classes or clusters are well described by the span of a few samples. Classical Grassmannian kernels, including the projection and Binet-Cauchy kernels, require full Gram matrices, which leads to prohibitive computational and memory costs for large high-dimensional subspace datasets. We address this limitation using random features based on rank-one projections of subspace projection matrices followed by bounded non-linear transforms, either periodic or binary, to control the resulting distributions. We show that inner products in the random feature space approximate well-defined rotation-invariant Grassmannian kernels that depend only on the principal angles between subspaces. When the number of features is sufficiently large relative to the intrinsic subspace dimension, the approximation holds uniformly over all fixed-dimensional subspaces with high probability. For periodic transforms, the approximated kernel has a closed-form expression with tunable behaviour between inverse Binet-Cauchy and Gaussian-type regimes. Binary transforms yield compact one-bit subspace features, although no closed-form kernel is known. Structured rank-one projections based on randomised fast Fourier transforms further reduce computation without sacrificing practical accuracy. Experiments on synthetic data and ETH-80 classification tasks show that these features accurately preserve Grassmannian geometry while reducing computation, memory, and storage. Rank-one embeddings therefore provide a practical and scalable alternative to classical Grassmannian kernels.

Rémi Delogne, Laurent Jacques

INMA/ICTEAM, UCLouvain, Belgium

Atom-Averaged Features Kernel Method Subspace Approximation Random

Jul 30, 2026cs.LG

Representations from Pretrained Machine-Learning Interatomic Potentials as Coarse Coordinates for Material Generation and Evaluation

Generative machine learning is increasingly used for inorganic crystal structure generation. Most models and the corresponding evaluation approaches rely on simple forms of crystal structure representation. In this paper, we showcase the power of atom-averaged features from pretrained Machine-Learning Interatomic Potentials (MLIPs), such as MACE, for such tasks. We first introduce a distance measure that assesses the output of material generative models by capturing both quality and novelty in a single distribution-based evaluation framework. In particular, we introduce the Coarse-Fine Transport Distance (CFTD) using two different featurizers, where the quality component is based on coarse MACE features. We showcase CFTD's versatility in capturing crystal-structure quality while also detecting memorization, and compare it with the recently introduced continuous SUN metrics. We further show that coarse MACE features can be used as guidance for a material generative model.

Paul Hagemann, Katharina Ueltzen, Simon Müller +2

Crystal Structure Interatomic Potentials Generative Models Atom-Averaged Features Latent Transport

Jul 29, 2026cs.LG

Universality and Approximation Rates of Graph Neural Networks with Random Features

We investigate message-passing graph neural networks with random node features. Random node features are known to enhance the expressiveness of graph neural networks (GNNs) both theoretically and empirically. Here, we establish a novel universality result focusing on permutation-equivariant neural networks (PENNs), a class of GNNs built from feedforward neural network components that subsumes many prominent GNN architectures. We show that PENNs, combined with partially random node features, can approximate arbitrarily well in probability any measurable permutation-invariant or permutation-equivariant function on directed graphs of fixed size with multidimensional node and edge features. For

k

-times continuously differentiable functions,

k\geq 2

, we also derive upper bounds on the approximation rates, relating the complexity of the feedforward components of a PENN in terms of layer depth and number of nonzero weights to the desired approximation accuracy.

Lukas Gonon, Thilo Meyer-Brandis, Niklas Weber

School of Computer Science University of St. Gallen St. Gallen, 9000, Switzerland · Department of Mathematics LMU Munich Munich, 80333, Germany

Graph Neural Networks Atom-Averaged Features Neural Approximations Approximation Neural Network Equivariant Neural Networks+1

Jul 26, 2026cs.LG

A Coulomb Particle Model for Learning Kernel Attention in Transformers

Randomized features provide a scalable approximation to kernel machines, but their performance depends strongly on the choice of feature distribution. We propose a particle-based method that learns this distribution by optimizing kernel-target alignment while regularizing particles with a Riesz/Coulomb repulsive potential. The resulting Hamiltonian yields diverse, task-adaptive random features and admits a mean-field description through a McKean--Vlasov equation. We instantiate the method in linearized Transformer attention by learning positive random-feature maps in a first alignment phase, then freezing the kernel and training the remaining network parameters with cross-entropy. Experiments on synthetic classification and sentence-level benchmarks show that learned kernelized attention can improve accuracy, calibration, and robustness for several feature maps while preserving linear-attention inference complexity.

Masoud Badiei Khuzani, Sharath Honnaiah, Atiq Islam +2

eBay Inc., San Jose, CA, USA

Atom-Averaged Features Transformer Attention Transformer Architectures Fokker--Planck Equation Value Alignment Friction

Jul 22, 2026math.NA

A Structure-Adaptive Random Feature Method for High-Dimensional Elliptic PDEs

Random-feature methods reduce high-dimensional elliptic PDE collocation to linear coefficient problems, but full-dimensional trial spaces overlook lower-dimensional structure. We introduce the Hierarchical Analysis-of-Variance Random Feature Method (HA-RFM), which selects coordinate blocks using closed Sobol indices of the PDE residual, identifies oblique low-rank features from fitted-predictor gradients, and couples all retained features in one regularized least-squares solve. Under structural and stability hypotheses, we establish an

L^2

error bound that links solution and residual truncation to finite-width approximation and regularized finite-sample fitting, and we derive guarantees for width and structure recovery. The resulting width is polynomial in the dimension at fixed interaction order, with dimension-independent higher-order contributions under uniform structural control. Residual screening achieves exact recovery of the prescribed three-pair support, while fitted-predictor gradients recover oblique directions through dimension

50

. In random-ridge tests, less than

1\%

additional width reduces errors by factors of

14

-

39

over coordinate blocks and

34

-

100

over equal-width full-dimensional RFM. Semilinear computations extend HA-RFM through dimension

100

, while dense and distributed interactions delineate the coordinate families required for broader structure.

Jiale Linghu, Hao Dong, Yangshuai Wang

School of Mathematics and Statistics, Xidian University, Xi’an 710071, China. · Department of Mathematics, National University of Singapore, 10 Lower Kent Ridge Road, 119076, Singapore

Atom-Averaged Features Low-Dimensional Structure Random

Jun 8, 2026cs.LG

Bernstein-Schur Kernels: Random Features by Sketched Modulation and Radial Randomization

Bernstein--Schur kernels are products of a finite-feature kernel and a completely monotone shift-invariant kernel: nonstationary kernels falling between the shift-invariant and dot-product templates random features exploit, so neither Bochner sampling nor polynomial sketching applies to the full kernel directly. We give one random-feature construction for the whole class that randomizes both factors: it sketches the finite modulation and samples the radial factor's one-dimensional Bernstein--Widder scale before applying Gaussian random Fourier features, giving feature dimension

Dm

, free of the

O(d^2)

size of the exact modulation feature. With the modulation kept exact (the

m\to\infty

limit), we prove unbiasedness, an exact variance, and a matrix-Bernstein operator-norm bound controlled by the top kernel and modulation eigenvalues and an intrinsic dimension rather than the crude

N\max_{ij}

route. Whitening this argument at the ridge makes the effective dimension

d_{\mathrm{eff}}(λ)

the \emph{exact} intrinsic dimension of the matrix variance, so

O((1+\|P\|_{\mathrm{op}}/λ)\log(d_{\mathrm{eff}}/δ))

radial draws preserve the kernel-ridge solution; tilting the draw by a closed-form whitened leverage improves this to the effective-dimension count

O((1+d_{\mathrm{eff}})\log(d_{\mathrm{eff}}/δ))

. Conditioning on the sketch carries every guarantee to the deployed doubly-randomized estimator up to one additive sketch term, and all hold for the whole class with the modulation Gram in place of the polynomial one. The flagship instance is the biased

yat

-kernel

k_{yat,b}(w,x)=(w^\top x+b)^2/(\|w-x\|^2+\varepsilon)

, whose family span contains the inverse-multiquadric kernel by finite differences in

b

.

Taha Bouhsine

Azetta AI

Atom-Averaged Features Kernel Hilbert Spaces Fourier Feature Learning Low-Rank Structure Random+2

May 14, 2026eess.SY

Randomized Atomic Feature Models for Physics-Informed Identification of Dynamic Systems

We present a physics-informed framework for system identification based on randomized stable atomic features. Impulse responses are represented as random superpositions of stable atoms, namely damped complex exponentials associated with poles sampled inside a prescribed disk. Identification is then cast as a convex regularized least-squares problem with optional linear, second-order-cone, and KYP constraints. The approach generalizes random Fourier and random Laplace features to the damped, nonstationary regime relevant to engineering systems while retaining modal interpretability and scalable finite-dimensional computation. The main analytic point is an operator-theoretic Disk-Bochner viewpoint: positive measures over stable poles generate positive-definite kernels with a radius-dependent shift defect, while a converse scalar disk moment representation for an arbitrary kernel is characterized by subnormality of the canonical shift. We prove this statement, establish an RKHS-to-l1 embedding, show that sampled poles induce a valid finite atomic gauge, discuss random-feature convergence, and state sparse-recovery guarantees conditionally on the restricted-eigenvalue properties of the realized disk-Vandermonde or input-output design matrix. We also connect the normalized transfer function problem to Nevanlinna-Pick interpolation and LFT set-membership. The framework directly encodes stability margins, modal localization, DC-gain bounds, monotonicity, passivity, relative degree, settling-time targets, and time/frequency-domain error bounds. Numerical comparisons illustrate how physically meaningful priors can compensate for poor excitation and improve constrained impulse-response recovery in an under-informative data setting.

Rajiv Singh, Mario Sznaier, Lennart Ljung

∗The MathWorks Inc., 1 Apple Hill Drive, Natick, MA 01760 USA · ∗∗ECE Dept., Northeastern University, Boston, MA 02115, USA · ∗∗∗Div. of Automatic Control, Linköping University, Sweden

System Identification Atom-Averaged Features Physics-Informed Learning Impulse Response Kernel Method Spectral Decomposition

May 11, 2026stat.ML

Characterizing the Generalization Error of Random Feature Regression with Arbitrary Data-Augmentation

This paper aims at analyzing the regularization effect that data augmentation induces on supervised regression methods in the proportional regime, where the number of covariates grows proportionally to the number of samples. We provide a tight characterization of the test error, measured in mean squared error, in terms only of the population quantities of the true data, as well as first and second order statistics of the augmentation scheme. Our results are valid under misspecified feature maps, and for any network architecture where only the last readout layer is trained, and the rest of the network is either frozen or randomly initialized. We specify our results in the case of Gaussian data, and show that our asymptotic characterization is tight in this setting.

Lucas Morisset, Alain Durmus, Adrien Hardy

Qube Research & Technologies · École Polytechnique

Data Augmentation Regularization Atom-Averaged Features Semi-Supervised Learning Gaussian Primitives

May 7, 2026stat.ML

Fourier Feature Methods for Nonlinear Causal Discovery: FFML Scoring, TRFF Scoring, and FFCI Testing in Mixed Data

Gaussian process (GP) marginal likelihood scores and kernel conditional independence tests are theoretically appealing for nonlinear causal discovery but computationally prohibitive at scale. We present three complementary RFF-based methods forming a practical toolkit for score-based, constraint-based, and hybrid causal discovery. The Fourier Feature Marginal Likelihood (FFML) score approximates the exact GP marginal likelihood by replacing the

n x n

kernel Gram matrix with a finite-dimensional feature representation, reducing cost to

O(nm^2 + m^3)

while retaining the probabilistic interpretation and automatic complexity penalty of the exact score. FFML extends to mixed (continuous and discrete) parent sets via a product-kernel construction, with a Kronecker path for small discrete parent sets and a Hadamard-product path otherwise. The Tetrad Random Fourier Feature (TRFF) score is a complementary BIC-style alternative using penalized Student-t regression with random Fourier features. TRFF offers robustness to heavy-tailed noise and faster runtime than FFML. Empirically, TRFF and FFML exhibit a complementary precision-recall profile: TRFF achieves higher precision while FFML achieves better recall and lower SHD overall. The Fourier Feature Conditional Independence (FFCI) test is a fast nonparametric CI test for mixed data, using ridge residualization in feature space and a Frobenius-norm cross-covariance statistic approximated as a weighted sum of chi-squared variables. Empirically, BOSS+FFML achieves the lowest SHD on nonlinear data, while BOSS+TRFF offers the highest precision. When run through PC-Max, FFCI and RCIT exhibit complementary precision-recall profiles: RCIT is more precise while FFCI achieves better recall and substantially lower SHD, at approximately twice the runtime.

Joseph D. Ramsey

Department of Philosophy Carnegie Mellon University Pittsburgh, PA 15213

Causal Discovery Methods False Discovery Rate Atom-Averaged Features Fourier Covariance Gaussian Process

Mar 24, 2026cs.LG

Asymptotic Learning Curves for Diffusion Models with Random Features Score and Manifold Data

We study the theoretical behavior of denoising score matching--the learning task associated to diffusion models--when the data distribution is supported on a low-dimensional manifold and the score is parameterized using a random feature neural network. We derive asymptotically exact expressions for the test, train, and score errors in the high-dimensional limit. Our analysis reveals that, for linear manifolds the sample complexity required to learn the score function scales linearly with the intrinsic dimension of the manifold, rather than with the ambient dimension. Perhaps surprisingly, the benefits of low-dimensional structure starts to diminish once we have a non-linear manifold. These results indicate that diffusion models can benefit from structured data; however, the dependence on the specific type of structure is subtle and intricate.

Anand Jerry George, Nicolas Macris

École Polytechnique Fédérale de Lausanne (EPFL), Lab for Statistical Mechanics of Inference in Large Systems (SMILS), CH-1015 Lausanne, Switzerland

Low-Dimensional Structure Data Manifold Atom-Averaged Features Diffusion Models Sample Complexity Neural Network

Mar 2, 2026cs.LG

Scaling Laws of SignSGD in Linear Regression: When Does It Outperform SGD?

We study scaling laws of signSGD under a power-law random features (PLRF) model that accounts for both feature and target decay. We analyze the population risk of a linear model trained with one-pass signSGD on Gaussian-sketched features. We express the risk as a function of model size, training steps, learning rate, and the feature and target decay parameters. Comparing against the SGD risk analyzed by Paquette et al. (2024), we identify a drift-normalization effect and a noise-reshaping effect unique to signSGD. We then obtain compute-optimal scaling laws under the optimal choice of learning rate. Our analysis shows that the noise-reshaping effect can make the compute-optimal slope of signSGD steeper than that of SGD in regimes where noise is dominant. Finally, we observe that the widely used warmup-stable-decay (WSD) schedule further reduces the noise term and sharpens the compute-optimal slope, when feature decay is fast but target decay is slow.

Jihwan Kim, Dogyoon Song, Chulhee Yun

Seoul National University & KAIST InnoCORE LLM · University of California, Davis · KAIST

Stochastic Gradient Descent Scaling Laws Atom-Averaged Features Signsgd

Oct 9, 2025cs.LG

Computationally-efficient Graph Modeling with Refined Graph Random Features

We propose refined GRFs (GRFs++), a new class of Graph Random Features (GRFs) for efficient and accurate computations involving kernels defined on the nodes of a graph. GRFs++ resolve some of the long-standing limitations of regular GRFs, including difficulty modeling relationships between more distant nodes. They reduce dependence on sampling long graph random walks via a novel walk-stitching technique, concatenating several shorter walks without breaking unbiasedness. By applying these techniques, GRFs++ inherit the approximation quality provided by longer walks but with greater efficiency, trading sequential, inefficient sampling of a long walk for parallel computation of short walks and matrix-matrix multiplication. Furthermore, GRFs++ extend the simplistic GRFs walk termination mechanism (Bernoulli schemes with fixed halting probabilities) to a broader class of strategies, applying general distributions on the walks' lengths. This improves the approximation accuracy of graph kernels, without incurring extra computational cost. We provide empirical evaluations to showcase all our claims and complement our results with theoretical analysis.

Krzysztof Choromanski, Avinava Dubey, Arijit Sehanobish +1

Google DeepMind, New York, NY, USA · Columbia University, New York, NY, USA · Google Research, Mountain View, CA, USA +2

Atom-Averaged Features Graph Representation Learning Approximation

Sep 29, 2020cs.LG

Random Polytope Descriptors

We introduce a class of random polytopes which simultaneously generalizes several known constructions. While being fairly general, these polytopes are also computationally exceptionally benign. We indicate how these properties can be exploited for classification and clustering tasks in data analysis. Crucially, our construction lets users smoothly trade off between a tighter description of the data and faster computation.

Michael Joswig, Marek Kaluba, Lukas Ruff

Technische Universität Berlin, Chair of Discrete Mathematics/Geometry · MPI MiS Leipzig · Adam Mickiewicz University, Poznań, Poland +1

Polytopes Clustering Atom-Averaged Features Point Clouds