Nonparametric Regression

Follow this topic Ask about this topic

Nonparametric regression studies how to estimate an unknown relationship between inputs and outcomes while allowing its shape to be determined by data and assumptions such as smoothness or additivity. Common approaches include kernel and spline smoothing, Gaussian processes and reproducing-kernel methods, tree ensembles, and neural networks; researchers also develop online, functional, and distributional variants for specialized data. A central theoretical goal is to match minimax rates, which quantify the best possible prediction accuracy under a given smoothness class, while adapting to unknown structure and handling dependence, missingness, or distribution shift. The main tension is that flexible models can achieve strong guarantees under suitable conditions, but high dimension, computational cost, and reliable uncertainty estimates remain difficult.

Latest papers

45 papers7 papers in the last four weeks, down 30% on the four weeks before. 0.0% of all new papers.

May 4, 2026

ParaRNN: An Interpretable and Parallelizable Recurrent Neural Network for Time-Dependent Data

The proliferation of large-scale and structurally complex data has spurred the integration of machine learning methods into statistical modeling. Recurrent neural networks (RNNs), a foundational class of models for time-dependent data, can be viewed as nonlinear extensions of classical autoregressive moving average models. Despite their flexibility and empirical success in machine learning, RNNs often suffer from limited interpretability and slow training, which hinders their use in statistics. This paper proposes the Parallelized RNN (ParaRNN), a novel model composed of multiple small recurrent units. ParaRNN admits an additive representation that decouples recurrent dynamics into interpretable components, whose behavior can be characterized through recurrence features. This interpretability enables its applications in nonparametric regression for time-dependent data, while the design also allows efficient parallelization. The approximation capacity and non-asymptotic prediction error bounds in a nonparametric regression setting are established for ParaRNN. Empirical results on three sequential modeling tasks further demonstrate that ParaRNN achieves performance comparable to vanilla RNNs while offering improved interpretability and efficiency.

Yuxi Cai, Lan Li, Feiqing Huang +1

Department of Statistics and Actuarial Science, University of Hong Kong

Interpretable ML Recurrent Neural Networks Nonparametric Regression Neural Network Approximation Theory

Ask PDF

Apr 28, 2026

Adversarial Robustness of NTK Neural Networks

Deep learning models are widely deployed in safety-critical domains, but remain vulnerable to adversarial attacks. In this paper, we study the adversarial robustness of NTK neural networks in the context of nonparametric regression. We establish minimax optimal rates for adversarial regression in Sobolev spaces and then show that NTK neural networks, trained via gradient flow with early stopping, can achieve this optimal rate. However, in the overfitting regime, we prove that the minimum norm interpolant is vulnerable to adversarial perturbations.

Yuxuan Hou

1Qiuzhen College, Tsinghua University

Neural Network Robustness Adversarial Robustness Nonparametric Regression Neural Tangent Kernel

Ask PDF

Apr 14, 2026

SMART Fine-tuning Factor Augmented Neural Lasso

Fine-tuning is a widely used strategy for adapting pre-trained models to new tasks, yet its methodology and theoretical properties in high-dimensional nonparametric settings with variable selection have not yet been developed. We propose a source-model-augmented residual tuning (SMART) framework, which incorporates the pre-trained source model as an augmented feature into the target learner and estimates only the residual target-specific component. The approach is widely applicable, from parametric and sparse models to neural networks and blackbox machine learning models. We focus on the development of fine-tuning factor-augmented neural Lasso, resulting in SMART-FAN-Lasso. This transfer-learning framework for high-dimensional nonparametric regression with variable selection simultaneously handles covariate and posterior shifts. We use a low-rank factor structure to manage high-dimensional dependent covariates and a residual tuning decomposition in which the target function is expressed as a function of source model and other target-specific variables, thereby reducing the effective complexity of the target task. We derive minimax-optimal excess risk bounds, characterizing the precise conditions, in terms of relative sample sizes and function complexities, under which fine-tuning yields statistical acceleration over single-task learning. Extensive numerical experiments across diverse covariate- and posterior-shift scenarios demonstrate that SMART-FAN-Lasso consistently outperforms standard baselines and achieves near-oracle performance even under severe target sample size constraints, empirically validating the derived rates.

Jinhang Chai, Jianqing Fan, Cheng Gao +1

Department of Operations Research and Financial Engineering, Princeton University

Fine-Tuning Transfer Learning Nonparametric Regression Feature Selection Statistical Learning Theory Residual Learning

Ask PDF

Apr 10, 2026

Online Quantile Regression for Nonparametric Additive Models

This paper introduces a projected functional gradient descent algorithm (P-FGD) for training nonparametric additive quantile regression models in online settings. This algorithm extends the functional stochastic gradient descent framework to the pinball loss. An advantage of P-FGD is that it does not need to store historical data while maintaining

O(J_t\ln J_t)

computational complexity per step where

J_t

denotes the number of basis functions. Besides, we only need

O(J_t)

computational time for quantile function prediction at time

t

. These properties show that P-FGD is much better than the commonly used RKHS in online learning. By leveraging a novel Hilbert space projection identity, we also prove that the proposed online quantile function estimator (P-FGD) achieves the minimax optimal consistency rate

O(t^{-\frac{2s}{2s+1}})

where

t

is the current time and

s

denotes the smoothness degree of the quantile function. Extensions to mini-batch learning are also established.

Haoran Zhan

Quantile Regression Nonparametric Regression Stochastic Gradient Descent

Ask PDF

Apr 8, 2026

The Theory and Practice of Highly Scalable Gaussian Process Regression with Nearest Neighbours

Gaussian process (

GP

) regression is a widely used non-parametric modeling tool, but its cubic complexity in the training size limits its use on massive data sets. A practical remedy is to predict using only the nearest neighbours of each test point, as in Nearest Neighbour Gaussian Process (

NNGP

) regression for geospatial problems and the related scalable

GPnn

method for more general machine-learning applications. Despite their strong empirical performance, the large-

n

theory of

NNGP/GPnn

remains incomplete. We develop a theoretical framework for

NNGP

and

GPnn

regression. Under mild regularity assumptions, we derive almost sure pointwise limits for three key predictive criteria: mean squared error (

MSE

), calibration coefficient (

CAL

), and negative log-likelihood (

NLL

). We then study the

L_2

-risk, prove universal consistency, and show that the risk attains Stone's minimax rate

n^{-2α/(2p+d)}

, where

α

and

p

capture regularity of the regression problem. We also prove uniform convergence of

MSE

over compact hyper-parameter sets and show that its derivatives with respect to lengthscale, kernel scale, and noise variance vanish asymptotically, with explicit rates. This explains the observed robustness of

GPnn

to hyper-parameter tuning. These results provide a rigorous statistical foundation for

NNGP/GPnn

as a highly scalable and principled alternative to full

GP

models.

Robert Allison, Tomasz Maciazek, Anthony Stephenson

School of Mathematics University of Bristol Bristol, BS8 1UG, UK

Gaussian Process Regression Nonparametric Regression Statistical Learning Theory

Ask PDF

Apr 3, 2026

Nonparametric Regression Discontinuity Designs with Survival Outcomes

Quasi-experimental evaluations are central for generating real-world causal evidence and complementing insights from randomized trials. The regression discontinuity design (RDD) is a quasi-experimental design that can be used to estimate the causal effect of treatments that are assigned based on a running variable crossing a threshold. Such threshold-based rules are ubiquitous in healthcare, where predictive and prognostic biomarkers frequently guide treatment decisions. However, standard RD estimators rely on complete outcome data, an assumption often violated in time-to-event analyses where censoring arises from loss to follow-up. To address this issue, we propose a nonparametric approach that leverages doubly robust censoring corrections and can be paired with existing RD estimators. Our approach can handle multiple survival endpoints, long follow-up times, and covariate-dependent variation in survival and censoring. We discuss the relevance of our approach across multiple areas of applications and demonstrate its usefulness through simulations and the prostate component of the Prostate, Lung, Colorectal and Ovarian (PLCO) Cancer Screening Trial where our new approach offers several advantages, including higher efficiency and robustness to misspecification. We have also developed an open-source software package,

\texttt{rdsurvival}

, for the

\texttt{R}

language.

Maximilian Schuessler, Erik Sverdrup, Robert Tibshirani +1

Causal Effect Estimation Survival Analysis Nonparametric Regression

Ask PDF

Mar 22, 2026

Gradient Descent with Projection Finds Over-Parameterized Neural Networks for Learning Low-Degree Polynomials with Nearly Minimax Optimal Rate

We study the problem of learning a low-degree spherical polynomial of degree

k_0 = Θ(1) \ge 1

defined on the unit sphere in

\RR^d

by training an over-parameterized two-layer neural network with augmented feature in this paper. Our main result is the significantly improved sample complexity for learning such low-degree polynomials. We show that, for any regression risk

\eps \in (0, Θ(d^{-k_0})]

, an over-parameterized two-layer neural network trained by a novel Gradient Descent with Projection (GDP) requires a sample complexity of

n \asymp Θ( \log(4/δ) \cdot d^{k_0}/\eps)

with probability

1-δ

for

δ\in (0,1)

, in contrast with the representative sample complexity

Θ(d^{k_0} \max\set{\eps^{-2},\log d})

. Moreover, such sample complexity is nearly unimprovable since the trained network renders a nearly optimal rate of the nonparametric regression risk of the order

\log({4}/δ) \cdot Θ(d^{k_0}/{n})

with probability at least

1-δ

. On the other hand, the minimax optimal rate for the regression risk with a kernel of rank

Θ(d^{k_0})

Θ(d^{k_0}/{n})

, so that the rate of the nonparametric regression risk of the network trained by GDP is nearly minimax optimal. In the case that the ground truth degree

k_0

is unknown, we present a novel and provable adaptive degree selection algorithm which identifies the true degree and achieves the same nearly optimal regression rate. To the best of our knowledge, this is the first time that a nearly optimal risk bound is obtained by training an over-parameterized neural network with a popular activation function (ReLU) and algorithmic guarantee for learning low-degree spherical polynomials. Due to the feature learning capability of GDP, our results are beyond the regular Neural Tangent Kernel (NTK) limit.

Yingzhen Yang, Ping Li

School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ 85281, USA · VecML Inc., Bellevue, WA 98004, USA

Low-Degree Polynomial Method Neural Network Generalization Nonparametric Regression Shallow Neural Networks Gradient Projection

Ask PDF

Mar 18, 2026

Consistency of the $k$ -Nearest Neighbor Regressor under Complex Survey Designs

We study the consistency of the

k

-nearest neighbor regressor under complex survey designs. While consistency results for this algorithm are well established for independent and identically distributed data, corresponding results for complex survey data are lacking. We show that the

k

-nearest neighbor regressor is consistent under regularity conditions on the sampling design and the distribution of the data. We derive lower bounds for the rate of convergence and show that these bounds exhibit the curse of dimensionality, as in the independent and identically distributed setting. Empirical studies based on simulated and real data illustrate our theoretical findings.

Caren Hasler

k-Nearest Neighbors Nonparametric Regression Statistical Learning Theory

Ask PDF

Mar 11, 2026

Deep regression learning from dependent observations with minimum error entropy principle

This paper considers nonparametric regression from strongly mixing observations. The proposed approach is based on deep neural networks with minimum error entropy (MEE) principle. We study two estimators: the non-penalized deep neural network (NPDNN) and the sparse-penalized deep neural network (SPDNN) predictors. Upper bounds of the expected excess risk are established for both estimators over the classes of Hölder and composition Hölder functions. For the models with Gaussian error, the rates of the upper bound obtained match (up to a logarithmic factor) with the lower bounds established in \cite{schmidt2020nonparametric}, showing that both the MEE-based NPDNN and SPDNN estimators from strongly mixing data can achieve the minimax optimal convergence rate.

William Kengne, Modou Wade

Institut Camille Jordan, Université Jean Monnet, 23 Rue Dr Paul Michelon 42023 Saint-Etienne Cedex 2, France · THEMA, CY Cergy Paris Université, 33 Boulevard du Port, 95011 Cergy-Pontoise Cedex, France

Nonparametric Regression Minimax Estimation Sparse Neural Networks

Ask PDF

Feb 24, 2026

Standard Transformers Achieve the Minimax Rate in Nonparametric Regression with $C^{s,λ}$ Targets

The tremendous success of Transformer models in fields such as large language models and computer vision necessitates a rigorous theoretical investigation. To the best of our knowledge, this paper is the first work proving that standard Transformers can approximate Hölder functions

C^{s,λ}\left([0,1]^{d\times n}\right) $$ (s\in\mathbb{N}_{\geq0},0<λ\leq1)

under the

L^t

distance (

t \in [1, \infty]

) with arbitrary precision. Building upon this approximation result, we demonstrate that standard Transformers achieve the minimax optimal rate in nonparametric regression for Hölder target functions. It is worth mentioning that, by introducing two metrics: the size tuple and the dimension vector, we provide a fine-grained characterization of Transformer structures, which facilitates future research on the generalization and optimization errors of Transformers with different structures. As intermediate results, we also derive the upper bounds for the Lipschitz constant of standard Transformers and their memorization capacity, which may be of independent interest. These findings provide theoretical justification for the powerful capabilities of Transformer models.

Yanming Lai, Defeng Sun

Department of Applied Mathematics, The Hong Kong Polytechnic University, Hung Hom, Hong Kong, China

Transformer Expressivity Neural Network Approximation Theory Nonparametric Regression Statistical Learning Theory

Ask PDF

Aug 31, 2025

When is Fine-Tuning Scalable? Certified Recovery from Noisy Data and Linear-Cost Fine-Tuning on Structured Task Shifts

Transformers often transfer across related tasks with little fine-tuning; we study this question in noisy nonparametric regression for continuous maps

f,g:[0,1]^d\to\mathbb{R}^D

, representing the source and shifted tasks, using a single-query, single-head transformer

\mathcal{T}(\cdot)=V\operatorname{Softmax}(-α(\cdot,K))

with

\ell^{\infty}

-based alignment scores

α

. Rather than assume that gradient-based training reaches an empirical risk minimizer, we give a model-specific two-stage procedure certifying source-task recovery and out-of-distribution (OOD) adaptation. With high probability, the resulting attention head recovers

f

to arbitrary accuracy

\varepsilon>0

in every fixed

L^p([0,1]^d)

norm,

1\le p<\infty

, from noisy observations. For Lipschitz targets, its rate is minimax-optimal up to logarithmic factors without assuming oracle access to an asymptotic ERM. If

D=1

and

f

is Pfaffian than the values matrix factors into

AC

with

A

scaling at-most linearly

\mathcal{O}(\varepsilon^{-1})

in the reciprocal approximation error

\varepsilon>0

. Importantly, our training procedure encodes all our identified (potentially) shared geometric structure from

f

only into the values matrix factor

C

and the keys matrix

K

allowing for rapid access, and all

f

-specific structure only into

A

. Our main result concerns the OOD phase; where we identify geometric structures, which if shared by the source

f

and OOD

g

task; then only only the values matrix factor

A

needs to be fine-tuned to guarantee an

L^p

-recovery of

g

. Critically, fine-tuning only

A

is a convex problem which involving an at-most linear number of parameters

\mathcal{O}(\varepsilon^{-1})

; dramatically cutting down on the worst-case scaling of

\mathcal{O}(\varepsilon^{-\mathbf{d}})

unavoidable in the source-training phase.

Anastasis Kratsios, Tin Sum Cheng, Bum Jun Kim +1

Department of Mathematics, McMaster University, Canada · Department of Mathematics and Computer Science, University of Basel, Switzerland · Department of Statistical Sciences, University of Toronto, Canada +1

Fine-Tuning OOD Generalization Transformer Parameter-Efficient Fine-Tuning Nonparametric Regression

Ask PDF

Jun 22, 2025

Identifiable Convex-Concave Regression via Sub-gradient Regularised Least Squares

We propose a novel nonparametric regression method that models complex input-output relationships as the sum of convex and concave components. The method-Identifiable Convex-Concave Nonparametric Least Squares (ICCNLS)-decomposes the target function into additive shape-constrained components, each represented via sub-gradient-constrained affine functions. To address the affine ambiguity inherent in convex-concave decompositions, we introduce global statistical orthogonality constraints, ensuring that residuals are uncorrelated with both intercept and input variables. This enforces decomposition identifiability and improves interpretability. We further incorporate L1, L2 and elastic net regularisation on sub-gradients to enhance generalisation and promote structural sparsity. The proposed method is evaluated on synthetic and real-world datasets, including healthcare pricing data, and demonstrates improved predictive accuracy and model simplicity compared to conventional CNLS and difference-of-convex (DC) regression approaches. Our results show that statistical identifiability, when paired with convex-concave structure and sub-gradient regularisation, yields interpretable models suited for forecasting, benchmarking, and policy evaluation.

William Chung

Department of Decision Analytics and Operations, City University of Hong Kong, Hong Kong

Parameter Identifiability Nonparametric Regression

Ask PDF

Apr 21, 2025

Deep learning with missing data

In the context of multivariate nonparametric regression with missing covariates, we propose Pattern Embedded Neural Networks (PENNs), which can be applied in conjunction with any existing imputation technique. In addition to a neural network trained on the imputed data, PENNs pass the vectors of observation indicators through a second neural network to provide a compact representation. The outputs are then combined in a third neural network to produce final predictions. Our main theoretical result exploits an assumption that the observation patterns can be partitioned into cells on which the Bayes regression function behaves similarly, and belongs to a compositional Hölder class. It provides a finite-sample excess risk bound that holds for an arbitrary missingness mechanism, and in combination with a complementary minimax lower bound, demonstrates that our PENN estimator attains in typical cases the minimax rate of convergence as if the cells of the partition were known in advance, up to a poly-logarithmic factor in the sample size. Numerical experiments on simulated, semi-synthetic and real data confirm that the PENN estimator consistently improves, often dramatically, on standard neural networks without pattern embedding. Code to reproduce our experiments, as well as a tutorial on how to apply our method, is publicly available.

Tianyi Ma, Tengyao Wang, Richard J. Samworth

Statistical Laboratory, University of Cambridge · Department of Statistics, London School of Economics

Neural Network Generalization Learning with Missing Data Nonparametric Regression

Ask PDF

Nov 15, 2024

Dense ReLU Neural Networks for Temporal-spatial Model

In this paper, we focus on fully connected deep neural networks utilizing the Rectified Linear Unit (ReLU) activation function for nonparametric estimation. We derive non-asymptotic bounds that lead to convergence rates, addressing both temporal and spatial dependence in the observed measurements. By accounting for dependencies across time and space, our models better reflect the complexities of real-world data, enhancing both predictive performance and theoretical robustness. We also tackle the curse of dimensionality by modeling the data on a manifold, exploring the intrinsic dimensionality of high-dimensional data. We broaden existing theoretical findings of temporal-spatial analysis by applying them to neural networks in more general contexts and demonstrate that our proof techniques are effective for models with short-range dependence. Our empirical simulations across various synthetic response functions underscore the superior performance of our method, outperforming established approaches in the existing literature. These findings provide valuable insights into the strong capabilities of dense neural networks (Dense NN) for temporal-spatial modeling across a broad range of function classes.

Carlos Misael Madrid Padilla, Zhi Zhang, Xiaokai Luo +2

Department of Statistics and Data Science, Washington University in St. Louis. · Department of Statistics and Data Science, University of California, Los Angeles · Department of Statistics, University of Notre Dame

Nonparametric Regression ReLU Neural Networks Neural Network Approximation Theory

Ask PDF

Oct 14, 2024

Statistical Properties of Deep Neural Networks with Dependent Data

This paper develops theory for deep neural network (DNN) estimators under dependent data. To provide theory applicable to a variety of DNN-based estimators, I first establish nonasymptotic probability bounds on the theoretical and empirical

\mathcal{L}^{2}

-errors of nonparametric sieve estimators for a general class of estimation problems under possibly nonstationary

β

-mixing data taking values in unbounded sets. I then apply the theory to fully connected and convolutional DNN estimators without bounds or sparsity restrictions on the DNN weights. For both DNN classes, I derive general results when the function to be estimated is Hölder smooth and the data are nonstationary, subgaussian, and

β

-mixing with either exponential or polynomial decay. I then specialize these to nonparametric regression, logistic regression, and quantile regression settings. Under exponential

β

-mixing, the resulting estimators attain the nonparametric minimax rate of Stone (1982) up to logarithmic factors.

Chad Brown

Department of Economics, University of Colorado Boulder, 80309, USA.

Nonparametric Regression Neural Network Approximation Theory Statistical Learning Theory

Ask PDF