Out-of-Distribution generalization of quantile regression with heavy tailed inputs: an SVM approach

Extreme value theory provides rigorous theory and statistical tools for extrapolation in machine learning, particularly in settings where traditional methods struggle due to data scarcity in the tails. A broad range of tasks benefit from these advances, including regression and classification beyond the training data, extreme quantile regression, supervised and unsupervised dimension reduction, generative artificial intelligence and anomaly detection. This review synthesizes recent developments in these fields at the intersection of statistical learning and extreme value theory, with a focus on principled methods based on asymptotically motivated representations of the tail of univariate and multivariate distributions. We consider different theoretical frameworks for both asymptotically dependent and independent data and discuss how they translate into efficient statistical methods for extrapolation to extreme regions. By addressing both theoretical and practical aspects, we offer a comprehensive overview of the state-of-the-art in this quickly evolving field, and identify promising directions for future research.

This work proposes deep nonparametric Instrumental variable quantile regression (IVQR), a two-stage estimator that combines conditional diffusion modeling with a kernel-smoothed conditional moment formulation. In the first stage, we estimate the joint conditional distribution of the outcome and endogenous covariates given the instrument using a variance-preserving conditional diffusion model. In the second stage, we approximate the conditional moment operator through Monte Carlo sampling and a kernel-smoothed surrogate for the indicator function, and then estimate the structural quantile function by empirical risk minimization over deep neural networks. We establish an excess-risk bound for the proposed estimator and derive end-to-end total variation guarantees for the conditional diffusion model under unbounded support, explicitly accounting for score estimation, early stopping, and discretization errors. Our theory is developed under a polynomial-tail envelope on the data distribution and degenerates continuously to the exponential setting: as the tail index grows, the obtained excess-risk rate converges to the minimax-optimal rate of nonparametric regression, thus our heavy-tailed theory covers the classical light-tailed nonparametric guarantees as a limiting case. Simulation studies and a real-data application demonstrate that the proposed method outperforms existing nonparametric IVQR approaches, with gains that become increasingly pronounced as the dimensionality of the covariates and instruments increases.

Constructing minimum-volume prediction regions that satisfy conditional coverage is a fundamental challenge in multivariate regression. Standard approaches rely on explicitly estimating the full conditional density and subsequently thresholding it. This two-step plug-in process is notoriously difficult, sensitive to estimation errors, and computationally expensive. One would like to instead optimize the region directly. Formulating a direct solution is challenging, however, because it requires minimizing a volume objective that is coupled with the conditional quantiles of the model's own estimation error. In this work, we address this challenge. We introduce super-level-set regression (SLS), a novel mathematical framework that successfully resolves this implicit coupling, allowing us to directly parameterize and optimize the geometric boundaries of the target conditional level sets. By bypassing full distribution estimation and leveraging flexible volume-preserving frontier functions, our approach natively captures complex, multimodal, and disjoint conditional structures end-to-end. Ultimately, SLS offers a new perspective on multivariate conditional quantile regression, replacing the restrictive assumptions of density-first methods with a direct geometric optimization strategy.