Accelerating Non-Smooth and Heavy-Tailed Sampling

Pervez Ali; Xiaoyu Wang; Yingli Wang; Lingjiong Zhu

stat.MLOct 8, 2026

Accelerating Non-Smooth and Heavy-Tailed Sampling

Authors: Pervez Ali, Xiaoyu Wang, Yingli Wang, Lingjiong Zhu

Organizations: Department of Mathematics, Florida State University, Tallahassee, Florida, United States of America · Fintech Thrust, Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong Province, People’s Republic of China · School of Mathematical Sciences, Fudan University, Shanghai, People’s Republic of China

Abstract

Anchored Langevin dynamics (ALD) is useful for non-smooth sampling where the density of the target distribution is possibly non-differentiable and heavy-tailed; reflected anchored Langevin dynamics (RALD) can sample possibly non-differentiable target density on a constrained domain. In this paper, we propose and study non-reversible anchored Langevin dynamics (NALD) for sampling possibly non-differentiable and heavy-tailed target density in the Euclidean space and the non-reversible reflected anchored Langevin dynamics (NRALD) for sampling possibly non-differentiable target density in the constrained space. Our construction adds a circulation drift generated by a possibly state-dependent divergence-free skew-symmetric matrix field and a stream potential. It preserves the target distribution without requiring derivatives of target density, admits a random-time-change representation, and applies both on the whole Euclidean space and on bounded domains with normal reflection. By breaking reversibility, we show that NALD and NRALD can converge to their target distributions faster than their reversible counterparts via finite-time non-asymptotic convergence analysis, a large deviations analysis and asymptotic variance reduction. Numerical experiments demonstrate the efficiency of the proposed algorithms.

Figures & tables

Figure 1: $2$ -Wasserstein distance between the particle distribution and an exact sample of the target ( 7.5 ) with the MCP regularizer ( 7.6 ), as a function of the iterate on logarithmic axes, for the reversible anchored Langevin algorithm ( $\alpha=0$ ) and NRALD with the constant skew-symmetric matrix $J_{a}$ ( $\alpha=1$ , $a=1.75$ ), averaged over $5$ replications of $N=5000$ particles, from the initial distributions $\mathcal{N}(0,10\,I_{d})$ (left) and $\mathcal{U}(-5,5)^{d}$ (right).

Figure 2: $\mathcal{W}_{1}$ distance ( 7.20 ) between the walkers and an exact sample of the target, averaged over $8$ replicas, as a function of the iterate on logarithmic axes, for the reversible projected anchored Langevin algorithm ( $\alpha=0$ ) and NRALD with the tensor $J_{s}$ in ( 2.26 ) with $s=8$ and $s=16$ ( $\alpha=1$ ), for the targets (A) ( 7.14 ) (left) and (B) ( 7.15 ) (right) on the ball ( 7.16 ). The dotted line is the two-sample floor and the dashed line is $3\times$ floor, the threshold of $\tau_{3}$ .

Figure 3: Visualized density plots of the first two coordinates $(x_{1},x_{2})$ for target (A), $U(x)=\|L^{-1}x\|_{2}$ in ( 7.14 ), restricted to the ball $K$ in ( 7.16 ). From left to right: $5000$ exact samples of the target obtained by rejection, and the walkers of the reversible projected anchored Langevin algorithm ( $J=0$ ) and of NRALD with $J_{s}$ in ( 2.26 ) for $s=8$ and $s=16$ after $5000$ iterates with $\eta=5\times 10^{-4}$ ( $8$ replicas of $5000$ walkers), shown as kernel density estimates with the same contour levels.

Figure 4: Visualized density plots of the first two coordinates $(x_{1},x_{2})$ for target (B), $U(x)=\|L^{-1}x\|_{1}$ in ( 7.15 ), restricted to the ball $K$ in ( 7.16 ). From left to right: $5000$ exact samples of the target obtained by rejection, and the walkers of the reversible projected anchored Langevin algorithm ( $J=0$ ) and of NRALD with $J_{s}$ in ( 2.26 ) for $s=8$ and $s=16$ after $5000$ iterates with $\eta=5\times 10^{-4}$ ( $8$ replicas of $5000$ walkers), shown as kernel density estimates with the same contour levels.

Figure 5: Accuracy over the training set and the test set for the synthetic data. The red part (solid line) denotes the mean and standard deviation of the non-reversible anchored Langevin algorithm with $J_{s}(x)$ ( $\alpha=1$ ) and the blue part (dashed line) denotes the mean and standard deviation of the reversible anchored Langevin algorithm ( $\alpha=0$ ).

Figure 6: Accuracy over the training set and the test set for the synthetic data. The red part (solid line) denotes the mean and standard deviation of the non-reversible anchored Langevin algorithm with $J_{g}(x)$ ( $\alpha=1$ ) and the blue part (dashed line) denotes the mean and standard deviation of the reversible anchored Langevin algorithm ( $\alpha=0$ ).

Figure 7: Accuracy over the training set and the test set for the Telescope dataset over the unit ball. The red part (solid line) denotes the mean and standard deviation of the non-reversible anchored Langevin algorithm with $J_{s}(x)$ ( $\alpha=1$ ), the blue part (dashed line) denotes the mean and standard deviation of the reversible anchored Langevin algorithm ( $\alpha=0$ ), and the grey dotted line is the test accuracy of the Bayesian logistic regression fit.

Figure 8: Accuracy over the training set and the test set for the Telescope dataset over the smoothed $l_{1}$ ball. The red part (solid line) denotes the mean and standard deviation of the non-reversible anchored Langevin algorithm with $J_{g}(x)$ ( $\alpha=1$ ), the blue part (dashed line) denotes the mean and standard deviation of the reversible anchored Langevin algorithm ( $\alpha=0$ ), and the grey dotted line is the test accuracy of the Bayesian logistic regression fit.

Figure 9: Accuracy over the training set and the test set for the Titanic dataset over the unit ball. The red part (solid line) denotes the mean and standard deviation of the non-reversible anchored Langevin algorithm with $J_{s}(x)$ ( $\alpha=1$ ), the blue part (dashed line) denotes the mean and standard deviation of the reversible anchored Langevin algorithm ( $\alpha=0$ ), and the grey dotted line is the test accuracy of the Bayesian logistic regression fit.

Figure 10: Accuracy over the training set and the test set for the Titanic dataset over smoothed $l_{1}$ ball. The red part (solid line) denotes the mean and standard deviation of the non-reversible anchored Langevin algorithm with $J_{g}(x)$ ( $\alpha=1$ ), the blue part (dashed line) denotes the mean and standard deviation of the reversible anchored Langevin algorithm ( $\alpha=0$ ), and the grey dotted line is the test accuracy of the Bayesian logistic regression fit.

Explore similar work

Reflected Anchored Langevin Algorithms
Oct 7, 2026Changwei Tu, Xiaoyu Wang, Yingli Wang +2Langevin Dynamics Markov Chain Monte Carlo
Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong Province, People’s Republic of China · School of Mathematical Sciences, Fudan University, Shanghai, People’s Republic of China · School of Mathematics and Statistics, Beijing Institute of Technology, Beijing 100081, People’s Republic of China +1
Penalized Nonreversible Langevin for Constrained Sampling
Sep 21, 2026Pervez Ali, Weihao Dong, Xiaoyu WangMarkov Chain Monte Carlo Stochastic Gradient Langevin Dynamics
Department of Mathematics, Florida State University, Tallahassee, Florida, United States of America · Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong, People’s Republic of China · FinTech Thrust, Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong, People’s Republic of China
Dimension-Uniform Discretization Analysis of Preconditioned Annealed Langevin Dynamics for Multimodal Gaussian Mixtures
May 15, 2026Lorenzo Baldassari, Josselin Garnier, Knut Solna +1Langevin Dynamics Diffusion Sampling
University of Basel · Ecole Polytechnique, IP Paris · University of California Irvine +1

Oct 7, 2026, stat.MLJ/K move · Enter open · S save

Reflected Anchored Langevin Algorithms

Changwei Tu, Xiaoyu Wang, Yingli Wang, Xicheng Zhang+1

Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong Province, People’s Republic of China · School of Mathematical Sciences, Fudan University, Shanghai, People’s Republic of China · School of Mathematics and Statistics, Beijing Institute of Technology, Beijing 100081, People’s Republic of China+1

Langevin Dynamics Markov Chain Monte Carlo

Open paper AskPDF arXiv

First order Langevin algorithms for constrained sampling in machine learning, such as projected Langevin Monte Carlo which are based on discretizations of reflected Langevin dynamics, require differentiable log densities that limits their applicability. This paper introduces reflected anchored Langevin dynamics (RALD), a reflected diffusion that converges to non-differentiable targets on constrained domains. The method uses a smooth anchored reference potential and multiplies the drift and noise covariance of its reflected Langevin dynamics by the same state dependent scaling factor. Its Euler-Maruyama discretization with projection gives reflected anchored Langevin Monte Carlo (RALMC) algorithm. We prove explicit convergence bounds and iteration complexity for RALMC in the 2-Wasserstein distance to the target distribution. Numerical experiments are provided to illustrate the theoretical predictions and the empirical performance of the method.

Abstract

Explore similar work

Reflected Anchored Langevin Algorithms

Penalized Nonreversible Langevin for Constrained Sampling

Dimension-Uniform Discretization Analysis of Preconditioned Annealed Langevin Dynamics for Multimodal Gaussian Mixtures