A Stein Identity for q-Gaussians with Bounded Support

Sophia Sklaviadis; Thomas Moellenhoff; Andre F. T. Martins; Mario A. T. Figueiredo; Mohammad Emtiyaz Khan

cs.LGMar 4, 2026

A Stein Identity for q-Gaussians with Bounded Support

Authors: Sophia Sklaviadis, Thomas Moellenhoff, Andre F. T. Martins, Mario A. T. Figueiredo, Mohammad Emtiyaz Khan

Organizations: RIKEN Center for AI Project, Tokyo, Japan · Instituto de Telecomunicações, Instituto Superior Técnico, Universidade de Lisboa, Portugal

Abstract

Stein's identity is a fundamental tool in machine learning with applications in generative models, stochastic optimization, and other problems involving gradients of expectations under Gaussian distributions. Less attention has been paid to problems with non-Gaussian expectations. Here, we consider the class of bounded-support $q$ -Gaussians and derive a new Stein identity leading to gradient estimators which have nearly identical forms to the Gaussian ones, and which are similarly easy to implement. We do this by extending the previous results of Landsman, Vanduffel, and Yao (2013) to prove new Bonnet- and Price-type theorems for q-Gaussians. We also simplify their forms by using escort distributions. Our experiments show that bounded-support distributions can reduce the variance of gradient estimators, which can potentially be useful for Bayesian deep learning and sharpness-aware minimization. Overall, our work simplifies the application of Stein's identity for an important class of non-Gaussian distributions.

Figures & tables

Figure 1: Bounded support

q

-Gaussians are a subclass of location-scale families that are elliptically contoured, that is, they are obtained by composing a quadratic

s(\mbox{

\mbox{

\mathbf{x}

}

})=(\mbox{

\mbox{

\mathbf{x}

}

}-\mbox{

\mbox{

\boldsymbol{\mu}

}

})^{\top}\mbox{

\mbox{

\boldsymbol{\Sigma}

}

}^{-1}(\mbox{

\mbox{

\mathbf{x}

}

}-\mbox{

\mbox{

\boldsymbol{\mu}

}

})

with a generator function

g

, thus

p(\mbox{

\mbox{

\mathbf{x}

}

})=g\bigl(s(\mbox{

\mbox{

\mathbf{x}

}

})\bigr)

. On the right, we show three examples of

q

-Gaussians for

q=0,0.5,

and

0.99

. We see for larger

q

the base densities (black curves) are less peaked and have larger support. We also show the first associated escort

2-q

-Gaussian densities (gray curves), which are slightly more peaked than their base densities. As

q\to 1

q

-Gaussians converge to Gaussians.

Figure 2: Synthetic logistic regression. Left: For

D\in\{10,50,200\}

and

q\in\{0.0,0.5,0.8,1\}

, we draw 8 Monte Carlo samples and compute the empirical per-coordinate gradient variance

\frac{1}{D}\sum_{j=1}^{D}\mbox{

\operatorname{Var}

}(\widehat{\nabla}F(\mbox{

\mbox{

\mathbf{w}

}

}^{\star})_{j})

, averaged over

50

independent repetitions; standard errors are all <.003. Right: Maximum radius of the

q

-Gaussian support against

D

Figure 3: Sharpness-Aware Minimization (SAM) [ 18 ] considers an adversarial perturbation over a compact ball. Variational stochastic gradient descent (VSGD) with Gaussian weight perturbation averages perturbations over whole space (unbounded support). Our proposed $q$ -VSGD uses $q$ -Gaussian weight perturbations, which have bounded support similarly to SAM but uses averages similarly to VSGD. The method combines the two complementary features of SAM and VSGD.

1 Monte-Carlo Sample
Method	Acc. (%) $\uparrow$	NLL $\downarrow$	ECE (%) $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$	Time $\downarrow$
SGD	92.0 $\pm$ 0.1	0.28 $\pm$ 0.00	3.84 $\pm$ 0.11	0.12 $\pm$ 0.00	92.1 $\pm$ 0.1	37s
IVON (1-MC) [ 29 ]	92.5 $\pm$ 0.1	0.26 $\pm$ 0.00	3.43 $\pm$ 0.09	0.12 $\pm$ 0.00	92.5 $\pm$ 0.1	43s
SAM [ 18 ]	92.6 $\pm$ 0.1	0.22 $\pm$ 0.00	1.56 $\pm$ 0.05	0.11 $\pm$ 0.00	92.5 $\pm$ 0.1	70s
$q$ -VSGD ( $q=0.0$ )	92.1 $\pm$ 0.0	0.25 $\pm$ 0.00	3.01 $\pm$ 0.05	0.12 $\pm$ 0.00	92.3 $\pm$ 0.0	44s
$q$ -VSGD ( $q=0.2$ )	92.1 $\pm$ 0.0	0.26 $\pm$ 0.00	2.99 $\pm$ 0.04	0.12 $\pm$ 0.00	92.3 $\pm$ 0.1	43s

Table 1: ResNet-20 on CIFAR-10 test performance. Mean

\pm

standard error over 10 seeds. Among the baselines (the first block at the top), SGD is the worst but fastest. SAM achieves the highest accuracy but is also slow due to two gradients needed in each iteration, while IVON performs reasonably well with a reasonable cost. In the second block, we show

q

-VSGD with 1 Monte-Carlo sample which achieves slightly better accuracy than its counterpart VSGD by increasing

q=0.6

, although it is comparable in other metrics. Going to 5-MC in the third block improves the performance but slows down the algorithm. The results are mixed and indicate more work is needed in improving performance via tweaking

q

Quantum Speedups for Stochastic Optimization with Heavy-Tailed Noise
Jul 28, 2026Bin Luo, Chengchang Liu, Jonathan Allcock +2Nonconvex Stochastic Optimization Heavy-Tailed Noise
The Chinese University of Hong Kong · Westlake University · Tencent Quantum Laboratory
Stability and Generalization of Straight-Through Estimators for Training Two-Layer Quantized Neural Networks
Sep 6, 2026Yiming YingStraight-Through Estimator Neural Network Generalization
School of Mathematics and Statistics University of Sydney
Stochastic Gradient Variational Inference with Price's Gradient Estimator from Bures-Wasserstein to Parameter Space
Feb 21, 2026Kyurae Kim, Qiang Fu, Yi-An Ma +2Variational Inference
University of Pennsylvania, Philadelphia, U.S. · Yale University, New Haven, U.S. · University of California San Diego, San Diego, U.S. +1

Jul 28, 2026J/K move · Enter open · S save

Quantum Speedups for Stochastic Optimization with Heavy-Tailed Noise

Bin Luo, Chengchang Liu, Jonathan Allcock, Shengyu Zhang+1

The Chinese University of Hong Kong · Westlake University · Tencent Quantum Laboratory

Nonconvex Stochastic Optimization Heavy-Tailed Noise Quantum Machine Learning Heavy-Tailed Stochastic Optimization

Open paper AskPDF arXiv

We study stochastic optimization with heavy-tailed gradient noise. We first propose a novel quantum mean estimator for multivariate heavy-tailed random variables that achieves lower query complexity than optimal classical estimators in the low-dimensional regime. We further develop an unbiased quantum mean estimator by applying a generalized multi-level Monte Carlo technique. We prove quantum lower bounds showing that, when the dimension

d

of the random vector is small and can be viewed as a constant, our quantum estimators are optimal up to logarithmic factors. We further derive stronger dimension-dependent lower bounds for tail index

p>4/3

, showing that a nontrivial dependence on the dimension is unavoidable in the low-dimensional regime. Based on these estimators, we propose a quantum normalized stochastic gradient descent method (

\texttt{QNSGD}

), which finds an

ε

-stationary point using

\tilde{\mathcal{O}}\big(\sqrt d\,ε^{-\frac{5p-4}{2p-2}}\big)

queries to the quantum stochastic gradient oracle. For a convex objective function, we propose a quantum projected stochastic gradient descent method (

\texttt{QPSGD}

), which computes a solution with

ε

-optimal solution using

\tilde{\mathcal{O}}\big(\sqrt d\,ε^{-\frac{3p-2}{2p-2}}+ε^{-2}\big)

queries in expectation. These sharper bounds improve upon the classical lower bounds

Ω\big(ε^{-\frac{3p-2}{p-1}}\big)

for nonconvex problems and

Ω\big(ε^{-\frac{p}{p-1}}\big)

for convex problems in the low-dimensional regimes

d\lesssimε^{-\frac{p}{p-1}}

and

d\lesssimε^{-\frac{2-p}{p-1}}

, respectively.

Abstract

Explore similar work

Quantum Speedups for Stochastic Optimization with Heavy-Tailed Noise

Stability and Generalization of Straight-Through Estimators for Training Two-Layer Quantized Neural Networks

Stochastic Gradient Variational Inference with Price's Gradient Estimator from Bures-Wasserstein to Parameter Space