Beyond ReinMax: Low-Variance Gradient Estimators for Discrete Latent Variables

Pathwise gradients are preferred for continuous random variables because they are unbiased, low variance, and work with a single sample. For discrete variables, however, the pathwise identity cannot generally be exact for every differentiable function. We propose a general framework to construct finite-order exact pathwise gradient estimators for a range of common discrete variables such as Poisson. The estimator is the least-norm solution among all solutions that are unbiased for polynomials of degree at most. The resulting estimators preserve the hard forward sample, require no temperature tuning, and can be implemented in a few lines of codes. Against other admissible solutions, our estimator is unique and minimizes weight variance; in contrast, prior works use categorical variables or augmented representations to approximate non-categorical variables that induces excess variance and computations. To understand approximation bias for functions beyond the prescribed class, we also derive a non-asymptotic bias bound. In experiments our low order methods match or improve tuned baselines across linear, nonlinear and hierarchical latent-variable models, while out-speeding competitors in every runtime benchmark.

Importance-weighted variational inference approximates intractable densities by optimizing multi-sample Monte Carlo objectives whose tightness increases with the number of importance samples $N$. When reparameterization is unavailable, optimization relies on score-function (REINFORCE) gradient estimators. In this paper, we provide a comprehensive analysis of score-function gradient estimators for importance-weighted objectives by characterizing their asymptotic expectation and variance. We establish that standard variational inference for Monte Carlo objectives (VIMCO) gradient estimators suffer from a vanishing signal-to-noise ratio (SNR) as $N$ increases, which arises from a suboptimal control variate choice and can impair optimization. To address this issue, we derive a general mechanism for constructing VIMCO gradient estimators that avoid SNR collapse. We identify the High-Effective Sample Size optimal variance importance sampling estimator as belonging to this category of estimators and establish its $\sqrt{N}$ SNR scaling. This estimator does not, however, have zero variance at optimality and achieving its $\sqrt{N}$ SNR rate may require prohibitively large $N$ when the importance weights are degenerate. We overcome these limitations by introducing VIMCO gradient estimators that attain zero variance at optimality and support importance weight tempering. Experiments with discrete latent-variable and state-space models validate the theoretical predictions and illustrate the benefits of our estimators.

For approximating a target distribution given only its unnormalized log-density, stochastic gradient-based variational inference (VI) algorithms are a popular approach. For example, Wasserstein VI (WVI) and black-box VI (BBVI) perform gradient descent in measure space (Bures-Wasserstein space) and parameter space, respectively. Previously, for the Gaussian variational family, convergence guarantees for WVI have shown superiority over existing results for black-box VI with the reparametrization gradient, suggesting the measure space approach might provide some unique benefits. In this work, however, we close this gap by obtaining identical state-of-the-art iteration complexity guarantees for both. In particular, we identify that WVI's superiority stems from the specific gradient estimator it uses, which BBVI can also leverage with minor modifications. The estimator in question is usually associated with Price's theorem and utilizes second-order information (Hessians) of the target log-density. We will refer to this as Price's gradient. On the flip side, WVI can be made more widely applicable by using the reparametrization gradient, which requires only gradients of the log-density. We empirically demonstrate that the use of Price's gradient is the major source of performance improvement.