Shrinkage prior for implicit generative models

Shreya Sinha-Roy; Christian P. Robert; Ritabrata Dutta

cs.LGNov 3, 2026

Shrinkage prior for implicit generative models

Authors: Shreya Sinha-Roy, Christian P. Robert, Ritabrata Dutta

Organizations: Department of Statistics, University of Warwick, Coventry, CV4 7AL, United Kingdom · CEREMADE, Université Paris Dauphine PSL, France

Abstract

Implicit generative models based on deep neural networks have revolutionized many scientific fields, from weather research to language generation. They successfully model complex random objects arising in real-world problems. In this work, we propose a sample-efficient training mechanism for these models. Our approach leverages regularized inference with shrinkage priors and generalized Bayes. In the large $p$ (parameter dimension) and small $n$ (sample size) setting, sparsity-inducing priors have been successfully used for linear regression when $p \gg n$ . However, their application to implicit generative models remains challenging due to the intractability of the likelihood function. To address this, we adapt a scoring rule posterior based on a strictly proper scoring rule, following the generalized Bayes framework. We then propose a block SGMCMC within Gibbs sampling mechanism. This enables efficient exploration of high-dimensional parameter spaces by identifying a sparse representation of neural implicit generative models. We demonstrate the performance of our proposed method through experiments on $p \gg n$ linear regression and an application of neural generative models.

Figures & tables

Figure 1: Exact versus approximate sampler for sparse linear regression. Log likelihood of the samples obtained from the exact (in pink) and the approximate (in blue) sampler versus runtime plot. The solid line represents the mean performance over $10$ independent runs, and the $90\ \%$ confidence intervals are shown in a lighter shade.

Method	$p=500$	$p=1000$	$p=3000$
Ridge	$-30.79\ (45.52)$	$-36.62\ (65.32)$	$-127.20\ (165.12)$
LASSO	$-34.70\ (36.70)$	$-98.18\ (80.73)$	$-111.54\ (116.90)$
Elastic Net	$-34.34\ (37.00)$	$-88.61\ (84.56)$	$-92.38\ (124.10)$
Horseshoe (exact Gibbs)	$\mathbf{-1.616}\ (0.001)$	$-1.830\ (0.032)$	$-23.290\ (1.169)$
Horseshoe (approximate, ours)	$-2.805\ (1.486)$	$\mathbf{-1.626}\ (0.035)$	$\mathbf{-1.612}\ (2\times 10^{-5})$

Table 1: Mean log-likelihood (standard deviation ) across 10 simulated datasets for the linear regression example. For the first three methods, the log-likelihood at the estimated parameter value is computed. For the Bayesian methods, the mean log-likelihood is computed over posterior samples after discarding the initial burn-in period. Higher values indicate better predictive performance.

	$p=500$	$p=1000$	$p=3000$
Exact Sampler	0.0987 (0.0019)	0.6736 (0.0218)	21.9139 (1.8124)
Approximate Sampler	0.0309 (0.0025)	0.0310 (0.0015)	0.0345 (0.0005)

Table 2: Mean runtime (in seconds) per iteration across 10 seeds for exact and approximate samplers for linear regression. Standard deviation over the 10 runs are shown in parentheses.

Figure 2: Predictive performance on the Hopper environment. We report the normalized root mean squared error (NRMSE), coefficient of determination $R^{2}$ , and average energy score on a test set of length $1000$ . Results are shown for the horseshoe prior (blue), standard Gaussian prior (yellow), Student- $t$ prior with 10 degrees of freedom (green), and an ensemble model initialized from the horseshoe prior and optimized using Adam (red). For visual clarity, the negative adjusted $R^{2}$ is plotted.

$n$	NRMSE	$R^{2}$	Energy score
50	16998.45 (3058.44)	-2.31e10 (8.50e9)	78153.29 (16198.17)
100	17301.19 (2740.90)	-2.58e10 (6.68e9)	80490.82 (13381.43)
500	16442.49 (3102.66)	-2.74e10 (1.05e10)	63382.63 (9684.82)
1000	18561.19 (2940.66)	-3.27e10 (1.11e10)	70151.97 (12845.99)

Table 3: Predictive performance of the posterior samples under a standard Gaussian prior for different training set sizes

(n)

. The Markov chain is run for

10000

iterations, and the last

1000

samples are used to compute the predictive metrics. Results are reported as mean (standard deviation) over five independent runs.

$n$	NRMSE	$R^{2}$	Energy score
50	0.212 (0.070)	-3.070 (3.260)	3.001 (0.079)
100	0.215 (0.074)	-3.120 (3.750)	2.984 (0.065)
500	0.152 (0.073)	-1.550 (3.200)	2.934 (0.046)
1000	0.115 (0.055)	-0.470 (1.890)	2.914 (0.031)

Table 4: Predictive performance of the posterior samples under the horseshoe prior with

b_{0}=b_{g}=100

. The Markov chain is run for

10000

iterations, and the last

1000

samples are used to compute the predictive metrics. Results are reported as mean (standard deviation) over five independent runs.

Method	$n=50$	$n=100$	$n=500$	$n=1000$
Gaussian prior	$0.062\ (0.002)$	$0.105\ (0.006)$	$0.487\ (0.022)$	$1.077\ (0.006)$
Horseshoe prior	$0.118\ (0.016)$	$0.153\ (0.012)$	$0.517\ (0.028)$	$1.208\ (0.050)$

Table 5: Average runtime (seconds) per iteration for the Gaussian and horseshoe priors. Values are reported as mean (standard deviation) over repeated runs.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Exact and approximate horseshoe sampler for linear regression with fixed $n=50$ . We have considered 3 pairs of different values of the hyperparameters ( $b_{0},b_{g}$ ) to define the horseshoe prior.

$n$	NRMSE	$R^{2}$	Energy score
$\mathbf{b_{0},b_{g}=1/p}$
50	0.543 (0.155)	-27.810 (23.520)	3.295 (0.155)
100	0.626 (0.157)	-38.750 (28.170)	3.407 (0.172)
500	0.674 (0.189)	-45.410 (30.210)	3.470 (0.150)
1000	0.651 (0.219)	-46.080 (31.280)	3.431 (0.148)
$\mathbf{b_{0},b_{g}=0.01}$

Appendix

Table 6: Predictive performance across different values of the horseshoe hyperparameters

\{b_{0},b_{g}\}

and observation sizes

n

for the Hopper experiment. Values are reported as mean (standard deviation) across independent runs.

Generative Modeling

Generative modeling learns data distributions to create new samples, from images and text to molecules, simulations, and structured records.

See all 519 papers

Controlling Dependence in Implicit Generative Models via Spread Mutual Information
Oct 7, 2026Jiahao Yu, Song Liu, José Miguel Hernández-Lobato +1Generative Modeling Mutual Information Estimation
University of Oxford · Work completed at the University of Cambridge. · University of Bristol +1
Random Process Flow Matching: Generative Implicit Representations of Multivariate Random Fields
May 27, 2026Julien Lalanne, David Picard, Lionel Boillot +3Flow Matching Uncertainty Quantification
Navier, CNRS, Univ Gustave Eiffel, ENPC, IP Paris, France · TotalEnergies, OneTech, France · LIGM, CNRS, Univ Gustave Eiffel, ENPC, IP Paris, France
Learning the Universe: Posterior Reliability of Neural Generative Models in High-Dimensional Field-Level Inference of Cosmic Initial Conditions
Jun 8, 2026Ludvig Doeser, Jens JascheSimulation-Based Inference Normalizing Flows
The Oskar Klein Centre, Department of Physics, Stockholm University, AlbaNova University Centre, SE 106 91 Stockholm, Sweden · Center for Computational Astrophysics, Flatiron Institute, 162 5th Avenue, New York, NY 10010, USA

Oct 7, 2026J/K move · Enter open · S save

Controlling Dependence in Implicit Generative Models via Spread Mutual Information

Jiahao Yu, Song Liu, José Miguel Hernández-Lobato, RuiKang OuYang

University of Oxford · Work completed at the University of Cambridge. · University of Bristol+1

Generative Modeling Mutual Information Estimation Mutual Information Maximization

Open paper AskPDF arXiv

Mutual information (MI) provides an objective for suppressing or encouraging statistical dependence in implicit generative models. However, direct MI evaluation is challenging in implicit models due to typically intractable densities. A remedy is estimating the generator gradient from the difference between conditional and marginal scores. This score difference can, in turn, be estimated by differentiating a log density ratio learned through classification. This construction nevertheless faces two difficulties: (i) singular distributions need not admit the required score functions, and (ii) poor overlap can hinder density-ratio estimation. We therefore introduce Spread Mutual Information (SMI), a weighted integral of MI across noise levels obtained by applying a common spreading kernel to the generated variable. Gaussian spreading yields smooth, strictly positive conditional and marginal densities, extending the gradient construction to distributions that may originally be singular. Across a variaty of experiments, SMI consistently achieves effective dependence control among MI-based methods and remains competitive with established task-specific approaches.

Abstract

Generative Modeling

Explore similar work

Controlling Dependence in Implicit Generative Models via Spread Mutual Information

Random Process Flow Matching: Generative Implicit Representations of Multivariate Random Fields

Learning the Universe: Posterior Reliability of Neural Generative Models in High-Dimensional Field-Level Inference of Cosmic Initial Conditions