Approximation of differential entropy in Bayesian optimal experimental design

Chuntao Chen; Tapio Helin; Nuutti Hyvönen; Yuya Suzuki

stat.MLOct 1, 2025

Approximation of differential entropy in Bayesian optimal experimental design

Authors: Chuntao Chen, Tapio Helin, Nuutti Hyvönen, Yuya Suzuki

Abstract

Bayesian optimal experimental design provides a principled framework for selecting experimental configurations that maximize obtained information. In this work, we consider settings where the likelihood entropy is independent of the design or can be evaluated explicitly, which applies, e.g. to additive noise models with design-independent or explicitly computable noise entropy. This reduces the problem to maximum entropy estimation, where the nested integral defining the expected information gain collapses into a single integral. Our study is motivated by large-scale inference problems, such as inverse problems, where the computational cost is dominated by expensive likelihood evaluations. Under additive Gaussian noise model, we propose a computational approach in which the evidence density is approximated by a Monte Carlo or quasi-Monte Carlo surrogate, while the differential entropy is evaluated using standard methods without additional likelihood evaluations. We prove that this strategy achieves convergence rates that are comparable to, or better than, state-of-the-art methods for full expected information gain estimation, particularly when the cost of entropy evaluation is negligible. Our approach relies only on mild smoothness of the forward map and avoids stronger technical assumptions required in earlier work. We also present numerical experiments, which confirm some of our theoretical findings.

Figures & tables

Method	Regime	Forward-solve cost
Nested MC	standard nested EIG	$\mathcal{O}(\mathrm{TOL}^{-3-\omega/\eta})$
Foster et al. [ 17 ]	variational / MC	$\mathcal{O}(\mathrm{TOL}^{-2-\omega/\eta})$
Kaarnioja–Schillings [ 35 ]	sparse tensor QMC, nested	$\mathcal{O}(\mathrm{TOL}^{-1-\omega/\eta})$
Bartuska et al. [ 3 ]	multilevel randomized QMC	$\mathcal{O}(\mathrm{TOL}^{-1-\omega/\eta})$
This work, MC	GMM surrogate	$\mathcal{O}(\mathrm{TOL}^{-2-\omega/\eta})$
This work, QMC lattice	GMM surrogate	$\mathcal{O}(\mathrm{TOL}^{-1-\omega/\eta-\varepsilon})$

Table 1: Simplified asymptotic forward-solve cost for achieving an RMSE tolerance

\mathrm{TOL}

. Here

\delta_{K}=O(K^{-\eta})

and the cost of evaluating

\mathcal{G}_{K}

O(K^{\omega})

. For the QMC results of this work,

\varepsilon>0

denotes an arbitrarily small loss in the convergence exponent. See the text for information on dimension-dependent constants, postprocessing costs, and method-specific assumptions.

\pi^{K}_{M}(y)=\frac{1}{\sqrt{(2\pi)^{d}|\Gamma|}}\,\frac{1}{M}\sum_{m=1}^{M}\exp\!\Big(-\frac{1}{2}\|z_{m}-y\|_{\Gamma}^{2}\Big);

Algorithm 1 A realization of the estimator ( 7 ) by MC or randomized QMC

Figure 1: The RMSEs and standard deviations as functions of

M

for the MC and randomized QMC estimators of the differential entropy

J^{K}

given in ( 62 ) for the linear model ( 61 ). For both methods, we choose large enough

N

so that the

M

-dependent terms dominate in ( 37 ) and ( 58 ).

Figure 2: The three optimal observation points (red dots) on top of the solution to ( 66 ) with one possible realization of $x$ . For comparison, the black dots depict the other possible measurement locations, cf. [ 35 ]

Figure 3: The RMSEs as functions of

M

for the MC and the two randomized QMC estimators in comparison to the reference differential entropy

\widetilde{J}^{K}_{\rm ref}

for the evidence of the nonlinear model ( 68 ). The employed QMC methods in the first part of Algorithm 1 are the randomized rank-1 lattice rule (first order method) and the randomized tent-transformed lattice rule (second order method). For all methods, we choose

N=2^{20}

, which suffices for the

M

-dependent terms to dominate in the estimation error (cf. ( 37 ) and ( 52 )). We consider three possible designs; the optimal design (left):

\{(0.50,0.50),(0.75,0.50),(0.50,0.75)\}

, the worst design (middle):

\{(0.25,0.25),(0.25,0.50),(0.25,0.75)\}

, and the 42nd best design (right):

\{(0.25,0.50),(0.75,0.25),(0.75,0.50)\}

. For the reference values, needed for computing the RMSEs, we used the tent-transformed lattice rule with

M_{0}=2^{13}

and the same generating vector

z

from [ 49 ] .

Figure 4: The RMSEs as functions of

M

for the MC and the two randomized QMC estimators in comparison to the reference differential entropy

\widetilde{J}^{K}_{\rm ref}

N=2^{20}

, which suffices for the

M

-dependent terms to dominate in the estimation error (cf. ( 37 ) and ( 52 )). We consider the optimal design

\{(0.50,0.50),(0.75,0.50),(0.50,0.75)\}

with three different combinations of the truncation index

K

in ( 67 ) and the mesh parameter

h

(K,h)=(50,1/64)

(50,1/128)

and

(100,1/128)

. For the reference values, needed for computing the RMSEs, we used the tent-transformed lattice rule with

M_{0}=2^{13}

and the same generating vector

z

from [ 49 ] .

Bayesian Experimental Design

Bayesian experimental design uses probabilistic models to choose measurements that improve learning or support better decisions.

See all 63 papers

Wasserstein Gradient Flows for Batch Bayesian Optimal Experimental Design
Mar 12, 2026Louis SharrockBayesian Experimental Design Wasserstein Gradient Flows
Department of Statistical Science, University College London.
Maximin Robust Bayesian Experimental Design
Mar 14, 2026Hany Abdulsamad, Sahel Iqbal, Christian A. Naesseth +2Bayesian Experimental Design Mutual Information
UvA–Bosch Delta Lab University of Amsterdam · Department of Statistics University of Oxford · School of Mathematics University of Edinburgh +1
Gauss--Hermite Quadrature for Gaussian-Mixture Entropy with an Action-Space Hermite Surrogate
Aug 21, 2026Jae Wan ShimNumerical Integration Surrogate-Assisted Optimization
Extreme Materials Research Center, Korea Institute of Science and Technology, 5 Hwarang-ro 14-gil, Seongbuk, Seoul, 02792, Republic of Korea · Climate and Environmental Research Institute, Korea Institute of Science and Technology, 5 Hwarang-ro 14-gil, Seongbuk, Seoul, 02792, Republic of Korea · Division of AI-Robotics, KIST Campus, University of Science and Technology, 5 Hwarang-ro 14-gil, Seongbuk, Seoul, 02792, Republic of Korea

Mar 12, 2026J/K move · Enter open · S save

Wasserstein Gradient Flows for Batch Bayesian Optimal Experimental Design

Louis Sharrock

Department of Statistical Science, University College London.

Bayesian Experimental Design Wasserstein Gradient Flows

Open paper AskPDF arXiv

Bayesian optimal experimental design (BOED) provides a powerful, decision-theoretic framework for selecting experiments so as to maximise the expected utility of the data to be collected. In practice, however, its applicability can be limited by the difficulty of optimising the chosen utility. The expected information gain (EIG), for example, is often high-dimensional and strongly non-convex. This challenge is particularly acute in the batch setting, where multiple experiments are to be designed simultaneously. In this paper, we introduce a new approach to batch EIG-based BOED via a probabilistic lifting of the original optimisation problem to the space of probability measures. In particular, we propose to optimise an entropic regularisation of the expected utility over the space of design measures. Under mild conditions, we show that this objective admits a unique minimiser, which can be explicitly characterised in the form of a Gibbs distribution. The resulting design law can be used directly as a randomised batch-design policy, or as a computational relaxation from which a deterministic batch is extracted. To obtain scalable approximations when the batch size is large, we then consider two tractable restrictions of the full batch distribution: a mean-field family, and an i.i.d. product family. For the i.i.d. objective, and formally for its mean-field extension, we derive the corresponding Wasserstein gradient flow, characterise its long-time behaviour, and obtain particle-based algorithms via space-time discretisations. We also introduce doubly stochastic variants that combine interacting particle updates with Monte Carlo estimators of the EIG gradient. Finally, we illustrate the performance of the proposed methods in several numerical experiments, demonstrating their ability to explore multimodal optimisation landscapes and obtain high-utility batches in challenging examples.

Abstract

Bayesian Experimental Design

Explore similar work

Wasserstein Gradient Flows for Batch Bayesian Optimal Experimental Design

Maximin Robust Bayesian Experimental Design

Gauss--Hermite Quadrature for Gaussian-Mixture Entropy with an Action-Space Hermite Surrogate