Repulsive Ensembles for Bayesian Inference in Physics-informed Neural Networks

Philipp Pilar; Markus Heinonen; Niklas Wahlström

stat.MLMay 22, 2025

Repulsive Ensembles for Bayesian Inference in Physics-informed Neural Networks

Authors: Philipp Pilar, Markus Heinonen, Niklas Wahlström

Organizations: Department of Information Technology Uppsala University, Uppsala, Sweden · Department of Computer Science Aalto University, Espoo, Finland

Abstract

Physics-informed neural networks (PINNs) have proven an effective tool for solving differential equations, in particular when considering the inverse problem. When inferring solutions and parameters of the differential equation from data, uncertainty estimates are preferable to point estimates, as they give an idea about the accuracy of the solution. In this work, we consider the inverse problem for a selection of basic differential equations and employ repulsive ensembles of PINNs (RE-PINN) to obtain such estimates. The repulsion is implemented by adding a particular repulsive term to the loss function, which has the property that the ensemble predictions correspond to the true Bayesian posterior in the limit of infinite ensemble members. Where available, we compare the ensemble predictions to Monte Carlo baselines. Whereas the standard ensemble tends to collapse to maximum-a-posteriori solutions, the repulsive ensemble produces significantly more accurate uncertainty estimates and exhibits higher sample diversity. The experimental results demonstrate that RE-PINN can accurately infer the Bayesian posterior in low-dimensional problems, establishing repulsive ensembles as a promising framework for uncertainty quantification in PINNs.

Figures & tables

Figure 1: Distribution of the repulsive ensemble when solving the exponential equation. The ensemble median is depicted, together with the 0.1-0.9 quantiles, and the 0.25-0.75 quantiles (shaded areas). The non-repulsive ensemble collapses to the MAP estimate. The dashed lines correspond to the MC baseline. The black dots depict training data. The ensembles have 50 members.

Method	MC	VI	ensemble	repulsive ensemble	other	Reference
B-PINN	✓	✓				Yang et al. (2021)
BPINN-VI		✓				Li et al. (2024)
MO-PINN	✓				✓	Yang and Foster (2022)
E-PINN			✓			Jiang et al. (2023)
Multi-solution PINN			✓			Zou et al. (2025)
EikonalNet			✓			Sahli Costabal et al. (2020)

Table 1: Overview of works that consider uncertainty quantification in PINNs. Green ticks indicate methods that constitute integral parts of the method described in the paper. Grey ticks indicate methods that were considered for comparison.

Repulsion	repulsive term
non-repulsive	$\log\rho(\boldsymbol{f},\lambda)\rightarrow 0$
repulsive ( $f$ )	$\log\rho(\boldsymbol{f},\lambda)\rightarrow\log\rho(\boldsymbol{f})$
repulsive ( $\lambda$ )	$\log\rho(\boldsymbol{f},\lambda)\rightarrow\log\rho(\lambda)$
repulsive ( $f,\lambda$ )	$\log\rho(\boldsymbol{f},\lambda)$
repulsive ( $f,\lambda$ ) - factorized	$\log\rho(\boldsymbol{f},\lambda)\approx\log\rho_{f}(\boldsymbol{f})+\log\rho_{\lambda}(\lambda)$
repulsive ( $f,\lambda$ ) - fully factorized	$\log\rho(\boldsymbol{f},\lambda)\approx\frac{1}{\sqrt{N_{d}}}\left(\sum_{j}\log\rho_{fj}(f_{j})+\sum_{l}\log\rho_{\lambda l}(\lambda_{l})\right)$

Table 2: Different variants of repulsive ensembles that we considered in our experiments. The column to the right gives the expression used for the repulsive term. The first three models denote variants that were implemented for the sake of comparison, where repulsion is either omitted or only takes place in a subspace. In the fourth model, the full joint density is kept in the repulsive term without approximation. The last two models correspond to different ways of approximating the full repulsive term by factorizing the density.

Table 4

Figure 3: The evaluation metrics when solving the exponential equation with repulsion in (

f

\lambda

)-space with different numbers of ensemble members. For better visibility, the non-repulsive ensemble has been omitted for logL/N (test), since the values are far below those of the repulsive ensembles.

Figure 4: Solving the damped harmonic oscillator ( 13 ) with repulsion in (

f

\lambda

)-space, using the repulsive ensemble with fully-factorized density. Left: The predictions in

f

-space are depicted. The light- and dark-shaded areas give the regions between the [0.1, 0.9]- and the [0.25, 0.75]-quantiles, respectively. The dashed lines depict the 0.1- and 0.9- quantiles as obtained via MCMC, together with the median. The non-repulsive ensemble effectively collapses to the MAP estimate. Black dots correspond to train data and red dots to test data. Middle: The predictions in

\lambda

-space are depicted. The ensemble distribution is compared to the distribution obtained via MCMC. Right: The individual ensemble members are depicted. The ensembles have 25 members.

Repulsion	RMSE (true)	logL/N (test)	$\|\lambda-\hat{\lambda}\|$	logL $\lambda$	W (f)	W $(\lambda)$
non-repulsive	0.059 $\pm$ 0.013	-1498.9 $\pm$ 822.7	0.016 $\pm$ 0.004	-2903.7 $\pm$ 2016.4	0.045 $\pm$ 0.004	0.024 $\pm$ 0.006
repulsive ( $f$ )	0.059 $\pm$ 0.014	-287.8 $\pm$ 234.2	0.016 $\pm$ 0.004	-65.5 $\pm$ 62.8	0.041 $\pm$ 0.005	0.022 $\pm$ 0.006
repulsive ( $\lambda$ )	0.062 $\pm$ 0.017	-17.3 $\pm$ 3.3	0.021 $\pm$ 0.007	4.7 $\pm$ 0.4	0.027 $\pm$ 0.004	0.010 $\pm$ 0.004
repulsive ( $f,\lambda$ )	0.060 $\pm$ 0.014	-860.5 $\pm$ 841.0	0.017 $\pm$ 0.004	-263.8 $\pm$ 247.6	0.043 $\pm$ 0.004	0.023 $\pm$ 0.006
repulsive ( $f,\lambda$ ) (factorized)	0.063 $\pm$ 0.017	-13.4 $\pm$ 1.1	0.021 $\pm$ 0.007	4.7 $\pm$ 0.4	0.026 $\pm$ 0.004	0.010 $\pm$ 0.004
repulsive ( $f,\lambda$ ) (fully factorized)	0.061 $\pm$ 0.016	-2.0 $\pm$ 1.5	0.020 $\pm$ 0.006	4.6 $\pm$ 0.3	0.025 $\pm$ 0.005	0.013 $\pm$ 0.003

Table 4: Evaluation metrics when solving the damped harmonic oscillator ( 13 ) with repulsion in (

f,\lambda

)-space, averaged over 5 runs with 25 ensemble members. The mean values plus-or-minus one standard deviation are given. The results for one of these runs are depicted in Figure 4 .

Figure 5: Example solution of the advection equation. Black dots correspond to training data and red dots to test data.

Figure 6: Solving the advection equation from ( 14 ) with repulsion in

(f,\lambda)

-space. The ensembles have 25 members. Red dots correspond to test data.

Repulsion	RMSE (true)	logL/N (test)	$\|\lambda-\hat{\lambda}\|$	logL $\lambda$
non-repulsive	0.125 $\pm$ 0.062	-85.8 $\pm$ 79.7	0.020 $\pm$ 0.030	-47.4 $\pm$ 84.6
repulsive (f)	0.122 $\pm$ 0.058	-36.3 $\pm$ 31.1	0.020 $\pm$ 0.030	-9.9 $\pm$ 22.4
repulsive ( $\lambda$ )	0.126 $\pm$ 0.064	-44.0 $\pm$ 33.1	0.020 $\pm$ 0.031	3.5 $\pm$ 0.2
repulsive (f and $\lambda$ )	0.123 $\pm$ 0.061	-33.6 $\pm$ 15.7	0.018 $\pm$ 0.029	-22.3 $\pm$ 45.1
repulsive (f and $\lambda$ ) (factorized)	0.121 $\pm$ 0.058	-40.3 $\pm$ 32.1	0.020 $\pm$ 0.031	3.5 $\pm$ 0.2
repulsive (f and $\lambda$ ) (fully factorized)	0.121 $\pm$ 0.057	-3.3 $\pm$ 3.1	0.020 $\pm$ 0.031	3.8 $\pm$ 0.6

Table 5: Evaluation metrics when solving the advection equation ( 14 ) with repulsion in (

f,\lambda

)-space, averaged over 5 runs with 25 ensemble members. The mean values plus-or-minus one standard deviation are given. The results for one of these runs are depicted in Figure 6 .

Figure 7: Solving the damped harmonic oscillator ( 13 ) with variational inference. Left: The predictions in

f

-space are depicted. The light- and dark-shaded areas give the regions between the [0.1, 0.9]- and the [0.25, 0.75]-quantiles, respectively. The dashed lines depict the 0.1- and 0.9-quantiles as obtained via MCMC, together with the median. Black dots correspond to train data and red dots to test data. Middle: The predictions in

\lambda

-space are depicted. The VI distribution is compared to the distribution obtained via MCMC. Right: Samples from the VI posterior are depicted.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	exponential equation	damped HO	advection	Lotka-Volterra
non-repulsive	0.073 $\pm$ 0.023	0.011 $\pm$ 0.002	0.055 $\pm$ 0.007	0.330 $\pm$ 0.021
repulsive (f)	0.135 $\pm$ 0.017	0.013 $\pm$ 0.002	0.055 $\pm$ 0.007	0.359 $\pm$ 0.061
repulsive ( $\lambda$ )	0.089 $\pm$ 0.019	0.014 $\pm$ 0.002	0.062 $\pm$ 0.006	0.313 $\pm$ 0.035
repulsive (f and $\lambda$ )	0.116 $\pm$ 0.016	0.111 $\pm$ 0.002	0.055 $\pm$ 0.008	0.335 $\pm$ 0.067
repulsive (f and $\lambda$ ) (factorized)	0.123 $\pm$ 0.017	0.013 $\pm$ 0.002	0.062 $\pm$ 0.006	0.359 $\pm$ 0.054
repulsive (f and $\lambda$ ) (fully factorized)	0.080 $\pm$ 0.021	0.012 $\pm$ 0.002	0.054 $\pm$ 0.007	0.399 $\pm$ 0.076

Appendix

Table 6: Adherence of individual posterior samples to the DE, as measured by the mean DE residuals

\frac{1}{N_{z}}\sum_{i=1}^{N_{z}}2\sigma_{\mathcal{F}}^{2}\mathcal{L}_{\mathcal{F}}^{i}

. Repulsion does not significantly impact the quality of individual samples. For samples from the VI posterior, the DE fulfillment is adversely affected.

	non-repulsive	repulsive
exponential equation ( 5.1 )	0.35	0.43
damped harmonic oscillator ( 5.2 )	0.69	0.78
advection equation ( 5.3 )	0.39	0.46
Lotka-Volterra ( E.6 )	0.69	0.78

Appendix

Table 7: Runtime (in hours) for one run contributing to the entries in Tables 3 - 8 , where

N_{e}=25

Figure 8: The ensemble members corresponding to the distributions in Figure 2 are depicted.

Figure 9: Solving the damped harmonic oscillator ( 13 ) with repulsion in (

f

\lambda

)-space. Top row: The predictions in

f

-space are depicted. The light- and dark-shaded areas give the regions between the [0.1, 0.9]- and the [0.25, 0.75]-quantiles, respectively. The dashed lines depict the 0.1- and 0.9-quantiles as obtained via MCMC, together with the median. Black dots correspond to train data and red dots to test data. Bottom row: The predictions in

\lambda

-space are depicted. The ensemble distribution is compared to the distribution obtained via MCMC. The corresponding evaluation metrics are given in Table 4 . The ensembles have 25 members.

Figure 10: A subset of the ensemble members corresponding to the distributions in Figure 6 is depicted.

Table 17

Uncertainty Quantification

Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.

See all 663 papers

Apr 18, 2026

Uncertainty Quantification in PINNs for Turbulent Flows: Bayesian Inference and Repulsive Ensembles

Physics-informed neural networks (PINNs) have emerged as a promising framework for solving inverse problems governed by partial differential equations (PDEs), including the reconstruction of turbulent flow fields from sparse data. However, most existing PINN formulations are deterministic and do not provide reliable quantification of epistemic uncertainty, which is critical for ill-posed problems such as data-driven Reynolds-averaged Navier-Stokes (RANS) modeling. In this work, we develop and systematically evaluate a set of probabilistic extensions of PINNs for uncertainty quantification in turbulence modeling. The proposed framework combines (i) Bayesian PINNs with Hamiltonian Monte Carlo sampling and a tempered multi-component likelihood, (ii) Monte Carlo dropout, and (iii) repulsive deep ensembles that enforce diversity in function space. Particular emphasis is placed on the role of ensemble diversity and likelihood tempering in improving uncertainty calibration for PDE-constrained inverse problems. The methods are assessed on a hierarchy of test cases, including the Van der Pol oscillator and turbulent flow past a circular cylinder at Reynolds numbers Re=3,900 (direct numerical simulation data) and Re = 10,000 (experimental particle image velocimetry data). The results demonstrate that Bayesian PINNs provide the most consistent uncertainty estimates across all inferred quantities, while function-space repulsive ensembles offer a computationally efficient approximation with competitive accuracy for primary flow variables. These findings provide quantitative insight into the trade-offs between accuracy, computational cost, and uncertainty calibration in physics-informed learning, and offer practical guidance for uncertainty quantification in data-driven turbulence modeling.

Khemraj Shukla, Zongren Zou, Theo Kaeufer +2

Division of Applied Mathematics, Brown University, Providence, RI, USA · Department of Computing and Mathematical Sciences, California Institute of Technology, Pasadena, CA, USA · Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA, USA

Uncertainty Quantification Deep Ensembles Physics-Informed ML Uncertainty Calibration PDE Inverse Problems Computational Fluid Dynamics

Ask PDF

Oct 8, 2026

Uncovering and Fixing Collider Bias in Bayesian PINNs

Bayesian physics-informed neural networks (B-PINNs) are a popular framework for parameter and state inference from sparse or noisy observations. They are commonly formulated via a collider structure, in which physical and trajectory parameters are assumed to be a priori independent and become coupled through virtual likelihoods on differential-equation residuals that enforce physical consistency. We show that this modeling choice can induce severe systematic bias in the posterior over physical parameters: even when the prior is favorably centered on the ground-truth parameters, the resulting posterior can drift away and concentrate far from them. As a remedy, we advocate a hierarchical chain model in which physics generates trajectories, which in turn generate observations. The chain model does not suffer from this posterior bias, but it poses a harder, so-called doubly intractable, inference problem due to a physics-dependent normalization constant. This challenge can be resolved by discretizing the underlying stochastic dynamics, after which the chain posterior can be sampled exactly with particle MCMC. We identify two distinct mechanisms characterizing the collider bias, derive analytical approximations of their magnitudes, and establish diagnostic criteria for predicting when standard B-PINNs remain reliable. Experiments confirm the predicted bias and show that the chain formulation successfully avoids it.

Michael Obermayr, Robert Peharz

TU Graz

Physics-Informed ML Bayesian Inverse Problems Hierarchical Bayesian Modeling

Ask PDF

May 8, 2026

Functional-prior-based approaches to Bayesian PDE-constrained inversion using physics-informed neural networks

Physics-informed neural networks (PINNs) provide a mesh-free framework for solving PDE-constrained inverse problems, but their extension to Bayesian inversion still faces a fundamental difficulty: prior distributions are typically defined in the weight space of neural networks, whereas physically meaningful prior assumptions are more naturally expressed in function space. In this study, we introduce a unified framework, termed functional-prior-based approaches to Bayesian PDE-constrained inversion using physics-informed neural networks (fpBPINN), to incorporate functional priors into Bayesian PINN-based inversion. We consider two complementary approaches. The first is a functional-prior-informed Bayesian PINN (FPI-BPINN), in which a neural network weight prior is learned to be consistent with a prescribed functional prior, and Bayesian inference is subsequently performed in weight space. The second is function-space particle-based variational inference for PINNs (fParVI-PINN), which performs Bayesian estimation using ParVI directly in function space. We also show that random Fourier features (RFF) play an important role in representing Gaussian functional priors with neural networks and in improving posterior approximation. We applied the proposed approaches to one-dimensional seismic traveltime tomography and two-dimensional Darcy-flow permeability inversion. These numerical experiments showed that both approaches accurately estimated posterior distributions, highlighting the significance of introducing physically interpretable functional priors into Bayesian PINN-based inverse problems. We also identified the contrasting advantages of FPI-BPINN and fParVI-PINN, namely flexibility and accuracy, respectively.

Ryoichiro Agata, Tomohisa Okazaki

Japan Agency for Marine-Earth Science and Technology, Japan · Disaster Prevention Research Institute, Kyoto University, Japan · RIKEN Center for Advanced Intelligence Project, Japan

Bayesian Inverse Problems Physics-Informed ML Variational Inference PDE Inverse Problems

Ask PDF

Abstract

Uncertainty Quantification

Explore similar work

Uncertainty Quantification in PINNs for Turbulent Flows: Bayesian Inference and Repulsive Ensembles

Uncovering and Fixing Collider Bias in Bayesian PINNs

Functional-prior-based approaches to Bayesian PDE-constrained inversion using physics-informed neural networks