Repulsive Ensembles for Bayesian Inference in Physics-informed Neural Networks

Philipp Pilar; Markus Heinonen; Niklas Wahlström

stat.MLMay 22, 2025

Repulsive Ensembles for Bayesian Inference in Physics-informed Neural Networks

Authors: Philipp Pilar, Markus Heinonen, Niklas Wahlström

Organizations: Department of Information Technology Uppsala University, Uppsala, Sweden · Department of Computer Science Aalto University, Espoo, Finland

Abstract

Physics-informed neural networks (PINNs) have proven an effective tool for solving differential equations, in particular when considering the inverse problem. When inferring solutions and parameters of the differential equation from data, uncertainty estimates are preferable to point estimates, as they give an idea about the accuracy of the solution. In this work, we consider the inverse problem for a selection of basic differential equations and employ repulsive ensembles of PINNs (RE-PINN) to obtain such estimates. The repulsion is implemented by adding a particular repulsive term to the loss function, which has the property that the ensemble predictions correspond to the true Bayesian posterior in the limit of infinite ensemble members. Where available, we compare the ensemble predictions to Monte Carlo baselines. Whereas the standard ensemble tends to collapse to maximum-a-posteriori solutions, the repulsive ensemble produces significantly more accurate uncertainty estimates and exhibits higher sample diversity. The experimental results demonstrate that RE-PINN can accurately infer the Bayesian posterior in low-dimensional problems, establishing repulsive ensembles as a promising framework for uncertainty quantification in PINNs.

Figures & tables

Figure 1: Distribution of the repulsive ensemble when solving the exponential equation. The ensemble median is depicted, together with the 0.1-0.9 quantiles, and the 0.25-0.75 quantiles (shaded areas). The non-repulsive ensemble collapses to the MAP estimate. The dashed lines correspond to the MC baseline. The black dots depict training data. The ensembles have 50 members.

Method	MC	VI	ensemble	repulsive ensemble	other	Reference
B-PINN	✓	✓				Yang et al. (2021)
BPINN-VI		✓				Li et al. (2024)
MO-PINN	✓				✓	Yang and Foster (2022)
E-PINN			✓			Jiang et al. (2023)
Multi-solution PINN			✓			Zou et al. (2025)
EikonalNet			✓			Sahli Costabal et al. (2020)

Table 1: Overview of works that consider uncertainty quantification in PINNs. Green ticks indicate methods that constitute integral parts of the method described in the paper. Grey ticks indicate methods that were considered for comparison.

Repulsion	repulsive term
non-repulsive	$\log\rho(\boldsymbol{f},\lambda)\rightarrow 0$
repulsive ( $f$ )	$\log\rho(\boldsymbol{f},\lambda)\rightarrow\log\rho(\boldsymbol{f})$
repulsive ( $\lambda$ )	$\log\rho(\boldsymbol{f},\lambda)\rightarrow\log\rho(\lambda)$
repulsive ( $f,\lambda$ )	$\log\rho(\boldsymbol{f},\lambda)$
repulsive ( $f,\lambda$ ) - factorized	$\log\rho(\boldsymbol{f},\lambda)\approx\log\rho_{f}(\boldsymbol{f})+\log\rho_{\lambda}(\lambda)$
repulsive ( $f,\lambda$ ) - fully factorized	$\log\rho(\boldsymbol{f},\lambda)\approx\frac{1}{\sqrt{N_{d}}}\left(\sum_{j}\log\rho_{fj}(f_{j})+\sum_{l}\log\rho_{\lambda l}(\lambda_{l})\right)$

Table 2: Different variants of repulsive ensembles that we considered in our experiments. The column to the right gives the expression used for the repulsive term. The first three models denote variants that were implemented for the sake of comparison, where repulsion is either omitted or only takes place in a subspace. In the fourth model, the full joint density is kept in the repulsive term without approximation. The last two models correspond to different ways of approximating the full repulsive term by factorizing the density.

Table 4

Figure 3: The evaluation metrics when solving the exponential equation with repulsion in (

f

\lambda

)-space with different numbers of ensemble members. For better visibility, the non-repulsive ensemble has been omitted for logL/N (test), since the values are far below those of the repulsive ensembles.

Figure 4: Solving the damped harmonic oscillator ( 13 ) with repulsion in (

f

\lambda

)-space, using the repulsive ensemble with fully-factorized density. Left: The predictions in

f

-space are depicted. The light- and dark-shaded areas give the regions between the [0.1, 0.9]- and the [0.25, 0.75]-quantiles, respectively. The dashed lines depict the 0.1- and 0.9- quantiles as obtained via MCMC, together with the median. The non-repulsive ensemble effectively collapses to the MAP estimate. Black dots correspond to train data and red dots to test data. Middle: The predictions in

\lambda

-space are depicted. The ensemble distribution is compared to the distribution obtained via MCMC. Right: The individual ensemble members are depicted. The ensembles have 25 members.

Repulsion	RMSE (true)	logL/N (test)	$\|\lambda-\hat{\lambda}\|$	logL $\lambda$	W (f)	W $(\lambda)$
non-repulsive	0.059 $\pm$ 0.013	-1498.9 $\pm$ 822.7	0.016 $\pm$ 0.004	-2903.7 $\pm$ 2016.4	0.045 $\pm$ 0.004	0.024 $\pm$ 0.006
repulsive ( $f$ )	0.059 $\pm$ 0.014	-287.8 $\pm$ 234.2	0.016 $\pm$ 0.004	-65.5 $\pm$ 62.8	0.041 $\pm$ 0.005	0.022 $\pm$ 0.006
repulsive ( $\lambda$ )	0.062 $\pm$ 0.017	-17.3 $\pm$ 3.3	0.021 $\pm$ 0.007	4.7 $\pm$ 0.4	0.027 $\pm$ 0.004	0.010 $\pm$ 0.004
repulsive ( $f,\lambda$ )	0.060 $\pm$ 0.014	-860.5 $\pm$ 841.0	0.017 $\pm$ 0.004	-263.8 $\pm$ 247.6	0.043 $\pm$ 0.004	0.023 $\pm$ 0.006
repulsive ( $f,\lambda$ ) (factorized)	0.063 $\pm$ 0.017	-13.4 $\pm$ 1.1	0.021 $\pm$ 0.007	4.7 $\pm$ 0.4	0.026 $\pm$ 0.004	0.010 $\pm$ 0.004
repulsive ( $f,\lambda$ ) (fully factorized)	0.061 $\pm$ 0.016	-2.0 $\pm$ 1.5	0.020 $\pm$ 0.006	4.6 $\pm$ 0.3	0.025 $\pm$ 0.005	0.013 $\pm$ 0.003

Table 4: Evaluation metrics when solving the damped harmonic oscillator ( 13 ) with repulsion in (

f,\lambda

)-space, averaged over 5 runs with 25 ensemble members. The mean values plus-or-minus one standard deviation are given. The results for one of these runs are depicted in Figure 4 .

Figure 5: Example solution of the advection equation. Black dots correspond to training data and red dots to test data.

Figure 6: Solving the advection equation from ( 14 ) with repulsion in

(f,\lambda)

-space. The ensembles have 25 members. Red dots correspond to test data.

Repulsion	RMSE (true)	logL/N (test)	$\|\lambda-\hat{\lambda}\|$	logL $\lambda$
non-repulsive	0.125 $\pm$ 0.062	-85.8 $\pm$ 79.7	0.020 $\pm$ 0.030	-47.4 $\pm$ 84.6
repulsive (f)	0.122 $\pm$ 0.058	-36.3 $\pm$ 31.1	0.020 $\pm$ 0.030	-9.9 $\pm$ 22.4
repulsive ( $\lambda$ )	0.126 $\pm$ 0.064	-44.0 $\pm$ 33.1	0.020 $\pm$ 0.031	3.5 $\pm$ 0.2
repulsive (f and $\lambda$ )	0.123 $\pm$ 0.061	-33.6 $\pm$ 15.7	0.018 $\pm$ 0.029	-22.3 $\pm$ 45.1
repulsive (f and $\lambda$ ) (factorized)	0.121 $\pm$ 0.058	-40.3 $\pm$ 32.1	0.020 $\pm$ 0.031	3.5 $\pm$ 0.2
repulsive (f and $\lambda$ ) (fully factorized)	0.121 $\pm$ 0.057	-3.3 $\pm$ 3.1	0.020 $\pm$ 0.031	3.8 $\pm$ 0.6

Table 5: Evaluation metrics when solving the advection equation ( 14 ) with repulsion in (

f,\lambda

)-space, averaged over 5 runs with 25 ensemble members. The mean values plus-or-minus one standard deviation are given. The results for one of these runs are depicted in Figure 6 .

Figure 7: Solving the damped harmonic oscillator ( 13 ) with variational inference. Left: The predictions in

f

-space are depicted. The light- and dark-shaded areas give the regions between the [0.1, 0.9]- and the [0.25, 0.75]-quantiles, respectively. The dashed lines depict the 0.1- and 0.9-quantiles as obtained via MCMC, together with the median. Black dots correspond to train data and red dots to test data. Middle: The predictions in

\lambda

-space are depicted. The VI distribution is compared to the distribution obtained via MCMC. Right: Samples from the VI posterior are depicted.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	exponential equation	damped HO	advection	Lotka-Volterra
non-repulsive	0.073 $\pm$ 0.023	0.011 $\pm$ 0.002	0.055 $\pm$ 0.007	0.330 $\pm$ 0.021
repulsive (f)	0.135 $\pm$ 0.017	0.013 $\pm$ 0.002	0.055 $\pm$ 0.007	0.359 $\pm$ 0.061
repulsive ( $\lambda$ )	0.089 $\pm$ 0.019	0.014 $\pm$ 0.002	0.062 $\pm$ 0.006	0.313 $\pm$ 0.035
repulsive (f and $\lambda$ )	0.116 $\pm$ 0.016	0.111 $\pm$ 0.002	0.055 $\pm$ 0.008	0.335 $\pm$ 0.067
repulsive (f and $\lambda$ ) (factorized)	0.123 $\pm$ 0.017	0.013 $\pm$ 0.002	0.062 $\pm$ 0.006	0.359 $\pm$ 0.054
repulsive (f and $\lambda$ ) (fully factorized)	0.080 $\pm$ 0.021	0.012 $\pm$ 0.002	0.054 $\pm$ 0.007	0.399 $\pm$ 0.076

Appendix

Table 6: Adherence of individual posterior samples to the DE, as measured by the mean DE residuals

\frac{1}{N_{z}}\sum_{i=1}^{N_{z}}2\sigma_{\mathcal{F}}^{2}\mathcal{L}_{\mathcal{F}}^{i}

. Repulsion does not significantly impact the quality of individual samples. For samples from the VI posterior, the DE fulfillment is adversely affected.

	non-repulsive	repulsive
exponential equation ( 5.1 )	0.35	0.43
damped harmonic oscillator ( 5.2 )	0.69	0.78
advection equation ( 5.3 )	0.39	0.46
Lotka-Volterra ( E.6 )	0.69	0.78

Appendix

Table 7: Runtime (in hours) for one run contributing to the entries in Tables 3 - 8 , where

N_{e}=25

Figure 8: The ensemble members corresponding to the distributions in Figure 2 are depicted.

Figure 9: Solving the damped harmonic oscillator ( 13 ) with repulsion in (

f

\lambda

)-space. Top row: The predictions in

f

-space are depicted. The light- and dark-shaded areas give the regions between the [0.1, 0.9]- and the [0.25, 0.75]-quantiles, respectively. The dashed lines depict the 0.1- and 0.9-quantiles as obtained via MCMC, together with the median. Black dots correspond to train data and red dots to test data. Bottom row: The predictions in

\lambda

-space are depicted. The ensemble distribution is compared to the distribution obtained via MCMC. The corresponding evaluation metrics are given in Table 4 . The ensembles have 25 members.

Figure 10: A subset of the ensemble members corresponding to the distributions in Figure 6 is depicted.

Table 17

Uncertainty Quantification

Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.

See all 663 papers

Uncertainty Quantification in PINNs for Turbulent Flows: Bayesian Inference and Repulsive Ensembles
Apr 18, 2026Khemraj Shukla, Zongren Zou, Theo Kaeufer +2Uncertainty Quantification Deep Ensembles
Division of Applied Mathematics, Brown University, Providence, RI, USA · Department of Computing and Mathematical Sciences, California Institute of Technology, Pasadena, CA, USA · Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA, USA
Uncovering and Fixing Collider Bias in Bayesian PINNs
Oct 8, 2026Michael Obermayr, Robert PeharzPhysics-Informed ML Bayesian Inverse Problems
TU Graz
Functional-prior-based approaches to Bayesian PDE-constrained inversion using physics-informed neural networks
May 8, 2026Ryoichiro Agata, Tomohisa OkazakiBayesian Inverse Problems Physics-Informed ML
Japan Agency for Marine-Earth Science and Technology, Japan · Disaster Prevention Research Institute, Kyoto University, Japan · RIKEN Center for Advanced Intelligence Project, Japan

Apr 18, 2026J/K move · Enter open · S save

Uncertainty Quantification in PINNs for Turbulent Flows: Bayesian Inference and Repulsive Ensembles

Khemraj Shukla, Zongren Zou, Theo Kaeufer, Michael Triantafyllou+1

Division of Applied Mathematics, Brown University, Providence, RI, USA · Department of Computing and Mathematical Sciences, California Institute of Technology, Pasadena, CA, USA · Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA, USA

Uncertainty Quantification Deep Ensembles Physics-Informed ML Uncertainty Calibration PDE Inverse Problems Computational Fluid Dynamics

Open paper AskPDF arXiv

Physics-informed neural networks (PINNs) have emerged as a promising framework for solving inverse problems governed by partial differential equations (PDEs), including the reconstruction of turbulent flow fields from sparse data. However, most existing PINN formulations are deterministic and do not provide reliable quantification of epistemic uncertainty, which is critical for ill-posed problems such as data-driven Reynolds-averaged Navier-Stokes (RANS) modeling. In this work, we develop and systematically evaluate a set of probabilistic extensions of PINNs for uncertainty quantification in turbulence modeling. The proposed framework combines (i) Bayesian PINNs with Hamiltonian Monte Carlo sampling and a tempered multi-component likelihood, (ii) Monte Carlo dropout, and (iii) repulsive deep ensembles that enforce diversity in function space. Particular emphasis is placed on the role of ensemble diversity and likelihood tempering in improving uncertainty calibration for PDE-constrained inverse problems. The methods are assessed on a hierarchy of test cases, including the Van der Pol oscillator and turbulent flow past a circular cylinder at Reynolds numbers Re=3,900 (direct numerical simulation data) and Re = 10,000 (experimental particle image velocimetry data). The results demonstrate that Bayesian PINNs provide the most consistent uncertainty estimates across all inferred quantities, while function-space repulsive ensembles offer a computationally efficient approximation with competitive accuracy for primary flow variables. These findings provide quantitative insight into the trade-offs between accuracy, computational cost, and uncertainty calibration in physics-informed learning, and offer practical guidance for uncertainty quantification in data-driven turbulence modeling.

Abstract

Uncertainty Quantification

Explore similar work

Uncertainty Quantification in PINNs for Turbulent Flows: Bayesian Inference and Repulsive Ensembles

Uncovering and Fixing Collider Bias in Bayesian PINNs

Functional-prior-based approaches to Bayesian PDE-constrained inversion using physics-informed neural networks