Gaussian Neural Networks

Peter Kuhn; Victoria Heusinger-Heß

cs.LGSep 28, 2026

Gaussian Neural Networks

Authors: Peter Kuhn, Victoria Heusinger-Heß

Organizations: Fraunhofer Institute for High-Speed Dynamics, Ernst-Mach-Institut, EMI, Germany

Abstract

Gaussian neural networks (GaNNs) are proposed as a novel regularization mechanism for neural networks. From a Bayesian perspective standard regularization techniques can be viewed as imposing priors over weight-space. Assuming priors over activation-space remains a largely unexplored possibility. GaNNs assume such priors. They do this by treating activities from earlier layers like signals with Gaussian noise and predicting the properties of the noise distribution using an additional unsupervised loss. While training, the unsupervised loss acts as a penalty on unexpected activities, allowing greater weight updates in less surprising directions. The paper demonstrates the superiority of Gaussian neural networks over standard neural networks on a variety of classification and regression tasks. We also investigate the ability of GaNNs to quantify uncertainty.

Figures & tables

Figure 1: Two models with two sparse (a) and dense (b) Gaussian hidden layers. Every box is a Gaussian neuron, containing an activity and an activity prior. In the sparse Gaussian layers, the variance-priors only depend on the associated activity. In the dense Gaussian layers, the variance-priors depend on all preceding nodes. In both architectures, the activities are independent of variances. The mean-priors only enter as constituents of the loss and are irrelevant in inference.

Regression
Name	Domain	N	Features	Epochs
Airfoil	tabular	1503	5	3000
Yacht	tabular	308	6	5000
UTKFace	vision	23708	12288	120
Wine Regression	tabular	6497	11	120
YearPredictionMSD	tabular	515345	90	120

Table 1: Summary of dataset properties and number of epochs employed.

Regression (MAE)
Dataset	GaNN-dense	GaNN-sparse	GaNN-sparse + D	ANN	ANN + D	ANN + WD	ANN + WD + D
Airfoil	$2.706\pm 0.099$	$2.335\pm 0.075$	$2.853\pm 0.075$	$2.494\pm 0.090$	$11.516\pm 0.416$	$2.494\pm 0.090$	$11.515\pm 0.416$
Yacht	$2.208\pm 0.154$	$0.804\pm 0.119$	$1.173\pm 0.107$	$0.879\pm 0.119$	$5.521\pm 0.386$	$0.880\pm 0.120$	$5.521\pm 0.386$
UTKFace	$10.164\pm 0.094$	$10.131\pm 0.124$	$10.227\pm 0.116$	$10.183\pm 0.078$	$16.719\pm 0.168$	$10.184\pm 0.078$	$16.719\pm 0.168$
Wine Regression	$0.573\pm 0.008$	$0.589\pm 0.008$	$0.596\pm 0.009$	$0.613\pm 0.005$	$0.929\pm 0.022$	$0.613\pm 0.005$	$0.929\pm 0.022$
YearPredictionMSD	$6.173\pm 0.025$	$6.159\pm 0.025$	$6.841\pm 0.217$	$7.029\pm 0.062$	$33.727\pm 1.161$	$7.024\pm 0.061$	$33.075\pm 1.157$

Table 2: Test performance (mean for best model

\pm

std). Top: Regression (MAE). Bottom: Classification (Accuracy). Note that for regression datasets, the ANN dropout columns use the variance-head variants.

Figure 2: Ablation studies on CIFAR100 using a sparse GaNN. Validation accuracy evolution for different values of $\alpha$ .

Regression (Gaussian NLL)
Dataset	GaNN-dense	GaNN-sparse	GaNN-sparse + D	ANN + varhead	ANN + varhead + D
Airfoil	3.36	3.28	3.26	4.05	4.10
Yacht	2.79	1.99	2.01	3.02	3.06
UTKFace	12.92	7.57	7.89	4.50	4.50
Wine Regression	1.23	1.27	1.28	1.61	1.63
YearPredictionMSD	5.05	5.05	5.18	5.38	6.06

Table 3: Test UQ (means). Top: Regression (Gaussian NLL). Bottom: Classification (NLL / AUROC).

Figure 3: Performance on the airfoil dataset, standard deviations are visualized as shaded regions. This dataset constitutes an extreme example of the predicted delayed convergence resulting in higher final performance. Delayed convergence is caused by first optimizing inner loss (left, dashed) and then optimizing the base loss.

Explore similar work

Gaussian Sheaf Neural Networks
May 20, 2026André Ribeiro, Ana Luiza Tenório, Tiago da Silva +1Sheaf Neural Networks Graph Representation Learning
Getulio Vargas Foundation · MBZUAI
Local Stability and Gaussian Smoothing of Quantized Neural Networks
Jul 22, 2026Sergey Salishev, Anton Makarov, Oleg GranichinSmoothing Neural Network
St. Petersburg State University, 199034, Saint Petersburg, Russia
Universality in Deep Neural Networks: An approach via the Lindeberg exchange principle
May 4, 2026Filippo Giovagnini, Sotirios Kotitsas, Marco RomitoInfinities Activation Functions

May 20, 2026, cs.LGJ/K move · Enter open · S save

Gaussian Sheaf Neural Networks

André Ribeiro, Ana Luiza Tenório, Tiago da Silva, Diego Mesquita

Getulio Vargas Foundation · MBZUAI

Sheaf Neural Networks Graph Representation Learning Sheaf Neural Network Inductive Bias

Open paper AskPDF arXiv

Graph Neural Networks (GNNs) have become the de facto standard for learning on relational data. While traditional GNNs' message passing is well suited for vector-valued node features, there are cases in which node features are better represented by probability distributions than real vectors. Concretely, when node features are Gaussians, characterized by a mean and a covariance matrix, naively concatenating their parameters into a single vector and applying standard message passing discards the geometric and algebraic structure that governs means and covariances. We propose Gaussian Sheaf Neural Networks (GSNNs), a principled framework that incorporates these inductive biases into graph-based learning. Building on the theory of cellular sheaves, we derive a new Laplacian operator that generalizes the sheaf Laplacian to this setting and preserves its key properties. We complement our theoretical contributions with experiments on synthetic and real-world data that illustrate the practical relevance of GSNNs.

Abstract

Explore similar work

Gaussian Sheaf Neural Networks

Local Stability and Gaussian Smoothing of Quantized Neural Networks

Universality in Deep Neural Networks: An approach via the Lindeberg exchange principle