Gaussian Neural Networks

Peter Kuhn; Victoria Heusinger-Heß

cs.LGSep 28, 2026

Gaussian Neural Networks

Authors: Peter Kuhn, Victoria Heusinger-Heß

Organizations: Fraunhofer Institute for High-Speed Dynamics, Ernst-Mach-Institut, EMI, Germany

Abstract

Gaussian neural networks (GaNNs) are proposed as a novel regularization mechanism for neural networks. From a Bayesian perspective standard regularization techniques can be viewed as imposing priors over weight-space. Assuming priors over activation-space remains a largely unexplored possibility. GaNNs assume such priors. They do this by treating activities from earlier layers like signals with Gaussian noise and predicting the properties of the noise distribution using an additional unsupervised loss. While training, the unsupervised loss acts as a penalty on unexpected activities, allowing greater weight updates in less surprising directions. The paper demonstrates the superiority of Gaussian neural networks over standard neural networks on a variety of classification and regression tasks. We also investigate the ability of GaNNs to quantify uncertainty.

Figures & tables

Figure 1: Two models with two sparse (a) and dense (b) Gaussian hidden layers. Every box is a Gaussian neuron, containing an activity and an activity prior. In the sparse Gaussian layers, the variance-priors only depend on the associated activity. In the dense Gaussian layers, the variance-priors depend on all preceding nodes. In both architectures, the activities are independent of variances. The mean-priors only enter as constituents of the loss and are irrelevant in inference.

Regression
Name	Domain	N	Features	Epochs
Airfoil	tabular	1503	5	3000
Yacht	tabular	308	6	5000
UTKFace	vision	23708	12288	120
Wine Regression	tabular	6497	11	120
YearPredictionMSD	tabular	515345	90	120

Table 1: Summary of dataset properties and number of epochs employed.

Regression (MAE)
Dataset	GaNN-dense	GaNN-sparse	GaNN-sparse + D	ANN	ANN + D	ANN + WD	ANN + WD + D
Airfoil	$2.706\pm 0.099$	$2.335\pm 0.075$	$2.853\pm 0.075$	$2.494\pm 0.090$	$11.516\pm 0.416$	$2.494\pm 0.090$	$11.515\pm 0.416$
Yacht	$2.208\pm 0.154$	$0.804\pm 0.119$	$1.173\pm 0.107$	$0.879\pm 0.119$	$5.521\pm 0.386$	$0.880\pm 0.120$	$5.521\pm 0.386$
UTKFace	$10.164\pm 0.094$	$10.131\pm 0.124$	$10.227\pm 0.116$	$10.183\pm 0.078$	$16.719\pm 0.168$	$10.184\pm 0.078$	$16.719\pm 0.168$
Wine Regression	$0.573\pm 0.008$	$0.589\pm 0.008$	$0.596\pm 0.009$	$0.613\pm 0.005$	$0.929\pm 0.022$	$0.613\pm 0.005$	$0.929\pm 0.022$
YearPredictionMSD	$6.173\pm 0.025$	$6.159\pm 0.025$	$6.841\pm 0.217$	$7.029\pm 0.062$	$33.727\pm 1.161$	$7.024\pm 0.061$	$33.075\pm 1.157$

Table 2: Test performance (mean for best model

\pm

std). Top: Regression (MAE). Bottom: Classification (Accuracy). Note that for regression datasets, the ANN dropout columns use the variance-head variants.

Figure 2: Ablation studies on CIFAR100 using a sparse GaNN. Validation accuracy evolution for different values of $\alpha$ .

Regression (Gaussian NLL)
Dataset	GaNN-dense	GaNN-sparse	GaNN-sparse + D	ANN + varhead	ANN + varhead + D
Airfoil	3.36	3.28	3.26	4.05	4.10
Yacht	2.79	1.99	2.01	3.02	3.06
UTKFace	12.92	7.57	7.89	4.50	4.50
Wine Regression	1.23	1.27	1.28	1.61	1.63
YearPredictionMSD	5.05	5.05	5.18	5.38	6.06

Table 3: Test UQ (means). Top: Regression (Gaussian NLL). Bottom: Classification (NLL / AUROC).

Figure 3: Performance on the airfoil dataset, standard deviations are visualized as shaded regions. This dataset constitutes an extreme example of the predicted delayed convergence resulting in higher final performance. Delayed convergence is caused by first optimizing inner loss (left, dashed) and then optimizing the base loss.

Explore similar work

May 20, 2026cs.LG

Gaussian Sheaf Neural Networks

Graph Neural Networks (GNNs) have become the de facto standard for learning on relational data. While traditional GNNs' message passing is well suited for vector-valued node features, there are cases in which node features are better represented by probability distributions than real vectors. Concretely, when node features are Gaussians, characterized by a mean and a covariance matrix, naively concatenating their parameters into a single vector and applying standard message passing discards the geometric and algebraic structure that governs means and covariances. We propose Gaussian Sheaf Neural Networks (GSNNs), a principled framework that incorporates these inductive biases into graph-based learning. Building on the theory of cellular sheaves, we derive a new Laplacian operator that generalizes the sheaf Laplacian to this setting and preserves its key properties. We complement our theoretical contributions with experiments on synthetic and real-world data that illustrate the practical relevance of GSNNs.

André Ribeiro, Ana Luiza Tenório, Tiago da Silva +1

Getulio Vargas Foundation · MBZUAI

Sheaf Neural Networks Graph Representation Learning Sheaf Neural Network Inductive Bias

Ask PDF

Jul 22, 2026cs.LG

Local Stability and Gaussian Smoothing of Quantized Neural Networks

We study Gaussian averaging as a smooth surrogate for quantized neural models. Under bounded local oscillation, we derive a local dimension-dependent bound on |f-g|, linking Gaussian smoothing to the stability analysis of discontinuous networks. We compute closed-form Gaussian averages of the rectified linear unit (ReLU) and sign activation functions, and illustrate the mechanism on a high-dimensional binary perceptron, where layer-preactivation aggregation under an explicit quantization-noise surrogate yields the Gaussian envelope used in inference-side smoothing and training-side smooth surrogate gradients.

Sergey Salishev, Anton Makarov, Oleg Granichin

St. Petersburg State University, 199034, Saint Petersburg, Russia

Smoothing Neural Network Surrogate Gradient

Ask PDF

May 4, 2026math.PR

Universality in Deep Neural Networks: An approach via the Lindeberg exchange principle

We consider the infinite-width limit of a fully connected deep neural network with general weights, and we prove quantitative general bounds on the

2

-Wasserstein distance between the network and its infinite-width Gaussian limit, under appropriate regularity assumptions on the activation function. Our main tool is a Lindeberg principle for Deep Neural Networks, which we use to successively replace the weights on each layer by Gaussian random variables.

Filippo Giovagnini, Sotirios Kotitsas, Marco Romito

Infinities Activation Functions

Ask PDF