cs.LGMar 19, 2026

Beyond Weighted Summation: Learnable Nonlinear Aggregation Functions for Robust Artificial Neurons

Authors: Berke Deniz Bozyigit

Organizations: Independent Researcher

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Weighted summation has remained the default input aggregation mechanism in artificial neurons since the earliest neural network models. While computationally efficient, this design implicitly behaves like a mean-based estimator and is therefore sensitive to noisy or extreme inputs. This paper investigates whether replacing fixed linear aggregation with learnable nonlinear alternatives can improve neural network robustness without sacrificing trainability. Two differentiable aggregation mechanisms are introduced: an F-Mean neuron based on a learnable power-weighted aggregation rule, and a Gaussian Support neuron based on distance-aware affinity weighting. To preserve the optimisation stability of standard neurons, hybrid neurons are proposed that interpolate between linear and nonlinear aggregation through a learnable blending parameter. Evaluated in multilayer perceptrons and convolutional neural networks on CIFAR-10 and a noisy CIFAR-10 variant with additive Gaussian corruption, hybrid neurons consistently improve robustness under noise while F-Mean hybrids also yield modest gains on clean data. The three-way hybrid achieves robustness scores of up to 0.991 compared to 0.890 for the standard baseline, and learned parameters converge consistently to sub-linear aggregation (p ≈\approx 0.43--0.50) and high novelty utilisation (αα ≈\approx 0.69--0.79). These findings suggest that neuron-level aggregation is a meaningful and underexplored design dimension for building more noise-tolerant neural networks.

Figures & tables

ModelClean (%)Noisy (%)ρ\rho
Baseline52.3051.450.984
F-Mean Hybrid55.1753.560.971
Gaussian Hybrid54.3053.300.982
Three-way Hybrid55.2154.720.991
Table 1: MLP results on clean and noisy CIFAR-10.
ModelClean (%)Noisy (%)ρ\rho
Standard CNN87.3377.730.890
F-Mean CNN87.6178.410.895
Gaussian CNN86.6077.330.893
Three-way CNN86.3777.520.898
Table 2: CNN results on clean and noisy CIFAR-10.
Architectureα\alphappσ\sigma
MLP F-Mean0.7620.476–
CNN F-Mean0.7860.430–
MLP Gaussian0.689–4.68
CNN Gaussian0.785–7.30
Three-way MLP0.70–0.790.4955.08
Three-way CNN0.70–0.790.4458.50
Table 3: Converged novel parameters across architectures.
More in this topic

Neural Network Robustness

Neural network robustness studies how models withstand adversarial inputs, data shifts, noise, and hardware faults while maintaining reliable predictions.

See all 237 papers

Explore similar work

CardsList
May 21, 2026

Partial Fusion of Neural Networks: Efficient Tradeoffs Between Ensembles and Weight Aggregation

Ensembles of neural networks typically outperform individual networks but incur large computational costs, whereas weight aggregation produces less costly, yet also less accurate, aggregate models. We introduce partial fusion of networks, which interpolates between ensembles and weight aggregation and thus allows for a flexible tradeoff between computational cost and performance. A direct way to achieve this is to extend existing weight aggregation methods based on neuron-level similarity between different networks, where partial fusion then only aggregates weights of neurons which are most similar. We showcase one particular method to jointly identify which neurons are most similar and match them via partial optimal transport. Further, we consider the more general perspective of weight aggregation and partial fusion as generalized pruning of ensemble models, where neurons cannot just be deleted, but also linearly combined. Finally, we show that generalized pruning applied to a single network yields similar benefits as partial fusion by allowing for a tradeoff between isolating, deleting, and linearly combining neurons based on similarity. Our code is available at https://github.com/Fabian-Mor/partial_fusion_nn.

Fabian Morelli, Stephan Eckstein

Department of Computer Science, University of Tübingen, Germany · Department of Mathematics, University of Tübingen, Germany.

Inverse Optimal TransportEnsemble LearningDeep EnsemblesModel MergingNeural Network CompressionNeural Network Pruning
AskPDF
Jun 9, 2026

Neural networks become centroid classifiers when trained on heavily corrupted inputs

Learning from imperfect data is a central theme in machine learning, connecting practical questions of robustness to fundamental questions of learnability. Here we examine attribute noise: learning from corrupted inputs while keeping the labels intact, a setting that has received considerably less analytical attention than its label-noise counterpart. We consider two types of corruption models: additive noise and replacement noise. Through experiments with multi-layer perceptrons (MLPs) on corrupted classification datasets, we find that neural networks remain robust, maintaining well-above-chance accuracy even when inputs are >90% corrupted -- far beyond human recognition. To understand this robustness, we analyze infinite-width networks in the heavy-corruption regime using a mean-field-inspired approach and derive a leading-order decision rule for the classification outcome: the network implements a prototype rule, the nearest-class-mean, assigning each test point to the class whose training-set average it most closely resembles. This leading-order decision rule is universal across a broad range of MLP architectures, holding for any depth, as well as a wide class of activation functions and noise distributions. The same centroid mechanism closely matches finite-width network behavior in our experiments and provides an interpretable and analytically tractable account of why learning can succeed even when individual training examples carry almost no signal.

Justin Tahmassebpur, Asadullah Bhuiyan, Hyejin Kim +1

Department of Applied and Engineering Physics, Cornell University, Ithaca, NY 14853, USA · Department of Physics, Cornell University, Ithaca, NY 14853, USA

Neural Network GeneralizationMultilayer PerceptronsPrototype-Based ClassificationNeural Network RobustnessCorruption RobustnessClassification+1
AskPDF
Jun 11, 2026

Neural Variability Enhances Artificial Network Robustness

Neural responses in cortex exhibit substantial trial-to-trial variability in response to repeated stimuli, while peripheral sensory neurons respond far more consistently, leading many to wonder whether stochasticity may carry meaning. Existing work has argued that noise and signal correlations may be optimized for discrimination in animals, whereas artificial neural network (ANN) studies have shown similar benefits of noise in machine learning tasks, although most ANN work has neglected the effects of correlations. Here we investigate whether correlated noise improves the robustness of artificial neural networks to adversarial attacks and naturalistic image modifications. Using the covariance of activations under modified versus clean inputs, we find that structured noise may significantly improve network robustness. Robustness to naturalistic image modifications benefits most from structure, but this structure transfers poorly across modification types. In contrast, noise structure from adversarial attacks can generalize to other kinds of attacks. These results suggest that structured noise in ANN activations generally improves robustness, establishing a biologically plausible strategy for creating robust artificial neural networks that only relies on local information.

Robin Preble, Praveen Venkatesh, Stefan Mihalas +1

Department of Computer Science, Western Washington University, Bellingham, WA 98225 · Allen Institute, Seattle, WA 98195

Distribution Shift RobustnessNeural Network RobustnessAdversarial RobustnessCorruption RobustnessNoise Robustness
AskPDF