Correlational Training of Morphological Neural Networks

Konstantinos Fotopoulos; Petros Maragos

cs.LGOct 8, 2026

Correlational Training of Morphological Neural Networks

Authors: Konstantinos Fotopoulos, Petros Maragos

Organizations: HERON - Hellenic Robotics Center of Excellence, Athena Research Center, Marousi, Greece · Institute of Robotics, Athena Research Center, Marousi, Greece · School of ECE, National Technical University of Athens, Athens, Greece

Abstract

Neural networks are typically trained using first-order methods and back-propagation. It is unclear whether this approach is optimal for morphological layers whose weight Jacobians are sparse and whose resulting parameter gradients can be poor. In this work, we propose a novel weight update method for morphological neural networks inspired from the Multiplicative Weights Update (MWU) scheme. We view each morphological perceptron as an instance of the learning from experts' advice problem in logarithmic space, and use a correlation-based reward that favors inputs aligned with the desired output change, regardless of whether a strong gradient signal has reached their weight. We empirically evaluate our approach by training fully connected layers both as stand-alone models and as parts of larger transformer networks. Across nine benchmarks, correlational training yields improvements on eight, by up to 32.84 percentage points, while substantially reducing run-to-run variability.

Figures & tables

\theta^{\prime}=\theta+\eta g_{\theta}.

Algorithm 1 MWU-adjusted morphological training.

Dataset	MLP / ViT	Morph. + BP	Morph. + Corr. (ours)	$\Delta$ vs. BP (pp)
CIFAR-10	$81.08\pm 0.33$	$63.96\pm 0.88$	$70.20\pm 0.62$	+6.24
Fashion-MNIST	$88.94\pm 0.34$	$62.66\pm 8.18$	$86.03\pm 0.36$	+23.37
MNIST	$98.19\pm 0.20$	$62.46\pm 8.90$	$95.30\pm 0.38$	+32.84
Adult	$78.99\pm 1.15$	$59.14\pm 12.52$	$78.13\pm 1.08$	+18.99
Chess	$98.96\pm 0.33$	$68.70\pm 8.29$	$87.89\pm 12.69$	+19.19
Ionosphere	$90.48\pm 2.94$	$78.38\pm 16.03$	$83.65\pm 4.74$	+5.27

Table 1: Test performance of linear models, morphological models trained using ordinary back-propagation (BP), and the proposed correlational update, both accelerated using Adam, across several benchmarks. The fourth column corresponds to percentage-point difference with respect to ordinary BP. For image datasets we report accuracy; for PMLB balanced accuracy.

Panel of Figure 1 — Figure 1: Training curves (mean $\pm$ std) of linear models (blue), morphological models under vanilla Adam (orange), and morphological models under our proposed method (green). Our method improves training dynamics over vanilla Adam.

Explore similar work

How Temporal Correlations Shape Memory in Linear Recurrent Neural Networks
Aug 31, 2026Arnol Manuel Fokam, Fasseu Sieyondji Akpevwoghene, Edem Fiifi DawsonRecurrent Neural Networks Linear RNNs
Independent Researcher, United Kingdom · Department of Electrical and Electronic Engineering, University of Buea, Cameroon · minoHealth AI Labs, Ghana
Momba: Network Modernization Improves Multi-Objective Reinforcement Learning
Aug 7, 2026Adam Štafa, Santeri Heiskanen, Petr Novotný +1Multi-Objective Reinforcement Learning
Faculty of Informatics, Masaryk University · Department of Electrical Engineering and Automation, Aalto University
Learning from almost nothing: How neural networks survive heavy input corruption
Jun 9, 2026Justin Tahmassebpur, Asadullah Bhuiyan, Hyejin Kim +1Neural Network Generalization Multilayer Perceptrons
Department of Applied and Engineering Physics, Cornell University, Ithaca, NY 14853, USA · Department of Physics, Cornell University, Ithaca, NY 14853, USA

Aug 31, 2026, cs.LGJ/K move · Enter open · S save

How Temporal Correlations Shape Memory in Linear Recurrent Neural Networks

Arnol Manuel Fokam, Fasseu Sieyondji Akpevwoghene, Edem Fiifi Dawson

Independent Researcher, United Kingdom · Department of Electrical and Electronic Engineering, University of Buea, Cameroon · minoHealth AI Labs, Ghana

Recurrent Neural Networks Linear RNNs Neural Network Training Dynamics

Open paper AskPDF arXiv

The linear recurrent neural network (LRNN) is a simple model for studying how much memory a network builds up as it trains. For uncorrelated inputs, earlier work found that training itself settles the network between keeping the past and reacting only to the present. Real sequences are correlated, and we solve the learning dynamics exactly for correlated inputs. In the solution, keeping the past carries a cost. The whole effect of correlation lands on that cost. This cost reduces to the earlier one when inputs are uncorrelated and grows once they are positively correlated. Three findings follow. (1) Correlation reshapes the course of learning, not only its end. Memory builds, overshoots, and is partly removed, and the settled network keeps less of the past. (2) Memory switches off at a threshold set by one number, how much each input resembles the one just before it. Neither sequence length nor longer-range correlation moves this threshold. Memory is worth keeping only when the task needs the previous input more than the current input already supplies it through correlation with the past. (3) The best network changes too. Zero error demands a feedthrough, a path that passes the current input straight to the network's output and remembers nothing, and training builds it unprompted when given one spare hidden dimension. Our work turns one property of the input into a prediction of whether a network learns memory and explains why correlated data turns recurrent networks into change detectors.

Abstract

Explore similar work

How Temporal Correlations Shape Memory in Linear Recurrent Neural Networks

Momba: Network Modernization Improves Multi-Objective Reinforcement Learning

Learning from almost nothing: How neural networks survive heavy input corruption