Throughout the literature on Neural Cellular Automata (NCAs), it is often taken for granted that the systems learn attractors. This is shown through evolving the system for many timesteps and noting visual similarity to the goal state. There remain many questions after such an analysis. Namely, what kind of attractors do we have? Is their behavior ordered or chaotic? Can we estimate stability over very long time horizons? What really happens in the attractor when perturbations are applied? In this paper, we present a case study to help answer these questions, with methods drawn from the literature on dynamical systems theory. We use the growing gecko NCA of Mordvintsev et al. (2020) with deterministic cell updates as a case study. To the best of the authors' knowledge, we present the first visualizations of NCA attractor dynamics. We also analyze them using the Lyapunov and Fourier spectra, to reveal that the NCA displays oscillatory, periodic and quasi-periodic behavior, and that these behaviors arise early during training. This challenges the belief that NCAs learn fixed point attractors. Finally, we show that large perturbations to the attractor states can throw the NCAs into a secondary mode separate from the original attractor. We hope that this initial foray into NCA attractor dynamics expands the toolkit for NCA researchers to analyze the robustness and stability of their systems.
Figures & tables
Figure 1: The burn-in of the attractor. From timestep t =0 to 200, the system moves a large distance. At timestep 60 (red circle), the system has already converged to the goal state. From timestep 200 to 400, the system is still moving (though the distances are small, notice the axes). From timestep 400 to 1000, the system finally converges, and from 1000 onward, the system no longer moves meaningfully along the rotational axis. Time intervals are transformed by separate PCA transforms.
Attractor type
Lyapunov Spectrum
Fixed point
−⋯−
Limit cycle/periodic
0−⋯−
k-torus/quasi-periodic
k≥10⋯0−⋯−
Strange attractor
l≥1+⋯+k≥10⋯0−⋯−
Table 1: Attractors and their characteristic spectrum. Strange attractors are a wider topic, and depending on how high the number l and k gets, there are more classifications of strange attractors within the cateory. Adapted from ( Sandri, 1996 ) .
Figure 2: Estimating rate of change using finite differences. Adapted from ( Datseris and Parlitz, 2022 , p. 39)
Figure 3: The Fourier spectra (right) of three attractors (left). From top to bottom: Van der Pol (periodic), a torus (quasi-periodic), and Lorenz (chaotic).
Figure 4: The attractors of the green gecko. Top: Version 1. Middle: Version 2. Bottom: Version 3. Unless otherwise specified, PCA axis 0 to 2 are used. Axes are only shown where relevant. Magenta, blue, and green are the first, second, and third most dominant PCA-axes respectively.
Figure 5: The unscaled version 1 attractor. x,y, and z axes are the first, second, and third most dominant PCA axes.
V
Spectrum, relevant
Simplified
1
λ1=0.0,λ2=−0.002 ,
0−⋯−
λ3−5=−0.007 , λ10=−0.013
2
λ1,2=−0.005 , λ3,4=−0.007
−⋯−
λ5=−0.01 , λ10=−0.012
3
λ1=−0.009 , λ2=−0.014
−⋯−
λ10=−0.016 , λ50=−0.031
Table 2: The top Lyapunov exponents of the three attractors. Only informative ones are included.
Figure 6: The sum of unscaled distances for version 2 (top) and 3 (bottom).
Figure 7: The Fourier power spectra of unscaled data. Top: version 1. Middle: version 2. Bottom: version 3.
Figure 8: Perturbations returning to system attractors. Top: version 1, unscaled. Bottom: version 3. Time is in colormap “plasma”, from purple to yellow. Perturbations’ start positions have a black, red, green, blue, or yellow ball. Original attractor in “viridis” in a, b, d, and e.
Figure 9: Attractor for model version 3 for 1 to 8*1000 epochs. The plots include 300 timesteps. The colormap is “plasma”, indicating the timestep number, from purple to yellow.
As Neural Cellular Automata (NCAs) are increasingly applied outside of the toy models in Artificial Life, there is a pressing need to understand how they behave and to build appropriate routes to interpret what they have learnt. By their very nature, the benefits of training NCAs are balanced with a lack of interpretability: we can engineer emergent behaviour, but have limited ability to understand what has been learnt. In this paper, we apply a variety of techniques to pry open the NCA black box and glean some understanding of what it has learnt to do. We apply techniques from manifold learning (principal components analysis and both dense and sparse autoencoders) along with techniques from topological data analysis (persistent homology) to capture the NCA's underlying behavioural manifold, with varying success. Results show that when analysis is performed at a macroscopic level (i.e. taking the entire NCA state as a single data point), the underlying manifold is often quite simple and can be captured and analysed quite well. When analysis is performed at a microscopic level (i.e. taking the state of individual cells as a single data point), the manifold is highly complex and more complicated techniques are required in order to make sense of it.
James Stovold, Mia-Katrin Kvalsund, Harald Michael Ludwig +2
University of York, UK · University of Oslo, Norway · Machine Learning Research Unit, TU Wien, Austria +2
Stochastic cell updates are often used throughout the life of a neural cellular automaton (NCA), from backpropagation through time to final rollout. This leaves two questions entangled: does update randomness help learn a useful rule, and must that randomness remain at execution? We separate training and evaluation update modes in controlled Growing NCA experiments, then vary the states shown during training. Under the standard constant-rate persist recipe, asynchronous training passes the short-horizon quality test in 10/10 runs, compared with 3/10 synchronous runs. All ten asynchronous models also retain the target for 4,096 steps under deterministic evaluation. For a scalar translation-invariant lattice, we derive an exact mean-square criterion: random masking can damp mean modes, but it also injects variance, and a mean-only test misclassifies four non-marginal settings. Finally, among 30 models that all pass the same reconstruction test, eight of ten grow-trained models become off-target at 4,096 steps, while all persist and regenerate models retain the target; damage recovery separates persist from regenerate. The results distinguish optimization reliability, execution mode, and task-specific behavior instead of treating them as one stability property.
Fei Zuo, Jiaqi Shi, Yujing Liu
Fudan University · Independent Researcher · Shanghai Pupuda Culture Communication Co., Ltd.
Stephen Wolfram proclaimed in his 2003 seminal work "A New Kind Of Science" that simple recursive programs in the form of Cellular Automata (CA) are a promising approach to replace currently used mathematical formalizations, e.g. differential equations, to improve the modeling of complex systems. Over two decades later, while Cellular Automata have still been waiting for a substantial breakthrough in scientific applications, recent research showed new and promising approaches which combine Wolfram's ideas with learnable Artificial Neural Networks: So-called Neural Cellular Automata (NCA) are able to learn the complex update rules of CA from data samples, allowing them to model complex, self-organizing generative systems. The aim of this paper is to review the existing work on NCA and provide a unified modular framework and notation, as well as a reference implementation in the open-source library NCAtorch. Supplementary materials, videos, and code are available at the project website: https://www.neural-cellular-automata.org/
Martin Spitznagel, Janis Keuper
Institute for Machine Learning and Analytics (IMLA), Offenburg University, Germany · University of Mannheim, Germany