Throughout the literature on Neural Cellular Automata (NCAs), it is often taken for granted that the systems learn attractors. This is shown through evolving the system for many timesteps and noting visual similarity to the goal state. There remain many questions after such an analysis. Namely, what kind of attractors do we have? Is their behavior ordered or chaotic? Can we estimate stability over very long time horizons? What really happens in the attractor when perturbations are applied? In this paper, we present a case study to help answer these questions, with methods drawn from the literature on dynamical systems theory. We use the growing gecko NCA of Mordvintsev et al. (2020) with deterministic cell updates as a case study. To the best of the authors' knowledge, we present the first visualizations of NCA attractor dynamics. We also analyze them using the Lyapunov and Fourier spectra, to reveal that the NCA displays oscillatory, periodic and quasi-periodic behavior, and that these behaviors arise early during training. This challenges the belief that NCAs learn fixed point attractors. Finally, we show that large perturbations to the attractor states can throw the NCAs into a secondary mode separate from the original attractor. We hope that this initial foray into NCA attractor dynamics expands the toolkit for NCA researchers to analyze the robustness and stability of their systems.
Figures & tables
Figure 1: The burn-in of the attractor. From timestep t =0 to 200, the system moves a large distance. At timestep 60 (red circle), the system has already converged to the goal state. From timestep 200 to 400, the system is still moving (though the distances are small, notice the axes). From timestep 400 to 1000, the system finally converges, and from 1000 onward, the system no longer moves meaningfully along the rotational axis. Time intervals are transformed by separate PCA transforms.
Attractor type
Lyapunov Spectrum
Fixed point
−⋯−
Limit cycle/periodic
0−⋯−
k-torus/quasi-periodic
k≥10⋯0−⋯−
Strange attractor
l≥1+⋯+k≥10⋯0−⋯−
Table 1: Attractors and their characteristic spectrum. Strange attractors are a wider topic, and depending on how high the number l and k gets, there are more classifications of strange attractors within the cateory. Adapted from ( Sandri, 1996 ) .
Figure 2: Estimating rate of change using finite differences. Adapted from ( Datseris and Parlitz, 2022 , p. 39)
Figure 3: The Fourier spectra (right) of three attractors (left). From top to bottom: Van der Pol (periodic), a torus (quasi-periodic), and Lorenz (chaotic).
Figure 4: The attractors of the green gecko. Top: Version 1. Middle: Version 2. Bottom: Version 3. Unless otherwise specified, PCA axis 0 to 2 are used. Axes are only shown where relevant. Magenta, blue, and green are the first, second, and third most dominant PCA-axes respectively.
Figure 5: The unscaled version 1 attractor. x,y, and z axes are the first, second, and third most dominant PCA axes.
V
Spectrum, relevant
Simplified
1
λ1=0.0,λ2=−0.002 ,
0−⋯−
λ3−5=−0.007 , λ10=−0.013
2
λ1,2=−0.005 , λ3,4=−0.007
−⋯−
λ5=−0.01 , λ10=−0.012
3
λ1=−0.009 , λ2=−0.014
−⋯−
λ10=−0.016 , λ50=−0.031
Table 2: The top Lyapunov exponents of the three attractors. Only informative ones are included.
Figure 6: The sum of unscaled distances for version 2 (top) and 3 (bottom).
Figure 7: The Fourier power spectra of unscaled data. Top: version 1. Middle: version 2. Bottom: version 3.
Figure 8: Perturbations returning to system attractors. Top: version 1, unscaled. Bottom: version 3. Time is in colormap “plasma”, from purple to yellow. Perturbations’ start positions have a black, red, green, blue, or yellow ball. Original attractor in “viridis” in a, b, d, and e.
Figure 9: Attractor for model version 3 for 1 to 8*1000 epochs. The plots include 300 timesteps. The colormap is “plasma”, indicating the timestep number, from purple to yellow.