The Continuous Thought Machine (CTM) is a recurrent network that attends to inputs and makes predictions based on the synchronization between its neurons. On tasks spanning many domains, it displays emergent thought processes that have not been explained. We report a number of findings about how the CTM classifies images, focusing on ImageNet. Thinking steps are attracting but strongly non-normal maps exhibiting balanced amplification (Murphy and Miller, 2009); while attraction is already present at CTM initialization, training highly promotes non-normality. By Banach's theorem, the neural state thus has a locally unique attracting fixed point, which moves at every step, and the thought trajectory is the curve these fixed points trace, followed one tick behind. Most of that curve's motion is shared across images, and visible as a previously reported traveling wave of activity across neurons. The attention schedule along the thought is largely generic: shuffling queries across images at every tick costs only 7 percent of classification accuracy, while image-independent increments predict the next query with R^2 = 0.98 (held out). Our results suggest that the CTM has found a single strategy to process all ImageNet images.
Figures & tables
Figure 1: One tick of the CTM. After a single backbone pass the model iterates against a frozen token bank: a query qt read from the action-side synchronization slice drives attention; a synapse model mixes the attention output ot with the state zt ; per-neuron networks produce the next state zt+1 ; the synchronization matrix St+1 is updated and its output slice is read to logits yt+1 .
Figure 2: Attracting but non-normal, and one tick late. (a) At each fixed point the spectral radius ρ(J) stays below one (the map is stable and the fixed point unique) while the operator norm σmax(J) exceeds one (it is not a Euclidean contraction); the gap is the non-normality. (b) State and fixed point projected on the leading axis of the fixed-point curve, with zt−1⋆ overlaid: the state’s trajectory is the fixed-point curve delayed by one tick. (c) The same claim measured: the mean distance ∥zt−zt−k⋆∥ to the fixed point k ticks earlier, against candidate lag k , is minimized at k=1 ( 5.9 , against 41.4 at k=0 and 40.7 at k=2 ).
Figure 3: The fixed-point curve. The recurrence’s residual field on the plane spanned by the path of successive fixed points, shown across ticks for one held-out image. Color is residual magnitude, streamlines the in-plane flow; the red curve is the path of the fixed point z⋆ , the star its current location, and the black dot the state, which stays within about one step of the moving attractor.
Figure 4: The attention scan is a shared schedule. For two held-out images (input, first column), the model’s true attention (first row of each pair) and the attention produced by the shared population-average query (second row), across ticks; one color scale per image. The shared schedule recovers the image-specific scan.
intact
frozen ( q1 )
shuffled (cross-image)
uniform
no-attention
top-1
0.723
0.480
0.675
0.145
0.001
rel. change
—
−34%
−7%
−80%
−100%
Table 1: Query/attention ablations on ImageNet (top-1 at tdec ; chance 0.001 ). Freezing the query’s schedule is costly; replacing its image-specific content costs little. n=1000 images; 95% intervals in Appendix D . Without attention the image never enters after the backbone and z1 is a learned constant, so every image follows the same trajectory and receives the same class; accuracy is that class’s frequency in the sample.
Figure 5: The traveling wave is the moving fixed point. One held-out image; neurons ordered by the phase of z⋆(At) . Left: activations zt over ticks — the reported wave. Right: the fixed points under the same ordering, indistinguishable from the left.
Figure 6: What sharpens the prediction. (a) Normalized entropy over ticks, intact and under each manipulation ( 512 images). (b) Fraction of images whose predicted class differs from its final value.
model
acc.
ρ(J) (med)
σmax(J) (med/max)
attn.
frozen
shuffled
MNIST ( D=128 )
0.95
0.10
0.49/0.77
0.97
+0%
+0%
CIFAR (ResNet-9)
0.84
0.068
0.36/3.06
0.90
+0.3%
+0.2%
ImageNet ( D=4096 )
0.72
0.15
3.2/74
0.86
−34%
−7%
Table 2: Cross-model comparison. The attracting curve ( ρ(J)<1 ) is universal; transient amplification ( σmax(J)>1 ) and query dependence emerge only on ImageNet. “attn.” is the entropy of the head-averaged attention weights over image patches, as a fraction of its maximum lnL ( 1 = uniform); “frozen” and “shuffled” are the accuracy changes from freezing the query at its first-tick value and from giving each image another image’s query.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
feed-forward / departure
antisymmetric
J (trained)
0.99
0.71
Ginibre (i.i.d.)
0.71
0.71
entry-shuffled J
0.81
0.71
random core (recurrence at init.)
0.93
—
Appendix
Table 3: Balanced-amplification descriptors of the tick-map Jacobian J at z⋆ (medians over image–tick pairs), against same-size random nulls and the random core (trained backbone, recurrence at initialization; feed-forward fraction only). The Schur feed-forward fraction and Henrici’s departure from normality coincide and are reported together. J is a near-maximal feed-forward non-normal operator whose rotational part is at chance.
Figure 7: The tick map is a feed-forward cascade in its Schur basis. Magnitudes ∣Tij∣ (log scale) of the real Schur form of the materialized Jacobian J=UTU∗ (one held-out image, mid-trajectory tick). The Schur form is upper triangular by construction; what is informative is where the mass sits within it. For a normal operator T would be diagonal, the whole norm carried by the eigenvalues. Here almost none is: the strictly upper-triangular part accounts for a fraction 0.99 of ∥J∥F (Table 3 ), so the operator is nearly all feed-forward coupling between Schur modes rather than modal decay. Magnitudes are shown, and individual mode positions should not be read as structure; see the text.
Figure 8: The shared schedule. Left: alignment of each image’s query with the population-average query at each tick, cos(qit,qˉt) . Right: the attention computed from the shared, population-average query matches each image’s true attention.
quantity
value
95% interval
n
top-1 at tdec (intact)
0.723
0.695 – 0.750
1000 images
top-1, frozen / shuffled query
0.480 / 0.675
0.450 – 0.511 / 0.646 – 0.704
1000 images
top-1, uniform / no attention
0.145 / 0.001
0.125 – 0.166 / 0.000 – 0.003
1000 images
ρ(J) at z⋆ (median)
0.147
0.139 – 0.155
3055 pairs ( 256 images)
σmax(J) at z⋆ (median)
3.21
3.09 – 3.35
3055 pairs
σmax(J)/ρ(J) per fixed point (median)
23.6
23.0 – 24.3
3055 pairs
Appendix
Table 4: Headline numbers, with the number of images (or image–tick pairs) each rests on and a bootstrap 95% interval. Accuracies are computed per image on one common sample.
Drawing on recent breakthroughs in cellular neurobiology and detailed biophysical modeling linking neocortical pyramidal neurons to distinct mental-state regimes, this work introduces a mathematically grounded formulation showing how models (e.g., Transformers) can implement computational principles underlying awake imaginative thought to pre-select relevant information before attention is applied via triadic modulation loops among queries (Q), keys (K), and values (V).~Scalability experiments on ImageNet-1K, benchmarked against a standard Vision Transformer (ViT), demonstrate significantly faster learning with reduced computational demand (fewer heads, layers, and tokens), consistent with our prior findings in reinforcement learning and language modeling. The approach operates at approximately O(N) complexity with respect to the number of input tokens N.
Ahsan Adeel, M. Bilal
TREND Project: Funded by the UK Advanced Research + Invention Agency (ARIA).
Recurrent neural networks typically compress information into a single vector-valued recurrent state, forcing short-term computation and long-term retention to share the same representation. Past extensions alleviate this bottleneck by increasing the memory capacity or separating timescales, but lack the combination of rapid neuron-level processing and longer-term retention found in biology. To that end, we introduce the Continuous Memory Machine (CMM), a recurrent architecture with matrix-valued short- and long-term memory states serving distinct functional roles. Building on the Continuous Thought Machine (CTM), the CMM's short-term memory tracks recent neural activity, with uniquely parameterized neuron-level models learning to use these activity patterns for computation. A persistent long-term memory stores information for later use, with a Transformer jointly updating both memory stores, providing an expressive bidirectional read--write mechanism such that each store can reorganize its own contents and both read from and write to the other. Across algorithmic, in-context learning, and recurrent reasoning tasks, the CMM outperforms a broad suite of baselines, exhibiting stronger generalization than prior memory-augmented networks while preserving the CTM's interpretable attention patterns. Code is available at https://github.com/SakanaAI/continuous-memory-machines.
Large Language Models (LLMs) have achieved remarkable reasoning capabilities by utilizing chain-of-thought (CoT) as a scratchpad for intermediate stages of thinking. However, CoT techniques require explicit supervision on thinking tokens, which requires rich, task-specific data. In this work, we propose Abstract Token Curriculum (ATC), a novel curriculum learning framework that elicits effective continuous intermediate representations without direct supervision or manual scratchpad design. ATC gradually increases problem complexity through a sequence of distributions, training the model to develop internal abstract thoughts'' in the continuous representation space. This paper provides both theoretical and experimental evidence for the benefits of ATC and its advantages over previous methods for training continuous thoughts. Theoretically, we show that for learning parity functions with single-layer softmax attention using ATC, attention naturally focuses on the CoT tokens in the context that provide the easiest path'' to predicting the next token. Experimentally, we show ATC's effectiveness on graph reachability and arithmetic learning tasks.