Infrared Universality of Collective Dynamics across Transformer and State-Space Architectures

Byung Gyu Chae

cs.LGAug 19, 2026

Infrared Universality of Collective Dynamics across Transformer and State-Space Architectures

Authors: Byung Gyu Chae

Organizations: Electronics and Telecommunications Research Institute, 218 Gajeong-ro, Yuseong-gu, Daejeon 34129, Republic of Korea

Abstract

Whether distinct neural architectures develop common collective dynamics remains an open question. Recent analysis of Transformer language models revealed a nearly flat, weakly infrared-enhanced time-scale density of states (TDOS) associated with near-marginal long-memory dynamics. Here we test whether a closely related organization emerges in Mamba, whose selective state-space dynamics provides a fundamentally different microscopic mechanism. Mamba allows relaxation dynamics to be resolved at three levels: the intrinsic spectrum of the learned state-space generator, its input-conditioned selective rescaling, and the collective TDOS of the complete block measured from its Jacobian. These spectra are not identical: selective dynamics and the remaining block transformations substantially reorganize the microscopic relaxation hierarchy. Nevertheless, the full block develops a reproducible slow-mode continuum whose infrared sector becomes progressively better resolved with increasing sequence length. Cumulative analysis yields $ρ(λ)\simλ^β$ , with the long-sequence Mamba exponent stabilizing near $β_{\rm M}\simeq-0.17$ . The corresponding memory dynamics follows $K(t)\sim t^{-(1+β)}$ , close to the marginal $1/t$ regime. Despite fundamentally different microscopic dynamics, Transformer full-block spectra exhibit closely related infrared organization, with representative exponents of order $β_{\rm Tr}\sim-0.1$ . These results separate explicit state-space memory from collective infrared organization and show that distinct sequence architectures can develop closely related near-marginal slow-mode dynamics. They extend infrared collective organization beyond Transformers and provide an independent test of the dynamical structure described by Cognitive Field Theory.

Figures & tables

Figure 1: Collective spectral analysis of a representative Mamba-370M block (block index $\ell=24$ , $L=128$ retained tokens). (a) Complex eigenvalue spectrum of the full-block Jacobian. (b) Eigenvalue magnitude as a function of phase angle. (c,d) Distributions of the real and imaginary parts of the complex eigenvalues. (e) Phase-angle distribution, strongly concentrated near $\theta=0$ . (f,g) Time-scale density of states (TDOS) obtained from the stable eigenvalue magnitudes through $\lambda=-\ln|\zeta|$ , shown on linear and logarithmic scales. The measured spectrum spans many orders of magnitude in relaxation rate and contains a broad population of slow collective modes. The reference scale $\lambda_{c}=5\times 10^{-2}$ is shown only as a visual guide to the slow-relaxation sector and does not define a physical spectral boundary. (h) Memory kernel obtained directly from the Laplace transform of the measured full-block TDOS, without assuming a particular spectral functional form. The quantitative sequence-length dependence, infrared convergence, and scaling of the collective slow-mode sector are analyzed in the following sections.

Configuration	Mamba-370M
Model parameters	370M
Number of Mamba layers, $N_{\rm layer}$	48
Hidden dimension, $d_{\rm model}$	1024
SSM state dimension, $d_{\rm state}$	16
Expansion factor	2
Local convolution width, $d_{\rm conv}$	4

Table 1: Configuration of the pretrained Mamba-370M model used throughout this work. The SSM parameters correspond to the standard Mamba configuration used by the released model.

Figure 2: Intrinsic relaxation spectrum of the learned SSM in Mamba-370M. (a) Distribution of the learned logarithmic state-space parameters $A_{\log}$ for a representative block. The broad distribution is mapped to intrinsic continuous-time relaxation rates through $\lambda_{A}=\exp(A_{\log})$ . (b) Corresponding intrinsic relaxation-rate density $\rho_{A}(\lambda_{A})$ on logarithmic axes. The spectrum extends over many orders of magnitude, from a slow-relaxation branch near $\lambda_{A}\sim 10^{-5}$ to rapidly relaxing modes beyond $\lambda_{A}\sim 10^{4}$ , demonstrating that a broad hierarchy of intrinsic time scales is already encoded in the learned SSM generator. (c) Intrinsic spectral relaxation kernel $K_{A}(t)=\int d\lambda_{A}\,\rho_{A}(\lambda_{A})e^{-\lambda_{A}t}$ computed directly from the measured intrinsic relaxation spectrum. The dashed $t^{-1}$ line is included only as a scaling reference. The intrinsic kernel reflects the nonuniform relaxation spectrum of the learned SSM and is not identified with the collective memory kernel or self-energy dressing associated with the full-block collective dynamics.

Figure 3: Intrinsic SSM relaxation spectra across all blocks of Mamba-370M. The intrinsic relaxation-rate density $\rho_{A}(\lambda_{A})$ , obtained from $\lambda_{A}=\exp(A_{\log})$ , is shown for all 48 Mamba blocks. Despite block-to-block variations, the spectra reproduce a common broad envelope over many orders of magnitude in relaxation rate, including a distributed slow-relaxation sector extending toward the deep infrared. The intermediate and rapidly relaxing sectors exhibit particularly strong overlap across depth, whereas larger fluctuations appear in the more sparsely populated infrared region. The persistence of the overall spectral structure throughout the network demonstrates that the broad intrinsic relaxation hierarchy is a distributed property of the learned Mamba state-space architecture rather than a feature of an isolated block.

Figure 4: Input-conditioned effective SSM relaxation dynamics in Mamba-370M at block index $\ell=24$ . (a) Effective relaxation-rate density constructed from the selectively rescaled rates $\lambda_{\rm eff}(x_{t})=\Delta(x_{t})e^{A_{\log}}$ for retained sequence lengths $L=8,16,32,64,128,256,$ and $512$ . The effective spectrum spans an exceptionally broad hierarchy of relaxation rates. Short sequences exhibit substantial fluctuations in the sparsely sampled infrared sector, whereas increasing sequence length produces progressively stronger overlap and convergence of the normalized spectral envelope. The dashed vertical line at $\lambda_{c}=0.05$ is shown only as a common reference for the slow-relaxation sector. (b) Distribution of the corresponding input-dependent selective time step $\Delta(x_{t})$ . The distributions are strongly weighted toward small $\Delta$ and become increasingly stable with sequence length. Because $\lambda_{\rm eff}=\Delta(x_{t})e^{A_{\log}}$ , small selective time steps shift intrinsic SSM modes toward smaller effective relaxation rates and thereby extend the effective spectrum toward longer time scales. Together, the two panels directly show how Mamba’s selective mechanism dynamically rescales the learned intrinsic relaxation hierarchy during sequence processing.

Figure 5: Sequence-length dependence of the full-block collective relaxation spectrum in Mamba-370M. The time-scale density of states is obtained from the eigenvalues

\zeta_{\alpha}

of the complete block Jacobian through

\lambda_{\alpha}=-\ln|\zeta_{\alpha}|

and is shown for sequence lengths

L=8

–

512

at layer 24. The measured collective spectrum spans approximately eight decades in relaxation rate, from

\lambda\sim 10^{-7}

\lambda\sim 10^{1}

. Short sequences exhibit visible finite-size fluctuations in the deep-infrared sector, whereas increasing sequence length progressively resolves slower modes and produces a smoother common spectral envelope. For

L=512

, the collective spectral profile remains comparatively smooth down to approximately

\lambda\sim 5\times 10^{-6}

, while individual modes extend further into the sparsely populated extreme-infrared regime near

\lambda\sim 10^{-7}

. The simultaneous extension and convergence of the low-

\lambda

sector indicate that the infrared continuum becomes progressively better resolved as a larger dynamical subspace is sampled.

Figure 6: Depth dependence of the full-block collective relaxation spectrum in Mamba-370M. The time-scale density of states is measured from full-block Jacobians at a fixed sequence length of $L=128$ for layer pairs distributed throughout the 48-layer network, from layers $1\!\to\!2$ to $45\!\to\!46$ . Despite quantitative variations in local spectral weight and in the high-relaxation-rate rolloff, the measured spectra reproduce a closely related organization across depth: a broad slow-relaxation sector, an extended weakly varying infrared region, and a rapid decay toward the ultraviolet. The collective spectra extend over multiple decades into the deep-infrared regime, while the overall spectral envelope remains similar across widely separated layers. The persistence of this low- $\lambda$ organization across widely separated depths indicates that the collective infrared structure is not restricted to a particular layer but is distributed throughout the trained Mamba model.

Figure 7: Cumulative infrared scaling of the full-block relaxation spectrum in Mamba-370M. The empirical cumulative number of stable relaxation modes, $N(<\lambda)$ , is shown at layer 24 for sequence lengths $L=8,16,32,64,128,256,$ and $512$ . For each sequence length, the cumulative spectrum is fitted over the same fixed infrared interval, $10^{-4}\leq\lambda\leq 5\times 10^{-2}$ , according to $N(<\lambda)\propto\lambda^{s}$ , with the corresponding TDOS exponent given by $\beta=s-1$ . The fitted exponent evolves from $\beta=-0.228$ at $L=8$ and stabilizes near $\beta\simeq-0.17$ for the longer sequences, while the coefficients of determination remain approximately $R^{2}\simeq 0.99$ throughout. Over the same measurements, the number of modes within the fixed fitting interval increases from $N_{\rm fit}=2,719$ at $L=8$ to $156,593$ at $L=512$ . Thus, the infrared statistics improve by nearly a factor of 58 while the cumulative scaling exponent remains essentially unchanged at the largest sequence lengths. The more sparsely sampled deep-infrared region below the common fitting interval is excluded from the exponent determination.

Tokens	$s$	$\beta$	$R^{2}$	$N_{\rm fit}$
8	0.772	-0.228	0.988	2,719
16	0.802	-0.198	0.989	5,194
32	0.818	-0.182	0.990	10,108
64	0.830	-0.170	0.991	19,801
128	0.835	-0.165	0.991	39,448
256	0.831	-0.169	0.992	78,388

Table 2: Infrared scaling of the Mamba-370M full-block Jacobian TDOS at layer 24. All cumulative spectra are fitted over the fixed interval

10^{-4}\leq\lambda\leq 5\times 10^{-2}

according to

N(<\lambda)\propto\lambda^{s}

, with

\beta=s-1

N_{\rm fit}

denotes the number of modes within the fitting interval.

Figure 8: Sequence-length dependence of the infrared TDOS exponent in the Mamba-370M full-block dynamics. The exponent $\beta$ is obtained from the cumulative scaling relation $N(<\lambda)\propto\lambda^{1+\beta}$ over the common fitting interval $10^{-4}\leq\lambda\leq 5\times 10^{-2}$ . As the retained sequence length increases from 8 to 512 tokens, $\beta$ evolves from $-0.228$ toward a stable value near $-0.17$ . For the four longest sequence lengths, $L=64,128,256,$ and $512$ , the measured values are approximately $\beta=-0.170,-0.165,-0.169,$ and $-0.168$ , respectively. Thus, despite an eightfold increase in sequence length from 64 to 512 tokens, the fitted exponent remains confined to a narrow interval below the exactly flat-TDOS limit. The dashed horizontal line denotes this flat-spectrum limit, $\beta=0$ . The stabilization of the measured exponent near $\beta_{\rm IR}\simeq-0.17$ , without a systematic drift toward zero, indicates a reproducible, nearly flat but weakly infrared-enhanced scaling regime over the resolved infrared window.

Figure 9: Comparison of the microscopic architectures and collective infrared dynamics of Transformer and Mamba. The two architectures generate hidden-state dynamics through fundamentally different computational mechanisms: Transformer blocks combine attention, nonlinear feed-forward transformations, and residual propagation, whereas Mamba blocks employ selective state-space evolution with input-dependent temporal modulation. Despite this difference, both architectures can be analyzed using the same full-block collective construction. For each block, the Jacobian of the complete hidden-state transformation is diagonalized and the collective relaxation rates are defined as $\lambda_{\alpha}=-\ln|\zeta_{\alpha}|$ , allowing the corresponding time-scale density of states $\rho(\lambda)$ to be compared on the same dynamical convention. The measured full-block spectra exhibit a closely related infrared organization: both contain a broad slow-mode continuum with a nearly flat but weakly enhanced TDOS toward small relaxation rates. Power-law analysis, $\rho(\lambda)\sim\lambda^{\beta}$ , yields weakly negative infrared exponents in both architectures. The Transformer measurements approach values of order $\beta_{\rm Tr}\sim-0.1$ , whereas the long-sequence Mamba measurements stabilize near $\beta_{\rm M}\sim-0.17$ . The numerical exponents are not identical; rather, the common feature is the regime $\beta<0$ with $|\beta|\ll 1$ . Thus, substantially different microscopic architectures produce closely related near-marginal collective infrared dynamics when examined through the complete hidden-state transformation.

Figure 10: Hierarchy from microscopic state-space relaxation to collective infrared organization in Mamba, together with the corresponding collective Transformer spectrum. The learned Mamba SSM first defines an intrinsic distribution of relaxation rates, $\lambda_{A}=\exp(A_{\log})$ . Input-dependent selective dynamics then temporally rescales these intrinsic modes according to $\lambda_{\rm eff}(t)=\Delta_{t}\lambda_{A}$ , producing an effective SSM spectrum conditioned on the processed sequence. After input and output projections, gating, nonlinear transformations, and residual propagation are included, the complete Mamba block is characterized independently by the eigenvalues of its full-block Jacobian and the collective rates $\lambda=-\ln|\zeta|$ . These three spectra represent distinct dynamical levels and should not be interpreted as identical spectral objects. Rather, the comparison shows that a broad slow-mode hierarchy present in the internal state-space dynamics remains manifest, after substantial spectral reorganization, in the collective response of the complete Mamba block. The Transformer full-block spectrum, obtained independently from the same Jacobian construction, exhibits a similarly broad and nearly flat infrared collective sector despite the absence of an explicit state-space relaxation hierarchy. The figure therefore illustrates the separation between microscopic dynamical implementation and the closely related infrared organization observed at the collective level.

Property	Transformer	Mamba-370M
Microscopic mechanism	Attention + MLP	Selective SSM
Temporal mechanism	Context-dependent mixing	Input-dependent state evolution
Collective observable	Full-block Jacobian TDOS	Full-block Jacobian TDOS
Block relaxation rate	$-\ln\|\zeta\|$	$-\ln\|\zeta\|$
Infrared form	$\rho(\lambda)\sim\lambda^{\beta}$	$\rho(\lambda)\sim\lambda^{\beta}$
Representative $\beta$	$\sim-0.1$	$\sim-0.17$

Table 3: Comparison of the collective infrared dynamics measured in Transformer and Mamba. The infrared exponents denote representative resolved scaling behavior and are not expected to be numerically identical between architectures. The corresponding long-time behavior follows from the TDOS–memory relation established in the preceding sections.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure C1: FP64 numerical validation of the Mamba-370M full-block Jacobian TDOS. (a) Full-block time-scale density of states at layer 24 obtained from FP64 Jacobian calculations for retained sequence lengths $L=32$ , $64$ , and $128$ . The corresponding FP32 spectrum for $L=128$ is overlaid as a black dashed curve with square markers. The FP32 and FP64 measurements recover essentially the same collective spectral envelope throughout the statistically well-resolved regime, including the broad slow-relaxation sector, the intermediate spectral region, and the ultraviolet rolloff. (b) The same spectra restricted to the infrared region $\lambda\leq 0.05$ using common logarithmic bins. The agreement between FP32 and FP64 persists throughout the resolved infrared continuum, while larger bin-to-bin fluctuations appear in the sparsely populated extreme-infrared tail. The FP64 measurements further resolve the collective spectrum toward relaxation rates of order $\lambda\sim 10^{-8}$ . The persistence of the same large-scale spectral organization under increased numerical precision demonstrates that the observed collective infrared structure is not an artifact of the arithmetic precision used in the full-block Jacobian calculation.

Figure D1: Memory kernels constructed from the full-block Jacobian relaxation spectrum of Mamba-370M at layer 24 for retained sequence lengths $L=8$ – $512$ . (a) Full-spectrum memory kernel $K_{\rm full}(t)=N^{-1}\sum_{\alpha}e^{-\lambda_{\alpha}t}$ , evaluated using all measured stable relaxation modes. Despite the substantial increase in the number of resolved modes with sequence length, the kernels exhibit closely overlapping short- and intermediate-time behavior, while differences become increasingly visible at long times as larger sequence subspaces resolve additional deep-infrared modes. The resulting extension of the long-time tail directly reflects the progressively resolved low-relaxation-rate spectral weight. (b) Infrared memory kernel $K_{\rm IR}(t)$ constructed using only modes within the common scaling interval $10^{-4}\leq\lambda\leq 5\times 10^{-2}$ employed for the cumulative TDOS analysis. For a power-law TDOS $\rho(\lambda)\sim\lambda^{\beta}$ , the corresponding intermediate-time kernel obeys $K_{\rm IR}(t)\sim t^{-(1+\beta)}$ over $\lambda_{\max}^{-1}\ll t\ll\lambda_{\min}^{-1}$ . The vertical dotted lines mark the inverse spectral boundaries, $t_{\rm UV}\simeq 20$ and $t_{\rm IR}=10^{4}$ . The dashed $t^{-0.83}$ reference corresponds to the measured long-sequence exponent $\beta_{\rm IR}\simeq-0.17$ , while the dotted $t^{-1}$ line denotes the exactly flat-TDOS limit $\beta=0$ . These reference lines are scaling guides rather than fits to the kernel. Beyond $t_{\rm IR}$ , the finite lower cutoff of the selected infrared window produces the expected rapid decay. Together, the two panels show how the measured collective relaxation spectrum is transformed into extended temporal memory and distinguish the long-time contribution of the complete deep-infrared spectrum from the scaling behavior associated with the fixed resolved infrared window.

Dynamical Systems

Dynamical systems studies how states evolve over time, including stability, feedback, chaos, and transitions in physical, biological, and engineered systems.

See all 478 papers

Jul 12, 2026

Infrared Organization and Critical Cognitive Field Formation in Transformer Dynamics

Large language models exhibit remarkable emergent behaviors, yet the physical mechanism governing their collective dynamics remains poorly understood. Cognitive Field Theory predicts that learning reorganizes the time-scale density of states (TDOS) through the infrared accumulation of slow relaxation modes, thereby enhancing the memory self-energy, reducing the cognitive forgetting gap, and strengthening the collective susceptibility. Using publicly available Pythia language models, we extract relaxation spectra directly from Transformer layer Jacobians throughout training, network depth, and model scale, allowing the TDOS, memory self-energy, forgetting gap, memory kernel, and infrared critical exponent to be measured quantitatively. The measurements reveal progressive infrared accumulation of slow relaxation modes, producing an approximately flat infrared TDOS with

ρ(λ)\simλ^{-0.1}

and scale-free memory kernels

K(t)\sim t^{-1}.

The memory self-energy exhibits a pronounced transient maximum during early optimization before relaxing toward a metastable near-critical regime, corresponding to the smallest cognitive forgetting gap and the largest collective susceptibility predicted by Cognitive Field Theory. These observations provide quantitative experimental evidence that Transformer dynamics are governed by infrared collective organization. The reproducibility of the same dynamical behavior across training, network depth, and model scales suggests that infrared slow-mode organization represents a universal collective principle of Transformer dynamics.

Byung Gyu Chae

Electronics and Telecommunications Research Institute, 218 Gajeong-ro, Yuseong-gu, Daejeon 34129, Republic of Korea

Transformer Neural Network Training Dynamics

Ask PDF

Sep 21, 2026

Comparing Latent Concept Formation in State Space Models and Transformers via Sparse Autoencoders

The quadratic scaling of Transformer self-attention has driven the adoption of sub-quadratic Selective State Space Models (SSMs) like Mamba, which compress past context into a fixed-size recurrent hidden state. This strict informational bottleneck raises a foundational question for mechanistic interpretability: do SSMs and Transformers learn fundamentally distinct latent representations? In this work, we employ Sparse Autoencoders (SAEs) to conduct a large-scale, feature-level correspondence analysis between Mamba-130m and Pythia-70m over a 10-million token corpus. Contrary to hypotheses predicting widespread architectural divergence, we find no evidence of systematic representational divergence between architectures: across the observed Jaccard distribution, 99.98% of Mamba features cluster toward the upper alignment boundary, providing preliminary feature-level support for the Universality Hypothesis. We further identify and qualitatively characterize this microscopic fraction (0.02%) of diverging features, finding patterns consistent with the hypothesis that the recurrent bottleneck selectively limits the parsing of rigid syntax rather than broad semantic ontology. We demonstrate that while Pythia's unconstrained attention permits the monosemantic decomposition of distinct formatting edge-cases, Mamba is forced to compress unrelated syntactical anomalies into polysemantic "junk drawer" neurons to preserve state capacity. Collectively, these results suggest that architectural routing mechanisms may have negligible impact on core semantic understanding, with representational divergence confined to extreme structural margins.

Rithin Nagaraj, Rupa Laalasa Oruganti, Prerna Subhashchandra Kunder +1

Department of Computer Science and Engineering PES University Bangalore, India

Transformer Interpretability Language Modeling Representation Geometry in Language Models Representation Alignment State Space Models

Ask PDF

Feb 14, 2025

From Markov to Laplace: How Mamba In-Context Learns Markov Chains

While transformer-based language models have driven the AI revolution thus far, their computational complexity has spurred growing interest in viable alternatives, such as structured state space sequence models (SSMs) and Selective SSMs. Among these, Mamba (S6) and its variant Mamba-2 have shown remarkable inference speed-ups over transformers while achieving comparable or superior performance on complex language modeling tasks. However, despite these architectural innovations and empirical successes, the fundamental learning capabilities of Mamba remain poorly understood. In this paper, we address this gap by studying in-context learning (ICL) on Markov chains and uncovering an interesting phenomenon: even a single-layer Mamba efficiently learns the in-context Laplacian smoothing estimator, which is both Bayes and minimax optimal. To explain this, we theoretically characterize the representation capacity of Mamba and reveal the fundamental role of convolution in enabling it to represent the optimal Laplacian smoothing. These theoretical insights align strongly with empirical results and, to the best of our knowledge, represent the first formal connection between Mamba and optimal statistical estimators. Finally, we outline promising research directions inspired by these findings.

Marco Bondaschi, Nived Rajaraman, Xiuying Wei +5

EPFL · UC Berkeley · Google DeepMind +1

Mamba Selective SSMs In-Context Learning

Ask PDF

Abstract

Dynamical Systems

Explore similar work

Infrared Organization and Critical Cognitive Field Formation in Transformer Dynamics

Comparing Latent Concept Formation in State Space Models and Transformers via Sparse Autoencoders

From Markov to Laplace: How Mamba In-Context Learns Markov Chains