cs.LGAug 19, 2026
SaveInfrared Universality of Collective Dynamics across Transformer and State-Space Architectures
Organizations: Electronics and Telecommunications Research Institute, 218 Gajeong-ro, Yuseong-gu, Daejeon 34129, Republic of Korea
Abstract
Whether distinct neural architectures develop common collective dynamics remains an open question. Recent analysis of Transformer language models revealed a nearly flat, weakly infrared-enhanced time-scale density of states (TDOS) associated with near-marginal long-memory dynamics. Here we test whether a closely related organization emerges in Mamba, whose selective state-space dynamics provides a fundamentally different microscopic mechanism. Mamba allows relaxation dynamics to be resolved at three levels: the intrinsic spectrum of the learned state-space generator, its input-conditioned selective rescaling, and the collective TDOS of the complete block measured from its Jacobian. These spectra are not identical: selective dynamics and the remaining block transformations substantially reorganize the microscopic relaxation hierarchy. Nevertheless, the full block develops a reproducible slow-mode continuum whose infrared sector becomes progressively better resolved with increasing sequence length. Cumulative analysis yields , with the long-sequence Mamba exponent stabilizing near . The corresponding memory dynamics follows , close to the marginal regime. Despite fundamentally different microscopic dynamics, Transformer full-block spectra exhibit closely related infrared organization, with representative exponents of order . These results separate explicit state-space memory from collective infrared organization and show that distinct sequence architectures can develop closely related near-marginal slow-mode dynamics. They extend infrared collective organization beyond Transformers and provide an independent test of the dynamical structure described by Cognitive Field Theory.
Figures & tables
Figure 1: Collective spectral analysis of a representative Mamba-370M block (block index , retained tokens). (a) Complex eigenvalue spectrum of the full-block Jacobian. (b) Eigenvalue magnitude as a function of phase angle. (c,d) Distributions of the real and imaginary parts of the complex eigenvalues. (e) Phase-angle distribution, strongly concentrated near . (f,g) Time-scale density of states (TDOS) obtained from the stable eigenvalue magnitudes through , shown on linear and logarithmic scales. The measured spectrum spans many orders of magnitude in relaxation rate and contains a broad population of slow collective modes. The reference scale is shown only as a visual guide to the slow-relaxation sector and does not define a physical spectral boundary. (h) Memory kernel obtained directly from the Laplace transform of the measured full-block TDOS, without assuming a particular spectral functional form. The quantitative sequence-length dependence, infrared convergence, and scaling of the collective slow-mode sector are analyzed in the following sections.
| Configuration | Mamba-370M |
| Model parameters | 370M |
| Number of Mamba layers, | 48 |
| Hidden dimension, | 1024 |
| SSM state dimension, | 16 |
| Expansion factor | 2 |
| Local convolution width, | 4 |
Table 1: Configuration of the pretrained Mamba-370M model used throughout this work. The SSM parameters correspond to the standard Mamba configuration used by the released model.
Figure 2: Intrinsic relaxation spectrum of the learned SSM in Mamba-370M. (a) Distribution of the learned logarithmic state-space parameters for a representative block. The broad distribution is mapped to intrinsic continuous-time relaxation rates through . (b) Corresponding intrinsic relaxation-rate density on logarithmic axes. The spectrum extends over many orders of magnitude, from a slow-relaxation branch near to rapidly relaxing modes beyond , demonstrating that a broad hierarchy of intrinsic time scales is already encoded in the learned SSM generator. (c) Intrinsic spectral relaxation kernel computed directly from the measured intrinsic relaxation spectrum. The dashed line is included only as a scaling reference. The intrinsic kernel reflects the nonuniform relaxation spectrum of the learned SSM and is not identified with the collective memory kernel or self-energy dressing associated with the full-block collective dynamics.
Figure 3: Intrinsic SSM relaxation spectra across all blocks of Mamba-370M. The intrinsic relaxation-rate density , obtained from , is shown for all 48 Mamba blocks. Despite block-to-block variations, the spectra reproduce a common broad envelope over many orders of magnitude in relaxation rate, including a distributed slow-relaxation sector extending toward the deep infrared. The intermediate and rapidly relaxing sectors exhibit particularly strong overlap across depth, whereas larger fluctuations appear in the more sparsely populated infrared region. The persistence of the overall spectral structure throughout the network demonstrates that the broad intrinsic relaxation hierarchy is a distributed property of the learned Mamba state-space architecture rather than a feature of an isolated block.
Figure 4: Input-conditioned effective SSM relaxation dynamics in Mamba-370M at block index . (a) Effective relaxation-rate density constructed from the selectively rescaled rates for retained sequence lengths and . The effective spectrum spans an exceptionally broad hierarchy of relaxation rates. Short sequences exhibit substantial fluctuations in the sparsely sampled infrared sector, whereas increasing sequence length produces progressively stronger overlap and convergence of the normalized spectral envelope. The dashed vertical line at is shown only as a common reference for the slow-relaxation sector. (b) Distribution of the corresponding input-dependent selective time step . The distributions are strongly weighted toward small and become increasingly stable with sequence length. Because , small selective time steps shift intrinsic SSM modes toward smaller effective relaxation rates and thereby extend the effective spectrum toward longer time scales. Together, the two panels directly show how Mamba’s selective mechanism dynamically rescales the learned intrinsic relaxation hierarchy during sequence processing.
Figure 5: Sequence-length dependence of the full-block collective relaxation spectrum in Mamba-370M. The time-scale density of states is obtained from the eigenvalues of the complete block Jacobian through and is shown for sequence lengths – at layer 24. The measured collective spectrum spans approximately eight decades in relaxation rate, from to . Short sequences exhibit visible finite-size fluctuations in the deep-infrared sector, whereas increasing sequence length progressively resolves slower modes and produces a smoother common spectral envelope. For , the collective spectral profile remains comparatively smooth down to approximately , while individual modes extend further into the sparsely populated extreme-infrared regime near . The simultaneous extension and convergence of the low- sector indicate that the infrared continuum becomes progressively better resolved as a larger dynamical subspace is sampled.
Figure 6: Depth dependence of the full-block collective relaxation spectrum in Mamba-370M. The time-scale density of states is measured from full-block Jacobians at a fixed sequence length of for layer pairs distributed throughout the 48-layer network, from layers to . Despite quantitative variations in local spectral weight and in the high-relaxation-rate rolloff, the measured spectra reproduce a closely related organization across depth: a broad slow-relaxation sector, an extended weakly varying infrared region, and a rapid decay toward the ultraviolet. The collective spectra extend over multiple decades into the deep-infrared regime, while the overall spectral envelope remains similar across widely separated layers. The persistence of this low- organization across widely separated depths indicates that the collective infrared structure is not restricted to a particular layer but is distributed throughout the trained Mamba model.
Figure 7: Cumulative infrared scaling of the full-block relaxation spectrum in Mamba-370M. The empirical cumulative number of stable relaxation modes, , is shown at layer 24 for sequence lengths and . For each sequence length, the cumulative spectrum is fitted over the same fixed infrared interval, , according to , with the corresponding TDOS exponent given by . The fitted exponent evolves from at and stabilizes near for the longer sequences, while the coefficients of determination remain approximately throughout. Over the same measurements, the number of modes within the fixed fitting interval increases from at to at . Thus, the infrared statistics improve by nearly a factor of 58 while the cumulative scaling exponent remains essentially unchanged at the largest sequence lengths. The more sparsely sampled deep-infrared region below the common fitting interval is excluded from the exponent determination.
| Tokens | ||||
|---|---|---|---|---|
| 8 | 0.772 | -0.228 | 0.988 | 2,719 |
| 16 | 0.802 | -0.198 | 0.989 | 5,194 |
| 32 | 0.818 | -0.182 | 0.990 | 10,108 |
| 64 | 0.830 | -0.170 | 0.991 | 19,801 |
| 128 | 0.835 | -0.165 | 0.991 | 39,448 |
| 256 | 0.831 | -0.169 | 0.992 | 78,388 |
Table 2: Infrared scaling of the Mamba-370M full-block Jacobian TDOS at layer 24. All cumulative spectra are fitted over the fixed interval according to , with . denotes the number of modes within the fitting interval.
Figure 8: Sequence-length dependence of the infrared TDOS exponent in the Mamba-370M full-block dynamics. The exponent is obtained from the cumulative scaling relation over the common fitting interval . As the retained sequence length increases from 8 to 512 tokens, evolves from toward a stable value near . For the four longest sequence lengths, and , the measured values are approximately and , respectively. Thus, despite an eightfold increase in sequence length from 64 to 512 tokens, the fitted exponent remains confined to a narrow interval below the exactly flat-TDOS limit. The dashed horizontal line denotes this flat-spectrum limit, . The stabilization of the measured exponent near , without a systematic drift toward zero, indicates a reproducible, nearly flat but weakly infrared-enhanced scaling regime over the resolved infrared window.
Figure 9: Comparison of the microscopic architectures and collective infrared dynamics of Transformer and Mamba. The two architectures generate hidden-state dynamics through fundamentally different computational mechanisms: Transformer blocks combine attention, nonlinear feed-forward transformations, and residual propagation, whereas Mamba blocks employ selective state-space evolution with input-dependent temporal modulation. Despite this difference, both architectures can be analyzed using the same full-block collective construction. For each block, the Jacobian of the complete hidden-state transformation is diagonalized and the collective relaxation rates are defined as , allowing the corresponding time-scale density of states to be compared on the same dynamical convention. The measured full-block spectra exhibit a closely related infrared organization: both contain a broad slow-mode continuum with a nearly flat but weakly enhanced TDOS toward small relaxation rates. Power-law analysis, , yields weakly negative infrared exponents in both architectures. The Transformer measurements approach values of order , whereas the long-sequence Mamba measurements stabilize near . The numerical exponents are not identical; rather, the common feature is the regime with . Thus, substantially different microscopic architectures produce closely related near-marginal collective infrared dynamics when examined through the complete hidden-state transformation.
Figure 10: Hierarchy from microscopic state-space relaxation to collective infrared organization in Mamba, together with the corresponding collective Transformer spectrum. The learned Mamba SSM first defines an intrinsic distribution of relaxation rates, . Input-dependent selective dynamics then temporally rescales these intrinsic modes according to , producing an effective SSM spectrum conditioned on the processed sequence. After input and output projections, gating, nonlinear transformations, and residual propagation are included, the complete Mamba block is characterized independently by the eigenvalues of its full-block Jacobian and the collective rates . These three spectra represent distinct dynamical levels and should not be interpreted as identical spectral objects. Rather, the comparison shows that a broad slow-mode hierarchy present in the internal state-space dynamics remains manifest, after substantial spectral reorganization, in the collective response of the complete Mamba block. The Transformer full-block spectrum, obtained independently from the same Jacobian construction, exhibits a similarly broad and nearly flat infrared collective sector despite the absence of an explicit state-space relaxation hierarchy. The figure therefore illustrates the separation between microscopic dynamical implementation and the closely related infrared organization observed at the collective level.
| Property | Transformer | Mamba-370M |
|---|---|---|
| Microscopic mechanism | Attention + MLP | Selective SSM |
| Temporal mechanism | Context-dependent mixing | Input-dependent state evolution |
| Collective observable | Full-block Jacobian TDOS | Full-block Jacobian TDOS |
| Block relaxation rate | ||
| Infrared form | ||
| Representative |
Table 3: Comparison of the collective infrared dynamics measured in Transformer and Mamba. The infrared exponents denote representative resolved scaling behavior and are not expected to be numerically identical between architectures. The corresponding long-time behavior follows from the TDOS–memory relation established in the preceding sections.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure C1: FP64 numerical validation of the Mamba-370M full-block Jacobian TDOS. (a) Full-block time-scale density of states at layer 24 obtained from FP64 Jacobian calculations for retained sequence lengths , , and . The corresponding FP32 spectrum for is overlaid as a black dashed curve with square markers. The FP32 and FP64 measurements recover essentially the same collective spectral envelope throughout the statistically well-resolved regime, including the broad slow-relaxation sector, the intermediate spectral region, and the ultraviolet rolloff. (b) The same spectra restricted to the infrared region using common logarithmic bins. The agreement between FP32 and FP64 persists throughout the resolved infrared continuum, while larger bin-to-bin fluctuations appear in the sparsely populated extreme-infrared tail. The FP64 measurements further resolve the collective spectrum toward relaxation rates of order . The persistence of the same large-scale spectral organization under increased numerical precision demonstrates that the observed collective infrared structure is not an artifact of the arithmetic precision used in the full-block Jacobian calculation.
Figure D1: Memory kernels constructed from the full-block Jacobian relaxation spectrum of Mamba-370M at layer 24 for retained sequence lengths – . (a) Full-spectrum memory kernel , evaluated using all measured stable relaxation modes. Despite the substantial increase in the number of resolved modes with sequence length, the kernels exhibit closely overlapping short- and intermediate-time behavior, while differences become increasingly visible at long times as larger sequence subspaces resolve additional deep-infrared modes. The resulting extension of the long-time tail directly reflects the progressively resolved low-relaxation-rate spectral weight. (b) Infrared memory kernel constructed using only modes within the common scaling interval employed for the cumulative TDOS analysis. For a power-law TDOS , the corresponding intermediate-time kernel obeys over . The vertical dotted lines mark the inverse spectral boundaries, and . The dashed reference corresponds to the measured long-sequence exponent , while the dotted line denotes the exactly flat-TDOS limit . These reference lines are scaling guides rather than fits to the kernel. Beyond , the finite lower cutoff of the selected infrared window produces the expected rapid decay. Together, the two panels show how the measured collective relaxation spectrum is transformed into extended temporal memory and distinguish the long-time contribution of the complete deep-infrared spectrum from the scaling behavior associated with the fixed resolved infrared window.
More in this topic
Dynamical Systems
Dynamical systems studies how states evolve over time, including stability, feedback, chaos, and transitions in physical, biological, and engineered systems.
See all 478 papers