MIRA: Mitigating Pole Geometry Bias in Recurrent Language Models

Róisín Luo; Ihsan Ullah; Christian Gagné; Dong Li; Karyn Morrissey

cs.AINov 6, 2026

MIRA: Mitigating Pole Geometry Bias in Recurrent Language Models

Authors: Róisín Luo, Ihsan Ullah, Christian Gagné, Dong Li, Karyn Morrissey

Organizations: University of Galway · Research Ireland – D-REAL · Université Laval (IID) · Canada–CIFAR AI Chair ·

Abstract

We show that the Transformer's recurrent successors --- such as selective state-space models (SSMs), Receptance Weighted Key Value (RWKV), and Retentive Network (RetNet) --- can be characterized as infinite-impulse-response (IIR) systems. The zero--pole geometries of these IIR systems characterize the temporal modes they can represent. However, to trade parallelism, the algebraic structures of the parameterizations of these recurrent systems often give rise to poles constrained to the real axis, thereby inducing an architectural bias in their recurrent dynamics. To unify these recurrent architectures and eliminate their pole geometry bias, inspired by the lattice--ladder realization of general IIR systems, we present Mira (Modal IIR Recurrent Architecture), a unified family of linear-time recurrent architectures for building large language models (LLMs) with a highly parallel algebraic structure. Each Mira block realizes a higher-order IIR system that represents temporal modes from token streams through a lattice--ladder structure, followed by a feed-forward network (FFN). This realization enables the model to represent the temporal modes associated with both real and complex-conjugate poles. We pretrain Mira-S, a 12-layer small-scale LLM with 0.14B parameters composed entirely of Mira blocks. Throughout the pretraining, Mira-S achieves lower perplexity than strong recurrent baselines, exhibits stronger context-window generalization beyond the training length, and attains higher throughput with a dedicated CUDA kernel.

Figures & tables

Method	$A(z)$	$B(z)$	Zeroes	Poles
SDPA ( Vaswani et al., 2017 )	$\displaystyle\sum_{\tau=0}^{T-1}\alpha_{t,\tau}z^{-\tau}$	$1$	Complex	–
Selective SSM (Mamba-1, S6) ( Gu and Dao, 2024 )	$\displaystyle C\,\operatorname{adj}(I-Az^{-1})B+D\,\det(I-Az^{-1})$	$\displaystyle\det(I-Az^{-1})$	Complex	Real $(0,1)$
RWKV (v4–v6) ( Peng et al., 2023 )	$1$	$\displaystyle I-Wz^{-1}$	–	Real $(0,1)$
RetNet (v1) ( Sun et al., 2023 )	$1$	$\displaystyle 1-\gamma z^{-1}$	–	Real $[0.968,1)$
HGRN2 (v2) ( Qin et al., 2024 )	$\displaystyle B_{t}$	$\displaystyle I-G_{t}z^{-1}$	–	Real $(0.125,1)$
Hyena (v1) ( Poli et al., 2023 )	$\displaystyle\sum_{\tau=0}^{T-1}h_{\tau}z^{-\tau}$	$1$	Complex	–

Table 1: Transfer-Function Characterization.

A(z)

and

B(z)

denote the numerator (FIR) and denominator (IIR) polynomials of the transfer function

H(z)=A(z)/B(z)

. The transfer functions are input-dependent. The roots of

A(z)=0

are the zeros and the roots of

B(z)=0

are the poles . We also verify the derived transfer functions numerically, comparing the sequence each predicts against the measured output of its recurrence, with a maximum discrepancy of

3.6\times 10^{-12}

; the pole ranges are verified formally, in Lean 4 ( Moura and Ullrich, 2021 ) and Z3 ( De Moura and Bjørner, 2008 ) .

Figure 1: Provable Failure of Causal Mode Retrieval. We synthesize

4

-dimensional continuous input tokens

x_{1:200}

and target tokens

y_{1:200}

of a length

200

. The ground-truth transfer function contains poles that are not on the real axis. We use parameter-matched single-layer models trained to predict next-token

y_{t+1}

from input sequence

x_{1:t}

by using MSE (mean squared error) loss. Each model is optimized for 1000 steps using AdamW ( Loshchilov and Hutter, 2019 ) with a learning rate of

10^{-2}

. Left: synthetic temporal signals of length 200. Middle: pole geometries of Mira , Mamba-S6, RWKV-7, and SDPA. Right: next-token prediction error with logarithmic scale. The results show that Mira admits complex poles and more accurately retrieves the temporal modes associated with complex poles. The provable failure signal synthesis is provided in Appendix A.6 .

Figure 2: Mira Block & Reference LLM. The diagram illustrates lattice circuit at stage

k

(top left), an order-

K

lattice module (bottom left), a Mira block (top right), and a reference implementation of the Mira LLM composed entirely of Mira blocks (bottom right). In the order-

K

lattice module, a rank-one signal mixer forms the input representations through channel-wise signal mixing, the dynamical-modulation signal modulates the lattice–ladder dynamics, the lattice module implements the order-

K

lattice dynamics, and the ladder readout produces the output representations.

Panel of Figure 3 — Figure 3: Algebraic Structure of the Lattice Realization. Each node $(k,t)$ represents the recursive state transition $\bigl(U_{t}^{(k)},V_{t-1}^{(k-1)}\bigr)\mapsto\bigl(U_{t}^{(k-1)},V_{t}^{(k)}\bigr)$ at lattice stage $k\in\{K,\ldots,1\}$ and token position $t\in\{1,\ldots,T\}$ , with edges encoding dependencies on $(k+1,t)$ and $(k-1,t-1)$ .

Figure 4: Effects of Lattice Order

K

, State-Write Bottleneck Size

d_{r}

, and Memory Size

d_{m}

. Using Mira-XS (65M), we independently vary the lattice order

K\in\{1,2,4,8\}

, state-write bottleneck size

d_{r}\in\{d/64,d/32,d/16,d/8\}

, and memory size

d_{m}\in\{8,16,32,64\}

. We match model sizes by varying only the model width

d

and pretrain each model for

10{,}000

steps on FineWeb-Edu using AdamW . The results show that models admitting complex-conjugate poles (

K>1

) achieve lower perplexity than the model with real poles (

K=1

). Among the evaluated state-write bottleneck sizes,

d_{r}=d/8

yields the lowest perplexity. Perplexity generally decreases as

d_{m}

increases, reaching its lowest value at

d_{m}=64

Figure 5: Performance of Dedicated CUDA Kernel. We compare Mira module with parameter-matched baseline modules on an NVIDIA RTX A6000 GPU. Execution time: at a context length of

32K

, Mira completes a forward pass in

11.37

ms, yielding speedups of

4.13\times

1.32\times

2.15\times

, and

2.85\times

over MHA (Multi-Head Attention) with FlashAttention, Mamba (S6), RWKV-6, and RWKV-7, respectively (left). Throughput: At the same context length, Mira reaches

8.36

million tokens per second, exceeding the throughput of the respective baselines by factors of

6.41\times

2.20\times

1.02\times

, and

1.28\times

(right).

Model	Size (M)	State (‰)	FineWeb-Edu						WikiText-103						HellaSwag
			512			32K			512			32K			512			32K
			Loss	PPL	Acc (%)	Loss	PPL	Acc (%)	Loss	PPL	Acc (%)	Loss	PPL	Acc (%)	Loss	PPL	Acc (%)	Loss	PPL	Acc (%)
Transformer (MHA)	142.52	0.000	3.400	29.96	35.31	7.317	$1.51\times 10^{3}$	4.58	3.468	32.08	36.01	7.136	$1.26\times 10^{3}$	6.25	3.434	31.00	38.39	7.134	$1.25\times 10^{3}$	1.43
RWKV-6	144.64	4.418	3.465	31.97	34.52	3.752	42.59	20.18	3.525	33.94	35.10	3.764	43.12	21.77	3.523	33.88	35.47	3.814	45.34	19.21
RWKV-7	142.66	3.634	3.485	32.63	34.26	3.423	30.65	34.47	3.587	36.12	34.58	3.519	33.75	34.16	3.644	38.23	35.11	3.544	34.59	34.94
Mamba-1	148.73	2.107	3.582	35.93	33.70	50.092	$5.68\times 10^{21}$	1.31	3.566	35.36	34.62	52.096	$4.22\times 10^{22}$	1.02	3.680	39.63	34.05	46.162	$1.12\times 10^{20}$	1.02

Table 2: Benchmark Performance. Validation loss, perplexity (PPL) and next-token accuracy at context lengths of 512 and 32,768 tokens. The training context window is 512 tokens. FineWeb-Edu is scored on the held-out last 5% of the corpus; WikiText-103 and HellaSwag on their test splits. State denotes the ratio of the total recurrent state size across all layers to the model parameter count, expressed in parts per thousand.

Figure 6: Context-Window Generalization. We evaluate the context-window generalization for the pretrained Mira on a holdout set of FineWeb-Edu , WikiText-103 , and HellaSwag , respectively. The dotted vertical line marks the training context length. Mira attains the lowest perplexity among the recurrent baselines and matches or improves on the MHA reference.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Dynamics of Pretraining Benchmark. We implement a small-scale reference LLM composed entirely of Mira blocks, denoted as Mira-S . Pretraining results show that Mira-S achieves the lowest perplexity among strong recurrent language model baselines and Transformer-based MHA, while attaining higher token-prediction accuracy than the recurrent baselines and comparable token-prediction accuracy to MHA.

Figure 8: Final-Stage Dynamics of the Pretraining Benchmark. Training loss over the final stage of pretraining for Mira-S and the baseline models.

Figure 9: Zero-Shot & Few-Shot Prompting.

Figure 10: ARC Reasoning with Zero-Shot & Few-Shot Prompting.

Panel of Figure 11 — Figure 11: Context-window generalization experiments.

Panel of Figure 12 — Figure 12: Extra context-window generalization experiments.

Figure 13: Profiling rotation-angle dynamics ( $\theta_{t}$ ).

Figure 14: Profiling pole-radius dynamics ( $r_{t}$ ).

Figure 15: Characteristic-horizon dynamics ( $H_{t}$ ).

Figure 16: Initializer ablation experiment.

Figure 17: Visualizing memory-state projection.

Figure 18: Visualizing the projection of GPT-2’s last-layer hidden states for reference.

Figure 19: BPE Tokenizer benchmark.

Figure 20: Modulus map.

Figure 21: Argument map.

$d$	$d_{e}$	$d_{r}$	$d_{m}$	$d_{\mathrm{ffn}}$	$K$	$P_{\mathrm{mixer}}$	$P_{\mathrm{layer}}$	$S_{\mathrm{layer}}$	$S_{\mathrm{layer}}/P_{\mathrm{layer}}$ (‰)
768	1536	48	16	3072	2	4.119 M	8.843 M	49.152 K	5.56
768	1536	48	16	3072	4	4.572 M	9.296 M	98.304 K	10.57
768	1536	48	16	3072	8	5.478 M	10.202 M	196.608 K	19.27
768	1536	96	16	3072	2	4.674 M	9.398 M	49.152 K	5.23
768	1536	96	16	3072	4	5.571 M	10.295 M	98.304 K	9.55
768	1536	96	16	3072	8	7.365 M	12.089 M	196.608 K	16.26

Appendix

Table 24

Model	Configuration	Parameter size	State size	State-to-parameter ratio
Mira-S	Width 768; FFN width 3072; lattice order 2; lattice width 1536; write bottleneck 48; memory dimension 16	144,715,392	589,824	0.004076
Transformer	Width 912; FFN width 2587; 12 attention heads; head dimension 76; RoPE	142,516,452	11,206,656	0.078634
RWKV-6	Width 832; FFN width 2836; 13 heads; head dimension 64	144,637,296	638,976	0.004418
RWKV-7	Width 720; FFN width 3277; 12 heads; head dimension 60	142,664,124	518,400	0.003634
Mamba-1	Width 816; FFN width 2891; inner width 1632; state dimension 16; convolution width 4; time-step rank 51	148,731,108	313,344	0.002107
Mamba-2	Width 816; FFN width 2891; inner width 1632; state dimension 128; 12 heads; head dimension 136; convolution width 4; one group	148,413,972	2,506,752	0.016890

Appendix

Table 3: Benchmark configurations. All models use 12 layers, a vocabulary of 50,259 tokens, tied input and output embeddings, and a training context window of 512 tokens. Parameter size counts all trainable parameters. State size counts recurrent-state elements per sequence across all layers, excluding convolution buffers; for the Transformer, it counts key–value cache elements at 512 tokens. The state-to-parameter ratio is the state size divided by the parameter size.

Panel of Figure 22 — Figure 22: Chunk summarization (i), boundary composition (ii) and chunk replay (iii) for one stream, read from the bottom up: $T$ tokens in chunks of $L=2$ tokens, and $P\leq 32$ threads of two chunks each (threads $4$ to $P-1$ not drawn). A box holds the pair $(\Phi_{s\to t},\mathbf{S}_{s\to t})$ of equation ( 5 ); two adjacent pairs compose by equation ( 4 ). In the successive scan steps, a thread composes its pair with that of the thread $1,2,\ldots,P/2$ positions to its left (dashed). For $P>32$ , the steps run within each warp of $32$ threads, and each thread also composes the pair of every earlier warp.

Recurrent Language Models

See all 40 papers

May 26, 2026

Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior

We study Latent Recurrent Transformer (LRT), a lightweight augmentation of autoregressive transformers that reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token. Because this state is already computed during ordinary decoding, LRT introduces a cross-token, cross-layer latent pathway while preserving the standard attention mechanism, KV-cache interface, and one model forward per generated token. To pretrain this recurrence without sequentially unrolling the full sequence, we introduce interleaved parallel training: one full-sequence initialization forward constructs a shared buffer, followed by sequential refinement of disjoint position subsets with parallel computation within each subset. This provides every token with recurrent-memory-aware supervision at approximately 2x ideal token compute. Across 1.3B- and 2.1B-parameter nanochat-style backbones and a wide range of training budgets, LRT improves both BPB and CORE under matched effective compute. Additionally, LRT outperforms two-forward PonderLM-2 and matches a three-loop Transformer in BPB, while retaining one-forward-per-token decoding with 9% latency overhead over the standard Transformer.

Zeyi Huang, Xuehai He, LiLiang Ren +8

Microsoft · University of Washington · University of Wisconsin-Madison

Language Model Pretraining Transformer Inference Autoregressive Language Modeling Recurrent Transformers Autoregressive Transformer

Ask PDF

May 9, 2026

Structured Recurrent Mixers for Massively Parallelized Sequence Generation

Over the last two decades, language modeling has experienced a shift from the use of predominantly recurrent architectures that process tokens sequentially during training and inference to non-recurrent models that process sequence elements in parallel during training, which results in greater training efficiency and stability at the expense of lower inference throughput. Here we introduce the Structured Recurrent Mixer, an architecture that allows for algebraic conversion between a sequence parallel representation at train time and a recurrent representation at inference, notably without the need for specialized kernels or device-specific memory management. We show experimentally that this dual representation allows for greater training efficiency, higher input information capacity, and larger inference throughput and concurrency when compared to other linear complexity models. We postulate that recurrent models are poorly suited to extended sequence length scaling for information-rich inputs typical of language, but are well suited to scaling in the sample (batch) dimension due to their constant memory per sample. We provide Mojo/MAX inference implementations of SRMs exhibiting 12x the throughput and 170x the concurrency of similarly powerful Transformers inferenced on vLLM, increases characteristic of Pytorch implementations resulting in a 30% increase in compute-constant GSM8k Pass@k. We conclude by demonstrating that SRMs are effective reinforcement learning training candidates.

Benjamin L. Badger

IBM

Recurrent Neural Networks Autoregressive Language Modeling Efficient Language Model Training LLM Inference Acceleration Recurrent Language Models

Ask PDF

Apr 8, 2026

Optimal Decay Spectra for Linear Recurrences

Linear recurrent models offer linear-time sequence processing but often suffer from suboptimal long-range memory. We trace this to the decay spectrum: for

N

channels, random initialization collapses the minimum spectral gap to

O(N^{-2})

, yielding sub-exponential error

\exp(-Ω(N/\log N))

; linear spacing avoids collapse but degrades to

\exp(-O(N/\sqrt{T}))

, practically algebraic over long contexts. We introduce Position-Adaptive Spectral Tapering (PoST), an architecture-agnostic framework combining two mechanisms: (1) Spectral Reparameterization, which structurally enforces geometrically spaced log-decay rates, proven minimax optimal at rate

O(\exp(-cN/\log T))

; and (2) Position-Adaptive Scaling, the provably unique mechanism that eliminates the scale mismatch of static spectra (where only

N\log t/\log T

N

channels are effective at position

t

) by stretching the spectrum to the actual dependency range, sharpening the rate to

O(\exp(-cN/\log t))

. This scaling natively induces fractional invariance: the impulse response becomes scale-free, with channels interpolating between relative and absolute temporal coordinates. PoST integrates into any diagonal linear recurrence without overhead. We instantiate it across Mamba-2, RWKV-7, Gated DeltaNet, Gated Linear Attention, and RetNet. Pre-training at 180M-440M scales shows consistent zero-shot language modeling improvements, significant long-context retrieval gains for Mamba-2 (MQAR and NIAH), and competitive or improved performance across other architectures. Code: https://github.com/SiLifen/PoST.

Yang Cao

Linear RNNs Long-Range Dependency Modeling Long-Context Language Modeling Autoregressive Language Modeling

Ask PDF

Abstract

Recurrent Language Models

Explore similar work

Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior

Structured Recurrent Mixers for Massively Parallelized Sequence Generation

Optimal Decay Spectra for Linear Recurrences