Shared Phase and Retention Control for Efficient Adaptive Spectral Recurrence

Wentao Wang; Hengyu Zhong; Yunhan Jiang; Jialiang An; Meng Lu

cs.LGSep 30, 2026

Shared Phase and Retention Control for Efficient Adaptive Spectral Recurrence

Authors: Wentao Wang, Hengyu Zhong, Yunhan Jiang, Jialiang An, Meng Lu

Organizations: Peking University · GigaAI

Abstract

As new evidence arrives, a sequence model must update what it remembers and how memory influences predictions. While Transformers incur computation and cache costs scaling with context length, fixed-state recurrent models offer constant-memory inference. However, linear and spectral recurrences traditionally rely on static transitions, failing to dynamically revise how stored representations decay or rotate. While recent selective architectures introduce input-dependent transitions, they assign independent controls to every memory mode, coupling control cost to state capacity. We show that high-dimensional spectral memory does not require high-dimensional control, and introduce Shared Phase and Retention Control for Efficient Adaptive Spectral Recurrence (SPARC). SPARC employs just two input-dependent scalar signals to coordinate memory retention and phase rotation across heterogeneous complex modes, while preserving mode-specific baseline timescales and frequencies. Its diagonal affine recurrence supports parallel associative scans for sequence-level BPTT as well as exact structured Real-Time Recurrent Learning (RTRL) for online credit assignment. Across partially observable continuous control, POPGym, and sequence classification, SPARC achieves a 9.09% relative return improvement on Walker-P and a 1.36% relative accuracy gain on FordA over second-best methods. On an NVIDIA Blackwell GPU, our implementation reduces recurrent-mixer training latency by 18.2%-34.2% in fixed-token workloads and accelerates scans by 3.1x-4.7x over an optimized RG-LRU baseline. These results show that two shared control signals can efficiently govern adaptive spectral memory across online and full-sequence settings. Code is available at https://github.com/Botwwt/sparc.

Figures & tables

Figure 1: Two complementary memory requirements. (a) Recall relevant events in order despite distractors. (b) Track event-driven changes in contraction and rotation. Task details are in Appendix C.3 .

Figure 2: Overview of SPARC. (a) The layer input supplies a shared control block and the recurrent block, which produces the updated state. (b) Two scalar control heads coordinate retention and phase across spectral modes. The Linear–SiLU pathway illustrates a possible controller architecture; Eq. ( 31 ) specifies the tanh-projection parameterization used in this paper. (c) Mode

j

multiplies its previous state by an input-dependent complex transition and adds the current write contribution.

Method	Ant-P	Walker-P	Hopper-P	Cheetah-P
RTU	3823.8	899.8	1266.7	2547.3
LRU	4476.5	655.5	1363.4	2688.9
RG-LRU	1953.7	865.5	1224.4	2539.5
GateLoop	4105.6	911.9	1325.1	2466.1
Mamba-3	1355.7	680.2	863.6	2019.9
GRU	1921.1	760.1	631.3	2174.1

Table 1: Partially observable continuous control.

Figure 3: Learning curves for Autoencode, CountRecall, RepeatFirst, RepeatPrevious, Noisy Pendulum, and HigherLower. Curves and shading use the same three-seed averaging and sample-standard-deviation convention as Figure 5 .

Method	FordA	SLC	UWave	pMNIST	CIFAR-10
RTU	95.38	99.67	93.33	96.75	59.32
LRU	95.38	99.67	92.22	96.82	60.11
RG-LRU	92.89	99.67	98.52	85.48	51.01
GateLoop	91.97	99.00	91.85	62.31	52.46
Mamba-3	90.58	99.33	98.15	88.51	48.65
GRU	93.54	99.00	94.81	90.92	62.69

Table 2: Full-sequence classification.

Figure 4: GPU efficiency on a single NVIDIA RTX PRO 6000 Blackwell Server Edition GPU. (a) Scan-only timing after coefficient generation. (b,c) Recurrent-mixer forward-plus-backward timing at fixed token count, for widths

2048

and

2560

. (d) Runtime and peak allocated-memory ratios, SPARC divided by RG-LRU, as batch size varies. Ratios below one favor SPARC.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Variant	Intervention and scope
Fixed shared controls	Set $r_{t}=p_{t}=0$ in every pathway; retain the learned modal spectrum and the selected content activation.
No adaptive retention	Set $e_{j,t}=\nu_{j}$ only in the transition, including its phase clock; retain the original adaptive write normalization and gate.
No adaptive phase	Set $\lambda_{j,t}=\rho_{j,t}e^{i\theta_{j}}$ ; retain $p_{t}$ in the write gate.
No phase clock	Replace $(1-\rho_{j,t})\kappa_{p}p_{t}$ by $\kappa_{p}p_{t}$ ; keep retention and all write terms.
Static phase clock	Use $(1-e^{-\nu_{j}})\kappa_{p}p_{t}$ ; retain adaptive retention and writing.
Static transition	Set $\lambda_{j}=e^{-\nu_{j}+i\theta_{j}}$ ; retain the original adaptive write pathway.

Appendix

Table 3: Component-removal definitions.

Setting	Continuous control	POPGym
Environment interaction steps	$4{,}999{,}168$	$3{,}999{,}744$
Physical rollout length	$2048$	$2048$
PPO epochs per rollout	$4$	$10$
Minibatches per epoch	$32$	$32$
Real-valued recurrent state coordinates	$384$	$128$
SPARC complex modes	$192$	$64$

Appendix

Table 4: Shared reinforcement-learning hyperparameters.

Dataset	Length	Classes	Fit	Validation
FordA	$500$	$2$	$3240$	$361$
StarLightCurves	$1024$	$3$	$900$	$100$
UWaveGestureLibraryAll	$945$	$8$	$806$	$90$
Permuted MNIST	$784$	$10$	$54{,}001$	$5999$
Seq. CIFAR-10 Gray	$1024$	$10$	$45{,}000$	$5000$

Appendix

Table 5: Classification data configurations.

Method	Autoenc.	CountRec.	Repeat1st	RepeatPrev.	NoisyPend.	HigherLow.
RTU	-0.427	-0.565	0.448	0.891	0.244	0.501
LRU	-0.431	-0.597	0.242	0.908	0.246	0.498
RG-LRU	-0.476	-0.644	0.550	-0.004	0.246	0.503
GateLoop	-0.438	-0.647	0.727	0.268	0.245	0.498
Mamba-3	-0.499	-0.811	0.471	-0.428	0.269	0.487
GRU	-0.469	-0.835	0.811	0.744	0.311	0.495

Appendix

Table 6: Partially observable memory tasks in POPGym.

Figure 5: Learning curves on Ant-P, Walker-P, Hopper-P, and Cheetah-P. Each curve averages three seed trajectories after applying a $100$ -rollout moving average within each seed. Shading denotes one sample standard deviation across the smoothed trajectories.

Variant	RepeatFirst	Walker-P	FordA	CIFAR-10
SPARC	$\mathbf{0.829}$	$\mathbf{994.79}$	$\mathbf{96.68}$	$\mathbf{59.92}$
Neutral write gate	$0.663$	$909.56$	$\mathbf{96.68}$	$59.35$
Linear content	$0.123$	$921.90$	$96.12$	$59.46$
Fixed write normalization	$0.628$	$\underline{943.45}$	$96.40$	$58.73$
No phase clock	$0.288$	$928.64$	$\underline{96.49}$	$49.03$
Retention-only transition	$\underline{0.756}$	$837.70$	$96.03$	$\underline{59.79}$

Appendix

Table 7: Selected component ablations.

Task	SPARC	Variant	$\Delta$ (variant $-$ SPARC)
Autoencode	$-0.431$ , $\pm,0.003$	$\mathbf{-0.424}$ , $\pm,0.023$	$0.007$ , $\pm,0.021$
CountRecall	$-0.515$ , $\pm,0.098$	$\mathbf{-0.493}$ , $\pm,0.148$	$0.022$ , $\pm,0.051$
RepeatFirst	$\mathbf{0.829}$ , $\pm,0.008$	$0.756$ , $\pm,0.051$	$-0.073$ , $\pm,0.058$
RepeatPrevious	$0.855$ , $\pm,0.033$	$\mathbf{0.938}$ , $\pm,0.051$	$0.083$ , $\pm,0.019$
Noisy Pendulum	$0.461$ , $\pm,0.039$	$\mathbf{0.504}$ , $\pm,0.053$	$0.043$ , $\pm,0.020$
CartPole	$\mathbf{0.959}$ , $\pm,0.026$	$0.921$ , $\pm,0.016$	$-0.038$ , $\pm,0.010$

Appendix

Table 8: Retention-only transition.

Task	SPARC	Variant	$\Delta$ (variant $-$ SPARC)
Autoencode	$-0.431$ , $\pm,0.003$	$\mathbf{-0.424}$ , $\pm,0.023$	$0.007$ , $\pm,0.021$
CountRecall	$\mathbf{-0.515}$ , $\pm,0.098$	$-0.568$ , $\pm,0.229$	$-0.053$ , $\pm,0.131$
RepeatFirst	$\mathbf{0.829}$ , $\pm,0.008$	$0.326$ , $\pm,0.515$	$-0.503$ , $\pm,0.522$
RepeatPrevious	$0.855$ , $\pm,0.033$	$\mathbf{0.902}$ , $\pm,0.024$	$0.047$ , $\pm,0.056$
Noisy Pendulum	$\mathbf{0.461}$ , $\pm,0.039$	$0.457$ , $\pm,0.097$	$-0.004$ , $\pm,0.102$
CartPole	$\mathbf{0.959}$ , $\pm,0.026$	$0.947$ , $\pm,0.005$	$-0.012$ , $\pm,0.032$

Appendix

Table 9: Phase-only transition.

Task	SPARC	Variant	$\Delta$ (variant $-$ SPARC)
Autoencode	$\mathbf{-0.431}$ , $\pm,0.003$	$-0.441$ , $\pm,0.019$	$-0.009$ , $\pm,0.016$
CountRecall	$-0.515$ , $\pm,0.098$	$\mathbf{-0.397}$ , $\pm,0.161$	$0.119$ , $\pm,0.064$
RepeatFirst	$\mathbf{0.829}$ , $\pm,0.008$	$0.534$ , $\pm,0.196$	$-0.294$ , $\pm,0.189$
RepeatPrevious	$0.855$ , $\pm,0.033$	$\mathbf{0.900}$ , $\pm,0.077$	$0.045$ , $\pm,0.061$
Noisy Pendulum	$0.461$ , $\pm,0.039$	$\mathbf{0.532}$ , $\pm,0.031$	$0.070$ , $\pm,0.050$
CartPole	$\mathbf{0.959}$ , $\pm,0.026$	$0.949$ , $\pm,0.012$	$-0.010$ , $\pm,0.038$

Appendix

Table 10: Static transition.

Task	SPARC	Variant	$\Delta$ (variant $-$ SPARC)
Autoencode	$\mathbf{-0.431}$ , $\pm,0.003$	$-0.454$ , $\pm,0.029$	$-0.022$ , $\pm,0.027$
CountRecall	$\mathbf{-0.515}$ , $\pm,0.098$	$-0.627$ , $\pm,0.151$	$-0.112$ , $\pm,0.145$
RepeatFirst	$\mathbf{0.829}$ , $\pm,0.008$	$0.288$ , $\pm,0.242$	$-0.541$ , $\pm,0.234$
RepeatPrevious	$\mathbf{0.855}$ , $\pm,0.033$	$0.661$ , $\pm,0.120$	$-0.194$ , $\pm,0.136$
Noisy Pendulum	$\mathbf{0.461}$ , $\pm,0.039$	$0.384$ , $\pm,0.017$	$-0.077$ , $\pm,0.056$
CartPole	$\mathbf{0.959}$ , $\pm,0.026$	$0.945$ , $\pm,0.008$	$-0.013$ , $\pm,0.018$

Appendix

Table 11: No phase clock.

Task	SPARC	Variant	$\Delta$ (variant $-$ SPARC)
Autoencode	$\mathbf{-0.431}$ , $\pm,0.003$	$-0.452$ , $\pm,0.015$	$-0.021$ , $\pm,0.017$
CountRecall	$-0.515$ , $\pm,0.098$	$\mathbf{-0.389}$ , $\pm,0.068$	$0.126$ , $\pm,0.081$
RepeatFirst	$\mathbf{0.829}$ , $\pm,0.008$	$0.383$ , $\pm,0.547$	$-0.446$ , $\pm,0.555$
RepeatPrevious	$\mathbf{0.855}$ , $\pm,0.033$	$0.845$ , $\pm,0.053$	$-0.010$ , $\pm,0.071$
Noisy Pendulum	$\mathbf{0.461}$ , $\pm,0.039$	$0.413$ , $\pm,0.087$	$-0.048$ , $\pm,0.069$
CartPole	$\mathbf{0.959}$ , $\pm,0.026$	$0.811$ , $\pm,0.016$	$-0.148$ , $\pm,0.042$

Appendix

Table 12: Static phase clock.

Task	SPARC	Variant	$\Delta$ (variant $-$ SPARC)
Autoencode	$-0.431$ , $\pm,0.003$	$\mathbf{-0.430}$ , $\pm,0.026$	$0.001$ , $\pm,0.025$
CountRecall	$-0.515$ , $\pm,0.098$	$\mathbf{-0.511}$ , $\pm,0.059$	$0.004$ , $\pm,0.073$
RepeatFirst	$\mathbf{0.829}$ , $\pm,0.008$	$0.663$ , $\pm,0.067$	$-0.166$ , $\pm,0.074$
RepeatPrevious	$0.855$ , $\pm,0.033$	$\mathbf{0.867}$ , $\pm,0.033$	$0.012$ , $\pm,0.001$
Noisy Pendulum	$\mathbf{0.461}$ , $\pm,0.039$	$0.425$ , $\pm,0.030$	$-0.036$ , $\pm,0.022$
CartPole	$\mathbf{0.959}$ , $\pm,0.026$	$0.915$ , $\pm,0.078$	$-0.044$ , $\pm,0.104$

Appendix

Table 13: Neutral write gate.

Task	SPARC	Variant	$\Delta$ (variant $-$ SPARC)
Autoencode	$\mathbf{-0.431}$ , $\pm,0.003$	$-0.447$ , $\pm,0.024$	$-0.016$ , $\pm,0.021$
CountRecall	$\mathbf{-0.515}$ , $\pm,0.098$	$-0.613$ , $\pm,0.114$	$-0.098$ , $\pm,0.051$
RepeatFirst	$\mathbf{0.829}$ , $\pm,0.008$	$0.721$ , $\pm,0.016$	$-0.107$ , $\pm,0.024$
RepeatPrevious	$0.855$ , $\pm,0.033$	$\mathbf{0.907}$ , $\pm,0.031$	$0.052$ , $\pm,0.052$
Noisy Pendulum	$0.461$ , $\pm,0.039$	$\mathbf{0.464}$ , $\pm,0.031$	$0.003$ , $\pm,0.041$
CartPole	$\mathbf{0.959}$ , $\pm,0.026$	$0.950$ , $\pm,0.027$	$-0.009$ , $\pm,0.000$

Appendix

Table 14: Frozen reference gain.

Task	SPARC	Variant	$\Delta$ (variant $-$ SPARC)
Autoencode	$\mathbf{-0.431}$ , $\pm,0.003$	$-0.477$ , $\pm,0.021$	$-0.045$ , $\pm,0.019$
CountRecall	$-0.515$ , $\pm,0.098$	$\mathbf{-0.452}$ , $\pm,0.036$	$0.063$ , $\pm,0.077$
RepeatFirst	$\mathbf{0.829}$ , $\pm,0.008$	$0.628$ , $\pm,0.049$	$-0.200$ , $\pm,0.057$
RepeatPrevious	$0.855$ , $\pm,0.033$	$\mathbf{0.878}$ , $\pm,0.065$	$0.023$ , $\pm,0.037$
Noisy Pendulum	$\mathbf{0.461}$ , $\pm,0.039$	$0.366$ , $\pm,0.145$	$-0.095$ , $\pm,0.133$
CartPole	$\mathbf{0.959}$ , $\pm,0.026$	$0.891$ , $\pm,0.065$	$-0.067$ , $\pm,0.039$

Appendix

Table 15: Fixed write normalization.

Task	SPARC	Variant	$\Delta$ (variant $-$ SPARC)
Autoencode	$\mathbf{-0.431}$ , $\pm,0.003$	$-0.442$ , $\pm,0.012$	$-0.011$ , $\pm,0.009$
CountRecall	$-0.515$ , $\pm,0.098$	$\mathbf{-0.465}$ , $\pm,0.060$	$0.050$ , $\pm,0.081$
RepeatFirst	$\mathbf{0.829}$ , $\pm,0.008$	$0.123$ , $\pm,0.857$	$-0.705$ , $\pm,0.865$
RepeatPrevious	$0.855$ , $\pm,0.033$	$\mathbf{0.867}$ , $\pm,0.070$	$0.012$ , $\pm,0.041$
Noisy Pendulum	$\mathbf{0.461}$ , $\pm,0.039$	$0.425$ , $\pm,0.060$	$-0.037$ , $\pm,0.079$
CartPole	$\mathbf{0.959}$ , $\pm,0.026$	$0.944$ , $\pm,0.001$	$-0.015$ , $\pm,0.027$

Appendix

Table 16: Linear content.

Task	Gate SD	Saturation (%)	Gate ratio	Content (%)
Autoencode	$0.0913,\text{$ \pm,0.0299 $}$	$32.67,\text{$ \pm,3.70 $}$	1.171	15.49
CountRecall	$0.0426,\text{$ \pm,0.0274 $}$	$23.50,\text{$ \pm,5.50 $}$	1.252	25.38
RepeatFirst	$0.4508,\text{$ \pm,0.0382 $}$	$95.73,\text{$ \pm,1.79 $}$	1.816	0.76
RepeatPrevious	$0.0081,\text{$ \pm,0.0013 $}$	$31.52,\text{$ \pm,4.34 $}$	1.244	16.43
Noisy Pendulum	$0.0890,\text{$ \pm,0.0203 $}$	$75.65,\text{$ \pm,1.48 $}$	1.079	4.23
CartPole	$0.0072,\text{$ \pm,0.0015 $}$	$28.33,\text{$ \pm,5.32 $}$	1.149	26.49

Appendix

Table 17: Write-gate activity and content saturation.

Task	Decay scale	Surv. (%)	Rel. surv.	Phase (deg)	Clock (%)
Autoencode	2.769	63.99	1.039	30.553	38.05
CountRecall	0.708	80.46	1.114	17.901	25.30
RepeatFirst	0.255	99.12	1.091	2.963	3.29
RepeatPrevious	0.223	84.66	1.584	6.026	9.74
Noisy Pendulum	4.416	54.92	0.967	30.287	41.12
CartPole	0.623	67.66	1.210	17.442	22.16

Appendix

Table 18: Retention and phase activity.

$B$	$T$	$d_{\mathrm{mix}}$	Naive	Accel.	Speedup	Naive	Accel.	Speedup
			Backward			Forward + backward
4	2048	2048	238.730	1.225	$194.81\times$	358.822	2.617	$137.13\times$
2	4096	2048	479.999	1.289	$372.50\times$	721.780	2.510	$287.54\times$
1	8192	2048	967.711	0.911	$1061.91\times$	1469.467	1.690	$869.62\times$
4	2048	2560	236.191	1.081	$218.45\times$	358.953	1.824	$196.78\times$
2	4096	2560	489.609	1.105	$442.89\times$	740.985	1.846	$401.39\times$

Appendix

Table 19: SPARC versus its naive implementation at a fixed budget of

8192

tokens.

$B$	Naive	Accel.	Speedup	Naive	Accel.	Speedup
	Backward			Forward + backward
1	228.853	0.853	$268.35\times$	344.970	1.900	$181.56\times$
2	240.696	0.942	$255.42\times$	361.919	2.026	$178.67\times$
4	236.798	0.890	$265.99\times$	355.549	1.541	$230.69\times$
8	239.711	1.710	$140.21\times$	365.954	2.548	$143.62\times$
16	244.437	3.044	$80.31\times$	366.240	4.570	$80.15\times$

Appendix

Table 20: SPARC versus its naive implementation across batch sizes, with

T=2048

and

d_{\mathrm{mix}}=2048

Quantity	Maximum absolute error	Normalized error
Forward output	$5.66\times 10^{-7}$	$5.66\times 10^{-7}$
Input gradient	$2.03\times 10^{-6}$	$5.28\times 10^{-7}$
Log-decay parameter gradient	$4.77\times 10^{-6}$	$1.17\times 10^{-6}$
Log-phase parameter gradient	$7.63\times 10^{-5}$	$6.58\times 10^{-7}$
Phase-controller gradient	$6.68\times 10^{-6}$	$7.09\times 10^{-7}$
Retention-controller gradient	$3.05\times 10^{-5}$	$1.43\times 10^{-6}$

Appendix

Table 21: Numerical agreement between the accelerated SPARC mixer and the naive reference.

Explore similar work

Sep 27, 2026cs.LG

SchemaMem: Schema-Indexed Recurrent Memory for Delayed State Retrieval

Attention provides direct access to past representations, but retaining an ever-growing history is costly. Recurrent models bound persistent state, yet must preserve selected information while processing subsequent inputs. We introduce SchemaMem, an attention-based recurrent memory architecture combining chunk-local attention with a persistent, schema-indexed phase state. Learned schema embeddings provide a shared representational reference for reading and writing. Reads use the current state, whereas writes use the layer input and static schema embeddings, excluding direct feedback from that layer's own state. Chunk-boundary commits aggregate bounded phase increments through forward computation. The same parameters also support full-history attention training before and during recurrent training. We studied selective updates, preservation, and delayed retrieval in a controlled address--value task, comparing three-layer models with approximately matched parameter counts and persistent-state dimensions. Across nine address/value settings and three training seeds, SchemaMem has higher mean written-value retention at four times the maximum training delay than both baselines, which are trained toward a higher in-range accuracy target. Updated-value recovery favors SchemaMem in all nine settings against Mamba-3 and seven against Gated DeltaNet. Defaults consistently favor Gated DeltaNet over SchemaMem at that delay, and SchemaMem requires substantially more optimization steps. These results identify a promising retention--optimization trade-off in schema-indexed recurrence.

Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung

College of Pharmacy, Chungnam National University Daejeon, Republic of Korea · Department of Computer Science & Engineering, Chungnam National University Daejeon, Republic of Korea

Recurrent Model Persistent State Retention Linear Attention Bidirectional Long Short-Term Memory

Ask PDF

May 9, 2026cs.CL

Structured Recurrent Mixers for Massively Parallelized Sequence Generation

Over the last two decades, language modeling has experienced a shift from the use of predominantly recurrent architectures that process tokens sequentially during training and inference to non-recurrent models that process sequence elements in parallel during training, which results in greater training efficiency and stability at the expense of lower inference throughput. Here we introduce the Structured Recurrent Mixer, an architecture that allows for algebraic conversion between a sequence parallel representation at train time and a recurrent representation at inference, notably without the need for specialized kernels or device-specific memory management. We show experimentally that this dual representation allows for greater training efficiency, higher input information capacity, and larger inference throughput and concurrency when compared to other linear complexity models. We postulate that recurrent models are poorly suited to extended sequence length scaling for information-rich inputs typical of language, but are well suited to scaling in the sample (batch) dimension due to their constant memory per sample. We provide Mojo/MAX inference implementations of SRMs exhibiting 12x the throughput and 170x the concurrency of similarly powerful Transformers inferenced on vLLM, increases characteristic of Pytorch implementations resulting in a 30% increase in compute-constant GSM8k Pass@k. We conclude by demonstrating that SRMs are effective reinforcement learning training candidates.

Benjamin L. Badger

IBM

Asymmetric Grouped-Attention Variate Mixer Recurrent Model Language Modeling Pytorch

Ask PDF

Apr 23, 2026cs.LG

The Recurrent Transformer: Greater Effective Depth and Efficient Decoding

Transformers process tokens in parallel but are temporally shallow: at position

t

, each layer attends to key-value pairs computed based on the previous layer, yielding a depth capped by the number of layers. Recurrent models offer unbounded temporal depth but suffer from optimization instability and historically underutilize modern accelerators. We introduce the Recurrent Transformer, a simple architectural change where each layer attends to key-value pairs computed off its own activations, yielding layerwise recurrent memory while preserving standard autoregressive decoding cost. We show that the architecture can emulate both (i) a conventional Transformer and (ii) token-to-token recurrent updates under mild assumptions, while avoiding optimization instability. Naively, prefill/training appears bandwidth-bound with effective arithmetic intensity near

1

because keys and values are revealed sequentially; we give an exact tiling-based algorithm that preserves the mathematical computation while reducing HBM traffic from

Θ(N^2)

Θ(N\log N)

, increasing effective arithmetic intensity to

Θ(N/\log N)

for sequence length

N

. On 150M and 300M parameter C4 pretraining, Recurrent Transformers improve cross-entropy over a parameter-matched Transformer baseline and achieve the improvement with fewer layers (fixed parameters), suggesting that recurrence can trade depth for width, thus reducing KV cache memory footprint and inference latency.

Costin-Andrei Oncescu, Depen Morwani, Samy Jelassi +3

Harvard University

Transformer Architectures Kv-Cache Management

Ask PDF