Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout

Tushaar Gangavarapu; Gil Tabak; Clemens JS Schaefer

cs.LGNov 5, 2026

Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout

Authors: Tushaar Gangavarapu, Gil Tabak, Clemens JS Schaefer

Organizations: University of Texas at Austin · Google

Abstract

Linear attention compresses context into a fixed-size key-value state for linear-time training and constant-memory inference. Gated delta-rule models decay this state and erase its prediction for an incoming key before writing a new value; Gated DeltaNet-2 (GDN-2) generalizes this with channel-wise decay and decoupled, full-rank erase and write gates. However, we show that state updates are far less sensitive to erasure than to writes: key activation and unit normalization concentrate key energy onto a sparse subset of channels, whereas value writes scale all channels, so the erase gate can be parameterized at lower rank. Furthermore, slow-decaying state rows accumulate much larger norms than fast rows, dominating unweighted readout and forcing stronger write-time erasure to suppress interference. Guided by this analysis, we introduce Muir, which fuses all gates into a sensitivity-allocated low-rank projection, re-weights state rows with a matched-filter readout gate, and reinvests the saved parameters in a wider feed-forward network. Because standard chunkwise algorithms become numerically unstable at the large chunk sizes needed to saturate wide matrix units, we design a parallel update for stable, efficient training. Across 166M-1.34B parameters, Muir lowers perplexity by 1.8-2.0% over GDN-2 and 2.5-3.6% over Transformer, raises commonsense accuracy by up to 1.9 points, and improves extractive recall over GDN-2. At 1.34B, Muir outperforms both baselines on multi-hop and multi-needle recall, extrapolates 2,560x from its 4K training length to 10M-token streams without perplexity drift, and maintains near-100% passkey retrieval to 1M tokens with static readout. Muir achieves these gains while matching GDN-2 throughput, reaching its 100B-token validation loss 1.2x faster, and training 2.7x faster than Transformer at 32K.

Figures & tables

Figure 1: Muir outperforms GDN-2 and Transformer on language modeling and long-context throughput while improving recall over GDN-2. (a) Sensitivity-allocated low-rank gates with FFN expansion drive the bulk of the validation NLL gain over GDN-2, while dynamic matched-filter readout drives the gain on extractive recall. (b) Across $166\text{M}\text{--}1.34\text{B}$ parameters, Muir lowers validation NLL by $18.0\text{--}20.4\text{ millinats}$ over GDN-2 and $25.0\text{--}36.2\text{ millinats}$ over Transformer. (c) On $8$ TPU7x chips ( $0.5\text{M}$ tokens/step) at $1.34\text{B}$ with our chunkwise kernel, Muir matches GDN-2 throughput despite an additional readout gate and a $30.4\%$ wider FFN, and trains $1.64\times$ and $2.74\times$ faster than Transformer at $16\text{K}$ and $32\text{K},$ respectively.

Figure 2: Matched-filter readout and low-rank erase gates improve synthetic associative retrieval and lower optimal erase strength. (a) Full whitening worsens retrieval error relative to unit gain (GDN-2), whereas half-power scaling and the Corollary 3.8 filter reduce error and shift ${\mu}^{\ast}$ from $0.60$ to $0.50,$ with the Corollary 3.8 filter matching the static oracle. (b) On a $50/50$ mixture of lags $\ell\in\{2,80\}$ at ${\mu}=0.60,$ the lag-matched oracle outperforms the static oracle across all horizons, while the Proposition 3.6 filter matches the lag-matched oracle at $\ell=80$ and beats the static oracle on long and mixed horizons. (c) Low-rank erase gates $r_{b}$ outperform dense gates at every bottleneck rank on long-lag recall, whereas low-rank write gates $r_{w}$ degrade below dense at $r_{w}\leq 8$ and recover once $r_{w}\geq 16.$

Figure 3: Muir co-designs asymmetric low-rank gate parameterization and matched-filter query readout with MXU tile alignment and iso-parameter FFN reallocation. (a) Standard residual block with a Muir token mixer. (b) Within the Muir block, ${\bm{x}}_{t}$ feeds linear- $\mathrm{Conv}$ - $\mathrm{SiLU}$ pathways for $({\bm{q}}_{t},{\bm{k}}_{t},{\bm{v}}_{t})$ alongside a single fused low-rank gate projection that produces all five gates $(\log{\bm{\alpha}}_{t},{\bm{b}}_{t},{\bm{g}}_{t},{\bm{w}}_{t},{\bm{\gamma}}_{t}).$ (c) Asymmetric gate ranks $(r_{f},r_{b},r_{g},r_{w},r_{q})=(64,64,128,128,128)$ sum to $R_{\mathrm{tot}}=512=2\times 256,$ which fills two $256$ -column MXU tiles in the fused down-projection and aligns $(f,b)$ and $(g,w,q)$ on $128$ -lane VMEM tile boundaries for batched up-projections. (d) At $d=2{,}048,$ replacing the dense $(b,w)$ gate matrices in GDN-2 cuts gate-projection parameters by $4.5\times$ ; this either reduces per-layer parameters and FLOPs by $14.6\%$ at fixed FFN width, or frees $+7.3\text{M}$ parameters per layer to widen the FFN under an iso-parameter budget.

Figure 4: Boundary-pivoted chunkwise kernel eliminates 3D decay tensors, division overflow, and VPU bottlenecks. (a) Memory hierarchy of a TPU7x chip. (b) On the $256\times 256$ MXU, an unfused $256\times 128$ right-hand side $\hat{\bm{V}}$ achieves only $50\%$ utilization due to zero-padded lanes, whereas the fused solve $\begin{bmatrix}{\bm{U}}&{\bm{Y}}\end{bmatrix}={\bm{T}}^{-1}\begin{bmatrix}\hat{\bm{V}}&\hat{\bm{K}}_{\odot\pi}\end{bmatrix}$ achieves $100\%$ utilization. (c) At chunk size $C=256,$ leaf size $8,$ and $d_{k}=d_{v}=128,$ replacing naive $C\times C\times d_{k}$ 3D VPU decay and sequential VPU triangular solves with boundary-pivoted MXU matrix multiplications, recursive block inversion, and a custom backward pass cuts forward-and-backward VPU compute by $24\times$ and total kernel FLOPs by $12\%,$ routing $98\%$ of chunkwise arithmetic onto the MXU. (d) Boundary-pivoted chunkwise execution combines pre-permutation VREG fusion for $\hat{\bm{q}}_{t}$ and $\hat{\bm{v}}_{t},$ MXU cumulative log-decay, numerically stable VPU decay splitting, recursive block inversion of ${\bm{T}}$ on the MXU, and a packed MXU scan carrying ${\bm{S}}_{0}\to{\bm{S}}_{C}$ in VMEM (saving ${\bm{S}}_{0}$ only for backward).

Figure 5: Muir consistently lowers validation NLL relative to GDN-2 and Transformer throughout $\bm{100\text{B}}$ -token pretraining. Held-out FineWeb-Edu validation token NLL difference relative to GDN-2 across $166\text{M}\text{--}1.34\text{B}$ parameters as a function of training tokens. All three Muir variants improve over GDN-2 within the first $10\text{B}$ tokens and reach a $12\text{--}20\text{ mnat}$ reduction by $100\text{B}$ tokens, while static and dynamic matched-filter readout provide additional reductions over Muir (no ${\bm{\gamma}}$ ).

		Language Modeling			Zero-Shot Commonsense Reasoning									5-Shot
	Model	FW-Edu	Wiki.	LMB.	LMB.	PIQA	Hella.	Wino.	ARC-e	ARC-c	OBQA	SIQA	Avg.	Wino.	TQA
		NLL $\downarrow$	ppl $\downarrow$	ppl $\downarrow$	acc $\uparrow$	acc $\uparrow$	acc n $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc n $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$
166M	Transformer	2.4569	31.64	52.69	28.47	65.67	35.95	50.99	55.35	26.28	20.20	36.39	39.91	49.96	4.74
	GDN-2	2.4466	30.84	39.89	30.60	64.74	37.02	50.28	56.61	26.45	20.40	37.92	40.50	50.12	5.80
	Muir (no ${\bm{\gamma}}$ )	2.4322	30.31	34.87	32.68	65.51	38.19	52.01	56.48	27.82	20.80	38.95	41.55	52.49	3.77
	Muir (static ${\bm{\gamma}}$ )	2.4273	29.83	40.98	30.58	66.43	37.92	51.78	55.30	26.54	20.80	38.18	40.94	51.54	4.32

Table 1: Muir improves language modeling and downstream reasoning over Transformer and GDN-2 across model scale. We report FineWeb-Edu validation token NLL (FW-Edu), WikiText (Wiki.) and LAMBADA (LMB.) word-level perplexity, zero-shot accuracy across eight commonsense reasoning benchmarks and their mean (Avg.), and 5-shot WinoGrande (Wino.) and TriviaQA (TQA) accuracy after

100\text{B}

training tokens. Best results per scale are in bold and second-best are underlined .

Figure 6: Muir improves synthetic single-needle, multi-hop, multi-needle, and real-world retrieval over GDN-2. (a) Averaged across the three single-needle tasks, all Muir variants mitigate the $4\text{K}$ degradation of GDN-2 and extrapolate to $8\text{K},$ where unextended RoPE Transformer degrades beyond its $4\text{K}$ training window. (b) On $5$ -hop variable-binding chains, static ${\bm{\gamma}}$ fails to match the varying hop lags, whereas dynamic ${\bm{\gamma}}_{t}$ leads all models at every context length. (c, d) When retrieving four values bound to a single key or four distinct keys queried jointly, dynamic ${\bm{\gamma}}_{t}$ matches or outperforms both GDN-2 and Transformer across $1\text{K}\text{--}4\text{K}.$ (e) Averaged across the RULER tasks (a–d) by needle depth and context length, Muir achieves the largest gains over GDN-2 on early needles ( $0\text{--}40\%$ depth) at $4\text{K}\text{--}8\text{K}.$ (f) On real-world document extraction at $2\text{K},$ Muir improves over GDN-2 on all six tasks, with dynamic ${\bm{\gamma}}_{t}$ driving $69\%$ of the gain.

Figure 7: Muir extrapolates to $\bm{128\text{K}}$ -token documents and $\bm{10\text{M}}$ -token streams with the lowest perplexity at a constant $\bm{25\text{ MB}}$ state. (a, c) Zero-shot from $4\text{K}$ pretraining, Dynamic YaRN Transformer diverges past $8\text{K}$ and unchunked evaluation exceeds HBM at $\geq\!128\text{K},$ while Muir (static ${\bm{\gamma}}$ ) drifts upward past $32\text{K}$ ; dynamic ${\bm{\gamma}}_{t}$ avoids this static-gain drift and outperforms GDN-2 across $128\text{K}$ documents and $10\text{M}$ -token streams ( $2{,}560\times$ training length). (b, d) After a $64\text{K}$ cooldown ( $0.3\%$ of pretraining), Transformer improves through $64\text{K}$ before unchunked evaluation exceeds HBM at $\geq\!128\text{K},$ static- ${\bm{\gamma}}$ drift vanishes, and all Muir variants improve monotonically below GDN-2 through $10\text{M}$ tokens ( $160\times$ beyond cooldown).

Figure 8: Static matched-filter readout extends passkey retrieval to $\bm{1\text{M}}$ tokens both zero-shot and after a $\bm{64\text{K}}$ cooldown. S-NIAH-1 accuracy across $10$ needle-depth bins and context lengths $1\text{K}\text{--}1\text{M}$ ( $256\times$ pretraining length) at $1.34\text{B}$ scale under (top) zero-shot evaluation from $4\text{K}$ pretraining and (bottom) pure-text continual cooldown at $64\text{K}.$ Cooldown extends Dynamic YaRN Transformer from $8\text{K}$ to $32\text{K}$ before early-depth accuracy drops at $64\text{K}$ and unchunked prefill exceeds HBM at $\geq\!128\text{K},$ whereas recurrent horizons change little under pure-text cooldown: GDN-2 collapses past $4\text{K}\text{--}8\text{K},$ Muir (no ${\bm{\gamma}}$ ) fades past $32\text{K},$ Muir (dynamic ${\bm{\gamma}}_{t},$ splitting $(r_{w},r_{q})=(128,128)$ for multi-lag recall) holds through $32\text{K}$ before tapering at $\geq\!64\text{K},$ and Muir (static ${\bm{\gamma}},$ keeping $r_{w}=256$ for single-lag lookup) maintains near-perfect recall to $1\text{M}$ tokens in both regimes.

Figure 9: Boundary-pivoted chunkwise execution eliminates decay overflow and maintains well-conditioned forward and backward passes. (a) Against an exact float64 sequential reference, naive unpivoted float32 decay factorization overflows and produces NaN on $25\%$ of heads, whereas the boundary-pivoted bfloat16 and float32 kernels match the reference to machine precision on both chunk outputs and final state. (b) Across synthetic stress regimes spanning $\kappa_{2}({\bm{T}})\in[1,10^{4}],$ the bfloat16 block-Neumann inverse stays near its precision floor throughout the well-conditioned regime ( $\kappa_{2}({\bm{T}})<40,$ shaded), which contains the sampled $100\text{B}$ -token checkpoints of GDN-2 and Muir (markers show medians). (c) Single-layer backward gradients ( $T\!=\!128,C\!=\!64,d_{k}\!=\!16$ ) match the float64 reference across all six inputs without log-decay drift.

		Configuration						FineWeb-Edu Validation NLL $\downarrow$				1.34B PPL $\downarrow$		1.34B Accuracy $\uparrow$
	Model	$r_{f}$	$r_{b}$	$r_{g}$	$r_{w}$	$r_{q}$	$d_{\mathrm{ffn}}$	166M	406M	834M	1.34B	Wiki.	LMB.	LM-Eval	Based	TQA
Base	Transformer	–	–	–	–	–	8,192	2.4569	2.2424	2.1186	2.0335	17.17	10.30	52.05	45.92	14.56
Base	GDN-2	128	$d$	128	$d$	–	5,888	2.4466	2.2354	2.1060	2.0177	16.80	10.38	52.46	32.08	16.48
Ours	Muir (no ${\bm{\gamma}}$ )	64	64	128	256	0	7,680	2.4322	2.2230	2.0876	2.0006	16.46	10.26	53.79	32.86	16.52
	Muir (static ${\bm{\gamma}}$ )	64	64	128	256	0	7,680	2.4273	2.2191	2.0869	1.9994	16.35	10.15	53.40	33.02	17.71
	Muir (ours)	64	64	128	128	128	7,680	2.4276	2.2174	2.0862	1.9973	16.41	10.00	53.84	34.60	19.32

Table 2: Sensitivity-allocated gate ranks and dynamic readout match or outperform wider output-gate and uniform-rank allocations while preserving MXU alignment and FFN width. For each configuration of gate bottleneck ranks and FFN width (shown at

d=2{,}048

), we report FineWeb-Edu validation token NLL across all four model scales, alongside

1.34\text{B}

WikiText (Wiki.) and LAMBADA (LMB.) word-level perplexity, 8-task zero-shot commonsense accuracy (LM-Eval), 6-task Based extractive recall accuracy (Based), and 5-shot TriviaQA accuracy (TQA) after

100\text{B}

training tokens.

Figure 10: Trained models exhibit the predicted write-erase sensitivity asymmetry under both SVD rank truncation and causal gate freezing. (a) Truncating each rank- $128$ gate projection of the uniform-rank Muir checkpoint via SVD (one gate at a time across all layers) shows that the decay ( $f$ ) and erase ( $b$ ) gates saturate by rank $64,$ whereas the output ( $g$ ), write ( $w$ ), and readout gain ( $q$ ) gates require rank $128.$ (b) Replacing the dynamic write gate with its token-averaged channel mean ( ${\bm{w}}_{t}\leftarrow\bar{\bm{w}}$ ) across all layers degrades validation NLL $14.5\times$ more than freezing the erase gate ( ${\bm{b}}_{t}\leftarrow\bar{\bm{b}}$ ) in both GDN-2 and Muir.

Figure 11: Stationary state row energies obey the closed-form law and are balanced by fractional power-law readout gains. (a) Across state channels, the analytical fixed point ${m}_{j}^{\ast}$ ( 5 ) closely tracks measured stationary row energy ${m}_{j}=\mathbb{E}[\|{\bm{S}}_{t,j,:}\|_{2}^{2}]$ in both GDN-2 and Muir. (b) Across all heads, weighting state rows by the learned readout gain compresses the within-head $\mathrm{p90}/\mathrm{p10}$ energy spread by $35\%$ relative to GDN-2. (c) Within each head, learned static gains $|{\gamma}_{j}|$ follow a fractional power law ${\gamma}_{j}\propto{\rho}_{j}^{-p}$ ( $p=0.35$ ) of stationary row RMS ${\rho}_{j},$ intermediate between unit readout ( $p=0$ ) and full whitening ( $p=1$ ).

Figure 12: Readout-exponent sweeps and erase-logit shifts confirm the optimality of fractional power-law readout and the early-vs.-late erasure trade-off. (a) Replacing the learned static gain in each head with an RMS-matched power law ${\gamma}_{j}(p)\propto{\rho}_{j}^{-p}$ yields a U-shaped loss curve minimized near the learned exponent ( $p\approx 0.35$ ), outperforming both unit readout ( $p=0$ ) and full whitening ( $p=1$ ). (b) Shifting the pre-activation erase logit ( ${\bm{z}}^{b}_{t}\leftarrow{\bm{z}}^{b}_{t}+\delta_{b}$ ) shows that weaker erasure degrades predictions across all positions, whereas stronger erasure leaves early tokens $[0,16)$ nearly unchanged while degrading late tokens $[256,4\text{K}).$

	Diagnostic	Statistic	Scale	GDN-2	Muir	Effect	Theory
Sens.	Key participation ratio	$\bar{\eta}_{\mathrm{eff}}=1/\mathbb{E}[\\|{\bm{k}}_{t}\\|_{4}^{4}]$	166M	7.43	5.98	$\ll\!128$ ch.	Remark 3.2
Sens.	Write-erase sensitivity	Layer-mean $\mathbb{E}[\\|\mathrm{D}_{{\bm{z}}^{w}}{\bm{S}}_{t}\\|_{F}^{2}]/\mathbb{E}[\\|\mathrm{D}_{{\bm{z}}^{b}}{\bm{S}}_{t}\\|_{F}^{2}]$	166M	13.2	13.3	$13\times$ $w$ vs. $b$	Proposition 3.1
Energy	Stationary row-law fit	Median within-head $r_{s}({m}_{j},{m}_{j}^{\ast})$	166M	0.957	0.960	$+\,0.003$	Proposition 3.4
	Raw row-energy spread	Median within-head $\mathrm{p90}/\mathrm{p10}$ of ${m}_{j}$	1.34B	14.7	17.2	$+$ 17.0%	Proposition 3.4
	Readout energy spread	Median within-head $\mathrm{p90}/\mathrm{p10}$ of $\bar{\gamma}_{j}^{2}{m}_{j}$	1.34B	14.7	9.6	$-$ 34.7%	Corollary 3.8
Erase	Parallel erase rate	$\bar{\mu}=\mathbb{E}[\langle{\bm{k}}_{t},{\bm{b}}_{t}\odot{\bm{k}}_{t}\rangle]$	1.34B	0.782	0.652	$-$ 16.6%	Proposition 3.10

Table 3: Trained GDN-2 and Muir checkpoints confirm the predicted write-erase sensitivity asymmetry, stationary row-energy law, and readout-induced erasure reduction. Diagnostics are measured on held-out FineWeb-Edu validation sequences (

T=4{,}096

) at

166\text{M}

(gate sensitivity and stationary row-law fit) and

1.34\text{B}

(within-head row-energy spread and write-time erasure).

Figure 13: Muir preserves early variable bindings and routes readout to the target hop at each answer step. (a) On 4-value MV-NIAH, Transformer drops on the final write $V_{4}$ and unit-readout recurrences degrade after $V_{1},$ whereas Muir remains accurate across $V_{1}\text{--}V_{4}.$ (b) On 5-hop RULER VT, Transformer recalls only the root assignment $H_{1},$ whereas Muir maintains accuracy across all five hops $H_{1}\text{--}H_{5}.$ (c) Within-model root-needle ( $H_{1}$ ) retention in ${\bm{S}}_{t}$ from write milestone $s_{1}$ ( $100\%$ ) to query milestone $t_{\mathrm{q}}$ across $16$ held-out prompts (interquartile bands across prompts). (d) Within-model retention of written needles $H_{1}\text{--}H_{5}$ from each write ( $100\%$ ) across sequence milestones $s_{1}\text{--}s_{5}$ to $t_{\mathrm{q}}.$ (e) Share of readout weight $\hat{\bm{q}}_{t}{\bm{S}}_{t}$ (over the haystack) on each needle $H_{1}\text{--}H_{5}$ across the four answer-generation steps $A_{1}\text{--}A_{4}$ that chain $H_{1}\to\dotsb\to H_{5}.$

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol		Definition
$L$	$\mathbb{N}$	Number of residual layers
$d$	$\mathbb{N}$	Model residual-stream width
$d_{\mathrm{ffn}}$	$\mathbb{N}$	FFN intermediate width ( $4d$ in Transformer, $3d-256$ in GDN-2, $4d-512$ in Muir)
$d_{k},d_{v}$	$\mathbb{N}$	Per-head key and value dimensions ( $d_{k}=d_{v}=128$ )
$N_{h}$	$\mathbb{N}$	Number of attention heads ( $N_{h}=d/d_{k}$ )
$T,C$	$\mathbb{N}$	Sequence length and chunk length ( $C=256$ )

Appendix

Table 4: Summary of notation. All vectors are row vectors; hats (

\hat{\cdot}

) denote gate-modulated vectors or decayed states, bars (

\bar{\cdot}

) denote expectations or empirical means, and asterisks ( ∗ ) denote stationary fixed points or optimal solutions.

	Per-Head Recurrence and Readout				Gate Projections
Model	Decay	Erase	Write	Readout	Scalar	Low-Rank	Dense
Lin. Attn. [ Katharopoulos et al., 2020 ]	$\mathbf{I}$	$\mathbf{I}$	${\bm{v}}_{t}$	${\bm{q}}_{t}{\bm{S}}_{t}$	–	–	–
RetNet [ Sun et al., 2023 ]	${\alpha}\mathbf{I}$	$\mathbf{I}$	${\bm{v}}_{t}$	${\bm{q}}_{t}{\bm{S}}_{t}$	– †	–	${\bm{g}}_{t}$
Mamba-2 [ Dao and Gu, 2024 ]	${\alpha}_{t}\mathbf{I}$	$\mathbf{I}$	${\beta}_{t}{\bm{v}}_{t}$	${\bm{q}}_{t}{\bm{S}}_{t}$	${\alpha}_{t},{\beta}_{t}$	–	${\bm{g}}_{t}$
GLA [ Yang et al., 2023 ]	$\Diag({\bm{\alpha}}_{t})$	$\mathbf{I}$	${\bm{v}}_{t}$	${\bm{q}}_{t}{\bm{S}}_{t}$	–	${\bm{\alpha}}_{t}$	${\bm{g}}_{t}$
DeltaNet [ Yang et al., 2024 ]	$\mathbf{I}$	$\mathbf{I}-{\beta}_{t}{\bm{k}}_{t}^{\top}{\bm{k}}_{t}$	${\beta}_{t}{\bm{v}}_{t}$	${\bm{q}}_{t}{\bm{S}}_{t}$	${\beta}_{t}$	–	${\bm{g}}_{t}$

Appendix

Table 5: Comparison of per-head state recurrence, query readout, and gate projection parameterizations across linear-time sequence models. For each model, we list the per-head decay, key-erasure, and value-write factors, the query readout, and whether each learned gate projection across heads is a per-head scalar (

d\to N_{h}

), low-rank bottleneck (

d\to r\to d

), or dense matrix (

d\to d

). ( † RetNet fixes its per-head decay scalar

{\alpha}

without an input projection; ‡ RWKV-7 combines decay and removal additively, writes with a decoupled replacement key

\tilde{\bm{k}}_{t}^{\top}{\bm{v}}_{t},

and includes a low-rank cross-layer value-residual gate.)

	Row-Energy Law	Key Rank	Max / Min	RMS Spread	Energy	Spearman	Log-Pearson
	Diagnostic ( ${\mu}=0.60$ )	$\mathbb{E}[\eta_{\mathrm{eff}}({\bm{k}}_{t})]$	$\max_{j}{m}_{j}/\min_{j}{m}_{j}$	$\mathrm{p90}/\mathrm{p10}({\rho}_{j})$	$\mathrm{p90}/\mathrm{p10}$	$r_{s}({m}_{j},{m}_{j}^{\ast})$	$r_{\log}$
	Stationary row energy ${m}_{j}$	12.24	52.4 $\times$	5.69 $\times$	32.3 $\times$	1.0000	1.0000
		Normalized Retrieval MSE $\downarrow$			Optimal Erase ${\mu}^{\ast}$
	Geometric Horizon Readout ${\bm{\gamma}}$	Horizon $H\!=\!10$	Horizon $H\!=\!27$	Horizon $H\!=\!81$	$H\!=\!10$	$H\!=\!27$	$H\!=\!81$
Geometric	Unit gain ${\gamma}_{j}=1$ (GDN-2)	0.3599 $\,\pm\,$ 0.0019	0.6389 $\,\pm\,$ 0.0017	0.8607 $\,\pm\,$ 0.0017	0.90	0.60	0.30
	Full whitening ${\gamma}_{j}\propto{\rho}_{j}^{-1}$ ( $p=1$ )	0.4068 $\,\pm\,$ 0.0025	0.6802 $\,\pm\,$ 0.0021	0.8817 $\,\pm\,$ 0.0016	0.80	0.50	0.20

Appendix

Table 6: Single-head stationary row-energy verification, geometric-horizon retrieval sweep, and bimodal two-lag readout comparison. All results are reported as

\text{mean}\pm\text{SD}

across

20

random seeds for a single recurrent head with

d_{k}=d_{v}=64.

(Top) Verification of the closed-form row-energy law at

{\mu}=0.60.

(Middle) Optimal erase strength

{\mu}^{\ast}

and normalized retrieval MSE under geometric lag distributions of mean

H\in\{10,27,81\}.

(Bottom) Retrieval MSE on short (

\ell=2

), long (

\ell=80

), and mixed (

50/50

) queries at

{\mu}=0.60,

along with the Spearman correlation

r_{s}({\bm{\gamma}},{\bm{\alpha}})

between readout gain

{\bm{\gamma}}

and the channel decay factor

{\bm{\alpha}}.

		Gate Config			Associative Recall Accuracy (%) $\uparrow$				Diagnostics
	Model	$r_{b}$	$r_{w}$	${\bm{\gamma}}_{t}$	Overall	Short	Medium	Long	Erase	Row Ener.
					acc $\uparrow$	$[8,32)$	$[32,128)$	$[128,256)$	tok var $\downarrow$	$\bar{m}_{T}$ $\downarrow$
	Dense $(b,w)$ (GDN-2)	64	64	${\bm{1}}$	90.40 $\,\pm\,$ 3.79	96.29 $\,\pm\,$ 2.77	77.32 $\,\pm\,$ 8.49	33.68 $\,\pm\,$ 9.47	27.9%	1,593.0
Erase $r_{b}$	Erase rank $r_{b}=1$	1	64	${\bm{1}}$	91.28 $\,\pm\,$ 3.26	97.06 $\,\pm\,$ 2.10	79.01 $\,\pm\,$ 8.12	35.77 $\,\pm\,$ 9.03	8.4%	1,451.0
	Erase rank $r_{b}=2$	2	64	${\bm{1}}$	91.33 $\,\pm\,$ 2.91	97.04 $\,\pm\,$ 1.68	79.30 $\,\pm\,$ 7.46	35.76 $\,\pm\,$ 9.45	3.0%	1,436.1
	Erase rank $r_{b}=4$	4	64	${\bm{1}}$	91.51 $\,\pm\,$ 3.37	97.03 $\,\pm\,$ 2.36	80.05 $\,\pm\,$ 7.91	36.91 $\,\pm\,$ 9.07	4.0%	1,446.3

Appendix

Table 7: Learned multi-query associative recall accuracy, erase-logit token variance, and state row energy across gate-rank and readout-gain configurations. All results are evaluated on a two-head recurrent model with

d_{k}=d_{v}=32

and residual width

d=128

across

20

random seeds (mean

\pm

SD; dense gates have full head-channel rank

N_{h}d_{k}=64

). We report read accuracy overall and by event lag across single-gate rank sweeps and readout-gain variants with low-rank

(b,w)

r_{b}=r_{w}=4

(

r_{q}=8

for dynamic

{\bm{\gamma}}_{t}

), along with the token-dependent fraction of projected erase-logit energy on write steps

\mathbb{E}[\|\tilde{\bm{z}}^{b}_{t}-\bar{\bm{z}}^{b}\|_{2}^{2}]/\mathbb{E}[\|\tilde{\bm{z}}^{b}_{t}\|_{2}^{2}]

(

\tilde{\bm{z}}^{b}_{t}={\bm{z}}^{b}_{t}-{\bm{c}}_{b}

) and mean terminal row energy

\bar{m}_{T}=(N_{h}d_{k})^{-1}\mathbb{E}[\|{\bm{S}}_{T}\|_{F}^{2}].

	Naive Chunkwise			Boundary-Pivoted Kernel (Ours)
Kernel Stage	MXU	VPU	MXU	MXU	VPU	MXU
	FLOPs	FLOPs	Share	FLOPs	FLOPs	Share
Diag. blocks ( $n_{0}$ ), ${\bm{T}}_{\mathrm{leaf}}^{-1}$ ( 12 )	$0$	${\mathcal{O}}(n_{0}d_{k})$	0.0%	$4n_{0}^{2}(\log_{2}n_{0}\!-\!1)$	${\mathcal{O}}(n_{0}d_{k})$	6.3%
Off-diag. blocks ( ${\bm{T}}_{\mathrm{cross}},{\bm{M}}_{\mathrm{cross}}$ )	$0$	$7(C\!-\!n_{0})d_{k}$	0.0%	$2(C\!-\!n_{0})d_{k}$	${\mathcal{O}}(rd_{k})$	96.6%
$2\!\times\!2$ block-tri. merges ( 11 )	$0$	$0$	–	$\tfrac{2}{3}(C^{2}\!-\!n_{0}^{2})$	$0$	100.0%
Tri. solve ${\bm{T}}{\bm{Z}}=[\hat{\bm{V}}\;\,\hat{\bm{K}}_{\odot\pi}]$	$0$	$C(d_{k}\!+\!d_{v})$	0.0%	$2C(d_{k}\!+\!d_{v})$	$0$	100.0%

Appendix

Table 8: Per-token, per-head FLOP breakdown of the boundary-pivoted chunkwise kernel and naive chunkwise execution. Stage rows report leading per-token MXU and VPU operation counts (given cumulative log-decay

{\bm{\Lambda}}

) as a function of chunk length

C,

head dimensions

(d_{k},d_{v}),

leaf size

n_{0},

and tree depth

r=\log_{2}(C/n_{0}).

MXU shares and totals are evaluated at

C=256,

d_{k}=d_{v}=128,

and

n_{0}=8

(

r=5

		Throughput ( K tokens/s )					XLA Profile ( $4\text{K}\to 32\text{K}$ )
	Model ( $1.34\text{B}$ )	$T\!=\!2\text{K}$	$T\!=\!4\text{K}$	$T\!=\!8\text{K}$	$T\!=\!16\text{K}$	$T\!=\!32\text{K}$	Compiled	Achieved	HBM Traffic
		( $B\!=\!256$ )	( $B\!=\!128$ )	( $B\!=\!64$ )	( $B\!=\!32$ )	( $B\!=\!16$ )	TFLOP/dev	MFU ( $\%$ )	(GB/dev)
$\mathrm{p50}$	Transformer ( $8{,}192$ )	523.5	396.9	267.9	163.6	93.6	348 $\,\to\,$ 652	18.2 $\,\to\,$ 8.6	613 $\,\to\,$ 750
	GDN-2 ( $5{,}888$ )	179.6	179.3	179.1	264.3	253.5	330 $\,\to\,$ 330	7.1 $\,\to\,$ 10.1	4,478 $\,\to\,$ 4,148
	Muir (matched, $5{,}888$ )	183.4	183.2	182.6	276.8	261.8	285 $\,\to\,$ 285	6.2 $\,\to\,$ 8.9	4,413 $\,\to\,$ 4,058
	Muir (ours; iso, $\bm{7{,}680}$ )	179.5	179.4	175.6	268.5	256.1	324 $\,\to\,$ 324	7.1 $\,\to\,$ 10.2	4,445 $\,\to\,$ 4,089

Appendix

Table 9: End-to-end training throughput, compiled XLA FLOPs, and HBM traffic across sequence lengths at

\bm{1.34\text{B}}

scale. All measurements are collected on

8

TPU7x chips over

100

-step pure-training intervals at a fixed global batch size of

B\times T=524{,}288\text{ tokens/step},

comparing iso-parameter Muir (

d_{\mathrm{ffn}}=7{,}680

) and matched-FFN Muir (

d_{\mathrm{ffn}}=5{,}888

) against Transformer (TPU Pallas Splash Attention) and GDN-2. (XLA profile metrics are reported per TensorCore device across the

16

TensorCores of the

8

TPU7x chips; compiled TFLOP/dev includes activation rematerialization, whereas MFU counts non-rematerialized model FLOPs. Bottom-row XLA profile ratios are evaluated at

T=32\text{K}.

)

		Configuration				Gate Ranks & FFN Width						Param Breakdown (M)
	Model	$L$	$d$	$N_{h}$	$d_{k}=d_{v}$	$r_{f}$	$r_{b}$	$r_{g}$	$r_{w}$	$r_{q}$	$d_{\mathrm{ffn}}$	Mixer	FFN	Total
												+ Gates	Params	Params
166M	Transformer	8	1,024	8	128	–	–	–	–	–	4,096	33.55	67.11	166.20
	GDN-2	8	1,024	8	128	128	1,024	128	1,024	–	2,816	54.53	46.14	166.20
	Muir (no ${\bm{\gamma}}$ )	8	1,024	8	128	64	64	128	256	0	3,584	41.94	58.72	166.20
	Muir (static ${\bm{\gamma}}$ )	8	1,024	8	128	64	64	128	256	0	3,584	41.95	58.72	166.21

Appendix

Table 11: Architectural specifications and parameter accounting across all four iso-parameter scales (

\bm{166\text{M}}

\bm{1.34\text{B}}

). All models at a given scale share depth

L,

residual width

d,

head dimensions

d_{k}=d_{v}=128,

head count

N_{h}=d/128

(so all recurrent models are strictly iso-state with

N_{h}d_{k}d_{v}=128d

state elements per layer), vocabulary size

V=32{,}000

(untied input/output embeddings), per-layer non-embedding budget

12d^{2},

global batch size

524{,}288\text{ tokens/step}

(

B=128,

T=4{,}096

), and

100\text{B}

training tokens.

		Perplexity		Zero-Shot Accuracy (%)
	Model	FW-Edu	WikiText	BoolQ	Avg. (9)
		Token PPL $\downarrow$	( $4\text{K}$ ) PPL $\downarrow$	acc $\uparrow$	w/ BoolQ $\uparrow$
166M	Transformer	11.67	30.15	59.11	42.04
	GDN-2	11.55	29.32	53.46	41.94
	Muir (no ${\bm{\gamma}}$ )	11.38	28.87	58.99	43.49
	Muir (static ${\bm{\gamma}}$ )	11.33	28.31	57.13	42.74

Appendix

Table 12: Supplementary language modeling perplexities, zero-shot BoolQ accuracy, and 9-task commonsense average across all four scales. Complementing Table 1 , we report FineWeb-Edu validation token perplexity (PPL), full-window (

T=4{,}096

) WikiText word-level perplexity, zero-shot BoolQ accuracy, and the 9-task zero-shot commonsense average including BoolQ (“Avg. (9)”).

	S-NIAH-1				S-NIAH-2				S-NIAH-3				S-NIAH-1/2/3 Avg.
Model ( $1.34\text{B}$ )	1K	2K	4K	8K	1K	2K	4K	8K	1K	2K	4K	8K	1K	2K	4K	8K	1–4K	1–8K
Transformer	100.0	100.0	99.6	0.0	99.8	100.0	100.0	0.0	98.0	96.4	80.8	0.0	99.3	98.8	93.5	0.0	97.2	72.9
GDN-2	99.8	99.2	86.2	40.8	100.0	99.6	83.6	51.6	77.4	57.6	20.6	9.8	92.4	85.5	63.5	34.1	80.4	68.9
Muir (no ${\bm{\gamma}}$ )	100.0	100.0	99.6	99.8	98.2	99.8	93.8	39.2	84.0	80.4	41.6	10.4	94.1	93.4	78.3	49.8	88.6	78.9
Muir (static ${\bm{\gamma}}$ )	100.0	100.0	100.0	99.6	100.0	100.0	99.0	41.0	96.2	80.6	43.8	13.6	98.7	93.5	80.9	51.4	91.1	81.2
Muir (ours)	100.0	100.0	100.0	100.0	100.0	99.8	96.2	32.0	88.2	67.4	37.8	7.0	96.1	89.1	78.0	46.3	87.7	77.4

Appendix

Table 13: Complete synthetic associative recall accuracy on the six RULER benchmarks at

\bm{1.34\text{B}}

parameters. (Top) Single-needle retrieval on S-NIAH-1/2/3, and their 3-task mean. (Bottom) Multi-hop VT, multi-value (MV-NIAH) and multi-query (MQ-NIAH) retrieval, and suite-wide averages across multi-needle retrieval (MV+MQ), all six tasks at

1\text{K}\text{--}4\text{K},

and all evaluated lengths (

1\text{K}\text{--}8\text{K}

		Extraction (%) $\uparrow$		Passage Q/A ( $2\text{K}$ Cut; %) $\uparrow$				6-Task Avg. (%) $\uparrow$
	Model	SWDE	FDA	SQuAD	TQA	NQ	DROP	$2\text{K}$ Cut	$1\text{K}$ Cut	$2\text{K}$ Trunc.
166M	Transformer	29.80	39.51	30.47	44.91	17.83	16.20	29.79	31.41	23.33
	GDN-2	12.37	9.90	25.56	44.25	12.13	17.01	20.20	21.60	19.41
	Muir (no ${\bm{\gamma}}$ )	12.00	8.90	27.38	46.80	13.97	18.97	21.34	22.29	20.41
	Muir (static ${\bm{\gamma}}$ )	10.50	7.63	25.26	45.85	12.48	17.87	19.93	22.01	18.96
	Muir (ours)	10.97	5.27	27.68	46.27	13.40	18.78	20.39	21.78	19.51

Appendix

Table 14: Complete real-world extractive recall accuracy on the six Based/JRT benchmarks across all four model scales. Per-task accuracies are reported under the official

2\text{K}

context-cut JRT protocol (

12{,}077

validation examples across structured attribute extraction and passage question answering; the

1.34\text{B}

2\text{K}

context-cut average is reported in Table 2 ), alongside the 6-task average under

1\text{K}

context-cut truncation and un-cut

2\text{K}

left-truncation.

Contiguous FineWeb-Edu Long Documents
		PPL $\downarrow$						$\Delta_{\mathrm{tail}}$ (nats) $\downarrow$
	Model	$4\text{K}$	$8\text{K}$	$16\text{K}$	$32\text{K}$	$64\text{K}$	$128\text{K}$
166M	Transformer	11.55	50.97	199.85	473.31	728.84	895.74	$+$ 4.5349 $\,\pm\,$ 0.0798
	GDN-2	11.69	11.44	11.32	11.29	11.28	11.28	$+$ 0.0051 $\,\pm\,$ 0.0018
	Muir (no ${\bm{\gamma}}$ )	11.49	11.25	11.14	11.11	11.10	11.10	$+$ 0.0051 $\,\pm\,$ 0.0015
	Muir (static ${\bm{\gamma}}$ )	11.39	11.14	11.03	11.00	10.99	10.99	$+$ 0.0076 $\,\pm\,$ 0.0020

Appendix

Table 15: Zero-shot length extrapolation from

\bm{4\text{K}}

\bm{128\text{K}}

on contiguous FineWeb-Edu long documents and packed control streams. For each evaluation window

T\in\{4\text{K},8\text{K},16\text{K},32\text{K},64\text{K},128\text{K}\},

we report aggregate token perplexity alongside the paired tail NLL differential

\Delta_{\mathrm{tail}}\mathrel{\mathchoice{\vbox{\hbox{

\displaystyle:

}}}{\vbox{\hbox{

\textstyle:

}}}{\vbox{\hbox{

\scriptstyle:

}}}{\vbox{\hbox{

\scriptscriptstyle:

}}}{=}}\mathrm{NLL}_{[127\mathrm{K},128\mathrm{K})}(T=128\text{K})-\mathrm{NLL}_{[127\mathrm{K},128\mathrm{K})}(T=4\text{K})

(mean

\pm

paired SE in nats across

N_{\mathrm{docs}}

sequences).

		RULER S-NIAH-1 Accuracy (%) $\uparrow$											Avg. (%) $\uparrow$
	Model ( $1.34\text{B}$ )	$1\text{K}$	$2\text{K}$	$4\text{K}$	$8\text{K}$	$16\text{K}$	$32\text{K}$	$64\text{K}$	$128\text{K}$	$256\text{K}$	$512\text{K}$	$1\text{M}$	$1\text{--}32\text{K}$	$1\text{K--}1\text{M}$
Zero-Shot	Transformer (YaRN)	100.0	100.0	90.0	90.6	78.8	68.1	45.6	OOM	OOM	OOM	OOM	87.9	–
	GDN-2	100.0	98.8	80.0	40.0	13.8	11.9	34.4	23.8	15.0	10.0	10.0	57.4	39.8
	Muir (no ${\bm{\gamma}}$ )	100.0	100.0	100.0	100.0	98.1	73.8	46.2	46.9	26.2	15.6	11.9	95.3	65.3
	Muir (static ${\bm{\gamma}}$ )	100.0	100.0	100.0	99.4	100.0	98.8	98.1	98.8	95.6	91.9	90.0	99.7	97.5
	Muir (ours)	100.0	100.0	100.0	100.0	100.0	85.6	26.2	22.5	17.5	13.1	12.5	97.6	61.6

Appendix

Table 17: Zero-shot and post-cooldown RULER S-NIAH-1 passkey retrieval from

\bm{1\text{K}}

\bm{1\text{M}}

tokens at

\bm{1.34\text{B}}

parameters. Complementing the depth-by-length heatmaps in Figure 8 , each context length

T\in\{1\text{K},\dotsc,1\text{M}\}

evaluates

N=160

examples stratified uniformly across

10

needle-depth bins under greedy decoding, both zero-shot (

4\text{K}

) and after the

64\text{K}

continual cooldown (

0.315\text{B}

FineWeb-Edu tokens); unchunked Transformer (YaRN) prefill exceeds HBM (OOM) at

T\geq 128\text{K}.

Under Zero-Shot, the bottom two rows freeze dynamic readout in Muir at inference time to

{\bm{\gamma}}^{\mathrm{stat}}

(

{\bm{z}}^{q}_{t}\leftarrow{\bm{0}}

) or its held-out text mean

\bar{\bm{\gamma}}=\mathbb{E}[{\bm{\gamma}}_{t}].

			Gate Ranks & FFN ( $d=1{,}536$ )						FineWeb-Edu Val			WikiText
	Model ( $406\text{M}$ )	Params	$r_{f}$	$r_{b}$	$r_{g}$	$r_{w}$	$r_{q}$	$d_{\mathrm{ffn}}$	Token	$\Delta\text{NLL}$	Token	( $4\text{K}$ )
		(M)							NLL $\downarrow$	(mnat) $\downarrow$	PPL $\downarrow$	PPL $\downarrow$
Base	Transformer	409.7	–	–	–	–	–	6,144	2.2424	$+$ 7.0	9.42	21.94
Base	GDN-2	409.7	128	$d$	128	$d$	–	4,352	2.2354	0.0	9.35	21.76
Ours	Muir (no ${\bm{\gamma}}$ )	409.7	64	64	128	256	0	5,632	2.2230	$-$ 12.4	9.24	21.16
	Muir (static ${\bm{\gamma}}$ )	409.7	64	64	128	256	0	5,632	2.2191	$-$ 16.3	9.20	21.24

Appendix

Table 18: Fixed-FFN gate-compression, dense dynamic-readout, and unconstrained wide-FFN controls at the

\bm{406\text{M}}

scale. Complementing the multi-scale rank ablations in Table 2 , we evaluate fixed-FFN gate compression (

d_{\mathrm{ffn}}=4{,}352

), dense dynamic-readout (

r_{q}=d,

d_{\mathrm{ffn}}=3{,}584

), and non-iso-parameter wide-FFN (

d_{\mathrm{ffn}}=5{,}504,

448.7\text{M}

) controls at the

406\text{M}

scale (

L=11,

d=1{,}536,

100\text{B}

tokens);

\Delta\text{NLL}

is reported in millinats relative to GDN-2, with best and second-best iso-parameter models in bold and underlined .

	Kept SVD Rank $r$ (Validation $\Delta\text{NLL}$ vs. $r=128$ in nats $\pm$ SE $\downarrow$ )
Gate	$r=0$ (bias)	$r=16$	$r=32$	$r=48$	$r=64$	$r=96$
Decay $f$ ( ${\bm{\alpha}}_{t}$ )	$+$ 0.5801 $\,\pm\,$ 0.2147	$+$ 0.0259 $\,\pm\,$ 0.0084	$+$ 0.0043 $\,\pm\,$ 0.0018	$+$ 0.0020 $\,\pm\,$ 0.0015	$-$ 0.0007 $\,\pm\,$ 0.0015	$-$ 0.0001 $\,\pm\,$ 0.0004
Erase $b$ ( ${\bm{b}}_{t}$ )	$+$ 0.1193 $\,\pm\,$ 0.0154	$+$ 0.0039 $\,\pm\,$ 0.0004	$+$ 0.0016 $\,\pm\,$ 0.0009	$+$ 0.0006 $\,\pm\,$ 0.0008	$+$ 0.0006 $\,\pm\,$ 0.0004	$+$ 0.0000 $\,\pm\,$ 0.0001
Write $w$ ( ${\bm{w}}_{t}$ )	$+$ 0.4454 $\,\pm\,$ 0.2248	$+$ 0.0486 $\,\pm\,$ 0.0048	$+$ 0.0185 $\,\pm\,$ 0.0022	$+$ 0.0108 $\,\pm\,$ 0.0015	$+$ 0.0064 $\,\pm\,$ 0.0010	$+$ 0.0013 $\,\pm\,$ 0.0007
Query $q$ ( ${\bm{\gamma}}_{t}$ )	$+$ 0.5639 $\,\pm\,$ 0.2352	$+$ 0.0296 $\,\pm\,$ 0.0057	$+$ 0.0118 $\,\pm\,$ 0.0023	$+$ 0.0077 $\,\pm\,$ 0.0013	$+$ 0.0045 $\,\pm\,$ 0.0013	$+$ 0.0015 $\,\pm\,$ 0.0008
Output $g$ ( ${\bm{g}}_{t}$ )	$+$ 2.7725 $\,\pm\,$ 0.2189	$+$ 0.7489 $\,\pm\,$ 0.2328	$+$ 0.4824 $\,\pm\,$ 0.2483	$+$ 0.3492 $\,\pm\,$ 0.2225	$+$ 0.2472 $\,\pm\,$ 0.1783	$+$ 0.0376 $\,\pm\,$ 0.0209

Appendix

Table 19: Post-hoc SVD gate-rank truncation sensitivity on the trained

\bm{406\text{M}}

uniform-rank checkpoint. For each gate

m\in\{f,b,g,w,q\},

we replace its rank-

128

factor

{\bm{W}}_{m\downarrow}{\bm{W}}_{m\uparrow}

across all

11

layers with its best rank-

r

SVD approximation (retaining the additive bias

{\bm{c}}_{m}

) and measure

\Delta\text{NLL}(r)\mathrel{\mathchoice{\vbox{\hbox{

\displaystyle:

}}}{\vbox{\hbox{

\textstyle:

}}}{\vbox{\hbox{

\scriptstyle:

}}}{\vbox{\hbox{

\scriptscriptstyle:

}}}{=}}\mathrm{NLL}(r)-\mathrm{NLL}(128)

(in nats

\pm

paired SE) on held-out FineWeb-Edu sequences.

Diagnostic	Within-Head Statistic	Scale	GDN-2	Muir	Effect	Theory
Full row-law log-Pearson $r$	$r(\log_{10}{m}_{j},\log_{10}{m}_{j}^{\ast})$	166M	0.951	0.951	$0.932/0.916$ pooled	Prop. 3.4
Decay-only log-Pearson $r$	$r(\log_{10}{m}_{j},-\log_{10}(1-{\lambda}_{j}))$	166M	0.840	0.803	$-0.13$ ( $0.869/0.827$ pooled)	Rem. 3.5
Static gain alignment	$r_{s}(\|{\gamma}_{j}\|,{\rho}_{j}^{-1})$	166M	–	$+$ 0.710	$100\%$ heads $\!>\!0$	Cor. 3.8
Readout power-law slope	$-\mathrm{d}\log\bar{\gamma}_{j}/\mathrm{d}\log{\rho}_{j}$	1.34B	0.00	0.44	$p\in(0,1)$	Cor. 3.8
Readout row-RMS spread	$\mathrm{p90}/\mathrm{p10}$ of $\bar{\gamma}_{j}{\rho}_{j}$	1.34B	3.83 $\times$	3.10 $\times$	$-$ 19.1%	Cor. 3.8

Appendix

Table 20: Supplementary stationary row-energy and readout-scaling diagnostics across trained GDN-2 and Muir checkpoints. Complementing Table 3 , row-law ablations and static-gain alignment are evaluated across live channels (

{\rho}_{j}\geq 10^{-10}

) on the

166\text{M}

GDN-2 and Muir (static

{\bm{\gamma}}

) checkpoints, and dynamic-gain scaling and readout row-RMS spread are evaluated across all

24

layers (

384

heads) at

1.34\text{B}

(with

\bar{\gamma}_{j}=1

in GDN-2); values report the median across heads.

				Validation $\Delta\text{NLL}$ ( $\text{nats}\pm\text{SE}$ ) Across Token Positions
	Intervention	Model	Mean Erase	Overall	Early	Mid	Late
		( $834\text{M}$ )	$\bar{\mu}$	$[0,4{,}096)$	$[0,16)$	$[16,256)$	$[256,4{,}096)$
Freeze	Freeze write gate	GDN-2	0.733	$+$ 0.8325 $\,\pm\,$ 0.0581	$+$ 0.5799 $\,\pm\,$ 0.1467	$+$ 0.9517 $\,\pm\,$ 0.1275	$+$ 0.8261 $\,\pm\,$ 0.0566
	Freeze erase gate	GDN-2	0.672	$+$ 0.0574 $\,\pm\,$ 0.0066	$+$ 0.0153 $\,\pm\,$ 0.0143	$+$ 0.0523 $\,\pm\,$ 0.0152	$+$ 0.0578 $\,\pm\,$ 0.0062
	Isotropic erase ${\bm{b}}_{t}\leftarrow{\mu}_{t}{\bm{1}}$	GDN-2	0.732	$+$ 0.0844 $\,\pm\,$ 0.0044	$+$ 0.0189 $\,\pm\,$ 0.0116	$+$ 0.0658 $\,\pm\,$ 0.0157	$+$ 0.0859 $\,\pm\,$ 0.0044
	Freeze write gate	Muir	0.610	$+$ 0.3845 $\,\pm\,$ 0.0368	$+$ 0.2534 $\,\pm\,$ 0.0767	$+$ 0.4877 $\,\pm\,$ 0.1309	$+$ 0.3786 $\,\pm\,$ 0.0328

Appendix

Table 21: Causal gate-freezing and erase-logit shift interventions on trained

\bm{834\text{M}}

checkpoints. (Top) Validation

\Delta\text{NLL}

and mean erase rate

\bar{\mu}

when replacing the dynamic write gate

{\bm{w}}_{t}

or erase gate

{\bm{b}}_{t}

with its stream mean (

\bar{\bm{w}}

\bar{\bm{b}}

) or isotropic key-space projection (

{\mu}_{t}{\bm{1}}

) across all layers. (Bottom) Additive erase-gate logit shifts

{\bm{z}}^{b}_{t}\leftarrow{\bm{z}}^{b}_{t}+\delta_{b}

for

\delta_{b}\in\{-1.0,-0.5,+0.5\},

reported overall (

[0,4{,}096)

) and stratified by token position bucket (

[0,16),

[16,256),

and

[256,4{,}096)

		MV-NIAH Recall (%) $\uparrow$					5-Hop VT Recall (%) $\uparrow$
	Model ( $1.34\text{B}$ )	$V_{1}$	$V_{2}$	$V_{3}$	$V_{4}$	Avg.	$H_{1}$	$H_{2}$	$H_{3}$	$H_{4}$	$H_{5}$	Avg.
1K	Transformer	84.8	87.2	85.6	66.0	80.9	25.8	5.8	6.0	6.6	5.2	9.9
	GDN-2	67.0	57.0	62.0	74.4	65.1	28.4	34.4	49.4	61.6	64.6	47.7
	Muir (no ${\bm{\gamma}}$ )	85.6	46.0	49.0	57.4	59.5	29.2	36.0	49.0	57.2	50.8	44.4
	Muir (static ${\bm{\gamma}}$ )	57.4	55.2	57.2	59.6	57.4	13.2	17.2	20.8	23.8	20.2	19.0
	Muir (ours)	82.4	84.4	86.6	91.6	86.2	48.0	46.0	61.2	69.2	75.4	60.0

Appendix

Table 23: Write-order MV-NIAH and hop-resolved variable tracking recall at

\bm{1.34\text{B}}

parameters. All results are evaluated over

N=500

examples per task at

1\text{K}

(top) and

4\text{K}

context lengths (bottom). MV-NIAH columns

V_{1}\text{--}V_{4}

order the four target values bound to the shared key from oldest write (

V_{1}

) to most recent write (

V_{4}

), and VT columns

H_{1}\text{--}H_{5}

order the five target variables from earliest root binding (

H_{1},

VAR_1 = V ) to latest alias (

H_{5},

VAR_5 = VAR_4 ).

Delta-Rule Linear Attention

Delta-rule linear attention updates a compact recurrent memory to retrieve and revise sequence information with linear-time processing.

See all 39 papers

May 21, 2026

Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory. The hard part is not just what to forget, but how to edit this compressed memory without scrambling existing associations. Delta-rule models subtract the current read before writing a new value, and Kimi Delta Attention (KDA) sharpens forgetting with channel-wise decay. But the active edit still uses a single scalar gate to control two different things: how much old content to erase on the key side and how much new content to commit on the value side. We introduce Gated DeltaNet-2, which generalizes both Gated DeltaNet and KDA by inheriting adaptive forgetting and channel-wise decay while addressing their shared limitation, the scalar tie between erasing and writing. Gated Delta Rule-2 separates these roles with a channel-wise erase gate b_t and a channel-wise write gate w_t, reducing to KDA when both gates collapse to the same scalar and to Gated DeltaNet when the decay also collapses. We derive a fast-weight update view, a chunkwise WY algorithm with channel-wise decay absorbed into asymmetric erase factors, and a gate-aware backward pass that preserves efficient parallel training. At 1.3B parameters trained on 100B FineWeb-Edu tokens, Gated DeltaNet-2 achieves the strongest overall results among Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants across language modeling, commonsense reasoning, and retrieval. Its advantage is most pronounced on long-context RULER needle-in-a-haystack benchmarks, where it improves the evaluated multi-key retrieval setting and remains strong in both recurrent and hybrid settings. Code is available at https://github.com/NVlabs/GatedDeltaNet-2.

Ali Hatamizadeh, Yejin Choi, Jan Kautz

NVIDIA

Long-Context Language Modeling Recurrent Transformers Linear Attention Delta-Rule Linear Attention Gated Delta Networks

Ask PDF

Oct 28, 2026

Dual DeltaNet: Value-Side Memory Reorganization for Linear Attention

Linear attention and state space models (SSMs) offer efficient alternatives to softmax attention by maintaining a fixed-size recurrent state, but this design requires careful management to preserve useful historical associations and limit interference. The delta rule helps reduce interference through correction of the value associated with each key, but it does not explicitly account for how related value features are distributed across the state. We observe that a state

\boldsymbol{S}\in\mathbb{R}^{d_k\times d_v}

can be viewed as a bank of

d_k

addresses storing

d_v

-dimensional values. While a key-side read

\boldsymbol{S}^\top\boldsymbol{k}

retrieves the value associated with

\boldsymbol{k}

, a value-side read

\boldsymbol{S}\boldsymbol{e}

aggregates addresses whose stored value features match the selector

\boldsymbol{e}

. Based on this observation, we propose Dual DeltaNet (DDN), which extends Kimi Delta Attention (KDA) with a two-sided delta rule. The key-side delta rule reads with the current key to correct its associated value, while the complementary value-side delta rule reads with selected features of the current value to reorganize the addresses it occupies. We derive a chunkwise parallel training algorithm and a rank-2 recurrence that increases arithmetic intensity during inference. Compared with KDA, GDN-2, and Mamba-3, DDN achieves the strongest overall performance across language modeling, zero-shot downstream tasks, and retrieval, while incurring modest training and inference overhead relative to KDA. Code: https://github.com/ReichtumQian/DualDeltaNet

Yixiao Qian, Yaxin Zhang, Song Chen +3

College of Control Science and Engineering, Zhejiang University · Polytechnic Institute, Zhejiang University · Department of Mathematics, National University of Singapore +1

Linear Attention Delta-Rule Linear Attention Language Modeling

Ask PDF

Oct 30, 2026

DAMI: Dual-Axis Memory Initialization for Enhanced Long-Context Retrieval in Gated Linear Attention

Linear attention scales linearly with sequence length, but its finite-state memory can limit long-context retrieval. In gated linear recurrences, memory retention depends on state decay, which is shaped jointly by gate parameters A_log and dt_bias and by input activations. In controlled Gated DeltaNet (GDN) experiments, both parameters remain highly correlated with their initial values after training, highlighting the importance of initialization. Therefore, we introduce Dual-Axis Memory Initialization (DAMI), which jointly configures these parameters before training. DAMI assigns geometric timescales across heads for varied retrieval distances and a low-high-low input-sensitivity profile across layers. A small activation sample calibrates the GDN-family implementation to match both behavioral targets without adding trainable parameters. Across GDN, KDA, GDN2, Mamba-2, and RWKV-7, DAMI improves average RULER-style retrieval accuracy by 5.29 to 16.71 percentage points over the corresponding random-initialization baselines. Performance on language modeling, commonsense reasoning, and 2K real-world retrieval remains broadly comparable to those baselines. Our code is available at https://speedai.com/research/dami.

Zhangchi Feng, Harry Jiang, Yangyifei Luo +2

SpeedAI · CCSE, School of Computer Science and Engineering, Beihang University

Long-Context Retrieval Linear Attention Gated Delta Networks

Ask PDF

Abstract

Delta-Rule Linear Attention

Explore similar work

Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

Dual DeltaNet: Value-Side Memory Reorganization for Linear Attention

DAMI: Dual-Axis Memory Initialization for Enhanced Long-Context Retrieval in Gated Linear Attention