Shallow Encoding Emits Deep Memories

Mingyuan Ma; Yangmin Li; Harris Nover; Zheyu Fu; Po-Han Huang; Alex Settle; Ashwin Nanjappa

cs.LGNov 4, 2026

Shallow Encoding Emits Deep Memories

Authors: Mingyuan Ma, Yangmin Li, Harris Nover, Zheyu Fu, Po-Han Huang, Alex Settle, Ashwin Nanjappa

Organizations: NVIDIA

Abstract

Hybrid language models interleave a few softmax-attention layers with many linear-attention or state-space layers, so every layer keeps a memory of earlier tokens: a key-value cache or a fixed-size recurrent state. Serving them spends more compute and high-bandwidth memory on these memories than the output requires. The prefill runs every layer for every prompt token, and every decode step loads and stores whole recurrent states, which on Qwen3.8-Flash-Next take as many bytes per request as the attention cache of thousands of tokens. We find two properties of trained hybrids that locate this excess. First, writes are easier than reads: writing a deep layer's memory of a token needs little of the deep computation that predicting the next token needs. Trained copies of the deep layers' write paths, which we call emitters, therefore write a prompt's deep memories from the residual stream of a shallower anchor layer, and all prompt tokens but the last run as few as 63% of the layers. Second, recurrent states have a state sink: most of their energy lies in a rank-one term along a fixed direction of each head, and the rest is a low-rank working memory. SEED builds on both: it makes the prefill shallow in depth and the decode state shallow in representation. Each recurrent state is stored as its sink plus low-rank factors, which the Gated DeltaNet, Kimi Delta Attention and Mamba-2 updates extend exactly; we prove that the open-loop error of the periodic compressions back to low rank is bounded independently of their number. SEED keeps the original weights frozen, compresses no attention cache, and runs all layers for every generated token. On six checkpoints of five production hybrids of 30B to over 300B parameters, it is nearly lossless on eleven official exams. Served with SGLang on NVIDIA GB300 GPUs on an agentic-coding trace, it raises the per-GPU throughput of Qwen3.8-Flash-Next 1.2-1.4x.

Figures & tables

Figure 1: SEED keeps a shallow encoding of a hybrid model’s memories in each phase. Left: original model; right: SEED; layer stacks as in Qwen3.8-Flash-Next, with three recurrent layers per attention layer. (a) Prefill, shallow in depth:

x_{1}..x_{N-1}

run layers

0..k{-}1

only. Emitters, trained copies of the deep layers’ write paths, write every deeper memory from

h^{(k)}

, the output of anchor layer

k{-}1

, directly or through an optional prompt code. At the end of the prefill, the states left by

x_{1}..x_{N-1}

are compressed to the shallow form of (b);

x_{N}

runs all

L

layers. (b) Decode, shallow in representation: each stored recurrent state is its sink

a\bar{v}^{\top}

plus rank-

R

factors

UV^{\top}

. A step updates the sink’s coefficient

a

, decays

U

and appends one column to each factor, which keeps the update exact; every

W

steps, the factors are compressed back to rank

R

. Every generated token runs all

L

layers, and no attention cache is compressed.

	Prefill (compute-bound)	Decode (HBM-bound)
Property	writes are easier than reads: a deep layer’s write needs little of the deep computation (§ 3 )	the state sink: a rank-one term along a fixed direction of each head holds most of the states’ energy; the rest is low-rank (§ 4 )
Model reads	caches and states of layers $k..L{-}1$ for each prompt token	read-out and update of each $d_{k}{\times}d_{v}$ recurrent state
Shallow encoding	in depth : prompt tokens $x_{1}..x_{N-1}$ run only layers $0..k{-}1$ ; their output $h^{(k)}$ stands in for layers $k..L{-}1$	in representation : the shallow state, the sink $a\bar{v}^{\top}$ plus rank- $R$ factors $U$ , $V$
Emitted by	emitters, trained copies of the deep layers’ write paths, write every deep cache and state from $h^{(k)}$	the factors, which give the read-out and the exact update; compressed to rank $R$ every $W$ steps
Saves	prompt tokens but the last run as few as 63% of the layers and 71% of the parameter FLOPs	the state is 2.0–3.9 $\times$ smaller after a compression, 1.3–2.6 $\times$ on average
States	every stored recurrent state is shallow: the states that $x_{1}..x_{N-1}$ leave are compressed to the sink plus rank- $R$ factors at the end of the prefill, and the decode keeps this form

Table 1: SEED in the two serving phases. In each phase, a property of trained hybrids lets SEED keep an encoding of the memories that is shallow along the phase’s bottleneck, compute in the prefill and HBM in the decode, and emit from it what the model reads.

	Qwen3.8- Flash-Next		Nemotron-3.5- Lightning		Kimi- Linear		GLM-5.3- Flash		Nemotron-3.5- Super
	Orig.	SEED	Orig.	SEED	Orig.	SEED	Orig.	SEED	Orig.	SEED
State at $R$ (MB)	113.2	29.2	48.2	18.8	41.9	21.3	142.6	72.4	167.8	65.5
mean, $R{+}W/2$ (MB)	113.2	43.4	48.2	27.9	41.9	31.8	142.6	108.1	167.8	97.0
peak, $R{+}W$ (MB)	113.2	57.5	48.2	36.9	41.9	42.3	142.6	143.7	167.8	128.5
Attn. cache / token (B)	25,344	25,344	6,144	6,144	8,064	8,064	11,627	11,627	8,192	8,192
Resident, 4K $+$ 4K (MB)	321	251	99	78	108	98	238	203	235	164

Table 2: SEED’s recurrent state is 2.0–3.9

\times

smaller right after a compression, and its prefill runs fewer layers; attention caches stay uncompressed. Bytes of one request (fp32 states, bf16 attention caches; MB

=10^{6}

bytes), for the representation, not an engine’s allocation. SEED’s state, the factors plus the explicit sink, is at rank

R

after a compression,

R{+}W/2

on average (the midpoint of a compression window; resident and traffic rows) and

R{+}W

at the peak, just before a compression. 4K

+

4K: a 4,096-token prompt after 4,096 generated tokens; 32K

+

8K: a 32,768-token prompt after 8,192. Traffic counts every state as read and written in full at every step, SEED’s factors included, although the update of § 5.1 rescales

U

, updates the sink’s coefficient and leaves

V

unchanged apart from the appended column. Qwen3.8-Flash-Next and GLM-5.3-Flash read 2048 selected tokens per sparse layer and step, plus the indexer keys of all tokens, pooled over blocks of four (GLM-5.3-Flash: in the model’s own fp8); the NVFP4 Qwen3.8-Flash-Next checkpoint has the same caches and states. Short-convolution windows (0.8–5.0 MB per request, equal in both columns) are not included. FLOPs: 2 per parameter that a token meets; compression per generated token in fp64 and in fp32 GFLOP (Appendix F ).

Figure 2: Writes are easier than reads in the hybrids we serve. (a, b) Per-token KL from the original model’s next-token distribution when one layer is skipped only for the prompt tokens, its memory still written (green), or only for the generated tokens (black); the deepest layers matter most for the generated tokens because they produce the next-token distribution. Dashed: the anchor depth. (c) KL at the anchor depth without training: the deep layers’ write paths fed

h^{(k)}

(green) against an untrained early exit, the output head applied to

h^{(k)}

(black), with the ratio of the two KLs. (d) The risk is retrieval: identifiers copied by the untrained shallow prefill, as a share of the original model’s; hollow: below three quarters. Protocols: Appendix A .

Figure 3: A rank-one sink along the head’s mean value holds most of a recurrent state’s energy, and the content left over lies in few directions (Qwen3.8-Flash-Next; the first 3,584 tokens of one of the six held-out web prompts of Appendix C ). (a) The $128\times 128$ state of head 7 of layer 24, a middle head of the layer by the sink’s share of the state’s energy. (b) Its sink $a_{T}\bar{v}^{\top}$ , the state’s projection on the head’s mean value $\bar{v}$ (estimated on other text). (c) The content $S_{T}-a_{T}\bar{v}^{\top}$ . Titles: shares of $\|S_{T}\|_{F}^{2}$ , which add to one. (a)–(c) share one colour scale, $\log(1+|x|/10^{-3})$ in the magnitude, nothing clipped. (d) $a_{T}$ and $\bar{v}$ , each divided by its largest magnitude. (e) Singular spectra of the state (solid) and the content (dashed), divided by the head’s top singular value; per-index medians over the layer’s 48 heads. Dotted: the rank SEED keeps. (f) Frobenius norms of the sink (solid) and the content (dashed) against the prompt length; medians over heads. (g) The sink’s share of the state’s energy and (h) $|\cos|$ between $v_{\star}$ and $\bar{v}$ , per head; bars: layer medians.

Figure 4: One recipe trains every component set. Only SEED’s components are trained, with the frozen model as the only teacher, in a forward pass that runs the prefill as served. Stage 1 distils on fixed text; stage 2 is on-policy, on the replies of the teacher and of the current set to the same prompts. Per-model terms: Appendix D .

Figure 5: Every set converges the same way: most of the KL reduction comes early in stage 1. Teacher-forced KL to the frozen model on held-out web and answer windows against training tokens (dashed: start of stage 2; dotted: start of a further round).

Qwen3.8-Flash-Next (bf16)
									RULER
	GPQA Diamond	AIME 2024+25	MATH-500	MMLU	IFEval	LiveCode Bench v6	HumanEval+	LongBench v2	32K	64K	128K
original	91.9	99.4	99.3	92.5	91.7	88.5	93.9	64.3	93.2	94.2	95.5
SEED	91.5	98.8	98.8	92.6	91.1	88.9	94.8	65.3	96.4	95.5	95.0
difference	$-0.3\!\pm\!0.4$	$-0.6\!\pm\!0.5$	$-0.5\!\pm\!0.1$	$+0.1\!\pm\!0.1$	$-0.6\!\pm\!0.3$	$+0.4\!\pm\!0.4$	$+0.9\!\pm\!0.4$	$+0.9\!\pm\!0.9$	$+3.2\!\pm\!0.6$	$+1.2\!\pm\!0.5$	$-0.5\!\pm\!0.4$
Qwen3.8-Flash-Next-NVFP4

Table 3: SEED matches the original model’s scores across the eleven exams: 63 of the 66 cells lie within two standard errors of the original model or above it. Each component set (on Nemotron-3.5-Super, its stage-1 set; § 5.2 ) is paired with its original model on the same questions and per-request seeds. Rows: original score, the set’s score and their paired difference (mean

\pm

standard error over questions). Eight repetitions per exam except LongBench v2 (four), MMLU and RULER (one each).

Figure 6: SEED matches the original model’s scores on every model, and its shallow state moves the next-token distribution less than an official NVFP4 checkpoint does. (a) The 66 paired differences of Table 3 (

\pm

one standard error; shaded:

\pm 1

point). (b) The perplexity of the original model’s own replies changes by less than 0.6% under the shallow state of the exams, less than under NVIDIA’s official NVFP4 checkpoints (Table 4 ; GLM-5.3-Flash above its full-rank factored path).

original model, dense state
	Qwen3.8- Flash-Next	Nemotron-3.5- Lightning	Kimi- Linear	GLM-5.3- Flash	Nemotron-3.5- Super
fp32 (exams)	0 100%	0 100%	0 100%	0 100%	0 100%
fp16	0.0006 50%	0.0004 s 50%	0.0002 50%	0.0006 50%	0.0002 s 50%
fp8	0.0073 s 25%	0.0487 25%	0.0048 s 25%	0.0055 s 25%	0.0745 25%
SEED, shallow state
fp32 (exams)	0.0056 26%	0.0060 39%	0.0017 51%	0.0027 51%	0.0070 39%

Table 4: The shallow state stays closer to the original model than the official NVFP4 checkpoints do, and it composes with 16-bit storage. Top: decode drift (§ 6.1 ) in nats per token; grey: stored state size as a share of the original fp32 state (shallow state: right after a compression). s : stochastic rounding (Table 9 gives both roundings). Bottom: change in the perplexity of the same replies. –: no NVFP4 release (Kimi-Linear) or an FP8 release, the checkpoint we use (GLM-5.3-Flash).

Figure 7: The shallow state’s drift grows far more slowly than its number of compressions. Decode drift of the released state form (fp32) per 1,024 decode steps of long replies of the original model (24 replies on both Qwen3.8-Flash-Next sets and Kimi-Linear, 16 on Nemotron-3.5-Lightning, 12 on Nemotron-3.5-Super;

R=W=16

, 32 on Kimi-Linear); dashed: the official NVFP4 checkpoint of the same model over its first 1,024 steps (Table 4 ).

Figure 8: SEED reaches 1.16–1.42

\times

the original model’s peak throughput per GPU, at 8 to 16 more sessions. Served Qwen3.8-Flash-Next with fp32 recurrent states. Rows: aggregated NVFP4 TP1 (a–d), aggregated bf16 TP2 (e–h), disaggregated bf16 1P:1D (i–l). Columns: throughput per GPU against the P90 output rate per user; throughput and prefix-cache hit against concurrency; throughput against the P90 time to first token (Pareto frontiers). Charcoal: the original model; dashed: shallow states alone and shallow prefill alone; green: SEED, the best of its three configurations at each concurrency (§ 7 ); shaded: its gain.

aggregated, NVFP4 TP1
	peak			at $C^{\star}$	at 40 tok/s
Configuration	sessions	tok/s/GPU	ratio	ratio	ratio
original	28	80,219	–	–	–
shallow states	32	92,343	$1.15\times$	$1.38\times$	$1.16\times$
shallow prefill	28	86,385	$1.08\times$	$1.17\times$	$1.06\times$
SEED (both)	36	95,989	$1.20\times$	$1.50\times$	$1.23\times$

Table 5: SEED raises the peak throughput per GPU of served Qwen3.8-Flash-Next in every deployment (Figure 8 ). Peak: each configuration’s highest throughput over its measured concurrencies (tokens/s/GPU), its concurrency and its ratio to the original model’s peak. At

C^{\star}

: the throughput ratio at SEED’s peak concurrency (36, 40 and 64 sessions), where the original model is past its own peak (P90 per-user rate 6.8, 6.7 and 1.2 tokens/s). At 40 tok/s: the throughput ratio at an equal P90 per-user rate of 40 tokens/s.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: The write–read asymmetry of Figure 2 a, b holds on Nemotron-3.5-Lightning, Nemotron-3.5-Super and GLM-5.3-Flash. KL from the original model when one layer’s output is skipped only for the prompt tokens, its memory still written (green), or only for the generated tokens (black); sampled layers and windows as in Appendix A . Nemotron-3.5-Super’s values include its engine’s run-to-run floor (grey). Dashed: the anchor depth.

Figure 10: Without training, a shallow prefill keeps a model’s direct answers and loses mainly retrieval. (a) Qwen3-8B-Base: held-out NLL change when one layer is skipped only for the prompt tokens (memory still written) or only for the generated tokens; the last layer’s prompt-side change is exactly 0 (marked below the axis). (b, c) Untrained shallow prefill at each anchor depth (hollow, dashed); filled: Qwen3-8B-Base at 22 of 36 layers with trained write paths. (b) Core average, % of the original model. (c) Needle recall minus the original model’s; dotted:

-3

points. A checkpoint’s markers share a horizontal shift of at most 1.4 points of depth.

		KL above the floor			cache error (%)
Set	floor	as served	states exact	code bypassed	first emitted layer	beyond
Qwen3.8-Flash-Next bf16	0.0033	0.0359	0.0216	0.0411	1.4–2.2	9.0–66.9
Qwen3.8-Flash-Next-NVFP4	0.0206	0.0393	0.0224	0.0445	1.7–2.6	10.0–73.5
Nemotron-3.5-Lightning	0.0018	0.0145	0.0100	0.0269	0.5–4.0	11.6–28.6
Kimi-Linear	0.0016	0.0063	0.0038	0.0119	1.2	11.5–67.7
GLM-5.3-Flash	0.0000	0.0330	0.0318	–	0.0–0.8	2.2–79.5

Appendix

Table 6: Every released set stays close to the original model although its memories are inexact. Teacher-forced KL to the original model (nats per token) above the floor, the KL of a second run of the original model (nonzero where the engine is not deterministic); windows of Appendix A (GLM-5.3-Flash: 16 windows, 63 tokens). As served: the emitters read the decoded code and every state is compressed as in the decode. States exact: no compression. Code bypassed: the emitters read the exact

h^{(k)}

(GLM-5.3-Flash has no code). Cache error: relative squared error of the emitted attention caches at the first emitted layer, whose original write path itself reads

h^{(k)}

, and beyond it (Nemotron-3.5-Super: no attention layer beyond; its code keeps 512 of 4,096 coordinates).

depth $k$
	Qwen3.8- Flash-Next	Nemotron-3.5- Lightning	Kimi- Linear	GLM-5.3- Flash	Nemotron-3.5- Super
gap at boundary	23: 0.102 27: 0.100 31: 0.074	19: 0.182 26: 0.127 33: 0.075	11: 0.174 15: 0.078 19: 0.031	19: 0.172 23: 0.115 27: 0.045 31: 0.035	58: 0.216 69: 0.091 78: 0.065
copies	31: 290 / 297	33: 1,498 / 1,500	19: 1,500 / 1,497	27: 269 / 595 31: 530 / 595	78: 1,494 / 1,496
code $r{+}m$
closed form	4096: 227 / 2 ( $p<0.001$ ) 6144: 102 / 7 ( $p<0.001$ ) 8192: 15 / 7 ( $p=0.134$ ) 10240: 4 / 4 ( $p=1.000$ )	1024: 17 / 1 ( $p<0.001$ ) 1536: 12 / 1 ( $p=0.003$ ) 2048: 5 / 1 ( $p=0.219$ ) 2688: 2 / 0 ( $p=0.500$ )	1024: 0 / 0 1536: 0 / 0 2048: 0 / 0 2304: 0 / 0	2048–4608: at least 528 / 0 ( $p<0.001$ ) 8192: 144 / 11 ( $p<0.001$ )	256: 5 / 0 ( $p=0.0625$ ) 512: 3 / 0 ( $p=0.250$ )
bytes	$8192{+}256$ : 5,380 (0.509 / 1.019)	$2048{+}128$ : 1,540 (0.752 / 1.504)	$2304{+}128$ : 1,684 (0.487 / 0.975)	none; $8192{+}256$ would take 5,380 (1.272 / 2.468)	$512{+}128$ : 676 (0.660 / 1.320)

Appendix

Table 7: Inputs and choices of the selection rule (Appendix A.1 ). The Qwen3.8-Flash-Next column serves its bf16 and NVFP4 sets (probes on bf16). Gap: the untrained gap at the attention-layer boundaries, nats per token. Copies: identifiers copied by the untrained shallow prefill / the original model. Closed form: items lost / gained by a closed-form code of rank

r

against the uncoded shallow prefill (

p

: exact McNemar). Bytes: nominal per token; in parentheses, over the bf16 / fp8 deep cache. Template / other: relative reconstruction error at

m=128

and the selected rank. Full offset: the code of rank

d

, over the fp8 deep cache. Drift: decode drift at rank

R

W=R

, against

\mathrm{gap}(k)/4

Figure 11: The anchor depth is the shallowest boundary that passes both depth clauses. (a)–(e) The untrained gap at the attention-layer boundaries (

\pm

standard error, grey lines) and other depths (open markers; GLM-5.3-Flash and Nemotron-3.5-Super: boundaries only); black: the D1 threshold; dashed: the chosen

k

. (f) D2: identifiers copied by the untrained shallow prefill over those copied by the original model; threshold

3/4

. On GLM-5.3-Flash, 27 passes D1 and fails D2; on Nemotron-3.5-Super, every boundary below 78 fails D1.

Set	Code	32K		64K		128K
NVFP4, released	$8192{+}256$	1 /	0	1 /	1	4 /	5
bf16, unreleased, stage 1	$4096{+}512$	21 /	4	18 /	4	19 /	11
bf16, unreleased, stage 2	$4096{+}512$	22 /	3	12 /	5	16 /	15
bf16, released	$8192{+}256$	1 /	0	1 /	2	1 /	5

Appendix

Table 8: RULER’s two UUID tasks on Qwen3.8-Flash-Next: wrong replies out of 100 documents per task and length, niah_single_3 / niah_multikey_3 (the original model: none wrong). The rows with the code

4096{+}512

are an earlier, unreleased bf16 set after each stage.

Figure 12: The state clause takes the smallest rank whose state drift fits the budget, else 32. (a)–(e) State drift (

\pm

standard error) at

R=8,16,32

with

W=R

; open squares: its mean over decode steps 1,025–4,096 of 24 replies, drawn at

1.2R

. Black: the budget

\mathrm{gap}(k)/4

. Dotted: the floor, a second run of the original model (on GLM-5.3-Flash, the exact state scanned in 16-token pieces). Star: the chosen

R

. (f) Share of the content energy (

C=S-a\bar{v}^{\top}

) beyond rank

R

in prompt-final states of held-out 4K windows, energy-weighted over heads and layers.

Figure 13: One direction carries most of the recurrent state, and it is nearly the same on every prompt tested (Qwen3.8-Flash-Next,

36\times 48

heads, six web prompts of 3,584 tokens). (a) Energy share of the state’s

i

-th singular direction; grey: one layer (median over heads and prompts); green: median over layers. (b) The top direction’s share over the 1,728 heads. (c)

|\cos|

between a head’s top output directions

v_{\star}

on two unrelated prompts (green), on a web prompt and a prompt of uniformly random tokens (charcoal outline), and for independent random directions in

\mathbb{R}^{128}

(grey). (d) Rank holding 90% and 99% of the energy of the content

C_{T}

(Lemma 1 ) per head, averaged over the prompts; box 25–75%, whiskers 5–95%, bar the median; dashed: Qwen3.8-Flash-Next’s

R=16

		Qwen3.8- Flash-Next		Nemotron-3.5- Lightning		Kimi- Linear		GLM-5.3- Flash		Nemotron-3.5- Super
format	rounding	dense	shallow	dense	shallow	dense	shallow	dense	shallow	dense	shallow
fp32	–	0	0.0056	0	0.0060	0	0.0017	0	0.0027	0	0.0070
fp16	nearest	0.0006	0.0056	0.0011	0.0062	0.0002	0.0017	0.0006	0.0021	0.0012	0.0069
fp16	stochastic	0.0006	0.0056	0.0004	0.0059	0.0002	0.0016	0.0010	0.0025	0.0002	0.0070
bf16	nearest	0.0008	0.0056	0.0057	0.0061	0.0008	0.0017	0.0008	0.0030	0.0066	0.0072
bf16	stochastic	0.0007	0.0056	0.0044	0.0065	0.0004	0.0017	0.0010	0.0024	0.0048	0.0073

Appendix

Table 9: The shallow state’s drift is the same with fp32 and fp16 factors under either rounding; a dense state’s depends on the format and, on the Mamba-2 models, on the rounding. Decode drift (nats per token above a second run of the original model) of the original model’s dense state, rounded after every token, and of the shallow state, rounded at every compression. The exams use the fp32 shallow state. Last row: size per head and layer in 16 bits, the shallow state’s right after a compression.

Set	Component parameters (M)	Peak LR		16K-window steps (%)	stage 1	stage 2	Auxiliary alignment
					answer-start (%)
Qwen3.8-Flash-Next bf16	619.0	$5\times 10^{-5}$ /	$10^{-5}$	30	50/30	70/60	–
Qwen3.8-Flash-Next-NVFP4	619.0	$5\times 10^{-5}$ /	$10^{-5}$	30	50/30	70/60	–
Nemotron-3.5-Lightning	235.8	$5\times 10^{-5}$ /	$10^{-5}$	40	50/40	70/60	states
Kimi-Linear	113.8	$5\times 10^{-5}$ /	$10^{-5}$	30	50/30	70/60	–
GLM-5.3-Flash	708.7	$5\times 10^{-5}$ /	$10^{-5}$	30	50/30	70/60	states, index keys

Appendix

Table 10: Hyperparameters of the released sets (Nemotron-3.5-Super: stage 1 only). Component parameters: every exported tensor, including the fixed

\mu

and sink buffers and rows without gradient (Mamba-2 gate rows: 88M on Nemotron-3.5-Lightning, 134.2M on Nemotron-3.5-Super; indexer query rows: 6.6M on each Qwen3.8-Flash-Next set). Peak LR: stage 1 / stage 2; second stage-2 rounds about

5\times 10^{-6}

. Answer-start windows: the share of the 4K / 16K windows that start at the beginning of an answer.

Stage	Stream	Sources
1	public text	FineWeb-Edu Penedo et al. (2024) , FineMath Ben Allal et al. (2025) ; synthetic retrieval facts in web documents; long web documents
1	reasoning/chat †	OpenMathReasoning Moshkov et al. (2025) , OpenScienceReasoning-2 NVIDIA (2025) , OpenCodeReasoning Ahmad et al. (2025) , OpenR1-Math Hugging Face (2025) , OpenThoughts3 Guha et al. (2026) and tulu-3-sft-mixture Lambert et al. (2025)
1	teacher replies †	Frozen-model responses to public code, mathematics, science and chat prompts; long Nemotron-3.5-Lightning and Nemotron-3.5-Super responses
2	paired generation †	Public prompts (code and science reused from stage 1) answered by the teacher and the current components; forward KL on both, reverse KL on the components’ replies
2	retained streams	Stage-1 web, retrieval and public-corpus text † and stage-1 teacher replies

Appendix

Table 11: Data recipe of both stages; windows are 4K/16K tokens. † : 13-gram filtered when built against GPQA, MMLU, MATH-500, AIME and IFEval (stage 2 also LiveCodeBench and IFBench). Table 12 gives each set’s shares.

stage 1, share of $6\times 10^{8}$ tokens (%)
	Qwen3.8- Flash-Next bf16	Qwen3.8- Flash-Next NVFP4	Nemotron-3.5- Lightning	Kimi- Linear	GLM-5.3- Flash	Nemotron-3.5- Super
web text and retrieval documents	14.1	14.1	9.9	15.6	15.6	9.9
long web documents	5.5	5.5	4.2	7.8	7.8	4.2
reasoning and chat corpora †	21.9	21.9	17.6	39.3	19.0	17.6
frozen model’s replies †	58.5	58.5	43.3	37.2	57.6	43.3
frozen model’s long replies †	–	–	25.0	–	–	25.0

Appendix

Table 12: The training data of every set. Stage 1: share of the tokens per data group; stage 2: rounds, prompts per round and the domain weights of the paired generations in the first round. † : 13-gram filtered when built against GPQA, MMLU, MATH-500, AIME and IFEval (stage 2 also LiveCodeBench and IFBench).

					held-out KL
Run	steps	tokens	nodes $\times$ min		step 0	final
Qwen3.8-Flash-Next bf16
stage 1	881	600M	16 $\times$	159.7	0.0315	0.0176
stage 2, round 1	221	151M	16 $\times$	58.2	0.0174	0.0172
stage 2, round 2	221	151M	16 $\times$	52.5	0.0173	0.0171
Qwen3.8-Flash-Next-NVFP4

Appendix

Table 13: Run metrics of the training stages from their histories (Nemotron-3.5-Super releases its stage-1 set). Nodes have four GPUs; stage 1 starts from the closed-form initialization and each stage-2 round from its predecessor. Held-out: 32 windows of 4K tokens, scored after token 2048 at the run’s training rank.

Qwen3.8-Flash-Next bf16
		error		floor
Set	state term	Frobenius, pooled	read-out, 64 tokens	pooled / read-out
untrained shallow prefill	–	0.615	0.076	0.021 / 0.001
released set	no	0.492	0.047
Qwen3.8-Flash-Next-NVFP4
untrained shallow prefill	–	0.616	0.075	0.021 / 0.001

Appendix

Table 14: Training moves the emitted prompt-final states towards the frozen model’s: the released sets’ pooled error is 0.20–0.52, against 0.34–1.29 for the untrained shallow prefill. Frobenius:

\|S_{\mathrm{emit}}-S\|_{F}^{2}/\|S\|_{F}^{2}

, pooled over a layer’s heads (above 1: farther from the frozen model’s state than a zero state). State term: trained with the emitted-state alignment term. Read-out: the error as the layer reads it over the next 64 teacher-forced tokens, relative to the state’s own read-out. Floor: the frozen model’s own state in the stored form.

Task	Items	Repetitions	Grading and configuration
GPQA-Diamond	198	8	simple-evals OpenAI (2024) prompt, extraction and option permutation
AIME 2024+25	60	8	prompt asks for a boxed answer; integer gold answers
MATH-500	500	8	prompt asks for a boxed answer; last boxed answer, Math-Verify Kydlíček (2025)
MMLU	14,042	1	test split, simple-evals prompt and extraction
IFEval	541	8	prompt-level strict
LiveCodeBench v6	454	8	problems of 2024-08 to 2025-04; official prompt, extraction and public/private tests

Appendix

Table 15: Task configurations. Sampled exams are paired by question and request seed; RULER is greedy at each length. The official grader gives the result.

Exam (questions)	score	paired	$-$ stage 1	score	paired	$-$ stage 1	score	paired	$-$ stage 1
	Qwen3.8-Flash-Next-NVFP4			Nemotron-3.5-Lightning			Kimi-Linear
	stage-1 set		released	stage-1 set		released	stage-1 set		released
GPQA-Diamond (198)	91.5	$-0.8\pm 0.6$	$-0.3\pm 0.6$	75.1	$-1.2\pm 1.1$	$-0.1\pm 0.9$	67.4	$-0.2\pm 1.2$	$+0.6\pm 1.3$
AIME 2024+25 (60)	99.6	$0.0\pm 0.4$	$0.0\pm 0.4$	87.1	$-2.5\pm 1.4$	$+1.7\pm 1.0$	66.0	$-1.5\pm 2.4$	$+0.8\pm 2.6$
MATH-500 (500)	99.1	$-0.1\pm 0.1$	$-0.2\pm 0.1$	98.4	$+0.2\pm 0.2$	$-0.1\pm 0.2$	92.5	$-0.6\pm 0.4$	$-0.1\pm 0.4$
MMLU (14,042)	92.4	$0.0\pm 0.2$	$+0.2\pm 0.2$	88.0	$-0.1\pm 0.2$	$-0.1\pm 0.2$	77.9	$+0.2\pm 0.4$	$+0.4\pm 0.4$

Appendix

Table 16: Stage 1 against stage 2: of the 27 released

-

stage-1 differences, only IFEval on Qwen3.8-Flash-Next-NVFP4 exceeds two standard errors (on unrounded values). Score, paired: the stage-1 set’s score and its paired difference to the original model, mean

\pm

standard error; released

-

stage 1: the paired difference between the two sets. Table 3 gives the original model’s and the released sets’ scores.

Quantity	Account
Dense state per head	$b_{s}d_{k}d_{v}$
Shallow state	$b_{s}[d_{k}+(d_{k}+d_{v})(R+w)]$ at $w$ steps after a compression, $0\leq w\leq W$ : the sink’s coefficient and the factors; the mean direction is shared by all requests and not counted
Resident request	$M_{\mathrm{r}}(T)$ (§ 2 ): the state bytes plus $T$ times the attention-cache bytes per token, plus the short-convolution windows (which Table 2 leaves out)
Decode traffic	cache reads, state reads/writes and the weights touched by the batch; sparse attention reads selected tokens and indexer keys
Expected routed experts	$n_{e}[1-(1-n_{a}/n_{e})^{B}]$ per layer for the $B$ tokens of a step, with $n_{e}$ routed experts of which each token activates $n_{a}$ : exact for independent tokens under uniform routing, an upper bound when independent tokens share a skewed routing distribution
Prompt computation	two FLOPs per parameter used in layers $0..k{-}1$ , code and emitters; no per-token output-head term

Appendix

Table 17: Analytical byte and FLOP account.

b_{s}

and

b_{c}

are the state and cache precisions in bytes; counts use checkpoint tensor shapes. For Mamba-2, use the orientation of Appendix B :

d_{k}=d_{B}=128

and

d_{v}=d_{x}=64

. Runtime is measured separately.

		Orig.	(i)	(ii)
Qwen3.8-Flash-Next	bf16 caches (B)	26,753	25,727	20,420 †
	fp8 caches (B)	14,081	13,055	13,028 †
	reuse (GFLOP)	0	0	1.07
Nemotron-3.5-Lightning	bf16 caches (B)	6,743	6,384	5,793 †
	fp8 caches (B)	3,671	3,312 †	3,745
	reuse (GFLOP)	0	0	0.46

Appendix

Table 18: With bf16 attention caches, form (ii) of a cached prefix takes up to 22% fewer bytes than form (i), and its reuse costs 0.22–1.07 GFLOP per prompt token; with fp8 caches the two differ by at most 3%, except on Nemotron-3.5-Lightning, where (i) is 12% smaller. A cached prefix of 80K tokens in each storage form: bytes per prompt token (the prefix’s states and convolution windows spread over its tokens) at bf16 and fp8 attention caches, and the GFLOP per prompt token that its reuse costs. (i) every layer’s cache, the deep ones as the emitters wrote them, and the shallow states; (ii) the caches and states of layers

0..k{-}1

and the code, from which the emitters rewrite the deep layers. † The smaller of (i) and (ii) in bytes. The NVFP4 set of Qwen3.8-Flash-Next has the bytes and FLOPs of the bf16 set; GLM-5.3-Flash has no code, so (i) is its only form, and its fp8 figures halve the latent only, its indexer keys being fp8 already.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,171 papers

Oct 6, 2026

Hybrid Latent Attention for Looped Language Models

Looped language models apply the same stack of layers T times to each token, which deepens the model without adding parameters but multiplies its key-value (KV) cache by T. The larger cache limits how many sequences a GPU can decode at once and slows each decoding step, which reads the whole cache. We propose Hybrid Latent Attention (HLA), which keeps exact keys and values within a sliding window of W recent tokens and stores each older token as a compact latent that the query of each loop reads directly, without reconstructing keys and values. We uptrain HLA on Ouro looped models (T=4) with 1.4B and 2.6B parameters, keeping the pretrained weights frozen and training only the added parameters to reproduce the original attention. The cache shrinks by 10.7x per token, fitting 4.0-8.8x as many concurrent sequences per GPU, and decoding throughput improves by 2.5x at 1K-token contexts and by up to 7.4x at 16K. HLA retains over 97% of the original accuracy on math, knowledge and reasoning benchmarks, and 96-100% on long-context retrieval up to 16K tokens. After supervised fine-tuning, it performs on par with the fine-tuned original model on competition-level math.

Yuhan Chen, Siyuan Zhang, Nan Wang +2

Virginia Tech · Independent Researcher

Long-Context Language Model Inference LLM Inference Acceleration KV-Cache Compression Hybrid Attention Recursive Language Models

Ask PDF

Sep 3, 2026

What Attention Recalls and Recurrence Controls in Hybrid Language Models

Hybrid language models combine attention with a fixed-size recurrent state, but the role of each channel remains unclear. We introduce two cache-level interventions. Split-prefill keeps only the KV cache or only the recurrent state from a prefilled context, then generates an answer. State-swap pairs the KV cache from one context with the recurrent state from another in a single forward pass. On Qwen3.5 and Falcon-H1, the two channels split sharply by function. Exact retrieval survives only through attention (64-98% of full accuracy) and collapses to zero through recurrence. Output language and persona reverse the pattern: both survive recurrence (70-80% and 3-5x) while KV-only drops to ~1% language accuracy. State-swap confirms this causally: the answer takes its value from the KV side and its language from the recurrent side. Recurrent-only generation also accepts words that were never in the context but share meaning or parts with seen items. Attention provides a lookup over what was said; the recurrent state shapes how the model says it next.

Kirill Afendulev, Alexey Dontsov, Elena Tutubalina +1

AIRI, Moscow, Russia · YSDA, Moscow, Russia · HSE University, Moscow, Russia

Hybrid Attention Recurrent Transformers In-Context Retrieval Causal Interventions in Language Models

Ask PDF

Jun 3, 2026

Depth-Attention: Cross-Layer Value Mixing for Language Models

Self-attention selects information freely across the sequence, but across depth, Transformers merely add each layer's output to the residual stream, so later layers cannot selectively reuse earlier-layer representations. Recent cross-layer methods improve this flow but operate on hidden states outside attention, adding state beyond the key-value cache at inference--a cost that becomes increasingly salient as modern LLMs compress the cache with grouped-query and multi-head latent attention. We introduce Depth-Attention, which performs this selection inside the attention module itself: before a layer attends over the sequence, its query attends over the keys of earlier layers at the same token position and mixes their values into the value that self-attention then reads. Because Depth-Attention reuses the standard attention queries, keys, and value-cache slots, storing depth-mixed values in place of the original values, it adds no parameters and introduces no persistent inference state beyond the standard key-value cache--the same cache size as a vanilla decoder and less than hidden-state-based cross-layer methods. On Qwen3-style decoders at 1.5B and 3B parameters, Depth-Attention attains the lowest perplexity and the highest average downstream accuracy, improving over the vanilla Transformer by up to 2.3 accuracy points and surpassing strong cross-layer baselines in perplexity and average accuracy, while adding under 0.01% extra arithmetic FLOPs and no additional persistent inference state. The gains hold from 360M to 3B parameters and extend to looped Transformers.

Boyi Zeng, Yiqin Hao, Zitong Wang +7

LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute +2

Efficient Transformer Inference Cross-Layer Attention Decoder-Only Language Models Transformer Attention

Ask PDF

Abstract

LLM Inference Acceleration

Explore similar work

Hybrid Latent Attention for Looped Language Models

What Attention Recalls and Recurrence Controls in Hybrid Language Models

Depth-Attention: Cross-Layer Value Mixing for Language Models