N-Gram Memories at Two Bits: Quantizing Lookup Tables in Large Language Models

Donghoon Han; Minseop Jung; Dong Woo Kwak; SeungJae Lee; Junhoo Lee

cs.LGOct 15, 2026

N-Gram Memories at Two Bits: Quantizing Lookup Tables in Large Language Models

Authors: Donghoon Han, Minseop Jung, Dong Woo Kwak, SeungJae Lee, Junhoo Lee

Organizations: Dnotitia Inc. · KAIST

Abstract

Conditional memory expands language-model capacity through selective lookup, but the full embedding table must remain available for every possible input. We introduce Conditional Program (CP), a compressed representation that reconstructs only the rows an input requests. CP represents each row by a short code and a local scale, with a reconstruction program shared across the table. We instantiate this representation with a Gaussian trellis, norm restoration and independently decodable row records. CP's two-bit code lowers output KL by 41--63% relative to 2-bit round-to-nearest. Analysis across three trained n-gram memory families shows that Gaussian-like coordinate statistics coexist with learned relationships among rows; at matched storage budgets, the shared trellis outperforms the fitted codebooks and low-rank residual variants we test. For DeepSeek, it reduces table storage from 203GB to 51GB, leaving room for larger workloads when the table is kept on GPU.

Figures & tables

Figure 1: Conditional Program (CP). An input selects row addresses in the compressed memory. Each record contains a trellis code and a bf16 scale; a shared decoder reconstructs the vector, and the scale restores its norm before the existing readout. (a) Offline, the encoder selects a low-error trellis path and stores its code with the norm-restoring scale. (b) Online, only addressed records are fetched, decoded and scaled. Trellis states and paths are schematic.

DeepSeek-V4.1-Flash
			Task score $\uparrow$
Method	Bits	Table GB $\downarrow$	PopQA	TriviaQA	NQ	MMLU	Belebele	GSM8K	HumanEval	Overall
Original	—	202.7	44.2	87.9	43.2	88.2	93.9	96.4	97.6	79.8
w/o memory	—	0.0	20.9	75.0	30.4	82.2	89.1	95.8	90.2	73.5
RTN	3	86.0	43.9	87.9	43.5	88.3	93.9	96.8	98.2	79.9
RTN	2	61.4	43.1	87.8	43.1	88.7	94.1	97.0	96.3	79.7

Table 1: Lookup-table storage and downstream performance. Bits denotes code bits per value; table GB includes scales and offsets. Scores are on a 0–100 scale; Belebele averages eight languages. Overall equally weights the five task-group means. Evaluation details and all 40 tasks appear in Appendices A and B .

Panel of Figure 2 — Figure 2: Table size versus six-domain geometric-mean output KL (log scale); lower and further left is better. CP labels include per-row scales. Dashed lines connect RTN settings. Purple markers: MXFP4 (left), NVFP4 (right), including block scales. Full values and paired comparisons are in Table 13 .

2-gram table	Energy (%)	Gaussian ref.
DeepSeek, layer 1	19.1%	6.25%
DeepSeek, layer 14	21.1%	6.25%
Qwen	33.3%	10.0%
LongCat	10.3%	6.25%

Table 2: Shared directions in frequently read rows. Energy captured by 16 fitted directions on the 4,096 most-read held-out rows. The Gaussian reference is

16/d

Representation	bpp	KL $\downarrow$
Shared trellis	2.10	0.0198
Fitted VQ, $256\times 4$	2.10	0.0238
Fitted VQ, $65{,}536\times 8$	2.10	0.0226
Mean + trellis	2.10	0.0193
Mean + four PCs + trellis	2.50	0.0188
Trellis reference †	2.50	0.0144

Table 3: Encoding at matched storage rates. Output KL on Qwen, evaluated on FineWeb-Edu. VQ labels denote codebook size

\times

block dimension.

		Output KL $\downarrow$
Rotate	Scale	DeepSeek	Qwen	LongCat
Yes	Global	0.0414	0.0347	0.0468
No	Global	0.0416	0.0346	0.0473
No	Row LS	0.0404	0.0349	0.0462
No	Row norm	0.0401	0.0327	0.0426

Table 4: Rotation and row scaling. Geometric-mean output KL over six domains with two-bit codes.

(a) Four GPUs: memory and latency $\downarrow$
Metric / workload	FP8	NVFP4	CP 2-bit
Table memory (GB/GPU)	50.7	27.6	12.7
Prefill, $8\times 4$ k (s)	64.66 $\pm$ 2.49	13.26 $\pm$ 0.28	13.27 $\pm$ 0.22
Decode, batch 8 (ms/step)	795.7 $\pm$ 11.1	788.9 $\pm$ 10.0	792.9 $\pm$ 6.3
Decode, batch 512 (s/step)	OOM	4.03 $\pm$ 0.03	4.05 $\pm$ 0.06
Prefill, 64k tokens (s)	OOM	59.81 $\pm$ 4.75	18.23 $\pm$ 0.01

Table 5: Memory and workload capacity on DeepSeek. Backbone and lookup tables reside on H200 GPUs. Four-GPU timings report mean

\pm

standard deviation over three runs.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Table	Rows	Row	Rows read	Original
	params		dim	per token	format
DeepSeek a	196.6B	768.0M	256	48	FP8 b
Qwen	51.2B	320.0M	160	16	BF16
LongCat	31.4B	122.7M	256	12	BF16

Appendix

Table 6: Lookup tables studied: the DeepSeek-V4.1-Flash Engram tables and the Qwen3.8-Flash-Next and LongCat-Flash-Lite n-gram tables. Each token reads one row per hashed head. Original size 202.7, 102.4 and 62.8 GB. a Two Engram layers, each 384.0M rows, 24 rows read per token. b e4m3 with a UE8M0 scale per 32 values.

Benchmark	Evaluation set	Items	Scoring
ARC-E / ARC-C	Test	1000 / 1172	Length-normalized likelihood
HellaSwag / PIQA	Validation	1000 each	Length-normalized likelihood
WinoGrande / BoolQ	Validation	1267 / 1000	Likelihood accuracy
LAMBADA	English test	1000	Last-word token match
MMLU	Stratified test	1000	Five-shot choice accuracy
PopQA	Test, five popularity bins	5000	Lowercase answer contains-match

Appendix

Table 7: Evaluation suite. Counts and protocols are fixed across quantization settings. English Belebele and MKQA are separate tasks; the main table’s Belebele score averages the eight non-English languages. QA uses three-shot prompts, with chat formatting for DeepSeek and Qwen and base prompts for LongCat. SQuAD uses zero-shot passage prompting. MGSM and GSM8K use native/base few-shot reasoning prompts or zero-shot chat instructions. All generation is greedy. Output limits are 16 tokens for short-answer QA, 24 for MKQA and 32 for SQuAD; GSM8K/MGSM use 512 tokens with chat prompts and 320 with base prompts, while HumanEval uses 768 and 384, respectively.

Commonsense
	DeepSeek-V4.1-Flash							Qwen3.8-Flash-Next							LongCat-Flash-Lite
		RTN			CP				RTN			CP				RTN			CP
Task	Orig.	Rem.	3	2	QTIP	2	1.5	Orig.	Rem.	3	2	QTIP	2	1.5	Orig.	Rem.	3	2	QTIP	2	1.5
ARC-E	.872	.872	.881	.878	.873	.884	.875	.811	.777	.802	.809	.811	.800	.813	.821	.714	.820	.819	.823	.824	.825
ARC-C	.695	.665	.686	.698	.692	.692	.691	.642	.589	.637	.642	.640	.640	.634	.616	.453	.602	.596	.605	.618	.615
HellaSwag	.883	.828	.882	.880	.886	.880	.888	.890	.889	.888	.892	.895	.894	.892	.810	.614	.822	.816	.815	.826	.809

Appendix

Table 8: All task results on the 0–1 scale. Scores use the task-specific metrics in Appendix A , including token F1 for SQuAD. Orig.: original table; Rem.: table removed. RTN codes use 3 or 2 bits; QTIP uses 2 bits; CP uses 2 or 1.5 bits. Belebele-

n

, MKQA-

n

and MGSM-

n

average

n

non-English languages. Task differences and their intervals are reported in Table 10 .

	DeepSeek					Qwen					LongCat
Setting	1	2	3	4	5	1	2	3	4	5	1	2	3	4	5
Original	.335	.333	.379	.482	.683	.261	.230	.269	.380	.674	.224	.157	.165	.243	.542
Table removed	.133	.096	.108	.190	.520	.188	.132	.149	.253	.606	.149	.080	.067	.065	.228
RTN 3-bit	.323	.330	.375	.481	.686	.259	.239	.272	.378	.673	.229	.155	.159	.243	.534
RTN 2-bit	.310	.312	.367	.470	.694	.245	.215	.254	.376	.663	.228	.153	.153	.239	.534
QTIP 2.0	.323	.332	.370	.478	.689	.254	.232	.262	.380	.676	.224	.159	.155	.246	.537

Appendix

Table 9: PopQA accuracy by entity popularity bin (1 = rarest). Settings match Table 8 .

Task	$n$	CP 2 $-$ RTN 2	CP 2 $-$ QTIP 2	CP 1.5 $-$ QTIP 2	CP 1.5 $-$ RTN 2
ARC-E	1000	+0.6 [-0.7, +1.9]	+1.1 [+0.0, +2.2]	+0.2 [-1.1, +1.5]	-0.3 [-1.6, +1.0]
ARC-C	1172	-0.6 [-2.2, +1.0]	+0.0 [-1.5, +1.5]	-0.1 [-1.7, +1.5]	-0.7 [-2.5, +1.2]
HellaSwag	1000	+0.0 [-0.9, +0.9]	-0.6 [-1.4, +0.2]	+0.2 [-0.6, +1.0]	+0.8 [-0.1, +1.7]
PIQA	1000	+0.9 [-0.1, +1.9]	+0.9 [-0.1, +2.0]	+0.4 [-0.7, +1.5]	+0.4 [-0.6, +1.4]
WinoGrande	1267	+0.5 [-1.1, +2.1]	+0.3 [-1.1, +1.7]	-0.6 [-2.1, +1.0]	-0.4 [-2.1, +1.3]
BoolQ	1000	+0.1 [-0.9, +1.1]	-0.3 [-1.2, +0.6]	+0.1 [-0.9, +1.1]	+0.5 [-0.5, +1.6]

Appendix

Table 10: Paired task-score differences for DeepSeek-V4.1-Flash in percentage points, with 95% intervals from 10,000 paired item-bootstrap replicates. Positive values favor the first method. Differences are computed before rounding, so they may differ from subtracting the displayed scores. Intervals are per comparison, without multiplicity adjustment; isolated differences across 40 tasks require caution.

Task	$n$	CP 2 $-$ RTN 2	CP 2 $-$ QTIP 2	CP 1.5 $-$ QTIP 2	CP 1.5 $-$ RTN 2
ARC-E	1000	-0.9 [-2.5, +0.6]	-1.1 [-2.5, +0.3]	+0.2 [-1.3, +1.7]	+0.4 [-1.1, +1.9]
ARC-C	1172	-0.2 [-1.9, +1.5]	+0.0 [-1.7, +1.7]	-0.6 [-2.3, +1.0]	-0.8 [-2.5, +0.9]
HellaSwag	1000	+0.2 [-0.6, +1.0]	-0.1 [-0.8, +0.6]	-0.3 [-1.3, +0.6]	+0.0 [-1.0, +1.0]
PIQA	1000	-0.6 [-2.0, +0.8]	+0.8 [-0.4, +2.0]	+0.6 [-0.7, +2.0]	-0.8 [-2.2, +0.6]
WinoGrande	1267	-2.1 [-4.3, +0.2]	-0.9 [-2.8, +0.9]	+1.3 [-0.6, +3.2]	+0.2 [-1.8, +2.3]
BoolQ	1000	+0.2 [-1.4, +1.9]	+0.5 [-0.9, +2.0]	-0.3 [-1.8, +1.2]	-0.6 [-2.2, +1.0]

Appendix

Table 11: Paired task-score differences for Qwen3.8-Flash-Next in percentage points, with 95% intervals from 10,000 paired item-bootstrap replicates. Positive values favor the first method. Differences are computed before rounding, so they may differ from subtracting the displayed scores. Intervals are per comparison, without multiplicity adjustment; isolated differences across 40 tasks require caution.

Task	$n$	CP 2 $-$ RTN 2	CP 2 $-$ QTIP 2	CP 1.5 $-$ QTIP 2	CP 1.5 $-$ RTN 2
ARC-E	1000	+0.5 [-1.0, +2.0]	+0.1 [-1.2, +1.4]	+0.2 [-1.2, +1.6]	+0.6 [-0.9, +2.1]
ARC-C	1172	+2.1 [+0.4, +3.8]	+1.3 [-0.2, +2.7]	+1.0 [-0.5, +2.6]	+1.9 [+0.2, +3.6]
HellaSwag	1000	+1.0 [-0.3, +2.4]	+1.1 [-0.1, +2.3]	-0.6 [-2.0, +0.7]	-0.7 [-2.1, +0.6]
PIQA	1000	+0.9 [-0.8, +2.6]	-1.2 [-2.7, +0.3]	-1.4 [-2.9, +0.1]	+0.7 [-0.8, +2.3]
WinoGrande	1267	+0.4 [-2.3, +3.1]	+0.4 [-2.0, +2.7]	+1.1 [-1.4, +3.6]	+1.1 [-1.5, +3.8]
BoolQ	1000	+1.4 [+0.0, +2.9]	+0.7 [-0.5, +1.9]	+0.7 [-0.6, +2.0]	+1.4 [-0.1, +2.9]

Appendix

Table 12: Paired task-score differences for LongCat-Flash-Lite in percentage points, with 95% intervals from 10,000 paired item-bootstrap replicates. Positive values favor the first method. Differences are computed before rounding, so they may differ from subtracting the displayed scores. Intervals are per comparison, without multiplicity adjustment; isolated differences across 40 tasks require caution.

	DeepSeek-V4.1-Flash					Qwen3.8-Flash-Next					LongCat-Flash-Lite
Setting	bpp	GB	KL	KL max	top-1	bpp	GB	KL	KL max	top-1	bpp	GB	KL	KL max	top-1
Original	8.25	202.7	0	0	1.000	16.00	102.4	0	0	1.000	16.00	62.8	0	0	1.000
Table removed	0	0	.773	1.08	.713	0	0	.308	.481	.815	0	0	2.89	3.57	.475
Mean row	0	0	.517	.743	.775	0	0	.310	.477	.819	0	0	3.44	4.07	.446
NVFP4	4.50	110.6	.0199	.0298	.954	4.50	28.8	.0166	.0289	.955	4.50	17.7	.0111	.0173	.968
MXFP4	4.25	104.4	.0220	.0347	.952	4.25	27.2	.0182	.0274	.953	4.25	16.7	.0144	.0216	.964

Appendix

Table 13: Full storage and output-fidelity results. KL is the geometric mean over six domains; KL max is the worst domain; top-1 is next-token agreement with the original. GB counts lookup-table storage. Rates include row/group metadata. QTIP uses within-row rotation and a global scale; CP restores each row norm. RTN uses groups of 64 (80 for Qwen). NVFP4 and MXFP4 include block scales. Bottom: paired KL ratios and 95% chunk-bootstrap intervals; below 1 favors the first setting.

Setting	DeepSeek	Qwen	LongCat
Original	2.611	4.671	13.316
Table removed	7.703	6.476	103.539
RTN 3-bit	2.835	4.757	13.681
RTN 2-bit	3.662	5.089	14.842
QTIP 2.0	3.089	4.827	13.325
CP, 2-bit code	3.054	4.830	13.538

Appendix

Table 14: WikiText-2 perplexity in its original units (lower is better). Settings match Table 1 ; no normalization by the table-removal baseline is applied.

Statistic	DeepSeek	Qwen	LongCat	Gaussian
$d$	256	160	256	256/160
Kurtosis	2.99	2.94–2.95	2.96–2.97	2.98/2.96
Kurt. (read-wtd.)	2.98–2.99	2.95	2.95–2.96	2.98/2.96
Max/RMS	3.00–3.01	2.84–2.85	2.99	3.00/2.86
Cov. eff. rank	239–250	152–158	253–254	256/160
KLT gain (dB)	0.10–0.25	0.18–0.53	0.04–0.21	0

Appendix

Table 15: Six statistics of the hashed tables. Ranges are over layers and n-gram orders (effective rank: over heads);

d

is the row dimension. Kurtosis and max/RMS (median over rows) are computed within each row (uniformly sampled rows; read-weighted: rows drawn in proportion to their read count). Effective rank is that of the row covariance of each head. The Gaussian column gives the value expected for i.i.d. Gaussian rows of the same dimension. KLT gain: coding gain of a covariance KLT at 2 bits, in the value-path readout metric. Cols.: columns with kurtosis above 5, out of 12,288, 2,560 and 3,072 columns. a Uniformly sampled rows; with read-weighted sampling, DeepSeek layer-14 order-2 heads have 2,048 such columns, an effect of mixing rows of different energy.

Representation	bpp	Qwen KL [95% CI]
Shared trellis	2.1	.0198 [.0163, .0234]
Fitted VQ, $256\times 4$	2.1	.0238 [.0193, .0285]
Fitted VQ, $65{,}536\times 8$	2.1	.0226 [.0184, .0269]
Mean + trellis	2.1	.0193 [.0160, .0229]
Mean + 4 PCs + trellis	2.5	.0188 [.0156, .0220]
Shared trellis curve	2.5	.0144

Appendix

Table 16: Source-code comparisons on held-out Qwen text. Rates include per-row gains and structural coefficients. The final row interpolates the trellis curve in log KL; its uncertainty is not reported. The fitted codebooks and structural variants use the protocol in Appendix D.2 .

Code	bpp	KL	KL max	top-1
QuIP# E8P a	2.00	.0413	.0605	.934
QuIP# E8P, norm restored b	2.10	.0373	.0518	.937
AQLM $1{\times}16$ c	2.10	.0385	.0545	.936
AQLM $1{\times}16$ , norm restored d	2.10	.0374	.0518	.937
HIGGS c	2.10	.0404	.0577	.935
HIGGS, norm restored d	2.10	.0391	.0544	.936

Appendix

Table 17: Codes considered for the Engram tables (Section 3 ) at about 2 bpp, on Qwen in the shared benchmark of Section 4 (six domains). KL is the geometric mean over domains of the per-token KL to the original model, KL max the worst domain, top-1 the next-token agreement. a Rotation and one scale per table. b As in CP: no rotation, one bf16 norm-restoring scale per row. c One scale per row (least squares for AQLM, RMS for HIGGS); HIGGS rotates, AQLM does not. d The same codes with each row’s norm restored. Ratios are paired, with document-chunk bootstrap 95% CIs.

	DeepSeek			Qwen			LongCat
Step	bpp	KL	ratio [95% CI]	bpp	KL	ratio [95% CI]	bpp	KL	ratio [95% CI]
QTIP	2.00	.0414	–	2.00	.0347	–	2.00	.0468	–
No rotation	2.00	.0416	1.005 [.995, 1.015]	2.00	.0346	.999 [.974, 1.027]	2.00	.0473	1.010 [1.000, 1.020]
Row LS gain	2.06	.0404	.972 [.964, .981]	2.10	.0349	1.008 [.984, 1.030]	2.06	.0462	.977 [.974, .981]
Norm restoration	2.06	.0401	.991 [.987, .995]	2.10	.0327	.937 [.904, .970]	2.06	.0426	.922 [.917, .928]
CP / QTIP	–	–	.968 [.959, .977]	–	–	.943 [.917, .969]	–	–	.910 [.900, .919]

Appendix

Table 18: Full ablation from QTIP to CP. The first three step ratios compare each setting with the preceding one: remove within-row rotation at global scale, introduce row-wise LS gain, then restore the norm on the same codes. KL is the geometric mean over six domains. Ratio intervals use paired chunk bootstraps. The lower block repeats gain comparisons at other rates. –: not applicable (the first row has no preceding setting; ratio-only rows have no bpp or KL).

	Web	Code	Wikipedia
LongCat	.904	.940	.904
	[.886, .923]	[.923, .958]	[.889, .918]
Qwen	.983	.944	.988
	[.975, .992]	[.928, .958]	[.970, 1.005]
DeepSeek	.997	.997	.988
	[.987, 1.006]	[.979, 1.014]	[.971, 1.004]

Appendix

Table 19: Norm-restoration KL divided by LS KL in the gain test, with paired 95% bootstrap intervals. Values below 1 favor norm restoration. Stored codes and storage costs are fixed within each model. Web is the held-out FineWeb-Edu test split; code and Wikipedia reuse the benchmark texts described above.

Tool	DeepSeek	Qwen	LongCat	Eval.
Rotation a	.999	.993	1.008	S
(Hadamard)	[.990, 1.008]	[.971, 1.015]	[.997, 1.019]
Second-order	–	1.46	1.11	E
rounding		Lloyd–Max	GPTQ
Frequency	–	1.044 b	–	S
allocation		[1.016, 1.076]

Appendix

Table 20: Supplementary tool comparisons on hashed tables (frequency allocation lowers KL only on web, the distribution its read counts come from: .915 there, 1.044 over all domains). Entries: KL with / without the tool (S rows: geometric mean over domains, 95% CI). S: shared benchmark; E: separate evaluation with its own baseline; –: not tested; details in Table 21 . A pure low-rank replacement was not evaluated end to end (covariance effective rank 239–254 of

d=256

and 152–158 of

d=160

); a mean + 4 PCs + trellis residual was (Section 4.3 ). a KL without / with rotation, both with norm restoration, as in Section 3 . b Trellis codes in three tiers by read count (top tier 35% of rows) vs the 2-bit code alone at the same mean rate; the 10% top tier and per-domain ratios are in Table 21 .

Tool	Motivation	Measurement	Evaluation setup
Rotation (random Hadamard within a row)	Gaussian-like row statistics motivate testing whether within-row rotation improves reconstruction.	KL ratio without/with rotation, both with norm restoration: DeepSeek .999 [.990, 1.008] † , Qwen .993 [.971, 1.015], LongCat 1.008 [.997, 1.019]. Table MSE ratio .998–1.001 on all 11 order tables.	KL: shared benchmark (six domains; token counts in Appendix A ), 2.06–2.10 bpp. MSE: CPU encode, 2,048–4,096 read-weighted rows per table.
Second-order rounding (GPTQ, LDLQ, metric-aware scalar)	A readout-only surrogate metric is $W^{\top}W$ : condition number 2.5 (LongCat), metric effective rank 243–254/256 and 155–158/160. This surrogate omits gates and subsequent layers.	KL with vs without the metric: LongCat GPTQ .128 vs .115 (vs RTN, 2.5 bpp); Qwen metric-whitened Lloyd–Max .1502 vs .1027 (2.1 bpp). Not tested on DeepSeek.	(E) Separate evaluation: about 32k web tokens, own baseline per row.
Low-rank decomposition	Covariance effective rank 239–254/256 and 152–158/160: high global rank; frequent-row structure is examined separately in Section 4.3 .	Global covariance alone does not determine compression value. A pure low-rank replacement was not evaluated end to end. A mean + 4 PCs + trellis residual was, at equal bytes (Section 4.3 ).	Qwen KL (Section 4.3 )
Frequency-based bit allocation	Frequently read rows have the same distribution (read-weighted kurtosis), so equal rate gives equal relative error (LongCat: .0737 unread vs .0722 for 10–99 reads).	(S) Qwen, KL with / without tiers: trellis codes at 2.6/2.1/1.6 bpp by read count vs the 2-bit code alone, same mean rate: top tier 35% of rows (89% of web reads) 1.044 [1.016, 1.076]; web .915, code 1.074, multilingual 1.092–1.123. With top tier 10% of rows (72% of web reads) 1.012 [.980, 1.046]; web .929, code 1.027, multilingual 1.016–1.048. (E) Scalar code, 2.1 bpp, vs the same code at a uniform rate: web .0853 vs .1030 ( $-$ 17%), but 1.33–1.44 $\times$ higher on code, Chinese and Japanese.	(S) Shared benchmark (six domains; token counts in Appendix A ), 2.10 bpp. (E) Separate evaluation (131,071 web tokens, plus code, Chinese, Japanese).
Start-of-sequence row protection	Protection tests whether repeatedly accessed start rows warrant additional precision.	Rows read at every sequence start kept at original precision (DeepSeek 24 per Engram layer, LongCat 27,811). KL ratio vs CP: DeepSeek .999 [.995, 1.002] (random-row control 1.000 [1.000, 1.000]); LongCat .991 [.988, .993] (control .999 [.997, 1.001]).	Shared benchmark (six domains; token counts in Appendix A ), on top of CP at 2.06 bpp.

Appendix

Table 21: Detail of Table 20 (Appendix E ). Supplementary tool comparisons on hashed lookup tables (frequency allocation lowers KL only on web, the distribution its read counts come from, and raises it on code and multilingual text). KL ratios are geometric means over the six domains with paired chunk-bootstrap 95% CIs; per-domain ratios are point values; rotation: KL without / with rotation; frequency allocation and start-row protection: KL with / without the tool. (S): shared benchmark of Section 4.1 ; (E): a separate evaluation with fewer tokens and its own baseline, given in the last column. † The expected outcome was fixed before this measurement.

Step	DeepSeek	Qwen	LongCat
Top-35% tiers / uniform	1.010	1.044	1.006
	[1.000, 1.020]	[1.016, 1.076]	[.995, 1.017]
Top-10% tiers / uniform	.991	1.012	–
	[.985, .997]	[.980, 1.046]
Start rows / uniform	–	1.008	–
		[.980, 1.039]

Appendix

Table 22: Extended allocation checks. Entries are geometric-mean KL ratios to uniform CP with paired 95% chunk-bootstrap intervals over six domains. Frequency tiers keep the same mean rate; the top-tier fraction refers to rows. Qwen start-row protection restores 37,750 selected rows to bf16 and adds .0019 bpp; the random-row control uses the same count.

Panel of Figure 3 — Figure 3: LongCat damage model $\mathrm{KL}=c_{0}+k\,S^{p}+k_{\parallel}\,dN^{2}$ with the measured small-perturbation offset $c_{0}=.0021$ and $k=.198$ , $p=.79$ , $k_{\parallel}=2.19$ fitted once on 29 codec configurations (LM, VQ and TCQ at 0.5–4 bits with the MSE-optimal scale or with norm restoration, plus one global gain, not drawn). Pure direction edits that keep every row norm (triangles) are out of sample. $S$ and $dN$ summarize direction and norm error using the codec-level approximations defined in the text; the direction controls set $S$ explicitly.

Tier	Who needs it	What 2 bpp changes (measured)	What it does not change
GPU HBM, 4 GPUs	RL rollouts and public inference code that keep the table on the GPU	FP8 needs 50.7 GB/GPU and runs out of memory at decode batch 512 and at 64k-token prefill; in the representative setting its prefill takes 64.66 $\pm$ 2.49 s (peak 147.0 GB). NVFP4 (27.6 GB/GPU) runs both, but its 64k-token prefill step takes 59.81 $\pm$ 4.75 s. CP needs 12.7 GB/GPU, runs every workload, and prefills in 13.27 $\pm$ 0.22 s (peak 109.5 GB) and 64k tokens in 18.23 $\pm$ 0.01 s.	Decode step time (795.7 vs. 792.9 ms, representative setting) and prefill steps up to 16k tokens (within 0.9%). Prefill slows when a large table leaves little HBM (left).
GPU HBM, 8 GPUs	Same, with the model spread over a full node	Both fit (25.3 vs. 6.3 GB/GPU); CP leaves 19.0 GB/GPU more for KV cache and raises the per-step ceiling from decode batch 1024 to $\geq$ 2048 and from 96k to $\geq$ 128k prefill tokens.	Step time (within 2.4% at every point both ran).
Host DRAM	Several model replicas per node; hosts with little RAM	Table size: 202.7 $\to$ 50.7 GB; Peak node RSS 225 $\to$ 75 GB. Lookup alone at 64k tokens: 53.5 $\to$ 19.4 ms. Smallest host that holds the table (calc.): 256 GB $\to$ 64 GB.	End-to-end time: the lookup is $\leq$ 0.8% of a forward step.
SSD	DRAM smaller than the table (long tail on SSD)	Nothing per read: each row is one 4 KB read in both formats, 48 reads per decoded token.	Lookup latency; with prefetch the exposed lookup at 16k-token prefill is 2.7 ms (FP8) and 1.5 ms (CP), vs. 199.2 ms without.
Hot-row cache (HBM) + SSD	Same, with spare HBM	At 12.9 GB the cache serves 50.6% $\to$ 70.7% of reads; a 51.5 GB cache holds the whole 50.7 GB CP table and serves 100% of reads (FP8 rows: 70.7%). Simulated SSD reads/token at 32 GB: 19.4 $\to$ 2.7.	Decode still misses every step (FP8 with 12.9 GB cache: 20.8 SSD reads/token at batch 64), so the off-GPU round trip stays on the critical path.

Appendix

Table 23: Placement summary (detail for Section 5 ): what compressing the DeepSeek tables to 2 bpp changes in each memory tier, and what it does not (8

\times

H200 node, reference inference code; representative setting: batch 8, 4k-token prompt, 256 decode steps). 4-GPU timings are mean

\pm

s.d. over 3 runs; other settings are one run each. Peak: maximum over GPUs of PyTorch’s allocated memory, in

10^{9}

bytes (an H200 has 141 GB nominal). “Calc.” marks a calculation (table only, 87.5% of installed RAM usable); everything else is measured.

Fused gather and decode: mean of run medians
		HBM (ms)		DRAM (ms)
Layout	KB/token	decode	prefill	prefill
Rows (CP)	3.2	0.0357	0.228	6.70
Tiles, whole tile	50.7 ( $16{\times}$ )	0.133	3.64 ( $16{\times}$ )	9.90 ( $1.48{\times}$ )
+ Hadamard, 256 rows	811 ( $256{\times}$ )	1.66	101 ( $443{\times}$ )	187 ( $28{\times}$ )
Rows (CP)	3.2	0.0252	0.19	6.54

Appendix

Table 24: DeepSeek lookup costs at two code bits (48 rows/token; one H200). HBM: GPU memory; DRAM: pinned host memory over PCIe. Decode: batch 64; prefill: 4,096 tokens. Top: separate gather/decode, single-session medians; parentheses give factors over row records. Bottom: mean of five independent run medians (100 calls after 20 warm-ups each); row windows use random payloads, CP uses model codes. Blocks use different nodes and protocols and should be compared only within each block. KB/token is computed code traffic, not measured bandwidth; this test does not compare output KL.

Tier	Capacity (node)	Bandwidth (measured)	Read unit	Latency	Lookup, 64k tok (ms) FP8 $\to$ CP	What compression changes
GPU HBM	1,128 GB (8 $\times$ H200)	4,579 GB/s read per GPU	$\leq$ 64 B a	–	9.70 $\to$ 3.23 b	Whether the table fits next to the model weights.
DRAM	2 TB	STREAM triad 191–315 GB/s c ; host $\to$ GPU 55.2 GB/s	64 B a	1.16 $\mu$ s (PCIe round trip)	53.5 $\to$ 19.4	Host RAM per replica (202.7 $\to$ 50.7 GB) and PCIe traffic (12.7 $\to$ 3.2 KB per token).
Local SSD	6 NVMe, RAID-0	4 KB random 2.56M IOPS; 64 KB reads 40 GB/s	4 KB	91 $\mu$ s (QD1)	1,243 $\to$ 1,048	Nothing on the read path: one 4 KB read per row in every format (16k-token prefill: reads 232 vs 228 ms). The gap is CPU-side row extraction (15.2 vs 1.0 ms). More rows fit in the same cache.

Appendix

Table 25: Memory tiers (detail for Section 5 ): one 8

\times

H200 node and the effect of compressing the DeepSeek Engram tables from FP8 (264 B/row) to CP at 2.06 bpp format (66 B/row). Lookup is the median time to fetch the 48 rows of each of 64k tokens on one GPU. Bandwidths: nvbandwidth (HBM, PCIe) and STREAM (DRAM); fio (SSD). a Access granularity of the memory system, not measured. b FP8 uses a quarter shard of the table, since the full FP8 table does not fit one GPU. c One NUMA node to both nodes; measured while other jobs used host memory, so a lower bound.

Panel of Figure 4 — Figure 4: DeepSeek tables behind a static hot-row cache (rows ranked by read count, fineweb held-out reads). (a) Share of reads served by a cache of a given size holding FP8 or 2-bpp rows (CP, 2.06 bpp); circles mark the 12.9 GB/node cache of the end-to-end runs. (b) Simulated SSD reads per token (one 4 KB read per missed row, 48 rows per token) for the FP8 table (202.7 GB) and the CP table (2.06 bpp, 50.7 GB); colors as in (a).

	No rotation			Rotation
Model	QTIP	+BOS	+top-100	QTIP	+top-100
E2B	.106	.031	.027	.033	.026
E2B-it	.227	.198	.140	.126	.123
E4B	.162	.030	.028	.029	.026
E4B-it	.067	.067	.061	.059	.056
3n-E4B	.108	.109	.105	.123	.116

Appendix

Table 26: Worst-domain KL of Gemma PLE tables at 2.0 bpp (six domains) when the BOS row, or the BOS row and the 100 most-read token rows (+top-100), are kept in bf16 in every layer (+0.006 bpp), without and with rotation. Models: Gemma-4 E2B, E4B and their instruction-tuned (-it) versions, gemma-3n-E4B. Bold: lowest of the first four columns.

Memory Compression

See all 24 papers

Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
Jan 12, 2026Xin Cheng, Rui Tian, Wangding Zeng +18Memory-Augmented Language Models Language Model Scaling Laws
Peking University · DeepSeek-AI
NGM: A Plug-and-Play Training-Free Memory Module for LLMs
May 16, 2026Yuwen Qu, Wenhui Dong, Chenyang Si +1Memory-Augmented Language Models Knowledge Augmentation for Language Models
Nanjing University
Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory
May 20, 2026Runxi Cheng, Yuchen Guan, Yongxian Wei +7Language Model Pretraining Memory-Augmented Language Models
Tsinghua University · Microsoft Reasearch Asia

Jan 12, 2026J/K move · Enter open · S save

Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

Xin Cheng, Rui Tian, Wangding Zeng, Damai Dai+17

Peking University · DeepSeek-AI

Memory-Augmented Language Models Language Model Scaling Laws Language Modeling

Open paper AskPDF arXiv

While Mixture-of-Experts (MoE) scales capacity via conditional computation, Transformers lack a native primitive for knowledge lookup, forcing them to inefficiently simulate retrieval through computation. To address this, we introduce conditional memory as a complementary sparsity axis, instantiated via Engram, a module that modernizes classic

N

-gram embedding for O(1) lookup. By formulating the Sparsity Allocation problem, we uncover a U-shaped scaling law that optimizes the trade-off between neural computation (MoE) and static memory (Engram). Guided by this law, we scale Engram to 27B parameters, achieving superior performance over a strictly iso-parameter and iso-FLOPs MoE baseline. Most notably, while the memory module is expected to aid knowledge retrieval (e.g., MMLU +3.4; CMMLU +4.0), we observe even larger gains in general reasoning (e.g., BBH +5.0; ARC-Challenge +3.7) and code/math domains~(HumanEval +3.0; MATH +2.4). Mechanistic analyses reveal that Engram relieves the backbone's early layers from static reconstruction, effectively deepening the network for complex reasoning. Furthermore, by delegating local dependencies to lookups, it frees up attention capacity for global context, substantially boosting long-context retrieval (e.g., Multi-Query NIAH: 84.2 to 97.0). Finally, Engram establishes infrastructure-aware efficiency: its deterministic addressing enables runtime prefetching from host memory, incurring negligible overhead. We envision conditional memory as an indispensable modeling primitive for next-generation sparse models.

Abstract

Memory Compression

Explore similar work

Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

NGM: A Plug-and-Play Training-Free Memory Module for LLMs

Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory