Evaluating Compressed Memory: Dependence, Utility, and Content Benefit

Minsu Oh; Seunghui Jwa; Chanjun Park; Yeo-Chan Yoon

cs.CLOct 14, 2026

Evaluating Compressed Memory: Dependence, Utility, and Content Benefit

Authors: Minsu Oh, Seunghui Jwa, Chanjun Park, Yeo-Chan Yoon

Organizations: Department of Artificial Intelligence, Jeju National University · School of Software, Soongsil University

Abstract

Does a language model answer better because its compressed memory contains useful document information? Replacing memory vectors with zeros can lower a trained model's score without showing that it outperforms a model trained without memory. We combine zeroing with separately trained models that cannot access stored text or receive the same constant input for every document. Across nine systems, some models rely on memory yet perform worse than the trained alternative. For example, in experiments that predict states in generated stories, zeroing memory worsens a model's prediction loss, yet a model trained without memory achieves lower loss than the original model with intact memory. On TriviaQA, giving xRAG memory from another question's passage raises F1, which measures word overlap with accepted answers, from 19.38 without a document to 24.78. Its effect on how often outputs contain an accepted answer is statistically uncertain. The question's retrieved passage scores higher still (29.88 F1). Thus a gain over no document alone does not establish that the gain requires the question's passage. Evaluations must therefore specify the comparison model, absolute performance, and generation and scoring conditions.

Figures & tables

Figure 1: Three comparisons for interpreting a memory gain. Zeroing memory tests whether an already trained model relies on it. Separately trained models without memory or with constant input test whether memory and document content improve performance.

System	Task adaptation / reference	Synthetic $U$	Synthetic $D$	PG-19 $U$	PG-19 $D$
StreamingLLM ( Xiao et al., 2024 )	KV / restricted decoder	$-0.610$	$+6.484$	$+0.333$	$+4.895$
H 2 O ( Zhang et al., 2023 )	KV / restricted decoder	$-0.219$	$+7.690$	$+4.560$	$+8.726$
xRAG ( Cheng et al., 2024 )	projector / suffix decoder	$-0.932$	$+6.790$	$-0.100$	$+1.237$
LoCoCo ( Cai et al., 2024 )	convolution / suffix adapter	$-1.414$	$+6.367$	$+0.676$	$+5.653$
CCM ( Kim et al., 2024 )	merge adapter / suffix adapter	$+2.153$	$+9.910$	$+0.462$	$+2.200$
ICAE ( Ge et al., 2024 )	decoder / suffix decoder	$+2.185$	$+7.767$	$-2.932$	$-2.326$

Table 1: Advantage over a trained restricted-input model (

U

) and zeroing damage (

D

), in nats; three seeds per row. Positive values favor intact memory. Study-specific simultaneous 95% intervals exclude zero for every sign (Table 19 ). Synthetic KV budgets: 24 updates with memory versus

183+64+512

for the restricted-input model. KV policies share weights trained with full context; the last four compressors remain frozen during adaptation. Access and optimization differ across rows (Appendix G ).

	Training objective
Model	All-token	Target-only
No-memory adapter	2.3189	3.2729
Pooled memory	2.3190	2.4921
Constant input	2.3106	2.4566

Table 2: Held-out answer-token NLL in nats (lower is better), averaged over ten paired training orders. With target-only training, all three models have higher loss, but the no-memory model worsens more. Thus a relative memory advantage can emerge without lower prediction loss. Appendix L retains accuracy, stopping records and uncertainty estimates.

Projector training input	Intact F1	Zeroed F1
Document	19.98	6.82
Same constant for all documents	12.54	6.82

Table 3: QASPER F1 on 979 questions, averaged over ten paired training orders. Zeroing the projector output leaves the same frozen LLM and prompt in both rows. Scores fall even when the learned input conveys no current-document facts. Appendix R retains the full results.

Input	Match (%)	EM (%)	F1
Closed book	57.01	1.31	19.38
Full text	65.87	1.64	21.86
Document	65.73	6.33	29.88
Question	60.53	1.66	22.43
Zero vector	57.61	1.28	19.49
Donor mean	56.45	5.11	24.78

Table 4: Released xRAG on 11,313 TriviaQA questions with fixed software and batch size four. Higher is better. Document encodes the retrieved passage; full text supplies it as text. Closed book omits the vector; question encodes the question instead. Donors average three assignments of other passages (Appendix K ).

	Donor minus closed book			Document minus donor
Dataset	Match	EM	F1	Match
TriviaQA	$-0.57\;[-1.50,0.37]$	$+3.80\;[3.22,4.42]$	$+5.40\;[4.64,6.17]$	$+9.28\;[8.26,10.31]$
NQ-open	$-0.20\;[-1.88,1.44]$	$+0.14\;[0.04,0.28]$	$+3.71\;[3.15,4.28]$	$+8.65\;[6.93,10.42]$
WebQuestions	$+3.99\;[1.49,6.59]$	$+0.15\;[0.00,0.35]$	$+3.98\;[2.91,4.84]$	$+5.22\;[3.05,7.47]$

Table 5: Score differences with fixed software and batch size four, in percentage or F1 points. Donors are other questions’ passages; closed book omits documents. Retrieved passages improve Match over donors in every dataset; donor gains over no document vary. Intervals adjust for fifteen TriviaQA or twelve combined NQ-open/WebQuestions comparisons, not a joint family across datasets.

Appendix figures & tables38 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	$L$	$k$	$H$	MLP width	Module (M)
Pythia-2.8B	32	10	2,560	4,098	20.98
Pythia-6.9B	32	10	4,096	6,402	52.45
Pythia-12B	36	8	5,120	7,938	81.30
Llama-3.2-1B	16	20	2,048	3,330	13.64
Llama-3.2-3B	28	11	3,072	4,866	29.90
Llama-3.1-8B	32	10	4,096	6,402	52.45

Appendix

Table 6: Controlled-state settings: layer count

L

, window

k

, backbone width

H

, and adapter intermediate width from saved tensors. Module counts exclude common LoRA. Parameter-count matching error is below 0.06%.

Backbone	Adapter NLL	Pooled NLL	Zeroed NLL	Gain $U$ [95% CI]	Zeroing $D$ [95% CI]
Pythia-2.8B	2.4284	2.5067	3.0140	$-0.0784\;[-0.0899,-0.0665]$	$+0.5073\;[+0.4814,+0.5337]$
Pythia-6.9B	2.3499	2.4293	2.7966	$-0.0795\;[-0.0923,-0.0671]$	$+0.3673\;[+0.3441,+0.3909]$
Pythia-12B	2.3736	2.4459	2.7628	$-0.0723\;[-0.0849,-0.0595]$	$+0.3169\;[+0.2946,+0.3394]$
Llama-3.2-1B	2.3019	2.3801	2.8610	$-0.0782\;[-0.0909,-0.0655]$	$+0.4809\;[+0.4539,+0.5073]$
Llama-3.2-3B	2.1926	2.3343	2.7986	$-0.1417\;[-0.1574,-0.1265]$	$+0.4643\;[+0.4395,+0.4892]$
Llama-3.1-8B	2.0586	2.0910	2.1906	$-0.0325\;[-0.0400,-0.0248]$	$+0.0996\;[+0.0848,+0.1145]$

Appendix

Table 7: Pooled PlanGen on state tokens (nats; marginal 95% document-bootstrap CIs, three fitted seeds).

U

compares pooled memory with the parameter-matched adapter;

D

compares its intact and zeroed inputs. All six have

U<0,D>0

. Candidate accuracy does not establish state recovery; mean uniform-candidate references are 11.35% (Pythia) and 17.24% (Llama). Table 8 reports all arms.

Backbone	LoRA only	Local adapter	Pooled	Indexed
Pythia-2.8B	2.6062 / 11.26	2.4284 / 10.86	2.5067 / 11.32	2.5168 / 11.29
Pythia-6.9B	2.4753 / 11.28	2.3499 / 11.00	2.4293 / 10.99	2.4261 / 11.99
Pythia-12B	2.5359 / 11.62	2.3736 / 11.44	2.4459 / 11.00	2.4516 / 11.11
Llama-3.2-1B	2.6037 / 13.54	2.3019 / 13.89	2.3801 / 13.77	2.4015 / 13.79
Llama-3.2-3B	2.6051 / 13.90	2.1926 / 14.48	2.3343 / 14.07	2.3449 / 14.11
Llama-3.1-8B	2.1252 / 13.71	2.0586 / 13.62	2.0910 / 13.63	2.0977 / 13.68

Appendix

Table 8: Controlled state targets: full-vocabulary NLL / candidate-token accuracy (%), averaged over seeds 71–73 on the same positions as Table 7 . All four arms are shown; raw NLLs are compared only within a backbone.

Training loss	Model	Target NLL	Accuracy (%)
All-token	Local adapter	2.1775	12.92
All-token	Normal pooled memory	2.2068	13.64
All-token	Document-independent memory	2.2055	13.48
Target-only	Local adapter	2.2422	16.83
Target-only	Normal pooled memory	2.1907	16.47
Target-only	Document-independent memory	2.1869	16.70

Appendix

Table 9: New-document content control: 1,024 targets from 588 stories, A6000, seeds 311–313. All 27 conditional 95% intervals are shown.

U

compares normal memory with the local adapter;

U_{\rm const}

compares document-independent memory with the same local adapter.

D_{\rm const}

is zeroing damage to that control.

C=\mathrm{NLL}_{\rm const}-\mathrm{NLL}_{\rm normal}

measures normal memory's advantage;

C_{\rm acc}

uses normal-minus-constant accuracy. Swap covers 1,015 targets; the other effects cover all 1,024. Accuracy is full-vocabulary first-token accuracy, unlike the candidate-restricted accuracy in Table 8 ; its contrasts use percentage points. An uninformed uniform full-vocabulary predictor has accuracy

100/|V|

percent, not the candidate-aware

100/K

reference.

QA effect (estimate)	Fixed-fit percentile 95%	Crossed simultaneous 95%
$U$ (F1) (+7.4436)	$[+6.1977,+8.7125]$	$[+5.4742,+9.4130]$
$C$ (F1) (+7.4336)	$[+6.1472,+8.7450]$	$[+5.4782,+9.3889]$
$D_{\rm zero}$ (F1) (+13.1594)	$[+11.6620,+14.6691]$	$[+10.9762,+15.3426]$
$D_{\rm swap}$ (F1) (+12.8842)	$[+11.5753,+14.2269]$	$[+10.8939,+14.8745]$
Constant zeroing (F1) (+5.7258)	$[+4.3961,+7.0852]$	$[+3.7904,+7.6612]$
Constant minus question (F1) (+0.0100)	$[-0.4945,+0.5263]$	$[-0.8445,+0.8645]$

Appendix

Table 10: All twelve trained-projector QA effects. Conditional percentile intervals hold ten fits fixed; crossed sensitivity resamples paired seeds and papers with adjustment across these twelve effects. Positive

U/C

favor document input, and positive intervention scores mean damage. F1 uses points and NLL nats. Fixed-storage LLM bounds appear in Tables 40 and 11 ; lexical bounds in Table 17 .

F1 contrast	Estimate	Fixed-fit paper 95%	Crossed simultaneous 95%
$R$ : intact recovery	$+7.08$	$[+5.74,+8.44]$	$[+5.22,+8.95]$
$T_{\rm LLM}$ : trained-reference gain	$+8.87$	$[+7.62,+10.12]$	$[+7.11,+10.63]$
$I_{\rm LLM}$ : intact-minus-donor recovery	$+10.83$	$[+9.55,+12.11]$	$[+9.07,+12.59]$
$J$ : training specificity	$+9.95$	$[+8.84,+11.05]$	$[+8.37,+11.52]$
$D_{\rm swap}$ : swap damage	$+14.14$	$[+12.81,+15.46]$	$[+12.32,+15.95]$
$D_{\rm mean}$ : mean damage	$+10.99$	$[+9.76,+12.20]$	$[+9.24,+12.73]$

Appendix

Table 11: All seven fixed-memory LLM effects, in F1 points. Each focal contrast in this earlier study (

R,T_{\rm LLM},I_{\rm LLM}

) is positive under both constructions. The seven-effect simultaneous family also resamples training-order seeds; neither interval covers arbitrary initialization or recipe selection. Two F1 points is a descriptive magnitude benchmark, not the existence criterion.

LLM and memory	F1	EM (%)	NLL	Tokens
Public LLM, intact	14.21	1.12	3.4513	48.90
Public LLM, swap	10.90	0.92	3.9922	48.47
Public LLM, mean	12.23	0.82	3.6518	40.51
Document-trained LLM, intact	21.29	4.58	2.0255	12.46
Document-trained LLM, swap	7.15	1.23	3.0156	11.10
Document-trained LLM, mean	10.30	1.97	2.7545	10.04

Appendix

Table 12: All nine absolute conditions on the same 979 questions. Adapted-LLM rows average ten fits. Intact uses the original projected document vector, swap uses a fixed different-paper vector, and mean uses the public projection of the training-only mean embedding. EM, NLL and length are descriptive. Lower swap scores for the document-trained LLM are retained.

Input	Match (%)	EM (%)	F1	Tokens
Closed-book	51.59	30.43	43.54	10.83
Full text	61.27	35.80	50.26	10.75
Document input	63.03	48.73	60.15	6.92
Question input	59.04	30.36	46.05	6.35
Zero vector	52.15	29.22	42.81	10.99
Donor 1	53.98	41.68	51.84	7.42

Appendix

Table 13: Released xRAG on all 11,313 repository TriviaQA questions. Match is token-substring answer inclusion; strict EM and F1 are descriptive secondary scores. No weights are fitted in this audit.

Effect	Estimate	Simultaneous 95% CI
Document $-$ closed	$+11.45$	$[+10.30,+12.59]$
Question $-$ closed	$+7.45$	$[+6.46,+8.44]$
Document $-$ question	$+4.00$	$[+2.98,+5.01]$
Document $-$ mean donor	$+9.00$	$[+8.05,+9.96]$
Mean donor $-$ closed	$+2.44$	$[+1.59,+3.30]$
Full text $-$ closed	$+9.68$	$[+8.37,+10.99]$

Appendix

Table 14: All eight released-model effects. The first seven are Match differences in percentage points; boost is a percentage conditional on closed-book errors. One maximum-standardized family uses 50,000 paired original-passage-cluster draws and a two-point benchmark.

Comparison	Estimate	Fixed-fit 95%	Adjusted sensitivity 95%
Question minus closed prefix F1	$+1.773$	$[+0.916,+2.633]$	$[+0.707,+2.838]$
Document minus closed prefix F1	$+15.331$	$[+14.505,+16.161]$	$[+14.301,+16.360]$
Donor minus closed prefix F1	$+6.575$	$[+5.926,+7.227]$	$[+5.766,+7.384]$
Document minus donor prefix F1	$+8.756$	$[+8.063,+9.445]$	$[+7.897,+9.614]$

Appendix

Table 15: Released-model oracle-prefix diagnostics. F1 uses points. All four rows form one separate follow-up family.

Panel	Input	NLL	Accuracy (%)
Original	Local input	2.7986	49.26
Original	Constant input	2.8214	50.41
Original	Remote input	2.6116	47.56
Original	Full-context reference	1.5019	66.36
Substituted	Local input	13.3199	0.00
Substituted	Constant input	13.0405	0.00

Appendix

Table 16: New-article first-token prediction. The three trained arms average ten paired training-order seeds. The full-context row is one frozen decoder receiving the whole prefix, not a local reference or a ten-seed fit.

Ten-fit PG-19 lexical study
Study / effect (estimate)	Fixed-fit percentile 95%	Crossed simultaneous 95%
$C$ (NLL) (+0.3057)	$[+0.2530,+0.3585]$	$[+0.2369,+0.3746]$
$C$ (accuracy pp) (+2.1255)	$[+1.1501,+3.0933]$	$[+0.8538,+3.3959]$
Constant-reference utility (NLL) (+0.0109)	$[-0.0042,+0.0259]$	$[-0.0095,+0.0318]$
Constant zeroing (NLL) (+1.2360)	$[+1.1719,+1.3010]$	$[+1.1544,+1.3180]$
NASA article lexical study

Appendix

Table 17: All four PG-19 and eighteen NASA effects, in separate families. Fixed-fit intervals resample documents; crossed intervals also resample ten paired order seeds with family adjustment. Original NASA

U/C

have positive fixed-fit lower bounds and negative crossed lower bounds. NLL uses nats and accuracy uses percentage points.

Task / score	$D_{\rm doc}$	$C$	$D_{\rm const}$	Ratio
QASPER (F1)	13.16	7.43	5.73	43.5%
PG-19 (NLL)	1.54	0.31	1.24	80.2%
NASA original (NLL)	0.59	0.21	0.38	64.3%

Appendix

Table 18: Shared-zero arithmetic:

D_{\rm doc}=C+D_{\rm const}

. Ratio is

D_{\rm const}/D_{\rm doc}

, a descriptive score ratio, not a causal fraction of stored information. Units differ across rows; complete intervals are in Appendices C and F .

System	Targets	Local NLL	Intact NLL	Gain $U$ [simultaneous CI]	Zeroing $D$ [simultaneous CI]
Synthetic: state-disambiguating token
StreamingLLM	1,652	1.849	2.459	$-0.610\;[-0.667,-0.552]$	$+6.484\;[+6.279,+6.694]$
H 2 O	1,652	1.849	2.068	$-0.219\;[-0.292,-0.147]$	$+7.690\;[+7.467,+7.916]$
xRAG	2,000	2.279	3.211	$-0.932\;[-1.010,-0.855]$	$+6.790\;[+6.582,+6.995]$
LoCoCo	2,000	2.232	3.646	$-1.414\;[-1.598,-1.234]$	$+6.367\;[+6.150,+6.591]$
CCM	2,000	2.231	0.078	$+2.153\;[+2.104,+2.197]$	$+9.910\;[+9.728,+10.089]$

Appendix

Table 19: Existing systems, three fitted seeds per row, in nats. Positive

U

favors memory; positive

D

means zeroing harms prediction. Synthetic rows score state-disambiguating tokens; PG-19 rows score repeated-word first tokens. Intervals are study-specific simultaneous 95% intervals. H 2 O uses full-prefix prefill followed by pruning.

System	Absent NLL	Zeroed NLL	$D_{\rm abs}$ [simultaneous CI]	$E$ [simultaneous CI]
Synthetic: state-disambiguating token
StreamingLLM	10.906	8.943	$+8.447\;[+8.307,+8.588]$	$+0.002\;[-0.004,+0.005]$
H 2 O	10.906	9.757	$+8.838\;[+8.684,+8.991]$	$+0.405\;[+0.353,+0.460]$
xRAG	9.971	10.001	$+6.760\;[+6.556,+6.962]$	$+1.268\;[+1.216,+1.321]$
PG-19: repeated-word first token
StreamingLLM	11.582	11.481	$+4.996\;[+4.687,+5.312]$	$+0.002\;[-0.003,+0.010]$

Appendix

Table 20: Additional interventions for StreamingLLM, H 2 O, and xRAG in Table 19 (nats). Positive

D_{\rm abs}

means absence harms prediction; positive

E

means evidence masking harms more than matched sham masking. The respective simultaneous families contain 8/8/32 effects.

Policy	Intact F1	$U_{\rm question}$	$C$ [crossed 95%]	$D_{\rm swap}-C$ [crossed 95%]
TF–IDF top-1	13.213	$+0.678$	$+0.668\;[-1.369,2.706]$	$+5.818\;[4.015,7.621]$
TF–IDF top-3	14.050	$+1.515$	$+1.505\;[-0.145,3.154]$	$+5.217\;[3.519,6.915]$
SFR top-1	15.597	$+3.062$	$+3.052\;[0.948,5.155]$	$+5.962\;[4.222,7.702]$
SFR top-3	15.880	$+3.345$	$+3.335\;[1.375,5.295]$	$+5.272\;[3.545,6.999]$

Appendix

Table 21: Retrieved-input deployment shift on the same 979 questions. Shared question/constant F1 is 12.535/12.545. Bounds belong to the full 28-effect seed–paper simultaneous family. All four policies have damage exceeding

C

, without establishing negative utility or

U/C

divergence. All 28 effects and absolute EM/NLL/length remain in the companion.

Comparison set	Adjusted family	Inference scope
QA projectors	12 F1/NLL effects	Ten paired order seeds and papers
Fixed-storage LLMs	7 original F1; 2 direct F1/EM	Each family paired over seeds and papers
Released QA model	8 Match/boost effects	One checkpoint; passage clusters
Lexical projectors	4 PG-19; 18 NASA	Separate tasks; paired seeds/documents
Optimized/trained constants	14 F1/NLL effects	Sample and interval choices use available results
Allocation / retrieval	6 / 28 F1 effects	Rule quality versus deployed inputs

Appendix

Table 22: Historical study families, supplemented by the six planned contrasts, fifteen pinned-generation effects and three CoQA effects in Appendix N . Secondary analyses are exploratory, not independent confirmations. No bound accounts for selection among analyses. Other studies retain their original fixed-fit adjustments; the numerical artifact lists all family members.

	Learned compression ( $n=19$ )				KV policy ( $n=21$ )
Category	Present	Not located	Unclear	n/a	Present	Not located	Unclear	n/a
Full input/cache	16	3	0	0	21	0	0	0
Frozen source restriction	4	12	1	2	3	18	0	0
Trained reference (broad)	16	3	0	0	3	0	0	18
Trained independent input	0	17	2	0	1	20	0	0
Source/order intervention	2	17	0	0	3	18	0	0

Appendix

Table 23: Sample descriptions under the stated broad definitions. Unclear and not-applicable cases are shown, not folded into absence. “Present” includes the qualified component/order controls described below. A missing control does not invalidate a paper’s performance or efficiency claim.

Contrast (F1 points)	Estimate	Crossed 95% CI	Six-primary adjusted CI
Intact minus same-paper swap	15.2365	$[13.7586,16.7300]$	$[13.2351,17.2696]$
Correct minus wrong document	7.4844	$[6.2284,8.7535]$	$[5.7842,9.1670]$
Correct document minus closed book	7.9442	$[6.6133,9.2603]$	$[6.1394,9.7217]$

Appendix

Table 24: Planned QASPER contrasts. Crossed resampling pairs order seeds and whole papers; the adjustment retains the original six-contrast family.

Decoder	Selected update: number of seeds
Correct document	64:10
Wrong document	64:10
Closed book	64:8, 128:2
Full text	64:7, 128:3

Appendix

Table 25: Development-selected checkpoints for the four decoder arms; each arm still completes all 512 updates.

Input	Match	EM	F1	Tokens
closed	57.01	1.31	19.38	27.35
full	65.87	1.64	21.86	27.82
document	65.73	6.33	29.88	21.17
question	60.53	1.66	22.43	15.53
zero	57.61	1.28	19.49	27.45
swap1	56.44	5.19	24.80	21.94

Appendix

Table 26: All input conditions with fixed software versions and batch size four. Closed supplies only the question; full supplies passage text; document/question supply the corresponding compressed vector; zero supplies a zero vector. Swap1–3 use vectors from different passages. Match measures answer inclusion, EM strict normalized exact match, and F1 token overlap. Donor comparisons average the three swaps.

Contrast	Metric	Estimate	Ordinary 95%	15-effect adjusted
donor minus closed	MATCH	-0.5657	$[-1.1804,0.0505]$	$[-1.5033,0.3651]$
donor minus closed	EM	+3.8039	$[3.4083,4.2064]$	$[3.2191,4.4162]$
donor minus closed	F1	+5.4011	$[4.8976,5.9070]$	$[4.6438,6.1705]$
document minus donor	MATCH	+9.2814	$[8.6003,9.9693]$	$[8.2599,10.3119]$
document minus donor	EM	+1.2169	$[0.7362,1.7047]$	$[0.5139,1.9439]$
document minus donor	F1	+5.0961	$[4.5498,5.6441]$	$[4.2815,5.9208]$

Appendix

Table 27: Pinned batch-four paired score contrasts in percentage points. Bonferroni intervals jointly cover the fifteen specified contrasts nominally at 95%; they are not a correction over every analysis in the paper.

Objective	Arm	NLL	Full accuracy (%)	Candidate accuracy (%)	Cap / 10	Plateau / 10
all-tokens	adapter	2.3189	15.51	15.51	8	2
all-tokens	pooled	2.3190	14.24	14.24	9	1
all-tokens	constant	2.3106	14.93	14.93	7	3
target	adapter	3.2729	14.00	18.37	7	3
target	pooled	2.4921	19.66	19.66	6	4
target	constant	2.4566	19.82	19.82	6	4

Appendix

Table 28: New-holdout target-token scores and stopping by arm; all ten fits retained. The candidate-informed uniform reference has mean NLL 1.8871 and accuracy 17.59%. It is not a uniform distribution over the full vocabulary.

Panel	Contrast (nats)	Estimate	Crossed 95%	Adjusted interval
New holdout	Objective difference in U	+0.7809	$[0.4656,1.0939]$	$[0.3668,1.1963]$
New holdout	All-token U	-0.0001	$[-0.0596,0.0625]$	$[-0.0797,0.0868]$
New holdout	Target-only U	+0.7808	$[0.4593,1.1087]$	$[0.3611,1.2187]$
New holdout	Target-only constant U	+0.8162	$[0.5059,1.1311]$	$[0.3619,1.2961]$
New holdout	Target-only C	-0.0354	$[-0.1101,0.0342]$	$[-0.1418,0.0590]$
Reused panel	Objective difference in U	+0.7469	$[0.4302,1.0664]$	$[0.3298,1.1722]$

Appendix

Table 29: Paired seed/story intervals. Constant utility uses its original nineteen-secondary-effect adjustment; other displayed adjusted intervals use six-comparison endpoints, with only the new-holdout objective difference a member of the six primary contrasts. Reused-panel estimates are secondary.

Figure 2: New holdout, ten paired fits per arm/objective. Target-only training worsens target-token NLL for all arms, most for the adapter; positive utility need not imply absolute improvement. Bars: crossed seed/story 95% intervals. Dotted NLL line: uniform-candidate reference. Appendix L gives adjusted intervals and accuracy.

Fit and evaluation input	F1
Projector: correct rationale	60.56
Projector: same-story non-rationale	5.14
Correct-document decoder	66.56
Wrong-document-trained decoder	34.22
Closed-book decoder	18.86

Appendix

Table 30: Absolute CoQA F1, averaged over ten fixed QASPER fits. Decoders trained with correct and wrong document vectors both receive the same correct rationale vector at evaluation.

Contrast (F1)	Estimate	Fixed-fit 95%	Crossed 95%	Three-effect adjusted
Rationale minus non-rationale projector	55.4226	$[51.9354,58.8963]$	$[51.8655,58.9748]$	$[51.0388,59.7601]$
Correct minus wrong-document training	32.3428	$[28.2753,36.4576]$	$[28.0996,36.6024]$	$[27.2110,37.5917]$
Correct document minus closed book	47.7046	$[43.6950,51.6845]$	$[43.5806,51.7956]$	$[42.6608,52.6938]$

Appendix

Table 31: Separate three-contrast family with paired story/seed resampling. Large gains are conditional on short oracle evidence and known upstream task exposure.

Planned contrast	Estimate	Ordinary crossed/clustered 95%	Six-contrast adjusted
Objective difference in utility (nats)	0.7809	$[0.4656,1.0939]$	$[0.3668,1.1963]$
QASPER intact minus same-paper swap (F1)	15.2365	$[13.7586,16.7300]$	$[13.2351,17.2696]$
QASPER correct minus wrong training (F1)	7.4844	$[6.2284,8.7535]$	$[5.7842,9.1670]$
QASPER correct minus closed book (F1)	7.9442	$[6.6133,9.2603]$	$[6.1394,9.7217]$
NQ-open/WebQuestions donor minus closed (EM)	7.8904	$[7.1712,8.6193]$	$[6.9193,8.8858]$
NQ-open/WebQuestions document minus donor (EM)	4.4157	$[3.6631,5.1656]$	$[3.4114,5.4279]$

Appendix

Table 32: All six planned comparisons, in the order defined above. The last two average NQ-open and WebQuestions equally under batch-one generation and pair passage clusters; they are not pinned TriviaQA Match claims. Score effects are percentage points except the first row.

System	Dataset	Method learning rate/decay	Local learning rate/decay
StreamingLLM	Syn.	1e-3/0.1	1e-5/0
H 2 O	Syn.	1e-3/0.1	1e-5/0
xRAG	Syn.	5e-5/0	5e-5/0
LoCoCo	Syn.	0.05/0	1e-3/0.1
CCM	Syn.	5e-05/0	1e-3/0
ICAE	Syn.	5e-05/0	5e-5/0

Appendix

Table 33: Final learning rates and weight decay for the nine-system comparison. Task adaptation, suffix access and development histories differ; these are finite configurations rather than an optimized architecture ranking.

System	Memory trainables	Local trainables	Synthetic updates (M/L)	PG-19 updates (M/L)
StreamingLLM	21,561,344	21,561,344	24 / (183+64+512)	128 / 128
H 2 O	21,561,344	21,561,344	24 / (183+64+512)	128 / 128
xRAG	33,562,624	33,559,552	128 / 128	128 / 128
LoCoCo	11,012,128	11,010,048	256 / 256	256 / 256
CCM	16,842,752	16,777,216	256 / 256	256 / 256
ICAE	218,636,288	218,636,288	128 / 128	64 / 64

Appendix

Table 34: Task-trainable parameters and actual updates of the final fits behind Table 1 , three seeds per row. M/L denotes memory/local. Counts exclude frozen backbone/compressor parameters and prior pretraining. Synthetic KV has unequal, staged fitting histories; parameter equality alone does not match optimization.

Learning rate	Document	Question	Constant
$2\times 10^{-4}$	2.1352	2.5067	2.4693
$8\times 10^{-4}$	2.1321	2.4550	2.4462
$3.2\times 10^{-3}$	2.3597	2.4405	2.4402
$1.28\times 10^{-2}$	2.4451	2.4540	2.4640
$5.12\times 10^{-2}$	2.5074	2.5078	2.5582

Appendix

Table 35: Minimum development canonical-answer NLL over the fixed checkpoints (lower is better). Every displayed cell selects update 128. These are development scores, not new test estimates.

Dataset	Input	Match	EM	F1	Mean output tokens
NQ-open	Closed book	30.2493	0.0000	7.9622	34.1532
NQ-open	Document	38.6981	0.6094	15.5835	28.1623
NQ-open	Donor 1	30.4432	0.1939	11.8873	28.1850
NQ-open	Donor 2	30.0554	0.0831	11.6641	28.9119
NQ-open	Donor 3	29.6399	0.1385	11.4745	28.8266
NQ-open	Donor mean	30.0462	0.1385	11.6753	28.6412

Appendix

Table 36: Absolute scores from every condition. Strict EM is near its floor on both datasets. Output length retokenizes saved answers and is not an independent mechanism test.

Dataset	Contrast	Score	Estimate	Ordinary 95%	Twelve-effect adjusted
NQ-open	Donor $-$ closed	MATCH	$-0.2031$	$[-1.3444,0.9311]$	$[-1.8828,1.4350]$
NQ-open	Donor $-$ closed	EM	$+0.1385$	$[0.0632,0.2307]$	$[0.0364,0.2820]$
NQ-open	Donor $-$ closed	F1	$+3.7131$	$[3.3262,4.1094]$	$[3.1503,4.2827]$
NQ-open	Document $-$ donor	MATCH	$+8.6519$	$[7.4687,9.8528]$	$[6.9336,10.4153]$
NQ-open	Document $-$ donor	EM	$+0.4709$	$[0.2227,0.7472]$	$[0.1285,0.8838]$
NQ-open	Document $-$ donor	F1	$+3.9082$	$[3.3504,4.4779]$	$[3.1030,4.7489]$

Appendix

Table 37: All twelve effects. The WebQuestions donor EM adjusted lower bound equals zero; its positive point estimate does not establish an adjusted positive effect.

Dataset	Input	Regroup
NQ-open	Closed	228
NQ-open	Document	221
NQ-open	Donor 1	216
NQ-open	Donor 2	223
NQ-open	Donor 3	224
WebQuestions	Closed	231

Appendix

Table 38: Changed raw answers among the first 256 questions under within-batch reversal or frozen regrouping; prompts and input tokens remain identical per item. All score changes are retained in the numerical supplement.

Input / intervention	F1	EM	Answer NLL
Document input	19.98	4.29	2.0979
Question input	12.53	3.15	2.4336
Constant input	12.54	3.13	2.4161
Shared zero-vector input	6.82	0.00	10.6852
Document, swapped	7.09	1.44	3.0507
Full-text reference	29.57	0.20	7.4681

Appendix

Table 39: Gold-answer QASPER with supplied evidence: all 979 test questions. Trained rows average ten training orders; full-text and closed-book rows share the frozen released decoder. F1/EM are percentages; NLL is example-mean answer-token loss. All arms receive the question. Zeroed projector outputs give the same frozen predictor, shown once (Equation 4 ).

Training input	F1	NLL	F1 gain
Correct document	21.21	1.9941	—
Wrong document	13.73	2.3289	$7.48\;[5.78,9.17]$
Closed book	13.27	2.4044	$7.94\;[6.14,9.72]$
Full text	62.54	0.6520	$-41.33$ (exploratory)

Appendix

Table 40: QASPER decoders: 979 test questions and ten training orders. Both document-trained rows use correct vectors at evaluation. F1 gain is the correct-document score minus each row. NLL is mean canonical-answer token loss in nats; lower is better. Gain intervals resample orders and papers, adjusting for six comparisons (Appendix N ). The full-text advantage has exploratory crossed 95% bounds

[38.95,43.68]

Compressed fit	Reference	Ref. F1	Gain
Projector (19.98)	Frozen closed book	6.60	13.37
Projector (19.98)	Question projector	12.53	7.44
Decoder (21.21)	Trained closed book	13.27	7.94
Decoder (21.21)	Wrong-document fit	13.73	7.48

Appendix

Table 41: QASPER gains over comparison models. Gain is compressed-model F1 minus reference F1, computed from unrounded means. All rows share questions; only rows training the same component hold the compressed model fixed.

Decoder	Correct	Other paper	Difference
Public, unadapted	14.21	10.90	3.31
Document-trained	21.29	7.15	14.14

Appendix

Table 42: F1 on identical QASPER inputs before and after decoder training. The trained row averages ten training orders from the earlier fixed-memory study; it is distinct from the later 21.21-F1 decoder comparison. The increase in the correct-minus-other difference is 10.83 points, with original crossed simultaneous 95% bounds

[9.07,12.59]

Memory-Augmented Language Models

Memory-augmented language models combine persistent or temporary memory with neural computation to retain, retrieve, and use information beyond the active context.

See all 292 papers

AdaMem: Adaptive Memory Token Allocation for Soft Compression in Retrieval-Augmented Generation
Aug 12, 2026Artem Sakhno, Grigorii Davydenko, Omar Zoloev +3RAG Context Compression Context Compression
BudgetMem: Training-Free Selective Memory for Cost-Efficient Long-Context Processing in Language Models
Nov 7, 2025Chandra Vamsi Krishna Alla, Harish Naidu Gaddam, Manohar Kommi +1Memory-Efficient Optimization LLM Compression
University of Texas at Arlington
Latent Compaction with Test-Time Training
Oct 14, 2026Yingfa Chen, Xingyu Shen, Chaojun Xiao +1LLM Context Compression KV-Cache Compression
NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China

Aug 12, 2026J/K move · Enter open · S save

AdaMem: Adaptive Memory Token Allocation for Soft Compression in Retrieval-Augmented Generation

Artem Sakhno, Grigorii Davydenko, Omar Zoloev, Julia Belikova+2

RAG Context Compression Context Compression Adaptive RAG Memory-Augmented Language Models LLM Inference Acceleration

Open paper AskPDF arXiv

Retrieval-augmented generation (RAG) improves language models with retrieved evidence, but processing many long passages is costly and can introduce distracting information. Soft compression addresses this challenge by encoding passages as compact sequences of continuous memory embeddings before generation. However, existing methods typically assign each retained passage an identical number of memory embeddings, irrespective of its query-specific relevance. To address this, we propose AdaMem, a relevance-guided soft-compression framework that maps learned passage-relevance estimates to a query-dependent allocation of a fixed memory-token budget. A shared query-conditioned compressor produces both continuous passage memories and relevance scores in a single pass; a deterministic allocation rule assigns more memory tokens to higher-scoring passages and can omit low-scoring ones. Across six open-domain QA benchmarks, AdaMem consistently outperforms OSCAR (the closely matched soft-compression baseline that uses uniform allocation) as well as other soft-compression methods at matched memory budgets. Under standard 16

\times

compression, AdaMem improves sub-string match by up to 3.2 points (5.5%) over uniform allocation baseline, with an average relative gain of 3.4%; under aggressive 64

\times

compression the average relative gain grows to 14.6%, with a maximum of 9.8 points (19.7%) on PopQA. AdaMem matches the answer quality of the uncompressed at up to 4

\times

lower inference latency than full context baseline. AdaMem retains an efficiency profile comparable to the uniform-compression baseline, while achieving up to

4\times

lower inference latency than full-context inference. Thus, relevance-guided memory allocation is particularly effective when retrieval pools are large and the available memory budget is tight.

Abstract

Memory-Augmented Language Models

Explore similar work

AdaMem: Adaptive Memory Token Allocation for Soft Compression in Retrieval-Augmented Generation

BudgetMem: Training-Free Selective Memory for Cost-Efficient Long-Context Processing in Language Models

Latent Compaction with Test-Time Training