Cache2LoRA: Understanding and Improving Context Internalization

Yi Fang; Gaurav Srivastava; Zhenyu Bi; Sirui Ding; Xuan Wang

cs.AIOct 16, 2026

Cache2LoRA: Understanding and Improving Context Internalization

Authors: Yi Fang, Gaurav Srivastava, Zhenyu Bi, Sirui Ding, Xuan Wang

Organizations: Department of Computer Science, Virginia Tech, Blacksburg, VA, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA, USA

Abstract

Context-to-LoRA methods encode documents as low-rank parameter updates, allowing language models to answer without retaining the source text in their decoding context. How reliably do these updates preserve facts and support their use under different instructions? Controlled diagnosis of existing context-to-LoRA methods separates fact discrimination, target retention, and response adaptation. Competing sources influence answers, while target-only construction and memory-state optimization improve recovery at fixed update size. Recovering facts also does not ensure that answers follow response instructions. We introduce Cache2LoRA, which uses each query to aggregate evidence from a reusable frozen document cache before generating LoRA updates. Trainable encoding operates only on the query and memory-token suffix, and multi-style supervision teaches both evidence preservation and answer expression. Across 17 tasks, Cache2LoRA improves on existing context-to-LoRA methods and achieves macro F1 of 0.663, approaching full-context in-context learning at 0.671. For 32K documents and 256-token outputs at batch size 16, its decoding throughput is 5.9 $\times$ that of full-context ICL. At matched memory size, generated LoRA preserves answer accuracy more reliably than the learned-KV variant under long, diverse intervening text and combines more effectively with retained document KV, yielding higher hybrid F1 and EM.

Figures & tables

Figure 1: Illustrative diagnostics and writing paradigms. (a) Fact discrimination, target retention, and response adaptation assess factual recovery and answer expression. Teal bars schematize relative target shares. (b) Queries guide answering in all three paradigms, memory routing in Doc-to-Atom, and memory writing in Cache2LoRA.

Figure 2: Internal readouts on FactSet-Homo. (a) Overlap between facts’ extraction contributions over 16 documents. (b) Variation in LoRA’s direct effect across questions, measured as a fraction of activation energy. Lines pair D2L and directly optimized LoRA of the same size on eight documents. Thick marks show means.

Figure 3: Changes to writing on 64 questions. (a) Attention share given to the target as statements are added. (b) F1/R-L with 15 added facts: unchanged writing, boosting target attention, averaging other statements’ vectors, or shifting attention toward the known target.

Figure 4: Adjusting the vectors used to generate LoRA. The answerer, generator, and LoRA size stay fixed. Supervised training changes only the generator’s input vectors. Bars count responses containing an accepted answer out of 64 questions (Appendix B.4 ).

Figure 5: Answer form under changed instructions. (a) Mean words across 13 tasks. (b) Short answers (

\leq

5 words) among FactSet-Homo responses covering the full reference (R-L=1). Target only generates LoRA from the relevant passage with D2L. Labels give counts.

Figure 6: Cache2LoRA overview. Multi-style teacher answers supervise query-specific LoRA generation. Encoding LoRA acts on the query-and-memory suffix over a frozen document cache. Each layer’s

M

memory states generate a rank-

M

update.

Method	F1	R-L	EM
ICL (full context)	0.671	0.845	0.518
Parameter writers
D2L	0.173	0.514	0.016
D2L + Further Training	0.391	0.468	0.272
KV baselines, 48 positions
SnapKV	0.636	0.789	0.485

Table 1: Comparison with parameter and KV baselines. 17-task Qwen3-4B averages (Appendix D.1 ). TF uses frozen D2L. KV methods and Cache2LoRA use 6.75 MiB. Teal marks best non-ICL scores.

Method	State (MiB)	F1	R-L	EM
ICL	423.7	0.449	0.855	0.242
D2L	9.0	0.294	0.535	0.125
OSCAR	32.0	0.218	0.762	0.010
Cache2LoRA	9.0	0.456	0.780	0.261

Table 2: Mistral-7B comparison. Averages over 17 tasks, with best non-ICL scores highlighted. State measures document memory.

Configuration	F1	R-L	EM
(a) Cache2LoRA writer variants
Cache2LoRA	0.663	0.774	0.510
w/o encoding LoRA	0.562	0.650	0.405
w/o query conditioning	0.182	0.233	0.106
w/ matcher	0.665	0.775	0.511
(b) Cache2LoRA teacher recipes

Table 3: Writer, supervision, and memory-form ablations. Qwen3-4B macro averages over the 17-task evaluation suite. Component comparisons share the training pool and step budget. Short-answer-only and D2L prompt are single-style controls; teacher recipes share documents, queries, encoder architecture, and step budget. Memory output forms share training data, training budget, and a 6.75 MiB output size.

Figure 7: Memory recall under intervening context. Macro F1 and EM over six tasks. Unrelated text precedes the question. LoRA, Memory Tokens KV, and selected document KV are fixed before insertion. LoRA and Memory Tokens KV each use 6.75 MiB. ICL retains the full document. DropKV retains 10% of document KV (minimum 48 positions).

Figure 8: Hybrid F1 and EM with SnapKV and DropKV. The horizontal axis gives total document-derived state (MiB, log scale). Generated LoRA and Memory Tokens KV each add 6.75 MiB to the same selected document cache at

B=48,128,512

Figure 9: Qwen3-4B efficiency on one H200. (a) Time from the query to the first answer token, excluding one-time document caching. Cache2LoRA generates and applies LoRA, while the control re-encodes the document for each query. (b) Batch-16 decoding throughput with 256 output tokens. SnapKV and DropKV retain 48 positions. Full results are in Table 28 b.

Style	Weight	User-message template
Evaluation short answer	0.20	$D$ Answer the following question. Output only the answer and do not output any other words. Question: $q$
Original D2L	0.15	You are an honest and helpful assistant. # Provided Information $D$ — # System Instruction - The information provided is up-to-date information and/or the user instruction. - When the provided information is not relevant to the question, ignore it and answer the question based on your knowledge. - If the provided information is related to the question, incorporate it in your response. - If the provided information is an instruction, follow the instruction carefully. — # User Input $q$
Direct answer	0.15	Document: $D$ Answer the question directly. $q$
Explicit short answer	0.10	Document: $D$ Answer the question directly. Output only the answer and do not output any other words. $q$
Entity or number	0.12	Document: $D$ Answer with only the name, entity, date or number that answers the question. Do not write a sentence. Question: $q$
Complete sentence	0.12	Document: $D$ Answer the question in one short, complete sentence. Question: $q$

Table 5: Teacher prompts and sampling weights. Each template is a single user message in the backbone’s chat template, followed by the assistant-generation prefix.

D

denotes the document. For the original-D2L, direct-answer, and explicit-short-answer styles,

q

includes a sampled D2L closed-QA instruction, except for PwC. Other styles use the raw question. Table 6 lists the closed-QA wrappers. Blank lines are condensed here.

No.	Template
1	Answer the question based on the given passages. Only give me the answer and do not output any other words. Question: $q_{0}$
2	Answer without any explanation. Question: $q_{0}$
3	Based on the provided text, what is the answer to the following question? Provide only the answer. Question: $q_{0}$
4	Extract the answer to the question from the text. Be concise. Do not explain. Question: $q_{0}$
5	What is the answer to this question, based on the context? Respond with the answer only. Question: $q_{0}$
6	Provide a direct answer to the question using the given passages. Do not give any explanation. Question: $q_{0}$

Table 6: D2L closed-QA wrappers. One of these 22 templates is sampled uniformly for the original-D2L, direct-answer, and explicit-short-answer styles.

q_{0}

is the raw question. The resulting string supplies

q

in Table 5 and serves as the student’s prompt. PwC uses its original query.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Wikipedia fact-count probe: 304 matched queries per count, using raw queries. Gray bands span the minimum and maximum ICL scores across the three writer/backbone configurations.

	ICL		D2L		D2L (oracle)
Task	F1	R-L	F1	R-L	F1	R-L
FactSet-Homo	1.000	1.000	0.081	0.184	0.517	0.982
Wikipedia	0.844	0.977	0.059	0.217	0.280	0.958
SQuAD-4	0.729	0.943	0.210	0.604	0.307	0.834
NQ	0.568	0.841	0.162	0.530	0.261	0.689
NewsQA	0.479	0.770	0.217	0.538	0.351	0.787

Appendix

Table 7: Full-document and target-only writing (Qwen3-4B). ICL answers with the source text; D2L writes from the full input, while D2L (oracle) writes only from annotated supporting evidence. SQuAD-4 combines four passages and NQ uses full pages.

Figure 11: Natural-text validation on 60 SQuAD targets. Generic paragraphs omit names, dates, and numbers. Enrichment divides attention share by target token fraction. The dashed line marks 1.

Method	F1	R-L	$\leq$ 5 words	Given R-L=1
No-document LM	0.111	0.208	0.291	0.409
ICL	0.751	0.917	0.742	0.757
D2L	0.187	0.553	0.043	0.045

Appendix

Table 8: Baseline short-answer diagnosis (13 non-LongBench tasks). The last two columns report the fraction of responses with at most five words, overall and conditional on R-L=1. Conditional rates omit tasks with no full-recall responses.

Figure 12: Fact discrimination and target retention in existing KV compressors. (a) F1 as a Wikipedia source grows from one to eight people. (b) R-L on 64 fixed targets as same-type facts are added. Both construct 48-position KV without the query.

Method	$K=1$		$K=2$		$K=4$		$K=8$		Borrowed
500xCompressor	0.797	0.905	0.715	0.827	0.633	0.732	0.445	0.502	22/152
CCM	0.812	0.914	0.678	0.780	0.515	0.601	0.387	0.436	19/152
Cache2LoRA w/o query	0.111	0.112	0.101	0.104	0.097	0.099	0.089	0.093	19/152
Cache2LoRA	0.825	0.931	0.815	0.927	0.825	0.934	0.819	0.934	2/152

Appendix

Table 9: Fact discrimination as source count increases. Birth-year and birthplace queries share a target passage with

K-1

added people. Pairs report F1 / R-L; borrowed counts cover 152 birth-year queries at

K=8

Method	Additions	$k=0$	1	3	7	15	31
500xCompressor
	Same-type	0.992	0.984	0.938	0.781	0.500	0.188
	Other-type	0.992	1.000	0.984	0.984	0.922	0.789
	Empty	0.992	1.000	1.000	0.992	0.969	0.914
CCM
	Same-type	1.000	0.844	0.742	0.586	0.352	0.094

Appendix

Table 10: Target retention under document expansion. R-L on 64 targets with

k

added facts or empty sentences.

Method	FactSet-Homo		Wiki (year)		SQuAD-4		NQ-4
500xCompressor	0.896	0.992	0.413	0.887	0.516	0.720	0.494	0.622
CCM	0.624	0.969	0.321	0.911	0.468	0.663	0.456	0.606
Cache2LoRA w/o query	0.001	0.005	0.089	0.118	0.199	0.226	0.231	0.247
Cache2LoRA	0.970	0.972	0.928	0.968	0.711	0.759	0.609	0.657

Appendix

Table 11: Full-document and target-only construction. Paired F1 scores use the full document / target evidence.

Method	Words		R-L
	Short	Explain	Short	Explain
500xCompressor (48)	4.63	46.93	0.69	0.77
CCM (48)	4.25	46.83	0.63	0.71
Cache2LoRA	5.09	46.37	0.86	0.89

Appendix

Table 12: Response adaptation across memory constructors. Macro averages over 13 non-LongBench tasks under short-answer and explanation instructions. Length is measured in words.

Task	ICL			D2L			D2L + Further Training			Cache2LoRA-TF			Cache2LoRA
	F1	R-L	EM	F1	R-L	EM	F1	R-L	EM	F1	R-L	EM	F1	R-L	EM
SQuAD	0.789	0.948	0.620	0.308	0.838	0.010	0.684	0.800	0.507	0.325	0.881	0.013	0.762	0.872	0.570
DROP	0.539	0.839	0.323	0.150	0.789	0.000	0.486	0.665	0.297	0.174	0.809	0.000	0.602	0.818	0.393
Wiki (birth year)	0.980	0.990	0.970	0.054	0.197	0.000	0.178	0.178	0.178	0.221	0.799	0.000	0.928	0.928	0.928
Wiki (birthplace)	0.708	0.964	0.283	0.065	0.237	0.000	0.163	0.206	0.092	0.266	0.913	0.020	0.678	0.921	0.270
FactSet-Homo	1.000	1.000	1.000	0.081	0.184	0.000	0.246	0.248	0.118	0.379	0.840	0.010	0.970	0.972	0.941

Appendix

Table 13: Per-task results on Qwen3-4B. Each method reports F1, R-L, and EM on the same queries. Teal cells mark the best rounded score for each task and metric, excluding ICL.

(a) Selected document KV
Task	SnapKV			DropKV			Fast KVzip
	F1	R-L	EM	F1	R-L	EM	F1	R-L	EM
SQuAD	0.766	0.925	0.580	0.737	0.933	0.550	0.779	0.942	0.613
DROP	0.548	0.815	0.357	0.491	0.797	0.297	0.529	0.756	0.340
Wiki (year)	0.975	0.980	0.970	0.986	0.990	0.984	0.711	0.737	0.704
Wiki (place)	0.692	0.947	0.260	0.668	0.951	0.211	0.167	0.345	0.076

Appendix

Table 14: Standalone memory comparison. Per-task F1, R-L, and EM for six methods, all using 6.75 MiB of document-derived state. KV methods use a 48-position budget. Teal cells mark the best rounded score across all six methods for each task and metric.

Writer / backbone	raw	Cache2LoRA-TF
D2L / Qwen3-4B	0.19	0.29
D2L / Gemma-2-2B	0.51	0.73
SHINE / Qwen3-8B	0.51	0.64

Appendix

Table 15: Cache2LoRA-TF across writers. Macro F1 over 13 non-LongBench tasks: three FactSet variants, two Wikipedia attributes, SQuAD, DROP, NewsQA, TriviaQA, NQ, SQuAD-4, SQuAD-Mix, and NQ-4. Tasks have equal weight; queries are matched per writer.

	ICL			D2L			Cache2LoRA-TF			Cache2LoRA
Task	F1	R-L	EM	F1	R-L	EM	F1	R-L	EM	F1	R-L	EM
SQuAD	0.882	0.917	0.753	0.769	0.799	0.633	0.828	0.848	0.700	0.841	0.861	0.690
DROP	0.691	0.754	0.553	0.535	0.587	0.433	0.601	0.639	0.483	0.686	0.747	0.530
Wiki (birth year)	0.969	0.970	0.967	0.181	0.181	0.181	0.559	0.559	0.559	0.931	0.931	0.931
Wiki (birthplace)	0.712	0.964	0.283	0.174	0.237	0.059	0.624	0.839	0.243	0.713	0.936	0.303
FactSet-Homo	1.000	1.000	1.000	0.418	0.426	0.319	0.938	0.939	0.891	0.996	0.997	0.990

Appendix

Table 16: Per-task benchmark results on Gemma-2-2B. Cache2LoRA uses the Gemma teacher with the same objective and training budget as the Qwen configuration.

	4K intervening tokens			32K intervening tokens
Task	Memory Tokens KV	RoPE-aligned KV	Cache2LoRA	Memory Tokens KV	RoPE-aligned KV	Cache2LoRA
SQuAD-4	.316 / .502	.647 / .859	.591 / .812	.054 / .201	.087 / .290	.574 / .811
NQ-4	.256 / .429	.446 / .625	.456 / .659	.115 / .295	.116 / .316	.405 / .648
Wikipedia-Year	.243 / .260	.893 / .898	.910 / .918	.012 / .026	.039 / .125	.875 / .875
FactSet-Homo	.269 / .280	.745 / .788	.957 / .957	.002 / .013	.004 / .018	.947 / .947
NewsQA	.192 / .308	.312 / .626	.349 / .565	.028 / .088	.056 / .150	.316 / .529

Appendix

Table 17: Position control under intervening context. Each cell reports F1 / R-L. RoPE-aligned KV shifts only the learned memory keys’ RoPE positions to restore their relative distance to the question without removing intervening text. Original KV and Cache2LoRA scores are from Figure 7 ; RoPE-aligned KV scores are from the additional position control.

Added content	Metric	$k=0$	$k=1$	$k=3$	$k=7$	$k=15$	$k=31$
D2L
Same-type facts	F1	0.407	0.303	0.188	0.036	0.020	0.004
	R-L	0.984	0.742	0.469	0.125	0.062	0.016
	EM	0.141	0.094	0.062	0.000	0.000	0.000
Other-type facts	F1	0.407	0.406	0.374	0.346	0.311	0.304
	R-L	0.984	0.977	0.914	0.844	0.781	0.758

Appendix

Table 18: Diagnostic repair under fixed-target expansion. All conditions retain the same 64 target queries and references.

k

counts added same-type facts, other-type facts, or empty units. The target-only result at

k=0

is shared across content types. Bold, shaded values mark the best method for each content type, metric, and

k

Method	F1	R-L	EM	Mean words
Short answer
ICL	0.751	0.917	0.602	7.9
D2L	0.187	0.553	0.018	15.1
Cache2LoRA	0.745	0.862	0.594	5.1
One sentence
ICL	0.306	0.927	0.000	13.1

Appendix

Table 19: Answer quality and response length under four instructions. All methods use the same 13 tasks and three training styles plus a query-only instruction. Length is measured in words.

	Short answer		Explanation
Method	Mean words	R-L	Mean words	R-L
Qwen3-4B
ICL	7.9	0.915	48.7	0.952
D2L	14.0	0.454	26.0	0.485
Cache2LoRA	5.3	0.844	46.8	0.875
Gemma-2-2B

Appendix

Table 20: Response length and answer coverage across backbones. Short-answer and explanation prompts use the same 904 queries from SQuAD-4, NQ-4, and Wikipedia birth year. Each backbone configuration uses one checkpoint for both instructions, which are represented in the multi-style recipe.

Method	Format success	F1	R-L	EM	Joint
Uppercase answer
ICL	0.954	0.450	0.863	0.190	0.186
D2L	0.907	0.137	0.385	0.009	0.009
Cache2LoRA	0.915	0.442	0.636	0.234	0.208
JSON object
ICL	0.993	0.724	0.905	0.603	0.603

Appendix

Table 21: Format compliance and answer quality under held-out formats. JSON and prefix use four matched tasks. Uppercase uses reference-eligible samples from three tasks. Content metrics score extracted answers independently of format compliance.

(a) Teacher setting and answering instruction
Instruction	Teacher setting	Mean words	F1	R-L	EM
Short answer	D2L prompt	19.6	0.307	0.777	0.088
	Multi-style	7.2	0.663	0.774	0.510
Entity	D2L prompt	9.6	0.579	0.734	0.431
	Multi-style	3.5	0.706	0.739	0.572
Sentence	D2L prompt	15.5	0.254	0.771	0.003

Appendix

Table 22: Teacher-recipe comparison across response instructions. D2L prompt denotes single-style supervision. All configurations use Cache2LoRA with the same source documents, queries, encoder architecture, and training-step budget. Panel (a) reports Qwen3-4B macro averages over the 17-task evaluation suite. Panel (b) follows the cohorts and scoring of Table 21 .

Same-document query swap: matched subset of eight tasks
Configuration	Snap F1	R-L	EM	Drop F1	R-L	EM
KV only	0.636	0.789	0.485	0.620	0.780	0.472
KV + Cache2LoRA	0.696	0.816	0.545	0.705	0.825	0.555
KV + Cache2LoRA (other document)	0.573	0.720	0.424	0.570	0.717	0.426
Cache2LoRA only	0.663	0.774	0.510	0.663	0.774	0.510
KV only	0.764	0.908	0.605	0.741	0.906	0.588

Appendix

Table 24: Cache2LoRA gains under the same answering pipeline. Macro averages over 17 matched tasks. Both KV methods use

B=48

. Within each KV method, all conditions use the same selected document KV and process the instruction and query over that cache. The lower block uses documents with multiple queries.

		KV only			KV + Cache2LoRA
Method	$B$	F1	R-L	EM	F1	R-L	EM
SnapKV	16	0.669	0.847	0.556	0.749	0.861	0.630
SnapKV	48	0.779	0.925	0.636	0.811	0.930	0.673
DropKV	16	0.550	0.756	0.452	0.700	0.815	0.597
DropKV	48	0.759	0.911	0.615	0.810	0.924	0.672

Appendix

Table 25: Answer-position retention control. Seven tasks with answer-matching KV positions prioritized within budget. Each pair shares the retained cache and query replay.

FactSet-Homo
Method	$B$	KV source	KV only	+ Cache2LoRA ( $X$ )	+ Cache2LoRA ( $Y$ )
SnapKV	16	X	0.54 / 0.00	0.87 / 0.00	0.45 / 0.06
SnapKV	16	Y	0.00 / 0.54	0.03 / 0.41	0.00 / 0.87
SnapKV	48	X	0.95 / 0.00	0.99 / 0.00	0.86 / 0.02
SnapKV	48	Y	0.00 / 0.93	0.01 / 0.87	0.00 / 0.97
DropKV	16	X	0.93 / 0.00	0.99 / 0.00	0.82 / 0.05

Appendix

Table 26: Counterfactual evidence conflicts. Cells report the fractions of responses containing the original (

X

) and counterfactual (

Y

) answers. KV and Cache2LoRA use independently chosen documents for each query.

(a) 64 output tokens: document lengths and KV budgets
		State (MiB)		Prefill	Prepare	Decode	Throughput
Doc.	Method	Total	Document	(ms)	(ms)	batch 1 (ms/tok)	batch 16 (tok/s)
4K	ICL (full document)	581.9	576.4	279	29.6	22.38	512
	SnapKV, $B{=}48$	12.7	6.8	279	47.0	22.35	627
	DropKV, $B{=}48$	12.7	6.8	279	59.7	23.02	644
	SnapKV, $B{=}128$	23.9	18.0	279	47.4	22.42	649

Appendix

Table 27: Preparation and decoding efficiency. Qwen3-4B on one H200, grouped by output length. Document prefill is paid once. Per-query preparation ends at the first answer token. State excludes model weights and document caches retained for future queries. Throughput uses batch 16; OOM denotes out of memory.

(b) 256 output tokens: document caching and per-query costs
		Preparation (ms)		Req. state	Throughput	Doc. peak $\Delta$
Doc.	Method	Document	Query	(MiB)	batch 16 (tok/s)	(MiB)
1K	ICL (full document)	36.3	29.1	149.91	630	222.2
	SnapKV	36.3	46.3	12.66	624	222.2
	DropKV	36.3	51.9	12.66	626	222.2
	CCM	171.9	29.5	12.52	607	475.2

Appendix

Table 28: Preparation and decoding efficiency (continued). KV baselines in (b) retain up to 48 document positions. State in (b) is measured at the first answer token. Doc. peak

\Delta

is the prefill peak allocation increment above the phase-start baseline; – denotes no separate document phase. In (c), query latency includes preparation and batch-one generation after document caching, and state includes the generated output.

LLM Context Compression

See all 50 papers

Rethinking LoRA Memory Through the Lens of KV Cache Compression
Jun 4, 2026Chunsheng Zuo, Liaoyaqi Wang, William Jurayj +2Retrieval-Augmented Generation Memory-Augmented Language Models
Johns Hopkins University
PAIR: Memory-Efficient and Fast Inference with Reading--Reasoning Fusion
Oct 12, 2026Sejun Park, Haocheng Xia, Yongjoo ParkMemory-Efficient Inference Long-Context Language Model Inference
Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, Urbana, IL 61801, USA
Rethinking Retrieval in the Long-Context Era: Training-Free Semantic Trees for Adaptive Context Construction
Oct 25, 2026Zhe Xu, Tianhao Liao, Zhenqing Ling +3Long-Context Retrieval Information Retrieval
Sun Yat-sen University · Alibaba Group

Jun 4, 2026J/K move · Enter open · S save

Rethinking LoRA Memory Through the Lens of KV Cache Compression

Chunsheng Zuo, Liaoyaqi Wang, William Jurayj, William Fleshman+1

Johns Hopkins University

Retrieval-Augmented Generation Memory-Augmented Language Models KV Caching Document QA Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Parametric retrieval augmentation encodes document information into lightweight, document-specific modules such as LoRA adapters, reducing the need to include all evidence as input context. However, it remains unclear how this parameter-side memory interacts with context-side memory stored in the KV cache. We study this interaction in document-level question answering by progressively evicting document key-value states and measuring when a document LoRA contributes beyond the retained context. We find that document LoRA adds little when the KV cache is largely intact, but becomes increasingly useful under aggressive compression, recovering 13-21 ROUGE-L points when no document context remains. The gain is largest when the base model encodes the document, and the adapter is applied only during answer generation, suggesting that document LoRA is better understood as decoding-time parametric memory than as a document encoder. Finally, QA-style supervision produces substantially stronger adapters than raw-context next-token-prediction. These results position document LoRA as a complementary memory channel whose value emerges precisely when context-side evidence is scarce.

Abstract

LLM Context Compression

Explore similar work

Rethinking LoRA Memory Through the Lens of KV Cache Compression

PAIR: Memory-Efficient and Fast Inference with Reading--Reasoning Fusion

Rethinking Retrieval in the Long-Context Era: Training-Free Semantic Trees for Adaptive Context Construction