Cache2LoRA: Understanding and Improving Context Internalization
Authors: Yi Fang, Gaurav Srivastava, Zhenyu Bi, Sirui Ding, Xuan Wang
Organizations: Department of Computer Science, Virginia Tech, Blacksburg, VA, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA, USA
Context-to-LoRA methods encode documents as low-rank parameter updates, allowing language models to answer without retaining the source text in their decoding context. How reliably do these updates preserve facts and support their use under different instructions? Controlled diagnosis of existing context-to-LoRA methods separates fact discrimination, target retention, and response adaptation. Competing sources influence answers, while target-only construction and memory-state optimization improve recovery at fixed update size. Recovering facts also does not ensure that answers follow response instructions. We introduce Cache2LoRA, which uses each query to aggregate evidence from a reusable frozen document cache before generating LoRA updates. Trainable encoding operates only on the query and memory-token suffix, and multi-style supervision teaches both evidence preservation and answer expression. Across 17 tasks, Cache2LoRA improves on existing context-to-LoRA methods and achieves macro F1 of 0.663, approaching full-context in-context learning at 0.671. For 32K documents and 256-token outputs at batch size 16, its decoding throughput is 5.9× that of full-context ICL. At matched memory size, generated LoRA preserves answer accuracy more reliably than the learned-KV variant under long, diverse intervening text and combines more effectively with retained document KV, yielding higher hybrid F1 and EM.
Figures & tables
Figure 1: Illustrative diagnostics and writing paradigms. (a) Fact discrimination, target retention, and response adaptation assess factual recovery and answer expression. Teal bars schematize relative target shares. (b) Queries guide answering in all three paradigms, memory routing in Doc-to-Atom, and memory writing in Cache2LoRA.
Figure 2: Internal readouts on FactSet-Homo. (a) Overlap between facts’ extraction contributions over 16 documents. (b) Variation in LoRA’s direct effect across questions, measured as a fraction of activation energy. Lines pair D2L and directly optimized LoRA of the same size on eight documents. Thick marks show means.
Figure 3: Changes to writing on 64 questions. (a) Attention share given to the target as statements are added. (b) F1/R-L with 15 added facts: unchanged writing, boosting target attention, averaging other statements’ vectors, or shifting attention toward the known target.
Figure 4: Adjusting the vectors used to generate LoRA. The answerer, generator, and LoRA size stay fixed. Supervised training changes only the generator’s input vectors. Bars count responses containing an accepted answer out of 64 questions (Appendix B.4 ).
Figure 5: Answer form under changed instructions. (a) Mean words across 13 tasks. (b) Short answers ( ≤ 5 words) among FactSet-Homo responses covering the full reference (R-L=1). Target only generates LoRA from the relevant passage with D2L. Labels give counts.
Figure 6: Cache2LoRA overview. Multi-style teacher answers supervise query-specific LoRA generation. Encoding LoRA acts on the query-and-memory suffix over a frozen document cache. Each layer’s M memory states generate a rank- M update.
Method
F1
R-L
EM
ICL (full context)
0.671
0.845
0.518
Parameter writers
D2L
0.173
0.514
0.016
D2L + Further Training
0.391
0.468
0.272
KV baselines, 48 positions
SnapKV
0.636
0.789
0.485
Table 1: Comparison with parameter and KV baselines. 17-task Qwen3-4B averages (Appendix D.1 ). TF uses frozen D2L. KV methods and Cache2LoRA use 6.75 MiB. Teal marks best non-ICL scores.
Method
State (MiB)
F1
R-L
EM
ICL
423.7
0.449
0.855
0.242
D2L
9.0
0.294
0.535
0.125
OSCAR
32.0
0.218
0.762
0.010
Cache2LoRA
9.0
0.456
0.780
0.261
Table 2: Mistral-7B comparison. Averages over 17 tasks, with best non-ICL scores highlighted. State measures document memory.
Configuration
F1
R-L
EM
(a) Cache2LoRA writer variants
Cache2LoRA
0.663
0.774
0.510
w/o encoding LoRA
0.562
0.650
0.405
w/o query conditioning
0.182
0.233
0.106
w/ matcher
0.665
0.775
0.511
(b) Cache2LoRA teacher recipes
Table 3: Writer, supervision, and memory-form ablations. Qwen3-4B macro averages over the 17-task evaluation suite. Component comparisons share the training pool and step budget. Short-answer-only and D2L prompt are single-style controls; teacher recipes share documents, queries, encoder architecture, and step budget. Memory output forms share training data, training budget, and a 6.75 MiB output size.
Figure 7: Memory recall under intervening context. Macro F1 and EM over six tasks. Unrelated text precedes the question. LoRA, Memory Tokens KV, and selected document KV are fixed before insertion. LoRA and Memory Tokens KV each use 6.75 MiB. ICL retains the full document. DropKV retains 10% of document KV (minimum 48 positions).
Figure 8: Hybrid F1 and EM with SnapKV and DropKV. The horizontal axis gives total document-derived state (MiB, log scale). Generated LoRA and Memory Tokens KV each add 6.75 MiB to the same selected document cache at B=48,128,512 .
Figure 9: Qwen3-4B efficiency on one H200. (a) Time from the query to the first answer token, excluding one-time document caching. Cache2LoRA generates and applies LoRA, while the control re-encodes the document for each query. (b) Batch-16 decoding throughput with 256 output tokens. SnapKV and DropKV retain 48 positions. Full results are in Table 28 b.
Style
Weight
User-message template
Evaluation short answer
0.20
D Answer the following question. Output only the answer and do not output any other words. Question: q
Original D2L
0.15
You are an honest and helpful assistant. # Provided Information D — # System Instruction - The information provided is up-to-date information and/or the user instruction. - When the provided information is not relevant to the question, ignore it and answer the question based on your knowledge. - If the provided information is related to the question, incorporate it in your response. - If the provided information is an instruction, follow the instruction carefully. — # User Input q
Direct answer
0.15
Document: D Answer the question directly. q
Explicit short answer
0.10
Document: D Answer the question directly. Output only the answer and do not output any other words. q
Entity or number
0.12
Document: D Answer with only the name, entity, date or number that answers the question. Do not write a sentence. Question: q
Complete sentence
0.12
Document: D Answer the question in one short, complete sentence. Question: q
Table 5: Teacher prompts and sampling weights. Each template is a single user message in the backbone’s chat template, followed by the assistant-generation prefix. D denotes the document. For the original-D2L, direct-answer, and explicit-short-answer styles, q includes a sampled D2L closed-QA instruction, except for PwC. Other styles use the raw question. Table 6 lists the closed-QA wrappers. Blank lines are condensed here.
No.
Template
1
Answer the question based on the given passages. Only give me the answer and do not output any other words. Question: q0
2
Answer without any explanation. Question: q0
3
Based on the provided text, what is the answer to the following question? Provide only the answer. Question: q0
4
Extract the answer to the question from the text. Be concise. Do not explain. Question: q0
5
What is the answer to this question, based on the context? Respond with the answer only. Question: q0
6
Provide a direct answer to the question using the given passages. Do not give any explanation. Question: q0
Table 6: D2L closed-QA wrappers. One of these 22 templates is sampled uniformly for the original-D2L, direct-answer, and explicit-short-answer styles. q0 is the raw question. The resulting string supplies q in Table 5 and serves as the student’s prompt. PwC uses its original query.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Wikipedia fact-count probe: 304 matched queries per count, using raw queries. Gray bands span the minimum and maximum ICL scores across the three writer/backbone configurations.
ICL
D2L
D2L (oracle)
Task
F1
R-L
F1
R-L
F1
R-L
FactSet-Homo
1.000
1.000
0.081
0.184
0.517
0.982
Wikipedia
0.844
0.977
0.059
0.217
0.280
0.958
SQuAD-4
0.729
0.943
0.210
0.604
0.307
0.834
NQ
0.568
0.841
0.162
0.530
0.261
0.689
NewsQA
0.479
0.770
0.217
0.538
0.351
0.787
Appendix
Table 7: Full-document and target-only writing (Qwen3-4B). ICL answers with the source text; D2L writes from the full input, while D2L (oracle) writes only from annotated supporting evidence. SQuAD-4 combines four passages and NQ uses full pages.
Figure 11: Natural-text validation on 60 SQuAD targets. Generic paragraphs omit names, dates, and numbers. Enrichment divides attention share by target token fraction. The dashed line marks 1.
Method
F1
R-L
≤ 5 words
Given R-L=1
No-document LM
0.111
0.208
0.291
0.409
ICL
0.751
0.917
0.742
0.757
D2L
0.187
0.553
0.043
0.045
Appendix
Table 8: Baseline short-answer diagnosis (13 non-LongBench tasks). The last two columns report the fraction of responses with at most five words, overall and conditional on R-L=1. Conditional rates omit tasks with no full-recall responses.
Figure 12: Fact discrimination and target retention in existing KV compressors. (a) F1 as a Wikipedia source grows from one to eight people. (b) R-L on 64 fixed targets as same-type facts are added. Both construct 48-position KV without the query.
Method
K=1
K=2
K=4
K=8
Borrowed
500xCompressor
0.797
0.905
0.715
0.827
0.633
0.732
0.445
0.502
22/152
CCM
0.812
0.914
0.678
0.780
0.515
0.601
0.387
0.436
19/152
Cache2LoRA w/o query
0.111
0.112
0.101
0.104
0.097
0.099
0.089
0.093
19/152
Cache2LoRA
0.825
0.931
0.815
0.927
0.825
0.934
0.819
0.934
2/152
Appendix
Table 9: Fact discrimination as source count increases. Birth-year and birthplace queries share a target passage with K−1 added people. Pairs report F1 / R-L; borrowed counts cover 152 birth-year queries at K=8 .
Method
Additions
k=0
1
3
7
15
31
500xCompressor
Same-type
0.992
0.984
0.938
0.781
0.500
0.188
Other-type
0.992
1.000
0.984
0.984
0.922
0.789
Empty
0.992
1.000
1.000
0.992
0.969
0.914
CCM
Same-type
1.000
0.844
0.742
0.586
0.352
0.094
Appendix
Table 10: Target retention under document expansion. R-L on 64 targets with k added facts or empty sentences.
Method
FactSet-Homo
Wiki (year)
SQuAD-4
NQ-4
500xCompressor
0.896
0.992
0.413
0.887
0.516
0.720
0.494
0.622
CCM
0.624
0.969
0.321
0.911
0.468
0.663
0.456
0.606
Cache2LoRA w/o query
0.001
0.005
0.089
0.118
0.199
0.226
0.231
0.247
Cache2LoRA
0.970
0.972
0.928
0.968
0.711
0.759
0.609
0.657
Appendix
Table 11: Full-document and target-only construction. Paired F1 scores use the full document / target evidence.
Method
Words
R-L
Short
Explain
Short
Explain
500xCompressor (48)
4.63
46.93
0.69
0.77
CCM (48)
4.25
46.83
0.63
0.71
Cache2LoRA
5.09
46.37
0.86
0.89
Appendix
Table 12: Response adaptation across memory constructors. Macro averages over 13 non-LongBench tasks under short-answer and explanation instructions. Length is measured in words.
Task
ICL
D2L
D2L + Further Training
Cache2LoRA-TF
Cache2LoRA
F1
R-L
EM
F1
R-L
EM
F1
R-L
EM
F1
R-L
EM
F1
R-L
EM
SQuAD
0.789
0.948
0.620
0.308
0.838
0.010
0.684
0.800
0.507
0.325
0.881
0.013
0.762
0.872
0.570
DROP
0.539
0.839
0.323
0.150
0.789
0.000
0.486
0.665
0.297
0.174
0.809
0.000
0.602
0.818
0.393
Wiki (birth year)
0.980
0.990
0.970
0.054
0.197
0.000
0.178
0.178
0.178
0.221
0.799
0.000
0.928
0.928
0.928
Wiki (birthplace)
0.708
0.964
0.283
0.065
0.237
0.000
0.163
0.206
0.092
0.266
0.913
0.020
0.678
0.921
0.270
FactSet-Homo
1.000
1.000
1.000
0.081
0.184
0.000
0.246
0.248
0.118
0.379
0.840
0.010
0.970
0.972
0.941
Appendix
Table 13: Per-task results on Qwen3-4B. Each method reports F1, R-L, and EM on the same queries. Teal cells mark the best rounded score for each task and metric, excluding ICL.
(a) Selected document KV
Task
SnapKV
DropKV
Fast KVzip
F1
R-L
EM
F1
R-L
EM
F1
R-L
EM
SQuAD
0.766
0.925
0.580
0.737
0.933
0.550
0.779
0.942
0.613
DROP
0.548
0.815
0.357
0.491
0.797
0.297
0.529
0.756
0.340
Wiki (year)
0.975
0.980
0.970
0.986
0.990
0.984
0.711
0.737
0.704
Wiki (place)
0.692
0.947
0.260
0.668
0.951
0.211
0.167
0.345
0.076
Appendix
Table 14: Standalone memory comparison. Per-task F1, R-L, and EM for six methods, all using 6.75 MiB of document-derived state. KV methods use a 48-position budget. Teal cells mark the best rounded score across all six methods for each task and metric.
Writer / backbone
raw
Cache2LoRA-TF
D2L / Qwen3-4B
0.19
0.29
D2L / Gemma-2-2B
0.51
0.73
SHINE / Qwen3-8B
0.51
0.64
Appendix
Table 15: Cache2LoRA-TF across writers. Macro F1 over 13 non-LongBench tasks: three FactSet variants, two Wikipedia attributes, SQuAD, DROP, NewsQA, TriviaQA, NQ, SQuAD-4, SQuAD-Mix, and NQ-4. Tasks have equal weight; queries are matched per writer.
ICL
D2L
Cache2LoRA-TF
Cache2LoRA
Task
F1
R-L
EM
F1
R-L
EM
F1
R-L
EM
F1
R-L
EM
SQuAD
0.882
0.917
0.753
0.769
0.799
0.633
0.828
0.848
0.700
0.841
0.861
0.690
DROP
0.691
0.754
0.553
0.535
0.587
0.433
0.601
0.639
0.483
0.686
0.747
0.530
Wiki (birth year)
0.969
0.970
0.967
0.181
0.181
0.181
0.559
0.559
0.559
0.931
0.931
0.931
Wiki (birthplace)
0.712
0.964
0.283
0.174
0.237
0.059
0.624
0.839
0.243
0.713
0.936
0.303
FactSet-Homo
1.000
1.000
1.000
0.418
0.426
0.319
0.938
0.939
0.891
0.996
0.997
0.990
Appendix
Table 16: Per-task benchmark results on Gemma-2-2B. Cache2LoRA uses the Gemma teacher with the same objective and training budget as the Qwen configuration.
4K intervening tokens
32K intervening tokens
Task
Memory Tokens KV
RoPE-aligned KV
Cache2LoRA
Memory Tokens KV
RoPE-aligned KV
Cache2LoRA
SQuAD-4
.316 / .502
.647 / .859
.591 / .812
.054 / .201
.087 / .290
.574 / .811
NQ-4
.256 / .429
.446 / .625
.456 / .659
.115 / .295
.116 / .316
.405 / .648
Wikipedia-Year
.243 / .260
.893 / .898
.910 / .918
.012 / .026
.039 / .125
.875 / .875
FactSet-Homo
.269 / .280
.745 / .788
.957 / .957
.002 / .013
.004 / .018
.947 / .947
NewsQA
.192 / .308
.312 / .626
.349 / .565
.028 / .088
.056 / .150
.316 / .529
Appendix
Table 17: Position control under intervening context. Each cell reports F1 / R-L. RoPE-aligned KV shifts only the learned memory keys’ RoPE positions to restore their relative distance to the question without removing intervening text. Original KV and Cache2LoRA scores are from Figure 7 ; RoPE-aligned KV scores are from the additional position control.
Added content
Metric
k=0
k=1
k=3
k=7
k=15
k=31
D2L
Same-type facts
F1
0.407
0.303
0.188
0.036
0.020
0.004
R-L
0.984
0.742
0.469
0.125
0.062
0.016
EM
0.141
0.094
0.062
0.000
0.000
0.000
Other-type facts
F1
0.407
0.406
0.374
0.346
0.311
0.304
R-L
0.984
0.977
0.914
0.844
0.781
0.758
Appendix
Table 18: Diagnostic repair under fixed-target expansion. All conditions retain the same 64 target queries and references. k counts added same-type facts, other-type facts, or empty units. The target-only result at k=0 is shared across content types. Bold, shaded values mark the best method for each content type, metric, and k .
Method
F1
R-L
EM
Mean words
Short answer
ICL
0.751
0.917
0.602
7.9
D2L
0.187
0.553
0.018
15.1
Cache2LoRA
0.745
0.862
0.594
5.1
One sentence
ICL
0.306
0.927
0.000
13.1
Appendix
Table 19: Answer quality and response length under four instructions. All methods use the same 13 tasks and three training styles plus a query-only instruction. Length is measured in words.
Short answer
Explanation
Method
Mean words
R-L
Mean words
R-L
Qwen3-4B
ICL
7.9
0.915
48.7
0.952
D2L
14.0
0.454
26.0
0.485
Cache2LoRA
5.3
0.844
46.8
0.875
Gemma-2-2B
Appendix
Table 20: Response length and answer coverage across backbones. Short-answer and explanation prompts use the same 904 queries from SQuAD-4, NQ-4, and Wikipedia birth year. Each backbone configuration uses one checkpoint for both instructions, which are represented in the multi-style recipe.
Method
Format success
F1
R-L
EM
Joint
Uppercase answer
ICL
0.954
0.450
0.863
0.190
0.186
D2L
0.907
0.137
0.385
0.009
0.009
Cache2LoRA
0.915
0.442
0.636
0.234
0.208
JSON object
ICL
0.993
0.724
0.905
0.603
0.603
Appendix
Table 21: Format compliance and answer quality under held-out formats. JSON and prefix use four matched tasks. Uppercase uses reference-eligible samples from three tasks. Content metrics score extracted answers independently of format compliance.
(a) Teacher setting and answering instruction
Instruction
Teacher setting
Mean words
F1
R-L
EM
Short answer
D2L prompt
19.6
0.307
0.777
0.088
Multi-style
7.2
0.663
0.774
0.510
Entity
D2L prompt
9.6
0.579
0.734
0.431
Multi-style
3.5
0.706
0.739
0.572
Sentence
D2L prompt
15.5
0.254
0.771
0.003
Appendix
Table 22: Teacher-recipe comparison across response instructions. D2L prompt denotes single-style supervision. All configurations use Cache2LoRA with the same source documents, queries, encoder architecture, and training-step budget. Panel (a) reports Qwen3-4B macro averages over the 17-task evaluation suite. Panel (b) follows the cohorts and scoring of Table 21 .
Configuration
Snap F1
R-L
EM
Drop F1
R-L
EM
KV only
0.636
0.789
0.485
0.620
0.780
0.472
KV + Cache2LoRA
0.696
0.816
0.545
0.705
0.825
0.555
KV + Cache2LoRA (other document)
0.573
0.720
0.424
0.570
0.717
0.426
Cache2LoRA only
0.663
0.774
0.510
0.663
0.774
0.510
Same-document query swap: matched subset of eight tasks
KV only
0.764
0.908
0.605
0.741
0.906
0.588
Appendix
Table 24: Cache2LoRA gains under the same answering pipeline. Macro averages over 17 matched tasks. Both KV methods use B=48 . Within each KV method, all conditions use the same selected document KV and process the instruction and query over that cache. The lower block uses documents with multiple queries.
KV only
KV + Cache2LoRA
Method
B
F1
R-L
EM
F1
R-L
EM
SnapKV
16
0.669
0.847
0.556
0.749
0.861
0.630
SnapKV
48
0.779
0.925
0.636
0.811
0.930
0.673
DropKV
16
0.550
0.756
0.452
0.700
0.815
0.597
DropKV
48
0.759
0.911
0.615
0.810
0.924
0.672
Appendix
Table 25: Answer-position retention control. Seven tasks with answer-matching KV positions prioritized within budget. Each pair shares the retained cache and query replay.
Method
B
KV source
KV only
+ Cache2LoRA ( X )
+ Cache2LoRA ( Y )
FactSet-Homo
SnapKV
16
X
0.54 / 0.00
0.87 / 0.00
0.45 / 0.06
SnapKV
16
Y
0.00 / 0.54
0.03 / 0.41
0.00 / 0.87
SnapKV
48
X
0.95 / 0.00
0.99 / 0.00
0.86 / 0.02
SnapKV
48
Y
0.00 / 0.93
0.01 / 0.87
0.00 / 0.97
DropKV
16
X
0.93 / 0.00
0.99 / 0.00
0.82 / 0.05
Appendix
Table 26: Counterfactual evidence conflicts. Cells report the fractions of responses containing the original ( X ) and counterfactual ( Y ) answers. KV and Cache2LoRA use independently chosen documents for each query.
(a) 64 output tokens: document lengths and KV budgets
State (MiB)
Prefill
Prepare
Decode
Throughput
Doc.
Method
Total
Document
(ms)
(ms)
batch 1 (ms/tok)
batch 16 (tok/s)
4K
ICL (full document)
581.9
576.4
279
29.6
22.38
512
SnapKV, B=48
12.7
6.8
279
47.0
22.35
627
DropKV, B=48
12.7
6.8
279
59.7
23.02
644
SnapKV, B=128
23.9
18.0
279
47.4
22.42
649
Appendix
Table 27: Preparation and decoding efficiency. Qwen3-4B on one H200, grouped by output length. Document prefill is paid once. Per-query preparation ends at the first answer token. State excludes model weights and document caches retained for future queries. Throughput uses batch 16; OOM denotes out of memory.
(b) 256 output tokens: document caching and per-query costs
Preparation (ms)
Req. state
Throughput
Doc. peak Δ
Doc.
Method
Document
Query
(MiB)
batch 16 (tok/s)
(MiB)
1K
ICL (full document)
36.3
29.1
149.91
630
222.2
SnapKV
36.3
46.3
12.66
624
222.2
DropKV
36.3
51.9
12.66
626
222.2
CCM
171.9
29.5
12.52
607
475.2
Appendix
Table 28: Preparation and decoding efficiency (continued). KV baselines in (b) retain up to 48 document positions. State in (b) is measured at the first answer token. Doc. peak Δ is the prefill peak allocation increment above the phase-start baseline; – denotes no separate document phase. In (c), query latency includes preparation and batch-one generation after document caching, and state includes the generated output.
Parametric retrieval augmentation encodes document information into lightweight, document-specific modules such as LoRA adapters, reducing the need to include all evidence as input context. However, it remains unclear how this parameter-side memory interacts with context-side memory stored in the KV cache. We study this interaction in document-level question answering by progressively evicting document key-value states and measuring when a document LoRA contributes beyond the retained context. We find that document LoRA adds little when the KV cache is largely intact, but becomes increasingly useful under aggressive compression, recovering 13-21 ROUGE-L points when no document context remains. The gain is largest when the base model encodes the document, and the adapter is applied only during answer generation, suggesting that document LoRA is better understood as decoding-time parametric memory than as a document encoder. Finally, QA-style supervision produces substantially stronger adapters than raw-context next-token-prediction. These results position document LoRA as a complementary memory channel whose value emerges precisely when context-side evidence is scarce.
For context-based question answering, using one large model for both evidence extraction and answer reasoning requires substantial memory and computation. Existing approaches trade accuracy for efficiency within a single model only. We propose a multi-model approach for efficient inference: our method, PAIR, employs two differently sized models for evidence extraction and answer reasoning, respectively. A smaller Reader model processes and retains the full context, while a larger Reasoner model sees only the question and generated sequence without directly attending to the context. These capabilities are fused at the logit level: at each token, both Reader and Reasoner produce a prediction, and their logits are combined into one, behaving as a single next-token-prediction model. Fine-tuning further boosts their performance. On HotpotQA, PAIR with a 32B Reasoner retains 96% of that same 32B model's full-context accuracy while using roughly one quarter of its KV memory. At 45.7K-token contexts, PAIR achieves up to 3.2× higher end-to-end throughput than the full-context 32B model on the same GPU. PAIR also improves the accuracy-memory and accuracy-throughput trade-offs over evaluated token-reduction baselines. Its accuracy-preserving behavior generalizes across Qwen2.5, OLMo-3, and Gemma-3 model families.
Sejun Park, Haocheng Xia, Yongjoo Park
Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, Urbana, IL 61801, USA
Long-context language models can increasingly process entire documents, making full-context prompting a practical alternative to passage retrieval. However, greater capacity determines how much information a model can receive, but not which information best supports a query. This distinction motivates rethinking retrieval: when the full document already fits, can selecting and organizing its evidence improve answering? The challenge is to retain supporting facts and relations while keeping the context focused on the question. To reconcile these goals, we introduce \ours, a training-free framework that uses semantic abstraction to guide selection while retaining original text for answering. \ours constructs a reusable semantic hierarchy from the full document in a single generation, linking node descriptions to source spans. Given a query, structural--semantic scoring and fine-grained lexical attribution nominate shared source nodes; their union with ancestor context preserves structural relations and complete local text. Across four frontier answer models, \ours improves average QA-F1 by 1.72 points over full-context prompting on three LongBench tasks, surpassing the evaluated baselines on the four-model average while reducing answer-context length by 58.55%. Our code is available at the repository.