Rethinking Retrieval in the Long-Context Era: Training-Free Semantic Trees for Adaptive Context Construction

Zhe Xu; Tianhao Liao; Zhenqing Ling; Ying Shen; Yaliang Li; Daoyuan Chen

cs.CLOct 25, 2026

Rethinking Retrieval in the Long-Context Era: Training-Free Semantic Trees for Adaptive Context Construction

Authors: Zhe Xu, Tianhao Liao, Zhenqing Ling, Ying Shen, Yaliang Li, Daoyuan Chen

Organizations: Sun Yat-sen University · Alibaba Group

Abstract

Long-context language models can increasingly process entire documents, making full-context prompting a practical alternative to passage retrieval. However, greater capacity determines how much information a model can receive, but not which information best supports a query. This distinction motivates rethinking retrieval: when the full document already fits, can selecting and organizing its evidence improve answering? The challenge is to retain supporting facts and relations while keeping the context focused on the question. To reconcile these goals, we introduce \ours, a training-free framework that uses semantic abstraction to guide selection while retaining original text for answering. \ours constructs a reusable semantic hierarchy from the full document in a single generation, linking node descriptions to source spans. Given a query, structural--semantic scoring and fine-grained lexical attribution nominate shared source nodes; their union with ancestor context preserves structural relations and complete local text. Across four frontier answer models, \ours improves average QA-F1 by 1.72 points over full-context prompting on three LongBench tasks, surpassing the evaluated baselines on the four-model average while reducing answer-context length by 58.55%. Our code is available at the repository.

Figures & tables

Figure 1: A schematic example contrasts full context with distractors, incomplete retrieval with missing support, and TreeFuse ’s focused context retaining both supporting statements.

Figure 2: Overview of TreeFuse. (a) A single generation constructs a semantic hierarchy whose nodes are aligned to original source spans. (b) Structural–semantic scoring and fine-grained lexical attribution independently nominate nodes from this shared representation. (c) Their union is completed with ancestor nodes and used to prune the original tree. Retained nodes preserve their structural relations and complete local source text for answering.

Dataset	Method	Qwen3.8-Max	DeepSeek-V4-Pro	Kimi-K2.6	GLM-5.3	Overall
Qasper	Raw	48.89	51.20	50.35	46.61	49.26
	LongLL \citep longllmlingua2024	29.75	26.55	24.58	27.72	27.15 ( $\downarrow$ 22.11 )
	Provence \citep provence2025	47.50	46.18	48.17	46.84	47.17 ( $\downarrow$ 2.09 )
	HiChunk \citep hichunk2025	49.70	49.88	49.24	46.85	48.92 ( $\downarrow$ 0.35 )
	BRIEF-Pro \citep briefpro2026	49.93	48.50	47.98	47.06	48.37 ( $\downarrow$ 0.90 )
	LongRefiner \citep longrefiner2025	31.52	31.00	29.91	29.59	30.50 ( $\downarrow$ 18.76 )

Table 1: LongBench QA-F1 (three-run means). Overall averages four models; parentheses show changes from Raw. Bold/underline mark the best/second-best means.

Method	Qwen	DS	Kimi	GLM	Overall
Raw	79.26	76.52	80.66	84.22	80.17
LongLL	64.45	61.14	62.30	64.93	63.20 ( $\downarrow$ 16.96 )
Provence	71.85	70.77	71.93	76.96	72.88 ( $\downarrow$ 7.29 )
HiChunk	72.15	72.23	73.85	77.59	73.96 ( $\downarrow$ 6.21 )
BRIEF-Pro	72.30	71.14	70.73	75.31	72.37 ( $\downarrow$ 7.79 )
LongRefiner	43.07	38.79	38.68	43.04	40.89 ( $\downarrow$ 39.27 )

Table 2: LaRA accuracy (%) averaged over three answer runs. Overall averages the four models, with changes from Raw in parentheses.

Dataset	Method	All-hit	Purity	ECP	Overall
Qasp.	LongLL	0.57	2.18	1.38	27.15
	Prov.	16.57	17.82	17.20	47.17
	HiChunk	96.00	5.80	50.90	48.92
	BRIEF	73.71	25.41	49.56	48.37
	LongRef.	20.57	5.19	12.88	30.50
	TreeFuse	96.57	9.33	52.95	50.25

Table 3: Evidence coverage, purity, and ECP (%). Overall QA-F1 is reused from Table 1 .

Variant	HotpotQA	2WikiMQA	Avg.
Raw	75.69	84.19	79.94
Semantic-only	76.59	83.97	80.28 ( $\uparrow$ 0.34 )
Lexical-only (BM)	73.54	76.38	74.96 ( $\downarrow$ 4.98 )
Semantic-only (BM)	74.93	81.87	78.40 ( $\downarrow$ 1.54 )
TreeFuse	77.39	84.85	81.12 ( $\uparrow$ 1.18 )

Table 4: Effects of semantic and lexical selection.

Variant	HotpotQA	2WikiMQA	Avg.
Raw	75.69	84.19	79.94
Fixed 200	72.77	79.47	76.12 ( $\downarrow$ 3.82 )
Chonkie	74.33	78.77	76.55 ( $\downarrow$ 3.39 )
TreeFuse (flat)	74.07	80.93	77.50 ( $\downarrow$ 2.44 )
TreeFuse	77.39	84.85	81.12 ( $\uparrow$ 1.18 )

Table 5: Effects of semantic units and hierarchy.

Variant	HotpotQA	2WikiMQA	Avg.
Raw	75.69	84.19	79.94
Multi-stage	75.47	82.33	78.90 ( $\downarrow$ 1.04 )
HiChunk adapter	74.15	82.54	78.34 ( $\downarrow$ 1.60 )
TreeFuse	77.39	84.85	81.12 ( $\uparrow$ 1.18 )

Table 6: Effects of tree-construction strategy.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Indexed representation	Query-time mechanism	Reader input	Reuse / Train
LongLLMLingua	Flat prompt	Token pruning	Retained tokens	No / No
Provence	Flat passages	Sentence pruning	Source sentences	No / Yes
RAPTOR	Summary tree	Dense node retrieval	Summaries / nodes	Yes / No
LongRefiner	Document hierarchy	Learned adaptive refinement	Refined spans	Yes / Yes
HiChunk	Learned hierarchy	Retrieval + Auto-Merge	Merged chunks	Yes / Yes
ACC-RAG	Multi-granular embeddings	Adaptive compression rate	Compressed features	Yes / Yes

Appendix

Table 7: Comparison with evaluated baselines and selected related methods. ‘Reuse’ denotes a document representation shared across queries. ‘Train’ denotes method-specific parameter updates.

Figure 3: Three paradigms for constructing an answer context. Flat methods organize text as independent units and retrieve, prune, or rewrite them; prior hierarchical methods build method-dependent trees and retrieve, expand, merge, or refine tree regions; TreeFuse constructs a reusable source-grounded semantic hierarchy from the full document, prunes it to an ancestor-closed retained subtree for each query, and serializes the complete local source spans while preserving the original parent–child relations and sibling order.

Interface	Fields
Construction	sections : records with structure , title , boundary_type , heading_verbatim , text_before_cut , summary , and facts .
Scoring input	Query, document directory, and node records containing node_id , title , and semantic material .
Scoring output	scores : records with node_id , integer relevance in $[0,10]$ , and reason .
Selection cutoff	Integer keep_threshold in $[1,10]$ .
Answer context	doc_id and tree , with level , title , source_text , and optional children per node.

Appendix

Table 8: Principal interfaces used for construction, selection, and answering. Only fields relevant to the all-node selection path are shown.

Setting	Value or behavior	Provenance
Constructor / scorer	Qwen3.6-35B-A3B	Reported setting
Construction facts	Typically 3–6, at most 8 requested	Construction prompt
Scoring material	Section summary, atomic facts, and parent context	Node metadata
Missing metadata	First 300 source characters when both summary and facts are absent	Scorer implementation
Batch size	40 nodes and 48,000 serialized-entry characters	Scorer defaults
Cutoff suggestions	3 attempts, minimum valid keep threshold	Selector default

Appendix

Table 9: Method settings. Entries marked as defaults refer to the implementation.

Method	Compression or retrieval model	Configuration
LongLL	Llama-2-7B	Requested retention rate 0.25.
Provence	DeBERTa-v3 reranker	Sentence threshold 0.1, title retention enabled, no reordering.
HiChunk	Youtu-HiChunk and BGE-M3	200-token chunks, maximum depth 10, top-128 retrieval and Auto-Merge, 4,096-token context budget.
BRIEF-Pro	BRIEF-Pro checkpoint	20-sentence instruction, 1,024-token output limit.
LongRefiner	Qwen2.5-3B-Instruct	Query analysis, document structuring, and global selection, with BGE reranking and a 2,048-token target.

Appendix

Table 10: Models and context settings for the main baseline comparison.

Dataset	Raw	LongLL	Provence	HiChunk	BRIEF-Pro	LongRefiner	TreeFuse
Qasper	49.26 $\pm$ 0.99	27.15 $\pm$ 1.05	47.17 $\pm$ 0.90	48.92 $\pm$ 0.48	48.37 $\pm$ 0.74	30.50 $\pm$ 0.35	50.25 $\pm$ 0.22
HotpotQA	69.51 $\pm$ 0.72	56.55 $\pm$ 1.08	66.01 $\pm$ 0.78	66.62 $\pm$ 2.41	68.18 $\pm$ 0.51	57.64 $\pm$ 1.87	71.23 $\pm$ 1.31
2WikiMQA	70.63 $\pm$ 0.73	56.54 $\pm$ 0.87	61.82 $\pm$ 0.70	69.69 $\pm$ 0.23	69.34 $\pm$ 1.16	45.80 $\pm$ 0.45	73.09 $\pm$ 0.09
LaRA	80.17 $\pm$ 0.30	63.20 $\pm$ 0.29	72.88 $\pm$ 0.28	73.96 $\pm$ 1.14	72.37 $\pm$ 1.19	40.89 $\pm$ 1.48	80.59 $\pm$ 0.37

Appendix

Table 11: Run-level variability for the main comparison. Each entry is the four-model Overall mean with its 95% Student-

t

interval over three decoding runs. The intervals characterize run variability on the fixed evaluation sample and contexts only.

Panel of Figure 4 — Figure 4: Compression–performance trade-off on Qasper, HotpotQA, and 2WikiMQA, averaged over Qwen3.8-Max, DeepSeek-V4-Pro, Kimi-K2.6, and GLM-5.3. Compression rate is the percentage of raw context characters removed. The vertical axis starts below 0% so Raw remains visible. Moving right indicates better QA-F1, whereas moving upward indicates stronger compression.

Benchmark	Method	Qwen	DeepSeek	Kimi	GLM	Context (%)
LB	Raw	69.59	67.56	52.42	63.02	100.00
	LongLLMLingua	63.11	55.90	42.69	53.21	38.76
	Provence	69.43	67.32	53.10	63.45	35.96
	HiChunk	66.54	64.62	50.27	61.11	38.96
	BRIEF-Pro	68.71	63.84	54.28	62.29	23.75
	LongRefiner	55.43	41.95	42.02	45.11	30.89

Appendix

Table 12: Additional comparisons with adjusted baseline context lengths. LongBench reports mean QA-F1 across the three tasks and LaRA reports accuracy. Context is the mean percentage of raw-context characters retained across the corresponding models and tasks.

Dataset	$D$	$N$	$N/D$	Construction	Selection	$C_{\mathrm{pre}}$
Qasper	148	200	1.351	4,865	14,318	19,183
HotpotQA	200	200	1.000	14,827	33,418	48,245
2WikiMQA	200	200	1.000	8,903	25,403	34,306
LaRA	64	887	13.859	2,209	19,485	21,694

Appendix

Table 13: Preprocessing input-token accounting. Construction and selection columns are per-question Qwen3.6 inputs.

D

and

N

describe the indexed collection used to amortize preprocessing. Values are rounded.

Answer model	Dataset	$\kappa^{*}$	Raw	TreeFuse	Modeled red. (%)
Qwen3.8-Max $\kappa=31.502$	Qasper	10.24	163,591	123,773	24.3
	HotpotQA	5.40	424,765	191,750	54.9
	2WikiMQA	6.36	238,521	102,885	56.9
	LaRA	1.66	923,552	532,473	42.3
DeepSeek-V4-Pro $\kappa=17.149$	Qasper	10.64	87,571	75,833	13.4
	HotpotQA	5.75	219,470	123,776	43.6

Appendix

Table 14: Break-even thresholds and modeled reductions in proxy-weighted successful-path input work under the stated

\kappa

. Raw and TreeFuse are Qwen3.6 input-token equivalents per question, including preprocessing for TreeFuse . Percentages are computed from the displayed rounded proxy totals.

Variant	Change relative to the full method
Semantic-only	Retain semantic selections and their ancestor context.
Lexical-only (BM)	Use lexical node selection with ancestor closure and an approximately matched character budget.
Semantic-only (BM)	Add fixed-seed random nodes to the semantic selection to approach the full method’s context length.
Fixed 200	Replace semantic units with 200-token chunks and apply the same selection framework.
Chonkie	Replace units with the configured Chonkie semantic chunks.
Flat	Retain the semantic units and source text, but remove their hierarchy.

Appendix

Table 15: Changes made by each ablation variant.

Full minus	Qasper	HotpotQA	2WikiMQA	Pooled
Semantic-only	+1.14 / +0.11	+1.86 / +0.40	+0.53 / +0.56	+1.17 / +0.36
Semantic-only (BM)	+1.14 / $-1.14$	$-0.07$ / +2.42	+0.22 / +2.76	+0.41 / +1.42
Lexical-only (BM)	+16.45 / +0.60	+18.96 / +4.17	+28.46 / +8.14	+21.50 / +4.45

Appendix

Table 16: Conditional evidence and answer differences with Qwen3.8-Max. Each cell reports

\Delta

Span Recall in percentage points /

\Delta

QA-F1 in score points. BM denotes approximate character-budget matching.

Condition	Evidence units	Span Recall	All-hit	Mean QA-F1	Context chars.
Lexical-only (BM)	0/4	0.00	No	17.57	5,050
Full ( TreeFuse )	4/4	100.00	Yes	35.33	12,552

Appendix

Table 17: Evidence retention for the Qasper weight-adjustment case. Span Recall and QA-F1 use a 0–100 scale; QA-F1 averages three Qwen3.8-Max answer runs. BM matches mean character budgets, not individual context lengths.

Long-Context Retrieval

Long-context retrieval finds and uses relevant evidence across lengthy documents, conversations, and model contexts.

See all 45 papers

UNREAL: Unifying Retrieval and Long-Context with a Single Model
Oct 6, 2026Edan Kinderman, Elad Hoffer, Yochai Blau +4Retrieval-Augmented Generation Knowledge Augmentation for Language Models
NVIDIA · Technion
ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning
Jul 2, 2026Yanjun Zhao, Ruizhong Qiu, Tianxin Wei +6Long-Context Inference Long-Context Reasoning in Language Models
University of Illinois Urbana-Champaign
Residual Vector-based Reconstruction as Long-Context Recall Regardless of Context Window Size
Sep 11, 2026MyungHoon Ryu, XinYu Piao, Jong-Kook KimLLM Memory Long-Context Modeling
Korea University South Korea

Oct 6, 2026J/K move · Enter open · S save

UNREAL: Unifying Retrieval and Long-Context with a Single Model

Edan Kinderman, Elad Hoffer, Yochai Blau, Brian Chmiel+3

NVIDIA · Technion

Retrieval-Augmented Generation Knowledge Augmentation for Language Models LLM Inference Acceleration Information Retrieval Long-Context Inference Long-Context Reasoning in Language Models

Open paper AskPDF arXiv

Long-context inference and Retrieval-Augmented Generation (RAG) handle evidence selection at vastly different scales, from a single long prompt to an entire corpus. We ask whether a single model-internal mechanism can select evidence across this range. We introduce UNifying REtrieval And Long-Context with a Single Model (UNREAL), a model-native evidence selection framework to span corpus retrieval and long-context inference. UNREAL encodes chunks and derives retrieval queries directly from the frozen LLM's internal representations. It adds fewer than 500K trainable parameters and leaves the backbone unchanged. On a 3B-token, 21M-chunk Wikipedia index, all four dense and hybrid UNREAL backbones outperform state-of-the-art retriever-reranker systems. The best model raises recall from 49.1% to 73.2% on HotpotQA, from 31.7% to 60.1% on 2WikiMultiHopQA, and from 8.8% to 14.4% on MuSiQue. Applied to long-context tasks, the same selection mechanism removes distractors before generation, raising NoLiMa accuracy from 1.0% to 24.83% at its maximum context length of 128K tokens, and LV-Eval's F1 score from 49.97% to 54.66% at 256K. UNREAL also reduces FLOPs and time-to-first-token relative to full-context inference from roughly 32K tokens onward, with larger gains as context grows. Together, these results establish model-internal evidence selection as a common foundation for corpus retrieval and evidence-sparse long-context inference.

Abstract

Long-Context Retrieval

Explore similar work

UNREAL: Unifying Retrieval and Long-Context with a Single Model

ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning

Residual Vector-based Reconstruction as Long-Context Recall Regardless of Context Window Size