Highlight-Then-Summarize: Learning to Compress Evidence for Long-Context Understanding

Zhaoyuan Xia; Qinghongbing Xie; Yung Xiang Hue; Jianguang Jiang; Gaofeng Lu; Zhenyu Jiao; Xing Yuan; Dai Dai

cs.CLSep 25, 2026

Highlight-Then-Summarize: Learning to Compress Evidence for Long-Context Understanding

Authors: Zhaoyuan Xia, Qinghongbing Xie, Yung Xiang Hue, Jianguang Jiang, Gaofeng Lu, Zhenyu Jiao, Xing Yuan, Dai Dai,

Organizations: Peking University · Baidu Inc · Baidu Inc. · Tsinghua University

Abstract

Long-context understanding requires large language models (LLMs) to reason over lengthy documents, conversations, and code, yet task-relevant evidence is often sparse and scattered amid substantial irrelevant and redundant content. We propose Highlight-Then-Summarize (H2S), a compress-then-reason paradigm that first identifies source-grounded, question-relevant evidence and then integrates it into a compact, question-conditioned summary before producing the final answer. To train this behavior, we construct H2S-Dataset, comprising 6,647 examples from 11 benchmark families with an average context length of 43.9K tokens, and introduce H2S-RL, which provides process-level rewards for evidence selection and summary construction in addition to final-answer correctness. We evaluate on H2S-Bench, a seven-task long-context suite. Under a shared 128K input and 4K output budget, H2S-14B achieves an average score of 32.60, outperforming Qwen3.8-27B by 10.17 points and obtaining the strongest overall result among the evaluated open-source models. H2S-14B also achieves the highest Evidence-Summary Quality score and retains 97.1% of its 16K-budget performance with only a 4K output budget. These results show that explicitly selecting and integrating evidence improves long-context reasoning while enabling more compact generation.

Figures & tables

Figure 1: Highlight-Then-Summarize. H2S transforms a block-structured long document into source-addressable evidence and then a question-conditioned summary before producing the final answer in one autoregressive trajectory.

Input to Qwen3.5-Flash	Context tokens	Acc.
LongDoc	15.19K	72.6
Claude Summary	0.46K	93.0
LongDoc + Claude Summary	15.19K + 0.46K	96.7
LongDoc + Self-Summary (S)	15.19K + 0.14K	69.3
LongDoc + Self-Summary (M)	15.19K + 0.60K	77.8
LongDoc + Self-Summary (L)	15.19K + 1.04K	85.9

Table 1: Even a few summary tokens can substantially improve long-document QA.

Panel of Figure 2 — Figure 2: Composition of H2S-Dataset. H2S-Dataset spans diverse long-context tasks and input lengths, with reasoning and precise retrieval accounting for 24.4% and 23.7% of the dataset, respectively, while examples cover input lengths from 0–32K to 64–128K+ tokens.

Figure 3: Dataset Construction of Highlight-Then-Summarize. Each training instance is constructed through semantic blocking, candidate retrieval, query-guided evidence extraction, question-conditioned summarization, and quality verification, with failed instances iteratively reconstructed until the required criteria are satisfied.

Category	Model	DocFinQA	Frames	LongCite	MRCR	AA-LCR	LongBenchV2	HELMET-Summ	Average
Closed-source	Claude Opus 5	56.11	27.89	12.17	100.00	26.45	64.18	15.13	43.13
	GPT-5.5	49.57	36.05	12.62	92.38	52.34	46.15	9.45	42.65
	DeepSeek-V4-Pro	34.83	37.41	6.78	72.14	47.77	48.56	12.48	37.14
	Kimi-K2.5	51.14	37.41	10.83	87.58	39.95	50.00	14.41	41.62
	GLM-5.2	49.57	36.05	12.62	92.38	40.85	48.80	16.52	42.40
Open-source	DeepSeek-R1-Distill-Qwen-7B	0.06	8.50	1.31	0.40	0.06	1.92	5.29	2.51

Table 2: Main results on seven long-context benchmarks. All models use a maximum input length of 128K and a maximum output length of 4K. Closed-source models are included for reference; among all open-source models, including our models, the best result in each column is shown in bold , and the second-best result is shown in bold and underlined .

Figure 4: Analysis of Summarization Quality. Final-answer score versus Evidence–Summary Quality (ESQ). Higher is better on both axes.

Model	DocFinQA	Frames	LongCite	MRCR	AA-LCR	LongBenchV2	HELMET-Summ	Avg
Baseline-7B	4.82	21.43	3.92	34.80	10.71	29.09	2.65	15.35
w/o evidence-7B	23.56	30.61	9.97	54.60	14.60	31.01	7.52	24.55
w/o summary-7B	13.64	29.93	4.23	42.40	10.98	21.63	7.81	18.66
H2S-7B	26.33	32.97	15.76	59.72	21.47	32.11	11.58	28.56
Baseline-14B	25.52	29.59	4.06	39.00	13.19	31.73	12.89	22.28
w/o evidence-14B	25.84	37.41	10.64	59.80	17.34	35.82	11.60	28.35

Table 3: Detailed ablation of evidence and summary generation. Baseline removes both evidence and summary generation; w/o evidence and w/o summary remove the corresponding intermediate component, respectively. Avg is the unweighted macro-average over the seven tasks.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A7: Benchmark and input-length distributions of the final training data. The left panel merges different task configurations from the same benchmark; LongBench-Pro contains T1–T11. The right panel shows the within-split length proportions for SFT and RL. The two splits have similar task-source and length distributions.

Category	Model	Final Answer $\uparrow$	E-Recall $\uparrow$	S-RL $\uparrow$	ESQ $\uparrow$
Closed-source	Claude Opus 5	43.13	56.31	23.93	25.45
	GPT-5.5	42.65	41.10	28.61	21.87
	DeepSeek-V4-Pro	37.14	45.47	33.20	27.18
	Kimi-K2.5	41.62	48.96	32.51	28.41
	GLM-5.2	42.40	51.75	34.88	30.50
Open-source	Qwen3-4B-Thinking-2507	14.59	14.94	11.70	6.44

Appendix

Table 4: Final-answer and evidence–summary quality. Final Answer is the seven-task Avg from Table 2 . E-Recall is the macro-average of per-example recall over reference evidence blocks; predictions are the 8-character block IDs parsed from <evidence> . Construction-time IDs are aligned to equivalent IDs in the actual inference blockization by normalized character matching before ID-only scoring. S-RL is ROUGE-L between the parsed <summary> and the reference summary. ESQ is the macro-average of the per-example harmonic mean of E-Recall and S-RL. Missing or unparsable components receive zero.

Model	Budget	DocFinQA	Frames	LongCite	MRCR	AA-LCR	LongBenchV2	HELMET-Summ	Avg	Complete
Qwen3-4B-Thinking-2507	4K	14.61	18.71	3.68	25.45	9.88	18.99	10.80	14.59	53.75%
	8K	24.30	25.51	4.82	47.49	12.31	31.73	13.73	22.84	86.60%
	16K	26.82	29.93	4.92	56.11	15.16	36.34	14.18	26.21	95.15%
Qwen3.8-27B	4K	40.52	32.31	4.94	46.89	15.69	15.14	1.51	22.43	41.17%
	8K	50.12	39.46	7.22	95.39	28.26	38.46	4.00	37.56	68.78%
	16K	51.38	42.18	10.25	97.60	42.14	54.33	8.16	43.72	86.60%

Appendix

Table 5: Detailed performance under different output budgets. Avg denotes the unweighted macro-average across the seven tasks, and Complete denotes the percentage of completed generations.

Answer model	Answer-time input	Context tokens	0–20K	20–40K	$>$ 40K	Avg	$\Delta$ Avg
Claude Opus 4.6	LongDoc	15.19K	98.9	89.9	78.3	89.0	–
Claude Opus 4.6	Claude Summary	0.46K	100.0	96.6	95.7	97.4	+8.4
Claude Opus 4.6	LongDoc + Claude Summary	15.19K + 0.46K	100.0	96.6	87.0	94.5	+5.5
Qwen3.5-Flash	LongDoc	15.19K	77.3	75.3	65.2	72.6	–
Qwen3.5-Flash	Claude Summary	0.46K	97.7	94.4	87.0	93.0	+20.4
Qwen3.5-Flash	LongDoc + Claude Summary	15.19K + 0.46K	97.7	96.6	95.7	96.7	+24.1

Appendix

Table 6: Full results for question-conditioned summaries on long-document QA. Accuracy is reported by document-length range and as their unweighted average; context tokens report the mean document and summary lengths, and

\Delta

Avg is measured against LongDoc for the same answer model.

Stage	Statistic	Unit	SFT	RL	Total
Blocks Retrieval	Raw candidates	case	5,300	3,000	8,300
	Segmentation	block	1,127,610	657,020	1,784,630
	Ranked retrieval	block	193,031	109,550	302,581
Claims Summaries	Subquery decompose	subquery	17,712	10,232	27,944
	claim extraction	claim	22,107	12,927	35,034
	Canonical claims	claim	21,211	12,325	33,536

Appendix

Table 7: Construction statistics for the SFT and RL data. A case is an independent question–context instance; a block is the unit used for document segmentation and retrieval; a subquery is a decomposed information need; a claim is an independently verifiable atomic fact; and a citation links a claim to its source text. Quality-control counts follow their evaluated objects: faithfulness counts citations that exactly match the source, coverage counts subqueries supported by cited evidence, and answerability counts cases that can be solved from the summary alone. An additional 7,436 citations pass normalized fuzzy-0.95 matching and are reported in the appendix.

Long-Context Reasoning in Language Models

See all 41 papers

CoEM: Empowering Long-Context Reasoning with Commit-on-Evidence Memory
Sep 29, 2026Jingguang Li, Yebo Wu, Zuyi Guo +7Long-Context Reasoning in Language Models Memory-Augmented Language Models
Korea Advanced Institute of Science & Technology · University of Macau · Massachusetts Institute of Technology
ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning
Jul 2, 2026Yanjun Zhao, Ruizhong Qiu, Tianxin Wei +6Long-Context Inference Long-Context Reasoning in Language Models
University of Illinois Urbana-Champaign
Learning Evidence Highlighting for Frozen LLMs
Apr 24, 2026Shaoang Li, Yanhang Shi, Yufei Li +10Long-Context QA LLM Prompting
Stony Brook University · Meta AI

Sep 29, 2026J/K move · Enter open · S save

CoEM: Empowering Long-Context Reasoning with Commit-on-Evidence Memory

Jingguang Li, Yebo Wu, Zuyi Guo, Kailang Ma+6

Korea Advanced Institute of Science & Technology · University of Macau · Massachusetts Institute of Technology

Long-Context Reasoning in Language Models Memory-Augmented Language Models Persistent Memory for Language Models LLM Context Management

Open paper AskPDF arXiv

Long-context reasoning is essential for complex and long-horizon tasks, yet the performance of large language models (LLMs) degrades as context length increases. Recent approaches address this by processing input chunk by chunk while maintaining a bounded textual memory in model context. However, premature information compression can discard critical details essential for subsequent reasoning. In this paper, we introduce Commit-on-Evidence Memory (CoEM), which learns when to convert source evidence into compact memory facts. Specifically, under a fixed context-memory budget, CoEM preserves potentially useful source excerpts verbatim in a pending set, allowing subsequent context to clarify their relevance before irreversible compression. As new context arrives, a learned policy revisits each pending excerpt and decides whether to promote it to the committed memory, retain it for further consideration, or discard it. A frozen verifier ensures proposed facts are accepted only if supported by retained excerpts and current context. To further guide effective memory management, we train this policy using reinforcement learning by combining fine-grained, step-level evidence rewards with final answer rewards. Extensive experiments demonstrate that CoEM consistently improves long-context reasoning. When evaluated on 6,400 documents long-context input, CoEM outperforms the strongest memory baseline by 10.4-11.4 F1 points on Qwen3.5-9B. Code repository: https://github.com/benmagnifico/CoEM.

Abstract

Long-Context Reasoning in Language Models

Explore similar work

CoEM: Empowering Long-Context Reasoning with Commit-on-Evidence Memory

ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning

Learning Evidence Highlighting for Frozen LLMs