Which Long Documents Matter? A Controlled Study of Data Selection for Context Extension

Junlong Jia; Jiang Zhou; Xing Wu; Chaochen Gao Tinghao Yu; Feng Zhang; Lilin Wang

cs.CLOct 17, 2026

Which Long Documents Matter? A Controlled Study of Data Selection for Context Extension

Authors: Junlong Jia, Jiang Zhou, Xing Wu, Chaochen Gao Tinghao Yu, Feng Zhang, Lilin Wang

Organizations: Hunyuan Team, Tencent

Abstract

Natural long documents are widely used for context extension, but which selection pipelines are effective remains unclear. We conduct a controlled comparison on Qwen2.5-3B at 128K under matched 6B-token continued-training budgets. Random sampling of natural long documents (random-long) outperforms repacking at longer evaluation lengths. Within the same candidate pool, selecting documents with high LongPPL key-token density yields further gains, unlike the evaluated generic-quality and LLM-judge selectors. We introduce truncLongPPL to reduce scoring cost by computing key-token density over a truncated prefix. At 128K, scoring the first half of the reference span yields downstream performance close to full-span scoring with a measured $2.28\times$ end-to-end scoring speedup. Across three training seeds, truncLongPPL achieves the highest mean scores among evaluated pipelines, exceeding random-long by 4.54 points on validation HELMET and 3.28 on held-out RULER, with gains most consistent in retrieval. At a 256K target context, it gains 2.82 average RULER points over random-long on Qwen2.5-3B. In short/long mixed training across Qwen2.5-3B, Qwen3-8B, and OLMo3-7B, it gains 1.47--7.31 average RULER points over random-long mixing, with seven-task short-context averages within 0.23 points of the corresponding baselines. After supervised fine-tuning (SFT), Qwen3-8B and OLMo3-7B models trained with selected long documents outperform their random-long counterparts by 1.13--1.67 points on LongBench v2 and 0.62--1.44 points on MRCR.

Figures & tables

Study	Backbone	Context	Tokens	Data setting
Controlled comparison	Qwen2.5-3B	128K	6B	Controlled pipelines
Length extension	Qwen2.5-3B	128K / 256K	6B	Natural long
Mixture validation	Qwen2.5-3B	128K	10B	Mixed, 66/34
Cross-model validation	Qwen3-8B	256K	20B	Mixed, 66/34
Cross-model validation	OLMo 3-7B	256K	20B	Mixed, 66/34

Table 1: Overview of experiments and continued-training settings. Mixture ratios are short:long. The dolma3-short controls pack only short documents and use the same total budget as their respective mixed configurations. The short-data source is described in Appendix I .

Figure 1: Validation HELMET (left) and held-out RULER (right) by evaluation length for random sampling of natural long documents and repack under a shared 6B-token budget. At 128K, random sampling is higher by approximately 2.5 and 3.3 points, respectively.

Figure 2: Three-seed comparison of data-selection pipelines at 128K and 6B training tokens. Bars show mean validation HELMET (left) and held-out RULER (right); error bars show one standard deviation across training seeds. Dashed lines mark the random-long means. truncLongPPL uses

r=1/2

prefix scoring at

\tau=0.05

; exact values appear in Appendix K .

Figure 3: Full-span (

r=1

L=128

K) LongPPL key-token-density threshold sweep under the 6B-token budget: macro-averaged validation HELMET (left), held-out RULER (right), and the number of unique documents retained (gray, secondary axes). Validation performance rises through

\tau=0.05

and then remains on a plateau through

0.08

as the retained pool continues to shrink.

Figure 4: Scores by evaluation length for front-prefix fractions

r\in\{1/8,1/4,1/2,1\}

L=128

K at

\tau=0.05

under the 6B-token budget: validation HELMET (left) and held-out RULER (right). At the 128K evaluation length,

r=1/2

and

r=1

are comparable, differing by 0.1 point on HELMET and 0.2 point on RULER.

Figure 5: Validation HELMET category scores by evaluation length for random sampling of natural long documents and truncLongPPL under the 6B-token budget. At 128K, recall is approximately 21 points higher with selection; long-document QA differs by at most 1.1 points across lengths.

Figure 6: Held-out RULER category scores by evaluation length for random sampling of natural long documents and truncLongPPL under the 6B-token budget.

	128K training	256K training
Selection pipeline	RULER	RULER	RULER 256K
random-long	$78.03\pm 0.08$	75.81	62.91
LongFilter	$79.68\pm 0.12$	77.25	65.75
truncLongPPL	$\mathbf{81.31\pm 0.14}$	78.63	67.05

Table 2: Length-extension validation on Qwen2.5-3B with 6B training tokens per configuration. The 128K column reports RULER mean

\pm

standard deviation over three training seeds. The 128K average covers 8K–128K; the 256K columns report the average over 8K–256K and the 256K endpoint. These averages cover different length sets.

Figure 7: Seven-task short-context average and macro-averaged validation HELMET and held-out RULER under a shared 10B-token budget. Bars are means over three independent training seeds and scatter points show the individual seeds. The two 66/34 mixed configurations use the same pool of natural long documents and differ only in random versus truncLongPPL-based selection. Summary statistics over the three training seeds are reported in Appendix J .

Configuration	Short	RULER	RULER 256K
dolma3-short	63.42	83.72	67.39
$+$ random-long	63.42	85.61	73.12
$+$ truncLongPPL	63.46	87.08	77.20

Table 3: Mixed-training validation on Qwen3-8B Base at 256K, with 20B tokens per configuration. Mixed configurations use short:long

=66/34

; dolma3-short uses the same total budget. Short is the seven-task short-context average; RULER is averaged over 8K–256K; RULER 256K is the target-length score. truncLongPPL uses a front-half scoring prefix (128K). Bold marks the highest value among the displayed configurations in each column.

Configuration	Short	RULER	RULER 256K
dolma3-short	62.11	60.60	43.73
$+$ random-long	61.65	65.68	53.06
$+$ truncLongPPL	61.42	72.99	55.94

Table 4: Cross-model validation on OLMo 3-7B Base at 256K and 20B tokens. Mixed configurations use short:long

=66/34

; dolma3-short uses the same total budget. Metric definitions follow Table 3 . Bold marks the highest value among the displayed configurations in each column.

Configuration	Overall	Short	Medium	Long
dolma3-short	33.20	42.93	28.67	25.93
$+$ random-long	33.87	43.33	28.67	28.43
$+$ truncLongPPL	35.00	42.60	30.53	31.20

Table 5: LongBench v2 accuracy (%) after SFT of Qwen3-8B 256K configurations (means over three evaluation seeds,

N=503

; per-cell standard deviations in Appendix M ). Columns are the overall score and the Short/Medium/Long splits. truncLongPPL uses a front-half scoring prefix (128K). Best means in each column are in bold.

Configuration	Overall	2-needle	4-needle	8-needle
dolma3-short	24.59	37.93	19.99	15.87
$+$ random-long	24.99	38.11	20.61	16.26
$+$ truncLongPPL	25.61	39.15	20.78	16.90

Table 6: MRCR accuracy (%) at context lengths up to 256K after SFT of Qwen3-8B (means over three evaluation seeds,

N=1779

; per-cell standard deviations in Appendix M ). Columns are the overall score and the 2-, 4-, and 8-needle subsets. Best means among the displayed configurations in each column are in bold.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Document signal	Selection rule or construction
Scoring-free constructions
repack	None	Pack documents into 128K sequences following the repacking strategy of MAI-Thinking-1; documents may be concatenated.
random-long (random in figures)	None	Uniformly sample documents from $\mathcal{C}$ without document scoring.
Generic-quality selectors
PPL (top40 / mid40)	Document-level perplexity under Gemma-4-26B-A4B (base)	Select scores above the 60th percentile or between the 30th and 70th percentiles, respectively.
gzip (mid60)	gzip compression ratio	Select scores between the 20th and 80th percentiles.

Appendix

Table 7: Definitions of the data-construction pipelines compared in Section 3 , grouped by their selection signals. The first column maps each method to the shorthand used in the result figures. Except for repack , every method operates on the candidate pool

\mathcal{C}

of natural long documents and preserves document boundaries. All percentile thresholds in this 128K comparison are computed globally over

\mathcal{C}

	HELMET		RULER
Training tokens	LongPPL-density	gzip	LongPPL-density	gzip
2B	50.41	49.24	76.67	75.86
4B	50.56	50.36	77.84	77.55
6B	50.40	50.71	77.91	77.45
8B	50.33	50.68	77.93	77.72
10B	50.54	51.05	78.15	78.01

Appendix

Table 8: OLMo-style middle-60% selection comparison in our training setup across budgets of 2B–10B tokens. Columns are grouped into validation HELMET and held-out RULER; within each group, the two subcolumns compare LongPPL-density and gzip selection. The higher value in each (budget, metric) pair is bold.

Figure 8: Raw mean LongPPL key-token density across 8K positional bins for 30,000 natural long documents truncated to 128K tokens. Density increases with position, but the available prefix also grows; the curve is therefore descriptive rather than evidence of a position-only effect.

Figure 9: Validation HELMET (left) and held-out RULER (right) by continued-training budget for front-half (

r=1/2

) LongPPL-density selection at

L=128

K. Low thresholds remain lower, while

\tau\in\{0.03,0.04,0.05,0.06\}

forms a higher cluster from approximately 6B tokens onward.

Figure 10: Macro-averaged validation HELMET (left) and held-out RULER (right) for scoring-window layouts at

\tau=0.05

under the 6B-token budget. Dashed lines mark random sampling of natural long documents. All layouts exceed this baseline. The front

\cap

tail label denotes the intersection of the document sets selected by the front-half and tail-half windows.

Threshold $\tau$	Scattered $8\times L/16$	Contiguous front-half
0.03	55.01 / 80.55	56.03 / 80.87
0.04	56.23 / 80.97	55.79 / 81.01
0.05	56.24 / 81.32	56.23 / 81.23

Appendix

Table 9: Scattered (

8\times L/16

, randomly located) versus contiguous (front-half) scoring at shared absolute thresholds,

L=128

K and 6B tokens. Both layouts have a nominal span of

L/2

; densities are normalized by their valid target counts as defined in Appendix D . Cells give validation HELMET / held-out RULER averaged over evaluation lengths.

Figure 11: Scores by evaluation length for prefix fractions

r\in\{1/8,1/4,1/2,1\}

(

L=128

\tau=0.03

, 6B training tokens). At 128K evaluation,

r=1/8

scores lower, while

r=1/2

and

r=1

remain close (Figure 4 ).

	128K training	256K training
Scoring configuration	RULER	RULER	RULER 256K
truncLongPPL ( $r=1/4$ )	$80.07^{\dagger}$	78.01	66.41
truncLongPPL ( $r=1/2$ , default)	$\mathbf{81.31\pm 0.14}$	78.63	67.05

Appendix

Table 10: Scoring-window comparison on Qwen2.5-3B with 6B training tokens per configuration. The 128K average covers 8K–128K, and the 256K average covers 8K–256K. In the 128K column,

\dagger

denotes the reference-run result from the

\tau=0.05

window ablation in Figure 4 ; the default configuration reports mean

\pm

standard deviation over three training seeds. The 256K columns retain the reported point estimates. Bold marks the higher score in each column.

Figure 12: Domain-mixture control for truncLongPPL at

L=128

K under the 6B-token budget. Resampling restores the candidate pool’s coarse domain proportions and changes validation HELMET (left) by at most 0.3 point and held-out RULER (right) by at most 0.2 point.

Figure 13: Seven-task short-context averages before and after 6B-token continued training. The base model scores 60.1; continued models score 59.0–59.5, including 59.2 for truncLongPPL.

Configuration	ARC-C	ARC-E	HellaSwag	LAMBADA	LogiQA	PIQA	WinoG.	Avg.
Base (no CT)	45.05	77.36	55.00	66.89	29.95	78.24	68.35	60.12
dolma3-short	48.21	79.17	54.27	66.31	26.27	78.18	68.19	60.09
$+$ random-long	47.01	78.58	54.17	65.63	27.34	78.02	68.19	59.85
$+$ truncLongPPL	46.84	78.45	54.10	65.26	27.34	78.13	68.43	59.79

Appendix

Table 11: Per-task short-context accuracy (%) for the uncontinued base model ( Base , no long-context extension) and the three fixed-mixture configurations in Section 7.1 under the shared 10B-token budget: dolma3-short, and the 66/34 mixtures adding random versus truncLongPPL-selected long documents. Values are for one representative training seed; Table 12 reports averages over three training seeds for the final column. Columns are the seven lm-evaluation-harness tasks and their unweighted average. The highest value in each column is bold.

Configuration	Short	HELMET	RULER
dolma3-short	60.04 $\pm$ 0.11	49.44 $\pm$ 0.12	77.16 $\pm$ 0.15
$+$ random-long	59.81 $\pm$ 0.04	50.72 $\pm$ 0.10	78.61 $\pm$ 0.15
$+$ truncLongPPL	59.79 $\pm$ 0.06	55.31 $\pm$ 0.27	81.94 $\pm$ 0.16

Appendix

Table 12: Seed stability of the three fixed-mixture configurations of Section 7.1 at the shared 10B-token budget, reported as mean

\pm

standard deviation over three independent training seeds. Short is the seven-task short-context average; HELMET is the macro-averaged validation score; RULER is the macro-averaged held-out score. The highest value in each column is in bold.

Pipeline	HELMET	RULER
truncLongPPL	$\mathbf{55.59\pm 0.55}$	$\mathbf{81.31\pm 0.14}$
LongFilter	$53.37\pm 0.30$	$79.68\pm 0.12$
Prospector	$51.98\pm 0.28$	$78.41\pm 0.05$
random-long	$51.05\pm 0.56$	$78.03\pm 0.08$
repack	$50.31\pm 0.45$	$76.81\pm 0.28$
gzip (mid60)	$50.08\pm 0.79$	$77.68\pm 0.26$

Appendix

Table 13: Mean

\pm

standard deviation over three training seeds for the nine pipelines on Qwen2.5-3B, 128K, 6B tokens.

	HELMET		RULER		Short
Short:long	Random	Selected	Random	Selected	Random	Selected
66/34	50.70	55.30	78.60	81.90	59.80	59.80
50/50	51.58	55.42	79.02	81.70	59.74	59.86
34/66	49.98	54.35	78.18	81.36	59.30	59.31

Appendix

Table 14: Mixture-ratio sensitivity at 128K and 10B tokens on Qwen2.5-3B. Each pair compares random-long and truncLongPPL-selected long documents. Short is the seven-task short-context average.

Metric	dolma3-short	$+$ random-long	$+$ truncLongPPL
Overall	33.20 $\pm$ 0.40	33.87 $\pm$ 0.31	35.00 $\pm$ 0.20
Easy	34.03 $\pm$ 0.64	34.73 $\pm$ 0.76	35.93 $\pm$ 1.38
Hard	32.70 $\pm$ 0.35	33.33 $\pm$ 0.50	34.40 $\pm$ 0.60
Short	42.93 $\pm$ 0.64	43.33 $\pm$ 0.55	42.60 $\pm$ 0.82
Medium	28.67 $\pm$ 1.21	28.67 $\pm$ 1.10	30.53 $\pm$ 0.58
Long	25.93 $\pm$ 1.62	28.43 $\pm$ 1.93	31.20 $\pm$ 3.26

Appendix

Table 15: Full LongBench v2 results (%) for the Qwen3-8B SFT configurations: mean

\pm

standard deviation over three evaluation seeds. Best means in each row are in bold.

Metric	dolma3-short	$+$ random-long	$+$ truncLongPPL
Overall	24.59 $\pm$ 0.30	24.99 $\pm$ 0.33	25.61 $\pm$ 0.29
2-needle	37.93 $\pm$ 0.43	38.11 $\pm$ 0.77	39.15 $\pm$ 0.51
4-needle	19.99 $\pm$ 0.60	20.61 $\pm$ 0.24	20.78 $\pm$ 0.42
8-needle	15.87 $\pm$ 0.21	16.26 $\pm$ 0.39	16.90 $\pm$ 0.21

Appendix

Table 16: MRCR overall and needle-subset results (%) for the Qwen3-8B SFT configurations: mean

\pm

standard deviation over three evaluation seeds. Best means in each row are in bold.

Metric	dolma3-short	$+$ random-long	$+$ truncLongPPL
Overall	27.27 $\pm$ 0.42	27.67 $\pm$ 0.42	29.33 $\pm$ 0.31
Easy	27.93 $\pm$ 2.12	29.13 $\pm$ 1.38	31.40 $\pm$ 1.21
Hard	26.90 $\pm$ 1.23	26.80 $\pm$ 0.85	28.10 $\pm$ 0.46
Short	32.03 $\pm$ 1.96	33.87 $\pm$ 0.55	34.07 $\pm$ 0.29
Medium	25.73 $\pm$ 1.60	25.27 $\pm$ 0.29	27.60 $\pm$ 0.52
Long	22.53 $\pm$ 1.91	22.20 $\pm$ 0.90	25.00 $\pm$ 0.90

Appendix

Table 17: LongBench v2 accuracy (%) after SFT of OLMo 3-7B: mean

\pm

standard deviation over three evaluation seeds (

N=503

). Best means in each row are in bold.

Metric	dolma3-short	$+$ random-long	$+$ truncLongPPL
Overall	13.02 $\pm$ 0.18	13.73 $\pm$ 0.27	15.16 $\pm$ 0.26
2-needle	17.99 $\pm$ 0.42	18.72 $\pm$ 1.13	22.03 $\pm$ 0.83
4-needle	11.55 $\pm$ 0.34	11.93 $\pm$ 0.29	12.68 $\pm$ 0.34
8-needle	9.54 $\pm$ 0.31	10.54 $\pm$ 0.47	10.79 $\pm$ 0.58

Appendix

Table 18: MRCR overall and needle-subset accuracy (%) up to 256K after SFT of OLMo 3-7B: mean

\pm

standard deviation over three evaluation seeds (

N=1780

). Best means in each row are in bold.

Training Data Selection

Training data selection studies how to choose, filter, weight, or schedule examples to improve learning while reducing data and compute costs.

See all 187 papers

Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context
May 13, 2026Zhaowei Wang, Lishu Luo, Haodong Duan +9Visual Question Answering Vision-Language Models
CSE Department, HKUST · ByteDance Seed
Highlight-Then-Summarize: Learning to Compress Evidence for Long-Context Understanding
Sep 25, 2026Zhaoyuan Xia, Qinghongbing Xie, Yung Xiang Hue +7Long-Context Reasoning in Language Models Long-Context Modeling
Peking University · Baidu Inc · Baidu Inc. +1
Rethinking Retrieval in the Long-Context Era: Training-Free Semantic Trees for Adaptive Context Construction
Oct 25, 2026Zhe Xu, Tianhao Liao, Zhenqing Ling +3Long-Context Retrieval Information Retrieval
Sun Yat-sen University · Alibaba Group

May 13, 2026J/K move · Enter open · S save

Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context

Zhaowei Wang, Lishu Luo, Haodong Duan, Weiwei Liu+8

CSE Department, HKUST · ByteDance Seed

Visual Question Answering Vision-Language Models Long-Document VQA Large Vision-Language Models Long-Context Modeling Vision-Language Pretraining+1

Open paper AskPDF arXiv

Long-context modeling is becoming a core capability of modern large vision-language models (LVLMs), enabling sustained context management across long-document understanding, video analysis, and multi-turn tool use in long-horizon agents. Yet practical training recipes remain insufficiently explored, particularly for designing and balancing long-context data mixtures. In this work, we present a systematic study of long-context continued pre-training (LongPT) for LVLMs, extending a 7B model from 32K to 128K context with extensive ablations on long-document data. We first show that long-document VQA is substantially more effective than OCR transcription. Building on this observation, our ablations further yield three key findings: i) for sequence-length distribution, balanced data outperforms target-length-focused data (e.g., 128K), suggesting that long-context ability requires generalizable key-information retrieval across various lengths and positions; ii) retrieval remains the primary bottleneck, favoring retrieval-heavy mixtures with modest reasoning data for task diversity; iii) pure long-document VQA largely preserves short-context capabilities, suggesting that instruction-formatted long data reduces the need for short-data mixing. Based on these findings, we introduce MMProLong, obtained by long-context continued pre-training from Qwen2.5-VL-7B with only a 5B-token budget. MMProLong improves long-document VQA scores by 7.11% and maintains strong performance at 256K and 512K contexts beyond its 128K training window, without additional training. It further generalizes to webpage-based multimodal needle retrieval, long-context vision-text compression, and long-video understanding without task-specific supervision. Overall, our study establishes a practical LongPT recipe and an empirical foundation for advancing long-context vision-language models.

Abstract

Training Data Selection

Explore similar work

Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context

Highlight-Then-Summarize: Learning to Compress Evidence for Long-Context Understanding

Rethinking Retrieval in the Long-Context Era: Training-Free Semantic Trees for Adaptive Context Construction