PolicyLong: Towards On-Policy Context Extension

Junlong Jia; Jiang Zhou; Ziyang Chen; Xing Wu; Chaochen Gao; TingHao Yu; Feng Zhang; Songlin Hu

cs.LGApr 9, 2026

PolicyLong: Towards On-Policy Context Extension

Authors: Junlong Jia, Jiang Zhou, Ziyang Chen, Xing Wu, Chaochen Gao, TingHao Yu, Feng Zhang, Songlin Hu

Organizations: Hunyuan Team, Tencent · Chinese Academy of Sciences

Abstract

Extending LLM context windows is hindered by scarce high-quality long-context data. Recent methods synthesize data with genuine long-range dependencies via information-theoretic verification, selecting contexts that reduce a base model's predictive entropy. However, their single-pass offline construction with a fixed model creates a fundamental off-policy gap: the static screening landscape misaligns with the model's evolving capabilities, causing the training distribution to drift. We propose PolicyLong, shifting data construction towards a dynamic on-policy paradigm. By iteratively re-executing data screening (entropy computation, retrieval, and verification) using the current model, PolicyLong ensures the training distribution tracks evolving capabilities, yielding an emergent self-curriculum. Crucially, both positive and hard negative contexts derive from the current model's entropy landscape, co-evolving what the model learns to exploit and resist. Experiments on RULER, HELMET, and LongBench-v2 (Qwen2.5-3B) show PolicyLong consistently outperforms EntropyLong and NExtLong, with gains growing at longer contexts (e.g., +2.54 at 128K on RULER), confirming the value of on-policy data evolution.

Figures & tables

Figure 1: Overview of PolicyLong. Top: iterative on-policy training across stages, where the current model $M_{t}$ re-executes data screening and progressively shifts training toward harder dependencies, yielding an implicit self-curriculum. Bottom: per-stage data construction, including entropy computation on root document $A$ , candidate retrieval, entropy-based positive verification, on-policy hard negative construction, and final sequence assembly.

Panel of Figure 2 — Figure 2: Empirical evidence for the off-policy gap. (a) The entropy distribution of positions initially identified as high-entropy by the base model shifts significantly across stages, demonstrating progressive data–model misalignment. (b) Target-word loss with the positive context decreases rapidly, indicating that base-model-identified dependencies become trivially easy for the trained model.

	RULER			HELMET			LongBench-v2
Model	64K	128K	Avg	64K	128K	Avg	Med.	Long	Avg
Qwen2.5-3B	65.36	63.22	64.29	40.53	38.99	39.76	21.5	20.6	21.1
NExtLong	70.17	61.71	65.94	45.35	41.24	43.30	23.7	26.9	25.3
EntropyLong	69.86	63.45	66.66	43.88	40.08	41.98	24.2	28.7	26.5
PolicyLong	71.07	65.99	68.53	46.51	42.57	44.54	27.1	31.5	29.3

Table 1: Long-context benchmark results (

>

32K) on Qwen2.5-3B. PolicyLong consistently outperforms all baselines, with the advantage widening at longer contexts (e.g., +2.54 over EntropyLong at 128K on RULER). RULER and HELMET are evaluated before SFT; LongBench-v2 is evaluated after SFT. LongBench-v2 Med.: 32K–128K input; Long: 128K–2M input.

	RULER				HELMET				LB-v2
Model	8K	16K	32K	Avg	8K	16K	32K	Avg	Short
Qwen2.5-3B	86.43	82.24	76.82	81.83	58.30	55.85	51.36	55.17	35.8
NExtLong	85.82	80.73	75.90	80.82	57.52	56.00	51.19	54.90	40.6
EntropyLong	85.20	80.49	75.67	80.45	58.11	54.67	48.83	53.87	40.0
PolicyLong	86.05	82.00	78.23	82.09	59.34	56.64	52.57	56.18	40.6

Table 2: Performance at shorter context lengths (

\leq

32K) on Qwen2.5-3B. PolicyLong achieves the best overall short-context performance among all synthesis methods, confirming that on-policy long-context training does not compromise short-context capability. RULER and HELMET are evaluated before SFT; LongBench-v2 Short is evaluated after SFT.

	RULER			HELMET			LongBench-v2
Model	64K	128K	Avg	64K	128K	Avg	Med.	Long	Avg
Longmino †	71.4	65.4	68.40	54.6	48.1	51.35	25.27	24.07	24.67
NExtLong	73.7	69.5	71.60	54.9	51.0	52.95	25.74	23.77	24.76
EntropyLong	71.4	68.7	70.05	56.0	52.2	54.10	28.68	25.00	26.84
PolicyLong	73.3	72.7	73.00	57.1	54.4	55.75	29.46	29.32	29.39

Table 3: Cross-family and cross-scale validation on OLMo3-7B (

>

32K), trained under a production-scale 50B-token budget. PolicyLong achieves the best long-context average on RULER, HELMET, and LongBench-v2, with its strongest gains at the longest context settings. RULER and HELMET are evaluated before SFT; LongBench-v2 is evaluated after SFT. Avg denotes the long-context average: (64K

+

128K)/2 for RULER/HELMET and (Med.

+

Long)/2 for LongBench-v2. † Longmino reuses OLMo3’s official long-context stage-extension data, retrained under the same pipeline.

Configuration	Iter.	HN	Curr.	64K	128K	Avg
EntropyLong	$\times$	$\times$	$\times$	69.86	63.45	66.66
• Iteration	$\checkmark$	$\times$	$\times$	70.66	63.62	67.14
• Hard Negatives	$\times$	$\checkmark$	$\times$	70.27	64.66	67.46
• Curriculum	$\times$	$\times$	$\checkmark$	70.01	63.75	66.88
• Iter. + Curriculum	$\checkmark$	$\times$	$\checkmark$	70.13	64.11	67.12
• Iter. + HN	$\checkmark$	$\checkmark$	$\times$	69.95	65.16	67.56

Table 4: Ablation on RULER at long contexts (

>

32K) for Qwen2.5-3B. Iter. = Iteration; EL = EntropyLong.

\checkmark

\times

denotes enabled/disabled.

Configuration	Iter.	HN	Curr.	8K	16K	32K	Avg
EntropyLong	$\times$	$\times$	$\times$	85.20	80.49	75.67	80.45
• Iteration	$\checkmark$	$\times$	$\times$	85.74	80.65	75.92	80.77
• Hard Negatives	$\times$	$\checkmark$	$\times$	85.30	81.44	76.49	81.08
• Curriculum	$\times$	$\times$	$\checkmark$	85.13	81.04	75.71	80.63
• Iter. + Curriculum	$\checkmark$	$\times$	$\checkmark$	85.61	81.23	76.51	81.12
• Iter. + HN	$\checkmark$	$\checkmark$	$\times$	85.81	81.68	76.51	81.33

Table 5: Ablation on RULER at short contexts (

\leq

32K) for Qwen2.5-3B. Iter. = Iteration; EL = EntropyLong.

\checkmark

\times

denotes enabled/disabled.

Panel of Figure 3 — Figure 3: Data difficulty progression. We measure the loss reduction from the base model to the progressively trained model on data generated at Stage 2 and Stage 3. PolicyLong yields a significantly larger loss gap than EntropyLong, indicating that it successfully surfaces harder, non-trivial dependencies.

Model	ARC-C	ARC-E	HellaSwag	PIQA	LogiQA	WinoGrande	Avg
Qwen2.5-3B	45.04	77.36	55.00	78.24	29.95	68.35	58.99
PolicyLong	45.90	78.45	53.97	78.02	29.95	67.96	59.04

Table 6: General short-text benchmark results on Qwen2.5-3B. We compare the base model and the full PolicyLong model to verify that long-context extension does not degrade general short-text capabilities.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Target Length	Model	8K	16K	32K	64K	128K	Avg
64K	EntropyLong	83.71	79.57	74.56	68.37	59.69	73.18
64K	PolicyLong	85.26	80.82	76.58	70.14	61.66	74.89
128K	EntropyLong	85.20	80.49	75.67	69.86	63.45	74.93
128K	PolicyLong	86.05	82.00	78.23	71.07	65.99	76.67
256K	EntropyLong	85.81	81.48	76.95	71.66	65.87	76.35
256K	PolicyLong	86.24	82.24	78.34	72.56	66.81	77.24

Appendix

Table 7: RULER benchmark results on Qwen2.5-3B across different training target context lengths. All results are at checkpoint-500.

Figure 4: Needle-in-a-Haystack evaluation results for PolicyLong. The heatmap shows retrieval accuracy across varying context lengths (x-axis) and needle depth positions (y-axis).

Panel of Figure 5 — Figure 5: RULER detailed analysis. (a) Subtask breakdown at 128K: PolicyLong achieves the largest gains on NIAH Multi-Key (+5.7) and Variable Tracking (+4.0). (b) Gain (PolicyLong $-$ EntropyLong) scaling with context length on the two most improved subtasks.

Method	seed 42	seed 43	seed 44	mean	std
EntropyLong	74.93	75.15	74.84	74.97	0.16
PolicyLong	76.67	76.60	76.65	76.64	0.04
$\Delta$ (PolicyLong $-$ EntropyLong)	+1.74	+1.45	+1.81	+1.67	–

Appendix

Table 8: Multi-seed RULER average (8K–128K) on Qwen2.5-3B. Each method is trained with three independent seeds. The per-method standard deviation (

\leq

0.16) is far smaller than the mean improvement of PolicyLong over EntropyLong (+1.67).

$T$ (stages)	RULER Avg	$\Delta$ vs $T=1$
1 (EntropyLong)	74.93	–
2	76.11	+1.18
3	76.55	+1.62
4 (main config)	76.67	+1.74
5	76.45	+1.52

Appendix

Table 9: Effect of the number of on-policy refresh stages

T

(RULER 8K–128K average, Qwen2.5-3B), at a fixed total token budget.

T=1

corresponds to the off-policy EntropyLong baseline.

Hard-negative source	RULER Avg	$\Delta$ vs PolicyLong
Random chunks (no secondary retrieval)	73.77	$-2.90$
Per-query random (higher diversity)	74.20	$-2.47$
Bottom- $k$ retrieval (least similar)	75.48	$-1.19$
BM25, fixed (off-policy)	74.01	$-2.66$
BM25, on-policy refresh	74.46	$-2.21$
PolicyLong (semantic-near, on-policy)	76.67	–

Appendix

Table 10: Hard-negative design choices on Qwen2.5-3B (RULER 8K–128K average). Only the hard-negative source is varied; all other components match the full PolicyLong pipeline. The BM25 pair differs only in fixed (off-policy) vs. on-policy refresh.

Long-Context Language Modeling

Long-context language modeling studies how language models can use, retain, and efficiently process information across very long sequences.

See all 189 papers

OPSDL: On-Policy Self-Distillation for Long-Context Language Models
Apr 19, 2026Xinsen Zhang, Zhenkai Ding, Tianjun Pan +4Long-Context Language Modeling On-Policy Self-Distillation
Baidu Inc.
EndPrompt: Efficient Long-Context Extension via Terminal Anchoring
May 14, 2026Han Tian, Luxuan Chen, Xinran Chen +10Long-Context Language Modeling LLM Fine-Tuning
1Nankai University · 2Baidu Inc. · 3Shanghai Jiao Tong University +1
Beyond Skill Evolution: Self-Evolving Context Management Policies for Long-Horizon Agent Harnesses
Sep 28, 2026Weiyuan Li, Jinghan Xu, Aili Chen +4Agent Harness Optimization Long-Horizon Agent Tasks
School of Data Science, Fudan University · Shanghai Key Laboratory of Data Science · College of Computer Science and Artificial Intelligence, Fudan University

Apr 19, 2026J/K move · Enter open · S save

OPSDL: On-Policy Self-Distillation for Long-Context Language Models

Xinsen Zhang, Zhenkai Ding, Tianjun Pan, Run Yang+3

Baidu Inc.

Long-Context Language Modeling On-Policy Self-Distillation Language Model Distillation Language Model Post-Training

Open paper AskPDF arXiv

Extending the effective context length of large language models (LLMs) remains a central challenge for real-world applications. While recent post-training methods have made progress in long-context scaling, they either rely on high-quality supervision data or sparse sequence-level rewards, leading to unstable and inefficient optimization. We propose OPSDL, an On-Policy Self-Distillation method for enhancing the Long-context capabilities of LLMs. Unlike other recent self-distillation methods that inject privileged information and rely on the model's in-context learning ability to act as a teacher, OPSDL leverages the model's own inherently strong short-context capability as a self-teacher to supervise its own generation in long-context scenarios. The model first generates responses conditioned on the full long-context, then the self-teacher provides per-token supervision signals via point-wise reverse KL divergence under the relevant extracted short-context. This dense token-level signal encourages faithful use of relevant evidence and mitigates hallucinations induced by irrelevant context. We evaluate OPSDL on long-context benchmarks across a range of models from 7B to 32B parameters. Results show consistent and substantial improvements across varying context lengths, outperforming standard post-training approaches such as SFT and DPO with higher sample efficiency. Notably, these gains are achieved without degrading general short-context performance. These findings highlight the effectiveness of OPSDL as a scalable and stable approach for long-context learning.

Abstract

Long-Context Language Modeling

Explore similar work

OPSDL: On-Policy Self-Distillation for Long-Context Language Models

EndPrompt: Efficient Long-Context Extension via Terminal Anchoring

Beyond Skill Evolution: Self-Evolving Context Management Policies for Long-Horizon Agent Harnesses