LongForge: Measuring and Improving Long-Context Instruction Data via Evidence-Assertion Reasoning Graphs

Ziyang Chen; Junlong Jia; Xing Wu; Tinghao Yu; Songlin Hu

cs.AIOct 16, 2026

LongForge: Measuring and Improving Long-Context Instruction Data via Evidence-Assertion Reasoning Graphs

Authors: Ziyang Chen, Junlong Jia, Xing Wu, Tinghao Yu, Songlin Hu

Abstract

Existing quality control for long-context instruction-tuning data is a one-shot judgment inside the generation pipeline that leaves only a scalar, revealing neither which document parts an answer uses nor how many reasoning steps it writes down; the data can thus be neither audited nor controlled. We formalize data quality as a property of the alignment structure among document, question, and answer, and build LongForge, which pairs an analyzer that measures long-context data with a synthesizer that generates it. The analyzer reverse-reconstructs each sample into an evidence-assertion reasoning graph and programmatically computes four dimensions on it: breadth, depth, correctness, and completeness. Every reading traces back to specific evidence spans and reasoning steps, making data auditable. Measuring seven mainstream datasets reveals two previously unquantified defects: breadth barely grows with document length, and depth is generally low, the two being nearly orthogonal. The synthesizer then embeds the analyzer into generation, gating on correctness and completeness and deferring breadth and depth to a training-time mixture over measured values, which makes the training distribution controllable. On two backbones, its two mixtures, LongForge-1 and LongForge-N, outperform the strongest baseline by 2.2-4.7 points on average over six long-context benchmarks, and in a controlled comparison that holds the gate and the candidate pool fixed, raising breadth and depth alone is worth a further 1.16 points, making the structural distribution of training data a lever on downstream quality rather than only a description of it.

Figures & tables

Figure 2: Overview. (a) Existing data keeps only one-shot judgments. (b) The analyzer reconstructs an evidence–assertion reasoning graph and computes four dimensions on it. (c) Each synthesized sample carries these readings and is auditable and controllable.

Figure 3: The analyzer pipeline (bottom) and an example (top) with $B=4$ , $P=3$ , $R=3/4$ , and $C=3/4$ .

Figure 4: Per-sample evidence count (a, log scale) and document utilization (b). Each row is a baseline; the lines in the middle connect the positions of the same baseline in the two panels.

Figure 5: Evidence-length elasticity. (a) Per-baseline scaling; the gray dashed lines are iso-utilization lines, the colored line is the median by document-length bin, and the black dashed line is the log–log fit, whose slope is the elasticity $\beta$ . (b) $\beta$ with 95% confidence intervals.

Figure 8: Composition of completeness values. (a) Share of samples with completeness

=1

, in

(0,1)

, and

=0

in each dataset; the right column gives the mean. (b) Value composition stratified by the number of information points in the question; the top strip gives the share of samples in each stratum (log scale).

Figure 9: Failure-mode matrix. Each row is a dataset and each column a failure axis; each cell is the value standardized across baselines within the axis (higher = worse), and the black box marks the dataset’s dominant defect.

Figure 10: Human-audit agreement: strict agreement and usable rates, pooled (top) and per source (bottom).

Figure 11: The synthesizer pipeline. Left: question generation and answering. Middle: measurement and gating by the analyzer, with failing QA pairs re-answered from their failure reasons. Right: the distribution before and after mixing by breadth and depth when the training set is built.

Figure 12: Joint distribution of breadth and depth per training sample. Each cell is the share of samples falling in it, with the marginal distributions on the top and right; the three panels share one set of bins and one probability scale. The version with the seven baselines shown separately is Figure 14 in Appendix C.1 .

	Seven baselines	LongForge-1	LongForge-N
Breadth $B$	7.18	12.2	95.5
Depth $P$	1.798	2.89	4.42
Single-step rate	0.434	0.154	0.015
Correctness $R$	0.786	0.928	0.918
Completeness $C$	0.913	1.000	1.000
Document utilization	0.121	0.102	0.519

Table 1: Data distributions of the two mixtures alongside the seven baselines. The single-step rate is the share of samples with depth 1; all other rows are sample means.

Llama3.1-8B
Training data	LongBench	LongBench v2	LooGLE	MRCR	HELMET	RULER	Avg.
+ LongAlign	35.37 $\pm$ 0.53	25.23 $\pm$ 1.66	61.67 $\pm$ 0.26	1.54 $\pm$ 1.29	42.15 $\pm$ 1.39	81.99 $\pm$ 1.45	41.32 $\pm$ 0.51
+ LongAlpaca	30.78 $\pm$ 0.27	26.17 $\pm$ 3.92	62.08 $\pm$ 0.73	0.02 $\pm$ 0.03	42.91 $\pm$ 1.47	80.33 $\pm$ 1.76	40.38 $\pm$ 0.52
+ LongReward	35.28 $\pm$ 0.73	26.53 $\pm$ 0.70	61.78 $\pm$ 0.30	4.60 $\pm$ 2.57	49.07 $\pm$ 1.51	83.81 $\pm$ 0.71	43.51 $\pm$ 0.82
+ SEALONG	23.02 $\pm$ 0.61	19.23 $\pm$ 1.45	43.35 $\pm$ 3.17	0.85 $\pm$ 0.73	44.32 $\pm$ 3.51	83.31 $\pm$ 0.75	35.68 $\pm$ 0.44
+ LongMIT	33.69 $\pm$ 0.91	24.27 $\pm$ 0.67	63.01 $\pm$ 0.56	11.98 $\pm$ 3.15	46.71 $\pm$ 0.93	86.40 $\pm$ 1.09	44.34 $\pm$ 0.46

Table 2: Six-benchmark results on the two backbones (mean and std over three seeds); shaded rows are ours, and bold marks the best per backbone.

Item	LongForge-1 (control)	Document source	Synthesis model	Distribution mixture
Document pool	Mixed pool	LongMagpie only	Mixed pool	Mixed pool
Synthesis model	DeepSeek-V4.1-Flash	DeepSeek-V4.1-Flash	Qwen3.8-27B	DeepSeek-V4.1-Flash
Breadth $B$ (mean / median)	12.2 / 10	11.7 / 10	11.2 / 9	4.41 / 5
Depth $P$	2.89	3.12	2.74	1.95
Single-step rate	0.154	0.092	0.210	0.462
LongBench	46.34 $\pm$ 0.35	45.67 $\pm$ 0.65	48.53 $\pm$ 0.52	45.57 $\pm$ 0.45

Table 3: Settings and results of the three ablation groups. Each group changes one setting (italic) and keeps the rest identical to the control; the first two groups do not change the distribution mixture, so breadth and depth fluctuate slightly with their candidate pools. The seven baselines pooled have breadth 7.18 / 5 (mean / median), depth 1.798, and single-step rate 0.434.

Figure 13: A LongMagpie (left) and a LongForge-1 (right) QA pair on the same source document. Top: the positions in the document of the evidence each pair uses. Middle: the question, the answer, and the reasoning graph the analyzer reconstructs from them, where diamonds are reasoning steps and red marks an unsupported one. Bottom: information-point coverage, the step judgments, and the readings of the four dimensions.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Round	Passed in round	Cumulative pass rate
1 (first answer)	17,912	87.46%
2	1,943	96.95%
3	346	98.64%
4	117	99.21%
Finally failed	162	–

Appendix

Table 4: Gate pass statistics of each answering round during synthesis.

Component	Stage	Per doc.	Per QA	Share
Synthesizer	Question generation	1.0	0.20	0.15%
	Answering (incl. re-answers)	5.8	1.17	0.86%
	Subtotal	6.8	1.37	1.01%
Analyzer	Evidence extraction	518.3	103.7	76.49%
	Graph assembly	25.5	5.1	3.76%
	Step judgment	126.9	25.4	18.73%

Appendix

Table 5: Model-call cost of synthesis and analysis. “Per QA” counts all 20,480 generated QA pairs, whereas the per-QA cost in the text is normalized by the pairs that pass the gate.

	Qwen tokenizer			Llama tokenizer
Data	Total	Input	Output	Total	Input	Output
LongAlign	16254	16072	182	16974	16773	201
LongAlpaca	12828	12559	270	12409	12144	265
LongReward	22136	21858	278	23555	23214	341
SEALONG	19421	19157	264	18465	18212	253
LongMIT	69699	69554	145	78131	77957	174

Appendix

Table 6: Token lengths of each training set.

Figure 14: Joint distribution of breadth and depth with the seven baselines shown separately; the last two panels are our two mixtures.

Dataset	Samples	Breadth (median / mean)	Depth (median / mean)	Single-step rate	Wide-and-deep	Modal cell (depth / breadth)
LongAlign	994	4.5 / 6.27	2 / 1.649	48.3%	28.6%	1 / 2–3
LongAlpaca	963	10 / 11.17	2 / 1.977	34.5%	55.9%	2 / 7–11
LongReward	986	10 / 12.43	1 / 1.595	54.4%	38.7%	1 / 7–11
SEALONG	1000	3 / 4.08	2 / 1.835	39.8%	15.0%	1 / 2–3
LongMIT	1000	6 / 6.44	1 / 1.621	51.2%	31.4%	1 / 2–3
LongMagpie	891	5 / 6.51	2 / 1.860	48.6%	25.6%	1 / 2–3

Appendix

Table 7: Summary readings of breadth and depth per dataset; the last row is the pooled total. Wide-and-deep means breadth at least 6 and depth at least 2; the modal cell is the cell with the largest share of samples in Figure 14 , given by its depth and breadth interval.

	Breadth	Depth	Correctness	Completeness
Breadth	–	0.189	$-0.134$	0.109
Depth	0.070	–	$-0.253$	0.010
Correctness	$-0.025$	$-0.166$	–	0.032
Completeness	0.100	$-0.007$	0.035	–

Appendix

Table 8: Correlations among the four dimensions over the 6,834 baseline samples (upper triangle: Spearman; lower triangle: Pearson).

Dimension	Axis	Computation (aggregated within a dataset)	Meaning
Breadth	Narrow	Share of samples with breadth at most 2	The answer uses at most two evidence spans
Breadth	Local	Sample mean of $1-$ evidence spread $/(M-1)$ ; the evidence spread is the index difference between the first and last chunks that yield evidence, and the denominator is 1 when $M=1$	Evidence is clustered in a small part of the document
Breadth	Doc-starved	Sample mean of $1-$ document utilization; document utilization is the number of chunks that yield evidence $/M$	Most of the document is never used
Depth	Shallow	Share of samples with depth 1	No explicit intermediate reasoning is written down
Correctness	Ungrounded	Share of isolated assertions among all assertions of a sample, averaged over samples	Some whole answer sentences connect to no evidence
Correctness	Unsupported	Sample mean of $1-$ correctness	An assertion connects to evidence, but the step does not derive it

Appendix

Table 9: Definitions of the seven failure axes.

M

is the number of chunks in the document; the axis names are the column names in Figure 9 .

Figure 15: One QA pair from LongMagpie contrasted with five turns of QA from LongForge-N on the same source document. On the right of each row are the positions in the document of the evidence used by that QA pair: dark chunks are new (unused in earlier turns), light chunks are reused, and the rightmost column gives the cumulative number of covered chunks. Bottom left: a local reasoning graph of an unsupported step in the baseline. Bottom right: the cumulative document coverage of the five turns.

Dimension	Meaning of the analyzer’s value	Consistent	Minor deviation	Major deviation
Breadth	Number of evidence spans counted toward breadth, excluding evidence discarded at assembly	The auditor’s evidence count equals the analyzer’s, or differs by 1 where that span is a borderline case (quoted range slightly wider or narrower)	Differs by 1–2 without changing the order-of-magnitude judgment of how much of the document the sample uses	Differs by 3 or more; or a counted evidence span is entirely irrelevant to the question; or a discarded evidence span is necessary for the answer
Depth	Number of hops of the longest reasoning path after paraphrase contraction	The longest chain reconstructed by the auditor has the same number of hops as the analyzer’s	Differs by 1 hop without crossing the boundary between 1 and 2	Differs by 2 or more hops; or crosses the boundary between 1 and 2 (1 hop means no intermediate reasoning and 2 hops at least one inference, a qualitative difference); or a step counted as reasoning is a pure paraphrase
Correctness	Share of supported steps among all reasoning steps	The auditor’s set of supported steps is identical to the analyzer’s	Exactly 1 step is judged differently, without crossing 0.5	2 or more steps are judged differently; or crosses 0.5 (majority supported vs. majority unsupported); or a step the analyzer judges supported is clearly underivable (evidence unrelated to the assertion, or a proper noun absent from the inputs)
Completeness	Share of answered information points among all information points	The auditor’s judgments of whether each information point is answered all match the analyzer’s	Some information points are judged differently, without crossing complete vs. incomplete	Crosses complete vs. incomplete (1.0 vs. below 1.0); or the question decomposition itself is wrong (e.g., what should be two information points is decomposed as one)

Appendix

Table 10: Audit criteria for the four dimensions.

Dimension	Usable rate (A / B / both)	Binary agreement	Binary AC1	Three-level agreement	Three-level AC1
Breadth	99.7% / 100% / 99.7%	0.997	0.997	0.997	0.997
Depth	97.4% / 98.6% / 96.2%	0.965	0.964	0.957	0.956
Correctness	89.6% / 95.4% / 87.9%	0.908	0.893	0.671	0.568
Completeness	98.0% / 100% / 98.0%	0.980	0.979	0.971	0.971

Appendix

Table 11: Agreement between the two auditors. The usable-rate column lists auditor A, auditor B, and the share rated usable by both; the binary scheme distinguishes only usable (consistent or minor deviation) from major deviation, and the three-level scheme distinguishes consistent, minor deviation, and major deviation.

Training data	Overall	2-needle	4-needle	8-needle
LongAlign	18.87 $\pm$ 5.42	26.56 $\pm$ 5.41	18.62 $\pm$ 6.85	11.57 $\pm$ 4.02
LongAlpaca	8.21 $\pm$ 3.48	11.18 $\pm$ 4.95	6.25 $\pm$ 4.46	7.28 $\pm$ 1.43
LongReward	17.11 $\pm$ 1.38	23.79 $\pm$ 3.11	15.55 $\pm$ 1.43	12.14 $\pm$ 0.10
SEALONG	0.26 $\pm$ 0.28	0.29 $\pm$ 0.31	0.36 $\pm$ 0.38	0.12 $\pm$ 0.15
LongMIT	15.54 $\pm$ 3.35	21.97 $\pm$ 5.14	13.32 $\pm$ 3.50	11.46 $\pm$ 1.63
LongMagpie	15.86 $\pm$ 1.38	20.98 $\pm$ 1.81	13.89 $\pm$ 1.70	12.81 $\pm$ 1.31

Appendix

Table 12: Results on the 256k bin of MRCR (Qwen3.5-4B).

Llama3.1-8B
Training data	HellaSwag	ARC-C	ARC-E	PIQA	WinoGrande	LogiQA	LAMBADA	MMLU	GSM8K	Avg.
+ LongAlign	80.09 $\pm$ 0.38	56.09 $\pm$ 2.06	57.48 $\pm$ 1.39	78.56 $\pm$ 0.55	74.19 $\pm$ 0.62	29.70 $\pm$ 0.18	77.20 $\pm$ 0.99	47.10 $\pm$ 4.50	38.62 $\pm$ 2.12	59.89 $\pm$ 0.42
+ LongAlpaca	81.79 $\pm$ 0.70	58.76 $\pm$ 1.11	69.73 $\pm$ 0.80	80.38 $\pm$ 0.23	74.88 $\pm$ 0.16	30.67 $\pm$ 2.00	76.28 $\pm$ 0.82	42.69 $\pm$ 6.33	42.30 $\pm$ 1.59	61.94 $\pm$ 0.58
+ LongReward	78.98 $\pm$ 0.48	56.97 $\pm$ 0.43	60.59 $\pm$ 0.97	78.53 $\pm$ 0.33	74.38 $\pm$ 1.23	30.01 $\pm$ 0.47	77.32 $\pm$ 1.22	44.63 $\pm$ 2.87	37.20 $\pm$ 5.08	59.84 $\pm$ 0.82
+ SEALONG	76.59 $\pm$ 0.43	52.82 $\pm$ 1.27	51.73 $\pm$ 0.34	77.98 $\pm$ 0.45	74.69 $\pm$ 0.36	30.01 $\pm$ 0.64	75.28 $\pm$ 0.59	34.27 $\pm$ 5.60	31.31 $\pm$ 0.95	56.07 $\pm$ 0.58
+ LongMIT	76.11 $\pm$ 0.82	55.18 $\pm$ 1.45	67.44 $\pm$ 3.02	79.72 $\pm$ 0.17	71.32 $\pm$ 2.73	30.57 $\pm$ 1.26	76.08 $\pm$ 0.67	51.28 $\pm$ 2.81	21.10 $\pm$ 3.22	58.76 $\pm$ 0.81

Appendix

Table 13: Results on nine short-context benchmarks.

Training samples	LongBench	LongBench v2	LooGLE	MRCR	HELMET	RULER	Avg.
2048 (control)	46.34 $\pm$ 0.35	37.00 $\pm$ 0.20	71.39 $\pm$ 0.56	35.18 $\pm$ 1.25	60.61 $\pm$ 0.61	90.87 $\pm$ 0.56	56.90 $\pm$ 0.10
4096	47.94 $\pm$ 1.21	36.73 $\pm$ 1.81	71.27 $\pm$ 0.83	36.07 $\pm$ 1.53	60.30 $\pm$ 0.86	91.20 $\pm$ 0.83	57.25 $\pm$ 0.33
8192	48.55 $\pm$ 0.36	36.80 $\pm$ 0.92	70.00 $\pm$ 0.78	35.50 $\pm$ 0.74	60.19 $\pm$ 0.40	91.08 $\pm$ 0.93	57.02 $\pm$ 0.37

Appendix

Table 14: Results on six benchmarks as the training size grows from 2,048 to 8,192 samples (LongForge-1, Qwen3.5-4B).

Llama3.1-8B
Training data	Short-dep QA	Long-dep QA
+ LongAlign	82.62 $\pm$ 0.98	40.72 $\pm$ 1.36
+ LongAlpaca	83.13 $\pm$ 0.18	41.03 $\pm$ 1.29
+ LongReward	80.94 $\pm$ 0.22	42.63 $\pm$ 0.41
+ SEALONG	59.45 $\pm$ 3.83	27.25 $\pm$ 2.54
+ LongMIT	84.67 $\pm$ 1.48	41.35 $\pm$ 2.57

Appendix

Table 15: Results on the two QA subtasks of LooGLE.

Llama3.1-8B
Training data	Avg	Single-Doc QA	Multi-Doc QA	Summarization	Few-shot	Synthetic	Code
+ LongAlign	35.37 $\pm$ 0.53	17.44 $\pm$ 0.12	13.95 $\pm$ 0.43	24.75 $\pm$ 0.20	60.45 $\pm$ 1.31	50.13 $\pm$ 2.51	63.05 $\pm$ 2.76
+ LongAlpaca	30.78 $\pm$ 0.27	16.74 $\pm$ 0.44	13.54 $\pm$ 0.89	21.12 $\pm$ 1.38	61.90 $\pm$ 0.34	20.54 $\pm$ 6.31	65.80 $\pm$ 2.33
+ LongReward	35.28 $\pm$ 0.73	19.52 $\pm$ 0.19	14.46 $\pm$ 0.16	24.60 $\pm$ 0.16	62.63 $\pm$ 0.95	39.80 $\pm$ 5.42	68.26 $\pm$ 2.13
+ SEALONG	23.02 $\pm$ 0.61	8.17 $\pm$ 0.64	3.31 $\pm$ 0.17	17.16 $\pm$ 0.59	60.66 $\pm$ 0.92	2.42 $\pm$ 1.64	59.48 $\pm$ 7.89
+ LongMIT	33.69 $\pm$ 0.91	32.78 $\pm$ 2.67	20.04 $\pm$ 4.90	22.76 $\pm$ 0.29	60.35 $\pm$ 0.46	12.07 $\pm$ 7.13	63.84 $\pm$ 1.08

Appendix

Table 16: Subtask results on LongBench.

Llama3.1-8B
Training data	Overall	Easy	Hard	Short	Medium	Long
+ LongAlign	25.23 $\pm$ 1.66	25.17 $\pm$ 2.12	25.30 $\pm$ 1.45	27.40 $\pm$ 3.59	24.07 $\pm$ 0.71	24.10 $\pm$ 2.80
+ LongAlpaca	26.17 $\pm$ 3.92	26.73 $\pm$ 5.62	25.80 $\pm$ 3.05	28.87 $\pm$ 4.72	23.87 $\pm$ 3.71	26.23 $\pm$ 3.52
+ LongReward	26.53 $\pm$ 0.70	27.60 $\pm$ 0.87	25.93 $\pm$ 1.19	28.17 $\pm$ 3.07	25.73 $\pm$ 0.71	25.63 $\pm$ 1.93
+ SEALONG	19.23 $\pm$ 1.45	20.33 $\pm$ 1.55	18.53 $\pm$ 2.56	18.33 $\pm$ 2.39	19.23 $\pm$ 2.41	20.70 $\pm$ 6.17
+ LongMIT	24.27 $\pm$ 0.67	23.97 $\pm$ 1.55	24.43 $\pm$ 2.12	21.30 $\pm$ 3.73	25.90 $\pm$ 1.49	25.93 $\pm$ 6.11

Appendix

Table 17: Subtask results on LongBench v2, sliced by difficulty and length.

Llama3.1-8B
Training data	Avg	8K	16K	32K	64K	128K
+ LongAlign	1.54 $\pm$ 1.29	3.65 $\pm$ 2.94	1.64 $\pm$ 1.60	0.82 $\pm$ 0.74	1.29 $\pm$ 1.01	0.13 $\pm$ 0.20
+ LongAlpaca	0.02 $\pm$ 0.03	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.13 $\pm$ 0.18	0.00 $\pm$ 0.00
+ LongReward	4.60 $\pm$ 2.57	2.64 $\pm$ 1.76	6.06 $\pm$ 3.93	5.73 $\pm$ 3.71	8.23 $\pm$ 3.88	0.58 $\pm$ 0.56
+ SEALONG	0.85 $\pm$ 0.73	0.94 $\pm$ 0.78	1.45 $\pm$ 1.24	1.39 $\pm$ 1.62	0.47 $\pm$ 0.41	0.00 $\pm$ 0.00
+ LongMIT	11.98 $\pm$ 3.15	17.52 $\pm$ 5.73	18.58 $\pm$ 3.34	12.66 $\pm$ 4.94	8.67 $\pm$ 1.31	2.12 $\pm$ 0.36

Appendix

Table 18: Per-bin results on MRCR up to 128k. Avg. is pooled over all samples in these five bins; the 256k bin is evaluated separately in Appendix D.1 .

Llama3.1-8B
Training data	Avg	RAG	ICL	Re-rank	LongQA
+ LongAlign	42.15 $\pm$ 1.39	49.06 $\pm$ 3.03	78.91 $\pm$ 3.34	29.55 $\pm$ 4.83	11.05 $\pm$ 3.04
+ LongAlpaca	42.91 $\pm$ 1.47	59.64 $\pm$ 0.38	72.10 $\pm$ 3.50	26.14 $\pm$ 1.39	13.75 $\pm$ 1.08
+ LongReward	49.07 $\pm$ 1.51	59.34 $\pm$ 1.40	80.36 $\pm$ 2.16	31.57 $\pm$ 1.93	25.01 $\pm$ 4.40
+ SEALONG	44.32 $\pm$ 3.51	49.48 $\pm$ 4.71	76.59 $\pm$ 0.63	30.00 $\pm$ 4.98	21.19 $\pm$ 3.86
+ LongMIT	46.71 $\pm$ 0.93	60.16 $\pm$ 0.27	77.75 $\pm$ 1.24	18.02 $\pm$ 2.28	30.90 $\pm$ 0.79

Appendix

Table 19: Results on the four HELMET categories.

Synthetic Data Generation for Language Models

Synthetic data generation for language models creates artificial text, tasks, and interaction traces to train or evaluate models.

See all 292 papers

Jul 7, 2026

LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis

Synthesizing long-context supervised fine-tuning (SFT) data is a scalable way to enhance the long-context understanding of large language models (LLMs), yet existing approaches share three limitations: narrow task coverage, insufficient instruction difficulty, and a lack of faithfulness supervision. We propose LongCrafter, a structured synthesis framework that couples a hierarchical task taxonomy with an evidence-grounded pipeline. The taxonomy organizes long-context understanding into local/shallow and global/deep levels and yields 32 fine-grained task types that serve as a global generative prior. Guided by this taxonomy, LongCrafter constructs task-aligned long contexts, decomposes them into explicit evidence graphs that model cross-paragraph dependencies, and generates instruction--response pairs strictly grounded in the located evidence spans, ensuring both controllable difficulty and faithful, traceable reasoning. Models fine-tuned on LongCrafter data outperform all SFT baselines and even the official post-trained models on LongBench, LongBench~v2, and LooGLE across both Qwen2.5-7B and LLaMA-3.1-8B, with the largest gains on high-difficulty tasks. Further analysis shows that LongCrafter data is more diverse and better spread across difficulty levels, and that the trained models locate evidence robustly regardless of position, effectively mitigating the ``lost in the middle'' problem.

Chenhao Yuan, Yinhao Xu, Shuwen Xu +8

University of Chinese Academy of Sciences, Beijing, China · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China

Lost-in-the-Middle Effect LLM Fine-Tuning Evidence-Grounded Generation Synthetic Data Generation for Language Models

Ask PDF

Oct 25, 2026

"High-Quality" Data for Early-Stage Instruction Tuning: An Empirical Study of Data Quality Metrics

"Data quality" is widely invoked in instruction tuning, but the term is often underspecified. We study it as an operational property by asking which simple example-level metrics, drawn from prior work on data selection, text quality, and instruction tuning, best predict training utility under a fixed finetuning budget. For each metric, we sort instruction-response examples by score, partition them into low, middle, and high bins with roughly equal token mass, finetune separately on each bin, and compare validation perplexity and LLM-as-judge scores for the models' generated answers as a function of cumulative training tokens. Our results suggest that answer-dense supervision, as captured by response ratio, is a stronger and more portable predictor of instruction-tuning utility than commonly used alternative proxies.

Vladimir Necula, David Gvadzabia, Sofia Serrano

University of Cambridge Department of Physics · Lafayette College Department of Computer Science

Instruction-Tuning Data Selection Instruction Tuning

Ask PDF

Jun 17, 2026

Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning

Long-context reasoning is an essential capability for large language models, particularly when they are deployed as autonomous agents that must reason over lengthy trajectories. Reinforcement learning (RL) has recently emerged as a dominant paradigm for improving this ability, yet existing work largely focuses on reward engineering while diverse training data remains scarce. We revisit this problem from a data-centric perspective and show that a simple yet effective data recipe alone, paired with a minimal outcome-based GRPO setup, suffices to substantially improve long-context reasoning. Our recipe targets three complementary task families -- retrieval, multi-evidence synthesis, and reasoning -- for which we construct and curate eight datasets totaling ~14K examples. Experiments on three models (Qwen3-4B/8B/30B-A3B) yield average gains of +7.2/+3.2/+6.4 points across seven long-context benchmarks, surpassing prior RL training sets. We further demonstrate that these gains transfer to agentic tasks, where continuing RL training on an agent-tuned model with our data recipe improves GAIA by +4.8 and BrowseComp by +7.0 points. We will release our datasets to facilitate future research.

Xiaoyue Xu, Sikui Zhang, Xiaorong Wang +2

OpenBMB · Tsinghua University

Training Data Curation RL for Language Model Reasoning Long-Context Modeling Long-Context Inference RL Fine-Tuning Long-Context Reasoning in Language Models

Ask PDF

Abstract

Synthetic Data Generation for Language Models

Explore similar work

LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis

"High-Quality" Data for Early-Stage Instruction Tuning: An Empirical Study of Data Quality Metrics

Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning