WEAVE: Weaving Real-World Code Evolution into Dependency-Structured Tasks for Benchmarking and Training Code Agents

Weihao Li; Zihang Xu; Zhongqiu Wang; Songwei Yu; Wenqiang Zhu; Yuechen Luo; Jingxuan Xu; Yanan Wu

cs.SEOct 15, 2026

WEAVE: Weaving Real-World Code Evolution into Dependency-Structured Tasks for Benchmarking and Training Code Agents

Authors: Weihao Li, Zihang Xu, Zhongqiu Wang, Songwei Yu, Wenqiang Zhu, Yuechen Luo, Jingxuan Xu, Yanan Wu,

Organizations: Independent Researchers · Shanghai Artificial Intelligence Laboratory · National University of Singapore · Tsinghua University

Abstract

Real-world software engineering rarely consists of isolated changes. Features, migrations, and systematic fixes often unfold across multiple pull requests (PRs), requiring coordinated modifications across a repository. Yet most executable repository-level benchmarks retain a single issue or PR as the task unit, while existing multi-PR settings rely on coarse aggregation and structure-agnostic evaluation. This mismatch obscures dependencies among changes and limits progress measurement and learning signals for long-horizon code agents. We introduce WEAVE, a framework that transforms verified PR histories into executable, dependency-structured tasks. WEAVE leverages repository-calibrated typed change loci to organize related PRs into bounded DAGs and instantiate each as a unified executable task. We further introduce Graph Credit, a deterministic, test-grounded metric that measures dependency-consistent partial progress and supports dense, verifiable rewards for reinforcement learning. Experiments on task construction, agent evaluation, supervised fine-tuning, and reinforcement learning show that WEAVE yields coherent executable tasks, enables more faithful progress measurement than flat scoring, and provides effective supervision and verifiable rewards for training code agents.

Figures & tables

Benchmark	Unit	Construction	Struct.	Exec.	Credit	Train	Lang.
SWE-bench Verified	Issue–PR	Issue linked	Atomic	✓	Binary	–	Py
Multi-SWE-bench	Issue–PR	Issue linked	Atomic	✓	Binary	✓	Multi
SWE-Gym	Issue–PR	Real issues	Atomic	✓	Binary	✓	Py
SWE-EVO	Release delta	Release based	Flat	✓	Flat F2P	–	Py
SWE-CI	Commit span	Evolution span	Sequence	✓	EvoScore	–	Py
SWE-Milestone	Milestone	Commit + LLM	DAG	✓	Milestone	–	Multi

Table 1: Comparison of executable software-engineering benchmarks Struct. denotes task structure retained for evaluation; Exec. indicates an executable oracle; and Train indicates available executable training data.

\sim

denotes partial support.

Figure 1: Real-world objectives span dependent PRs, but existing proxies obscure task structure and credit assignment. WEAVE constructs executable dependency-structured tasks for verifiable evaluation and rewards.

Figure 2: Overview of WEAVE . Validated PRs are represented by typed change loci, coupled into bounded dependency DAGs, and instantiated as executable tasks with unified specifications and node-attributed test oracles. The resulting graphs support structure-aware evaluation and reinforcement learning.

Figure 3: Composition and structural characteristics of WEAVE-Bench . (a) Language balance and PR scale. (b) Dependency-DAG complexity and patch scope.

Scaffold	Model	Resolved $\uparrow$	Flat-F2P $\uparrow$	GraphCr. $\uparrow$	OverCr. $\downarrow$	Turns $\downarrow$	Tok (M) $\downarrow$
OpenCode	MiniMax-2.7	12.3	24.2	13.4	10.8	90.9	5.98
	DeepSeek-V4-Flash	21.0	21.8	14.6	7.2	78.7	6.03
	Kimi-K2.5	14.0	12.4	5.1	7.3	88.6	6.65
	Kimi-K2.6	29.3	30.1	21.4	8.7	96.3	8.16
	GLM-5.1	28.7	24.1	14.5	9.6	114.8	7.88
	GLM-5.2	34.0	32.5	23.8	8.7	145.1	12.17

Table 2: Agent performance on WEAVE-Bench (8 models

\times

2 scaffolds). Resolved : all F2P pass, no P2P regressions. Flat-F2P : attributed F2P pass rate. Graph Credit : dependency-conditioned partial score (§ 4.1 ). OverCredit : phantom credit

=

Flat-F2P

-

Graph Credit. Turns / Tok : per-task averages. Models grouped by scaffold and ordered by model family (MiniMax, DeepSeek, Kimi, GLM, Claude), with earlier versions listed first within each family; best per-column bolded within each scaffold.

Method	AP $\uparrow$	Coverage $\uparrow$	P@50 $\uparrow$
Temporal Window	5.1	73.1	2.0
Traceability Links	17.2	12.8	28.0
File Overlap	26.6	73.5	10.0
Symbol Overlap	24.0	69.0	30.0
WEAVE (Ours)	37.7 ∗∗	84.5	38.0

Table 3: Dependency recovery quality (30 repos, 219 gold edges, Fleiss’

\kappa

= 0.791). AP: per-repo macro-average. Coverage: gold-edge recall at emission. P@50: pooled top-50 precision. Paired bootstrap + Holm correction; ∗∗

p{<}0.01

Figure 4: SFT comparison across different data construction strategies.

Figure 5: RL reward ablation on two model scales. All Runs start from the WEAVE -SFT checkpoint with matched rollout budgets.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Language	$\mathcal{L}_{\mathrm{ast}}$	$\mathcal{L}_{\mathrm{sym}}$	$\mathcal{L}_{\mathrm{cfg}}$	$\mathcal{L}_{\mathrm{decl}}$	$\mathcal{L}_{\mathrm{dep}}$
Python	✓	✓	✓	✓	✓
TypeScript	✓	✓	✓	✓	✓
JavaScript	✓	✓	✓	$\sim$	✓
Go	✓	✓	$\sim$	✓	✓
Java	✓	✓	$\sim$	✓	✓
Kotlin	✓	✓	$\sim$	✓	✓

Appendix

Table 4: Language support matrix for typed change loci extraction. ✓: full support;

\sim

: partial (heuristic-based).

Operator	Spearman $\rho$ $\uparrow$	Resid. OverCredit $\downarrow$
Linearized (no graph)	0.69	10.5 pp
Depth-weighted	0.73	7.1 pp
Mean-parent	0.77	3.2 pp
Min-bottleneck (ours)	0.82	0.0 pp

Appendix

Table 5: Comparison of graph operators. Spearman

\rho

with binary Resolved (higher

=

better diagnostic); residual OverCredit is the operator score minus the min-bottleneck score, i.e., the phantom credit the operator fails to remove (lower

=

better). The linearized row therefore recovers the corpus-level OverCredit of 10.5 pp reported in the main paper.

Parameter	Value
Max interaction turns	200
Total task timeout	2 hours
Test suite timeout	30 minutes
Tool interface	Shell + File Edit
Context window	Model-provider default
Temperature	Model-provider default

Appendix

Table 6: Scaffold evaluation configurations.

$\theta$	AP $\uparrow$	Coverage $\uparrow$	Exec. Rate $\uparrow$
0.05	31.4	92.0	62.4
0.10	35.6	88.0	71.2
0.15	37.6	84.0	76.8
0.20	37.2	78.0	82.4
0.25	34.0	68.0	80.6
0.30	30.2	58.0	78.8

Appendix

Table 7: Sensitivity to coupling threshold

\theta

(dev set,

\Delta{=}90

days). Bold : selected configuration.

Figure 6: Sensitivity of construction quality to coupling threshold

\theta

. Left axis: AP (solid) and Coverage (dashed); right axis: executable construction rate. The selected

\theta=0.20

(vertical dotted line) balances precision and recall while maintaining high executability. Shaded region indicates

\pm

1 standard deviation across development-set repositories.

$\Delta$ (days)	AP $\uparrow$	Coverage $\uparrow$	Candidates
7	38.4	48.0	326
14	38.0	60.0	708
30	37.6	72.0	1,482
60	37.4	77.4	3,716
90	37.2	78.0	6,218

Appendix

Table 8: Sensitivity to search radius

\Delta

(dev set,

\theta{=}0.20

). Bold : selected configuration.

Method	P@10	P@20	P@50	P@100
Temporal Window	10.0	5.0	2.0	2.0
Traceability Links	60.0	45.0	28.0	17.0
File Overlap	20.0	15.0	10.0	8.0
Symbol Overlap	50.0	40.0	30.0	22.0
WEAVE	70.0	50.0	38.0	29.0

Appendix

Table 9: Precision at various rank cutoffs. Candidate edges from all 30 repositories are merged into one score-ranked list; P@K is the fraction of gold dependency edges among its top

K

entries.

Single type
Configuration	AP $\uparrow$	Coverage $\uparrow$
$\mathcal{L}_{\mathrm{ast}}$ only	28.4	62.1
$\mathcal{L}_{\mathrm{sym}}$ only	22.6	55.8
$\mathcal{L}_{\mathrm{cfg}}$ only	14.2	38.4
$\mathcal{L}_{\mathrm{decl}}$ only	18.8	44.2
$\mathcal{L}_{\mathrm{dep}}$ only	12.4	32.6

Appendix

Table 10: Locus-type ablation. “Single”: using only that type. “

-

Type”: removing that type from the full set.

Qwen3.5-4B
Model	Training Data	SWE-V	SWE-Pro	SWE-ML	WB-Res	WB-GC
	Base (no SFT)	50.4	24.2	35.3	4.0	8.6
External Reference
	SWE-rebench V2	51.8	25.4	36.6	4.7	9.5
	daVinci-Agency	50.6	24.0	35.7	3.7	8.2
Controlled (same repos, matched tokens)

Appendix

Table 11: Full SFT comparison across data construction strategies (matched token budget). All values are percentages (%). Bold : best per model scale; underline : best overall.

Budget	25%	50%	75%	100%
Atomic-PR	9.2	10.8	11.7	12.4
Temporal-Window	9.9	11.7	12.8	13.5
WEAVE	12.6	15.2	16.7	17.8

Appendix

Table 12: Token-scaling analysis (Qwen3.5-4B, WB-GC metric).

Figure 7: Token-scaling curves for SFT training (Qwen3.5-4B, measured by WEAVE-Bench Graph Credit). WEAVE (solid blue) achieves at 25% of the token budget what Atomic-PR (dashed orange) achieves at 100%, indicating a token-efficiency advantage of at least

\sim

\times

. Temporal-Window (dotted green) falls between the two. The three curves differ in slope rather than offset, indicating that dependency-structured supervision is denser per token rather than merely better initialized. Error bars show

\pm

1 std across 3 random seeds.

Reward Config	WB-Res	WB-GC	SWE-V	$\Delta$
SFT only (no RL)	9.0	17.8	57.8	$-$ 3.7
$R_{\mathrm{WEAVE}}$ (full)	12.7	23.4	61.0	–
$-$ GraphCredit	11.0	19.0	59.4	$-$ 1.7
$-$ Resolved	11.6	22.3	60.1	$-$ 1.1
$-$ FlatF2P	12.1	22.7	60.5	$-$ 0.6
$-$ Reg P2P	12.4	23.1	60.7	$-$ 0.3

Appendix

Table 13: Reward component ablation (Qwen3.5-4B, after RL). All runs start from the same WEAVE -SFT checkpoint with matched rollout budgets.

\Delta

is the WB-Res difference relative to full

R_{\mathrm{WEAVE}}

. After each removal the remaining coefficients are renormalized to sum to one, so differences reflect the missing signal rather than a change in reward scale.

Scheduling	WB-Res	WB-GC	SWE-V
Flat (uniform random)	11.7	22.2	60.2
3-phase curriculum	12.7	23.4	61.0
$\Delta$	+1.0 pp	+1.2 pp	+0.8 pp

Appendix

Table 14: Curriculum scheduling comparison (Qwen3.5-4B). Phases defined by DAG complexity: P1 (depth

\leq

|V|\leq

3), P2 (depth

\leq

|V|\leq

5), P3 (all tasks).

Stage	Retained	Pass rate
Rollouts generated (3,200 $\times$ 3)	9,600	–
Dependency-consistent success	4,320	45.0%
Length	4,058	93.9%
Regression	3,857	95.0%
Degenerate pattern	3,699	95.9%
Leakage	3,644	98.5%

Appendix

Table 15: Trajectory generation and filtering yield. All rows count rollouts except the last, which counts one retained trajectory per task.

Parameter	4B	35B-A3B	Search range
Learning rate	5e-6	3e-6	{2,3,5,10}e-6
LR schedule	Cosine, 10% floor		–
Warmup ratio	0.03	0.03	{0.01, 0.03, 0.05}
Global batch (seq.)	16	16	–
Training epochs	3	3	{2, 3, 4}
Max sequence length	131,072		–

Appendix

Table 16: SFT training configuration. Search ranges are given for parameters tuned on the held-out calibration set; “–” marks values fixed a priori.

Parameter	4B	35B-A3B	Search range
RL prompt pool	3,200 tasks		–
Rollouts per task ( $G$ )	8	8	–
Tasks per update	16	16	–
Passes over pool	2	2	–
$\ast$ Policy updates	400	400	–
$\ast$ Total episodes	51,200		–

Appendix

Table 17: RL (GRPO) training configuration. Rows marked

\ast

are derived from the rows above them. Search ranges are given for parameters tuned on the calibration set.

Language	WEAVE-Bench	WEAVE-Train
Python	50 (16.7%)	720 (22.5%)
TypeScript	50 (16.7%)	680 (21.3%)
JavaScript	50 (16.7%)	640 (20.0%)
Go	50 (16.7%)	480 (15.0%)
Kotlin	50 (16.7%)	360 (11.3%)
Java	50 (16.7%)	320 (10.0%)

Appendix

Table 18: Language distribution across dataset splits.

Metric	WEAVE-Bench		WEAVE-Train
	Mean	Median	Mean	Median
Nodes ( $\|V\|$ )	4.91	5	4.16	4
Edges ( $\|E\|$ )	4.99	5	4.28	4
Depth (path nodes)	3.93	4	3.71	4
Width (max anti-chain)	1.50	1	1.31	1

Appendix

Table 19: DAG structural statistics across splits. Depth counts nodes on the longest path, matching the benchmark distribution in Figure 3 . Training depth is converted from edge count by adding one; curriculum thresholds use edge count.

$\|V\|$	2	3	4	5	6	7	8
Tasks	640	704	640	480	352	224	160
Share	20.0	22.0	20.0	15.0	11.0	7.0	5.0

Appendix

Table 20: Node-count distribution of WEAVE-Train (

|V|

mean 4.16, median 4).

Metric	Mean	Median	P25	P75
Gold patch (lines)	378	274	139	528
Files modified	8.4	6	3	12
F2P tests	18.6	12	6	24
P2P tests	142	87	34	198
Attributed tests (%)	72.8	76.4	61.2	86.3

Appendix

Table 21: Patch and test statistics for WEAVE-Bench.

Figure 8: Dependency graphs for Case Studies 1 and 2. Left : 4-node DAG (Case 1). Node A (DB migration)

\to

Node B (ORM model)

\to

Node C (API serializer)

\to

Node D (frontend). Green fill indicates agent-completed nodes; red fill indicates unresolved. Right : 3-node DAG (Case 2). Node A (utility refactor) is partially resolved (4/6 tests) and branches to Node B (primary consumer, 5/6 tests) and Node C (secondary consumer, 2/6 tests). Effective credits are

c_{A}=c_{B}=2/3

and

c_{C}=1/3

; Node B is capped by its upstream prerequisite. Right-panel fill intensity encodes

c_{v}

Software Engineering Benchmarks

Software engineering benchmarks evaluate how well AI systems understand, build, repair, test, and maintain software in realistic settings.

See all 155 papers

Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments
Jul 30, 2026Haomin Qi, Xingliang Wang, Xuanqi Gao +9Code Generation Benchmarks Agent Benchmarks
Microsoft · University of California San Diego · Zhejiang University +2
DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks
Jul 8, 2026Wenqi Huang, Charley Lee, Leonard Tng +1AI Coding Agents Software Engineering Benchmarks
1Datacurve
SWE-Next: Scalable Real-World Software Engineering Tasks for Agents
Mar 21, 2026Jiarong Liang, Zhiheng Lyu, Zijie Liu +4Automated Software Engineering Software Engineering Agents
University of Waterloo · University of North Carolina at Chapel Hill · Independent

Jul 30, 2026J/K move · Enter open · S save

Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments

Haomin Qi, Xingliang Wang, Xuanqi Gao, Baihui Sang+8

Microsoft · University of California San Diego · Zhejiang University+2

Code Generation Benchmarks Agent Benchmarks

Open paper AskPDF arXiv

Scaling coding agents requires a continuing supply of executable data for training, benchmarking, and continuous evaluation. Each task must couple a realistic software state with a specification, development tools, and reliable verification. To expand this supply, we present Change2Task, a system grounded in repository history that converts merged pull requests (PRs) into verified tasks on healthy modern revisions of the same repository. It aligns historical evidence with evolved code, reconstructs task states through Patch Reversal, Code Mapping, or Agent Reconstruction, and validates the lifecycle from a healthy base to a task state and a restored state. By deriving multiple tasks grounded in developer evidence from maintained environments, Change2Task provides executable data for coding agent training and evaluation while reducing repeated environment setup, storage, and task construction effort. We evaluate five coding agent task families: Bug Fix, Feature Addition, Test Generation, Application Programming Interface (API) Migration, and Security Repair. Starting from 1,130 source changes eligible for construction, Change2Task achieves 79.6% verified task construction success across these task families. On a matched candidate set, it recovers 29.2% more verified tasks than a construction baseline based on PRs. Historical and reconstructed cases achieve 89.7% aggregate matched outcome agreement under agent evaluation, while reuse of modern bases reduces measured expenditure across the complete pipeline by 10.8%. Four matched studies examine construction stages, semantic review, branch consistency, and resource trade-offs. An industrial deployment further constructs 1,000 tasks from private PR histories and evaluates four models on a fixed 500-task cohort.

Abstract

Software Engineering Benchmarks

Explore similar work

Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments

DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks

SWE-Next: Scalable Real-World Software Engineering Tasks for Agents