Skill Dreaming: Evolving Repository Skills through Pull Request Streams

Xinyuan Song; Hasibul Haque; Jay Somasundaram; Ryan Turner; Bowen Zhu; Liang Zhao

cs.AIOct 26, 2026

Skill Dreaming: Evolving Repository Skills through Pull Request Streams

Authors: Xinyuan Song, Hasibul Haque, Jay Somasundaram, Ryan Turner, Bowen Zhu, Liang Zhao

Organizations: Causal Dynamics Lab, USA

Abstract

Large language model (LLM) agents increasingly rely on reusable skills for complex software engineering tasks, making skill libraries essential for repository development and maintenance. However, repository-specific skills must evolve with changing code structures, implementation conventions, and maintenance requirements. Pull requests (PRs) provide a natural source of such knowledge by recording how repositories evolve through implementation and review. Meanwhile, new PRs must follow repository-specific implementation patterns, modify related components consistently, and ensure that existing tests continue to pass while adding coverage for new functionality. This creates a mutually reinforcing cycle: PRs provide knowledge for updating skills, while these skills improve future PRs, maintain repositories, and support further skill evolution. To enable this co-evolution of PRs, repositories, and skills, we propose Skill Dreaming, a framework that learns from PR streams to guide continuous repository development and maintenance. Given a repository and its PR history, Skill Dreaming builds a repository-specific skill library by extracting executable knowledge from validated PRs and continuously refines the skills as new PR information becomes available. These skills guide coding agents not only in implementing requested features but also in maintaining associated tests, documentation, compatibility, and repository conventions. By integrating PR analysis, skill management, code execution, and continuous integration, Skill Dreaming provides an end-to-end workflow for applying learned skills to repository changes and incorporating subsequent PR outcomes into future skill updates.

Figures & tables

Figure 1. Overview of Skill Dreaming : PR-driven skill evolution. Top: PR streams provide development records. Middle: Skills evolve through generation, reuse, revision, and replacement. Bottom: Evolving skills guide repository development and maintenance, generating feedback for further evolution. Three interconnected layers illustrate the continuous evolution of PR streams, repository-specific skills, and software repositories.

Method	PR Learning	Skill Library	Skill Reuse	Skill Evolution	Version Awareness	Repo Maintenance	Training-Free
ReasoningBank ( Ouyang et al., 2025 )	✗	✓	✓	✓	✗	✗	✓
Trace2Skill ( Ni et al., 2026 )	✗	✓	✓	✓	✗	✗	✓
Learning to Commit ( Li et al., 2026a )	✓	✓	✓	✓	✗	✓	✓
CODESKILL ( Li et al., 2026c )	✗	✓	✓	✓	✗	✓	✗
Socratic-SWE ( Xiao et al., 2026 )	✗	✓	✓	✗	✗	✓	✗
SkillLens ( Yu et al., 2026 )	✗	✓	✓	✓	✗	✗	✓

Table 1. Comparison with existing skill-learning and repository-editing methods. ✓ indicates an explicitly supported mechanism, while ✗ indicates otherwise.

Operation	Decision and state change
Add	Create a new skill when no applicable skill exists.
Update	Revise an existing skill using new information.
Specialize	Narrow a skill’s trigger for a specific task or context.
Extend	Add compatible procedures or maintenance requirements.
Conflict	Identify incompatible rules requiring resolution.
Supersede	Replace outdated rules while preserving their history.

Table 2. Skill alignment operations. Actions determine how new PR information updates the existing skill set.

Benchmark	Backbone	Base	Ours	$\Delta$
SWE-bench Pro ( Deng et al., 2025 )	Claude Opus 5	0.79	0.88	+9.0%
SWE-bench Verified ( OpenAI, 2024a )	Claude Opus 5	0.83	0.92	+9.0%
SWE-bench Verified ( OpenAI, 2024a )	Claude Sonnet 4.6	0.77	0.79	+2.0%

Table 3. Official resolution rates on SWE-bench Pro and SWE-bench Verified. Resolved rates are evaluated using the official benchmark harness. Base denotes the default mini-agent without skills, and

\Delta

represents the improvement achieved by Ours over Base.

Method	Resolved Rate $\uparrow$	Safe Rate $\uparrow$	Existing $\uparrow$	Surf. F1 $\uparrow$	Algn. $\uparrow$
Base Agent	0.83	0.83	0.83	0.62	0.38
Append-only	0.88	0.94	0.94	0.72	0.44
Trace2Skill ( Ni et al., 2026 )	0.76	0.78	0.78	0.60	0.37
Learning to Commit ( Li et al., 2026a )	0.72	0.67	0.67	0.64	0.49
SkillLens ( Yu et al., 2026 )	0.90	0.61	0.61	0.59	0.37
CODESKILL ( Li et al., 2026c )	0.79	0.56	0.56	0.46	0.32

Table 4. Main results with Claude Opus 5 on SWE-bench Verified. Base Agent uses the built-in mini-agent without skills; Append-only adds new skills without updating existing ones. Resolved Rate is the fraction of tasks resolved by the official benchmark harness; Safe Rate (

R_{\rm safe}

) is the fraction of PRs satisfying the safety requirements in Eq. ( 6 ); Existing is the historical existing-test pass rate; Surf. F1 is the mean Maintenance Surface F1 defined in Eq. ( 9 ); Algn. measures historical alignment. Best results are shown in bold.

Repository	Category	Base TS	Ours TS	Base F1	Ours F1
Click	CLI / Dev Tools	0.72	0.83	0.70	0.72
Black	CLI / Formatter	0.53	0.60	0.48	0.67
Rich	CLI / Rendering	0.90	1.00	0.78	0.90
HTTPX	Web / HTTP	0.80	0.93	0.72	0.80
Arrow	Data / Scientific	0.58	0.75	0.71	0.74
Agents	Agent Framework	0.50	0.70	0.85	0.88

Table 5. Cross-repository generalization with Claude Opus 5. Task Success (TS) measures task completion, and Surf. F1 denotes the Maintenance Surface F1 defined in Eq. ( 9 ).

Granularity	Top-1	Top-3	R@5	R@10	MRR
Module	1.00	1.00	0.99	0.99	1.00
File	0.64	0.90	0.92	0.96	0.77
Function	0.25	0.39	0.33	0.33	0.33

Table 6. Code localization at different granularities. Top-

k

indicates whether a correct location is retrieved, R@

k

measures the fraction of correct locations retrieved, and MRR measures the reciprocal rank of the first correct result.

PR Category	Precision	Recall	F1	Support
Usable	0.93	0.94	0.94	260
Repairable	0.87	0.90	0.88	50
Outdated	0.90	0.91	0.90	190
Irrelevant	0.96	0.93	0.94	450
Unsafe	0.85	0.88	0.86	50

Table 7. PR quality classification results. Precision, recall, and F1 for SWE-bench Pro ansible dataset of 1,000 PRs.

Method	TS	Surface F1
Single Sample	0.94	0.80
Best-of- $K$	1.00	0.87
Trace2Skill ( Ni et al., 2026 )	0.78	0.60
SkillLens-inspired composition ( Yu et al., 2026 )	0.61	0.71
Skill Dreaming	1.00	0.87

Table 8. Comparison of PR implementation methods. Task Success (TS) measures successful task completion, and Maintenance Surface F1 evaluates the coverage and precision of required repository changes.

Information Source / Method	TS	Reuse Precision	Recall
Positive PRs only	0.84	0.67	0.78
Review / reply only	0.83	0.48	0.56
Trace2Skill ( Ni et al., 2026 )	0.72	0.61	0.76
Socratic ( Xiao et al., 2026 )	0.83	0.67	0.77
Skill Dreaming	0.89	0.72	0.82

Table 9. Impact of information sources for skill learning. Comparison of task success (TS), skill reuse precision, and recall across different information sources and methods.

	Skill Filtering			Hybrid Retrieval
Method	TS	Surface F1	Precision	TS	Surface F1	Precision
No Supersession	0.67	0.51	0.92	0.83	0.74	0.97
No Stale Filtering	0.72	0.64	0.95	0.78	0.71	0.92
Neither Filter	0.78	0.69	0.93	0.89	0.74	0.97
Append-only	0.89	0.75	0.92	0.89	0.74	0.92
Skill Dreaming (Full)	0.93	0.75	0.93	0.94	0.77	0.90

Table 10. Impact of skill relation management. Comparison of skill filtering strategies across two settings. TS denotes Task Success, and Surface F1 denotes Maintenance Surface F1.

Skill Update Method	TS
Append-only	0.89
Parent-only	0.87
Rewrite-all	0.82
SkillLens-style selective ( Yu et al., 2026 )	0.83
No conflict/supersede	0.78
Learning to Commit-style ( Li et al., 2026a )	0.78

Table 11. Comparison of skill updating methods. TS denotes Task Success.

Method	TS	Surface F1	Tokens	Time (s)
SkillLens ( Yu et al., 2026 )	0.61	0.71	14,300	32.90
Skill Dreaming	0.93	0.80	26,894	36.00

Table 12. Efficiency comparison. Comparison of task success, maintenance quality (Surface F1), token consumption (Tokens), and execution time.

Figure 2. Continuous skill evolution through PR streams. TS, resolved rate, and Surface F1 of Skill Dreaming with Claude Opus 5 on SWE-bench Pro as new PRs arrive and repository-specific skills are incrementally updated.

Automated Software Engineering

Automated Software Engineering uses AI, search, and verification methods to automate tasks across the software development lifecycle.

See all 82 papers

Repo2Skill-Evo: Repository Skills Go Stale in Silence
Aug 22, 2026Chenyuan Duan, Ge Shi, Zineng Mao +10Software Engineering Agents LLM Agent Skill Learning
ByteDance · Peking University · Beijing Jiaotong University
Hierarchical Skill Evolution for Coding Agent
Oct 26, 2026Jun Gao, Yun Peng, Changhai Zhou +10LLM Agent Skill Learning Software Engineering Agents
CODESKILL: Learning Self-Evolving Skills for Coding Agents
May 25, 2026Yanzhou Li, Yiran Zhang, Xiaoyu Zhang +2Coding Agents LLM Agent Skill Learning
Nanyang Technological University · 2Zhejiang University

Aug 22, 2026J/K move · Enter open · S save

Repo2Skill-Evo: Repository Skills Go Stale in Silence

Chenyuan Duan, Ge Shi, Zineng Mao, Ge Zhang+9

ByteDance · Peking University · Beijing Jiaotong University

Software Engineering Agents LLM Agent Skill Learning LLM Agent Evaluation Continual Learning for LLM Agents

Open paper AskPDF arXiv

Large language model (LLM) agents increasingly operate over evolving software repositories, where success depends on repository-specific procedural knowledge: which APIs to call, which scripts to run, and which conventions the current release expects. Agent skills externalize this knowledge into reusable units, and prior work shows that they can improve agent performance. What remains unclear is whether that improvement is durable. The same version specificity that makes a skill useful also makes it fragile: after a release, it may become stale without raising any explicit signal, while continuing to provide obsolete guidance. Externalizing knowledge into a skill can therefore make its decay invisible. We study whether agents can keep this externalized knowledge current. Repo2Skill-Evo casts each release transition as a skill-maintenance task: given a V1 skill set and the official V1-to-V2 patch, an agent must update obsolete skill content while preserving guidance that remains valid. Across 57 real-world repositories and 105 selected release transitions, every evaluated transition invalidates part of the V1 skill set. Yet six frontier agents reach only 29.9%-69.7% avg@3 macro F1 under a patch-grounded removal metric that balances stale-content recall against over-editing precision. Across runs, two opposing errors dominate: incomplete coverage of affected files in the skill set leaves stale content untouched, while overbroad editing is associated with higher recall but lower precision. Repository skills go stale in silence, and even frontier agents cannot reliably maintain them.

Abstract

Automated Software Engineering

Explore similar work

Repo2Skill-Evo: Repository Skills Go Stale in Silence

Hierarchical Skill Evolution for Coding Agent

CODESKILL: Learning Self-Evolving Skills for Coding Agents