How Feedback-Free Self-Evolution Reshapes Agent Skills: A Systematic Study
Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Knowin · The Hong Kong University of Science and Technology
Abstract
Agent skills are documents in which experts write down procedural knowledge, and models increasingly maintain them by rewriting them without feedback. Such rewriting could adapt a skill to the model that uses it or erode the expert knowledge that makes it useful, yet how skills evolve under this feedback-free self-evolution has not been studied. We let models rewrite expert skills round after round without any task, run or judgment and run agents built on the same models with the rewritten skills. Our experiments show that the skills lose value rather than improve, as every model we test performs worse with them on every benchmark, by 15 points on average on SkillsBench after thirty rounds. Sampling each model's prior with skills it writes from the topic alone, we find that rewriting pulls skills toward this prior and gives up first the knowledge outside it. Putting this knowledge back recovers much of the lost pass rate, so an expert's skill is valuable largely for what the model does not write on its own. Building on this insight, SkillDiff marks this knowledge in place and raises the pass rate on SkillsBench by 4.8 points on average without feedback or any change to the skill's text. Project repository: https://github.com/LitaoGuo/How-Self-Evolution-Reshapes-Skills.
Figures & tables
| SkillsBench | LogicBench | MedCalc-Bench | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | 0/10 | 10/10 | 0/10 | 10/10 | 0/10 | 10/10 | |||
| GPT-5.6-Luna | 10 | 83 | 19 | 80 | 20 | 85 | |||
| GPT-6.1-Sol | 7 | 81 | 11 | 82 | 15 | 85 | |||
| MiMo-V2.5-Pro | 32 | 90 | 27 | 90 | 21 | 90 | |||
| Qwen3.7-Plus | 21 | 86 | 22 | 88 | 38 | 87 | |||
| Claude Haiku 5.5 | 37 | 82 | 56 | 92 | 52 | 88 | |||
| Round 30, lost items restored | |||||
|---|---|---|---|---|---|
| Benchmark | Original | Round 30 | 1/10 | 0/10 | All |
| SkillsBench | 40.2 | 25.7 | 30.1 | 34.5 | 35.8 |
| LogicBench | 86.2 | 71.4 | 75.0 | 80.4 | 80.6 |
| MedCalc-Bench | 89.7 | 78.7 | 80.0 | 83.1 | 84.4 |
| Model | pass rate, pp |
|---|---|
| GPT-5.6-Luna | |
| GPT-6.1-Sol | |
| MiMo-V2.5-Pro | |
| Qwen3.7-Plus | |
| Claude Haiku 5.5 | |
| Qwen3.8-Flash |
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| SkillsBench | LogicBench | MedCalc-Bench | ||||
|---|---|---|---|---|---|---|
| Model | R1 | R30 | R1 | R30 | R1 | R30 |
| GPT-5.6-Luna | ||||||
| GPT-6.1-Sol | ||||||
| MiMo-V2.5-Pro | ||||||
| Qwen3.7-Plus | ||||||
| Claude Haiku 5.5 | ||||||
| SkillsBench | LogicBench | MedCalc-Bench | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | Orig. | R30 | Orig. | R30 | Orig. | R30 | |||
| GPT-5.6-Luna | 1.50 | 1.18 | 1.38 | 1.23 | 1.36 | 1.18 | |||
| GPT-6.1-Sol | 1.68 | 1.34 | 1.43 | 1.22 | 1.68 | 1.39 | |||
| MiMo-V2.5-Pro | 1.29 | 1.18 | 1.21 | 1.07 | 1.22 | 1.10 | |||
| Qwen3.7-Plus | 1.41 | 1.28 | 1.32 | 1.18 | 1.31 | 1.23 | |||
| Claude Haiku 5.5 | 1.45 | 1.35 | 1.25 | 1.22 | 1.41 | 1.36 | |||
| SkillsBench | LogicBench | MedCalc-Bench | |||||||
| Model | 0/10 | 1/10 | 0/10 | 1/10 | 0/10 | 1/10 | |||
| After round 1 | |||||||||
| GPT-5.6-Luna | 73.1 | 31.6 | 39.4 | 15.6 | 45.5 | 21.8 | |||
| GPT-6.1-Sol | 75.4 | 33.3 | 40.9 | 16.6 | 57.5 | 18.9 | |||
| MiMo-V2.5-Pro | 23.1 | 11.9 | 25.7 | 4.3 | 26.0 | 8.7 | |||
| Qwen3.7-Plus | 37.5 | 20.1 | 38.3 | 8.7 | 30.6 | 13.2 | |||
| List entries | Paragraphs | Code blocks | Headings | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | 0/10 | 1/10 | 0/10 | 1/10 | 0/10 | 1/10 | 0/10 | 1/10 | ||||
| GPT-5.6-Luna | 90.3 | 42.6 | 89.0 | 41.7 | 92.4 | 59.5 | 85.5 | 40.3 | ||||
| GPT-6.1-Sol | 94.5 | 47.2 | 93.3 | 49.6 | 90.4 | 57.4 | 92.4 | 47.2 | ||||
| MiMo-V2.5-Pro | 68.5 | 33.5 | 68.5 | 26.9 | 73.2 | 30.2 | 50.6 | 23.3 | ||||
| Qwen3.7-Plus | 80.8 | 45.2 | 79.5 | 41.3 | 81.5 | 52.8 | 55.0 | 30.0 | ||||
| Claude Haiku 5.5 | 59.3 | 34.2 | 65.1 | 36.5 | 73.5 | 51.7 | 49.0 | 29.2 | ||||
| Second judge | Main judge, rerun | Third judge | |
|---|---|---|---|
| GPT-6.1-Sol | GPT-5.6-Luna | Claude Haiku 5.5 | |
| Agreement on self-written skills, % | 89.7 | 92.7 | 87.2 |
| Cohen’s | 0.78 | 0.84 | 0.72 |
| Agreement on 0/10 class, % | 88.9 | 94.8 | 86.0 |
| Agreement on rewritten skills, % | 87.2 | 89.8 | 82.4 |
| Cohen’s | 0.74 | 0.79 | 0.64 |
| Main judge | Second judge | Main judge, rerun | Third judge | |
| GPT-5.6-Luna | GPT-6.1-Sol | GPT-5.6-Luna | Claude Haiku 5.5 | |
| 0/10 | 89.2 | 93.3 | 89.9 | 93.4 |
| 1/10 | 39.8 | 40.9 | 38.4 | 52.6 |
| , pp |
| Samples of ten | Agreement on 0/10 class, % | Cohen’s |
|---|---|---|
| Original vs. further 1 | 88.1 | 0.76 |
| Original vs. further 2 | 88.2 | 0.76 |
| Original vs. further 3 | 87.7 | 0.75 |
| Further 1 vs. 2 | 92.0 | 0.84 |
| Further 1 vs. 3 | 91.7 | 0.83 |
| Further 2 vs. 3 | 91.4 | 0.83 |
| Class under the original ten | Items | Stated by the further thirty, % |
|---|---|---|
| 0/10 | 2,757 | 1.1 |
| 1–9/10 | 2,584 | 39.6 |
| 10/10 | 1,264 | 95.1 |
| Dropped outright | Weakened | |||
|---|---|---|---|---|
| Model | 0/10 | 1/10 | 0/10 | 1/10 |
| GPT-5.6-Luna | 55 | 37 | 28 | 48 |
| GPT-6.1-Sol | 60 | 39 | 24 | 43 |
| MiMo-V2.5-Pro | 47 | 40 | 32 | 42 |
| Qwen3.7-Plus | 52 | 43 | 27 | 39 |
| Claude Haiku 5.5 | 40 | 41 | 38 | 42 |
| Model | 0/10 | 1/10 |
|---|---|---|
| GPT-5.6-Luna | 3 | 26 |
| GPT-6.1-Sol | 2 | 21 |
| MiMo-V2.5-Pro | 10 | 42 |
| Qwen3.7-Plus | 6 | 29 |
| Claude Haiku 5.5 | 12 | 38 |
| Qwen3.8-Flash | 2 | 26 |
| 0/10 | Also 0/10 for another model, % | ||
|---|---|---|---|
| Model | items | Observed | Expected |
| GPT-5.6-Luna | 2,757 | 87 | 49 |
| GPT-6.1-Sol | 3,019 | 84 | 48 |
| MiMo-V2.5-Pro | 3,419 | 78 | 47 |
| Qwen3.7-Plus | 3,410 | 78 | 47 |
| Claude Haiku 5.5 | 2,924 | 84 | 48 |
| Held-out | Distance | Pairs closer than the original | ||||
|---|---|---|---|---|---|---|
| Measure | self-written skill | Original | Round 5 | Round 30 | Round 5 | Round 30 |
| TF-IDF in main text | 1.00 | 1.37 | 1.19 | 1.22 | 18/18 | 18/18 |
| Embedding | 1.01 | 2.24 | 1.78 | 1.93 | 17/18 | 15/18 |
| Word frequency | 1.00 | 2.42 | 1.62 | 1.77 | 18/18 | 16/18 |
| Kept after round 30, % | |||
| Model | 0/10 | 10/10 | , pp |
| LogicBench, three chains per skill | |||
| GPT-5.6-Luna | 18–21 | 77–83 | 59–62 |
| GPT-6.1-Sol | 11–11 | 80–83 | 69–72 |
| MiMo-V2.5-Pro | 25–28 | 85–93 | 60–65 |
| Qwen3.7-Plus | 18–26 | 88–90 | 62–72 |
| pass rate, pp | Both models | GPT-5.6-Luna | GPT-6.1-Sol |
|---|---|---|---|
| Against the original skill | |||
| 0/10 items deleted | |||
| 1/10 items deleted | |||
| Between the two deletions | |||
| 0/10 1/10 | |||
| Round-30 skill rewritten by | own | |||
|---|---|---|---|---|
| Agent | Original | GPT-5.6-Luna | GPT-6.1-Sol | other’s, pp |
| GPT-5.6-Luna | 36.8 | 31.0 | 27.6 | |
| GPT-6.1-Sol | 44.8 | 42.5 | 37.9 | |
LLM Agent Skill Learning
Large language model agent skill learning studies how agents acquire, represent, reuse, and refine procedures for multi-step tasks.
See all 431 papers