Cortex: Measurement-Gated Evolution and Pruning of Coding-Agent Skills and Rules
Organizations: Independent researcher
Abstract
Coding agents such as Claude Code also follow text files that developers write for them: a project file, skills the agent may choose to open, and rules that load automatically when the agent reads matching files. We call these files the agent's harness. Developers add to it whenever they correct the agent, but one success proves little: the same agent can pass a task once and fail it the next time. We present Cortex, a tool that tests changes to the harness instead of trusting them. It saves fixes from the developer's sessions as replayable tasks, finds the corrections that recur, writes one skill or rule at a time, and keeps it only if repeated runs with and without it show a net gain and no task made noticeably worse. Existing text is tested the same way before it is deleted, and Cortex neither adds nor deletes text the agent never used. We tested Cortex with Claude Haiku 4.5 on a repository built for the purpose, with conventions the model tends to break, one area with nothing to learn, and a script playing the developer. On held-out tasks, the harnesses it built made the agent follow the conventions far more often than no skills or rules did, left the comparison area unchanged, and added no text read on every turn. Rules, which load automatically, did most of the work; skills helped only when offered with the files a task touches, and those offered on every turn were rarely opened. Cortex kept none of the useless or harmful suggestions we planted, but the agent never opened the useless ones or the harmful skills, and it reached no verdict on the harmful rules, so the test shows that Cortex keeps no unused text, not that it detects damage. Under the stronger Claude Sonnet 5 a single run's harness added little, and on a real open-source project, where the agent was rarely corrected, Cortex found nothing to learn. We release the tool, the lab and its data.
Figures & tables
| Tier | What sits in context | When |
|---|---|---|
| root CLAUDE.md | the whole file | every turn |
| always-on skill | its description; the body when invoked | every turn |
| path-less rule | the whole rule | every turn |
| path-gated skill | its description; the body when invoked | after a matching file is read or written |
| path-scoped rule | the whole rule | after a matching file is read or @-mentioned |
| Term | Meaning | See |
|---|---|---|
| harness | the text files that steer the agent: CLAUDE.md , skills and rules | Section 2 |
| item | a skill or a rule, the harness files Cortex measures | Section 3.2 |
| live harness | the harness the developer is using now | Section 3.4 |
| loading tier | when an item’s text enters the agent’s context ( Table 1 ) | Section 2 |
| task | an episode /harvest saved so it can be replayed: a broken commit, the request, a check and the fix | Section 3.3 |
| check | a script that exits zero when an attempt solved the task | Section 3.3 |
| Hypothesis | Margin | Result | Verdict | Sec. | |
|---|---|---|---|---|---|
| H0 | A check of the lab, not of Cortex: with no skills or rules ( none ), the agent often breaks the house rules, so families A, B, C and E often fail, but it fixes the plain bugs of the control family, D. | On the held-out scenarios under none , D passes at least 90% of the time, and at least three of A, B, C and E pass less than 50%. | Pass rates under none : D 100%; A 0%, B 26%, C 2%, E 55%. D is above its bar, and 3 of the four rule families are below 50%; E is the one above it. | supported | 5.1 |
| H1 | Primary. What Cortex learns carries over to new work. On the held-out scenarios of A, B, C and E together, which no session played, the agent passes more often with the harness a run evolved ( evolved ) than with none . | The gain, evolved minus none , has an interval whose lower end is above zero: even the lowest value the data leave plausible is a gain. | On the 23 rule-family scenarios valid in every run, the pass rate rose from 21.5% to 76.5%: a gain of +55.0 [+34.1, +74.1]. Permutation , the smallest the test can return. | supported | 5.1 |
| H2 | Cortex does no harm where there is nothing to learn. D has no house rule and already passes under none , so the evolved harness cannot help there; it should not make D worse, and no run should keep an item written for D. | The change in D’s pass rate has an interval whose lower end is above −10 points, so that a loss that large is not plausible; and no run keeps an item for D. | D changed by −0.8 [−5.0, 0.0]: the lower end is above −10. The upper end cannot be above zero, since none passes D 100% of the time and leaves nothing to gain. No run kept an item for D. | supported | 5.1 |
| H3 | Cortex puts each kept item in a sensible loading tier, which decides when its text enters the agent’s context, and the item fires (a rule loads, a skill is opened) mainly in the work of its own family. | Every kept item meets three criteria: its form (rule or skill) fits how it loads; its scope is limited; it fires more often on its own family’s rollouts than on the others’. | 15 of the 18 kept items meet all three. The rest are rules on shop/** , a scope wide enough to load on almost every task of every family, and they fail only the third criterion. | not supported | 5.2 |
| H4 | The harness spares the developer corrections while it is being built. In the evaluation runs Cortex’s items go live as they are kept, so the scripted developer should correct the agent less often there than in the control run, which plays the same sessions with /evolve off. | Fewer corrections per session in the evaluation runs than in the control run, counting every session, including those before any item was kept. | 60 corrections in 104 sessions (0.58 per session) in the evaluation runs, against 23 in 26 (0.88) in the control run. With 1 control run, a permutation test cannot return below 0.2, so the difference is descriptive, not a tested effect. | supported | 5.1 |
| H5 | The gates, the rules that decide whether a candidate is kept, give the right verdict on candidates whose right verdict is known: 20 placebos (plausible advice that touches no house rule), 5 harmful ones (advice that contradicts a house rule) and 4 known-good ones (the items of the hand-written harness ideal ). They should rarely keep a placebo, kill every harmful candidate and keep the known-good ones. | The rate at which placebos are kept has an interval whose upper end is below 25%; every harmful candidate is killed; and at least 3 of the 4 known-good ones are kept. | Placebos: 0 of 20 kept; with so few, the rate could still be as high as 17%, below 25%: met. Harmful: none kept, but only 2 of 5 killed, with no verdict on the other 3: not met. Known-good: 2 of 4 kept: not met. | not supported | 5.3 |
| Hypothesis | Margin | Result | Verdict | Sec. | |
|---|---|---|---|---|---|
| H8 | Two harnesses state all four house rules: kitchen , which appends the contribution guide to CLAUDE.md , and ideal , written by hand with the rules known. evolved should come close to them on the rule families, with less always-on text. | evolved minus the better of the two has an interval whose lower end is above −10 points, and evolved has fewer always-on characters than either. | In R1, pass rates: evolved 87.5%, kitchen 76.4%, ideal 48.6%. evolved leads both: kitchen , the better of the two, by 11.1 points [0.0, +22.5], an interval that reaches zero. Always-on characters, in that order: 559, 2,519 and 728. | supported | 5.4 |
| H9 | What the loop learns is repeatable. The 4 evaluation runs are independent, each starting from a fresh copy of the repository; they should learn the same families, as items of similar form. | A run has learned a family when it keeps an item for it. Of the families learned in any run, at least half are learned in every run. | All four rule families were learned in every run: each of the 4 runs kept an item for A, B, C and E, and none for D. Every run but R3 chose the same scopes. | supported | 5.2 |
| H10 | /prune , which re-measures live items by removal, deletes only dead weight. Beside R1’s own 4 items, which it should keep, the prune test plants 4 items: 2 placebo always-on skills, to delete; a gated skill that no task reaches, to skip; and an always-on skill that never fires, to report as unmeasured. | Every planted item gets the verdict expected of it, and no item that never fired is deleted. | /prune kept R1’s own 4 items. Of the 4 planted items, 2 got the expected verdict (6 of 8 overall): one placebo was deleted and the unreachable skill skipped. The other placebo was kept: 2 tasks lost runs without it, and the agent never opened it. The skill that never fires got no verdict. No item that never fired was deleted. | not supported | 5.5 |
| H11 | A skill is offered to the agent by its description, one line in a listing; its body enters the context only if the agent opens it. The description should carry much of the skill’s effect by itself. desc-only is evolved with each skill’s body replaced by neutral filler. | desc-only recovers at least 50% of evolved ’s gain over none , with the lower end of that share’s interval above zero. | It recovered 79% [58%, 100%], which meets the letter of the margin. But only 1 of R1’s 4 items is a skill and could be rewritten; the rest are rules and kept their words. In C, the rewritten skill’s family, it recovered 16%. One skill cannot settle the question. | inconclusive | 5.4 |
| H12 | The development run, D0, used an earlier version of Cortex than v1.0-eval , the version evaluated. Its harness, measured on the same held-out scenarios with the evaluation’s tooling, should gain in line with the evaluation runs. | D0’s gain lies inside the interval of the evaluation runs’ gain, taken over every valid scenario. | D0’s harness gained +42.5 points, inside the evaluation runs’ interval, [+35.4, +75.0], and within the range of the single runs’ own gains, +32.5 to +66.7. | supported | 5.1 |
| H13 | A harness evolved for one model may be partly redundant under a stronger one. /prune , run on R1’s harness with its rollouts on Claude Sonnet 5 instead of Claude Haiku 4.5, should find items the weaker model needed and the stronger one no longer does. | /prune finds at least one such item removable. | 2 of the harness’s 4 items are removable under Claude Sonnet 5: the money rule and the exporter checklist. The money rule is the clear case, since the stronger model passed every held-out money scenario with no harness; the exporter checklist is less certain. | supported | 5.5 |
| Pass rate | ||||||||
|---|---|---|---|---|---|---|---|---|
| Family | Scenarios | Rollouts | none | evolved | Gain | 95% CI | Holm | |
| A, changelog | 5 | 200 | 0% | 97% | +97.0 | [+91.0, +100.0] | 0.0001 | 0.0004 |
| B, money | 6 | 240 | 26% | 79% | +53.3 | [+17.5, +88.3] | 0.0001 | 0.0004 |
| C, exporters | 6 | 240 | 2% | 52% | +50.0 | [+25.0, +73.3] | 0.0001 | 0.0004 |
| E, time | 6 | 240 | 55% | 82% | +26.7 | [+1.7, +60.0] | 0.0001 | 0.0004 |
| A, B, C, E together | 23 | 920 | 22% | 77% | +55.0 | [+34.1, +74.1] | 0.0001 | |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Candidate | Tier | Fired | KEEP | KILL | RERUN | Recorded KILL | |
|---|---|---|---|---|---|---|---|
| placebo, about a topic | always-on skill | 10 | 0 | 0 | 7 | 3 | 10 |
| placebo, about a moment | always-on skill | 10 | 0 | 0 | 5 | 5 | 10 |
| harmful | rule | 3 | 3 | 0 | 0 | 3 | 3 |
| harmful | always-on skill | 2 | 0 | 0 | 2 | 0 | 2 |
| known-good | rule | 1 | 1 | 1 | 0 | 0 | 0 |
| known-good | gated skill | 1 | 1 | 1 | 0 | 0 | 0 |
| Fires (%) | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Run | Fam. | Item | Form | Scope | Chars | own | other | Score | Held-out |
| R1 | A | changelog-requirement | rule | shop/** | 518 | 98 | 97 | 3 | 25/25 |
| R1 | B | billing-helpers | rule | shop/billing/** | 317 | 96 | 21 | 3 | 28/30 |
| R1 | C | exporter-checklist | skill | shop/plugins/** | 1,135 | 100 | 0 | 3 | 30/30 |
| R1 | E | shop-clock-rule | rule | shop/** | 327 | 98 | 97 | 3 | 29/30 |
| R2 | A | changelog-user-visible | rule | shop/** | 529 | 98 | 97 | 3 | 25/25 |
LLM Agent Harnesses
LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.
See all 357 papers