cs.SEOct 9, 2026

Cortex: Measurement-Gated Evolution and Pruning of Coding-Agent Skills and Rules

Authors: Samuele Giampieri

Organizations: Independent researcher

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Coding agents such as Claude Code also follow text files that developers write for them: a project file, skills the agent may choose to open, and rules that load automatically when the agent reads matching files. We call these files the agent's harness. Developers add to it whenever they correct the agent, but one success proves little: the same agent can pass a task once and fail it the next time. We present Cortex, a tool that tests changes to the harness instead of trusting them. It saves fixes from the developer's sessions as replayable tasks, finds the corrections that recur, writes one skill or rule at a time, and keeps it only if repeated runs with and without it show a net gain and no task made noticeably worse. Existing text is tested the same way before it is deleted, and Cortex neither adds nor deletes text the agent never used. We tested Cortex with Claude Haiku 4.5 on a repository built for the purpose, with conventions the model tends to break, one area with nothing to learn, and a script playing the developer. On held-out tasks, the harnesses it built made the agent follow the conventions far more often than no skills or rules did, left the comparison area unchanged, and added no text read on every turn. Rules, which load automatically, did most of the work; skills helped only when offered with the files a task touches, and those offered on every turn were rarely opened. Cortex kept none of the useless or harmful suggestions we planted, but the agent never opened the useless ones or the harmful skills, and it reached no verdict on the harmful rules, so the test shows that Cortex keeps no unused text, not that it detects damage. Under the stronger Claude Sonnet 5 a single run's harness added little, and on a real open-source project, where the agent was rarely corrected, Cortex found nothing to learn. We release the tool, the lab and its data.

Figures & tables

Figure 1: The Cortex loop. /harvest turns a session that went from broken to working into a task; /evolve measures one candidate against the live harness and keeps it only if it helped, fired, and cost no task more than a small tolerance; /prune re-measures live items by removal and keeps any item it never saw fire.
TierWhat sits in contextWhen
root CLAUDE.mdthe whole fileevery turn
always-on skillits description; the body when invokedevery turn
path-less rulethe whole ruleevery turn
path-gated skillits description; the body when invokedafter a matching file is read or written
path-scoped rulethe whole ruleafter a matching file is read or @-mentioned
Table 1: Loading tiers in Claude Code [ 4 , 5 ] . The rows for items with paths are as Cortex verified them on version 2.1.276, the oldest it accepts; the current documentation also counts writes and edits among the triggers of a path-scoped rule. The tier, not the text, decides how often an item is paid for, and whether it is loaded (rules) or must be chosen by the agent (skills).
TermMeaningSee
harnessthe text files that steer the agent: CLAUDE.md , skills and rulesSection 2
itema skill or a rule, the harness files Cortex measuresSection 3.2
live harnessthe harness the developer is using nowSection 3.4
loading tierwhen an item’s text enters the agent’s context ( Table 1 )Section 2
taskan episode /harvest saved so it can be replayed: a broken commit, the request, a check and the fixSection 3.3
checka script that exits zero when an attempt solved the taskSection 3.3
Figure 2: The terms this paper uses for Cortex, in roughly the order a reader meets them, with the subsection that explains each in full.
Figure 3: What /harvest does at the end of a working session. It asks two questions about the session (step 1). A : did something go from broken to working? B : did the developer correct the agent, even once? In most sessions the answer to both is no, and nothing is recorded. If the answer to A is yes, the command saves that change as a task that can be replayed later (step 2): the developer’s request, word for word; the commit where the code was still broken; the fix that made it work; and a check, a short script that says whether an attempt solved the problem. Before the task is kept, preflight tests the check itself in a throwaway copy of the repository (step 3). The check must fail on the broken code, pass once the fix is applied, and still pass when the fix’s changes to test code are left out, so that an agent is judged on the code it writes, not on reproducing the developer’s own test. A task that fails any of the three is set aside and gets one repair; if it fails again, it is deleted ( Section 3.3 ). If the answer to B is yes, the command writes one line per correction to a log of lessons : the date, the task, what the agent did, and what it should have done (step 4). A lesson is written even when the same one is already there, because /evolve counts the repeats to find the problems that recur. Finally, the command reports which items of the current harness already cover the files the fix touched (step 5). /harvest only records: it never runs the agent on a task and never writes a skill or rule, which is /evolve ’s job. The tasks it saves are used next by /evolve , to measure new candidates, and by /prune , to measure what removing an item costs.
Figure 4: One /evolve cycle, which tests a single new skill or rule before it may join the harness. The model proposes and writes the candidate, but never decides its fate: every verdict comes from a scoring script ( Table 2 ). Steps 1 to 4 prepare the cycle and run the agent on no task; the first three can end the cycle early (grey badges). Step 1 re-runs preflight to make sure the saved tasks still tell broken code from fixed code; if too few are left, the cycle stops. Step 2 looks for one theme , a problem that recurs: a lesson logged at least 3 times, or a cause behind 2 or more failing tasks. If none clears that bar, the cycle ends as BARREN and runs no rollouts. Step 3 chooses where the candidate should load: if the fixes behind the theme sit in one folder, it becomes a rule or a skill tied to that folder; if they are spread across the repository, it becomes a skill offered on every turn. A fact the agent breaks during unrelated work becomes a line in CLAUDE.md , and a check that must always run becomes a hook , a command Claude Code runs by itself at a fixed point of a session; such a change is written and recorded in the journal , Cortex’s running log of every cycle and verdict ( .evolve/journal.md ), but not measured. Step 4 writes the candidate under a new name, outside the live harness. Steps 5 to 7 measure it: the agent attempts each task with the live harness ( base ) and with the harness plus the candidate ( cand ), each attempt in its worker’s own clone of the repository, reset and verified before every attempt ( Figure 6 ). The screen (step 5) is cheap, 2 runs per task on the theme’s tasks only; a candidate that gained nothing, or that never fired (a rule never loaded, a skill never opened), is killed here. The confirm (step 6) is stricter, 3 runs per task on every valid task, so that it also sees what the candidate might break. The candidate is kept only if it wins overall, no task collapses, no task that never failed starts failing, and it fired at least once. Because one unlucky run can look like a broken task, a candidate that failed only on such a drop gets a recheck (step 7, drawn dashed): the tasks that dropped are run again, and the candidate is killed only if the drop repeats. A sweep whose data cannot be trusted, with runs missing or invalid, or in which no task the candidate reached could show a gain, is repeated rather than decided (RERUN). Step 8 acts on the verdict: a KEEP moves the candidate into the live harness and records the confirm sweep’s scores as the new baseline, with no new rollouts; a KILL buries it in the graveyard , the folder of rejected ideas ( .evolve/graveyard/ ), with its reason, so the same idea is not proposed again unchanged. Step 9 records the cycle in the journal; after 2 cycles in a row without a KEEP, cortex cycle signals the loop to stop. Badge colours follow Figure 1 : green, the item stays; red, it goes; grey, no decision.
Figure 5: One /prune pass, which tests the items already in the harness and deletes one only when removing it costs nothing measurable. It is meant to run monthly and after every change of model, since a stronger model may already do what an item teaches. Steps 1 to 3 plan the pass and run the agent on no task. Step 1 checks that the tasks are still valid, that the routing is sound (no name is both a skill and a rule, no symlink leaves the repository or dangles), and that the rollouts are not pinned to a model other than the one in use; if any check fails, the pass stops. Step 2 chooses which items to test: after a model change, every skill and rule, the largest always-on items first; otherwise, the few most in doubt. Each item is tested only on the tasks where it can load; an always-on skill, only on the tasks where earlier sweeps saw it fire, and not at all if they never did. How often an item is used helps choose what to test, but never decides a deletion. Step 3 shows the developer the plan with its expected rollouts, time and cost, and nothing runs until the developer approves it, possibly dropping some items. Steps 4 to 6 then take one item at a time: the agent attempts each of its tasks 3 times with the live harness ( base ) and without the item ( cand ), and the scoring script gives the verdict. ACCEPT: the item fired in at least one base run, and removing it lost nothing measurable, so it is deleted and buried in the graveyard. REJECT: removing it lost too many runs, made a task collapse, or made a task that never failed start failing, so it stays. UNMEASURED: the item never fired in any base run, so the test says nothing about it, and it stays; an item for a rare situation looks exactly like dead weight until that situation comes. RERUN: the data cannot be trusted, and the sweep is repeated. A deleted item is removed before the next one is measured, so every test starts from the harness as it really is. Step 7 can also move advice to a better place: an always-on skill used in only one folder can become a rule or a skill tied to that folder, and overlapping items can be merged; each such change is approved and measured on its own. Step 8 records every verdict in the journal and closes the plan, noting the model it ran on ( Section 3.6 ).
Figure 6: What happens in one rollout, a single attempt by the agent at a task. The base and cand arms take exactly the same steps, in alternation, and differ only in the harness that is installed. Once per sweep, Cortex freezes a copy of the harness and of the tasks (step 1), so that an edit made while the sweep runs cannot reach it, and gives each worker its own clone of the repository, which cannot push to the real one (step 2). Then, before every rollout, the clone is reset to the task’s broken commit and that arm’s harness is installed (step 3), and the task’s check is run once to confirm that it still fails (step 4): a task that already passes before the agent starts would measure nothing. The agent then works alone, with no human, on the task’s request, within a limit of 10 minutes (step 5). Cortex records which skills and rules entered the agent’s context during the attempt (step 6). Finally, the task’s own check decides the result, after the cached build files are deleted, since a stale cache could make old code look as if it passes (step 7). Each attempt ends in one of two ways. It is scored : a pass if the check succeeds, a fail if it does not or the agent ran out of time. Or it is invalid , because it could not run fairly: the reset failed, the task already passed, a service the task needs was down, the agent exited with an error (for example a login failure or a rate limit), the check hung, or the agent ran out of time while the machine was oversubscribed. Invalid is not the same as fail, and a sweep with even one invalid rollout is run again rather than scored.
Figure 8
Figure 9: Adding and removing are one test, pointed both ways. /evolve compares the live harness with the harness plus a candidate; /prune compares it with the harness minus a live item. Both count only the tasks on which the item was in the agent’s context: a candidate that never fired is never kept, and a live item that never fired is never deleted.
Figure 10: The lab. Top: the repository, and its five task families: each rule family is built around one house rule written in CONTRIBUTING.md , with the mistake an agent typically makes, what catches it, and the counts of training and held-out scenarios; family D, plain bugs with no rule, is the control. Bottom left: one correction from start to finish, for the rule on time. Bottom right: the training scenarios are played as sessions; the held-out scenarios are kept for the benchmark.
Figure 11: One run, end to end. (1) The rounds, and the family of each session. (2) One session: the teammate’s commit, the agent, the scripted reviewer, whose fixed reply sends the agent back at most 3 times, and /harvest . (3) The /evolve cycle after each round’s sessions ( Figure 4 ). (4) The runs: evaluation and control share one setup; the development run is reported apart.
Figure 12: The benchmark. Top: how each run’s final harness is examined, from the scenarios to the gain, which is the difference between the pass rates of evolved and none . Bottom: the arms, which share the scenarios, the model and the oracle and differ only in the harness, with the question each one answers, the runs it was measured in, and its number kk of attempts per held-out scenario. none and evolved also run the training scenarios.
Figure 13: The four further experiments. The gate test and the prune test run the real pipeline on copies of finished runs, and each item’s right verdict is known before it runs; the model change and the second repository ask how far the results carry. The badges give each item’s right verdict, not the verdict the pipeline reached ( Sections 5.3 and 5.5 ).
 HypothesisMarginResultVerdictSec.
H0A check of the lab, not of Cortex: with no skills or rules ( none ), the agent often breaks the house rules, so families A, B, C and E often fail, but it fixes the plain bugs of the control family, D.On the held-out scenarios under none , D passes at least 90% of the time, and at least three of A, B, C and E pass less than 50%.Pass rates under none : D 100%; A 0%, B 26%, C 2%, E 55%. D is above its bar, and 3 of the four rule families are below 50%; E is the one above it.supported5.1
H1Primary. What Cortex learns carries over to new work. On the held-out scenarios of A, B, C and E together, which no session played, the agent passes more often with the harness a run evolved ( evolved ) than with none .The gain, evolved minus none , has an interval whose lower end is above zero: even the lowest value the data leave plausible is a gain.On the 23 rule-family scenarios valid in every run, the pass rate rose from 21.5% to 76.5%: a gain of +55.0 [+34.1, +74.1]. Permutation p=0.0001p=0.0001 , the smallest the test can return.supported5.1
H2Cortex does no harm where there is nothing to learn. D has no house rule and already passes under none , so the evolved harness cannot help there; it should not make D worse, and no run should keep an item written for D.The change in D’s pass rate has an interval whose lower end is above −10 points, so that a loss that large is not plausible; and no run keeps an item for D.D changed by −0.8 [−5.0, 0.0]: the lower end is above −10. The upper end cannot be above zero, since none passes D 100% of the time and leaves nothing to gain. No run kept an item for D.supported5.1
H3Cortex puts each kept item in a sensible loading tier, which decides when its text enters the agent’s context, and the item fires (a rule loads, a skill is opened) mainly in the work of its own family.Every kept item meets three criteria: its form (rule or skill) fits how it loads; its scope is limited; it fires more often on its own family’s rollouts than on the others’.15 of the 18 kept items meet all three. The rest are rules on shop/** , a scope wide enough to load on almost every task of every family, and they fail only the third criterion.not supported5.2
H4The harness spares the developer corrections while it is being built. In the evaluation runs Cortex’s items go live as they are kept, so the scripted developer should correct the agent less often there than in the control run, which plays the same sessions with /evolve off.Fewer corrections per session in the evaluation runs than in the control run, counting every session, including those before any item was kept.60 corrections in 104 sessions (0.58 per session) in the evaluation runs, against 23 in 26 (0.88) in the control run. With 1 control run, a permutation test cannot return pp below 0.2, so the difference is descriptive, not a tested effect.supported5.1
H5The gates, the rules that decide whether a candidate is kept, give the right verdict on candidates whose right verdict is known: 20 placebos (plausible advice that touches no house rule), 5 harmful ones (advice that contradicts a house rule) and 4 known-good ones (the items of the hand-written harness ideal ). They should rarely keep a placebo, kill every harmful candidate and keep the known-good ones.The rate at which placebos are kept has an interval whose upper end is below 25%; every harmful candidate is killed; and at least 3 of the 4 known-good ones are kept.Placebos: 0 of 20 kept; with so few, the rate could still be as high as 17%, below 25%: met. Harmful: none kept, but only 2 of 5 killed, with no verdict on the other 3: not met. Known-good: 2 of 4 kept: not met.not supported5.3
Table 3: The hypotheses tested. The margin is the result a measurement must reach for the hypothesis to count as supported, and the verdict is what the margin gives, or inconclusive when the measurement could not decide the question. none is the harness with no skills or rules, and evolved the one a run built; A, B, C and E are the four rule families, and D is the control. Differences are in percentage points of pass rate, evolved minus none unless the row says otherwise, and the two numbers in brackets are the lower and upper ends of a 95% CI. The last column is the subsection that reports each one.
 HypothesisMarginResultVerdictSec.
H8Two harnesses state all four house rules: kitchen , which appends the contribution guide to CLAUDE.md , and ideal , written by hand with the rules known. evolved should come close to them on the rule families, with less always-on text.evolved minus the better of the two has an interval whose lower end is above −10 points, and evolved has fewer always-on characters than either.In R1, pass rates: evolved 87.5%, kitchen 76.4%, ideal 48.6%. evolved leads both: kitchen , the better of the two, by 11.1 points [0.0, +22.5], an interval that reaches zero. Always-on characters, in that order: 559, 2,519 and 728.supported5.4
H9What the loop learns is repeatable. The 4 evaluation runs are independent, each starting from a fresh copy of the repository; they should learn the same families, as items of similar form.A run has learned a family when it keeps an item for it. Of the families learned in any run, at least half are learned in every run.All four rule families were learned in every run: each of the 4 runs kept an item for A, B, C and E, and none for D. Every run but R3 chose the same scopes.supported5.2
H10/prune , which re-measures live items by removal, deletes only dead weight. Beside R1’s own 4 items, which it should keep, the prune test plants 4 items: 2 placebo always-on skills, to delete; a gated skill that no task reaches, to skip; and an always-on skill that never fires, to report as unmeasured.Every planted item gets the verdict expected of it, and no item that never fired is deleted./prune kept R1’s own 4 items. Of the 4 planted items, 2 got the expected verdict (6 of 8 overall): one placebo was deleted and the unreachable skill skipped. The other placebo was kept: 2 tasks lost runs without it, and the agent never opened it. The skill that never fires got no verdict. No item that never fired was deleted.not supported5.5
H11A skill is offered to the agent by its description, one line in a listing; its body enters the context only if the agent opens it. The description should carry much of the skill’s effect by itself. desc-only is evolved with each skill’s body replaced by neutral filler.desc-only recovers at least 50% of evolved ’s gain over none , with the lower end of that share’s interval above zero.It recovered 79% [58%, 100%], which meets the letter of the margin. But only 1 of R1’s 4 items is a skill and could be rewritten; the rest are rules and kept their words. In C, the rewritten skill’s family, it recovered 16%. One skill cannot settle the question.inconclusive5.4
H12The development run, D0, used an earlier version of Cortex than v1.0-eval , the version evaluated. Its harness, measured on the same held-out scenarios with the evaluation’s tooling, should gain in line with the evaluation runs.D0’s gain lies inside the interval of the evaluation runs’ gain, taken over every valid scenario.D0’s harness gained +42.5 points, inside the evaluation runs’ interval, [+35.4, +75.0], and within the range of the single runs’ own gains, +32.5 to +66.7.supported5.1
H13A harness evolved for one model may be partly redundant under a stronger one. /prune , run on R1’s harness with its rollouts on Claude Sonnet 5 instead of Claude Haiku 4.5, should find items the weaker model needed and the stronger one no longer does./prune finds at least one such item removable.2 of the harness’s 4 items are removable under Claude Sonnet 5: the money rule and the exporter checklist. The money rule is the clear case, since the stronger model passed every held-out money scenario with no harness; the exporter checklist is less certain.supported5.5
Table 4: The hypotheses tested, continued.
   Pass rate    
FamilyScenariosRolloutsnoneevolvedGain95% CIppHolm pp
A, changelog52000%97%+97.0[+91.0, +100.0]0.00010.0004
B, money624026%79%+53.3[+17.5, +88.3]0.00010.0004
C, exporters62402%52%+50.0[+25.0, +73.3]0.00010.0004
E, time624055%82%+26.7[+1.7, +60.0]0.00010.0004
A, B, C, E together2392022%77%+55.0[+34.1, +74.1]0.0001 
Table 5: The held-out gain, family by family: evolved against none on the scenarios valid in every evaluation run, R1 to R4. Pass rates are over rollouts, and Rollouts counts both arms. The gain is the paired difference in pass rate, in points, averaged over the scenarios of each run and then over the runs; its 95% CI is the two-way bootstrap and pp the permutation test ( Section 4 ). Holm’s correction adjusts the four rule families’ pp -values for having made four tests. In the rule-family rows no shuffle reached the observed gain, so each pp is the smallest the test can return with 10,000 shuffles; the test keeps the measured scenarios and runs fixed, while the interval also allows for which ones were drawn. Family D is the control.
Figure 14: Held-out results on the scenarios valid in every run. Left: pass rate; each bar averages the 4 evaluation runs, and each dot is one run, R1 to R4 from left to right. Right: evolved minus none , with the two-way bootstrap interval over runs and scenarios. The bottom row is the A/A check, the none harness measured twice in R1 over all five families, whose interval resamples scenarios only. Family D is the control.
Figure 15: Why held-out rollouts fail, in the 4 evaluation runs, on the scenarios valid in every run. Each bar is all of an arm’s rollouts on a family, split by the first check the oracle found failing: the agent edited the teammate’s test (hatched), left it unchanged but did not make it pass (light), or passed the tests and broke the house rule (solid); white is a pass. The number inside a segment counts its rollouts, and the number at the right is the share that failed. Without a harness, nearly every rule-family failure passed its tests and broke the house rule.
Figure 19
Figure 18: What each evaluation run kept, by house rule, and whether it reached the held-out scenarios: the share of the family’s held-out evolved rollouts (scenarios valid in every run) in which the run’s items for that family fired (a rule loaded, a skill opened), and the run’s gain on the family. R3’s money rules, which name 5 files, loaded in none.
Figure 19: A rule is loaded; a skill must be chosen. Claude Code injects a path-scoped rule whenever the agent reads a matching file. A skill is offered by its description, and its body enters the context only if the agent opens it: in their own sweeps and in the gate test, path-gated candidates were opened in some of the rollouts that listed them, and always-on ones in none. Of the items the loop kept, 16 are rules and 2 are path-gated skills; most candidates were rules too.
Figure 22
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

CandidateTiernnFiredKEEPKILLRERUNRecorded KILL
placebo, about a topicalways-on skill10007310
placebo, about a momentalways-on skill10005510
harmfulrule330033
harmfulalways-on skill200202
known-goodrule111000
known-goodgated skill111000
Appendix
Table 6: The gate test by kind of candidate. nn is the number of candidates of that kind. Fired counts the candidates that fired at least once on their screen (a skill opened, a rule loaded). KEEP, KILL and RERUN are score.sh ’s verdicts, after the confirm sweep for the candidates that reached one; the last column is the verdict the test’s procedure recorded, which stops at the screen when the gain is not positive or the candidate never fired. The harmful rule small-changes-need-no-changelog has no paths and loads on every turn.
Figure 22: Every confirm sweep of /evolve in the 4 evaluation runs, scored again from its rollouts: the runs the candidate won net ( kNkN in Equation 3 ) and its worst drop on one task ( DD ), marked by the candidate’s final verdict (KEEP a dot, KILL a cross, RERUN a hollow circle), after a recheck where one ran. The dashed line is gate 4’s threshold. use-rate-helpers , whose sweep score.sh left unscored and the /evolve agent buried, sits among the KEEPs.
      Fires (%)  
RunFam.ItemFormScopeCharsownotherScoreHeld-out
R1Achangelog-requirementruleshop/**5189897325/25
R1Bbilling-helpersruleshop/billing/**3179621328/30
R1Cexporter-checklistskillshop/plugins/**1,1351000330/30
R1Eshop-clock-ruleruleshop/**3279897329/30
R2Achangelog-user-visibleruleshop/**5299897325/25
Appendix
Table 7: Every item the loop kept in the 4 evaluation runs. Chars is the item’s length; none is always-on, since every item has paths . Fires is the share of the run’s evolved benchmark rollouts, held-out and training, on the item’s own family and on the others, in which it fired (a rule loaded, a skill opened): the tier rubric’s third part. Score is how many of the rubric’s three criteria the item meets, out of three ( Section 5.2 ). Held-out counts the family’s held-out evolved rollouts, on the scenarios valid in every run, in which the item fired.
Figure 23: Every held-out scenario under every arm of the 4 evaluation runs: the pass rate of each scenario, in percent, in the run that measured the arm, darker for a higher rate. HA6 was excluded from R4 in every arm ( Section 4 ). R3’s money scenarios scored the same under both arms: the run’s money rules loaded in none of those rollouts ( Section 5.2 ).
More in this topic

LLM Agent Harnesses

LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.

See all 357 papers

Explore similar work

CardsList
Apr 13, 2026

Guardrails Beat Guidance: A Large-Scale Study of Rules, Skills, and Persistent Configuration for Coding Agents

Random rules improve a coding agent's task performance as much as expert-curated ones (both +13.8+13.8pp on a discriminative subset of SWE-bench Verified), and in our data every individually beneficial rule is a negative constraint ("do not refactor unrelated code"), while every individually harmful one is a positive directive ("follow code style"). We arrive at these findings through the first large-scale controlled study of agent rule files (CLAUDE.md, .cursorrules, and the broader family of agent skills, plugin manifests, and persona definitions): we scrape 679 rule files (25{,}532 rules) from GitHub and conduct over 5{,}000 agent runs of Claude Code with Claude Opus 4.6 on SWE-bench Verified. Three patterns emerge. (i) Rule polarity cleanly separates beneficial from harmful rules; we read this through the lens of potential-based reward shaping (PBRS). (ii) Performance gains are largely content-independent: random, shuffled, mismatched-domain, and unconverted-format rule files all match curated rules, pointing to a context priming mechanism. (iii) Individual rules often appear harmful in isolation yet do not visibly accumulate damage in ensemble: pass rates remain stable across rule counts from 0 to 50. These findings expose a hidden reliability risk in the rapidly growing ecosystem of community-authored rules and skills, and they yield a clear principle for safer agent configuration: constrain what agents must not do, rather than prescribing what they should.

Xing Zhang, Guanghui Wang, Yanwei Cui +4

AWS Generative AI Innovation Center · HSBC Holdings Plc., HSBC Technology Center, China.

Software Engineering AgentsLLM Agent ReliabilityLLM Guardrails
AskPDF
Oct 16, 2026

Cortex: A Fixed-Point Theory of Governed Coding Agents - Technical Report

A coding agent combines a large language model (LLM) with a harness that plans, edits, and executes code. We study Cortex, a supervisory layer for pre-execution governance, requirement tracking, and iterative validation and repair. We formalize requirement closure on a finite lattice. For a fixed requirement set and a sound, monotone validation rule with persistent evidence, exhaustive iteration reaches the least fixed point with no more strict increases than there are requirements. Fair consequence scheduling reaches the same closure. A separate stochastic model bounds expected completion time under a uniform positive-progress assumption. These results depend on their stated assumptions; artifact repair, fallible validation, and finite execution budgets do not satisfy them automatically. We define Gauntlet's trajectory-similarity and build-gated scores, attack-success estimators, Wilson intervals, and case-cluster bootstrap summaries. Effective-feedback accounting counts newly certified requirements, rather than estimating their information value or utility. The five evaluation families support raw-versus-governed comparisons. We report retained benchmark observations, including rescoring of existing artifacts; constructed outcomes are excluded. Unequal observation sets and incomplete provenance prevent causal claims about governance or measured long-horizon scaling.

Marius-Constantin Dinu, Florian Zeba

Alpha Omega Labs

AI Coding AgentsAI Agent GovernanceProcess-Based Agent Evaluation
AskPDF
Jul 12, 2026

When Does Restricting a Coding Agent to execute_code Help? A Regime ×\times Agent-Design Ablation

Modern coding agents expose multiple tool surfaces -- IDE primitives, bash, and Model Context Protocol (MCP) code-execution -- and the field has shipped three contradictory claims about which one matters. We run the missing crossed comparison: an integrity-clean three-arm ablation (baseline / bash_only / code_only) on synthetic computation tasks and SWE-bench Mini modification tasks, holding model, harness, and prompts fixed, with two agents (Claude Code, OpenAI Codex CLI) so the comparison spans both regime and agent-design axes. Across the four resulting (regime, agent) cells, restricting the agent to a single execute_code MCP tool is cheaper than -- or statistically tied with -- its cheapest tool-rich rival in three cells (significantly on Artifact/Claude and SWE-bench/Codex; directionally on Artifact/Codex), with pass rates statistically tied within each cell. The lone exception is SWE-bench/Claude, where code_only is directionally costlier (+14.4%, not significant); a conditional-cost analysis localizes that gap to failure-cost on doomed-run trajectories, not a per-edit tax on successful runs. Two implications: the cheapest tool surface is jointly determined by task regime and agent design rather than by either axis alone, and the headline cost signal lives in cache-adjusted cost -- not pass rate, which is invariant across surfaces at the model sizes we evaluate. The benchmark harness, task suite, and analysis code are available at https://github.com/hyang0129/onlycodes.

Hong Yang, Qi Yu, Travis Desell

Rochester Institute of Technology Rochester, New York, USA

Coding AgentsTool-Using AgentsCode Generation EvaluationTool SelectionCode Generation Benchmarks
AskPDF