Random rules improve a coding agent's task performance as much as expert-curated ones (both +13.8pp on a discriminative subset of SWE-bench Verified), and in our data every individually beneficial rule is a negative constraint ("do not refactor unrelated code"), while every individually harmful one is a positive directive ("follow code style"). We arrive at these findings through the first large-scale controlled study of agent rule files (CLAUDE.md, .cursorrules, and the broader family of agent skills, plugin manifests, and persona definitions): we scrape 679 rule files (25{,}532 rules) from GitHub and conduct over 5{,}000 agent runs of Claude Code with Claude Opus 4.6 on SWE-bench Verified. Three patterns emerge. (i) Rule polarity cleanly separates beneficial from harmful rules; we read this through the lens of potential-based reward shaping (PBRS). (ii) Performance gains are largely content-independent: random, shuffled, mismatched-domain, and unconverted-format rule files all match curated rules, pointing to a context priming mechanism. (iii) Individual rules often appear harmful in isolation yet do not visibly accumulate damage in ensemble: pass rates remain stable across rule counts from 0 to 50. These findings expose a hidden reliability risk in the rapidly growing ecosystem of community-authored rules and skills, and they yield a clear principle for safer agent configuration: constrain what agents must not do, rather than prescribing what they should.
Figures & tables
Figure 1: Headline findings. (a) All seven rule conditions outperform the no-rule baseline by 7–14pp, yet random rules match curated rules (both 63.8%), pointing to a context priming effect rather than content-specific instruction. (b) Per-rule ablation on 18 curated rules: shaping rules (green, removal hurts) are all negative constraints; distorting rules (orange, removal helps) are all positive directives.
Figure 2: Taxonomy of 25,532 rules from 679 GitHub rule files, classified by an LLM classifier (87% agreement with human labels). Project-specific rules dominate (64.9%); our experiments use the five transferable categories.
Figure 3: Rule quantity and type ( n=58 tasks each). (a) Pass rate vs. rule count: individual seeds (dots), mean ± std (line with shading). No phase transition; 50 rules slightly outperform 0 rules. (b) Pass rate by rule type. State-dependent tool/process rules score highest; state-independent architecture rules score lowest (10.4pp spread), consistent with P1.
Rule
Polarity
Type
Effect
p
No unrelated refactor
“don’t”
safety
shaping
.016
No new dependencies
“don’t”
safety
shaping
.508
No unrelated files
“don’t”
safety
shaping
.688
Read test files
“do”
process
distorting
.227
Follow code style
“do”
style
distorting
.180
Handle edge cases
“do”
style
distorting
.289
Table 1: Non-inert rules from per-rule ablation ( n=35 tasks; cf. Figure 1 b). Shaping : removal hurts performance; distorting : removal helps. p -values from McNemar’s exact test. The 11 inert rules ( ∣Δ∣≤5 pp) are omitted; see Appendix D for the complete table.
Figure 4: Individual rule effects ( n=35 tasks). (a) Per-rule distortion: each of the 18 curated rules applied individually to 17 baseline-solved tasks. 14/18 rules break ≥ 2 tasks; the worst break 4/17. (b) Superposition test on 5 rule pairs: observed vs. additive-expected pass rate. Three pairs ( + ) are approximately additive; two ( × ) show strong non-linearity (max error 22.9pp).
Box 1: Rule authoring guidelines for coding agents.
Scoped to Claude Code-class agents on SWE-bench-style tasks.
Do
Write negative constraints (“do not refactor unrelated code”).
Table 4: Experiment 4a: complete per-rule ablation ( n=35 tasks). Positive Δ means the rule is helpful (removal hurts); sorted by Δ descending. Threshold for shaping / distorting : ∣Δ∣>5 pp.
None
A
B
A+B
Exp.
Err.
Additive?
(%)
(%)
(%)
(%)
(%)
(pp)
Pair 0
68.6
71.4
74.3
65.7
77.1
11.4
No
Pair 1
71.4
54.3
65.7
71.4
48.6
22.9
No
Pair 2
71.4
65.7
68.6
68.6
62.9
5.7
Yes
Pair 3
74.3
62.9
68.6
54.3
57.1
2.9
Yes
Pair 4
74.3
80.0
60.0
62.9
65.7
2.9
Yes
Appendix
Table 5: Experiment 4c: superposition test ( n=35 tasks). Expected rate under additivity: Pexp=PA+PB−Pnone . A pair is judged approximately additive if ∣PA+B−Pexp∣<10 pp.
Coding agents such as Claude Code also follow text files that developers write for them: a project file, skills the agent may choose to open, and rules that load automatically when the agent reads matching files. We call these files the agent's harness. Developers add to it whenever they correct the agent, but one success proves little: the same agent can pass a task once and fail it the next time. We present Cortex, a tool that tests changes to the harness instead of trusting them. It saves fixes from the developer's sessions as replayable tasks, finds the corrections that recur, writes one skill or rule at a time, and keeps it only if repeated runs with and without it show a net gain and no task made noticeably worse. Existing text is tested the same way before it is deleted, and Cortex neither adds nor deletes text the agent never used. We tested Cortex with Claude Haiku 4.5 on a repository built for the purpose, with conventions the model tends to break, one area with nothing to learn, and a script playing the developer. On held-out tasks, the harnesses it built made the agent follow the conventions far more often than no skills or rules did, left the comparison area unchanged, and added no text read on every turn. Rules, which load automatically, did most of the work; skills helped only when offered with the files a task touches, and those offered on every turn were rarely opened. Cortex kept none of the useless or harmful suggestions we planted, but the agent never opened the useless ones or the harmful skills, and it reached no verdict on the harmful rules, so the test shows that Cortex keeps no unused text, not that it detects damage. Under the stronger Claude Sonnet 5 a single run's harness added little, and on a real open-source project, where the agent was rarely corrected, Cortex found nothing to learn. We release the tool, the lab and its data.
As autonomous coding agents see rapid adoption, their evaluation has primarily focused on task completion rates holding the target codebase fixed. This leaves a critical question unanswered: does the structural and stylistic quality, or ``cleanliness'' of the underlying code affect an agent's ability to navigate and modify it? To isolate the effect of code cleanliness from agent capability, we introduce an evaluation protocol built around minimal pairs: repositories that match on architecture, dependencies, and external behaviour, but differ on static-analysis rule violations and cognitive complexity. The pairs are constructed in both directions, by agent pipelines that either degrade a clean repository or clean a messy one. We author 33 tasks across six such pairs, evaluated through hidden tests at the application's public surface. Across 660 trials with Claude Code, code cleanliness does not change the agent's pass rate. However, it substantially alters the agent's operational footprint: agents working on cleaner code use 7 to 8% fewer tokens and reduce file revisitations by 34%. Our findings suggest that traditional maintainability principles remain highly relevant in the era of AI-driven development, shaping the computational cost and navigational efficiency of coding agents. Code cleanliness joins model choice, harness, and prompting as a factor that materially affects agent behaviours.
When a coding agent obeys a rule, it may simply have been going to do that anyway. Existing instruction-following benchmarks cannot tell the difference: they concentrate rules in the user turn, while coding-agent benchmarks emphasize final task success. We introduce Harness-IF, which scores operational rules one at a time from execution evidence: 60 realistic multi-turn coding items drawn from a 642-rule library, 256 rules receiving verdicts, placed on the five configurable surfaces a deployed agent reads. To separate compliance from coincidence we introduce Against-Prior Accuracy (AP-Acc), which scores only rules labeled as opposing unprompted defaults, observed by re-running tasks with the rule withheld across nine probe builds and curated otherwise. Across 12 frontier models, accuracy spans 72.1-85.9% and AP-Acc 66.1-78.6%; every model is worse on against-prior rules, by 3.6 to 7.4 points (mean 5.81), and the direction survives a common-support analysis with item-clustered intervals. Aggregate scores therefore overstate compliance by a model-specific margin: prior control leaves the top build unchanged and exchanges three adjacent rank pairs. A counterbalanced conflict pilot on nine separate builds adds a second result: pooled precedence does not follow prompt depth, with system prompts, project files, and user instructions ahead of tool and skill descriptions.
Zining Huang, Haoran Que, Hong Zeng +8
ByteDance Seed · Tsinghua University · Peking University