Guardrails Beat Guidance: A Large-Scale Study of Rules, Skills, and Persistent Configuration for Coding Agents

Xing Zhang; Guanghui Wang; Yanwei Cui; Wei Qiu; Ziyuan Li; Bing Zhu; Peiyang He

cs.AIApr 13, 2026

Guardrails Beat Guidance: A Large-Scale Study of Rules, Skills, and Persistent Configuration for Coding Agents

Authors: Xing Zhang, Guanghui Wang, Yanwei Cui, Wei Qiu, Ziyuan Li, Bing Zhu, Peiyang He

Organizations: AWS Generative AI Innovation Center · HSBC Holdings Plc., HSBC Technology Center, China.

Abstract

Random rules improve a coding agent's task performance as much as expert-curated ones (both $+13.8$ pp on a discriminative subset of SWE-bench Verified), and in our data every individually beneficial rule is a negative constraint ("do not refactor unrelated code"), while every individually harmful one is a positive directive ("follow code style"). We arrive at these findings through the first large-scale controlled study of agent rule files (CLAUDE.md, .cursorrules, and the broader family of agent skills, plugin manifests, and persona definitions): we scrape 679 rule files (25{,}532 rules) from GitHub and conduct over 5{,}000 agent runs of Claude Code with Claude Opus 4.6 on SWE-bench Verified. Three patterns emerge. (i) Rule polarity cleanly separates beneficial from harmful rules; we read this through the lens of potential-based reward shaping (PBRS). (ii) Performance gains are largely content-independent: random, shuffled, mismatched-domain, and unconverted-format rule files all match curated rules, pointing to a context priming mechanism. (iii) Individual rules often appear harmful in isolation yet do not visibly accumulate damage in ensemble: pass rates remain stable across rule counts from 0 to 50. These findings expose a hidden reliability risk in the rapidly growing ecosystem of community-authored rules and skills, and they yield a clear principle for safer agent configuration: constrain what agents must not do, rather than prescribing what they should.

Figures & tables

Figure 1: Headline findings. (a) All seven rule conditions outperform the no-rule baseline by 7–14pp, yet random rules match curated rules (both 63.8%), pointing to a context priming effect rather than content-specific instruction. (b) Per-rule ablation on 18 curated rules: shaping rules (green, removal hurts) are all negative constraints; distorting rules (orange, removal helps) are all positive directives.

Figure 2: Taxonomy of 25,532 rules from 679 GitHub rule files, classified by an LLM classifier (87% agreement with human labels). Project-specific rules dominate (64.9%); our experiments use the five transferable categories.

Figure 3: Rule quantity and type (

n=58

tasks each). (a) Pass rate vs. rule count: individual seeds (dots), mean

\pm

std (line with shading). No phase transition; 50 rules slightly outperform 0 rules. (b) Pass rate by rule type. State-dependent tool/process rules score highest; state-independent architecture rules score lowest (10.4pp spread), consistent with P1.

Rule	Polarity	Type	Effect	$p$
No unrelated refactor	“don’t”	safety	shaping	.016
No new dependencies	“don’t”	safety	shaping	.508
No unrelated files	“don’t”	safety	shaping	.688
Read test files	“do”	process	distorting	.227
Follow code style	“do”	style	distorting	.180
Handle edge cases	“do”	style	distorting	.289

Table 1: Non-inert rules from per-rule ablation (

n=35

tasks; cf. Figure 1 b). Shaping : removal hurts performance; distorting : removal helps.

p

-values from McNemar’s exact test. The 11 inert rules (

|\Delta|\leq 5

pp) are omitted; see Appendix D for the complete table.

Figure 4: Individual rule effects (

n=35

tasks). (a) Per-rule distortion: each of the 18 curated rules applied individually to 17 baseline-solved tasks. 14/18 rules break

\geq

2 tasks; the worst break 4/17. (b) Superposition test on 5 rule pairs: observed vs. additive-expected pass rate. Three pairs (

+

) are approximately additive; two (

\times

) show strong non-linearity (max error 22.9pp).

Box 1: Rule authoring guidelines for coding agents.
Scoped to Claude Code-class agents on SWE-bench-style tasks.
Do	Write negative constraints (“do not refactor unrelated code”).
Don’t	Write positive directives (“follow code style,” “handle edge cases”) without auditing them.
Do	Include rules; even random or mismatched-domain rules help.
Don’t	Spend effort optimizing wording; presence beats phrasing.
Do	Use up to 50 rules; no degradation observed.

Table 6

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	Pass %	Helped	Hurt	Disc.	$p$
Baseline (none)	50.0	—	—	—	—
Random	63.8	12	4	16	.077
Curated	63.8	14	6	20	.115
Popular	60.3	18	12	30	.361
Mismatched	58.6	15	10	25	.424
Matched	56.9	11	7	18	.481

Appendix

Table 2: Experiment 1: complete results (

n=58

tasks). Pass rates and pairwise McNemar tests vs. the no-rule baseline, sorted by pass rate.

Rule type	PBRS category	Pass %
Tool / process	State-dependent	63.8
Mixed (all types)	Mixed	62.1
Safety	Constraint-based	60.3
Code style	State-independent	56.9
Architecture	State-independent	53.4

Appendix

Table 3: Experiment 3: pass rate by rule type (

n=58

tasks, 10 rules each). PBRS categories follow Section 4 .

#	Rule (abbreviated)	$\Delta$ (pp)	Status	$p$
6	No unrelated refactor	$+$ 20.0	Shaping	.016
17	No new dependencies	$+$ 8.6	Shaping	.508
15	No unrelated files	$+$ 5.7	Shaping	.688
4	Minimal changes	$+$ 2.9	Inert	1.0
13	Keep funcs concise	$+$ 2.9	Inert	1.0
14	Good naming	$+$ 2.9	Inert	1.0

Appendix

Table 4: Experiment 4a: complete per-rule ablation (

n=35

tasks). Positive

\Delta

means the rule is helpful (removal hurts); sorted by

\Delta

descending. Threshold for shaping / distorting :

|\Delta|>5

pp.

	None	A	B	A+B	Exp.	Err.	Additive?
	(%)	(%)	(%)	(%)	(%)	(pp)
Pair 0	68.6	71.4	74.3	65.7	77.1	11.4	No
Pair 1	71.4	54.3	65.7	71.4	48.6	22.9	No
Pair 2	71.4	65.7	68.6	68.6	62.9	5.7	Yes
Pair 3	74.3	62.9	68.6	54.3	57.1	2.9	Yes
Pair 4	74.3	80.0	60.0	62.9	65.7	2.9	Yes

Appendix

Table 5: Experiment 4c: superposition test (

n=35

tasks). Expected rate under additivity:

P_{\text{exp}}=P_{A}+P_{B}-P_{\text{none}}

. A pair is judged approximately additive if

|P_{\text{A+B}}-P_{\text{exp}}|<10

pp.

Software Engineering Agents

Software Engineering Agents are AI systems that use code, tools, and feedback to perform software development tasks.

See all 323 papers

Cortex: Measurement-Gated Evolution and Pruning of Coding-Agent Skills and Rules
Oct 9, 2026Samuele GiampieriLLM Agent Harnesses LLM Agent Skill Learning
Independent researcher
Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Study
May 19, 2026Priyansh Trivedi, Olivier SchmittCoding Agents AI Agent Evaluation
SonarSource
Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents
Aug 12, 2026Zining Huang, Haoran Que, Hong Zeng +8AI Coding Agents LLM Agent Evaluation
ByteDance Seed · Tsinghua University · Peking University

Oct 9, 2026J/K move · Enter open · S save

Cortex: Measurement-Gated Evolution and Pruning of Coding-Agent Skills and Rules

Samuele Giampieri

Independent researcher

LLM Agent Harnesses LLM Agent Skill Learning LLM Agent Self-Improvement

Open paper AskPDF arXiv

Coding agents such as Claude Code also follow text files that developers write for them: a project file, skills the agent may choose to open, and rules that load automatically when the agent reads matching files. We call these files the agent's harness. Developers add to it whenever they correct the agent, but one success proves little: the same agent can pass a task once and fail it the next time. We present Cortex, a tool that tests changes to the harness instead of trusting them. It saves fixes from the developer's sessions as replayable tasks, finds the corrections that recur, writes one skill or rule at a time, and keeps it only if repeated runs with and without it show a net gain and no task made noticeably worse. Existing text is tested the same way before it is deleted, and Cortex neither adds nor deletes text the agent never used. We tested Cortex with Claude Haiku 4.5 on a repository built for the purpose, with conventions the model tends to break, one area with nothing to learn, and a script playing the developer. On held-out tasks, the harnesses it built made the agent follow the conventions far more often than no skills or rules did, left the comparison area unchanged, and added no text read on every turn. Rules, which load automatically, did most of the work; skills helped only when offered with the files a task touches, and those offered on every turn were rarely opened. Cortex kept none of the useless or harmful suggestions we planted, but the agent never opened the useless ones or the harmful skills, and it reached no verdict on the harmful rules, so the test shows that Cortex keeps no unused text, not that it detects damage. Under the stronger Claude Sonnet 5 a single run's harness added little, and on a real open-source project, where the agent was rarely corrected, Cortex found nothing to learn. We release the tool, the lab and its data.

Abstract

Software Engineering Agents

Explore similar work

Cortex: Measurement-Gated Evolution and Pruning of Coding-Agent Skills and Rules

Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Study

Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents