ContextCov: Deriving and Enforcing Executable Constraints from Agent Instruction Files

Reshabh K Sharma

cs.SEFeb 28, 2026

ContextCov: Deriving and Enforcing Executable Constraints from Agent Instruction Files

Authors: Reshabh K Sharma

Organizations: University of Washington Seattle, Washington, USA

Abstract

As Large Language Model (LLM) agents increasingly execute complex, autonomous software engineering tasks, developers rely on natural language instruction files such as AGENTS.md to express project-specific coding conventions, tooling restrictions, and architectural boundaries. However, because these instructions remain passive text, agents frequently violate documented constraints due to context window saturation or conflicting local context. In autonomous settings without real-time human supervision, such violations rapidly compound into technical debt. To ground autonomous agents in repository constraints, we introduce ContextCov, a framework that transforms passive natural language instructions into executable guardrails. Unlike prompt-only or reflection-only compliance approaches, ContextCov compiles documented constraints into three complementary checks: static AST queries for code patterns, runtime shell shims that intercept prohibited commands, and architectural validators that enforce structural rules. Acting as an automated, continuous reviewer, ContextCov intercepts agent actions and returns immediate, reproducible violation traces, enabling self-correction before non-compliant changes are finalized. We evaluate ContextCov on SWE-bench Lite (12 repositories, 300 tasks). Compared to prompt-only and LLM reflection baselines, ContextCov achieves 88.3% constraint compliance (vs. 67.0% and 50.3%) with 3.4x lower feedback cost, while maintaining functional correctness. The source code and evaluation results are available at https://github.com/reSHARMA/ContextCov.

Figures & tables

Figure 2. Architecture of ContextCov . The system operates in two phases. First, the Check Generation phase parses Agent Instructions into a Markdown AST, refines constraints via an LLM, and synthesizes executable checks using domain-specialized generators. Second, the Runtime Enforcement phase actively intercepts the agent’s shell commands and file modifications, validating actions against the deployed checks and returning deterministic errors as immediate feedback for self-correction.

	Extraction & Synthesis (RQ1)							Harness Resolution (RQ3)
Repository	Lines	Seg.	Src	Proc	A-D	A-S	Chk	$n$	Van.	%	LR	CC	$\Delta$
astropy	151	54	25	17	15	12	69	6	3	50	3	2	$-1$
django	110	61	35	22	5	11	73	114	68	60	70	77	$+9$
matplotlib	157	71	30	24	20	9	83	23	8	35	10	9	$+1$
seaborn	106	44	15	23	13	0	51	4	2	50	2	2	0
flask	102	52	25	38	15	1	79	3	1	33	1	1	0

Table 1. Repository-level statistics: extraction & check synthesis (RQ1, left) and harness resolution (RQ3, right). Van. = Vanilla, LR = LLM Reflection, CC = ContextCov .

	Oracle Compliance & Feedback Rounds						Post-hoc LLM Critic
Method	Clean	Rate	Avg. viol.	Rounds	Rnds ${}_{\textrm{c}}$	Hit cap	Patch source	Clean	$n$	Avg.	Total
ContextCov	265	88.3%	0.14	1.67	1.31	29 (10%)	ContextCov finals	30.1% (88/292)	292	1.15	336
Vanilla	201	67.0%	0.62	—	—	—	Vanilla finals	29.2% (87/298)	298	1.15	344
LLM Reflection	151	50.3%	1.01	2.85	2.30	85 (28%)	LLM Refl. finals	50.3% (151/300)	300	1.01	304

Table 2. RQ2: Enforcement efficiency—oracle compliance, feedback rounds, and post-hoc LLM critic analysis (300 instances).

Figure 3. RQ2: Harness resolution rate versus number of feedback rounds.

Figure 4. RQ2: Head-to-head wins when one method resolves an instance another does not.

Method	Resolved	Rate	vs Van.	vs other FB	Unique
ContextCov	172	57.3%	+22	+32	9
LLM Reflection	157	52.3%	+23	+17	10
Vanilla	159	53.0%	—	+25	2
McNemar paired tests (discordant cells: A only / B only)
Vanilla vs. ContextCov : 9 / 22				$p=0.031$
Vanilla vs. LLM Reflection: 25 / 23				$p=0.885$

Table 3. RQ3: Functional correctness—harness resolution, pairwise wins, and McNemar tests (

n{=}300

Method	Oracle	Pass	Fail	Total
ContextCov	Clean	156	109	265
ContextCov	Dirty	16	19	35
LLM Refl.	Clean	85	66	151
LLM Refl.	Dirty	72	77	149
Vanilla	Clean	110	91	201
Vanilla	Dirty	49	50	99

Table 4. RQ3: Oracle cleanliness

\times

harness resolution (

n{=}300

per method).

Metric	Method	Mean	Sum
LLM input chars	LLM Refl.	19,455	5.84M
LLM input chars	ContextCov	6,292	1.89M
Feedback time (s)	LLM Refl.	133.7	40,110
Feedback time (s)	ContextCov	27.7	8,310
		Mean	Median
End-to-end (s)	LLM Refl.	461.4	379.2

Table 5. RQ4: Feedback cost proxies (300 instances).

Figure 5. RQ4: Mean logged LLM input characters (left) and mean feedback-generation wall time (right) for LLM Reflection versus ContextCov .

Runtime Enforcement for AI Agents

Runtime enforcement for AI agents checks and constrains actions as they execute, helping prevent unsafe or unauthorized effects.

See all 273 papers

Correct Code, Broken Contributions? SWE-CC: Benchmarking Repository Policy Compliance for Coding Agents
Oct 5, 2026Hai Dang Truong, Rayner Goh, Thanh Le-Cong +1AI Agent Auditing Coding Agents
Singapore Management University · Singapore University of Technology and Design
AgentForge: Execution-Grounded Multi-Agent LLM Framework for Autonomous Software Engineering
Apr 13, 2026Rajesh Kumar, Waqar Ali, Junaid Ahmed +2Software Engineering Agents Multi-Agent Coding
International Research Center for Complexity Sciences, Hangzhou International Innovation Institute, Beihang University, Hangzhou, 311115, China · Department of Computer Science, College of Science, Mathematics and Technology, Wenzhou-Kean University, Wenzhou 325060, China · Fakulti Teknologi Maklumat dan Komunikasi, Universiti Teknikal Malaysia Melaka, Melaka 76100, Malaysia +2
Look Before You Leap: Pre-Action Verification for LLM Agents
Sep 14, 2026Asaad AlthoubiLLM Agent Verification AI Agent Benchmarks
Oklahoma State University Stillwater, OK USA

Oct 5, 2026J/K move · Enter open · S save

Correct Code, Broken Contributions? SWE-CC: Benchmarking Repository Policy Compliance for Coding Agents

Hai Dang Truong, Rayner Goh, Thanh Le-Cong, Yintong Huo

Singapore Management University · Singapore University of Technology and Design

AI Agent Auditing Coding Agents Software Engineering Benchmarks

Open paper AskPDF arXiv

Autonomous coding agents now resolve a substantial share of real-world GitHub issues. However, passing functional tests differs fundamentally from producing a high-quality contribution acceptable for merging. Mature open-source projects publish repository-specific contribution policies, spanning style, git, testing workflows, to ensure code quality and long-term maintainability. Because existing benchmarks evaluate patches solely on unit tests, agent compliance with repository governance remains unknown. In this paper, we introduce SWE-CC, a benchmark evaluating code and process compliance in autonomous software engineering. We develop a semi-automated pipeline that converts developer documentation across 12 open-source repositories into 823 machine-checkable atomic policies. SWE-CC introduces two features: 1) lightweight, deterministic checker functions that represent each policy, 2) a comprehensive auditing mechanism that inspects both agent runtime behaviors and final deliverables. We evaluate the compliance of agent workflows in 500 end-to-end software contribution tasks extended from SWE-bench Verified. Our evaluation of four LLMs under two agent scaffolds shows that modern agents suffer from coding compliance issues: although agents produce functionally correct patches, they still violate 43.1 percent of applicable project policies, with nearly half of all violations occurring during intermediate execution steps. These results show that functional correctness does not guarantee real-world readiness, highlighting that future software engineering agents must reliably conform to repository governance to enable safe and trustworthy deployment.

Abstract

Runtime Enforcement for AI Agents

Explore similar work

Correct Code, Broken Contributions? SWE-CC: Benchmarking Repository Policy Compliance for Coding Agents

AgentForge: Execution-Grounded Multi-Agent LLM Framework for Autonomous Software Engineering

Look Before You Leap: Pre-Action Verification for LLM Agents