Correct Code, Broken Contributions? SWE-CC: Benchmarking Repository Policy Compliance for Coding Agents

Hai Dang Truong; Rayner Goh; Thanh Le-Cong; Yintong Huo

cs.SEOct 5, 2026

Correct Code, Broken Contributions? SWE-CC: Benchmarking Repository Policy Compliance for Coding Agents

Authors: Hai Dang Truong, Rayner Goh, Thanh Le-Cong, Yintong Huo

Organizations: Singapore Management University · Singapore University of Technology and Design

Abstract

Autonomous coding agents now resolve a substantial share of real-world GitHub issues. However, passing functional tests differs fundamentally from producing a high-quality contribution acceptable for merging. Mature open-source projects publish repository-specific contribution policies, spanning style, git, testing workflows, to ensure code quality and long-term maintainability. Because existing benchmarks evaluate patches solely on unit tests, agent compliance with repository governance remains unknown. In this paper, we introduce SWE-CC, a benchmark evaluating code and process compliance in autonomous software engineering. We develop a semi-automated pipeline that converts developer documentation across 12 open-source repositories into 823 machine-checkable atomic policies. SWE-CC introduces two features: 1) lightweight, deterministic checker functions that represent each policy, 2) a comprehensive auditing mechanism that inspects both agent runtime behaviors and final deliverables. We evaluate the compliance of agent workflows in 500 end-to-end software contribution tasks extended from SWE-bench Verified. Our evaluation of four LLMs under two agent scaffolds shows that modern agents suffer from coding compliance issues: although agents produce functionally correct patches, they still violate 43.1 percent of applicable project policies, with nearly half of all violations occurring during intermediate execution steps. These results show that functional correctness does not guarantee real-world readiness, highlighting that future software engineering agents must reliably conform to repository governance to enable safe and trustworthy deployment.

Figures & tables

Figure 1: A SWE-bench-resolved patch violating repository policies. The generated patch passes the full test suite, but violates Django’s guideline requiring assertIs for Boolean assertions.

Benchmark	Evaluation scope	Policy source	# Policy	Agent behavior	Policy retrieval	Policy evaluator
SWE-NFI	Patch quality	Literature, common practice	92	$\circ$	$\circ$	Executable
SWE-Gate	Patch quality	Code review history	303	$\circ$	$\circ$	Executable
RepoComplianceBench	AI declaration	Contribution file	455	$\circ$	$\bullet$	Executable + LLM
SWE-SHIELD	Patch quality	Code review history	1,787	$\circ$	$\circ$	LLM-as-a-judge
SWE-CC (ours)	Coding workflow	Project documentation	823	$\bullet$	$\bullet$	Executable

Table 1: Comparing SWE-CC against the closely-related benchmarks. Evaluation scope is the evaluation subject, and Policy source is where its policies come from. Agent behavior marks whether the agent’s trajectory is audited, and Policy retrieval indicates whether the agent must find the policies itself (

\circ

not supported,

\bullet

supported). Policy evaluator records how compliance is evaluated.

Figure 2: SWE-CC benchmark construction pipeline. An LLM agent extracts atomic policies from project documentation, filters them to 823 checkable, mandatory policies, and compiles each into a checker function. On a SWE-bench issue, the checker functions grade the agent’s final output and trajectory. We illustrate this with one policy example: commit summary $\leq$ 71 characters.

	mini-SWE-agent			OpenHands
Agent models	Triggering rate (%)	Compliance rate (%)	Resolve rate (%)	Triggering rate (%)	Compliance rate (%)	Resolve rate (%)
Native Setting
GPT-5.6 Luna	23.5	51.8	78.6	24.4	55.3	86.6
Gemini 3.7 Flash	23.8	53.9	80.2	24.3	56.3	78.4
DeepSeek V4 Flash	25.6	61.3	93.4	26.0	63.8	94.4
Kimi K2.5	24.6	55.9	74.6	24.8	57.2	76.2

Table 3: Triggering, compliance, and resolve rate per model and scaffold.

\Delta

shows the performance gain of Consolidated. The highest rate per column is in bold , marked separately for Native and Consolidated.

		mini-SWE-agent		OpenHands
Category		Native (%)	Consolidated (%)	Native (%)	Consolidated (%)
Git and commit conventions	Triggering rate	52.3	59.7	55.7	60.4
	Compliance rate	59.9	87.0	63.4	90.1
PR and release metadata	Triggering rate	36.3	45.6	37.5	51.8
	Compliance rate	36.4	51.6	39.6	58.9
Code and quality	Triggering rate	83.7	83.8	82.5	82.8

Table 4: Triggering and compliance rate by governance domain, averaging over four models within each scaffold. For each domain, the highest compliance rate is in bold and the second highest is underlined .

		Native (autonomously discover)		Consolidated
Model	Scaffold	Attempt (%)	Retrieve (%)	Opened (%)	Delivered (%)
GPT-5.6 Luna	mini-SWE-agent	50.0	13.2	100.0	100.0
	OpenHands	75.6	43.2	100.0	99.9
Gemini 3.7 Flash	mini-SWE-agent	10.0	0.4	100.0	100.0
	OpenHands	2.8	1.0	98.6	100.0
DeepSeek V4 Flash	mini-SWE-agent	34.8	10.6	100.0	99.7

Table 5: The portion of the policies that reach model context. Native columns are progressive: the agent attempted to fetch a policy document and retrieved policy text from it (highest in bold , second underlined ). Consolidated reports whether the mounted file was opened and what share of its content was delivered into context.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

project	instances	extracted	in scope	binding	output	diff.	traj.	approx.
		rules			evidence type
astropy	22	255	129	111	103	5	3	85%
django	231	143	89	78	67	8	3	55%
matplotlib	34	293	199	167	148	14	5	82%
scikit-learn	32	274	169	146	131	8	7	83%
sympy	75	279	188	142	112	11	19	24%

Appendix

Table 6: The twelve corpora of SWE-CC . Extracted counts atomic rules from the pages that bind contributors, in scope those a run produces evidence for, and binding those stating a requirement; the evidence-type columns partition the binding rules. Approx. is the share of checks that approximate their rule (Appendix B.5.3 ).

category	#	example policy
Documentation and docstrings	224	Give every public class, method, and function a docstring. (ASTROPY-C085)
Language and framework style	163	Do not raise the bare Exception class. (ASTROPY-C100)
Tests and test style	160	Name test modules test_.py or _test.py . (ASTROPY-C002)
Specialized changes	107	Keep in-repository data files under about 100 kB and host anything larger off the repository. (ASTROPY-C092)
PR and release metadata	57	Add a changelog fragment under docs/changes/<sub-package>/ describing your change. (ASTROPY-C061)
Git and commit conventions	40	Phrase commit subject lines in past tense and end them with a period. (DJANGO-C046)

Appendix

Table 7: One mandatory policy per category, as extracted. Counts are repeated from Table 3 .

		Native		Consolidated		$\Delta$
Scaffold	Model	All	Exact	All	Exact	All	Exact
mini-SWE-agent	GPT-5.6 Luna	51.8	52.0	67.6	69.9	15.8	17.9
	Gemini 3.7 Flash	53.9	50.7	65.3	69.0	11.4	18.3
	DeepSeek V4 Flash	61.3	56.6	67.7	66.1	6.4	9.5
	Kimi K2.5	55.9	51.8	56.9	55.2	1.0	3.4
OpenHands	GPT-5.6 Luna	55.3	55.8	67.5	69.2	12.2	13.4

Appendix

Table 8: Compliance rate (%) over all 823 checks and over the 294 exact checks.

\Delta

is the gain from consolidated policies.

Commit subject	Outcome	Reason
Fixed #123 -- Added truncation to Truncator.	satisfied
Made Truncator keep HTML entities.	satisfied	irregular verb
Add truncation to Truncator.	violated	not in the past tense
Added truncation to Truncator	violated	no final period
(no commit)	inapplicable

Appendix

Table 9: Authored cases for DJANGO-C046.

Technique	Policy	What the check tests	Case it gets wrong
Word list	DJANGO-C046: commit subjects in the past tense	the leading verb ends in -ed or is a listed irregular form	misses past forms outside the list
Text pattern	ASTROPY-C153: astropy for the package, Astropy for the Project	each mention matches one of the two sanctioned spellings	cannot tell which meaning a sentence intends
Code structure	SCIKIT-LEARN-C158: estimators inherit from BaseEstimator	a new class that defines fit is an estimator	selects other classes with a fit method
File location	MATPLOTLIB-C262: imported code carries a compatible licence	a new file under extern/ is imported code	misses imported code placed elsewhere
Co-change	SCIKIT-LEARN-C118: deprecations are listed in the API reference	a @deprecated name comes with an edit to the reference list	does not check that the right name was added

Appendix

Table 10: Techniques used by approximate checks, with one policy each.

project	checks	drawn	diff.	traj.
matplotlib	167	24	2	1
scikit-learn	146	21	1	1
SymPy	142	21	1	3
astropy	111	18	1	1
Django	78	14	2	1
pylint	49	11	1	1

Appendix

Table 11: Composition of the checker audit sample. Projects with ten or fewer checks are censused rather than sampled. Diff. and traj. give the differential and trajectory checks drawn, the two evidence types that chance alone would leave out.

	raw agr. (%)	Cohen’s $\kappa$
verdict (accept / reject)	94.0	0.72
C1 encodes the rule	99.3	0.89
C2 precondition	96.0	0.73
C3 post-condition	98.0	0.72
C4 exact or approximate	96.7	0.93

Appendix

Table 12: Agreement between the two raters on the 150 audited checks, on the verdict and on each of the four questions.

Checkpoint	Checks
1. Manifest (author)	every URL is at the pinned version; no maintainer or triage page admitted whole; partial pages name their in-scope sections; configuration files are context-only; nothing is claimed about a file that was not retrieved
2. Extraction (author)	row count plausible against the other repositories; section context and notes populated on every row; every rule records its source sentence and URL; identifiers contiguous; every category from the closed list; every rule cross-referenced with its source sentence on the live page; mid-sentence quotations checked against the preceding clause
3. Acceptance (automated)	rules-to-source-length ratio within band; judgment and in scope never co-occur; modal-verb and rubric levels differ on some rules, with direction recorded; reasoning distinct across rows; not-observable counted separately from other out-of-scope reasons; schema conformance

Appendix

Table 13: Checkpoints in the extraction workflow.

Symptom	Cause	Mitigation
Version drift mid-run	stable documentation retrieved	pin the developer documentation
False version mismatch	pinned at build rather than release	halt only on a different release
Template rules missing or inverted	rendered view strips comments	retrieve raw text in the pre-pass
Configuration files never retrieved	not reachable from navigation	resolve by path in the pre-pass
Agent-directed policy missing	agent files outside navigation	admit them in traversal and the pre-pass
Rules asserted unread	model answers from prior familiarity	require retrieval or an explicit non-retrieval

Appendix

Table 14: Failure modes observed during rule extraction.

Figure 7: Runs with an evaluation gap, by cause, over all 8,000 runs. No functional verdict : the SWE-bench harness returns no result. Compliance partly withheld : the run is graded, but checkers that need a parsed file return no verdict.

Model	Ours (Native)	Vals AI	$\Delta$
GPT-5.6 Luna	78.6	93.0	$-14.4$
Gemini 3.7 Flash	80.2	80.8	$-0.6$
DeepSeek V4 Flash 0731	93.4	88.8	$+4.6$
Kimi K2.5	74.6	70.0	$+4.6$

Appendix

Table 15: Resolve rate (%) on SWE-bench Verified under mini-SWE-agent.

Category	Policies	Never triggered	Share
Specialized changes	107	81	76%
Documentation and docstrings	224	93	42%
Language and framework style	163	59	36%
Tests and test style	160	50	31%
Git and commit conventions	40	10	25%
Code and quality	35	5	14%

Appendix

Table 16: Policies that never trigger on any of the 8,000 runs, per category.

		Native		Consolidated
Category	Scaffold	Triggered	Graded	Triggered	Graded
Git and commit conventions	mini-SWE-agent	4,100	4,092	4,676	4,671
	OpenHands	4,361	4,353	4,730	4,722
PR and release metadata	mini-SWE-agent	4,619	4,619	5,793	5,793
	OpenHands	4,764	4,764	6,586	6,586
Code and quality	mini-SWE-agent	5,385	1,453	5,388	1,484

Appendix

Table 17: Policies triggered and graded per category, pooled over twelve projects and four models within each scaffold. The gap is the policies whose checker finds too little evidence in the run to return a verdict.

Value	What the check opens
output	the final files or the commit message, read once, with no comparison
differential	two states, compared: pre-patch against post-patch
trajectory	a record of what the agent did, not only what it produced
judgment	nothing; the standard is not written down and a human decides

Appendix

Table 18: Evidence Type.

	Condition	What the annotator does
C1	observability	names the file, diff, or output that the check opens
C2	decidability	writes down the failure condition

Appendix

Table 19: Conditions for a rule to be in scope.

Exists	Does not exist
repository working tree, the agent’s diff	pull request object, PR template, review thread
one commit and its message	multiple commits, branch history, rebase or squash
a local test suite run	CI service, coverage bot, issue tracker
the files the agent writes	built or rendered documentation, browser, screenshots
	a second human, contributor identity

Appendix

Table 20: What a patch-only harness does and does not produce.

Route	The rule…
M1	says so outright ( must , required )
M2	forbids something ( never , do not , must not )
M3	is a condition of acceptance: a pre-merge or review checklist, or a named CI job
M4	names an exact thing, limit, form, or ordering
M5	is stated non-mandatorily, but admits only one satisfying state
M6	states a consequence in the source

Appendix

Table 21: Routes to must .

Route	The rule…
D1	uses preference wording
D2	permits deviation in its documentation
D3	has a fixed form, but whether it applies is a judgment call
D4	instructs on how , not what

Appendix

Table 22: Routes to should .

Explore similar work

Sep 3, 2026cs.SE

SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents

Repository-level software engineering benchmarks have significantly advanced the evaluation of coding agents, but existing benchmarks primarily measure whether generated patches pass functional tests and overlook review-derived acceptance constraints (review constraints) that often influence whether a patch is acceptable in real-world software development. We introduce SWE-Gate, a repository-level benchmark for software engineering agents that explicitly evaluates review constraint compliance alongside functional correctness. SWE-Gate derives review constraints from real pull request review comments and synthesizes repository-level repair instances around these constraints. Each instance provides separate functional and constraint tests, together with non-compliant and gold patches, enabling explicit separation between issue resolution capability and review constraint compliance. We construct SWE-Gate with 303 repository-level repair instances spanning 75 open-source Python repositories across diverse software domains. Experiments with four LLM backends spanning different capability levels under a common coding-agent scaffold reveal a substantial gap between functional success and success under the complete repair specification: among 644 repairs that pass the functional tests, 221 fail to satisfy the provided review constraints. These findings show that functional-only evaluation overestimates agents' ability to satisfy the full requirements of repository-level repair tasks. The replication package including code, data, and experimental results is available at https://github.com/DeepSoftwareAnalytics/SWE-Gate.

Xin He, Yanlin Wang, Mingwei Liu +3

School of Software Engineering, Sun Yat-sen University · College of Computer Science and Technology, Zhejiang University · School of Big Data and Software Engineering, Chongqing University +1

Coding Agents Repair Acceptance

Ask PDF

Sep 24, 2026cs.AI

SWE-Prometheus: Measuring Engineering Governance Improvements in Real-World Repositories

Large language model based coding agents have made substantial progress on repository-level software engineering tasks. Existing repository benchmarks, however, usually start from a human-identified issue and evaluate whether a patch satisfies a functional signal. We present SWE-Prometheus, a benchmark for the broader task of improving repository engineering governance. Each task provides a fixed snapshot and an open-ended objective, requiring the agent to identify risks, prioritize interventions, and verify the resulting changes. SWE-Prometheus evaluates six governance dimensions through paired evidence, clean-environment probes, behavior gates, and two independent teacher ratings of the same evidence. The benchmark contains 60 repositories; ten models are evaluated on a shared 22-repository public subset, where mean Normalized Governance Improvement ranges from 0.0568 to 0.5760 and observed behavior-breakage rates range from 0% to 23%. On a frozen ten-repository batch, a repository-blind template obtains mean NGI 0.272, but its gains concentrate in Tests & CI, Quality Gates, and Documentation; it improves Reproducible Environment and Dependency & Security on none of the repositories. This baseline makes the distinction between adding governance artifacts and producing execution-backed improvements measurable. The no-op condition has median NGI zero and standard deviation 0.073; two teachers agree exactly on 57 of 60 dimension scores for the same no-op evidence. For the two highest conditional-mean systems, common-valid NGI is similar, while full-pool comparisons that include behavior failures favor Kimi-K3. These results show why repository-governance evaluation should report improvement, behavior preservation, evidence quality, and coverage together.

Jiajun Wu, Leixin Sun, Zihan Tan +9

CosmosMind · ModCraft · Peking University +2

Repository-Level Code Understanding Governance Improvement

Ask PDF

Jul 8, 2026cs.SE

DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks

DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents. Most public agentic coding benchmarks follow SWE-bench in mining merged fixes from public GitHub repositories, which creates two problems: the fixes and their discussion were likely seen during pretraining, so a high score can reflect recall rather than problem-solving; and each task is graded by the tests that shipped with its merged fix, which were written to confirm one specific fix rather than grade an arbitrary solution, so they can fail a correct alternative or pass an incomplete one. DeepSWE avoids both. Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream, so their reference solutions stay out of the public record that model training scrapes; and each task is graded by a hand-written verifier that checks the requested functionality and accepts any implementation that provides it. When an independent LLM judge re-reviews graded runs, it disagrees with DeepSWE's verifier about an order of magnitude less often than with SWE-Bench Pro's inherited tests (1.4% versus 32.4%). Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code, and the benchmark separates frontier agents across a wider score band than the leaderboards on which they otherwise cluster. We release the benchmark, its verifiers, and the full record of evaluation trajectories.

Wenqi Huang, Charley Lee, Leonard Tng +1

1Datacurve

Swe-Bench Verified Coding Agents

Ask PDF