ANCHOR: Assessing Naturalistic Constraint Hierarchies and Ordering in Repository-Grounded Agentic Coding
Authors: Wenqiang Zhu, Xinping Lei, Zhiyi Lai, Zhengshuo Gong, Tianzhuang He, Songwei Yu, Weihao Li, Jingxuan Xu, Zihang Xu, Gang Wu, Zhongqiu Wang, Yuhui Bai, Jie Li, Zhao Liu, Junhao Fan, Yan Wu, Yanan Wu, Jiaheng Liu, Quan Gu, Ariel Deng
Organizations: Southern University of Science and Technology · Nanjing University · Independent Researcher · National University of Singapore · Beihang University · University of Michigan
For repository-grounded coding agents, a functionally correct patch is not necessarily an acceptable one: developers also constrain how the task must be carried out-modification scope, workflow order, validation, tool use, and reporting. This capability, which we call engineering constraint following, remains largely invisible to current evaluations. We introduce ANCHOR (Assessing Naturalistic Constraint Hierarchies and Ordering in Repository-Grounded Agentic Coding), a benchmark built on naturalistic, multilingual repository tasks spanning eight language groups. ANCHOR combines (i) a hierarchical taxonomy of 10 categories and 78 leaf types distilled from real developer interactions, (ii) an automated pipeline that instantiates task-grounded atomic constraints and composes them into typed AND/THEN/PRIORITY relations, and (iii) a trajectory-grounded evaluation framework that checks both the code diff and the execution trajectory with rule-based and LLM judges. Across 19 agent configurations spanning 2 scaffolds and 10 foundation models on 358 tasks, the strongest measured system, Claude Opus 4.8 under Cline, reaches 70.1% atomic compliance and 63.7% relation compliance, while 19.0% of tasks pass every scored checklist item under the strict all-item FCCR criterion. A five-engineer audit of constraints from 50 instances reports 94% feasibility, 94% repository consistency, and 96% realism. In the primary paired set (14 configurations), the mean signed change in task completion after adding constraints is +0.7 percentage points; individual changes vary in both directions. These results distinguish functional success from constraint compliance, with failures concentrated in workflow ordering, tool-use discipline, and reporting, and substantial interactions between models and scaffolds.
Figures & tables
Figure 1: Illustrative schematic. A correct patch is not necessarily an acceptable engineering solution. Two agents fix the same bug and pass tests. Their observed behavior differs on API preservation, reproduction order, regression testing, and command reporting. ANCHOR distinguishes task completion from engineering constraint following.
Benchmark
Task
Real
Scope
Comp.
Auto.
Traj.
Eval.
Scale
Lang.
IFEval
NL gen.
✗
✗
✗
✗
✗
Rule
541
EN
FollowBench
NL gen.
✗
✗
✗
✗
✗
LLM
820
EN
ComplexBench
NL gen.
✗
✗
✓
✓
✗
LLM+Rule
∼ 1K
ZH/EN
BigCodeBench
Code gen.
✗
Limited
✗
✗
✗
Exec.
1140
Python
CIFE
Code gen.
✗
Artifact
✗
✓
✗
LLM
1000
Python
CodeIF
Code gen.
✗
Limited
✗
✗
✗
Rule
–
Multi
Table 1: Comparison with prior instruction-following and coding benchmarks.
Figure 2: Constraint footprint in the source developer queries. Left: constraints per original query. Right: extracted atomic constraints across the ten top-level categories. These panels describe the source-query corpus; the instantiated benchmark taxonomy is shown separately in Figure 3 .
Figure 3: Instantiated ANCHOR taxonomy: 2,836 atomic constraints across ten categories. Category areas are proportional to their atomic counts. The three largest leaf types are labelled within each large category; the remaining leaves are combined into an explicitly labelled “Other” tile. The three smallest categories show their totals. Colours match Figure 2 ; Table 9 lists all 78 leaves.
Code
Category
#Leaves
#Atomic
C1
Tool Execution Protocol
8
371
C2
Workflow & Behaviour
9
512
C3
Modification Scope & Change Budget
7
518
C4
Code Style & Engineering Standards
13
223
C5
Code Implementation
13
283
C6
Environment
6
70
Table 2: Top-level engineering constraint taxonomy in ANCHOR, with leaf counts and instantiated atomic-constraint counts from the current release.
Figure 4: Overview of the ANCHOR construction pipeline. Starting from a repository task drawn from real GitHub issues (problem statement, gold patch, language, repository documentation), five stages produce each benchmark instance: (1) repository documentation is distilled into a compact project card; (2) a high-recall subset of taxonomy leaves is selected from task, repository, and scope evidence; (3) leaves are instantiated into task-grounded atomic constraints under explicit quality gates; (4) related atomics are composed into connected composite graphs with typed AND / THEN / PRIORITY relations; and (5) the result is rendered as a natural developer instruction paired with a structured constraint object, which Section 5 operationalizes into an executable checklist. A running example from 33cn/chain33 (rejecting duplicate crypto-type registration in Go) is threaded through the figure as the teal example lane at the bottom.
Dimension
Validity
Raw agreement
Gwet AC1
Feasibility
94.0
92.0
0.91
Repository consistency
94.0
96.4
0.96
Realism
96.0
97.6
0.97
Table 3: Audit of atomic constraints from 50 instances, independently labelled by five engineers. Validity and raw agreement are percentages; Gwet AC1 is a coefficient.
Figure 5: Overview of the ANCHOR evaluation framework.
Judge type
Description
path_filter
Whether modified paths satisfy a whitelist/blacklist
file_count
Whether the number of modified files satisfies a threshold
line_count
Whether the number of changed lines satisfies a threshold
command_presence
Whether a command appears or does not appear in the trajectory
command_arg_check
Whether a command uses required or forbidden flags
action_sequence
Whether observable actions occur in the required order
Table 4: Rule-based judge function types used in ANCHOR.
Figure 6: ANCHOR dataset overview. Stacked-bar widths are proportional to item counts; the relation-type bar expands the 600 relation items. Per-task counts are means rounded to one decimal place. The taxonomy spans ten top-level categories, 34 mid-level categories, and 78 leaves, all represented in the benchmark. C and C++ are counted as one language group.
Scaffold
Model
Task Completion
Constraint Compliance
Task-only
Constrained
ACR
CCR
IPR
FCCR
OpenCode
DeepSeek-V3.2
24.3
23.2
40.7
28.3
38.5
1.7
GLM-5
31.8
33.5
59.2
52.5
58.0
4.2
GLM-5.1
36.0
35.5
60.6
57.3
60.0
6.1
GPT-5.3-Codex
35.5
37.4
68.9
63.4
67.9
12.3
GPT-5.4
34.9
36.3
67.4
62.6
66.6
11.5
Table 5: Main results on ANCHOR. Task-only completion is measured on the original task objective without engineering constraints. Constrained completion is measured with the full ANCHOR developer instruction. ACR, CCR, IPR, and FCCR are reported on constrained runs. All numbers are percentages; higher is better. ACR, CCR, and IPR pool checklist items; FCCR uses the literal all-item criterion. Gold , silver , and bronze cells mark the best, second-best, and third-best values in each column.
Figure 7: Left: category-level Macro ACR for the primary paired set. Rates are averaged over task–configuration pairs containing at least one atomic item from the corresponding category. Error bars denote 95% bootstrap confidence intervals over tasks. Middle and right: reference-patch scope distributions over all 358 tasks. File counts include every modified file; changed lines count additions plus deletions within diff hunks. Each scope panel contains one observation per task.
Figure 8: Per-configuration top-level category breakdown. Each cell reports Macro ACR (%) for one scaffold–model configuration and one top-level category, macro-averaged over tasks containing at least one atomic constraint from that category. Columns use the C1–C10 shorthand from Table 2 . The heatmap covers the extended paired set (18 configurations). It makes it easier to identify whether a category-level trend in Figure 7 is broad-based or driven by a subset of configurations.
Category
Atomics
Tasks
Macro ACR
Typical evidence and checking methods
C1
371
234
55.7
Command presence/argument rules; semantic result verification may need an LLM.
C2
512
284
50.1
Observable action sequences; semantic ordering, explanations, and precedence need an LLM.
C3
518
274
66.6
Path/file/line checks; semantic scope and API compatibility need deeper inspection.
C4
223
167
65.7
Diff patterns and lint evidence; naming, documentation quality, and design need semantic judgment.
C5
283
174
63.9
Named imports and code patterns; algorithms, error handling, and side effects need semantic judgment.
C6
70
66
63.2
Paths, commands, and configuration; runtime compatibility may require execution or semantic evidence.
Table 6: Category counts, Macro ACR (%), and typical evidence and checking methods. Macro ACR averages over task–configuration pairs containing the corresponding category. The Tasks column counts distinct instances containing at least one such atomic item. Category names appear in Table 2 .
Model
Δ ACR
Δ CCR
ΔIPR
Δ FCCR
DeepSeek-V3.2
+10.5
+10.2
+10.5
+0.8
GLM-5
+2.2
+4.2
+2.6
+1.4
GLM-5.1
+0.6
−0.3
+0.5
+1.7
GPT-5.3-Codex
−4.8
−5.9
−5.0
−3.1
GPT-5.4
−3.1
−1.7
−2.9
+0.5
Kimi-K2.5
+1.8
+7.7
+2.8
−0.3
Table 7: Scaffold differences (Cline minus OpenCode) in percentage points.
Figure 9: Semantic breakdown of composite relation types for three models under both scaffolds.
Figure 10: Structural diagnosis of composite constraints. From left to right: FCCR as a function of task-level graph load, computed on the union graph induced by all composite constraints in a task; ACR as a function of the number of atomic nodes in a composite; and CCR as a function of the number of relation edges in a composite. Each scaffold curve averages the seven models in the primary paired set under that scaffold. Sample counts are shown in the x-axis tick labels. The displayed counts sum to 358 tasks in the graph-load panel, 531 observations grouped by composite node count, and 596 grouped by composite edge count. The latter two are panel-specific observation counts, distinct from the benchmark-wide atomic and relation-item totals. The ACR and CCR panels use truncated y-axes for readability.
Figure 11: Per-configuration composite-constraint breakdown. Each cell reports mean CCR (%) for one scaffold–model configuration and one composite bucket. The first three columns correspond to relation types ( AND , THEN , PRIORITY ); the remaining columns correspond to retained structural-load bins, where load is defined as ∣V∣+∣E∣ . The heatmap shows that the THEN penalty is broad-based across the extended paired set, while performance on higher-load composites is much more model-sensitive.
Figure 12: Constrained task completion versus IPR for the full evaluation set in Table 5 . Colors identify models, shapes identify scaffolds, and lines connect configurations using the same foundation model across scaffolds. Each point represents a complete agent configuration.
Model
OpenCode
Cline
FCCR (S)
FCCR (F)
FCCR (S)
FCCR (F)
DeepSeek-V3.2
4.8
0.7
6.0
1.5
GLM-5
6.7
2.9
4.2
6.3
GLM-5.1
3.9
7.4
7.4
8.0
GPT-5.3-Codex
11.9
12.5
11.9
7.6
GPT-5.4
13.8
10.1
12.9
11.5
Table 8: Literal all-item FCCR on task-successful (S) and task-failed (F) trajectories for the extended paired set (18 configurations). All values are percentages.
Figure 13: Attributed behavioral patterns across failed ANCHOR checklist items in the primary paired set. Left: overall distribution over 20,726 failures. Middle: row-normalized composition by scaffold. Right: row-normalized composition by relation type for 4,026 failed composite items. E1 = not perceived; E2 = misunderstood; E3 = deprioritized; E4 = capability limitation; E5 = scaffold interference; E6 = partial compliance.
Figure 14: Row-normalized error attribution by model for the primary paired set. Rows aggregate both scaffolds; cells show the percentage of attributed failures assigned to E1–E6, and counts in parentheses show row totals. The labels describe behavioral patterns, not causal effects of model capability.
Figure 15: Row-normalized error attribution by atomic category. Cells show the percentage of attributed failures assigned to E1–E6; counts in parentheses show row totals. Workflow and Testing are dominated by E1, Modification Scope by E3, and Output Format & Reporting has a heavier mix of E5 and E6.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Subcategory
Leaf category
Count
Example constraint
C1 Tool Execution Protocol
1.1 Tool selection
1.1.1 Tool Preference & Substitution
44
Use rg rather than grep for search; use fd rather than find .
1.1.2 Tool prohibition
1
Do not use sed -i for bulk replacement; do not use curl .
1.1.3 Tool Composition & Pipeline
1
First locate the target with rg , then edit the file.
1.2 Invocation parameters
1.2.1 Required parameters
74
Run pytest with --timeout=30 .
1.2.2 Forbidden parameters
36
Do not run git commit with --no-verify .
Appendix
Table 9: Mid- and leaf-level categories in the ANCHOR taxonomy, with instantiated counts and representative example constraints.
Repository-level software engineering benchmarks have significantly advanced the evaluation of coding agents, but existing benchmarks primarily measure whether generated patches pass functional tests and overlook review-derived acceptance constraints (review constraints) that often influence whether a patch is acceptable in real-world software development. We introduce SWE-Gate, a repository-level benchmark for software engineering agents that explicitly evaluates review constraint compliance alongside functional correctness. SWE-Gate derives review constraints from real pull request review comments and synthesizes repository-level repair instances around these constraints. Each instance provides separate functional and constraint tests, together with non-compliant and gold patches, enabling explicit separation between issue resolution capability and review constraint compliance. We construct SWE-Gate with 303 repository-level repair instances spanning 75 open-source Python repositories across diverse software domains. Experiments with four LLM backends spanning different capability levels under a common coding-agent scaffold reveal a substantial gap between functional success and success under the complete repair specification: among 644 repairs that pass the functional tests, 221 fail to satisfy the provided review constraints. These findings show that functional-only evaluation overestimates agents' ability to satisfy the full requirements of repository-level repair tasks. The replication package including code, data, and experimental results is available at https://github.com/DeepSoftwareAnalytics/SWE-Gate.
Xin He, Yanlin Wang, Mingwei Liu +3
School of Software Engineering, Sun Yat-sen University · College of Computer Science and Technology, Zhejiang University · School of Big Data and Software Engineering, Chongqing University +1
Developers increasingly delegate real maintenance work to product-grade coding agents, and many state tasks in their native language, in the style of a customer request rather than a curated English issue. Existing repository-level agentic benchmarks do not measure this setting: their task statements are English by design. We introduce RuBench 1.0, a benchmark of 25 tasks mined from recent fix commits in five live open-source repositories (aiohttp, aiogram, Laravel, NestJS, Fastify; Python, PHP, TypeScript, JavaScript), where each task is specified natively in Russian -- written from scratch in the style of an actual customer request, not translated -- and judged by the upstream maintainer's regression tests, which we withhold from release. All 25 fix commits postdate the training-data cutoffs of every evaluated model, giving a contamination argument that holds task-by-task. We evaluate deployed product configurations (CLI agent + model + reasoning effort) -- Claude Code with Opus 4.8, Sonnet 5, and Haiku 4.5, and Codex CLI with GPT-5.5 -- with three independent runs each, reporting pass@1 with task-level confidence intervals, paired comparisons, dollar cost, and token usage. The best configuration resolves 78.7% of tasks; at N=25 only the gaps to the weakest model are statistically resolvable, which we state explicitly. Auditing full trajectories of a fifth, hors-concours configuration (Claude Code + Fable 5, July 2, 2026 release), we caught the product silently substituting the model: on 5 of 25 tasks (20%) an official safeguard fallback re-routed routine HTTP-protocol fixes to Opus 4.8 -- direct, reproducible evidence that the deployed product, not the model, is the unit actually measured. We release task statements, metadata, full agent trajectories, and diffs; grading oracles are withheld, with a SHA-256 manifest committed at publication time.
Coding agents are increasingly asked to build software rather than patch it, yet benchmarks for from-scratch repository construction are mostly limited to a single language and depend on manually curated tasks. We introduce Zero2Repo, a benchmark in which an agent receives a product requirements document, an interface contract, and an empty workspace, and must deliver a complete repository in the project's native ecosystem. Tasks are produced by a language-agnostic authoring pipeline that converts real, version-pinned open-source projects into behavioral specifications, reproducible environments, and hidden acceptance tests. Each task is validated by execution: a reference implementation derived from the upstream project must pass, and adversarial validation must show that the tests reject incorrect implementations. Evaluation runs production coding agents in isolated containers, withholds the acceptance tests until an explicit submission, and assigns a binary reward only when every test passes, with no LLM judge. The pipeline and harness make no language-specific assumptions and apply to mainstream programming ecosystems; the current release contains Python, TypeScript, Go, and C++ tasks. Even on 11 tasks drawn from repositories that frontier models have very likely seen during training, the strongest agent solves only 10, and every failing submission passes 90-99% of the hidden tests; for the two strongest agents, 67-100% of failed tests trace to a single omission or a low-frequency rule stated in the specification rather than to a missing subsystem, so each failure is a concrete target for improvement.
Pei Yang, Tianyu Shi, Yuhang Yao +23
Gradient Data · McGill University · Carnegie Mellon University +6