AI coding agents can now complete complex programming tasks, but existing evaluations largely emphasize behavioral correctness and often overlook maintainability risks such as weak modularity or testability. We present Needle in the Repo (NITR), a diagnostic probe-and-oracle framework for evaluating whether behaviorally correct repository edits preserve maintainable structure. NITR distills recurring software engineering wisdom into controlled probes embedded in small, realistic multi-file codebases, each designed so that success depends primarily on one targeted maintainability dimension. Each probe is paired with a hidden evaluation harness that combines functional tests for required behavior with structural oracles that encode the targeted maintainability constraint and return interpretable diagnoses. Using NITR, we evaluate 23 coding configurations across GPT, Claude, Gemini, and Qwen families in both direct-inference and agent-based settings. Current AI coding systems remain far from robust: on average, configurations solve only 36.2% of cases, the best reaches 57.1%, and performance drops from 53.5% on micro cases to 20.6% on multi-step cases. The hardest pressures are architectural rather than local edits, especially dependency control (4.3%) and responsibility decomposition (15.2%). Moreover, 64/483 outcomes (13.3%) pass all functional tests yet fail the structural oracle. Under our harness, agent-mode configurations improve average performance from 28.2% to 45.0%, but do not eliminate these architectural failures. These results show that progress in code generation is not yet progress in maintainable code evolution, and that NITR exposes a critical failure surface missed by conventional evaluation.
Figures & tables
Figure 1: Existing evaluations rank models by behavioral success, but leaderboard performance says little about maintainability risk. NITR uses curated probes and structural oracles to expose failure modes beyond test passing.
Figure 3: From maintainability pressure to diagnostic probe. NITR summarizes SE practices into 9 recurring repository-evolution pressures, instantiates each target pressure as a compact probe with starter code and an agent-facing task, and pairs the probe with functional and structural oracles.
Figure 4: Anatomy of a maintainability probe. Each probe contains starter code, an agent-visible task ( TASK.md ), an author-facing specification ( SPEC.md ), and a hidden evaluator. During evaluation, the model sees only the starter code and TASK.md ; the probe passes only if the generated edits satisfy both the functional tests and the structural probes.
Figure 5: Multi-step probe execution. Each step is applied to the codebase produced by the previous step, so early design choices persist across later changes. The final codebase is then evaluated with tests and structural oracles.
Dim.
Name
#Cases
#Multi-step
D1
Change Locality
3
1
D2
Reuse & Repo Awareness
3
2
D3
Responsibility Decomposition
2
1
D4
Extension Structure
2
1
D5
Interface & Substitutability
2
1
D6
Dependency Control
2
1
Table 1: The nine maintainability dimensions used as probe-construction axes in NITR . Each case is assigned exactly one primary dimension. The last column reports how many cases in that dimension involve multiple steps.
001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
Dim.
D1
D2
D2
D3
D4
D5
D5
D6
D7
D9
D1
D8
D1
D4
D6
D7
D8
D7
D9
D3
D2
Type
m
m
m
m
m
μ
m
μ
μ
μ
μ
μ
μ
μ
m
m
m
m
m
μ
μ
Pass
5
0
22
0
0
2
10
2
21
21
10
23
18
19
0
8
4
2
1
7
0
S
13
0
0
0
0
4
0
4
0
0
0
0
0
0
0
0
0
0
0
11
0
Table 2: All 21 repositories in NITR with their dimension, granularity, and aggregate outcomes across 23 evaluated configurations. S counts are bolded where ≥4 , highlighting probes where functionally correct code still fails the maintainability oracle.
Figure 7: Both API-only and agent-mode systems perform substantially better on micro cases than on multi-step cases, indicating that the main difficulty lies in evolutionary tasks rather than isolated edits.
Model
Mode
#
Rate
D1
D2
D3
D4
D5
D6
D7
D8
D9
GPT-5.3-Cx
Agent
12
57%
2/3
1/3
1/2
1/2
1/2
1/2
2/3
2/2
1/2
GPT-5.2-Cx
Agent
11
52%
3/3
1/3
0/2
1/2
1/2
1/2
2/3
1/2
1/2
GPT-5
Agent
11
52%
2/3
1/3
1/2
1/2
1/2
0/2
3/3
1/2
1/2
GPT-5.4
Agent
10
48%
2/3
1/3
1/2
1/2
1/2
0/2
2/3
1/2
1/2
GPT-5.4
API
7
33%
1/3
1/3
0/2
1/2
0/2
0/2
1/3
2/2
1/2
GPT-5-Mini
API
6
29%
0/3
1/3
0/2
1/2
1/2
0/2
1/3
1/2
1/2
Table 3: Pass rates of evaluated configurations on NITR (21 cases). Agent uses agentic scaffolding; API uses direct inference. Per-dimension cell background: x all pass, x majority, x minority, x none.
Model
001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
GPT-5.3-Cx (Agt)
S
S
✓
F
B
S
✓
✓
✓
✓
✓
✓
✓
✓
F
✓
✓
F
F
✓
F
GPT-5.3-Cx (API)
S
S
✓
F
B
B
✓
F
✓
✓
F
✓
F
✓
F
F
S
F
B
S
F
GPT-5.4 (Agt)
S
S
✓
F
B
S
✓
F
✓
✓
✓
✓
✓
✓
F
✓
S
F
B
✓
F
GPT-5.4 (API)
S
S
✓
F
B
E
F
F
✓
✓
F
✓
✓
✓
F
F
✓
F
B
S
F
GPT-5.2-Cx (Agt)
✓
B
✓
F
B
S
✓
✓
✓
✓
✓
✓
✓
✓
F
✓
S
F
B
S
F
GPT-5 (Agt)
S
B
✓
E
B
S
✓
B
✓
✓
✓
✓
✓
✓
F
✓
S
✓
B
✓
F
Figure 8: Pass/fail heatmap of 23 evaluated configurations across 21 cases. X Pass, X S-category failure (functional tests pass, maintainability oracle fails), X Functional test failure, X Both fail, X Build failure. Cases marked X confirm that functional tests alone miss these maintainability failures, which are detectable only via structural probes.
Failure archetype
Defining behavior
Representative cases
Outcome
Shortcut over reuse or abstraction
Clones logic or creates a parallel path instead of routing the change through a reusable or repository-aligned abstraction
001, 002, 011, 021
S / F
Boundary contamination
Preserves visible behavior but violates responsibility, ownership, or side-effect boundaries
004, 019, 020
S / F / B
Incomplete abstraction refactor
Starts the right abstraction shape but does not propagate the change consistently across the repository
006, 008, 014, 015
E / B
Later-step regression
Works for earlier milestones but breaks baseline behavior or collapses under later extensions
005, 007, 016, 017, 018, 019
F / B
Missing test seam
Leaves time, randomness, or configuration-sensitive behavior hardwired in core logic
009, 016, 018
F
Table 4: The table summarizes recurring qualitative patterns and representative cases; it is not an exhaustive coded annotation of all failures.
Large language model (LLM)-powered agents have demonstrated strong capabilities in automating software engineering tasks such as static bug fixing. However, in the real world, the development of mature software is typically predicated on complex requirement changes and long-term feature iterations -- a process that static, one-shot repair paradigms fail to capture. To bridge this gap, we propose SWE-CI, the first repository-level benchmark built upon the Continuous Integration loop, aiming to shift the evaluation paradigm for code generation from static, short-term functional correctness toward dynamic, long-term maintainability. The key insight is simple: Maintainability can be revealed by tracking how functional correctness changes over time. The benchmark comprises 100 tasks, each deriving from a real-world code repository with a development history spanning an average of 233 days and 71 consecutive commits. SWE-CI requires agents to systematically resolve these tasks through dozens of rounds of analysis and coding iterations. SWE-CI provides valuable insights into how well agents can sustain code quality throughout long-term evolution.
Jialong Chen, Xander Xu, Hu Wei +2
Sun Yat-sen University · Skylenage · Alibaba Group
Coding agents are increasingly deployed for iterative development on real repositories, yet existing evaluation barely answers a basic question: do coding agents reuse existing code or reinvent the wheel? The question matters: every duplicated implementation is a fix applied twice and agents produce code far faster than humans can audit, so redundancy accumulates unsupervised. Thus, we present RepoReuse, a multi-turn benchmark for auditing code reuse in real repositories, where requirements are revealed turn by turn and the workspace accumulates across turns. It is built by a fully automated pipeline combining AST-based dependency graphs, guided evidence collection, and execution-verified task synthesis, and scales readily to new repositories. Beyond pass rates, we measure the reuse rate together with recall and cross-turn structural redundancy. An audit over 3{,}000 turns shows that agents progressively stop exploring relevant repository code, reuse their own history less even when it is fully in the workspace, and leave duplicated logic in 50.8% of task chains by turn~5---all while pass rates barely move. Such deficiencies are invisible to pass rates, underscoring the need to evaluate code generation beyond functional correctness.
Dongsheng Ma, Sizhe Wang, Xinyi Huang +5
Peking University · Fudan University · Shanghai Jiao Tong University +3
As autonomous coding agents see rapid adoption, their evaluation has primarily focused on task completion rates holding the target codebase fixed. This leaves a critical question unanswered: does the structural and stylistic quality, or ``cleanliness'' of the underlying code affect an agent's ability to navigate and modify it? To isolate the effect of code cleanliness from agent capability, we introduce an evaluation protocol built around minimal pairs: repositories that match on architecture, dependencies, and external behaviour, but differ on static-analysis rule violations and cognitive complexity. The pairs are constructed in both directions, by agent pipelines that either degrade a clean repository or clean a messy one. We author 33 tasks across six such pairs, evaluated through hidden tests at the application's public surface. Across 660 trials with Claude Code, code cleanliness does not change the agent's pass rate. However, it substantially alters the agent's operational footprint: agents working on cleaner code use 7 to 8% fewer tokens and reduce file revisitations by 34%. Our findings suggest that traditional maintainability principles remain highly relevant in the era of AI-driven development, shaping the computational cost and navigational efficiency of coding agents. Code cleanliness joins model choice, harness, and prompting as a factor that materially affects agent behaviours.