Beyond Surface Metrics: Multi-Condition Requirements in Enterprise Maintenance and AI Coding Benchmarks

Heeyong Eun

cs.SENov 2, 2026

Beyond Surface Metrics: Multi-Condition Requirements in Enterprise Maintenance and AI Coding Benchmarks

Authors: Heeyong Eun

Organizations: KPMG Korea

Abstract

Repository-level AI coding benchmarks measure task success but do not directly capture the conditions required for software maintenance completion. We study multi-condition fulfillment, where a maintenance task must satisfy multiple operational conditions, sometimes with directional dependencies. Using a fixed 18-criterion taxonomy, we analyze 2,925 Apache Fineract issues and 944 tasks from SWE-bench Pro, SWE-PolyBench Java, and SWE-EVO. We examine condition breadth, corpus-level co-occurrence, and source-grounded directional requirement relationships. Among 1,177 benchmark task pairs matched on issue classification, modified-file count, and code churn, the mean fulfillment-set Jaccard distance is 0.8138, and 57.4% have disjoint condition sets. Condition count alone does not explain this contrast: SWE-bench Pro averages 2.47 conditions per task, versus 1.78 for Fineract. Temporal (B4) and numerical (B2) conditions have a phi coefficient of +0.430 in Fineract versus -0.013 in the pooled benchmark corpus. Among temporally tagged tasks, numerical conditions are also specified in 48.85% (254/520) of Fineract cases versus 5.81% (5/86) of benchmark cases. We document 16 directional relationships in Fineract requirements involving time, calculation, and state preservation, and inspect four benchmark cases to illustrate how relevant conditions appear in evaluation assertions. These examples do not quantify benchmark-wide oracle omissions. The results show that similar surface task profiles can mask substantially different fulfillment-condition sets. We recommend reporting condition breadth, co-occurrence, and directional relationships separately. The corpus comparisons are descriptive and may reflect domain differences; they do not establish causality or production failure rates.

Figures & tables

Analytical Dimension	SNC Audit [ 5 ]	UTBoost [ 7 ]	This Work
Primary Problem	Repository topography ambiguity and task demand categorization	Incomplete tests and false pass judgments	Multi-condition structure of maintenance completion
Unit of Analysis	Repository graph metrics (AST distance, spread, novelty)	Tests and candidate patches in SWE-bench tasks	Requirement fulfillment invariants ( $A\rightarrow\Delta S\rightarrow B$ )
Requirement Perspective	Surface issue labels (Bug, Feature, Enhancement)	Task-specific tests and patch evaluation	Ontological taxonomy (ISO 14764 / 25010) with verified quotes
Oracle Evaluation Scope	Accepts existing benchmark test harnesses as ground truth	Augments benchmark tests to flag incorrect passes	Secondary case-level assertion inspection
Empirical Scope	SWE-bench Lite/Verified (Python repository graphs)	SWE-bench Python repos (LLM-augmented test suites)	Analyzed AI tasks ( $N=944$ ) vs. Fineract issues ( $N=2,925$ )

Table 1: Research positioning by unit of analysis and primary contribution.

Panel of Figure 1 — Figure 1: Three complementary lenses for multi-condition fulfillment: breadth, co-occurrence, and source-grounded directionality. The illustrated dependency chain synthesizes FINERACT-753 and FINERACT-2338 rather than quoting a single task; oracle inspection is a downstream application.

Code	Codebook Criterion Name	AI All ( $N=944$ )	Pro ( $N=731$ )	Poly ( $N=165$ )	EVO ( $N=48$ )
A1	Product Operation & Business Rules	426 (45.13%)	300 (41.04%)	84 (50.91%)	42 (87.50%)
A2	Structure & Implementation Method	572 (60.59%)	505 (69.08%)	50 (30.30%)	17 (35.42%)
A3	Testing & Quality Assurance	94 (9.96%)	70 (9.58%)	10 (6.06%)	14 (29.17%)
A4	Execution & Operations	122 (12.92%)	93 (12.72%)	20 (12.12%)	9 (18.75%)
A5	Deployment & Environment Migration	60 (6.36%)	39 (5.34%)	9 (5.45%)	12 (25.00%)
A6	Development Process & Docs	53 (5.61%)	25 (3.42%)	17 (10.30%)	11 (22.92%)

Table 2: Stratified 18-Criteria Distribution Across Benchmark Families (

N=944

Metric / Bin	Fineract ( $N=2,925$ )	AI All ( $N=944$ )	Pro ( $N=731$ )	Poly ( $N=165$ )	EVO ( $N=48$ )
Mean $\|B\|$	1.78 (1.7798)	2.24 (2.2447)	2.47 (2.4733)	1.22 (1.2182)	2.29 (2.2917)
Median $\|B\|$	2.0	2.0	2.0	1.0	2.0
$\|B\|=0$	331 (11.32%)	3 (0.32%)	0 (0.00%)	3 (1.82%)	0 (0.00%)
$\|B\|=1$	872 (29.81%)	248 (26.27%)	109 (14.91%)	126 (76.36%)	13 (27.08%)
$\|B\|=2$	1,097 (37.50%)	357 (37.82%)	304 (41.59%)	33 (20.00%)	20 (41.67%)
$\|B\|=3$	451 (15.42%)	231 (24.47%)	219 (29.96%)	3 (1.82%)	9 (18.75%)

Table 3: Fulfillment-Condition Breadth: Distribution of

|B|

(not a measure of dependency strength).

Pair	Condition Pair (non-directional)	Fineract ( $N=2,925$ )	AI Bench ( $N=944$ )
		Co-occur / $\phi$	Co-occur / $\phi$ ( $p$ )
$B_{4}\leftrightarrow B_{2}$	Temporal / Numerical	254 / $\mathbf{+0.430}$	5 / $\mathbf{-0.013}$ ( $0.825$ )
$A_{1}\leftrightarrow B_{4}$	Business Rule / Temporal	471 / $\mathbf{+0.309}$	42 / $\mathbf{+0.024}$ ( $0.496$ )
$A_{1}\leftrightarrow B_{2}$	Business Rule / Numerical	440 / $\mathbf{+0.343}$	34 / $\mathbf{+0.039}$ ( $0.246$ )
$A_{1}\leftrightarrow B_{3}$	Business Rule / Consistency	538 / $\mathbf{+0.329}$	115 / $\mathbf{+0.094}$ ( $0.005$ )
$B_{4}\leftrightarrow B_{3}$	Temporal / Consistency	138 / $\mathbf{+0.069}$	22 / $\mathbf{+0.022}$ ( $0.501$ )

Table 4: Pairwise Co-occurrence (

\phi

); associations do not establish causal direction.

ID	Issue	Classification	Structural Chain	Verbatim Jira Excerpt
Causal Cascade Focus: Multi-Tier Loan Modification (FINERACT-2338)
DEP-01	FINERACT-2338	Explicit Dependency	$A_{1}\rightarrow B_{4}$	“Backdate can be done on already paid Installments as well”
DEP-02	FINERACT-2338	Explicit Dependency	$B_{4}\rightarrow B_{2}$	“Repayment schedule will be recalculated with New EMI and Interest from the Interest applied schedule date”
DEP-03	FINERACT-2338	Explicit Dependency	$B_{2}\rightarrow B_{6}$	“No of Installments will remain the same, only EMI and Interest would get affected”
DEP-04	FINERACT-2338	Explicit Dependency	$B_{4}\rightarrow B_{3}$	“standard accounting rules and not charge-off accounting rules”
Cross-Entity Invariant Preservation & Transaction Isolation

Table 5: Representative Source-Grounded Requirement Dependencies in Apache Fineract Maintenance (Full set of 16 documented relationships in the replication package).

Axis / Task	Specification Mandate ( $R_{\text{spec}}$ )	Evaluation Test Oracle ( $R_{\text{eval}}$ )	Observed Scope	Construct Alignment or Gap Analysis
Axis 1 (Direct) NodeBB #0030	Explicit ( $A_{5}\rightarrow B_{6}$ ) : DB migration script ( chat_allow_list.js , 44 LOC) migrating legacy restrictChat records.	Leaf Unit Assertion : Asserts canMessageUser() on mock objects in memory ( test/messaging.js , L103–121).	In-memory leaf	Construct Underrepresentation : Direct test patch asserts leaf permissions; DB migration script unasserted in analyzed test patch.
Axis 2 (Scope) Element-Web #0139	Explicit badge update; inferred cache persistence : Recalculate unread badges; IndexedDB cache synchronization is inferred rather than explicitly mandated.	Mocked State : Asserts helper return values under synthetic event mocks ( test/Unread-test.ts ).	In-memory mock	Mocked Logic vs. Persistent Cache : In-memory transitions verified; asynchronous IndexedDB cache persistence outside evaluation scope.
Axis 3 (Hierarchy) RocketMQ #0819	Explicit ( $B_{7},B_{2}$ ) : Thread-safe counter insertion ( StatsItemSet , 20kL) under JVM thread contention.	Thread Stress Test : 100 worker threads executing 10,000 tasks ( StatsItemSetTest.java ).	JVM thread contention	Single JVM Concurrency Met : Checks concurrent counter accumulation within a single JVM; distributed transaction boundaries are not mandated by the task.
Axis 4 (Aligned) Navidrome #0495	Explicit ( $A_{4},B_{6},B_{10}$ ) : Online SQLite DB backup, destructive DDL recovery, and retention pruning.	Integrated Engine Test : Executes backup, runs destructive DDL, restores DB, asserts pruning ( db/backup_test.go ).	Persistent DB state	Aligned Invariant Testing : Positive case in which the inspected tests directly address the explicitly required persistent-state operations.

Table 6: Comparative Case Analysis: Test Oracle Construct Coverage Across Four Qualitative Benchmark Axes (Unabridged matrix in replication package).

Software Maintenance

See all 8 papers

Oct 16, 2026

Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks

Benchmarks drive AI progress, but are hampered by task under-specification and over-specification. Task under-specification causes guesswork, where solutions have to make things up, or worse, resort to contamination or reward hacking. Task over-specification causes spoilers, where solutions can bypass some of the harder steps the benchmark was supposed to measure. Since future-looking benchmarks should be difficult, over-specification reduces a benchmark's shelflife of community adoption. Repository-level coding benchmarks have helped the field progress but experience unnecessary churn due to both over- and under-specification. For instance, SWE-bench Pro, FEA-bench, Doc2Feat-bench, and Multi-SWE-bench all tackle under-specification by augmenting task statements with hints derived from ground truth. We show that leaking ground-truth information into the task statement causes over-specification in some of these benchmarks. We experiment with various specification levels, trying to thread the needle between spoilers and guesswork. Based on these results, we recommend techniques for deriving parsimonious hints. We will release our hints so the community can obtain more reliable signals from popular benchmarks.

Yuanzhe Liu, Bastian Lipka, Avraham Shinnar +3

Rensselaer Polytechnic Institute · IBM

Benchmark Design Software Engineering Benchmarks Code Generation Benchmarks Benchmark Validity

Ask PDF

Oct 16, 2026

SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark

SWE-Bench Pro is a widely used benchmark of long-horizon software engineering tasks for coding agents. As agents improve, they reach task issues earlier agents never touched and interact with the evaluation environment in ways the original harness did not anticipate. We present SWE-Bench Pro V2, a revised public split built to keep the benchmark accurate as capabilities advance. Agentic audits, expert adjudication and blind re-implementation repaired or retired 480 of the 731 tasks: 23 contracted engineers spent 404 hours, each working in a language they write professionally, and every revised instruction was verified by a second engineer solving the task from the text alone. A hardened protocol restricts network access to the model endpoint and re-grades every patch in a clean container, so agents cannot retrieve upstream fixes or get credit for changes outside their patch. On the 642 V2 tasks frontier agents resolve up to 99.4% while other systems stay clearly separated, and a 51-task Hard split spreads systems from 52.9% to 98.0%. We release the benchmark and distill lessons for maintaining agent benchmarks over time.

Soham Dan, Jeff Da, Sanyam Bhutani +8

Scale AI · Reflection AI

Coding Agent Benchmarks Software Engineering Benchmarks Code Generation Benchmarks Long-Horizon Agent Evaluation Software Engineering Agents

Ask PDF

Mar 29, 2026

Needle in the Repo: A Benchmark for Maintainability in AI-Generated Repository Edits

AI coding agents can now complete complex programming tasks, but existing evaluations largely emphasize behavioral correctness and often overlook maintainability risks such as weak modularity or testability. We present Needle in the Repo (NITR), a diagnostic probe-and-oracle framework for evaluating whether behaviorally correct repository edits preserve maintainable structure. NITR distills recurring software engineering wisdom into controlled probes embedded in small, realistic multi-file codebases, each designed so that success depends primarily on one targeted maintainability dimension. Each probe is paired with a hidden evaluation harness that combines functional tests for required behavior with structural oracles that encode the targeted maintainability constraint and return interpretable diagnoses. Using NITR, we evaluate 23 coding configurations across GPT, Claude, Gemini, and Qwen families in both direct-inference and agent-based settings. Current AI coding systems remain far from robust: on average, configurations solve only 36.2% of cases, the best reaches 57.1%, and performance drops from 53.5% on micro cases to 20.6% on multi-step cases. The hardest pressures are architectural rather than local edits, especially dependency control (4.3%) and responsibility decomposition (15.2%). Moreover, 64/483 outcomes (13.3%) pass all functional tests yet fail the structural oracle. Under our harness, agent-mode configurations improve average performance from 28.2% to 45.0%, but do not eliminate these architectural failures. These results show that progress in code generation is not yet progress in maintainable code evolution, and that NITR exposes a critical failure surface missed by conventional evaluation.

Haichao Zhu, Qian Zhang, Jiyuan Wang +2

Independent Researcher · UC Riverside · Tulane University

Software Engineering AI Coding Agents Code Generation Benchmarks Repository-Level Code Generation

Ask PDF

Abstract

Software Maintenance

Explore similar work

Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks

SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark

Needle in the Repo: A Benchmark for Maintainability in AI-Generated Repository Edits