cs.SEOct 15, 2026

Agentic Interpretation: Harness-Governed Evidence Construction for Software Agents

Authors: Yifeng He, Yanjun Wang, Siwei Cui, Aritra Sengupta, Murali Krishna Ramanathan

Organizations: University of California, Davis, Davis, California, USA · AWS AI Labs, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Software agents answer behavioral questions about code, such as whether a patch passes tests or which lines caused a failure. These tasks require repository-wide evidence, yet most scaffolds leave intermediate findings in free-form trajectories, making them hard to reuse and easy to disconnect from final answers. We present Agentic Interpretation (AgInt), a framework for harness-governed evidence construction. The model proposes typed records, while a deterministic harness admits, grades, and persists them, reports unresolved requirements, and licenses answers under task-specific rules. The resulting certificate gives the model persistent external state and leaves an auditable decision record. Formally, append-only certificate states form a grounding lattice ordered by record inclusion, where each admitted record extends the state by union. We evaluate AgInt with four models on SWE-bench-Live and DeepSWE for patch acceptance and Defects4J and GitBug-Java for fault localization. AgInt outperforms the plain agent and Agentic Code Reasoning on all reported patch-acceptance metrics and ranks responsible lines higher while inspecting fewer irrelevant lines in fault localization. For GLM-5, reject-class F1 rises from 0.256 to 0.543 on SWE-bench-Live, All@1 rises from 0.140 to 0.400 on Defects4J, and wasted inspected lines fall from 31.5 to 11.7 on GitBug-Java. A matched-template control separates the mechanism: the persistent scaffold drives most fault-localization gains, while the certificate body and gate govern patch-acceptance calibration. With execution licensed, AgInt produces harness-reproduced witnesses for 293 of 475 null-dereference candidates versus 253 for a matched shell agent, and 62 of 117 resource-leak candidates versus 58. Across reported efficiency measurements, it uses 35% fewer executions and reduces cost per candidate by 59%.

Figures & tables

Figure 1. A toy grounding lattice for three binary slots.
Figure 2. The licensing check at the model–harness boundary. Each column reads from certificate state EE , through model proposal pp , to the harness decision. Close refuses a verdict that contradicts EE or leaves an obligation open. Prose that claims reproduction remains cited because only a successful harness-run witness can earn the witnessed grade. Three vertical trajectories show a contradicted commit, a commit with an open obligation, and a model claim that cannot self-assign the witnessed grade. Each trajectory ends in a harness refusal.
Figure 3. How the harness decides at runtime which proposed record to admit and which to refuse. The model proposes a record through a certificate tool; the harness admits it as atom aa and unions it into EE , or refuses it. The harness, not the model, performs the union.
Figure 4. The certificate template of each evaluated task, condensed. Each field is a slot or slot family; Region.R{n} and Claim.k are repeated records whose children carry their own obligations. The harness enforces each gate; task instructions supply each field’s natural-language meaning. Three monospaced templates. Patch acceptance lists definition, premise, region, adversarial, and conclusion fields and a gate that maps region outcomes to a verdict. Fault localization lists ranked regions with cited analyses and a gate that binds each rank to its analysis. Bug witnessing lists claims, a harness-only WITNESS slot, and a gate that requires a certified witness.
Figure 5. Scaffold-versus-body decomposition on GLM-5.
bug typeanalyzer#AgIntbaseline
null dereferenceSonarQube197119 (60.4%)101 (51.3%)
Infer278174 (62.6%)152 (54.7%)
resource leakSonarQube7030 (42.9%)27 (38.6%)
Infer4732 (68.1%)31 (66.0%)
turns/candidate34.737.0
executions/candidate15.924.5
Table 3. Bug-witness yield and efficiency using GLM-5 for both AgInt and the matched shell-agent baseline on the fixed 3030 -project corpus. # is the pooled, deduplicated candidate count; rates are witnesses/#. Efficiency values are averages over the candidates in their respective evaluations.
real bug?witnessednot witnessed
✓1420
✗0142
Table 4. Soundness-floor confusion matrix on the 142142 matched Juliet pairs ( 142142 sites per row).
More in this topic

Software Engineering Agents

Software Engineering Agents are AI systems that use code, tools, and feedback to perform software development tasks.

See all 337 papers

Explore similar work

CardsList
Mar 25, 2026

Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code

Code agents resolve 65-70% of SWE-bench Verified issues, but Pass@1 cannot tell us why the rest fail, and, as we show, capable-model failures are systematically misdiagnosed without trajectory data. We introduce TRAJEVAL, a training-free decomposition of agent trajectories into reference-patch-aligned search, read, and edit stages, and apply it across 16,758 trajectories spanning three architectures and seven models. The dominant failure of capable models is not localization: 60-69% of failures on SWE-Agent and OpenHands reach and edit the correct functions yet still produce incorrect patches, and the pattern persists for most models on the bash-only LiveSWEAgent. Within this Edit-Quality residual, we identify Coherence Collapse, where the agent reaches correct code and then overwrites or thrashes it, as the largest theme, replicating across SWE-bench Verified and the multilingual PolyBench Verified. In 5 cases, the agent produces a patch bit-identical to the gold reference mid-trajectory and destroys it later; an edit-commit checkpoint recovers all 5 against the SWE-bench Docker harness. A reference-free consensus-driven variant yields a directional +3.0 pp Pass@1 measurement on GPT-5 (p=0.08).

Myeongsoo Kim, Dingmin Wang, Siwei Cui +6

AWS AI Labs · Monash University

Agent Failure AnalysisAI Coding AgentsCode Generation EvaluationAgent Trajectory AnalysisSoftware Engineering Benchmarks
AskPDF
Sep 3, 2026

SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents

Repository-level software engineering benchmarks have significantly advanced the evaluation of coding agents, but existing benchmarks primarily measure whether generated patches pass functional tests and overlook review-derived acceptance constraints (review constraints) that often influence whether a patch is acceptable in real-world software development. We introduce SWE-Gate, a repository-level benchmark for software engineering agents that explicitly evaluates review constraint compliance alongside functional correctness. SWE-Gate derives review constraints from real pull request review comments and synthesizes repository-level repair instances around these constraints. Each instance provides separate functional and constraint tests, together with non-compliant and gold patches, enabling explicit separation between issue resolution capability and review constraint compliance. We construct SWE-Gate with 303 repository-level repair instances spanning 75 open-source Python repositories across diverse software domains. Experiments with four LLM backends spanning different capability levels under a common coding-agent scaffold reveal a substantial gap between functional success and success under the complete repair specification: among 644 repairs that pass the functional tests, 221 fail to satisfy the provided review constraints. These findings show that functional-only evaluation overestimates agents' ability to satisfy the full requirements of repository-level repair tasks. The replication package including code, data, and experimental results is available at https://github.com/DeepSoftwareAnalytics/SWE-Gate.

Xin He, Yanlin Wang, Mingwei Liu +3

School of Software Engineering, Sun Yat-sen University · College of Computer Science and Technology, Zhejiang University · School of Big Data and Software Engineering, Chongqing University +1

Software Engineering AgentsAutomated Program RepairSoftware Engineering BenchmarksLLM Agent EvaluationRepository-Level Program Repair
AskPDF
Oct 6, 2026

Humanize: Judgement Engineering for Agentic Coding

Agentic coding makes code generation cheap, but reliable completion remains difficult: the agent that writes the code is a weak judge of whether it is done. We present Humanize, a multi-agent orchestration workflow for agentic coding built around judgement engineering: explicit, mechanically enforced decisions at the boundaries between planning, implementation, review, and learning. A human approves a plan contract, a builder agent implements it in rounds, and a reviewer agent from another vendor decides completion; deterministic hooks, not a model, route work between these roles and enforce 72 mechanical gates. Viewed as a Markov chain over repository states, alternating builder and reviewer samples jointly from two models, so a defect survives only if both miss it. We study Humanize through its deployment, 118 public postmortems of real loops, and its applications. Over 68 versions in 108 days, it gathered 1,468 GitHub stars. Applications include a 567-file gem5 build-system migration under upstream review; Kernel Design Agents, which extend the loop with a kernel knowledge base and profiling feedback and placed in the top three of all three Full-Agent tracks of the MLSys 2026 FlashInfer contest; and, through Humanize Olympiad Agents (HOA), full scores in IOI 2026, IMO 2026, IPhO 2026, and IBO 2024, 418.5/437 in IChO 2026 (gold-medal). Humanize also achieves 672/672 on PutnamBench and ranks first (251/303) on Lean-Eval's leaderboard even competiting with professional mathematicians. The postmortems show that independent review catches unsupported builder claims, but stopping remains a key weakness. In reports that separate rounds by phase, two thirds of rounds occurred after implementation was accepted. This evidence is observational, not a controlled comparison of workflows.

Sihao Liu, Ligeng Zhu, Zijian Zhang +6

NVIDIA · UCLA · Tsinghua University +2

Multi-Agent CodingLLM Agent OrchestrationLLM Agent ReliabilityHuman-in-the-Loop AIRuntime Enforcement for AI Agents
AskPDF