A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations

Hasan Najib Mahmud; Shreya Gupta; Isha Chaudhary; Nathaniel Enis; Ravi Mangal; Gagandeep Singh; Corina Pasareanu

cs.AIAug 18, 2026

A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations

Authors: Hasan Najib Mahmud, Shreya Gupta, Isha Chaudhary, Nathaniel Enis, Ravi Mangal, Gagandeep Singh, Corina Pasareanu

Organizations: Colorado State University · Microsoft · University of Illinois Urbana-Champaign · Carnegie Mellon University

Abstract

AI code agents are increasingly deployed to resolve real software issues, yet their reliability under superficial code variations remains poorly understood. We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding codebase is rewritten into a semantically equivalent form. We introduce a random variant sampler that applies common semantics-preserving transformations (SPTs) - spanning control-flow rewrites, dead-code injection, and identifier renaming - to produce perturbed variants. We evaluate two agentic scaffolds (mini-SWE agent and OpenCode) each backed by one of four frontier models (Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, and Qwen 3.6-27B) across instances drawn from SWE-bench Verified and SWE-bench Pro. For each instance, the agent is run multiple times on the unperturbed and perturbed variants, yielding paired resolve-rate estimates that isolate the perturbation effect from intrinsic stochasticity. We find small degradation in most configurations: up to 6.7 percentage points mean resolve-rate drop in the most affected configurations with statistically significant degradations in 6 of 16 configurations of model, scaffold, and dataset. Crucially, no single model ranking by robustness holds across scaffolds - Qwen is among the most robust under mini-SWE agent on SWE-bench Verified yet the most brittle under OpenCode - revealing a jagged robustness frontier. The simpler scaffold (mini-SWE agent) is more robust to perturbation. Our results demonstrate that even top frontier models are susceptible to semantics-preserving perturbations although the effect is not uniform, raising concerns about the deployment reliability of AI code agents in diverse real-world codebases.

Figures & tables

Transformation	Summary
If Else Switcher	Swaps if/else branches and negates the condition
For Loop Rewriting	Rewrites a for loop using an explicit iterator
And Condition Splitter	Decomposes if A and B into nested if s
Comparison Swapper	Swaps operands and inverts the operator
While Loop Unrolling	Unrolls one iteration of a while loop
Double Negation Injector	Wraps a condition in not not ( $\cdot$ )

Table 1: Catalog of semantics-preserving transformations (SPTs).

Figure 1: Mean degradation $\bar{\Delta}$ (percentage points) for each model under each scaffold on both benchmarks. Error bars are fixed-population 95% bootstrap confidence intervals.

Panel of Figure 2 — Figure 2: Mean relative change in agent steps ( $\bar{\delta}_{\text{step}}$ ) and cost ( $\bar{\delta}_{\text{cost}}$ ) after perturbation, for each scaffold–model configuration on the 28 SWE-bench Verified instances (left) and the 26 SWE-bench Pro instances (right), restricted to runs the agent resolved in both cases. Positive values mean perturbed runs take more steps, or cost more, than unperturbed runs on the same instance. Error bars are fixed-population 95% bootstrap confidence intervals.

Category	mini-SWE	OpenCode
Architecture	Single, centralized agent	Primary agents invoke tools and subagents for specific tasks (e.g., Build, Plan, General, Explore)
Available Tools	bash	bash , edit , write , read , grep , glob , lsp , apply_patch , skill , todo_write , web_fetch , web_search , question
Access Restrictions	None	Available tools determined by an agent’s role and user configuration
Context Management	Continuous, append-only	Multiple, dynamically changing contexts for different subagents

Table 2: Comparison of mini-SWE and OpenCode Agent Scaffolds

Figure 3: Mean resolve rate (%) on unperturbed and perturbed runs for each model under each scaffold, on SWE-bench Verified (top) and SWE-bench Pro (bottom).

Panel of Figure 4 — Figure 4: Mean relative change in agent steps ( $\bar{\delta}_{\text{step}}$ ) and cost ( $\bar{\delta}_{\text{cost}}$ ) after perturbation over all runs, for each scaffold–model configuration on the 28 SWE-bench Verified instances (left) and the 26 SWE-bench Pro instances (right). Positive values mean perturbed runs take more steps, or cost more, than unperturbed runs on the same instance. Error bars are fixed-population 95% bootstrap confidence intervals.

Figure 5: Per-instance degradation $\Delta(i)$ for the 28 SWE-bench Verified instances under each configuration. Error bars are Newcombe 95% confidence intervals for the difference between the unperturbed and perturbed resolve proportions ( $n{=}20$ runs per condition).

Figure 6: Per-instance degradation $\Delta(i)$ for the 26 SWE-bench Pro instances under each configuration. Error bars are Newcombe 95% confidence intervals for the difference between the unperturbed and perturbed resolve proportions ( $n{=}20$ runs per condition).

AI Coding Agents

AI coding agents use language models and software tools to plan, edit, test, and maintain code with varying levels of autonomy.

See all 327 papers

Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code
Mar 25, 2026Myeongsoo Kim, Dingmin Wang, Siwei Cui +6Agent Failure Analysis AI Coding Agents
AWS AI Labs · Monash University
Teaching Agents to Code Reliably
Oct 2, 2026Muhammad Ahmed Mohsin, Myeongsoo Kim, Kangrui Ruan +3Repository-Level Program Repair AI Coding Agents
Stanford University · AWS AI Labs
RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations
May 25, 2026Hanyu Li, Yichi Zhang, Speed Zhu +3AI Coding Agents Software Engineering Benchmarks
Beijing University of Posts and Telecommunications · Tsinghua University · Tencent

Mar 25, 2026J/K move · Enter open · S save

Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code

Myeongsoo Kim, Dingmin Wang, Siwei Cui, Farima Farmahinifarahani+5

AWS AI Labs · Monash University

Agent Failure Analysis AI Coding Agents Code Generation Evaluation Agent Trajectory Analysis Software Engineering Benchmarks

Open paper AskPDF arXiv

Code agents resolve 65-70% of SWE-bench Verified issues, but Pass@1 cannot tell us why the rest fail, and, as we show, capable-model failures are systematically misdiagnosed without trajectory data. We introduce TRAJEVAL, a training-free decomposition of agent trajectories into reference-patch-aligned search, read, and edit stages, and apply it across 16,758 trajectories spanning three architectures and seven models. The dominant failure of capable models is not localization: 60-69% of failures on SWE-Agent and OpenHands reach and edit the correct functions yet still produce incorrect patches, and the pattern persists for most models on the bash-only LiveSWEAgent. Within this Edit-Quality residual, we identify Coherence Collapse, where the agent reaches correct code and then overwrites or thrashes it, as the largest theme, replicating across SWE-bench Verified and the multilingual PolyBench Verified. In 5 cases, the agent produces a patch bit-identical to the gold reference mid-trajectory and destroys it later; an edit-commit checkpoint recovers all 5 against the SWE-bench Docker harness. A reference-free consensus-driven variant yields a directional +3.0 pp Pass@1 measurement on GPT-5 (p=0.08).

Abstract

AI Coding Agents

Explore similar work

Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code

Teaching Agents to Code Reliably

RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations