cs.LGJun 5, 2026

Strained Coherence: A Pre-Failure Signal in Coding Agent Execution Trajectories

Authors: Marut PandyaKasey ZhangBaiqing Lyu

Abstract

LLM-based coding agents sometimes acknowledge a problem in their own reasoning and then proceed anyway. We call this pattern strained coherence: a safety-relevant failure mode in which an agent has information that should change its behavior, states that information, and still acts against it. The pattern overlaps with verbalized reward hacking, where an agent names a tension between a task proxy and the underlying goal yet optimizes the proxy anyway. We give an operational definition, build a Claude Sonnet 4.6 judge that reads full trajectories and flags spans where the pattern occurs, and evaluate it on 44 Terminal-bench-2 trajectories using a Qwen3.5-35B-A3B backbone. Flagged trajectories fail 94% of the time versus 46% for unflagged trajectories (47-point gap, Fisher's exact p = 0.003; 46 points after excluding three prompt-embedded examples, p = 0.006). At matched selectivity, the detector reaches 94% precision versus 88% for a lexical discourse-marker baseline; the 10-trajectory intersection of the two methods has a 100% failure rate (Clopper-Pearson 95% CI [69%, 100%]). We replicate on Gemma4-31B with 43 trajectories: the overall signal is directionally consistent but not significant (20-point gap, p = 0.31), with attenuation driven largely by 13 trajectories with zero think content, where the detector has no substrate to analyze. In the high-verbosity Gemma tertile, the gap is +30 points; in the mid- and high-verbosity Qwen tertiles, it is +40 points each. The first flag appears at a median of 83-84% of elapsed trajectory time across both models, and the binary flag survives paraphrases that soften explicit conflict markers (8/8 trajectories). Unlike univariate predictors, the detector emits interpretable span-level output -- quoted acknowledgment, quoted action, and typed conflict -- showing what the agent saw and ignored.

Explore similar work

Mar 26, 2026cs.SE

Confident and Wrong: Silent Semantic Failures in Coding Agents

As coding agents move into production workflows, teams need to know not only whether an agent completes a task, but whether its action can be trusted. We show that completion and trustworthiness diverge sharply and systematically. Across 1,750 trajectories on 50 SWE-bench Verified tasks, we compare four frontier models over repeated runs and separate submit rate from test-verified resolve rate. GPT-5 submits a patch on 100% of runs but resolves only 44%; Llama 4 submits on 99% but resolves 18%; and Gemini, despite submitting least often at 70%, resolves more tasks than GPT-5 (50% versus 44%). These gaps are not random: they concentrate in one dangerous failure mode we call silent semantic failure. Qualitatively, on a buggy task the agent submits a plausible-looking patch on all five runs, yet none pass, the same misinterpretation repeated rather than random error. Quantitatively, it dominates failure, covering 80% of Llama 4's failing runs and 68% of GPT-5's, and it is invisible: the outcomes are confidently and consistently wrong, so completion-based and consistency-based monitoring both look healthy exactly when the agent should not be trusted. Lightweight pre-edit prompts do not close the gap. A second probe isolates the instinct to act: given an already-fixed bug, where the right move is to abstain, most models still edit the correct code. This action bias, acting when no action is warranted, is exactly what completion metrics reward. The throughline is measurement: submit rate captures action, but trust requires validity. So evaluation must catch up: score agents by test-verified correctness over repeated runs, report its uncertainty, and reward those that know when not to act.
Aman Mehta
Jul 10, 2026cs.SE

Failure as a Process: An Anatomy of CLI Coding Agent Trajectories

Large language model (LLM) coding agents are increasingly deployed to autonomously perform software engineering tasks in terminal-based environments, making their reliability a growing concern. Existing empirical studies investigate why coding agents fail, yet they largely treat failure as a final outcome rather than a temporal process, providing limited insight into how failures emerge, evolve, and become unrecoverable. We present the first large-scale empirical study of CLI coding-agent failure trajectories, introducing a process-oriented framework that analyzes failure through its onset, evolution, and recovery across execution trajectories. We first collect 3,843 execution trajectories generated by seven frontier models across three coding-agent scaffolds (OpenHands, MiniSWE, and Terminus2) on Terminal-Bench, then carefully filter them to obtain 1,794 complete and valid trajectories for manual annotation (over 63,000 execution steps), from which we derive 14 findings spanning failure occurrence, root causes, recovery, and cross-system consistency. Our findings show that coding-agent failures are predominantly driven by epistemic errors, typically begin within the first few execution steps, and often remain hidden until recovery is no longer possible, suggesting that improving coding-agent reliability requires earlier validation and intervention rather than relying solely on final-outcome evaluation.
Xiangxin Zhao, Han Li, Shuaiting Li +4
Jun 11, 2026cs.LG

Getting Better at Working With You: Compiling User Corrections into Runtime Enforcement for Coding Agents

Interactive LLM agents are becoming part of daily work, but they do not reliably become easier to work with over time: a correction remembered in one session may still be violated in the next. We study this gap between preference access and preference compliance. In tasks derived from anonymized real-user friction cases, Mem0 memory still leaves 57.5% of applicable preference checks violated. We introduce Test-time Rule Acquisition and Compiled Enforcement (TRACE), a drop-in skill-layer pipeline for coding-agent runtimes that mines user corrections, rewrites them as atomic rules, and compiles them into runtime checks that must pass before an agent completes future tasks. Unlike runtime checks written ahead of time by developers, TRACE skills come from the user's own chat corrections. We evaluate TRACE with simulated user-in-the-loop experiments on ClawArena coding-agent tasks and MemoryArena-derived memory-intensive tasks. On ClawArena, TRACE reduces held-out preference violation from 100.0% to 37.6% on in-distribution tasks and from 100.0% to 2.0% on out-of-distribution tasks. On MemoryArena-derived tasks, TRACE reduces in-distribution violation from 100.0% to 60.5% while matching or exceeding the strongest memory baseline on task pass. These results suggest that compiling corrections into runtime enforcement can address a repeated-friction failure mode that memory alone does not reliably solve, reducing the need for users to restate the same correction across future sessions. Experiment code is available at https://github.com/YujunZhou/TRACE_exp, and the deployable skill is available at https://github.com/YujunZhou/tellonce.
Yujun Zhou, Kehan Guo, Haomin Zhuang +8