cs.AINov 17, 2026

Done, But Still Working: Anytime Evaluation of When Coding Agents Should Stop

Authors: Zerui Yang, Quan Xue, Miaomiao Dong, Chaorui Zhang, Xuanyu Zheng, Xiyuan Wang, Haolin Ruan, Xueyan Niu, 

Organizations: Huawei Technologies Ltd.

Abstract

Coding agents often continue working after their own tests pass. Does this additional work improve the patch, and how much could be saved by stopping earlier? We study these questions through anytime evaluation: replaying recorded runs, scoring intermediate patches with hidden tests, and measuring the cost incurred at each step. For policies that only stop a run or select an earlier patch, this allows paired comparisons without generating new trajectories, provided replay preserves the relevant states. Across about 6,000 runs of eight open models in two scaffolds on SWE-bench Verified, we find substantial differences in the value of continued work. The hindsight saving available at passing self-tests ranges from under 1% to 57%, depending on the model and scaffold. A rule that stops after the same patch passes twice saves 23% in a preregistered replication, but the preregistered quality test does not establish non-inferiority. We explain these differences by measuring how often a rule fires, how often the patch changes afterwards, and whether those changes repair or break it. Fine-tuning agents to imitate the rule saves 30% over six seeds, while quality preservation remains uncertain. Our results help identify both the opportunities and the risks of early stopping.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Termination Lag: Coding Agents Solve Early but Stop Late

    Nov 11, 2026Musa Cim, Burak Topcu, Mahmut Taylan KandemirLLM AgentsLLM Agent Efficiency

  2. SWE-Shift: Evaluating Coding Agents in Evolving Workspaces

    Oct 23, 2026Victor Shea-Jay Huang, Rongqing Li, Zhangyue Yin +7Coding AgentsSoftware Engineering Benchmarks

  3. Knowing When to Stop: The Evidence-Action Gap in Coding Agents

    Oct 22, 2026Tianyu Chen, Xinran Zhao, Vijay Viswanathan +3AI Coding AgentsProcess-Based Agent Evaluation