Coding agents often continue working after their own tests pass. Does this additional work improve the patch, and how much could be saved by stopping earlier? We study these questions through anytime evaluation: replaying recorded runs, scoring intermediate patches with hidden tests, and measuring the cost incurred at each step. For policies that only stop a run or select an earlier patch, this allows paired comparisons without generating new trajectories, provided replay preserves the relevant states. Across about 6,000 runs of eight open models in two scaffolds on SWE-bench Verified, we find substantial differences in the value of continued work. The hindsight saving available at passing self-tests ranges from under 1% to 57%, depending on the model and scaffold. A rule that stops after the same patch passes twice saves 23% in a preregistered replication, but the preregistered quality test does not establish non-inferiority. We explain these differences by measuring how often a rule fires, how often the patch changes afterwards, and whether those changes repair or break it. Fine-tuning agents to imitate the rule saves 30% over six seeds, while quality preservation remains uncertain. Our results help identify both the opportunities and the risks of early stopping.
Figures & tables
Figure 1: Stopping saves cost, but its effect on patch quality depends on the agent. Two rules are compared with running to the end on the same recorded runs: stop at the first passing self-test, or after two passes on the same patch. Left: inference cost saved at c=0.1 . Right: change in local resolution under corrected scoring D′ (percentage points). Point estimates use all runs of the five model–scaffold combinations shown in Figure 4 , including derivation runs; these descriptive results are not the held-out confirmation test. “mini denotes mini-SWE-agent.
Figure 2: Anytime evaluation turns recorded commands into a patch-and-cost record. We recover intermediate patch states, score distinct patches with hidden tests offline, and align their scores with cumulative costs. The illustrative ledger contains four observations of two patches: repeated states reuse a score while cost continues to accumulate. Hidden-test results never enter the agent’s input; interventions that change its behavior require new runs.
Figure 3: Three rules applied to one illustrative trajectory. First pass stops at the first passing self-test; twice passed waits for two passes on the same patch. Both submit patch A and skip later work. Last pass runs to completion, then chooses A because the later patch B has no passing self-test. This selection provides no early-stopping cost saving. Self-test success does not establish hidden-test correctness.
Agent
Tasks
Offline
Online
Factor
Qwen3.6, mini
178
0.40
2.68
45.6
Qwen3.6, SWE-agent
193
1.28
2.58
4.0
Qwen3.8
371
0.23
1.81
60.5
Qwen3.5-9B
115
0.62
4.15
45.1
DeepSeek-V4-Flash
114
0.88
3.94
20.2
Table 1: Standard error (points) of a change in resolution for the twice-passed rule against running to the end, two runs per task: offline on the same runs, or online from independent runs (rule on one run, end on the other). Factor: tasks saved, (SEonline/SEoffline)2 . Qwen3.6 in mini-SWE-agent uses the 178 tasks of Section 4.5 ; rules that almost never fire are omitted.
Figure 4: Cost saved by the hindsight bounds and by two rules, with the change in resolution (points) at the end of each bar; all runs of each agent under D′ (Qwen3.6, mini-SWE-agent: 1,484 runs including the derivation runs). Oany is omitted for Qwen3.8, whose intermediate patches are not all scored.
Rule
Derivation (712)
Fresh tasks (230)
New repos (156)
New runs (386)
Self-test passed
+0.4 / −23.9%
−3.9 / −31.1%
−1.3 / −27.3%
−1.8 / −30.5%
Self-test passed twice on one patch
+0.7 / −17.5%
−1.7 / −23.1%
−1.3 / −17.6%
−1.0 / −22.8%
Last pass (no stopping)
+0.4 / 0%
+0.0 / 0%
+0.0 / 0%
+0.3 / 0%
Table 2: Paired rule comparisons on recorded runs of Qwen3.6-27B in mini-SWE-agent: change in local resolution (points) / change in cost at c=0.1 , relative to running to the end, under D′ . Rules were designed on the derivation runs; the fresh tasks (Django, SymPy) and the new repositories (78 tasks from seven repositories never run in our sandbox) test them, and the new runs (two further runs per task on both sets) replicate the preregistered tests. “Last pass” does not stop; it submits the last patch that passed a self-test if the agent changed it afterwards.
Figure 5: The decomposition ΔQ=−r⋅u⋅δ ( Equation 3 ) under D′ . Horizontal: share of all runs in which the rule stopped and the patch later changed; vertical: net repair per change. Dotted lines: equal loss in resolution (points). Twice-passed points lie left of the first-pass point of the same agent.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Experiment
Primary test
Expected
Verdict
Stopping, 712 runs
learned policy: ≤−15% cost, ≥−0.01 resolved, beats best simple rule
Table 3: Historical preregistered tests under the original D convention. These original verdicts are preserved for provenance; the D′ reanalysis in Appendix R supersedes affected quality verdicts. Paired bootstrap over tasks (2,000 resamples); thresholds on 95% CIs.
Figure 6: Stopping supervision for Qwen3.6-27B. Prefixes of the agent’s own runs, cut where its tests had passed twice on the same patch or earlier, with the targets shown; prefixes longer than 32K tokens are dropped.
178 tasks
256 tasks
Training seed
resolved
cost
resolved
cost
1
−0.4
−30.7%
−1.1
−31.4%
2
−0.7
−29.4%
−3.1
−30.0%
3
+2.4
−29.0%
+1.2
−28.1%
4
−2.4
−34.5%
−2.3
−33.2%
5
−1.3
−30.6%
−2.1
−29.3%
Appendix
Table 4: Stop distillation with six training seeds against four runs per task of the base model: change in local resolution (points) and cost at c=0.1 . Seed 1 has four runs per task, seeds 2–6 two. Below: each evaluation of seed 1 against the base model’s runs of the same period, and the base model’s second evaluation against its first.
Comparison
runs per task
1 point
2 points
3 points
Twice passed vs. end, same runs
2
221
56
25
First pass vs. end, same runs
2
554
139
62
Distilled vs. base agent, online
2
4,735
1,184
527
Distilled vs. base agent, online
8
1,184
296
132
Appendix
Table 5: Tasks needed to detect a change in resolution ( 80% power, two-sided 5% ), from the per-task variances on the 178 tasks.
Label
Setting
Teacher
Stopping
Online
e
Twice passed (XL)
mini, Qwen3.6
−0.6
−0.1
+1.0
0.02
Twice passed
mini, Qwen3.8
−0.3
−0.6
−0.3
0.02
Exact advantage, hidden / own tests
mini, Qwen3.6
+0.6 / −0.3
−0.3 / −0.3
−0.3 / −0.6
0.01 / 0.03
Random / matched position
mini, Qwen3.6
–
−0.3 / −0.3
−61.0 / −12.1
0.01 / 0.03
Twice passed
SWE-agent
−3.2
−3.9
−2.6
0.04
Hidden-test labels
SWE-agent
+0.6
−3.9
−3.2
0.09
Appendix
Table 6: Historical D diagnostic (not recomputed under D′ ): where a distilled agent’s change in resolution comes from (points against the base agent run to the end; held-out tasks, two runs each: 178 for mini-SWE-agent against the base agent’s runs 0–1, 78 for SWE-agent). Teacher: the labelling rule on the base agent’s runs. Stopping part: the fine-tuned model deciding only at the passes of the same runs. Online: the fine-tuned agent’s own runs. e : share of runs where the stopping part submits a different patch than the teacher. Imitation error is stopping part minus teacher; the rest is online minus stopping part, which includes sampling noise ( ≈ 1.6 points, Appendix O ). XL: mean of six training seeds; ∗ exploratory. This table is not a decomposition of the corrected six-seed result.
Comparison
D
D′
Fresh first pass
−3.0
−3.9
New-repository last pass
+3.2
0.0
Replication twice passed
0.0
−1.0
Replication last pass
+1.3
+0.3
Six-seed distillation, 256 tasks
−0.3
−1.2
Appendix
Table 7: Quality changes (points) under the original and corrected conventions. Test verdicts use the original prespecified thresholds. All corrected scores are local.
Rule
D
D′
P2
First pass
+3.0
+1.7
+3.5
Twice passed
+1.7
+0.9
+3.0
Last pass
+3.9
+2.6
+0.9
Appendix
Table 8: DeepSeek quality changes in points. Each convention uses its own run-to-end baseline.