cs.AIOct 6, 2026
SaveRippleCP: Measuring Counterfactual Checkpoint Advantage in Coding Agents
Organizations: Florida International University, Miami, USA
Abstract
Agent checkpoint systems decide what state is recovery-relevant, how to snapshot it, and whether rollback is admissible. None decides which of the safe boundaries they expose are worth materializing. We formulate this as counterfactual checkpoint advantage, the reduction in future recovery cost obtained by checkpointing a candidate rather than skipping it, and measure it by driving a CP branch and a SKIP branch to the same logical failure and recovering both under matched model, tool, verifier, and stopping conditions. On a frozen pilot of 12 SWE-bench Verified tasks and 106 real recovery branches, checkpointing saves 49.4 s per task, and that figure resolves into two regimes two orders of magnitude apart. The first checkpoint returns 100.0 s on 156.5 s of protected work, a conversion of 0.64; a second one step later returns-1.1 s on 41.8 s, a conversion of -0.03. Recovery is a re-derivation rather than a replay, so preserved work is a poor guide to saved work, and the classical elapsed-work rule misprices the second checkpoint by its full nominal cost. We identify where placement can pay, and set the bar a placement policy must clear.
Figures & tables
| Contrast | Tasks | CP (s) | SKIP (s) | Marginal value (s) | Conversion |
|---|---|---|---|---|---|
| First checkpoint vs. none | 10 | 100.8 | 200.8 | 100.0 [72.4, 131.1] | 0.64 |
| One additional, one step later | 10 | 100.6 | 99.5 | [ , ] | |
| Pooled over both strata | 10 | 100.7 | 150.2 | 49.4 [34.1, 65.5] |
Table 1: Marginal value of a checkpoint, by what the Skip world falls back to. Each row is a within-cell paired contrast over the 10 tasks whose recovery reached the endpoint verified by the task’s hidden tests. Brackets are 10,000-replicate task-bootstrap 95% intervals; advantage is net of measured creation cost, and conversion is advantage over the nominal work protected.
| Proxy | First checkpoint ( ) | Additional ( ) | Drop largest |
|---|---|---|---|
| Tool calls | ( ) | ||
| Files modified | ( ) | ||
| Bytes changed | ( ) | ||
| Agent importance | ( ) |
Table 2: Spearman correlation between a cheap online proxy and measured advantage, within stratum over verified-endpoint cells; pooling the strata would manufacture a correlation from the split alone. -values are permutation tests over relabelings; the last column drops the largest cell.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 1: Counterfactual labeling. From the previous checkpoint and candidate we run two matched worlds, CP , which materializes , and SKIP , which does not, to the same future failure, then recover both and measure and . Their difference is the label .
| Group | Examples |
|---|---|
| Execution cost | elapsed seconds; tokens since last checkpoint; tool calls; retries; errors; expensive-tool count |
| Physical state | files touched; bytes modified; artifact size; optional process-state size |
| Verified progress | newly passing tests; completed subgoals; verifier events; newly validated artifacts/facts |
| Ripple / graph | new descendant count; dependency fanout; downstream verified nodes; affected artifact count; centrality proxy |
| Context | distance from last checkpoint; time since last progress; branch depth; recent error trend |
Table 3: Proposed online feature families for advantage prediction. Not evaluated in this paper.
Figure 2: Marginal value of a checkpoint by what the Skip world falls back to, over the 10 verified-endpoint tasks, with task-bootstrap 95% intervals. The dashed rule is measured creation cost. These are the first two rows of Table 1 .
| Step | Trajectories | Mean (s) | Share | Cumulative |
|---|---|---|---|---|
| 1 | 12 | 172.1 | 73.9% | 73.9% |
| 2 | 12 | 45.5 | 19.5% | 93.4% |
| 3 | 12 | 13.9 | 6.0% | 99.4% |
| 4 | 3 | 5.4 | 0.6% | 100% |
Table 4: Distribution of nominal (pre-failure) work over logical steps, across all 12 frozen trajectories. A checkpoint at candidate step protects exactly step ’s work: the Skip world restores at and re-executes step before continuing to the same failure. The workload is close to a single expensive step followed by verification.
| Restore point | Branches | Cost (s) | Tokens re-executed |
|---|---|---|---|
| Pristine (start over) | 24 | 226.5 | 640,651 |
| After step 1 | 45 | 108.6 | 162,656 |
| After step 2 | 29 | 103.3 | 154,439 |
| After step 3 | 8 | 95.6 | 118,976 |
Table 5: Mean recovery cost by how much progress the restore point preserved, over all 106 branches. Descriptive only: these are not within-task paired contrasts, and the deepest bucket is drawn from the three four-step tasks alone. The paired contrasts in Table 1 are the controlled version of the same comparison.
| Single-checkpoint policy | Verified (s) | All (s) |
|---|---|---|
| Checkpoint at first safe boundary | 105.9 | 112.1 |
| Checkpoint at second safe boundary | 155.9 | 175.6 |
Table 6: Retrospective single-checkpoint placement on the 9 three-step tasks (7 of which reached the verified endpoint), under a uniform prior over the two measured failure points. Lower is better. An oracle that picks the better boundary per task coincides with the first-boundary row on 9/9 tasks, so it is not shown separately.
Figure 3: Retrospective one-checkpoint budget, per task, on the 9 three-step tasks. Placing the single checkpoint at the first safe boundary is cheaper on every task. Because the second-boundary policy holds no checkpoint at the step-2 failure, this margin restates the first-checkpoint contrast rather than measuring placement.
Figure 4: Per-task, per-stratum checkpoint advantage ( ), by repository and stratum. Hollow markers are the two tasks whose nominal trajectory never reached the endpoint verified by the hidden tests; they are excluded from the primary estimate and shown here for completeness. The legend’s “Early” and “Late” are the first-checkpoint and additional-checkpoint contrasts of Table 1 . Additional-checkpoint points sit near zero in every repository (Django mean s, scikit-learn s, SymPy s), and 5 of the 12 are individually positive.
| Task | Repository | Verified | Pairs | CP (s) | SKIP (s) | Adv. (s) |
|---|---|---|---|---|---|---|
| django-15731 | django/django | yes | 5 | 107.5 | 115.9 | 8.3 |
| django-15814 | django/django | yes | 4 | 92.1 | 181.8 | 89.7 |
| django-16136 | django/django | yes | 5 | 88.0 | 118.1 | 30.1 |
| django-16315 | django/django | yes | 4 | 132.5 | 214.8 | 82.3 |
| sklearn-25747 | scikit-learn | no | 4 | 148.4 | 212.1 | 63.7 |
| sklearn-25931 | scikit-learn | yes | 4 | 102.6 | 146.0 | 43.5 |
Table 7: Per-task means over the task’s two frozen cells. Advantage is the mean of across those cells; positive favors checkpointing. “Verified” indicates whether both recovery branches reached the endpoint verified by the hidden tests (identical to the nominal outcome on this workload). “Pairs” counts repeated CP/SKIP measurements and sums to 53 across the table.
| Repository | Tasks | CP (s) | SKIP (s) | Advantage (s) | Verified |
|---|---|---|---|---|---|
| django/django | 4 | 105.0 | 157.6 | 52.6 | 4/4 |
| scikit-learn/scikit-learn | 4 | 116.8 | 165.4 | 48.5 | 3/4 |
| sympy/sympy | 4 | 94.3 | 177.1 | 82.7 | 3/4 |
Table 8: Per-repository means over all frozen cells, including the two tasks that neither branch solved (4 tasks each). Restricted to verified-endpoint tasks the advantages are 52.6 s, 43.5 s, and 51.2 s respectively; the SymPy figure below is inflated by the unsolved sympy-23950 . Shown directly rather than bootstrapped: three repositories cannot support a repository-level resampling interval.
Figure 5: RippleCP intuition. A large physical state change may have low checkpoint value when it is cheap to reproduce (left), whereas a small semantic decision can have high value when many subsequent actions depend on it (right). RippleCP estimates whether checkpointing the current state avoids enough future recovery cost to justify materializing it.