cs.AIOct 20, 2026
SaveBack Up Before You Regret: Measuring Backup Awareness and Automating Backup in Coding Agents
Organizations: Northeastern University · University of California, Berkeley · University of Chicago · University of Illinois at Urbana-Champaign
Abstract
Coding agents now complete tasks autonomously, modifying real-world state such as artifacts, software environments, and configurations along the way. Yet they show limited backup awareness: recognizing that an action will modify state that may later be needed, and preserving it beforehand. To evaluate backup awareness, we introduce RegretBench, a benchmark of 205 executable tasks grounded in regret moments reported on platforms such as X.com, GitHub, and Stack Overflow, where users explicitly regretted not preserving state they later needed. Across six frontier models, agents complete 86.1% of tasks on average yet back up before these moments in only 15.2%, showing that task completion ability does not imply backup awareness. Because it is hard to predict which changes will be regretted, we propose BackUpper, a harness plug-in backup layer that checks every tool call and preserves affected state before modification. It combines rule-based strategies for common operations with a lightweight subagent for complex cases, and its rule table can expand with subagent decisions over time. On RegretBench, BackUpper covers every regret moment action and restores states with 92.2% average similarity to the original. Achieving this costs only $0.014 per task, with 7x fewer tokens than the main-agent self-backup baseline. When users have preferences on what to back up, BackUpper can also narrow its scope accordingly, matching human annotations on whether an action is worth preserving with an 84.5% F1 score. These results suggest that agents can keep actions reversible at low cost while respecting users' preferences.
Figures & tables
Figure 1: Agents act on real-world state without preserving it, leaving users regretting the loss of state they later need. To evaluate whether agents back up before acting, we construct RegretBench , the first benchmark grounded in 205 real-world regret moments , and find that frontier agents back up in only 15.2% of tasks. To close this gap, we propose BackUpper , a plug-in backup layer that intercepts every tool call and covers every regret moment action, with restored state reaching 92.2% average similarity to the original.
| Harness | Workspace | Native Edit | External State |
| Hermes \citep nousresearch2025hermes | ✓ | ✗ | ✗ |
| OpenClaw \citep openclaw2025openclaw | ✓ ∗ | ✓ ∗ | ✗ |
| OpenHands \citep wang2025openhands | ✗ | ✓ | ✗ |
| Claude Code \citep anthropic2025claudecode | ✓ | ✓ | ✗ |
| Codex \citep openai2025codex | ✓ | ✗ | ✗ |
| Ours: Any Harness + BackUpper | ✓ | ✓ | ✓ |
Table 1: Coverage of native backup mechanisms among five mainstream agent harnesses. Most are scoped to local workspace or native edits; none covers state modified through external tools. ✓ ∗ denotes scheduled backups that run independently of agent actions.
Figure 2: RegretBench construction pipeline. From each documented regret moment , we extract the triggering action and affected state, then embed them into a real-world environment and interaction trajectory. In this example, upgrading ipykernel silently replaces the installed traitlets ; the agent completes the request but leaves no copy of the previous version for later recovery.
Figure 3: Source and instruction diversity in RegretBench.
Figure 4: PR and RMBR (a), and average tool calls per task (b) on RegretBench.
Figure 5: Harness and model generation ablations on a 60-task subset of RegretBench.
Figure 6: BackUpper: a plug-in backup layer. BackUpper intercepts every tool call before execution. Predefined rules trigger command rewriting (Tier 1) or command pairing (Tier 2); otherwise, workspace changes trigger snapshots (Tier 3). Remaining calls pass through an LLM read-only checker, which invokes a backup subagent (Tier 4) when needed. Agents can explicitly request backups, while automatic checks catch overlooked backup needs.
Figure 7: Backup coverage across models (a) and restoration fidelity across tools (b) of BackUpper.
Figure 8: BackUpper’s performance against baseline methods (a) and under personalization (b).
Figure 9: BackUpper’s rule-based coverage under self-improvement (a) and storage usage (b).
Table 2: BackUpper reduces backup-related overhead (a) and preserves task performance (b). In (a), tokens and cost are per task and latency is per regret moment action; is BackUpper minus Skill-Based. In (b), tasks come from Terminal-Bench 4 and DeepSWE; is w/ minus w/o BackUpper.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Keywords | Example query |
| Social & Community | lost, deleted, accidentally, overwritten, no backup, only copy, clean up, git clean, emptied trash, retention, cascade delete, migrate reset, terraform state, version history, folder structure, tags, Claude, Cursor, OpenClaw | site:reddit.com "Claude" "overwrote files" |
| Creation & Productivity | deleted comments, recover, downsampling, aggregated, condense, lost tags, lost detail, lost historical detail, GitHub, Trello, InfluxDB, QuickBooks, Graphite | QuickBooks condense lost detail |
| Records & Media | annotations lost, notes lost, book update, replaced, reimport, certificate cleanup, certbot delete, archive lost, audit history, survey questions changed, questions overwritten, shared template | certificates deleted cleanup site:theregister.com |
Table 3: Search keywords used to collect regret moments by source category.
| Harness | Version | Models |
| Claude Code | v2.1.220 | Claude Opus-5, Claude Sonnet-5 |
| Codex | v0.146.0 | GPT 5.6-Sol, GPT 5.6-Terra |
| Hermes | v0.19.0 | Gemini 3.7-Flash, DeepSeek V4-Pro |
Table 4: Harness versions and the models run in each harness in the main experiment.
| Judged backed up | Judged not | |||
| Evaluation Method | BackUpper | Main Agent | Main Agent | Overall |
| LLM-as-Judge | 100% (200/200) | 100% (150/150) | 100% (350/350) | 100% (700/700) |
| Human-Eval | 94.0% (188/200) | 92.0% (138/150) | 97.4% (341/350) | 95.3% (667/700) |
Table 5: Validation of the backup judge on 700 sampled actions: 350 the judge labeled as backed up and 350 it labeled as not backed up.
Figure 10: Share of runs by task completion and regret moment backup, pooled over six models on all 205 tasks. 72.0% of runs complete the task without backing up the affected state.
Figure 11: Regret Moment Backup Rate broken down by tool category and application scenario. Row labels show the average across models. In both views, differences between models far exceed differences between categories.
Figure 12: Backup routing in BackUpper: tier distribution (a) and read-only checker contribution (b).
Figure 13: BackUpper’s rule self-reflection pipeline. After each subagent backup, the subagent reviews the action, its backup trajectory, and the current rule table, and decides to add a rule, update an existing one, or do nothing. Added or updated rules are then generalized to related action patterns and tools, expanding rule-based coverage for subsequent calls.
| Tool | Typical metadata |
| Terminal | modification time |
| SQL | sequence values |
| DevOps | creationTimestamp , resourceVersion |
| read and important flags | |
| Browser | object IDs, creation time, vote scores |
Table 6: Typical metadata captured in snapshots that documented recovery needs rarely mention.
Figure 14: A main agent’s own backup on an Email task: the message text is saved, but the read and flagged state is never captured, so it cannot be restored.
Figure 15: BackUpper on a Browser task: the subagent captures every deleted comment, but the authorship, timestamps, ids, and scores of other users’ comments cannot be written back.
Figure 16: Backup failure analysis: Skill-Based failure causes (a) and subagent ablation (b).
Figure 17: State similarity by restoration approach, with its main overhead per task: per-step checkpointing (AgentRewind) and Post-hoc Recovery compared with BackUpper.
| BackUpper | Skill-Based | Baseline | ||||||||
| Model | Mode | P | R | F1 | P | R | F1 | P | R | F1 |
| Opus-5 | High-Impact | 84.7 | 95.5 | 89.8 | 69.1 | 78.3 | 73.4 | 78.4 | 33.2 | 46.6 |
| Beyond-Request | 66.0 | 90.1 | 76.2 | 60.0 | 71.2 | 65.1 | 2.3 | 3.8 | 2.9 | |
| Human-Work | 83.0 | 100.0 | 90.7 | 94.9 | 84.9 | 89.7 | 80.7 | 17.7 | 29.0 | |
| Sol | High-Impact | 95.3 | 96.0 | 95.7 | 68.9 | 77.5 | 72.9 | 100.0 | 4.8 | 9.1 |
| Beyond-Request | 71.4 | 83.3 | 76.9 | 63.1 | 70.9 | 66.8 | 0.0 | 0.0 | 0.0 | |
Table 7: Backup-trigger precision, recall, and F1 under each personalized mode. Skill-Based receives the preference through its backup skill in the main agent’s context. Baseline is the original agent.
Table 8: Backup actions per task: (a) tool calls issued by each method on all 205 tasks; (b) subagent invocations under each mode on the 60-task subset. is relative to BackUpper in (a) and to Full mode in (b).
Figure 18: BackUpper on a Terminal trajectory: rule-based and subagent backups across Tiers 2–4.
Figure 19: BackUpper on an Email trajectory: read-only calls pass without backup, and the subagent exports the Temp-works folder before the regret moment deletion.
Figure 20: Prompt template for the task judge. The judge assigns each instruction DONE, PARTIAL, or NOT_DONE from observed output and derives the task-level completion verdict used for PR.
Figure 21: Prompt template for the backup judge. The judge evaluates each out-of-workspace action against four criteria and produces a per-action coverage assessment.
Figure 22: System prompt for backup marker injection. The agent marks each tool call whose side effects reach state outside the auto-captured workspace; a subagent then backs it up beforehand.
Figure 23: System prompt for the backup subagent. It is dispatched before each marked call, captures pre-state and transitive dependents under a budget, and records recovery commands.
Figure 24: System prompt for the backup rule learner. After each subagent attempt, it reflects on what was captured and generalizes the backup strategy into a reusable, parameterized rule or updates an existing one.
Figure 25: System prompt for the backup rule generator. Starting from one observed action, it derives parameterized backup rules for the action itself ( self ), other subcommands of the same tool (axis A), alternative entry points to the same store (B1), and analogous stores (B2).
More in this topic
AI Coding Agents
AI coding agents use language models and software tools to plan, edit, test, and maintain code with varying levels of autonomy.
See all 327 papers