cs.AISep 26, 2026
SaveSRE-Marathon: A Continuous, Change-Driven Benchmark for Autonomous Site Reliability Agents
Organizations: University of Toronto · The Hong Kong University of Science and Technology
Abstract
Benchmarks for site reliability engineering (SRE) agents are typically episodic: one fault is injected, the agent receives an incident task, and its response is scored. Production operation is not. Incidents surface through noisy alerts, overlap in time, and often originate from code or configuration changes. We present SRE-Marathon, a benchmark for long-horizon, continuous SRE operation. An agent is invoked at a fixed cadence with cumulative alert history and a persistent workspace while operating a live two-zone Kubernetes deployment as a fault orchestrator injects overlapping faults according to a seeded, production-calibrated schedule. Curated code and configuration changes deployed through the same build pipeline available for repair. Each run is recorded into a sealed bundle and scored offline: Marathon-Score credits each injected fault for ordered progress through correlation, localization, and repair, with all metrics computed deterministically from recorded system evidence. Across three applications and about sixty faults per run, the best of 10 methods reaches only 41.3 out of 100. Agents often correlate and localize faults, but almost never complete repairs while the faults remain active.
Figures & tables
Figure 1: One SRE-Marathon run using the ReAct ( Yao et al., 2023 ) agent with Kimi K2.5 on Astronomy Shop. Above: every fault from occurrence to revert. Below: the agent’s actions. The wall-clock axis runs longer than a nominal 90-minute run because the schedule drifts (Appendix A ).
Figure 2: Overview of SRE-Marathon. A fault orchestrator continuously introduces problems into a live microservice system under load. The SRE agent observes their effects through telemetry and alerts and interacts with the system only through instrumented tools.
Figure 3: Example fault specification. The highlighted fields define its category, injection mechanism, target service and affected scope, symptom probe, and responsible artifact.
| Stage metrics | Median time (s) | Degrad. Level (%) | Cost per run | Risk | Marathon-Score (Out of 100) | |||||||
| Method | Backbone | Corr | Loc | Rep | MTTD | MTTL | MTTR | Tokens | $ | |||
| Runbook | — | 0.658 | 0.364 | 0.024 | 167 | 267 | 560 | 111.0 | — | — | 0.028 | 33.5 |
| ReAct | kimi-k2.5 | 0.694 | 0.079 | 0.006 | 189 | 283 | 709 | 86.9 | 6.5M | 4.02 | 0.217 | 25.4 |
| ( Yao et al., 2023 ) | gpt-5.6-luna | 0.419 | 0.125 | 0.000 | 163 | 293 | — | 100.8 | 6.3M | 1.45 | 0.349 | 16.3 |
| RAG | kimi-k2.5 | 0.426 | 0.398 | 0.000 | 200 | 68 | — | 150.5 | 569k | 0.46 | 0.015 | 23.5 |
| ( Lewis et al., 2020 ) | gpt-5.6-luna | 0.199 | 0.105 | 0.000 | 159 | 65 | — | 93.5 | 730k | 0.23 | 0.000 | 8.2 |
Table 2: SRE-Marathon leaderboard: each row averages a method’s marathon runs on three microservice applications; the best value per column is in bold.
Figure 4: Agent actions over time during a marathon run: tool calls per invocation, by tier, in ten-minute bins of the schedule, for every method of Table 2 on Kimi K2.5 and for Runbook.
Figure 5: Stage-wise success over time in fault handling. Each panel shows the % of faults injected so far that have been successfully correlated, located, or repaired by each method on Kimi K2.5.
Figure 6: Two marathon runs containing the same fault type and correct remedy. The shaded focal window marks the lifetime of the fault discussed in each case. From top to bottom, the lanes show active faults (grey) and those with a failing symptom probe (black), RCA (Root Cause Analysis) reports, executed repairs, and user-facing service health.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Category and defect subtype | Astro | OB | TT | Total | Example |
| Code bug | |||||
| Feature or logic error | 48 | 6 | 6 | 60 | A new currency lookup omits a supported currency, so requests in that currency receive an incorrect default value. |
| Incorrect constant or flag | 5 | 6 | 6 | 17 | A new length limit on cart identifiers is set too low, so the cart rejects identifiers it previously accepted. |
| Component incompatibility | 23 | 6 | 6 | 35 | New code reads database rows by column name while the adapter still returns values by position, so the queries fail. |
| Type, validation, or exception error | 29 | 6 | 6 | 41 | A new price check reads only the whole-number part of a price, so valid prices below one dollar are rejected. |
| Backward incompatibility | 45 | 8 | 8 | 61 | An update changes the request fields a service expects without accepting the older format, so callers still sending it are rejected. |
Table 3: Catalog faults by category, application, and defect subtype, with an example of each.
| Rule | Fires when | Severity | Holds for | Applications |
| HighRequestErrorRate | 5xx share of entry-point requests above 10% | Critical | 1m | All |
| EndpointErrorRate | 5xx share of one entry-point route above 50% | Critical | 1m | All |
| HighRequestLatency | 95th-percentile latency of a service above 3000 ms | Critical | 1m | All |
| PodStatusError | A container is waiting with a reported reason | Critical | 1m | All |
| FailedPodsDetected | A pod is in phase Failed | Critical | 1m | All |
| DeploymentNotReady | A deployment with replicas requested has none available | Critical | 1m | All |
Table 4: The rule templates of the frozen pack. The condition paraphrases the PromQL expression, and the error-share rules also require a minimum request rate.
| Tool | Tier | Function |
|---|---|---|
| code_search | Read | Search the source of one service. |
| get_alerts | Read | Alerts fired in full cumulative alert history, with their labels. |
| get_topology | Read | The service call graph derived from traces. |
| kubectl_describe | Read | Describe one Kubernetes object with its recent events. |
| kubectl_get | Read | Read the current state of one Kubernetes object. |
| query_logs | Read | Recent log lines of one deployment. |
Table 5: The published tools by tier. The policy enforcement layer computes every mutating call’s target set from its typed arguments, and every guarded call must name a reported incident.
| Alert rule | Action | Reported category |
|---|---|---|
| DeploymentNotReady | scale_deployment to 2 | deployment |
| DeploymentReplicasUnavailable | scale_deployment to 2 | deployment |
| DeploymentGenerationMismatch | rollback | deployment |
| ContainerOOMKilled | restart_pod | code-bug |
| PodStatusError | restart_pod | code-bug |
| PodNotReady | restart_pod | code-bug |
Table 6: Runbook’s rule table in priority order: when an incident’s alerts fire several rules, the first listed wins. The Online Boutique and Train Ticket rules carry an application prefix and take the row of their unprefixed name.
Figure 7: One invocation of Orbit . Alert merging and both publication steps are deterministic, and the model decides only inside the loop. Everything kept across invocations lives in the workspace.
| Seed | Venue | Corr | Loc | Rep | MTTD | MTTL | MTTR | Risk | Marathon-Score |
|---|---|---|---|---|---|---|---|---|---|
| Shared | Single-host | 0.517 | 0.121 | 0.017 | 231 | 329 | 709 | 0.136 | 20.1 |
| 17 | Single-host | 0.203 | 0.034 | 0.051 | 202 | 625 | 593 | 0.593 | 7.3 |
| 27 | Multi-node | 0.455 | 0.073 | 0.018 | 208 | 324 | 2486 | 0.250 | 17.0 |
Table 7: ReAct on Kimi K2.5, one Astronomy Shop marathon run per seed. Columns are as in Table 2 .
| Regime | Runs | Corr | Loc | Rep | Marathon-Score |
|---|---|---|---|---|---|
| Short run, 5 faults | 6 | 0.800 | 0.200 | 0.500 | 38.9 |
| Short run, 10 faults | 6 | 0.728 | 0.283 | 0.183 | 34.8 |
| Marathon run, about 60 faults | 3 | 0.694 | 0.079 | 0.006 | 25.4 |
Table 8: ReAct on Kimi K2.5 by schedule regime. A short-run row averages six short runs, two per application, and the marathon-run row averages the three marathon runs of Table 2 . Columns are as in Table 2 .
Figure 8: Every method of Table 2 on Kimi K2.5, and Runbook. Top: executed actions per invocation in the ten-minute schedule bins of Figure 4 ; unlike its act tier, calls that were denied or failed are not counted. Bottom: every executed action of a method’s marathon runs by what it touched, with the total at the bar’s end. An action is on a live fault when its target set holds the faulted service of a fault injected and not yet reverted. Matching is by service, so it says where an action went and not whether the remedy fit.
Figure 9: One outcome per injected fault, by the first rule that matches: handled, meaning all three stages passed; repaired but not handled; acted on, meaning an executed action targeted its faulted service while it was live, but not repaired; located but not acted on; correlated only; or missed. Rows are the LLM-based methods of Table 2 on both backbones, plus the two ReAct short-run rows.
More in this topic
Long-Horizon Agent Evaluation
Long-Horizon Agent Evaluation measures whether AI agents can plan, act, remember, and stay reliable across extended tasks and changing environments.
See all 359 papers