Large language model (LLM) agents offer a promising data-driven approach to automating Site Reliability Engineering (SRE), yet their enterprise deployment is constrained by three challenges: restricted access to proprietary data, unsafe action execution under permission-governed environments, and the inability of closed systems to improve from failures. We present AOI (Autonomous Operations Intelligence), a trainable multi-agent framework formulating automated operations as a structured trajectory learning problem under security constraints. Our approach integrates three key components. First, a trainable diagnostic system applies Group Relative Policy Optimization (GRPO) to distill expert-level knowledge into locally deployed open-source models, enabling preference-based learning without exposing sensitive data. Second, a read-write separated execution architecture decomposes operational trajectories into observation, reasoning, and action phases, allowing safe learning while preventing unauthorized state mutation. Third, a Failure Trajectory Closed-Loop Evolver mines unsuccessful trajectories and converts them into corrective supervision signals, enabling continual data augmentation. Evaluated on the AIOpsLab benchmark, our contributions yield cumulative gains. (1) The AOI runtime alone achieves 66.3% best@5 success on all 86 tasks, outperforming the prior state-of-the-art (41.9%) by 24.4 points. (2) Adding Observer GRPO training, a locally deployed 14B model reaches 42.9% avg@1 on 63 held-out tasks with unseen fault types, surpassing Claude Sonnet 4.5. (3) The Evolver converts 37 failed trajectories into diagnostic guidance, improving end-to-end avg@5 by 4.8 points while reducing variance by 35%.
Figures & tables
Figure 1: AOI System Overview. Left: Closed-Loop Evolution Pipeline —a Judge classifies SRE troubleshooting workflows by outcome. Failed workflows are repaired by the Evolver into corrected command sequences that serve as diagnostic guidance at inference time. Successful workflows are distilled by the Purifier into optimal diagnostic paths that serve as training data. Right: Multi-Agent Runtime —the Observer coordinates read-only diagnosis (Probe) and write-gated remediation (Executor). The Observer is trained via GRPO, and at inference time receives the Evolver’s corrected plans as structured prompts.
Figure 2: AOI Runtime Agent Architecture. The Observer coordinates diagnosis through the Probe (read-only) and Executor (write-gated) agents. The Compressor maintains context efficiency via dual-timescale memory.
Agent
Responsibility
Permission
Mraw
Mtask
Mcomp
Observer ( O )
Planning, hypothesis tracking
Read compressed
-
R/W
R
Probe ( P )
Diagnostic exploration
Read-only exec
W
R
-
Executor ( X )
Remediation actions
Write (gated)
W
R
-
Compressor ( C )
Context distillation
Read raw, write comp.
R
-
W
Table 1: Agent roles and permissions in AOI. Right columns show the access control matrix over three memory stores (R = read, W = write, - = no access): Mraw stores raw environment outputs (e.g., full kubectl responses); Mtask maintains the diagnostic task queue and hypothesis state; Mcomp holds compressed context produced by the Compressor, serving as the Observer’s sole information source.
Model
Method
Detection
Localization
RCA
Mitigation
Overall
GPT-4o-mini
AOL-agent
25.0
9.5
7.7
7.7
14.7
GPT-4o-mini
STRATUS
78.1
25.0
15.4
23.1
43.0
GPT-4o-mini
AOI
90.6
32.1
38.5
53.8
58.1
Claude Sonnet 4.5
AOL-agent
68.8
53.6
15.4
76.9
57.0
Qwen3-14B (best@5/avg@5)
STRATUS
75.0/41.3
32.1/11.4
7.7/4.6
15.4/15.4
41.9/22.1
Qwen3-14B (best@5/avg@5)
AOI
100/66.9
53.6/27.9
30.8/7.7
46.2/23.1
66.3/38.6
Table 2: AOI runtime comparison on full benchmark (86 tasks, %). GPT-4o-mini OpenAI (2023) , Claude Sonnet 4.5 uses single-run; Qwen3-14B uses 5-round sampling (best@5/avg@5).
Table 4: Component ablation on Devolvertest (37 tasks, best / avg %). First three rows use 5 runs; last row uses 4 runs.
Figure 7
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Trajectory-Corrective Evolver Architecture. The Evolver observes failed command sequences from the Observer’s execution history, generates G candidate corrections via GRPO sampling (sampling from a policy trained on successful trajectories), and provides the highest-scoring correction as a structured prompt to guide the Observer’s next attempt. Key insight: This closed-loop mechanism converts failed diagnostic attempts into learning opportunities without requiring manual expert intervention.
Parameter
Value
AOI Runtime
Max iterations
15
Max Probe rounds/iteration
5
Context budget/iteration
4096 tokens
Long-term memory capacity
10 summaries
Executor whitelist
47 command patterns
Appendix
Table 5: Complete hyperparameter settings.
Task Type
Count
Detection
10
Localization
13
Mitigation
3
RCA
11
Total
37
Appendix
Table 10
Figure 6: Nested data partition structure. Dall (86 tasks) contains Devolvertrain (49 successful trajectories from Claude Sonnet 4.5), which in turn contains Dobstrain (23 tasks covering 11 fault types). The strict subset relationship ensures that no test data appears in any training set. Visual interpretation: The outer circle represents all benchmark tasks; the middle ring shows Evolver training data (all successful Sonnet trajectories); the innermost core shows Observer training data (carefully selected to cover diverse fault types while maintaining strict hold-out). This design enables fair evaluation of the combined Observer+Evolver system.
Fault Type
GT Tasks
Total Tasks
k8s_target_port-misconfig
9
12
scale_pod_zero_social_net
3
4
misconfig_app_hotel_res
3
4
astronomy_shop_cart_service_failure
1
2
astronomy_shop_payment_service_failure
1
2
astronomy_shop_product_catalog_failure
1
2
Appendix
Table 6: Training fault types (11 types → 23 successful trajectories). These fault types were selected to provide diverse coverage of common failure modes while ensuring sufficient test data remains for evaluation.
Fault Type
Tasks
revoke_auth_mongodb
8
user_unregistered_mongodb
8
assign_to_non_existent_node_social_net
4
wrong_bin_usage
4
astronomy_shop_ad_service_high_cpu
2
astronomy_shop_ad_service_manual_gc
2
Appendix
Table 7: Test fault types (15 types → 48 tasks) plus 15 tasks from training fault types where Sonnet failed, totaling 63 held-out tasks. Critical: All test fault types are completely unseen during Observer training, ensuring we evaluate true generalization capability.
Figure 7: best@ k success rate vs number of runs. The curve shows clear diminishing returns: the first retry ( k =1 → 2) provides +19.8pp improvement, while subsequent retries ( k =2 → 5) each provide only +2-3pp. Interpretation: The rapid initial improvement suggests many tasks have “near-miss” failure modes that succeed on retry, while the flattening curve indicates some tasks are fundamentally beyond current capability. The inflection point at k =2-3 provides clear guidance for resource-constrained deployments.
Round
Det.
Loc.
RCA
Mit.
Overall
R1
53.1
21.4
7.7
23.1
31.4
R2
75.0
32.1
0.0
23.1
41.9
R3
59.4
32.1
15.4
23.1
38.4
R4
75.0
21.4
15.4
23.1
40.7
R5
71.9
32.1
0.0
23.1
40.7
Avg
66.9
27.9
7.7
23.1
38.6
Appendix
Table 8: Per-round success rates (%) by task type.
Figure 8: Per-round success rate variance by task type. Detection exhibits the widest range (22pp), indicating high stochastic sensitivity—tasks that fail in one run often succeed in another due to different exploration paths. Mitigation’s flat line confirms that current failures are deterministic capability gaps rather than sampling artifacts. Actionable insight: Detection tasks benefit most from multi-round sampling, while Mitigation requires architectural improvements rather than more attempts.
Success
Category
Count
%
5/5
Consistently solved
14
16.3
3–4/5
Mostly solved
16
18.6
1–2/5
Occasionally solved
27
31.4
0/5
Never solved
29
33.7
Appendix
Table 9: Task stability distribution (86 tasks total).
Figure 9: Task stability distribution across 86 benchmark tasks. The large middle segment (50% of tasks with 1-4/5 success) validates our multi-round sampling strategy—these tasks are neither trivially easy nor impossibly hard, but rather exhibit outcome variance across runs. Design implication: Future work should focus on converting the 0/5 category (capability gaps) into the 1-2/5 category (stochastic successes), which can then be reliably solved via multi-round sampling.
Figure 10: Observer GRPO training effect by task type (best@5). Detection improves dramatically (+25.5pp) as the Observer learns systematic anomaly-detection patterns. Localization decreases ( − 4.5pp) due to over-exploration causing multi-anomaly confusion (detailed in Section E.4 ). RCA improves (+8.4pp) from better evidence-gathering strategies. Critical insight: GRPO training is not uniformly beneficial—it helps exploration-heavy tasks but can harm precision-critical tasks where thoroughness conflicts with efficiency.
Figure 11: Task-level changes after GRPO training. Net +9 tasks improved (11 gains, 2 losses). Both degraded tasks are Localization, confirming that GRPO’s deep-exploration strategy harms precision-critical fault localization. Interpretation: The asymmetric gains (strong Detection improvements, Localization degradations) suggest task-type-aware training as future work.
Task Type
Training Tasks
Test Tasks
Detection
10 (43.5%)
22
Localization
6 (26.1%)
22
Analysis (RCA)
1 (4.3%)
12
Mitigation
6 (26.1%)
7
Total
23
63
Appendix
Table 10: Training set composition by task type. The heavy Detection bias (43.5%) explains why Detection benefits most from GRPO.
Figure 12: Observer-GRPO vs Base performance by task type (best@5). Detection improves by +36.4pp (from 59.1% to 95.5%), while Localization degrades by − 18.2pp (50.0% to 31.8%). Key takeaway: GRPO benefits evidence-gathering tasks (Detection, Analysis) but harms precision tasks (Localization) where over-exploration introduces confusion. This heterogeneity motivates task-type-aware exploration policies.
Task Type
Obs-GRPO
Base
Diff.
Detection
11.0
1.9
+9.1
Localization
11.2
1.8
+9.4
Analysis
9.6
1.1
+8.5
Mitigation
12.0
2.8
+9.2
Average
10.9
1.9
+9.0
Appendix
Table 11: Average exploration steps by task type (lower = more efficient).
Figure 13: Exploration depth vs success rate trade-off. Each point represents a task type; arrows show the transition from Base to Observer-GRPO. GRPO uniformly increases exploration depth (arrows point right), but success direction varies: upward for Detection/Analysis (more exploration helps), downward for Localization (more exploration harms). Critical implication: A single exploration strategy cannot optimize all task types simultaneously—future work should adapt exploration depth based on detected task category.
Figure 14: Task outcome distribution for Localization vs Analysis. Localization shows Base advantage (6 exclusive wins vs 2), while Analysis shows Obs-GRPO advantage (2 vs 0). Interpretation: Confirms that deep exploration helps Analysis (which benefits from comprehensive evidence) but harms Localization (where first instinct is often correct). The asymmetry validates task-type-specific exploration strategies.
Task Type
Optimal Strategy
Why
Detection
Shallow OK
Any anomaly suffices
Localization
Shallow better
Avoid multi-anomaly confusion
Analysis
Deep better
Need comprehensive evidence
Mitigation
Deep required
Must understand before fixing
Appendix
Table 14: Exploration depth trade-off by task type.
Figure 15: Exploration strategy effectiveness heatmap. Green = strategy helps; Red = strategy harms. Localization uniquely benefits from shallow exploration, while Analysis requires deep exploration. This heterogeneity strongly motivates task-type-aware policies as future work.
Problem ID
R1
R2
R3
R4
R5
Best
assign_to_non_existent_node-det-1
T
T
T
T
T
5/5
astronomy_shop_ad_service_failure-det-1
F
T
F
T
F
2/5
astronomy_shop_ad_service_high_cpu-det-1
T
T
F
T
T
4/5
astronomy_shop_ad_service_manual_gc-det-1
F
T
T
T
F
3/5
astronomy_shop_cart_service_failure-det-1
F
T
F
T
F
2/5
astronomy_shop_image_slow_load-det-1
F
T
T
T
T
4/5
Appendix
Table 15: Detailed results: Detection tasks (32 total). High variance across runs indicates stochastic sensitivity to exploration paths.
Cloud-native systems have made operational work both more powerful and harder to automate: incidents unfold across microservices, logs and metrics arrive faster than operators can inspect them, and recovery actions must be coordinated without losing the causal context that makes them safe. We present AOI (AI-Oriented Operations), a context-aware multi-agent framework for autonomous IT operations. AOI separates operational responsibility across an Observer, a read-only Probe, and a guarded Executor, and connects them through dynamic scheduling and a hierarchical memory system with LLM-based context compression. This design turns long-running incident response into an iterative loop of observation, evidence gathering, safe intervention, and memory update. Across AIOpsLab simulations and real-world Loghub-derived scenarios, AOI improves task success to 94.2%, reduces mean time to resolution by 34.4% relative to the strongest baseline, and compresses operational context by 72.4% while preserving 92.8% of diagnostic information. Ablations show that these gains come from the combination of agent specialization, adaptive scheduling, and memory-aware compression rather than from any single module alone. The results suggest that autonomous operations systems can move beyond alert classification toward reliable, context-preserving recovery in complex infrastructure.
Zishan Bai, Hanxuan Chen, Jiayi Gu +9
Columbia University · Hunan University · Central University of Finance and Economics +8
LLM-based agents are reshaping microservice operations into AgentOps, where benchmarks are key to evaluating failure diagnosis over multimodal observability data. However, existing benchmarks remain largely outcome-oriented: they score only the final answer and fail to assess the systematic reasoning process in failure diagnosis. We address this gap by introducing two large-scale datasets (AIOps2025 and RCA100) under a reasoning-process evaluation paradigm that assesses agentic diagnostic capability along three dimensions: Localization (where the fault occurs), Identification (what type of fault it is), and Reason (whether the reasoning trace is grounded in relevant evidence). Together, the two datasets comprise over 500 expert-labeled failure cases across two representative microservice systems (HipsterShop and the OpenTelemetry Demo Store). They cover diverse fault scenarios across resource, network, runtime, middleware/database, and application-logic categories and provide fine-grained causal evidence to support agent learning and reasoning-process evaluation. Beyond scale and coverage, the datasets have been carefully labelled by domain experts and validated through large-scale competitions, supporting more than 6,000 participating teams. This makes them not only expert-labeled diagnostic datasets, but also competition-validated benchmarks for evaluating agentic failure diagnosis in real-world microservice environments. Datasets are available at https://www.aiops.cn/gitlab/aiops-live-benchmark/agenticopseval.
Yuanhong Cai, Xiaohui Nie, Kanglin Yin +8
Computer Network Information Center, Chinese Academy of Sciences, Beijing, China · Key Laboratory for Satellite Digitalization Technology, Chinese Academy of Sciences, Shanghai, China · Nankai University, Tianjin, China +2
AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy tool outputs. We address this gap by manually annotating failed agent runs and release a novel benchmark of 170 trajectories across 11 diverse task settings, including structured API workflows, incident management, and open-ended web/file tasks. Each trajectory is annotated with a critical failure step and a category from a grounded-theory derived, cross-domain failure taxonomy. To mitigate the human cost of failure attribution, we present AgentRx, an automated diagnostic framework that pinpoints the critical failure step in a failed agent trajectory. It synthesizes constraints, evaluates them step-by-step, and produces an auditable validation log of constraint violations with associated evidence; an LLM-based judge uses this log to localize the critical step and category. AgentRx improves step localization by 75% on average over prior work, while providing failure category attribution.