Organizations: Peking University; Key Laboratory of Data Intelligence and Security Beijing, China · Alibaba Group China · Peking University; Key Laboratory of Data Space Technology and System Beijing, China
Contemporary microservice systems continue to grow in scale and complexity, leading to increasingly frequent and costly failures. While recent LLM-based auto-remediation approaches have emerged, they primarily translate textual instructions into executable Ansible playbooks and rely on expert-crafted prompts, lacking runtime knowledge guidance and depending on large-scale general-purpose LLMs, which limits their accuracy and efficiency. We introduce End-to-End Microservice Remediation (E2E-MR), a new task that requires directly generating executable playbooks from diagnosis reports to autonomously restore faulty systems. To enable rigorous evaluation, we build MicroRemed, a benchmark that automates microservice deployment, failure injection, playbook execution, and post-repair verification. We further propose E2E-REME, an end-to-end auto-remediation model trained via experience-simulation reinforcement fine-tuning. Experiments on public and industrial microservice platforms, compared with nine representative LLMs, show that E2E-REME achieves superior accuracy and efficiency.
Figures & tables
Figure 1. Previous microservice remediation workflow compared with the end-to-end microservice remediation pipeline proposed in this paper.
Figure 3. MicroRemed Benchmark Pipeline: the benchmark launches a real microservice; Failure Injection injects faults and produces a Failure Report; the Failure Report together with Auxiliary Context is provided to the Candidate Remediation LLM which generates an Ansible Playbook; the Execution Engine executes the playbook; Status Verification checks remediation success; Evaluation and Recovery restores the system for the next run.
Figure 4. Runtime pipeline of E2E-REME. The model acts as a coordinator within a multi-agent workflow, ThinkRemed , which organizes the remediation process through the coordination of probing, execution, and verification agents.
No.
Category
Failure Types
1
Resource-Level
CPU Saturation
2
Memory Saturation
3
IO Saturation
4
Network-Level
Network Loss
5
Network Delay
6
Application-Level
Pod Failure
Table 1. Benchmark statistics on failure types
Figure 5. Overall framework of Experience-Simulation RFT
LLM Backbone
Train-Ticket
Online-Boutique
Simple-Micro
Accuracy (%)
Latency (s)
Accuracy (%)
Latency (s)
Accuracy (%)
Latency (s)
Easy
Med
Hard
Easy
Med
Hard
Easy
Med
Hard
Easy
Med
Hard
Easy
Med
Hard
Easy
Med
Hard
Closed-Sourced LLMs
Qwen3-Plus
47.83
30.61
31.17
79.83
77.44
81.22
43.48
43.75
31.58
53.35
83.77
59.35
47.83
36.17
38.03
71.26
75.79
78.18
Qwen3-Max
47.83
28.57
30.77
69.87
86.38
12.36
39.13
37.50
25.32
34.8 2
46.49
48.20
30.43
22.92
17.91
37.02
39.73
52.43
Qwen3-Flash
21.74
16.33
13.16
43.24
84.96
290.7
34.78
30.61
21.33
42.33
77.26
98.64
22.72
14.58
8.86
50.52
61.12
65.33
Table 2. Remediation Accuracy (left columns) and Latency (right columns) across closed-source and open-source LLM backbones
Figure 6. Latency–accuracy trade-off of various large language models on the Online-Boutique microservice
Backbone
Train-Ticket
Online-Boutique
Easy
Medium
Hard
Easy
Medium
Hard
Closed-Sourced LLMs
Qwen3-Plus
3363
4988
4359
18822
127299
108053
Qwen3-Max
3583
2145
3256
4821
5366
5758
Qwen3-Flash
2948
3651
3891
3490
3003
3378
Open-Sourced LLMs
Table 3. Average Token Consumption per remediation results
Method
Accuracy (%)
Latency (s)
Easy
Medium
Hard
Easy
Medium
Hard
Qwen3-8B
30.43
20.83
15.56
359.35
319.24
343.13
+ SFT
34.78
24.49
22.50
263.72
250.08
285.43
+ Sim-RFT
86.96
85.71
69.62
36.37
50.47
51.84
+ Real-RFT
95.65
89.80
78.75
39.44
38.31
40.56
Table 4. Ablation study on the Online-Boutique microservice
Method
Train-Ticket
Online-Boutique
Easy
Medium
Hard
Easy
Medium
Hard
ThinkRemed
47.83
30.61
31.17
43.48
43.75
31.58
w/o Probe
43.48
34.69
30.38
39.13
40.43
30.38
w/o Reflection
43.48
28.57
26.92
34.78
36.17
25.32
w/o P. & R.
39.13
33.33
20.51
30.43
35.42
20.51
Table 5. ThinkRemed’s ablation study
Method
Remediation Time Reduction (%)
Microservice A
Microservice B
Microservice C
MAPE-Ansible
67.21
55.83
48.95
WCA-Ansible
73.76
60.90
56.74
E2E-REME( ours )
81.33
79.58
76.03
Table 6. Evaluation on 3 realistic industrial workloads and microservices: Percentage reduction in remediation time
Large language model (LLM)-based program synthesis turns natural-language goals and examples into executable programs, often using tests or formal constraints.
Computer Network Information Center, Chinese Academy of Sciences, Beijing, China · Key Laboratory for Satellite Digitalization Technology, Chinese Academy of Sciences, Shanghai, China · Nankai University, Tianjin, China +2