SLS-Bench: A Benchmark for Incident Log Summarization with Synthetic Observability Data

Valerii Iakovlev; Markus Spanring; Martin Flechl

cs.AIOct 12, 2026

SLS-Bench: A Benchmark for Incident Log Summarization with Synthetic Observability Data

Authors: Valerii Iakovlev, Markus Spanring, Martin Flechl

Organizations: Dynatrace Research

Abstract

As IT systems grow in complexity, automated agent-driven incident response becomes increasingly critical. A key capability for these agents is summarizing large log streams into human-readable reports. Further progress requires meaningfully evaluating this function, yet existing benchmarks leave end-to-end summarization from large log streams underexplored due to the prohibitive cost and complexity of collecting real-world data. We introduce SLS-Bench, a benchmark for incident log summarization featuring 90 tasks grounded in real-world incident reports, each with a synthetic log stream and a reference summary. To construct SLS-Bench, we developed a data generation pipeline that converts incident reports into structured incident models, which guide the generation of synthetic logs and reference summaries. This pipeline combines narrative grounding, explicit incident modeling, generator-verifier loops, and statistical comparisons with real-world log streams to produce realistic and controllable observability data. Our evaluation of 14 LLMs on SLS-Bench shows that while frontier models can produce useful summaries, they do so at substantial cost and latency, and open-weight models lag significantly. SLS-Bench provides a framework for systematic evaluation of log analysis agents and a scalable blueprint for future synthetic observability benchmarks.

Figures & tables

Panel of Figure 1 — Figure 1: SLS-Bench task format, evaluation protocol, and cost-quality trade-off. a) Each benchmark problem combines a question, a large log file, and a reference answer. The question is fixed across problems, while the logs and answers are problem-specific. b) The agent reads the question and logs, and writes its own incident summary that is scored against the reference by an LLM judge. c) Benchmark scores versus cost per problem. Filled points define the Pareto frontier, with an approximately logarithmic trend with respect to cost.

Panel of Figure 2 — Figure 2: Problem generation pipeline. a) Each problem is seeded with a unique incident report. The report is converted into an incident model, which is then used to generate both a reference answer and an incident simulator; running the simulator produces the log file. The question template is fixed and shared across problems. Together, the question, logs, and reference answer constitute one benchmark problem. Solid outlines mark objects produced by the generator–verifier loop; double outlines mark deterministic outputs. b) The generator–verifier loop alternates between proposal and verification: the generator proposes an object, the verifier checks it, and any failed check returns feedback for revision. An object is accepted only after passing verification in $K$ consecutive rounds.

Panel of Figure 3 — Figure 3: Incident model. a) System structure : a static graph of log-emitting components, with example log templates and edges indicating interactions. b) Log emission mechanisms : logs are emitted as bursts, as background activity, or as ordered paths corresponding to requests traversing multiple components. a) Phased timeline : emission behavior changes across normal, trigger, and progression phases, allowing log rates, error activity, and path latencies to evolve over time.

Panel of Figure 4 — Figure 4: Increasing the verifier threshold $K$ improves the pass rate.

Figure 5: Comparing Loghub-2.0 and SLS-Bench on representative statistics.

Panel of Figure 6 — Figure 6: Top: Judge prediction vs. ground-truth on calibration set. Bottom: Judge agreement across evaluated LLMs.

Model	Score ( $\uparrow$ )	Cost [ $] ($ \downarrow$ )	Time [Min] ( $\downarrow$ )	Score/$	Score/Min
OpenAI
GPT-5.4 (high)	$0.73\pm 0.01$	$2.19$	$9.5\pm 1.6$	$0.33$	$0.08$
GPT-5.4 (medium) ⋆	$0.73\pm 0.01$	$1.56$	$6.9\pm 1.0$	$0.47$	$0.10$
GPT-5.4 (low) ⋆	$0.67\pm 0.01$	$0.85$	$3.4\pm 0.5$	$0.79$	$0.19$
GPT-5.4 Mini (high) ⋆	$0.61\pm 0.01$	$0.37$	$5.2\pm 1.3$	$1.65$	$0.11$
GPT-5.4 Mini (medium) ⋆	$0.57\pm 0.01$	$0.21$	$3.0\pm 0.7$	$2.71$	$0.18$

Table 1: Benchmark scores with cost and time per problem. Score is mean

\pm

SEM, time is mean

\pm

SD; cost and time are averages per problem. Open-weight model parameters are total/active for mixture-of-expert models. A

\star

denotes Pareto-optimal models under the cost–score trade-off (Fig. 13 ). We use the sample variance formula to compute standard deviation and standard error of the mean.

Figure 7: Overall score split into its constituent axes as evaluated by the LLM judge.

Panel of Figure 8 — Figure 8: Top: Precomputable Python tool outputs. Bottom: After initial orientation, sequential tool calls can be parallelized into independent streams.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Incident report statistics. By type (top) and organization (bottom).

Panel of Figure 10 — Figure 10: Compact system topology for incident model 0b9e6bdf , annotated by architectural plane. The dashboard data plane comprises the user and the middle-row components. The Agent API operates as an independent data-plane workload, sharing only the PostgreSQL database with the primary dashboard path. Auxiliary systems are positioned off the main request path, with external monitoring located in the observation plane and the Auto Scaling Group in the control plane.

Panel of Figure 11 — Figure 11: Compact timeline of incident model 0b9e6bdf . The figure illustrates the normal phase and all five ordered events of the 50-minute scenario. Thick lines highlight the dominant incident trajectory, thin lines represent secondary background activities, and stars denote one-shot log bursts at events 1, 3, and 5. Log and flow IDs are abbreviated for brevity.

Quantity	Mean	SD	Median	Range
Components	6.11	1.45	6.0	3–10
Hosts	16.38	23.66	12.5	7–211
Component edges	7.91	2.90	7.0	3–16
Log templates	26.96	7.85	27.0	12–46
Variable slots	81.84	25.28	78.0	42–180
Flow variants	8.34	3.44	8.0	3–18

Appendix

Table 2: Structural statistics of the 90 accepted incident models.

Figure 12: Distributional comparison between Loghub-2.0 and SLS-Bench . Panels show message length, unique templates per file, variables per template, variable cardinality, normalized template rank–frequency curves, and adjacent-template predictability.

Axis	Definition
Structure	Compliance with the required report form: headings, subsections, certainty lines, and one synopsis bullet per reference phase. Semantic correctness is scored on the other axes.
System reconstruction	Recovery of the same main system purpose, actors, and interactions. Extra plausible names or implementation details are allowed unless they change the main system, recenter a dependent component, or reverse a key dependency.
Timeline structure	Agreement with the reference phase count, order, and time ranges. Small boundary drift and reference-preserving phase splits or merges are minor unless they hide an important state transition.
Timeline progression	Recovery of the same baseline, trigger, major phase-to-phase changes, and final observable state. Missing or contradicting a major progression step is material.
Diagnosis	Recovery of the same core fault chain. A more specific restatement is acceptable when it stays within the reference diagnosis and does not turn an unresolved mechanism into a required causal step.
Support alignment	Whether the cited observations and reasoning support the same qualitative conclusions as the reference. This axis handles extra exactness, magnitude claims, local sequencing, and other log-surface details.

Appendix

Table 3: Evaluation axes used by the LLM judge.

Verdict	Definition
Strong match	A faithful substitute for the reference: the answer is semantically and structurally equivalent to the reference and covers all material content.
Good match	A reliable recovery of the reference, but not a full substitute. The main incident story is the same, and any remaining semantic issues are minor and non-material.
Partial match	A useful answer that gets the main causal story broadly right, but a material bounded defect or an accumulation of smaller semantic defects prevents using it as a reliable substitute.
Weak match	The reference is still recognizable, but major defects dominate. The answer preserves fragments, observations, or broad phases, but does not cleanly recover the incident story.
Mismatch	The prediction tells a materially different incident story and is not usable as a recovery of the reference. This includes switching to a different primary system, trigger, or fault chain.

Appendix

Table 4: Overall verdicts used by the LLM judge.

			Tokens				Cost [$]
Stage	Category	Calls	Input	Cached	Output	Reasoning	Input	Cached	Output	Total
Incident model	Generator	643	11,458,543	3,187,968	8,254,739	2,754,053	14.47	0.57	115.57	130.61
	Verifier	2,730	44,687,360	18,951,040	14,351,580	13,511,415	45.04	3.41	200.92	249.37
	Subtotal	3,373	56,145,903	22,139,008	22,606,319	16,265,468	59.51	3.99	316.49	379.99
Incident simulator	Generator	373	9,188,128	2,078,464	6,793,850	1,837,210	12.44	0.37	95.11	107.93
	Verifier	1,855	47,917,225	19,464,960	12,079,385	11,704,445	49.79	3.50	169.11	222.41

Appendix

Table 5: Token usage and cost breakdown across the three stages of the data generation pipeline. Each stage reports the generator and the verifier usage and cost. Cached tokens are a subset of the input tokens, and reasoning tokens are a subset of the output tokens.

Model	Effort	Budget 5	Budget 10	Budget 25	Budget 50
Nemotron 3 Nano	N/A	0.02	0.04	0.06	0.07
DeepSeek V3.2	N/A	0.09	0.17	0.20	0.29
GPT-5.4 Nano	medium	0.19	0.31	0.38	0.40
Haiku 4.5	N/A	0.32	0.40	0.41	0.43
Opus 4.6	low	0.44	0.45	0.50	0.50
GPT-5.4	low	0.56	0.62	0.63	0.65

Appendix

Table 6: SLS-Bench scores by tool-call budget. Values are means over the 30-problem sample.

Model	Effort	Budget 0	Budget 5	Budget 10	Budget 25	Budget 50
Nemotron 3 Nano	N/A	–	4.23	6.57	12.73	19.53
DeepSeek V3.2	N/A	–	5.00	9.59	20.87	37.10
GPT-5.4 Nano	medium	–	5.00	10.00	24.93	36.83
Haiku 4.5	N/A	–	4.07	7.30	13.70	15.50
Opus 4.6	low	–	4.23	6.93	10.40	10.97
GPT-5.4	low	–	5.00	9.73	20.00	22.37

Appendix

Table 7: Tool calls used by budget. Mean number of Python interpreter calls per problem.

Model	Effort	Budget 0	Budget 5	Budget 10	Budget 25	Budget 50
Nemotron 3 Nano	N/A	–	957 ( $1.00\times$ )	1020 ( $1.07\times$ )	852 ( $0.89\times$ )	845 ( $0.88\times$ )
DeepSeek V3.2	N/A	–	1270 ( $1.00\times$ )	1066 ( $0.84\times$ )	728 ( $0.57\times$ )	634 ( $0.50\times$ )
GPT-5.4 Nano	medium	–	1152 ( $1.00\times$ )	1096 ( $0.95\times$ )	1160 ( $1.01\times$ )	1054 ( $0.91\times$ )
Haiku 4.5	N/A	–	3743 ( $1.00\times$ )	3669 ( $0.98\times$ )	3104 ( $0.83\times$ )	3257 ( $0.87\times$ )
Opus 4.6	low	–	3887 ( $1.00\times$ )	2688 ( $0.69\times$ )	2333 ( $0.60\times$ )	2335 ( $0.60\times$ )
GPT-5.4	low	–	6629 ( $1.00\times$ )	5577 ( $0.84\times$ )	4126 ( $0.62\times$ )	3904 ( $0.59\times$ )

Appendix

Table 8: Tool-output size by budget. Mean characters per tool output; parenthesized multipliers are relative to the five-call budget.

Trajectory	Problem ID	Tool outputs	Precomputable	Prefix	Prefix fraction
T1	03a9b743	39	7	10	25.6%
T2	1cddedc3	36	10	10	27.8%
T3	0ff91f6a	35	7	9	25.7%
T4	0a7c69ff	42	8	11	26.2%
T5	2d21ffed	32	7	11	34.4%
T6	0b9e6bdf	39	7	11	28.2%

Appendix

Table 9: Trajectory-level efficiency counts. Tool outputs are Python interpreter outputs in the raw trajectory. Precomputable outputs are generic log artifacts available before agent reasoning. Prefix outputs are the initial orientation-and-planning calls before the main evidence-collection fan-out.

Stage / stream	Calls	Code functionality	Concrete role in summary construction
Orientation and planning	13	Load and parse the dataframe; compute schema, level/service/template summaries, coarse per-minute views, and short anomaly windows; build reusable service slices and phase anchors.	Establishes the shared state used by every stream: rtm-3 websocket failures and a SIGSEGV crash near 12:20 ; later session-validation/history failures tied to DB pressure near 12:27 ; and mitigation/recovery near 12:34 .
Failure-chain traces	8	Select anomalous request families—failed websocket upgrades, rejected upgrades, and history warnings—then read all rows with the same trace_id .	Produces request-level chains: failed upgrades target rtm-3 ; rejected upgrades pass through session validate timeout with db_err=pool_exhausted ; history warnings line up with app-api timeouts and gateway 503/504 responses.
Service timelines	3	Group service-specific slices by minute or phase using parsed fields such as rtm_target , HTTP status, DB counters, and monitor target.	Shows component evolution: in [12:20,12:27) , rtm-3 has 420 failed upgrades and 0 successful upgrades while other RTM targets continue serving; later timelines expose DB/app-api degradation.
Before/after traces	4	Sample representative healthy traces before the incident and recovery traces after mitigation, then read the complete trace rows.	Gives contrastive examples: normal history, websocket, and message-ingest paths before the crash, plus a successful rtm-3 websocket handshake after the 12:34 mitigation boundary.
Per-phase statistics	5	Loop over phase windows and count levels, templates, HTTP statuses, websocket outcomes, API events, and DB event types.	Compresses the trajectory into phase evidence: baseline behavior, RTM-targeted trigger, DB/auth/API degradation, and partial recovery.

Appendix

Table 10: Code-level construction of Fig. 8 (bottom) for T9. Calls are Python tool-output events. The bottom four rows match the stream labels in the figure and sum to the 20 outputs after the 13-call orientation prefix.

Modification	Word-Jaccard	GPT-5.2	Sonnet 4.6	Average
Meaning-preserving	0.56	0.98	0.99	0.99
Meaning-altering	0.87	0.14	0.20	0.17

Appendix

Table 11: Judge robustness to reference modifications. Word-Jaccard is the average lexical similarity between original and modified summaries. Judge scores use GPT-5.2 (medium) and Sonnet 4.6 (low); Average is their mean, as used for SLS-Bench scoring.

Model	SLS-Bench score	Human score
Nemotron 3 Nano	0.00	0.10
DeepSeek V3.2	0.10	0.20
GPT-5.4 Nano (medium)	0.38	0.35
Opus 4.6 (low)	0.53	0.65
GPT-5.4 (high)	0.63	0.95

Appendix

Table 12: LLM judge and human evaluation scores. Both columns are means over the same five sampled SLS-Bench problems. The SLS-Bench score averages the two LLM judges.

Model	SLS-Bench score	o11y-bench $\mathrm{Pass}^{3}$
OpenAI
GPT-5.4 (high)	0.73	$0.63^{*}$
GPT-5.4 (low)	0.67	$0.56^{*}$
GPT-5.4 Mini (high)	0.61	$0.29^{*}$
GPT-5.4 Mini (low)	0.48	$0.21^{*}$
GPT-5.4 Nano (high)	0.44	$0.48^{*}$

Appendix

Table 13: SLS-Bench and o11y-bench scores. A superscript

*

marks o11y-bench scores from the published leaderboard; unmarked o11y-bench scores are from our runs.

Model	SLS-Bench score	RCA Correct	RCA Partial
GPT-5.4 (low)	0.67	0.14	0.22
Sonnet 4.6 (low)	0.58	0.14	0.21
Haiku 4.5	0.46	0.04	0.08
GPT-5.4 Nano (low)	0.28	0.12	0.20
Nemotron 3 Nano	0.06	0.02	0.07

Appendix

Table 14: SLS-Bench and OpenRCA scores. OpenRCA results cover the 51-problem Telemetry subset; SLS-Bench scores are from the full benchmark.

Figure 13: Benchmark score vs. average cost per problem for all evaluated models.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,217 papers

SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response
Jul 29, 2026Lehan Wang, Boli Chen, Ruixue Ding +7LLM Agent Evaluation Cybersecurity
Tongyi Lab, Alibaba Group · Alibaba Cloud Computing, Alibaba Group · The Hong Kong University of Science and Technology
Benchmarking and Exploring the Capabilities of LLMs for Attack Investigations
Jun 9, 2026Aniket Anand, Yiwei Hou, Daniel Fields +4LLM Evaluation Cybersecurity
University of Chicago · USA · University of California, Berkeley +1
Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps
Apr 21, 2026Alankrit Chona, Igor Kozlov, Ambuj KumarAI Agent Security Benchmarks LLMs for Cybersecurity
Simbian AI

Jul 29, 2026J/K move · Enter open · S save

SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response

Lehan Wang, Boli Chen, Ruixue Ding, Pengjun Xie+6

Tongyi Lab, Alibaba Group · Alibaba Cloud Computing, Alibaba Group · The Hong Kong University of Science and Technology

LLM Agent Evaluation Cybersecurity Digital Forensics AI Agent Security Benchmarks

Open paper AskPDF arXiv

Large Language Model (LLM) agents are increasingly adopted in real-world security operations with access to host artifacts and command-line interfaces (CLIs), making it critical to thoroughly assess their security capabilities. However, existing cybersecurity benchmarks focus on pre-compromise settings where agents are placed in a clean and idealized environment before an attack occurs. This leaves the post-compromise setting underexplored. To address this gap, we introduce SecRespond, the first benchmark for evaluating LLM agents on the post-compromise incident-response workflow. Given a forensic disk snapshot of a compromised host together with the alerts, vulnerability scans, and baseline checks reported by a host security product, agents are required to produce forensic reports on intrusions, baseline risks, and vulnerability risks, together with a remediation plan. We instantiate this task across 10 cyber ranges, each constructed from a distinct compromised cloud host, spanning 4 entry-point types, 21 ATT&CK techniques, and 5 operating systems. We evaluate 23 frontier LLMs on the OpenCode agent harness. Experimental results show that although current agents can reliably uncover the problems exposed by alerts, they struggle to proactively investigate the disk for silent intrusions and to produce comprehensive, verified remediation plans, with no model achieving complete detection and remediation on any single range. This reveals a fundamental bottleneck in building agents for real-world incident response. The benchmark is publicly available at https://github.com/Alibaba-NLP/qqr/tree/main/data/secrespond.

Abstract

LLM Agent Evaluation

Explore similar work

SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response

Benchmarking and Exploring the Capabilities of LLMs for Attack Investigations

Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps