FaulT-Bench: Towards Benchmarking Network Troubleshooting LLM Agents under Unreliable User Tickets

Kuan-Hao Tseng; Niruth Bogahawatta; Yasod Ginige; Kunjan Patel; Kosta Dakic; Suranga Seneviratne

cs.NIAug 27, 2026

FaulT-Bench: Towards Benchmarking Network Troubleshooting LLM Agents under Unreliable User Tickets

Authors: Kuan-Hao Tseng, Niruth Bogahawatta, Yasod Ginige, Kunjan Patel, Kosta Dakic, Suranga Seneviratne

Organizations: The University of Sydney Australia

Abstract

LLM-based agents are increasingly proposed for network fault diagnosis, but existing benchmarks evaluate them only on accurate tickets and always assume a fault is present, conditions rarely met in practice. We present FaulT-Bench, a benchmark of 200 troubleshooting scenarios across eight network topologies, five reimplemented from public practitioner labs, spanning genuine faults, false fault reports, incorrect device attribution, and incorrect root-cause claims. To isolate how ticket wording affects diagnosis, we further rewrite 72 false-premise tickets into five reporter personas that vary reporter confidence and verifiable detail one factor at a time, holding the network state fixed. Our automated harness deploys each scenario in Kathará, lets agents interact through the NIKA tool interface, and scores free-text diagnoses with an LLM judge across outcome, fix, and reasoning quality. Evaluating SADE, ReAct, and Claude Code, we find all three are near-saturated on accurate tickets and robust to misdirection, yet degrade sharply when the network is healthy and the ticket is wrong, probing until a benign condition can be promoted to a root cause rather than concluding nothing is wrong. Persona rewrites show that how a ticket is written matters more than what it claims: a confidently wrong report is handled about as well as an accurate one, while a vague, underspecified report degrades performance sharply. The three agents also fail differently, from constant over-diagnosis to unanswered runs, at very different cost. These results position FaulT-Bench as a benchmark for developing agentic systems that can reason reliably over the noisy, unreliable tickets of real-world network troubleshooting.

Figures & tables

Network	Devices	Routers	Protocols and features
Kath1 ( Jeremy’s IT Lab, 2024 )	20	12	OSPF, VRRP, DHCP, dual ISP, ACLs
Kath2 ( katejay, 2019 )	30	3	RIP; web, FTP, DNS services
Kath3 ( imsiddhant, 2020a )	32	9	RIP, VLANs, ACL firewall
Kath4 ( imsiddhant, 2020b )	28	9	OSPF; SSH-managed switches
Kath5 ( Internetworks, 2024 )	15	13	OSPF, RIP, EIGRP redistribution
nika_ospf ( Wang et al., 2025a )	15	6	OSPF, static routes

Table 1. Summary of the eight network topologies.

Panel of Figure 1 — Figure 1. Anatomy of a scenario file. Only the ticket reaches the agent; every other block exists to inject, verify, and grade.

Category	#	ID	Network state	Ticket characteristics
Correct fault	80	—	injected fault	accurately describes the fault
False premise	72	3.2	verified healthy	reports a nonexistent fault (standard)
Wrong device	24	—	injected fault	blames a healthy device
Wrong cause	24	—	injected fault	asserts the wrong cause
Novice	72	1.9	verified healthy	non-technical, uncertain; omits cause
confident	72	1.9	verified healthy	novice content; demanding tone

Table 2. Dataset composition: 200 core scenarios; the five persona-rewrite sets are a controlled variant layer. ID: mean distinct verifiable identifiers per ticket.

Panel of Figure 2 — Figure 2. Six ticket forms of one false-premise scenario: the standard ticket and the novice pair (top), the naive chain (bottom). Each control changes one aspect relative to its parent, while the network and the ground truth remain the same.

Figure 3. System overview of FaulT-Bench.

Panel of Figure 4 — Figure 4. One run through the three scores. The judge sees the ground truth, the diagnosis, and for the reasoning score, a digest of the investigation log.

Agent	Base model	Beyond the 22 tools	Pairing isolates
SADE ( Tseng et al., 2026 )	claude-sonnet-4-6	skills + helpers	—
CC-Baseline ( Anthropic, 2025 )	claude-sonnet-4-6	nothing	skills layer
ReAct ( Yao et al., 2023 ; Wang et al., 2025a )	gpt-5	nothing	model + loop

Table 3. Evaluated agents: Access the same 22-tool interface, receive identical inputs, and have the same 20-turn budget.

Class ( $N$ )	SADE	CC-B.	ReAct	All	$\Delta$	Over-d.	T.O.
	Outcome, runs that answered
Correct-Fault (80)	.912	.942	.941	.932	—	—	2%
False-Premise (72)	.745	.870	.848	.820	$-.112$	17%	5%
Wrong-Device (24)	.948	.993	.931	.957	$+.025$	—	0%
Wrong-Cause (24)	.901	.981	.850	.911	$-.021$	—	0%

Table 4. E1: outcome by class and agent on core scenarios.

Persona	ID	SADE	CC-B.	ReAct	All	Over-d.	T.O.
		Outcome, runs that answered
Standard	3.2	.745	.870	.848	.820	17%	5%
Naive	2.1	.773	.911	.789	.820	16%	9%
Novice	1.9	.643	.793	.609	.675	31%	11%
Novice-confident	1.9	.704	.794	.654	.714	28%	13%
Naive-unsure	2.1	.698	.486	.670	.638	35%	19%

Table 5. E2: outcome by persona on the 72 healthy networks

	SADE		CC-B.		ReAct
Persona	Over-d.	T.O.	Over-d.	T.O.	Over-d.	T.O.
Standard	24%	3%	11%	8%	16%	3%
Naive	20%	4%	7%	17%	20%	6%
Novice	32%	6%	20%	21%	39%	6%
Novice-confident	27%	6%	19%	22%	37%	10%
Naive-unsure	28%	8%	51%	43%	32%	7%

Table 6. Over-diagnosis and Turn budget exhaustion.

	Fix				Reasoning
Class / persona	SADE	CC-B.	ReAct	All	SADE	CC-B.	ReAct	All
Correct-Fault (80)	.952	.962	.924	.946	.937	.917	.821	.891
False-Premise (72)	—	—	—	—	.933	.892	.845	.890
Wrong-Device (24)	.903	.979	.873	.919	.944	.957	.868	.923
Wrong-Cause (24)	.896	.958	.837	.897	.934	.929	.798	.887
Naive	—	—	—	—	.911	.894	.817	.873

Table 7. Fix and reasoning scores by class and persona.

Agent	answered	timeout	answered	timeout	Spend on t.o.
	Tool calls		US$
SADE	16.6	38.0	0.45	1.04	10%
CC-B.	22.1	49.2	0.43	1.07	36%
ReAct	19.5	45.5	0.12	0.38	14%

Table 8. Effort per agent: Tool calls and Cost.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,132 papers

May 6, 2026

SADE: Symptom-Aware Diagnostic Escalation for LLM-Based Network Troubleshooting

Large language model (LLM) agents are increasingly applied to network troubleshooting, but root-cause localization on public benchmarks remains well below practical deployment thresholds. We argue this is because existing agents do not encode the disciplined, layer-by-layer methodology that human network engineers use, and instead rely on free-form deliberation that conflates evidence acquisition with hypothesis commitment. We present SADE (Symptom-Aware Diagnostic Escalation), an agent that encodes the classical Cisco troubleshooting methodology as an explicit policy. SADE pairs a phase-gated diagnostic workflow, which separates evidence acquisition from hypothesis commitment, with a routed library of fault-family skills and high-yield diagnostic helpers. On a held-out 523 incident set of the public NIKA benchmark covering eleven unseen scenarios, SADE improves root-cause F1 by 37 percentage points over a ReAct + GPT-5 baseline; a model-controlled comparison against the same Claude Sonnet backend without the SADE policy attributes 22 of those points to the diagnostic policy alone, showing that the gain is not a side-effect of the model upgrade.

Kuan-Hao Tseng, Niruth Bogahawatta, Yasod Ginige +3

University of Sydney, Sydney, Australia · University of New South Wales, Sydney, Australia

LLM Agent Orchestration LLM Agents Root Cause Analysis LLM Agent Routing Fault Diagnosis

Ask PDF

Aug 12, 2026

CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations

Agents are increasingly considered for automating network operations and maintenance, where engineers must diagnose network faults, optimize configurations to enhance services, and reduce operational costs while acting under strict constraints. However, existing evaluations fail to accurately model real network characteristics or assess agents under partially observable telecom environments with diverse vendors, devices, protocols, and interfaces. In this paper, we introduce CTBench, a public benchmark for assessing whether an agent behaves like a competent telecom troubleshooting engineer. CTBench focuses on root cause analysis and path restoration. Each task is constructed by experts and annotated with rich task metadata, including golden evidence steps. CTBench uses expert-grounded metrics that evaluate both final answers and the diagnostic evidence. Experiments with representative harness-model combinations show that state-of-the-art agents perform very well at identifying endpoints in path-restoration tasks but, more generally, underperform in root cause analysis. In particular, agents struggle with interface state, link-layer, service-management, and other operational faults. Most importantly, even when agents produce plausible or correct final answers, they often fail to provide the evidence-grounded diagnoses required in operational practice. Our results further show that path restoration is generally more resource expensive, yet larger resource usage does not necessarily translate into better diagnosis.

Xingyu Yan, Tingting Dai, Antonio De Domenico +16

*Huawei Technologies, China · †Paris Research Center, Huawei Technologies, Boulogne-Billancourt, France · +Queen Mary University of London

AI Agent Benchmarks

Ask PDF

Aug 24, 2026

NetConfArena: An Executable Benchmark for LLM Agents in Closed-Loop Network Configuration

Large language model (LLM) agents are increasingly attractive for automating network configuration, yet their reliability and failure patterns are poorly understood. An essential prerequisite is to assess such agents in a realistic but risk-free environment. Existing benchmarks, however, fall short: they often treat configuration as static command generation or rely on overly simplified settings. Such evaluations understate the core challenges of network configuration, where correctness requires reasoning about protocol complexity and topology dependence. We present NetConfArena, an executable benchmark for evaluating LLM agents in closed-loop network configuration. NetConfArena places agents in emulated multi-device networks, provides a standardized and compact action interface for task execution, and evaluates the resulting network behavior with hidden task-specific executable test cases. The benchmark relies on an LLM-assisted, emulation-grounded pipeline, which converts human-oriented network materials into reusable parameterized task templates. We evaluate representative LLM agents on 480 task instances instantiated from 96 protocol-focused task templates, yielding 3840 execution trajectories, and show that failures are not limited to command errors. The failures also reveal gaps in task-specification adherence and robust planning and execution. These findings suggest two future directions: using validated trajectories as supervision signals to improve foundation models, and designing harness mechanisms that make agent execution more reliable and accountable.

Chang Liu, Xiaohui Xie, Xinyi Chen +1

Department of Computer Science and Technology, Tsinghua University, Beijing, China

LLM Agent Evaluation AI Agent Benchmarks

Ask PDF

Abstract

LLM Agent Evaluation

Explore similar work

SADE: Symptom-Aware Diagnostic Escalation for LLM-Based Network Troubleshooting

CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations

NetConfArena: An Executable Benchmark for LLM Agents in Closed-Loop Network Configuration