NetAgentBench: A State-Centric Benchmark for Evaluating Agentic Network Configuration

Ahmed Twabi; Yepeng Ding; Tohru Kondo

cs.NIApr 3, 2026

NetAgentBench: A State-Centric Benchmark for Evaluating Agentic Network Configuration

Authors: Ahmed Twabi, Yepeng Ding, Tohru Kondo

Organizations: Informatics & Data Science, Hiroshima University, Hiroshima, Japan

Abstract

As agentic network management gains popularity, there is a critical need for evaluation frameworks that transcend static, one-shot testing. To address this, we introduce NetAgentBench, a dynamic benchmark that evaluates agent interactions through a Finite State Machine (FSM) formalization guaranteeing determinism, correctness, and bounded execution. This provides the networking landscape with a rigorous foundation to measure complex, multi-turn operational behaviors. Our empirical evaluation of four state-of-the-art LLM agents through diverse network configuration tasks reveals stark deficiencies: while agents can solve basic tasks, they suffer severe exploration meltdowns and coherence collapse during expert-level configurations. Ultimately, NetAgentBench demonstrates that systematically evaluating multi-turn behavioral stability is an indispensable step toward realizing trustworthy, fully autonomous networks.

Figures & tables

Fig. 1: Benchmark architecture: $\mathcal{M}_{\text{bench}}$ orchestrates $\mathcal{M}_{\text{infra}}$ and $\mathcal{M}_{\text{SUT}}$ via the Initialize bridge function.

Fig. 2: Event-Driven Convergence: A command transitions the FSM from $s_{\text{stable}}$ to $s_{\text{pending}}$ , followed by an autonomous $\varepsilon$ -transition to a new stable or error state.

Fig. 3: Implementation architecture: Topology provisioning via Containerlab and state interaction via Docker.

Model	Score $\mu{\pm}\sigma$	Succ.	Melt.	Turns	Tok.(K)
GPT-5	$0.508{\pm}0.290$	24.0%	38.7%	12.4	503.0
Qwen3-Coder-30b	$0.310{\pm}0.302$	12.0%	29.3%	12.8	400.2
Llama-3.3	$0.231{\pm}0.262$	6.7%	21.3%	3.6	30.1
Qwen3-Coder-Next	$0.221{\pm}0.261$	5.3%	24.0%	6.3	167.6

TABLE I: Overall Model Performance (N=75 per model)

Fig. 4: Model performance across task difficulty levels. All models degrade across tiers; OSPF (Intermediate) creates a near-universal success collapse.

Fig. 5: Network coherence trajectories (mean $\pm 1\sigma$ ). GPT-5 maintains the most monotonic progression; all models exhibit coherence drops on OSPF tasks.

Fig. 6: Distribution of meltdown signals across models. GPT-5 suffers the highest absolute meltdown rate (38.7%), driven by destructive spirals under complex configurations.

Fig. 7: Token efficiency (score per 1K tokens) vs. overall score. Models in the upper-right quadrant are both high-performing and efficient.

Finite-State Machine

See all 18 papers

NetConfArena: An Executable Benchmark for LLM Agents in Closed-Loop Network Configuration
Aug 24, 2026Chang Liu, Xiaohui Xie, Xinyi Chen +1LLM Agent Evaluation AI Agent Benchmarks
Department of Computer Science and Technology, Tsinghua University, Beijing, China
AgentBetta: Verification-Driven Adaptive Configuration of an AI Nano-Agent through Selective Expansion and Verified Contraction
Sep 20, 2026Md. Ashraful BabuLLM Agents LLM Agent Verification
Department of Physical Sciences · Explainable Artificial Intelligence and Optimization Research (XAiOR)
Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows
May 27, 2026Yilun Yao, Xinyu Tan, Chao-Hsuan Liu +10Computer-Use Agent Benchmarks AI Agent Reliability
Peking University · Qiyuan Tech

Aug 24, 2026J/K move · Enter open · S save

NetConfArena: An Executable Benchmark for LLM Agents in Closed-Loop Network Configuration

Chang Liu, Xiaohui Xie, Xinyi Chen, Yong Cui

Department of Computer Science and Technology, Tsinghua University, Beijing, China

LLM Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

Large language model (LLM) agents are increasingly attractive for automating network configuration, yet their reliability and failure patterns are poorly understood. An essential prerequisite is to assess such agents in a realistic but risk-free environment. Existing benchmarks, however, fall short: they often treat configuration as static command generation or rely on overly simplified settings. Such evaluations understate the core challenges of network configuration, where correctness requires reasoning about protocol complexity and topology dependence. We present NetConfArena, an executable benchmark for evaluating LLM agents in closed-loop network configuration. NetConfArena places agents in emulated multi-device networks, provides a standardized and compact action interface for task execution, and evaluates the resulting network behavior with hidden task-specific executable test cases. The benchmark relies on an LLM-assisted, emulation-grounded pipeline, which converts human-oriented network materials into reusable parameterized task templates. We evaluate representative LLM agents on 480 task instances instantiated from 96 protocol-focused task templates, yielding 3840 execution trajectories, and show that failures are not limited to command errors. The failures also reveal gaps in task-specification adherence and robust planning and execution. These findings suggest two future directions: using validated trajectories as supervision signals to improve foundation models, and designing harness mechanisms that make agent execution more reliable and accountable.

Abstract

Finite-State Machine

Explore similar work

NetConfArena: An Executable Benchmark for LLM Agents in Closed-Loop Network Configuration

AgentBetta: Verification-Driven Adaptive Configuration of an AI Nano-Agent through Selective Expansion and Verified Contraction

Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows