Large language model (LLM) agents are increasingly attractive for automating network configuration, yet their reliability and failure patterns are poorly understood. An essential prerequisite is to assess such agents in a realistic but risk-free environment. Existing benchmarks, however, fall short: they often treat configuration as static command generation or rely on overly simplified settings. Such evaluations understate the core challenges of network configuration, where correctness requires reasoning about protocol complexity and topology dependence. We present NetConfArena, an executable benchmark for evaluating LLM agents in closed-loop network configuration. NetConfArena places agents in emulated multi-device networks, provides a standardized and compact action interface for task execution, and evaluates the resulting network behavior with hidden task-specific executable test cases. The benchmark relies on an LLM-assisted, emulation-grounded pipeline, which converts human-oriented network materials into reusable parameterized task templates. We evaluate representative LLM agents on 480 task instances instantiated from 96 protocol-focused task templates, yielding 3840 execution trajectories, and show that failures are not limited to command errors. The failures also reveal gaps in task-specification adherence and robust planning and execution. These findings suggest two future directions: using validated trajectories as supervision signals to improve foundation models, and designing harness mechanisms that make agent execution more reliable and accountable.
Figures & tables
Fig. 1: Motivation example from NetConfArena, illustrating the closed-loop workflow of a network configuration agent.
Benchmark
R1
R2
R3
NetConfEval
✗
◐
✗
NetLLMBench
✗
✗
◐
NetArena
✓
✗
◐
NetConfArena
✓
✓
✓
TABLE I: Comparison with existing benchmarks for LLM-based network configuration.
Fig. 2: Overview of the NetConfArena framework. NetConfArena connects the Benchmark, LLM Agent, Environment, and Evaluator components across three stages: task setup, agent execution, and agent evaluation.
Fig. 3: Benchmark task construction pipeline in NetConfArena: human-oriented configuration materials are converted into structured task specifications with executable test cases, validated through emulation, and parameterized under constraints to produce reusable templates and seeded task instances.
Model
Think
Score
Pass rate
Cfg. F1
Turns
Err. act.
Rep. act.
Limit
Early sub.
Corr. sub.
Err. sub.
qwen3-8B
✗
0.337
0.046
0.508
18.26
0.455
0.496
0.833
0.000
0.046
0.121
✓
0.457
0.094
0.582
14.21
0.317
0.124
0.446
0.000
0.094
0.460
qwen3-32B
✗
0.656
0.233
0.684
15.61
0.090
0.017
0.473
0.000
0.154
0.373
✓
0.728
0.421
0.682
10.88
0.139
0.020
0.208
0.013
0.379
0.400
deepseek-v4-flash
✗
0.914
0.656
0.759
14.72
0.027
0.000
0.350
0.000
0.502
0.148
✓
0.933
0.794
0.739
14.42
0.038
0.001
0.246
0.000
0.675
0.079
TABLE II: Outcome and process metrics for evaluated LLM agents in NetConfArena.
Fig. 4: Action distribution across models and thinking modes. The distribution excludes get_running_config and submit actions. For each model, the left bar corresponds to thinking disabled and the right bar corresponds to thinking enabled.
Fig. 5: Task overhead comparison between thinking-disabled and thinking-enabled settings.
Field
Template-level content
Scenario
Defines the task category, e.g., BGP Local Preference and MED.
Goals
Specifies the natural-language intent with placeholders, including iBGP over loopbacks, eBGP sessions, loopback advertisement, reachability, MED-based inbound preference, and Local Preference-based outbound preference.
Topology model
Describes abstract nodes and links before rendering, including four routers and five logical links whose media are parameterized.
Startup configurations
Defines initial per-router configuration templates using placeholders for interfaces, loopbacks, link addresses, masks, and EIGRP networks.
Reference configurations
Defines reference configuration templates for BGP neighbors, update sources, advertised loopbacks, route maps, MED, and Local Preference.
Test cases
Defines diagnostic commands and expected structured outcomes for BGP sessions, route advertisements, path selection, and policy effects.
TABLE III: Structure of a parameterized task template before rendering.
Intent
Device
Diagnostic command
Deterministic predicate
iBGP sessions use loopback endpoints inside AS100
router4_name
show ip bgp neighbors
Output contains established AS100 neighbors for router2_loopback0_ip and router1_loopback0_ip , with the local endpoint equal to router4_loopback0_ip .
eBGP session between router2_name and router3_name
router2_name
show ip bgp neighbors
Neighbor link5_ip2 is established with remote AS 200 , local IP link5_ip1 , and remote IP link5_ip2 .
eBGP session between router1_name and router3_name
router1_name
show ip bgp neighbors
Neighbor link3_ip2 is established with remote AS 200 , local IP link3_ip1 , and remote IP link3_ip2 .
Fig. 6: Generalizability analysis based on the test-case scores from the main evaluation. (a) Hierarchical score-variance decomposition; “8B” denotes qwen3-8B, “32B” denotes qwen3-32B, “Flash” denotes deepseek-v4-flash, and “Pro” denotes deepseek-v4-pro. The suffix “D” denotes reasoning disabled and “E” denotes reasoning enabled. (b) Spearman rank consistency for the 16 subcategories in the Table V order; full subcategory names are omitted to keep the panel legible. The dashed line denotes ρ=0.9 .