A Reproducible Semantic Benchmark for Multivendor DSM-to-CLI Translation

Jerônimo Menezes; Leonardo Bitzki; Diego Kreutz; Gefte Almeida; Marcio Pohlmann; Rodrigo Mansilha

cs.NIApr 5, 2026

A Reproducible Semantic Benchmark for Multivendor DSM-to-CLI Translation

Authors: Jerônimo Menezes, Leonardo Bitzki, Diego Kreutz, Gefte Almeida, Marcio Pohlmann, Rodrigo Mansilha

Organizations: Federal University of Rio Grande do Sul (UFRGS) · AI Horizon Labs and PPGES – Federal University of Pampa (UNIPAMPA)

Abstract

Translating high-level network intents into correct multivendor configurations remains a central challenge in network automation, as syntactically valid outputs may still violate the intended operational state. Despite recent advances in Large Language Models (LLMs), the field still lacks reproducible semantic benchmarks for rigorous cross-vendor evaluation. This paper presents a reproducible DSM-to-CLI semantic benchmark covering five cloud LLMs, three vendors, five representative use cases, and ten repeated runs per experimental cell under fixed judges and an explicit failure taxonomy. Our results show that semantic quality and operational reliability are orthogonal, vendor effects dominate use-case effects, and repeated-run dispersion strongly predicts vote instability, with Huawei VRP exposing failure modes hidden by aggregate metrics. These findings demonstrate that multivendor, repeated-execution semantic benchmarks are essential for scientifically rigorous comparison of LLM-based network configuration systems.

Figures & tables

Paper	Input $\rightarrow$ Output	Multivendor	Semantic Evaluation	Repeated Runs	Benchmark Protocol
Lin et al.	NL $\rightarrow$ YANG $\rightarrow$ CLI	Partial	No	Limited (3 trials)	No
Net2d-LLM	DSM $\rightarrow$ CLI	No	No	No	No
NetConfEval	NL $\rightarrow$ Spec/API/CLI	No	Lexical only	Limited	Partial
PeeringLLM-Bench	NL $\rightarrow$ CLI	Yes	Binary/functional	Yes	Yes
NetLLMBench	NL $\rightarrow$ Execution	No	No	No	No
INTA	CLI $\rightarrow$ CLI	Limited	No	No	No

TABLE I: Comparative analysis of LLM-based network configuration approaches.

Outcome	Raw	Final	Rate
Correct (semantic)	655	655	87.3%
Semantic incorrect	29	12	1.6%
Judge fault (infra)	0	17	2.3%
Pipeline error	66	66	8.8%
Total	750	750	100.0%

TABLE II: Execution outcome summary and analysis subsets.

Translator	Corr./Tot.	Rate	Sem.	Pipe.	$\mu$ votes	$\sigma$ votes
grok-4-1-fast-reasoning	149/150	99.3%	1	0	2.660	0.489
gpt-5	143/150	95.3%	3	0	2.507	0.588
gemini-2.5-pro	142/150	94.7%	3	0	2.520	0.599
deepseek-chat	121/150	80.7%	5	16	2.455	0.667
claude-opus-4-6	100/150	66.7% / 100%*	0	50	2.990	0.100
* Rate over valid non-outage runs.

TABLE III: Translator-level correctness, semantic failures, and pipeline errors.

Fig. 1: Semantic versus end-to-end correctness by translator.

Vendor	Corr./Tot.	Rate	SI	PE	$\mu$ votes
Cisco NX-OS	234/250	93.6%	0	16	2.705
Arista EOS	231/250	92.4%	1	17	2.674
Huawei VRP	190/250	76.0%	11	33	2.419

TABLE IV: Aggregate results by vendor.

Contrast	$\hat{p}_{1}$	$\hat{p}_{2}$	$\Delta$ (95% CI) / $p$
Cisco vs Huawei	0.936	0.760	$+$ 0.176 [0.115, 0.237] / $<$ 0.001
Arista vs Huawei	0.924	0.760	$+$ 0.164 [0.102, 0.226] / $<$ 0.001
Cisco vs Arista	0.936	0.924	$+$ 0.012 [ $-$ 0.033, 0.057] / 0.599

TABLE V: Planned vendor contrasts for H1.

Fig. 2: Correctness rate by translator and vendor.

UC	Description	Corr./Tot.	Rate	SI	PE	$\mu$ votes
UC1	L2 access	132/150	88.0%	1	11	2.583
UC2	L2 trunk + VLAN	135/150	90.0%	3	10	2.629
UC3	SVI IPv4	130/150	86.7%	6	10	2.564
UC4	SVI IPv6	127/150	84.7%	1	20	2.638
UC5	Dual-stack SVI	131/150	87.3%	1	15	2.607

TABLE VI: Aggregate results by use case.

Contrast	$\hat{p}_{L2}$	$\hat{p}_{SVI}$	$\Delta$ (95% CI)	$p$ -value
L2 vs SVI (all runs)	0.890	0.862	$+$ 0.028 [ $-$ 0.020, 0.075]	0.253
L2 vs SVI (excl. outage)	0.950	0.908	$+$ 0.042 [ $+$ 0.001, 0.082]	0.044

TABLE VII: Planned contrast for H2 (Layer 2 vs SVI).

Fig. 3: Outcome dispersion vs. vote stability across 75 cells.

	View A ( $n{=}667$ )			View B ( $n{=}483$ )
Stratum	$n$	$\kappa$	Splits	$n$	$\kappa$	Splits
Overall	667	$-$ 0.077	225	483	$+$ 0.197	41
Cisco NX-OS	234	$-$ 0.109	69	165	$+$ 1.000	0
Arista EOS	232	$-$ 0.104	73	162	$+$ 0.244	3
Huawei VRP	201	$-$ 0.046	83	156	$+$ 0.134	38
UC1	133	$-$ 0.106	45	98	$+$ 0.056	10

TABLE VIII: Inter-judge agreement under full and binary-only views.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

NetConfArena: An Executable Benchmark for LLM Agents in Closed-Loop Network Configuration
Aug 24, 2026Chang Liu, Xiaohui Xie, Xinyi Chen +1LLM Agent Evaluation AI Agent Benchmarks
Department of Computer Science and Technology, Tsinghua University, Beijing, China
Evaluating Agentic Configuration Repair for Computer Networks
Jun 4, 2026Rufat Asadli, Benjamin Hoffman, Ioannis Protogeros +1LLM Agent Evaluation AI Agent Benchmarks
Department of Information Technology and Electrical Engineering, ETH Zurich.
Uncertainty Signals for Network Intent Translation: Risk Ranking and Ambiguity Localization
Sep 3, 2026Ala' A. Alsamarneh, Omar AlhusseinIntent-Based Networking LLM Uncertainty Estimation
Department of Computer Science, College of Computing and Mathematical Sciences Khalifa University, Abu Dhabi, United Arab Emirates

Aug 24, 2026J/K move · Enter open · S save

NetConfArena: An Executable Benchmark for LLM Agents in Closed-Loop Network Configuration

Chang Liu, Xiaohui Xie, Xinyi Chen, Yong Cui

Department of Computer Science and Technology, Tsinghua University, Beijing, China

LLM Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

Large language model (LLM) agents are increasingly attractive for automating network configuration, yet their reliability and failure patterns are poorly understood. An essential prerequisite is to assess such agents in a realistic but risk-free environment. Existing benchmarks, however, fall short: they often treat configuration as static command generation or rely on overly simplified settings. Such evaluations understate the core challenges of network configuration, where correctness requires reasoning about protocol complexity and topology dependence. We present NetConfArena, an executable benchmark for evaluating LLM agents in closed-loop network configuration. NetConfArena places agents in emulated multi-device networks, provides a standardized and compact action interface for task execution, and evaluates the resulting network behavior with hidden task-specific executable test cases. The benchmark relies on an LLM-assisted, emulation-grounded pipeline, which converts human-oriented network materials into reusable parameterized task templates. We evaluate representative LLM agents on 480 task instances instantiated from 96 protocol-focused task templates, yielding 3840 execution trajectories, and show that failures are not limited to command errors. The failures also reveal gaps in task-specification adherence and robust planning and execution. These findings suggest two future directions: using validated trajectories as supervision signals to improve foundation models, and designing harness mechanisms that make agent execution more reliable and accountable.

Abstract

LLM Evaluation

Explore similar work

NetConfArena: An Executable Benchmark for LLM Agents in Closed-Loop Network Configuration

Evaluating Agentic Configuration Repair for Computer Networks

Uncertainty Signals for Network Intent Translation: Risk Ranking and Ambiguity Localization