cs.NIOct 17, 2026

Blackbox Performance Analysis of LLM-based Network Control

Authors: Mahdi Alizadeh, Pooria Namyar, Arvin Ghavidel, Ramesh Govindan

Organizations: University of Southern California · Microsoft Research

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

LLMs are becoming an indispensable part of network control systems due to their impressive reasoning. Before such LLM-enabled systems can be deployed, it is imperative to develop tools that can audit their performance on previously unseen inputs. However, auditing these systems is challenging due to the inherent opacity of LLMs. LENS is the first step towards black-box performance analysis of LLM-enabled network control systems. It uses novel methods to extend deep reinforcement learning to guide the search for performance gaps and inputs that make LLMs struggle. LENS finds large gaps for LLM-based traffic engineering and GPU allocation systems, and even matches existing white-box analyzers designed for hand-crafted TE heuristics.

Figures & tables

ToolNo Additional Formal ModelingNo Source Code AccessStructure Aware Search
LENS (ours)✓✓✓
White-box analysis ( Namyar et al., 2024a ; Goel et al., 2024 ; Tahmasbi Arashloo et al., 2023 )✗✓✓
Source code analysis ( Karimi et al., 2026 )✓✗✓
Gray-box analysis ( Namyar et al., 2024c )✗✓✓
Traditional black-box ( Davis, 1991 ; Kirkpatrick et al., 1983 )✓✓✗
Table 1. Related work on performance analyzers.
Figure 1. LLM-based traffic engineering workflow.
 TEGPU Scheduling
StateDemand matrix of src–dest traffic volumes.Job matrix of throughput and GPU requirements.
ActionEdit multiple demand entries.Edit throughput/GPU use of multiple jobs.
RewardRouted flow difference (OPT vs. LLM).Selected throughput difference (OPT vs. LLM).
Table 2. Mapping RL to specific problem domains.
Figure 2. LENS architecture. The policy edits a structured input, the environment projects the edit back into the valid input space, and the QOPT(I)Q_{\mathrm{OPT}}(I) and QLLM(I)Q_{\mathrm{LLM}}(I) executors evaluate the result. The measured gap is stored as a reward and reused by the learner.
Figure 3. LENS outperforms black-box baselines for (a) traffic engineering targets, LMTE and Teal, and (b) cluster scheduling with LLMCoSolver. Teal is the learned target discussed in Section 4.2 .
Figure 4. LENS matches MetaOpt, while outperforming MetaEase and black-box baselines under the same time budget.
Figure 5. LENS’s multi-stage refinement finds larger gaps than vanilla RL under the same search setup. (a) Traffic engineering, (b) Cluster scheduling with different job counts.
Figure 6. Sparse reward frequency kk trades search quality for runtime on LMTE.
More in this topic

Black-Box Model Auditing

Black-box model auditing evaluates AI systems through their inputs and outputs to assess privacy, fairness, safety, reliability, and provenance.

See all 96 papers

Explore similar work

CardsList

  1. Improving LLM Performance Through Black-Box Online Tuning: A Case for Adding System Specs to Factsheets for Trusted AI

    Mar 11, 2026Yonas Atinafu, Henry Lin, Robin CohenLLM Inference ServingLLM Serving

    David R. Cheriton School of Computer Science, University of Waterloo, Waterloo, Canada

  2. White-Box Sensitivity Auditing with Steering Vectors

    Jan 23, 2026Hannah Cyberey, Yangfeng Ji, David EvansAlgorithmic AuditingLLM Auditing

    University of Virginia

  3. Measuring Activation Control in Large Language Models

    Aug 21, 2026Marek Mateusz Kowalski, Joshua Fonseca Rivera, Uzay Macar +1LLM Safety EvaluationLLM Auditing

    UK AI Security Institute