Improving LLM Performance Through Black-Box Online Tuning: A Case for Adding System Specs to Factsheets for Trusted AI

Yonas Atinafu; Henry Lin; Robin Cohen

cs.AIMar 11, 2026

Improving LLM Performance Through Black-Box Online Tuning: A Case for Adding System Specs to Factsheets for Trusted AI

Authors: Yonas Atinafu, Henry Lin, Robin Cohen

Organizations: David R. Cheriton School of Computer Science, University of Waterloo, Waterloo, Canada

Abstract

In this paper, we present a novel black-box online controller that uses only end-to-end measurements over short segments, without internal instrumentation, and hill climbing to maximize goodput, defined as the throughput of requests that satisfy the service-level objective. We provide empirical evidence that this design is well-founded. Using this advance in LLM serving as a concrete example, we then discuss the importance of integrating system performance and sustainability metrics into Factsheets for organizations adopting AI systems.

Figures & tables

Panel of Figure 1 — Fig. 1: SLO-Tuner treats the server as a black box and tunes concurrency, batch size, and speculative width from tail latency and goodput.

Panel of Figure 2 — Fig. 2: Simulator goodput–p99 tradeoffs (steady vs. bursty).

Panel of Figure 3 — Fig. 3: Simulator hill-climb trajectories.

Panel of Figure 4 — Fig. 4: Simulator ablations over speculative width, verifier cadence, batch size.

Panel of Figure 5 — Fig. 5: vLLM experiments: SLO-Tuner trajectory and ablations over concurrency, batch size, and speculative width.

Knob setting	p99 (s)	Goodput (rps)
Baseline: spec width 8, spec on	$\approx$ 1.36	$\approx$ 8.1
Tuned: spec width 0, spec on	$\approx$ 0.70	$\approx$ 15.0

TABLE I: Headline vLLM results (TinyLlama on vLLM, 1.2 s p99 SLO).

Panel of Figure 7 — Fig. 6: MLX vs. simulator: concurrency, lookahead $k$ , and top- $k$ width. Dashed=sim, solid=MLX.

LLM Inference Serving

Large language model inference serving studies how to run model requests efficiently while meeting latency, throughput, cost, and energy goals.

See all 72 papers

Blackbox Performance Analysis of LLM-based Network Control
Oct 17, 2026Mahdi Alizadeh, Pooria Namyar, Arvin Ghavidel +1Black-Box Model Auditing Language Model-Based Control
University of Southern California · Microsoft Research
Human-Less LLM Serving: Quantifying the Human Tax on Throughput
May 3, 2026Jianhui Lian, Li Chen, Dan Li +1LLM Inference Efficiency LLM Serving
Tsinghua SIGS · Beijing Zhongguancun Laboratory · Tsinghua University
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Aug 6, 2026Varun Ursekar, Apaar Shanker, Yash Maurya +4Large Language Model-Guided Optimization LLM Agent Evaluation
Scale AI

Oct 17, 2026J/K move · Enter open · S save

Blackbox Performance Analysis of LLM-based Network Control

Mahdi Alizadeh, Pooria Namyar, Arvin Ghavidel, Ramesh Govindan

University of Southern California · Microsoft Research

Black-Box Model Auditing Language Model-Based Control

Open paper AskPDF arXiv

LLMs are becoming an indispensable part of network control systems due to their impressive reasoning. Before such LLM-enabled systems can be deployed, it is imperative to develop tools that can audit their performance on previously unseen inputs. However, auditing these systems is challenging due to the inherent opacity of LLMs. LENS is the first step towards black-box performance analysis of LLM-enabled network control systems. It uses novel methods to extend deep reinforcement learning to guide the search for performance gaps and inputs that make LLMs struggle. LENS finds large gaps for LLM-based traffic engineering and GPU allocation systems, and even matches existing white-box analyzers designed for hand-crafted TE heuristics.

Abstract

LLM Inference Serving

Explore similar work

Blackbox Performance Analysis of LLM-based Network Control

Human-Less LLM Serving: Quantifying the Human Tax on Throughput

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization