AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs

Cheuk Hang Lau; Zeyu Cao; Kevin Wong Cheuk Yin; Yao Lai; Haoran Wu; Nicholas D. Lane; Robert D. Mullins; Ilia Shumailov

cs.LGSep 28, 2026

AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs

Authors: Cheuk Hang Lau, Zeyu Cao, Kevin Wong Cheuk Yin, Yao Lai, Haoran Wu, Nicholas D. Lane, Robert D. Mullins, Ilia Shumailov, +1 more

Organizations: Imperial College London · University of Cambridge · University of Oxford

Abstract

The optimization of LLM serving engines, such as vLLM and SGLang, is largely benchmark-driven: optimizations, scheduling policies, hardware and system designs are all selected based on representative workloads. However, a significant mismatch has emerged in the agentic era. Existing benchmarks primarily focus on simple single-turn chatbot workloads. LLM applications are increasingly agentic: coding agents, terminal execution systems, and tool-use agents issue multi-turn requests with growing context lengths. We introduce AgentPerfBench, a benchmark suite for agentic inference. It uses real traces from agentic benchmarks, such as SWE-Bench and TerminalBench, alongside standard chat baselines. This enables benchmarking of models on multi-turn tasks involving tool calling, skill utilization, and increasing context lengths. AgentPerfBench also samples from empirical distributions of input length, output length, and turn count derived from the real traces, generating representative synthetic profiles for cheap and accurate measurements on new hardware. In addition, we further find that several existing benchmarks fail to accurately reflect real hardware performance for two key reasons: 1) they do not account for realistic context-length growth, and 2) they measure inference performance without operating at hardware saturation. We discuss these issues in detail and provide rich kernel-level Nsight Compute (NCU) traces to construct a new multi-dimensional roofline model that captures hardware-system limitations in both memory bandwidth and memory capacity footprint. The benchmarking suite then includes automated scripts to identify potential bottleneck conditions on emerging hardware when evaluated with diverse agentic traces. Together, these contributions quantify the chat-to-agentic gap in current inference benchmarks and characterise per-kernel GPU resource utilisation via roofline analysis.

Figures & tables

		Chat			Coding agent			Terminal agent			Computer-use agent
Model	TP	TTFT	TPOT	Req/s	TTFT	TPOT	Req/s	TTFT	TPOT	Req/s	TTFT	TPOT	Req/s
Dense
LLaMA-3.1-8B	1	343	9.4	13.5	11719	227.3	2.4	6889	197.2	3.0	1037	22.8	6.5
Qwen3.5-9B	1	953	12.0	11.0	6954	168.7	3.2	2877	47.1	4.6	1539	22.5	5.9
Qwen3.5-27B	2	1518	21.5	6.0	2729	62.3	5.3	2519	52.0	4.7	2709	33.2	3.6
MoE

Table 4: Medium multi-turn trace_replay saturation metrics on H100-SXM5-80GB with vLLM v0.19.0 and prefix caching enabled. Each cell reports the observed saturation point for one model/profile pair; TTFT and TPOT are medians in ms. Column groups use the workload names from the main text; exact profile IDs are listed in Appendix Table 10 .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Type	Params	$d_{\text{model}}$	Layers	Heads	KV	$d_{\text{ffn}}$	Experts	top- $k$
LLaMA-3.1-8B	Dense	8B	4,096	32	32	8	14,336	—	—
LLaMA-3.1-70B	Dense	70B	8,192	80	64	8	28,672	—	—
LLaMA-3.3-70B	Dense	70B	8,192	80	64	8	28,672	—	—
Qwen2.5-72B	Dense	72B	8,192	80	64	8	29,568	—	—
Qwen3.5-9B	Dense	9B	4,096	32	16	4	12,288	—	—
Qwen3.5-27B	Dense	27B	5,120	64	24	4	17,408	—	—

Appendix

Table 8: Model configurations evaluated. Dense models use grouped-query attention (GQA); MoE models use top-

k

expert routing. Models marked with a dagger ( † ) are included only in the NCU kernel profiling dataset ( kernels_labeled ) and were not run through the serving benchmark.

Model	2080Ti	RTX 3090	A100-40GB	H100
LLaMA-3.1-8B	1, 2, 4	1, 2, 4, 8	1, 2, 4, 8	1, 2, 4
LLaMA-3.1-70B	—	8	4, 8	1, 2, 4
LLaMA-3.3-70B	—	—	4, 8	1, 2, 4
Qwen2.5-72B	—	—	4, 8	1, 2, 4
Mixtral-8 $\times$ 7B	—	8	4, 8	2, 4
Qwen3.5-9B	2, 4	1, 2, 4, 8	1, 2, 4, 8	1, 2

Appendix

Table 9: Tensor-parallel configurations actually run in the released serving summaries. Entries list observed TP degrees after collapsing multi-GPU labels within each platform family. gpt-oss models use MXFP4 weights; all others bfloat16.

Profile	$\textbf{OI}_{\textbf{eff}}$	Ridge multiple
Chat multi-turn	181	$0.6\times$
Computer-use multi-turn	886	$3.0\times$
Terminal multi-turn	1,809	$6.1\times$
Coding-agent multi-turn	2,409	$8.2\times$

Appendix

Table 12: Effective serving OI at

C{=}80

for multi-turn profiles (LLaMA-3.1-8B, H100, TP=1, vLLM, bf16). Values correspond to Figure 4 a.

Profile	ISL	$\textbf{OI}_{\textbf{prefill}}$	$\textbf{OI}_{\textbf{decode}}$	Gap
Chat multi-turn	833	634	78	$8\times$
Computer-use multi-turn	4,666	1,695	78	$22\times$
Terminal multi-turn	10,472	2,123	78	$27\times$
Coding-agent multi-turn	12,929	2,208	78	$28\times$

Appendix

Table 13: Per-phase OI gap at

C{=}80

(LLaMA-3.1-8B, H100, BF16, TP=1).

\text{OI}_{\text{prefill}}

is the per-layer GEMM aggregate at

L{=}\text{ISL}

tokens;

\text{OI}_{\text{decode}}

is the GEMM aggregate at decode batch

L{=}C{=}80

tokens. The H100 ridge is 295 FLOP/byte; prefill is compute-bound for all profiles, decode is bandwidth-bound.

Explore similar work

Sep 16, 2026cs.AI

AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines

Large language model agents tune GPU kernels and serving engines through a closed loop of propose, measure, and keep, but the measurements behind this loop are not trustworthy. We characterize four failure modes from a four-day pilot corpus of 619 model calls: strawman baselines manufacture speedups, absolute times do not transfer across machines, saturated tasks nullify comparisons, and infrastructure defects impersonate science. We present AutoTuneBench, a benchmark and measurement protocol that makes trust architectural. The protocol is frozen as code with test-enforced provenance; a database-level validator rejects out-of-protocol results; anti-cheat checks run outside the agent's modification surface; comparisons follow pre-registered readouts; and measurements anchor to externally published results, grounded in paired-seed statistics with a 5% cross-run coefficient-of-variation cap. Honest measurement rewrites the headlines: our best kernel reads 10.6x against a naive baseline but 2.03x against the honest one; one configuration delivers 1.174x on one machine and 1.0049x on another; a pre-registered on/off comparison nulls at a shared wall (2.4840 vs 2.4957,ms); and the KernelBench Level-1 suite admits 51% of tasks with median speedup 1.0001x over PyTorch eager. The protocol, the two-engine corpus (vLLM and SGLang), and its audit trail are released as open artifacts.

Li Chen

Agentic Benchmarks Large Language Model Benchmarks Tuning Graphics Processing Unit Kernels Large Language Model Serving Text Corpora

Ask PDF

Jul 2, 2026cs.AI

PACE: A Proxy for Agentic Capability Evaluation

Evaluating LLM agents on benchmarks like SWE-Bench and GAIA can be expensive, time-consuming, and requires complex infrastructure. A single evaluation can cost thousands of dollars and take days to complete. In contrast, non-agentic LLM benchmarks that test individual capabilities (e.g., reasoning, code generation) are fast and cheap to run. In this paper, we investigate whether performance on expensive agentic benchmarks can be accurately predicted by the performance on a small, carefully selected subset of atomic evaluation instances. We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations whose aggregate scores most reliably predict model performances on agentic benchmarks. Given a pool of candidate instances spanning atomic capabilities, PACE fits a regression that maps a model's scores on a compact subset of source instances to its score on the target agentic benchmark. The subset itself is curated by combining two complementary instance-selection strategies, target-relevance local selection and globally informative global selection. We apply PACE to the 4 target agentic benchmarks in this paper, which yields PACE-Bench, the concrete proxy benchmark that we evaluate in the paper. Experiments across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks show that PACE-Bench predicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under 4%, Spearman correlation above 0.80, and pairwise model-ranking accuracy around 85%, all at much less than 1% of the full agentic evaluation cost. We further analyze the selected proxy instances, revealing which skills each agentic benchmark uniquely demands. PACE enables practitioners to obtain reliable estimates of agentic performance during model development, selection, and routing, without the overhead of full agent evaluation.

Yueqi Song, Lintang Sutawika, Jiarui Liu +8

Carnegie Mellon University · Salesforce AI Research

Agentic Benchmarks Agentic Evaluations Proxy Benchmarks

Ask PDF

May 27, 2026cs.AI

Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows

LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, performance depends not only on the base model, but also on the harness: the system layer that manages context, tools, state, constraints, permissions, tracing, and recovery. However, existing benchmarks typically abstract away execution, compare complete agent systems, or hold the harness fixed, making execution-layer variation difficult to study. We introduce Harness-Bench, a diagnostic benchmark for evaluating configuration-level harness effects in realistic agent workflows. Harness-Bench evaluates representative harness configurations across multiple model backends under shared task environments, budgets, and evaluation protocols, while preserving each harness's native execution behavior. The benchmark contains 106 sandboxed offline tasks constructed from practical agent-use patterns and manually reviewed for realism, solvability, oracle-checkability, and integrity. Each run records final artifacts, execution traces, usage statistics, and validator outputs, enabling analysis beyond final completion. Across 5,194 execution trajectories, we observe substantial variation in completion, process quality, efficiency, and failure behavior across model-harness pairings. These results suggest that agent capability should be reported at the model-harness configuration level rather than attributed to the base model alone. Our analysis further identifies recurring execution-alignment failures, where plausible reasoning becomes decoupled from tool feedback, workspace state, evidence, or verifiable output contracts. Harness-Bench provides a reproducible foundation for diagnosing and improving reliable, efficient, and auditable agent execution stacks.

Yilun Yao, Xinyu Tan, Chao-Hsuan Liu +9

1Peking University · 2Qiyuan Tech

Agent Harness Agentic Benchmarks Agentic Workflow Design

Ask PDF