Serving Agentic Robot Fleets: Workload Characterization, KV-Cache Thrashing, and Admission Control

Sida He

cs.RONov 1, 2026

Serving Agentic Robot Fleets: Workload Characterization, KV-Cache Thrashing, and Admission Control

Authors: Sida He

Organizations: Huawei Inc., China

Abstract

Agentic robots combine a vision-language brain, a tool-using harness, and a vision-language-action (VLA) policy. Serving a fleet of these agents introduces two coupled resource constraints: the brain repeatedly revisits long, growing key-value (KV) cache states, while the VLA executes action chunks under tight control deadlines. We characterize 80 LIBERO-PRO episodes generated by GPT-6-Astra, RPent, and pi0.5 (1,862 brain turns and 1,582 VLA inference chunks), profile both layers on H800 graphics processing units (GPUs), and replay the resulting agent programs on a vLLM serving engine. Brain contexts reach 215k tokens; the median Astra request takes 7.78 s, while the intervening environment-step gap is only 1.0 s. Under a 0.25 s action-chunk deadline, one VLA GPU meets the 1% starvation target for one robot; under a 1.0 s deadline, its measured capacity reaches at least 32. In sustained brain replay, increasing concurrency from 20 to 32 robots reduces throughput from about 2.4 to 0.6 requests/s as repetitive prefill displaces generation. A static active-episode cap restores 2.24-2.25 requests/s and raises completed trace-program throughput from 1.30 to 5.00-5.55 episodes/min. Our measurements expose the workload-dependent shift between action deadlines and brain KV residency, and establish practical principles for capacity planning and admission in sustained agentic robot fleets.

Figures & tables

	Robot	Agent KV	Sustained	Brain+VLA
	deadlines	residency	fleet replay	analysis
ROSA [ 2 ]	$\checkmark$	–	$\checkmark$	pipeline
CONCUR [ 6 ]	–	$\checkmark$	batch jobs	–
CacheWise [ 7 ]	–	$\checkmark$	agent traces	–
EfficientAgent [ 8 ]	–	$\checkmark$	agent traces	–
This work	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Table 1: Coverage of relevant serving research. Columns indicate robotic deadlines, agent key–value (KV) residency, sustained fleet replay, and analysis of both serving layers.

Panel of Figure 1 — Figure 1: The embodied-agent program issues growing-context requests to the brain and action-chunk requests to the vision–language–action (VLA) service. Both layers are profiled on graphics processing units (GPUs); dependent brain requests are replayed on vLLM.

Trace timing quantity	Median	P90
Astra request latency	7.78	13.84
Environment-step gap	1.00	3.10
Simulator tool wall gap	8.40	22.20
Astra + environment gap	9.40	15.38
Astra + simulator wall gap	17.96	32.77

Table 2: Timing of the 80-episode trace. Medians and 90th percentiles (P90) are in seconds. A paired response interval sums each logged Astra call with the preceding tool gap.

Hardware	8 $\times$ H800 80 GB; brain study on GPUs 4–7
Brain proxy	Qwen3.6-27B, BF16, TP=4, vLLM 0.17.1
Runtime	NVIDIA 535.161.08 (driver CUDA 12.3); PyTorch 2.10.0+cu128 (user-space CUDA 12.8)
KV cache	Enabled; hybrid blocks of 784 tokens, with observed pools of 4,270 and 4,396 blocks
Trace brain	GPT-6-Astra, low effort, RPent/no-SAM3
Action model	$\pi_{0.5}$ ; five denoising steps, five-step chunks
Robot control	20 Hz base; evaluated chunk budgets 0.25, 0.50, and 1.00 s

Table 3: Primary serving configuration. BF16 denotes bfloat16, and TP denotes tensor parallelism. The CUDA version reported by nvidia-smi describes driver support; PyTorch specifies the user-space runtime.

Suite	Success	Brain turns	Peak context	Total input	VLA chunks	Environment progression (s)
object_swap	19/20	17.0	89.5k	1.07M	8	16.1
spatial_swap	20/20	18.5	91.1k	1.27M	8	14.9
goal_swap	19/20	19.0	95.5k	1.32M	15	21.8
10_swap (long)	16/20	33.5	136.3k	3.05M	27	44.0
All	74/80	19.0	95k	1.31M	12.5	19.5

Table 4: The 80-episode RPent/Astra/

\pi_{0.5}

workload. Statistics except success are per-episode medians. Input tokens are summed across an episode’s brain turns; robot execution time is environment steps divided by 20 Hz.

Figure 2: Within-episode tool gaps between brain turns. The empirical cumulative distribution functions (ECDFs) show positive environment-step gaps and simulator wall-time gaps; 38.6% of environment gaps advance zero steps. The plotted gap excludes the brain’s own request latency.

Figure 3: Representative episode timeline using environment steps as a common horizontal axis. Brain turns and VLA inference chunks form separate, task-driven event streams.

Figure 4: Input context length over brain turns, with per-suite medians and interquartile ranges among episodes reaching each turn.

Figure 5: Per-turn incremental input length and the full-prompt/incremental-token ratio. These distributions quantify how much prefill work retained key–value cache state can save.

Figure 6: Measured

\pi_{0.5}

inference latency by batch size. The seven plotted sizes are directly measured medians over 50 runs each, preceded by 10 warm-up runs per shape.

Chunk deadline	1 GPU	2 GPUs
0.25 s (20 Hz)	1	no passing test at $N\geq 8$
0.50 s (10 Hz)	8	16
1.00 s (5 Hz)	$\geq 32$	$\geq 32$

Table 5: Deadline-qualified VLA capacity under a 1% starvation service-level objective (SLO), with five-step prefetch and zero round-trip time (RTT). Capacity is the largest tested robot count passing the

\leq 1\%

starvation target in both repetitions. One-GPU tests cover

N\in\{1,2,4,8,16,32\}

; two-GPU tests cover

N\in\{8,16,32\}

Figure 7: Control-tick starvation as robot count grows. Each point represents the worse of two runs; dashed horizontal lines mark the 1% service-level objective. Each panel uses its labeled chunk deadline and replica configuration.

Context	40k	78k	130k	215k
TTFT, evicted	1.34	2.85	5.32	10.26
TTFT, retained	0.18	0.22	0.27	0.35
Call, evicted	2.66	4.23	6.78	11.86
Call, retained	1.50	1.61	1.73	1.95

Table 6: Brain proxy cost at tensor parallelism four. Time to first token (TTFT) is measured; a 150-output-token call combines TTFT with 149 measured batch-one decode intervals. The retained-prefix case adds 3.3k tokens. All entries are seconds.

Figure 8: Estimated 150-token brain call duration from measured TTFT and batch-one decoding costs, under retained and evicted KV prefixes.

Figure 9: Full-prefix survival under cyclic, ordered revisits. Batch priming and sequential priming are shown separately, and each curve identifies the underlying KV-cache block pool.

Figure 10: Sustained-fleet request throughput and full-prefix hit rate. Circles represent the historical 20-minute sweep across 8–48 virtual robots; crosses represent 30-minute replication runs. The two series retain their original durations and seed provenance.

Figure 11: Kaplan–Meier survival estimate for the first 120-second window with full-prefix hit rate below 80%. The

N=24

group has four events and two right-censored runs;

N=28

has two observed events.

Figure 12: Full-prefix hit rate (120-second windows) and vLLM token computation (60-second windows) for healthy

N=20

and thrashing

N=32

runs. Prompt computation rises while useful generated-token throughput declines.

Figure 13: Completed fixed-trace robot programs per minute and Jain’s fairness index across all target robots. Each horizontal bar is one independent run; A and B distinguish separate runs at the same setting. Fully replayed episodes count toward throughput, while unfinished instances are censored.

Cap $K$	Requests/s	Full-hit (%)	Runs
12	2.027	100.00	1
16	2.269	100.00	1
20	2.377	99.95	2
24	2.132	98.01	1

Table 7: Static active-episode admission for 48 virtual robots. The

K=20

results average two valid runs; other caps each have one valid run. Full-hit percentages exclude the first brain call of each episode.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Suite	Turns	Env. time (s)	Turns/s env	Astra (s)	Paired (s)
Object	336	332.80	1.010	8.54	10.21
Spatial	385	337.40	1.141	7.33	8.59
Goal	417	508.35	0.820	8.55	10.44
Long	724	1130.50	0.640	7.22	9.05
All	1862	2309.05	0.806	7.78	9.40

Appendix

Table 8: Trace-time normalization by suite. The paired median is calculated over within-episode intervals by adding each recorded Astra request latency to its preceding environment-step gap.

LLM Agent Serving

Large language model agent serving runs tool-using, multi-step agents efficiently by coordinating inference, memory, and workflow execution.

See all 39 papers

Oct 4, 2026

Beyond LLM Serving: Characterizing Vision-Language-Action Workloads for Embodied AI System Design

Vision-language-action (VLA) models translate multimodal observations into low-level robot actions. During robot operation, each control period sets an inference deadline, and overruns leave the robot acting on stale observations, reducing task success. Meeting this deadline motivates on-device or nearby edge execution, where a single robot requires batch-1 inference outside the design point of LLM serving systems. Although VLA architectures combine familiar vision-language, autoregressive, and diffusion-style components, their runtime behavior in this batch-1 control setting remains uncharacterized. We characterize four representative VLA models on an edge GPU server and two onboard SoCs, using single-inference profiling and 43,200 closed-loop episodes. Action tensor dimensionality determines whether a stage is memory- or compute-bound, platform balance can shift that bottleneck, and GPU frequency scaling yields a platform-dependent energy-latency sweet spot. In closed-loop operation, overlapping inference with action execution creates an accuracy-speed-energy tradeoff, and no configuration is Pareto-dominant across deployment SLOs. These results guide joint design of VLA model architectures, hardware, and runtime policies.

Seonghun Jung, Sieun Moon, Jiyoung Jeong +2

KAIST, Daejeon, Republic of Korea

Efficient VLA Model Inference Vision-Language-Action Models VLMs for Robotics

Ask PDF

Sep 14, 2026

AgentKV: Phase-Aware KV Eviction for Agentic LLMs

Agentic serving can consume orders of magnitude more tokens than chatbot workloads, stressing both KV-cache capacity and decode-time bandwidth. Most KV-eviction methods score cached keys against representative queries drawn from the most recent tokens, assuming future attention resembles recent attention. We show that agentic generation violates this assumption: future queries form a mixture over think, act, tool, and others phases, and principal-angle analysis shows these components occupy measurably different query subspaces, so recency representatives systematically undervalue keys that upcoming phases will need. We propose AGENTKV, which maintains a small query buffer per phase and scores cached keys against their union. We further implement AGENTKV in a persistent multi-turn serving path that carries compressed KV state across turns and compacts retained KV pages online. Across two models, six task domains, and three KV budgets each, AGENTKV improves task score by 5.5 points on average over R-KV and 5.3 over Tri-attention. Relative to upstream full-KV SGLang, AGENTKV improves output-token throughput by up to 1.80x. Code: https://github.com/LiuTaowen-Tony/agentkv.

Taowen Tony Liu, Jeffrey T. H. Wong, Can Xiao +3

Imperial College London, London, United Kingdom · Columbia University, New York, NY, USA

KV Caching KV-Cache Eviction LLM Inference Acceleration LLM Agent Serving LLM Agents

Ask PDF

Jul 31, 2026

Action Chunk Scheduling for Batched Robot Policy Serving

Deploying robot foundation models at scale is the next step towards realizing the potential of general-purpose robots. However, Vision-Language-Action (VLA) and other foundation models are computationally demanding, and on-device compute is constrained by power and space. In this paper, we introduce the problem of serving a robot policy to multiple robots from a remote GPU and formulate it as a scheduling problem. We build Armory, a serving system validated on fleets of both simulated and real robots. Our experiments show that naive scheduling heuristics perform well when all robots are the same, but fall short when robots consume action chunks at different rates, uncovering a mismatch between conventional batching methods and the closed-loop requirements of robot policy execution. To address this, we propose a scheduling algorithm that accounts for this heterogeneity and improves overall system throughput by up to

18\%

in real-world experiments. Additional details are available at https://gatech-rl2.github.io/actionchunkscheduling.

Rohan Bansal, David He, Nadun Ranawaka Arachchige +4

Georgia Institute of Technology

Multi-Robot Systems Robot Foundation Models Efficient VLA Model Inference

Ask PDF

Abstract

LLM Agent Serving

Explore similar work

Beyond LLM Serving: Characterizing Vision-Language-Action Workloads for Embodied AI System Design

AgentKV: Phase-Aware KV Eviction for Agentic LLMs

Action Chunk Scheduling for Batched Robot Policy Serving