KerColle: Unlocking Fine-Grained GPU Concurrency in Vision-Language-Action Models

Anna Li; Christina Giannoula; Nandita Vijaykumar

cs.ARSep 16, 2026

KerColle: Unlocking Fine-Grained GPU Concurrency in Vision-Language-Action Models

Authors: Anna Li, Christina Giannoula, Nandita Vijaykumar

Organizations: University of Toronto · Max Planck Institute for Software Systems (MPI-SWS)

Abstract

Vision-Language-Action (VLA) models have emerged as foundational models for next-generation robotics. High VLA inference throughput is critical for meeting the control-rate requirements of robots. VLA models comprise two phases, a vision-language model (VLM) and an action head, that can be decoupled and executed asynchronously and concurrently across independent robot requests. Through a detailed characterization of four state-of-the-art VLAs, we observe that GPUs are severely underutilized in VLA inference as the Cooperative Thread Array (CTA) scheduler of GPU is unable to fully overlap the two independent phases of VLA execution. We identify that this inefficiency is caused by head-of-line blocking in the hardware thread-block dispatcher. We demonstrate that prior scheduling frameworks do not address the challenges posed by VLA concurrency. First, the independent phases across different robot requests each comprise numerous kernels, and at any given time, there are different combinations of kernels that are executed in parallel. This makes static or ahead-of-time scheduling policies largely ineffective. Second, many of the action-head operators are short-running kernels and there are numerous such kernels. This leaves no headroom for online profiling or preemption-based mechanisms. To address these challenges, we present KerColle, a lightweight GPU scheduling framework that leverages online Streaming Multiprocessor (SM) utilization and individual kernel resource requirements to intelligently and dynamically co-schedule kernels to efficiently overlap the two phases of execution by (1) mitigating head-of-line blocking, and (2) co-scheduling kernels with complementary resource requirements. We demonstrate in simulation, across two GPU architectures, for 4 state-of-the-art VLA models, that KerColle delivers throughput gains of up to $28\%$ .

Figures & tables

Figure 1. Representative VLA architecture decomposed into two phases.

Figure 2. Execution timelines for action chunking and asynchronous inference.

Figure 3. High-level overview of Streaming Multiprocessor (SM) and processing cores on modern GPUs.

Figure 4. CDF of inference time vs. SM throughput for a single-image inference on the RTX 4090.

Figure 5. Resource utilization profiles of the VLM and Action Head phases for CogACT (top) and

\pi 0.5

(bottom) across RTX 4090, H100, and RTX 5090.

Figure 6. Roofline of VLA kernels on RTX 4090. Large markers are time-weighted centroids.

Figure 7. Execution patterns for asynchronous VLA inference. (a) Sequential execution serializes the two phases. (b) Naive two-stream overlap leaves the action stream lagging behind. (c) Ideal overlap fully hides action-head latency behind the next frame’s VLM.

Figure 8. (a) Default GPU scheduling. Head-of-line blocking behavior. (b) Ideal co-scheduling. Fully overlapped execution within an SM.

Workload	Median	$<$ 5K	$<$ 10K	$<$ 25K	$<$ 50K
	(cyc)	(%)	(%)	(%)	(%)
CogACT ( Li et al., 2024 )	7,156	18.9	67.2	88.9	95.2
Octo ( Team et al., 2024 )	6,832	44.3	69.8	85.4	100.0
$\pi_{0.5}$ ( Intelligence et al., 2025 )	9,348	17.9	51.7	73.7	90.2
GRASP-VLA ( Deng et al., 2025 )	12,407	23.9	46.3	77.6	88.4

Table 1. Execution profile of micro-kernels across different VLA workloads after applying a CUDA Graph capture.

Scheduler	Mechanism	Unit Aware	Micro-K. Resp.	Black- Box
SMK ( Wang et al., 2016b )	HW Ctx Switch	$\times$	$\times$	$\checkmark$
Warped-Slicer ( Xu et al., 2016 )	Epoch + Static	$\times$	$\times$	$\checkmark$
Maestro ( Park et al., 2017 )	Epoch + 2-Stack	$\times$	$\times$	$\checkmark$
GPUPool ( Tan et al., 2023 )	ML + TB/SM Cap	$\checkmark$	$\times$	$\checkmark$
Kitsune ( Davies et al., 2025 )	Dataflow + L2 Q	$\checkmark$	$\checkmark$	$\times$
KerColle (Ours)	Dynamic Score	$\checkmark$	$\checkmark$	$\checkmark$

Table 2. Qualitative comparison of prior GPU co-scheduling frameworks

Figure 9. High-level overview of KerColle design.

Field	Description
$\tau_{TC}$	Tensor Core (TC) pipeline intensity. Fraction of peak TC throughput that CTAs consume.
$\tau_{ALU}$	ALU pipeline intensity. Fraction of peak ALU (CUDA Core) throughput consumed per CTA.
$\tau_{MEM}$	Memory bandwidth intensity. Fraction of peak DRAM bandwidth consumed per CTA.

Table 3. A kernel’s Intensity Descriptor. Each field (4 bits) encodes a normalized intensity level in the range 0–15.

Register	Description
$U_{TC}[j]$	Estimated Tensor Core utilization on SM $j$ .
$U_{ALU}[j]$	Estimated ALU pipeline utilization on SM $j$ .
$U_{MEM}[j]$	Estimated memory bandwidth utilization on SM $j$ .

Table 4. Per-SM 8-bit utilization registers (range

[0,255]

) maintained by the scheduler.

Parameter	NVIDIA RTX 4090	NVIDIA RTX 5090
Generation	Ada Lovelace	Blackwell
SM Clock Frequency	2520 MHz	2407 MHz
SMs/GPU	128	170
Warp Schedulers/SM	4	4
Warp Scheduling Policy	GTO	GTO
Max #Thread Blocks/SM	24	32

Table 5. GPU Microarchitectural Parameters in Accel-Sim

Figure 10. Left: VLA inference throughput on RTX 4090. Right: Percentage increase in throughput gain of KerColle and Warped-Slicer relative to Baseline.

Figure 11. Left: VLA inference throughput on RTX 5090. Right: Throughput gain of KerColle and Warped-Slicer relative to Baseline.

Figure 12. Concurrent SM utilization under the baseline scheduler and KerColle across all models and GPUs (left y-axis), and KerColle throughput gain relative to baseline (right y-axis).

Figure 13. Left: Distribution of concurrent kernel pairs grouped by the smaller kernel’s thread block count. Right: Throughput gains of KerColle over baseline for four models correlated to the median thread block count of each model’s smaller kernels.

Figure 14. Warp occupancy of different KerColle configurations and Warped-Slicer on RTX 4090.

Figure 15. Warp occupancy of different KerColle configurations and Warped-Slicer on RTX 5090.

Efficient VLA Model Inference

Efficient vision-language-action (VLA) model inference reduces robot-control latency, memory use, and energy while preserving task performance.

See all 180 papers

Characterizing Vision-Language-Action Models across XPUs: Constraints and Acceleration for On-Robot Deployment
Apr 27, 2026Kaijun Zhou, Qiwei Chen, Da Peng +3AI Accelerator Inference Edge Inference
School of Computer Science, Shanghai Jiao Tong University
vla.cpp: A Unified Inference Runtime for Vision-Language-Action Models
Jun 6, 2026Khanh D. Nguyen, Hung T. Ho, Chinh T. Nguyen +5AI Accelerator Inference Efficient VLM Inference
VinRobotics, Vietnam · Max Planck Research School for Intelligent Systems, Germany · University of Stuttgart, Germany +3
Beyond LLM Serving: Characterizing Vision-Language-Action Workloads for Embodied AI System Design
Oct 4, 2026Seonghun Jung, Sieun Moon, Jiyoung Jeong +2Efficient VLA Model Inference Vision-Language-Action Models
KAIST, Daejeon, Republic of Korea

Apr 27, 2026J/K move · Enter open · S save

Characterizing Vision-Language-Action Models across XPUs: Constraints and Acceleration for On-Robot Deployment

Kaijun Zhou, Qiwei Chen, Da Peng, Zhiyang Li+2

School of Computer Science, Shanghai Jiao Tong University

AI Accelerator Inference Edge Inference Efficient VLA Model Inference Vision-Language-Action Models VLMs for Robotics Diffusion Model Inference Acceleration+1

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models are promising for generalist robot control, but on-robot deployment is bottlenecked by real-time inference under tight cost and energy budgets. Most prior evaluations rely on desktop-grade GPUs, obscuring the trade-offs and opportunities offered by heterogeneous edge accelerators (GPUs/XPUs/NPUs). We present a systematic analysis for low-cost VLA deployment via model-hardware co-characterization. First, we build a cross-accelerator leaderboard and evaluate model-hardware pairs under CET (Cost, Energy, Time), showing that right-sized edge devices can be more cost-/energy-efficient than flagship GPUs while meeting control-rate constraints. Second, using in-depth profiling, we uncover a consistent two-phase inference pattern: a compute-bound VLM backbone followed by a memory-bound Action Expert, which induces phase-dependent underutilization and hardware inefficiency. Finally, guided by these insights, we propose DP-Cache and V-AEFusion to reduce diffusion redundancy and enable asynchronous pipeline parallelism, achieving up to 2.9x speedup on GPUs and 6x on edge NPUs with only marginal success degradation. The example leaderboard website is available at: https://vla-leaderboard-01.vercel.app/.

Abstract

Efficient VLA Model Inference

Explore similar work

Characterizing Vision-Language-Action Models across XPUs: Constraints and Acceleration for On-Robot Deployment

vla.cpp: A Unified Inference Runtime for Vision-Language-Action Models

Beyond LLM Serving: Characterizing Vision-Language-Action Workloads for Embodied AI System Design