FlashRT-PAI: Fast Kernels Are Not Fast Pipelines for Physical AI Inference

Liang Su; David Chen; Jason Li; Kevin Guo; Teresa Lv

cs.AIOct 17, 2026

FlashRT-PAI: Fast Kernels Are Not Fast Pipelines for Physical AI Inference

Authors: Liang Su, David Chen, Jason Li, Kevin Guo, Teresa Lv

Organizations: AnyDevice AI & Alps Lab

Abstract

Vision-language-action (VLA), video-action (VA), and world action models (WAMs) form the backbone of modern physical AI systems, yet their real-time single-batch inference remains a defining bottleneck. Latency is dominated not by individual kernel computation, but by system-level overhead: thousands of short kernel launches, intermediate data movement across operator boundaries, and redundant recomputation across consecutive decisions. General-purpose graph compilers cannot address these costs, as they optimize individual calls in isolation and lack visibility into cross-call state semantics and global pipeline structure. We present FLASHRT-PAI, a low-latency inference engine built around model-hardware pipeline co-design rather than isolated kernel optimization. FLASHRT-PAI combines three core mechanisms: deep kernel fusion with launch overlap eliminates inter-operator boundaries and launch gaps; an execution contract formalizes the pipeline's stage DAG, update rates, and named buffers to enable cross-decision state reuse and selective refresh; and execution-state capsules enable fast snapshot, restore, and fork of persistent pipeline state. The system also supports drop-in acceleration of unmodified PyTorch models via parity- and latency-gated kernel replacement. Evaluation covers six policies on RTX 5090 and four on Jetson AGX Thor. At matched 16-bit precision, FLASHRT-PAI is 1.3-3.7x faster than official compiled paths. Combining pipeline co-design with reduced precision delivers 3.38-12.66x speedup over official eager implementations. In LIBERO-10, four-bit pi0.5 achieves 94.04% success on Thor and 93.32% on RTX 5090, compared with 92.4% for the measured official FP16 reference. FLASHRT-PAI is open source, distributed via Hugging Face Kernel Hub, and its opt-in pi0.5 kernels have been adopted into the LeRobot framework.

Figures & tables

Panel of Figure 1 — Figure 1: One decision of (a) a VLA, (b) a VA, and (c) a WAM, with work grouped by how often it has to run. Results of the upper rows are kept in static buffers and state regions that the per-decision stage only reads; inputs and outputs pass through swap windows, buffers whose contents are replaced without touching the captured graphs. The rows are the same for all three kinds of policy; what differs is the cost of a denoising step and the state: the VA keeps a key/value cache over earlier chunks that grows across decisions. A monolithic graph replayed unchanged reruns the per-prompt row at every decision. Dashed outlines mark captured CUDA Graphs. FlashRT-PAI records this structure in the execution contract (§ 3.4 ).

	ms	%
Kernel execution	172.50	77.48
of which, kernels between GEMMs	70.10	31.49
Gaps between kernels (estimated)	50.14	22.52
Total	222.64	100

Table 1: Estimated time breakdown of one

\pi_{0.5}

decision with the official OpenPI eager implementation on Jetson AGX Thor. Kernels between GEMMs are the small kernels that normalize, add residuals, change layouts, or convert precision; gaps are the time when no kernel runs. Because profiling inflates the gaps, we estimate them as the unprofiled total latency minus the measured kernel time.

Panel of Figure 2 — Figure 2: Building and running a FlashRT-PAI model–hardware pipeline. Setup constructs the pipeline from a checkpoint, chooses stage boundaries and precision, validates the model, and captures the selected stages. The execution contract records the stage DAG, named buffers, and export interface. At each control tick, the host fills swap windows and reads the action chunk while the session manager schedules replay over frozen execution state and can copy declared state for capsules.

Panel of Figure 3 — Figure 3: Kernel timelines for one $\pi_{0.5}$ action-expert decoder layer on Jetson AGX Thor (10 action rows), drawn on one time axis. The upper rows compare the PyTorch FP8 baseline with FlashRT-PAI ’s FP8 and NVFP4 pipelines. Bars show measured CUPTI traces under CUDA Graph replay, and hatching marks kernel boundaries. The bottom pair holds the 11-kernel NVFP4 path fixed and enables programmatic dependent launch (PDL) at the FFN and adjacent O/QKV boundaries; green marks overlap. Separately, the PyTorch step recomputes 37 AdaRMS modulation projections in a 1.87 ms FP32-weight kernel; FlashRT-PAI prepares them once per prompt.

Panel of Figure 4 — Figure 4: Fixed-buffer reuse in $\pi_{0.5}$ . (a) Circled numbers link stages to their buffer accesses; yellow K/V is shared by the denoising steps within a call. (b) The next call processes a fresh image and rebuilds K/V, while reusing the unchanged instruction’s embeddings and fixed step-conditioning tables. The connector carries these prepared values across the call boundary.

Class	Model	Params	Steps	Actions	Views
VLA	$\pi_{0.5}$	3.35B	10	10	2
	GR00T N1.7	3B	4	16	1/2
VA	LingBot-VA	5.3B	20/50	16	2
WAM	FLUX 3 Action	7B	4+CFG	32	3
	Cosmos3-Edge-Policy	4B	4	32	3
	Motus	$\sim$ 8B	10	16	3

Table 2: Policies evaluated. Steps: denoising steps per decision (video/action for LingBot-VA); actions: actions returned per decision; views: cameras (Thor/RTX 5090 for GR00T N1.7).

		Jetson AGX Thor					RTX 5090
Class	Model	Eager	Compile	TRT	FlashRT-PAI	Speedup	Eager	Compile	TRT	FlashRT-PAI	Speedup
VLA	$\pi_{0.5}$	222.64	129.17	$\sim 49^{\dagger}$	20.33	10.95	104.21	58.26	–	8.23	12.66
	GR00T N1.7	$\sim 126^{\dagger}$	$\sim 105^{\dagger}$	$39.9^{\dagger}$	24.39	$\sim 5.2^{\dagger}$	41.10	32.01	–	7.67	5.36
VA	LingBot-VA	9309.53	6721.44	–	920.9	10.11	2608.53	2666.62	–	455.70	5.72
WAM	FLUX 3 Action	–	–	–	–	–	1353.57	1148.95	–	275.51	4.91
	Cosmos3-Edge-Policy	1223.00	1263.49	–	362.15	3.38	586.28	514.93	–	121.25	4.84

Table 3: Reported deployment latency (ms). Speedup is eager/ FlashRT-PAI for each model and GPU, combining pipeline and precision changes.

\dagger

marks published values and ratios computed from them; other latencies are measured locally. –: no result. Timing intervals: Appendix A.2 .

	$\pi_{0.5}$		GR00T N1.7		LingBot-VA		FLUX 3 Action
Configuration	Thor	5090	Thor	5090	Thor	5090	5090
Official compile, 16-bit	129.17	58.26	$\sim$ 105	32.01	6721.44	2666.62	1148.95
FlashRT-PAI 16-bit	67.47	25.08	54.77	18.85	3174.26	726.10	900.34
FlashRT-PAI 8-bit	40.71	13.85	36.98	12.91	2029.42	532.69	423.92
FlashRT-PAI 4-bit	20.33	8.23	24.39	7.67	920.9	455.70	275.51
16-bit vs. 16-bit compile	1.91 $\times$	2.32 $\times$	$\sim$ 1.9 $\times$	1.70 $\times$	2.12 $\times$	3.67 $\times$	1.28 $\times$

Table 4: Latency (ms) across pipeline configurations at 16, 8, and 4 bits. The 16-bit comparison matches FP16/BF16 precision; lower tiers change precision and kernels. Baseline sources follow Table 3 .

Policy	GPU	Eager	FlashRT-PAI	Reduction
$\pi_{0.5}$	Thor	13,414	2,610	81%
	RTX 5090	13,134	2,216	83%
GR00T N1.7	Thor	5,190	2,148	59%
	RTX 5090	4,978	1,579	68%
LingBot-VA	Thor	180,245	27,089	85%
	RTX 5090	220,504	35,557	84%

Table 5: GPU kernels per decision of the official eager implementation and FlashRT-PAI , counting every kernel inside CUDA Graphs.

Capsule copy cost		MB	Snapshot	Restore	Fork
Use	Policy/GPU	Baseline	Frozen state	Speedup
Reuse values	$\pi_{0.5}$ /Thor	37.05	21.61	1.71 $\times$
Resume after overwrite	$\pi_{0.5}$ /Thor	21.48	9.73	2.21 $\times$
Resume after overwrite	LingBot-VA/Thor	1809.32	925.27	1.96 $\times$
$\pi_{0.5}$	Thor	14.40	0.48	0.13	0.50
LingBot-VA	Thor	1410.65	18.77	11.29	36.03

Table 6: Benefits of frozen execution state (ms): reusing values across decisions and resuming a session after another session overwrites its buffers, with the cost of capsule copies. Restored and reused actions match the reference bit for bit. Paired experiment protocol: Appendix A.2 .

Panel of Figure 5 — Figure 5: LingBot-VA on Thor: time until the history state is ready again after another session overwrites it (log scale, median of 10 runs). Generating the next chunk (0.78–0.92 s) is not included. Restore uses the same state-ready interval as Table 6 : 11.29 ms at 16 chunks.

Class	Model	Thor	RTX 5090
VLA	$\pi_{0.5}$	.999713 / .999360	.999850 / .999201 d
	GR00T N1.7	.9997 ( $n{=}1$ )	.99999 d ( $n{=}1$ )
VA	LingBot-VA, BF16	.999949 / .999920	.999502 / .952046 c
	LingBot-VA, 4-bit mix	.995361 / .993777	.998907 / .941478 c
WAM	FLUX 3 Action	–	.999850 / .998939
	Cosmos3-Edge-Policy	.998533 / .976768	.998138 / .988953

Table 7: Action agreement: global / worst-sample cosine under matched noise.

n

marks per-input scores; Motus gives the range over four inputs. c : six arm channels, 717 chunks, official FP32 reference; full actions: Appendix A.2.1 . d : FP16 frontend reference. Bottom: LIBERO-10 success (100 matched episodes for LingBot-VA). Protocols: Appendix A.2 .

Project	How it uses FlashRT-PAI
MindOn Mind-1 [link]	Inference backend for a robot model
EagleVLA-Edge [link]	Edge VLA engine that plugs into FlashRT-PAI
LeRobot [link]	Optional FP8 kernels for $\pi_{0.5}$
FineART-VLA [link]	Optional kernels for training and inference
Jetson AI Lab [link]	TensorRT guide for $\pi_{0.5}$ , inspired by FlashRT-PAI
APXInf [link]	Edge inference engine that credits FlashRT-PAI

Table 8: Public uses of FlashRT-PAI as of October 2026. Links beside project names open the corresponding public sources.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Buffer	Role / writer	Refresh event
Images	Swap / host	Fresh observation
Noise	Swap / host	New action generation
Prompt embedding	Static / preparation	Prompt change
Step conditioning	Static / preparation	Schedule change
Context K/V	State / context	Image or prompt change
Actions	Swap / action	Action replay

Appendix

Table 9: Buffer roles and update events in the split

\pi_{0.5}

plan.

Interval	Samples	p50	p95
Deployed $\pi_{0.5}$ infer	1000	20.33	20.54
PDL off, decoder	200	10.124	10.137
PDL FFN + O/QKV, decoder	200	9.207	9.220
PDL all, decoder	200	9.009	9.024
Rebuild + action	64	21.484	21.533
Restore + action	64	9.726	9.778

Appendix

Table 10: Short-run Thor latency distributions (ms). Rows retain their own timing intervals; decoder p95 is not whole-call p95.

Policy	Recorded validation
$\pi_{0.5}$	Recorded-input action comparisons; LIBERO-10 rollouts against the measured FP16 reference.
GR00T N1.7	Single-view Thor and two-view RTX 5090 LIBERO observations; single-observation cosine.
LingBot-VA	RTX 5090: held-out LIBERO traces for arm/full-action agreement; paired 100-episode LIBERO-10 success comparison.
FLUX 3 Action	Returned-action agreement against the official implementation under matched inputs and noise.
Cosmos3-Edge-Policy	DROID observations: 114 observation/noise requests on Thor and 60 on RTX 5090.
Motus	Four RoboTwin2 input bundles; action and decoded-video agreement against saved upstream outputs.

Appendix

Table 11: Recorded validation by policy.

Configuration	Six arm channels	Full action
BF16	.999502 / .952046	.997388 / .643595
FP8/INT4 mix	.998907 / .941478	.994571 / .405055

Appendix

Table 12: Unfiltered LingBot-VA/RTX 5090 action agreement against official FP32. Entries are global / worst-chunk cosine. The mixed configuration matches the 455.70 ms deployment.

Input	Action cosine	Video cosine
sample_00	.999929	.999117
sample_01	.999935	.998644
sample_02	.999921	.999144
sample_03	.999913	.998844

Appendix

Table 13: Motus agreement on four RoboTwin2 inputs, compared with saved upstream outputs. The evaluated configuration retains all ten denoising steps.

Efficient Neural Network Inference

Efficient neural network inference reduces the time, memory, and energy needed to run trained models across hardware and deployment settings.

See all 471 papers

PhyAI: Real-Time Physical AI at the Edge, Scalable Rollouts in the Cloud
Aug 4, 2026Chenghua Wang, Daliang Xu, Dongqi Cai +21Efficient VLA Model Inference On-Device Inference
1Beijing University of Posts and Telecommunications · 2Nanjing University · 6ModelBest +3
VLA-HWBench: Where the Time Goes When VLA Models Run on Edge Hardware
Oct 26, 2026Borui Xu, Hongshi Tan, Wenqi Jiang +4Efficient VLA Model Inference On-Device Inference
National University of Singapore
vla.cpp: A Unified Inference Runtime for Vision-Language-Action Models
Jun 6, 2026Khanh D. Nguyen, Hung T. Ho, Chinh T. Nguyen +5AI Accelerator Inference Efficient VLM Inference
VinRobotics, Vietnam · Max Planck Research School for Intelligent Systems, Germany · University of Stuttgart, Germany +3

Aug 4, 2026J/K move · Enter open · S save

PhyAI: Real-Time Physical AI at the Edge, Scalable Rollouts in the Cloud

Chenghua Wang, Daliang Xu, Dongqi Cai, Duojin Sun+20

1Beijing University of Posts and Telecommunications · 2Nanjing University · 6ModelBest+3

Efficient VLA Model Inference On-Device Inference

Open paper AskPDF arXiv

Physical AI policies require inference throughout their lifecycle, including model evaluation, cloud reinforcement learning rollout, edge GPU serving, and onboard deployment. Although these settings share the same checkpoint and action semantics, they often rely on separate inference programs. To unify them, we build PhyAI, a Physical AI inference engine with a single runtime that keeps architecture-specific conditioning, solver, cache, and output logic in model adapters while sharing graph execution, kernels, memory management, and parallel services. The same codebase runs vision-language-action (VLA) models and world-action models (WAMs) on single or multiple GPUs across onboard, edge, and cloud deployments. We used the adapter interface to add MiniCPM-Robot on the day of its release. PhyAI achieves 1.40x-4.65x speedups over the official implementations of pi0, pi0.5, GR00T N1.7, and MiniCPM-Robot. On Cosmos3-Nano-Policy-DROID it reduces latency from 2.46 to 1.18 s on eight H20 GPUs (CFG=2, TP=4), a 2.08x speedup. Specialized runtimes remain faster in several configurations, so our goal is one runtime with competitive latency rather than the fastest result in every case. Detailed profiles reveal why different models need different execution policies: on a Hopper-series GPU at batch size one, the pi0.5 action expert accounts for 8.8% of FLOPs but 57.2% of latency; at batch size 32 its share drops to 13.5% and throughput reaches about 100 samples/s. Cosmos3 remains generation-dominated and gains only 14.3% throughput as batch size increases from 1 to 16. We further introduce the control-time Roofline, which distinguishes inference-bound from environment-bound control; the measured pi0.5 points on four LIBERO suites are environment-bound while Cosmos3 stays inference-bound. Code and benchmarks: https://github.com/mingti-org/phyai.

Abstract

Efficient Neural Network Inference

Explore similar work

PhyAI: Real-Time Physical AI at the Edge, Scalable Rollouts in the Cloud

VLA-HWBench: Where the Time Goes When VLA Models Run on Edge Hardware

vla.cpp: A Unified Inference Runtime for Vision-Language-Action Models