SlackDrive: Reclaiming Runtime Slack for Adaptive Driving Inference

Xiaohuan Pei; Hengguang Zhou; Yuanhao Ban; Justin Cui; Jiaqi Feng; Haoyu Xie; Tao Huang; Pichao Wang

cs.AISep 23, 2026

SlackDrive: Reclaiming Runtime Slack for Adaptive Driving Inference

Authors: Xiaohuan Pei, Hengguang Zhou, Yuanhao Ban, Justin Cui, Jiaqi Feng, Haoyu Xie, Tao Huang, Pichao Wang,

Organizations: The University of Sydney · University of California, Los Angeles · Shanghai Jiao Tong University · NVIDIA ·

Abstract

Driving world-action models improve planning by coupling multimodal reasoning with future prediction, but their growing inference cost increasingly conflicts with the real-time latency requirements of vehicle control. Existing acceleration methods reduce tokens, layers, or sampling steps with policies selected prior to deployment, yet leave residual runtime variation largely unexploited after offline profiling and static scheduling on shared onboard compute. We observe that the largest admissible compute budget varies systematically with the residual runtime state, while recent realized latency provides a direct signal of the available compute slack. Motivated by this observation, we propose SlackDrive, a pre-inference compute allocator that reuses realized latency to select the compute budget of each control step before model execution. SlackDrive profiles the latency and planning utility of a small discrete budget set once, estimates online compute state from completed forwards, and selects the highest-utility budget predicted to remain within the admissible latency envelope, complementing existing profiling and resource scheduling while preserving the driving backbone and its compute actuator. On NAVSIM v2 with DriveDreamer-Policy, SlackDrive improves latency-constrained EPDMS by $21.7\%$ over the strongest baseline under a stringent latency regime, while the full-budget model and preconfigured token-pruning baselines exceed the admissible latency envelope under runtime contention.

Figures & tables

Panel of Figure 1 — Figure 1: Motivation and formulation. (a) Full-token uses maximum compute, prior efficient methods use a fixed budget, and SlackDrive adapts the budget per step. (b) The admissible budget changes with measured inference latency. SlackDrive selects the largest feasible budget before the next forward pass. (c) SlackDrive conditions world-action inference on the measured latency state $\mathbf{c}_{t-1}$ with a frozen backbone.

Figure 2: Overview of SlackDrive. (a) Visual extraction obtains dense attention cues from the vision encoder for subsequent token selection. ( b) Latency-conditional budget selection adapts the attention mask and compute budget to the current latency target, yielding different configurations under relaxed, moderate, and tight schedules. (c) Runtime-state update and next-budget allocation feeds the realized inference latency back into the runtime state, from which a utility-maximizing policy selects the feasible budget for the next step.

Method	Compute Profile			Latency-Valid NAVSIM Submetrics $\uparrow$									Quality $\uparrow$
Method	Lat. $\downarrow$	FLOPs $\downarrow$	Mem. $\downarrow$	NC	DAC	DDC	TLC	EP	TTC	LK	HC	EC	EPDMS@ $\tau_{\max}$
	ms/fr	TF	GB
Admissible latency $\tau_{\max}=180$ ms
Full-token	226	56.9	11.0	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
FastV	188	47.3	10.4	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
SparseVLM	188	47.3	10.4	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00

Table 1: Main results on NAVSIM v2 under latency constraints. All methods use the same

N=4{,}966

held-out scenes and fluctuating-load trace. EPDMS@

\tau_{\max}

follows Eq. 4 . Underlined values denote the best baseline; full-token rows are gray and EfficientDrive rows are lavender.

Panel of Figure 3 — Figure 3: Sensitivity to the latency constraint. From left to right: latency-valid utility, violation rate, gain over fixed budgets, and retained utility.

Panel of Figure 4 — Figure 4: Sensitivity to runtime conditions. From left to right: mean load, load variability, their joint effect, and six representative runtime regimes.

Panel of Figure 5 — Figure 5: Controller design. From left to right: profile density, uncertainty margin, compute-state estimator, and EWMA rate.

Figure 6: Runtime-aware allocation in practice. (a) SlackDrive converts available runtime headroom into additional compute and avoids a collision. (b) Component removal identifies the profile and runtime state that enable this adaptation.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: The cheapest configuration is not always sufficient. Shown are $24$ fixed-seed samples from $1{,}052$ scenes in which Static@135 collides while Static@432 remains collision-free with EPDMS $\geq 80$ . The comparison motivates selecting the highest-utility configuration inside the feasible set rather than always minimizing compute.

Figure 8: Population structure of static compute budgets. All $10{,}439$ navtest scenes are included. The left panel shows the per-scene EPDMS difference between Static@432 and full-token inference. The right panel reports collision outcomes for each pair of operating points. Population-level utility increases with compute, while individual scenes need not follow the same ordering.

Figure 9: Budget-dependent trajectories in the camera view. Twelve scenes are selected from the $88$ scenes with synchronized camera data according to the separation between the three four-second trajectory endpoints. Semantic inputs are identical across configurations; only $b_{t}$ changes.

Figure 10: Static-budget examples. Twenty-four scenes are uniformly sampled from all $10{,}439$ navtest scenes with a fixed random seed and without outcome-based filtering. The three trajectories frequently overlap, indicating that moderate budget reduction often preserves the resulting plan.

Figure 11: More compute does not guarantee higher per-scene utility. All $21$ scenes satisfying the selection rule are shown. Full-token inference collides, while Static@432 remains collision-free with EPDMS $\geq 80$ . These examples establish existence rather than a population-level safety trend.

Figure 12: The population trend favors additional compute. Twelve fixed-seed samples are shown from $315$ scenes in which Static@432 collides while full-token inference remains collision-free with EPDMS $\geq 80$ .

Figure 13: Representative raw sensor observations. Six observations are evenly spaced in token order from the available synchronized sensor log. The same visual input underlies all three compute configurations.

Budget	Configuration	$\bm{L_{0}(b)}$ (ms)	$\bm{U(b)}$
$b_{1}$	Static@135	54	62.88
$b_{2}$	Static@432	84	81.18
$b_{3}$	Full-token	226	88.48

Appendix

Table 2: Static compute profile used in the case analysis.

L_{0}(b)

is nominal latency and

U(b)

is mean NAVSIM-v2 EPDMS over the analyzed navtest scenes.

Budget pair	Both safe	Cheap only	Expensive only	Both collide
Static@432 vs. Full-token	9,888	352	26	149
Static@135 vs. Static@432	8,702	1,133	74	426
Static@135 vs. Full-token	8,751	1,414	29	145

Appendix

Table 3: Pairwise collision outcomes over all

10{,}439

scenes. “Cheap only” denotes an at-fault collision from the lower-compute configuration but not the higher-compute configuration; “expensive only” denotes the converse.

Inference-Time Compute Allocation

Inference-time compute allocation decides where, when, and how much computation models spend during prediction to improve quality under resource limits.

See all 82 papers

FlashDrive: Flash Vision-Language-Action Inference for Autonomous Driving
Aug 13, 2026Zekai Li, Yihao Liang, Hongfei Zhang +3Efficient VLA Model Inference Autonomous Driving
1UC San Diego · 2Princeton · 3Independent Researcher
Scene-Adaptive VLA: Efficient Autonomous Driving via Dynamic Layer Routing
Oct 17, 2026Yujia Yang, Quan Yuan, Jiawei Fan +5Efficient VLA Model Inference VLMs for Autonomous Driving
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications
Think at 5 Hz, Act at 20 Hz: Asynchronous Fast-Slow Vision-Language-Action Inference for Closed-Loop Driving
Jul 17, 2026Yun Li, Jiachen Gong, Simon Thompson +7Efficient VLA Model Inference Vision-Language-Action Models
The University of Tokyo, Tokyo, Japan · TIER IV, Inc., Tokyo, Japan · Huawei

Aug 13, 2026J/K move · Enter open · S save

FlashDrive: Flash Vision-Language-Action Inference for Autonomous Driving

Zekai Li, Yihao Liang, Hongfei Zhang, Jian Chen+2

1UC San Diego · 2Princeton · 3Independent Researcher

Efficient VLA Model Inference Autonomous Driving Vision-Language-Action Models Diffusion Model Inference Acceleration

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models promise to bring end-to-end reasoning to autonomous driving, but their computational cost remains far too high for real-time control. The core challenge is structural: VLA inference is not a single bottleneck but a cascade of four. Visual encoding wastes compute on overlapping video frames; language-model prefill recomputes context that could be carried over from the previous timestep; reasoning tokens are generated serially despite low entropy; and flow-matching denoising applies uniform compute to a non-uniform velocity field. Addressing any one stage in isolation leaves the others untouched. We propose FlashDrive, an algorithm-system co-design framework that targets all four stages simultaneously. Our key insight is that each bottleneck admits a distinct, lightweight algorithmic shortcut: temporal overlap enables streaming KV-cache reuse across frames; the low per-token entropy and strong intra-block correlations of driving-domain reasoning make a non-autoregressive diffusion drafter highly effective for speculative decoding; and the velocity field's structure---sharp at the endpoints, flat in the middle---permits adaptive step caching that concentrates compute where it matters. Layered on system-level CUDA Graph compilation and kernel fusion, these techniques compound. Applied to Alpamayo 1.5-10B with W4A8 quantization, FlashDrive reduces end-to-end latency from 717ms to 151ms (4.7x) while leaving accuracy essentially unchanged: [email protected] shifts by only 0.08m, minADE1 improves, and closed-loop collision and off-road rates improve in simulation. By raising a 10B-parameter reasoning VLA from 1.4~~Hz to 6.6~~Hz on a single GPU, FlashDrive moves end-to-end autonomous driving substantially closer to real-time deployment.

Abstract

Inference-Time Compute Allocation

Explore similar work

FlashDrive: Flash Vision-Language-Action Inference for Autonomous Driving

Scene-Adaptive VLA: Efficient Autonomous Driving via Dynamic Layer Routing

Think at 5 Hz, Act at 20 Hz: Asynchronous Fast-Slow Vision-Language-Action Inference for Closed-Loop Driving