Driving world-action models improve planning by coupling multimodal reasoning with future prediction, but their growing inference cost increasingly conflicts with the real-time latency requirements of vehicle control. Existing acceleration methods reduce tokens, layers, or sampling steps with policies selected prior to deployment, yet leave residual runtime variation largely unexploited after offline profiling and static scheduling on shared onboard compute. We observe that the largest admissible compute budget varies systematically with the residual runtime state, while recent realized latency provides a direct signal of the available compute slack. Motivated by this observation, we propose SlackDrive, a pre-inference compute allocator that reuses realized latency to select the compute budget of each control step before model execution. SlackDrive profiles the latency and planning utility of a small discrete budget set once, estimates online compute state from completed forwards, and selects the highest-utility budget predicted to remain within the admissible latency envelope, complementing existing profiling and resource scheduling while preserving the driving backbone and its compute actuator. On NAVSIM v2 with DriveDreamer-Policy, SlackDrive improves latency-constrained EPDMS by 21.7% over the strongest baseline under a stringent latency regime, while the full-budget model and preconfigured token-pruning baselines exceed the admissible latency envelope under runtime contention.
Figures & tables
Figure 1: Motivation and formulation. (a) Full-token uses maximum compute, prior efficient methods use a fixed budget, and SlackDrive adapts the budget per step. (b) The admissible budget changes with measured inference latency. SlackDrive selects the largest feasible budget before the next forward pass. (c) SlackDrive conditions world-action inference on the measured latency state ct−1 with a frozen backbone.
Figure 2: Overview of SlackDrive. (a) Visual extraction obtains dense attention cues from the vision encoder for subsequent token selection. ( b) Latency-conditional budget selection adapts the attention mask and compute budget to the current latency target, yielding different configurations under relaxed, moderate, and tight schedules. (c) Runtime-state update and next-budget allocation feeds the realized inference latency back into the runtime state, from which a utility-maximizing policy selects the feasible budget for the next step.
Method
Compute Profile
Latency-Valid NAVSIM Submetrics ↑
Quality ↑
Lat. ↓
FLOPs ↓
Mem. ↓
NC
DAC
DDC
TLC
EP
TTC
LK
HC
EC
EPDMS@ τmax
ms/fr
TF
GB
Admissible latency τmax=180 ms
Full-token
226
56.9
11.0
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
FastV
188
47.3
10.4
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
SparseVLM
188
47.3
10.4
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Table 1: Main results on NAVSIM v2 under latency constraints. All methods use the same N=4,966 held-out scenes and fluctuating-load trace. EPDMS@ τmax follows Eq. 4 . Underlined values denote the best baseline; full-token rows are gray and EfficientDrive rows are lavender.
Figure 3: Sensitivity to the latency constraint. From left to right: latency-valid utility, violation rate, gain over fixed budgets, and retained utility.
Figure 4: Sensitivity to runtime conditions. From left to right: mean load, load variability, their joint effect, and six representative runtime regimes.
Figure 5: Controller design. From left to right: profile density, uncertainty margin, compute-state estimator, and EWMA rate.
Figure 6: Runtime-aware allocation in practice. (a) SlackDrive converts available runtime headroom into additional compute and avoids a collision. (b) Component removal identifies the profile and runtime state that enable this adaptation.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: The cheapest configuration is not always sufficient. Shown are 24 fixed-seed samples from 1,052 scenes in which Static@135 collides while Static@432 remains collision-free with EPDMS ≥80 . The comparison motivates selecting the highest-utility configuration inside the feasible set rather than always minimizing compute.
Figure 8: Population structure of static compute budgets. All 10,439 navtest scenes are included. The left panel shows the per-scene EPDMS difference between Static@432 and full-token inference. The right panel reports collision outcomes for each pair of operating points. Population-level utility increases with compute, while individual scenes need not follow the same ordering.
Figure 9: Budget-dependent trajectories in the camera view. Twelve scenes are selected from the 88 scenes with synchronized camera data according to the separation between the three four-second trajectory endpoints. Semantic inputs are identical across configurations; only bt changes.
Figure 10: Static-budget examples. Twenty-four scenes are uniformly sampled from all 10,439 navtest scenes with a fixed random seed and without outcome-based filtering. The three trajectories frequently overlap, indicating that moderate budget reduction often preserves the resulting plan.
Figure 11: More compute does not guarantee higher per-scene utility. All 21 scenes satisfying the selection rule are shown. Full-token inference collides, while Static@432 remains collision-free with EPDMS ≥80 . These examples establish existence rather than a population-level safety trend.
Figure 12: The population trend favors additional compute. Twelve fixed-seed samples are shown from 315 scenes in which Static@432 collides while full-token inference remains collision-free with EPDMS ≥80 .
Figure 13: Representative raw sensor observations. Six observations are evenly spaced in token order from the available synchronized sensor log. The same visual input underlies all three compute configurations.
Budget
Configuration
L0(b) (ms)
U(b)
b1
Static@135
54
62.88
b2
Static@432
84
81.18
b3
Full-token
226
88.48
Appendix
Table 2: Static compute profile used in the case analysis. L0(b) is nominal latency and U(b) is mean NAVSIM-v2 EPDMS over the analyzed navtest scenes.
Budget pair
Both safe
Cheap only
Expensive only
Both collide
Static@432 vs. Full-token
9,888
352
26
149
Static@135 vs. Static@432
8,702
1,133
74
426
Static@135 vs. Full-token
8,751
1,414
29
145
Appendix
Table 3: Pairwise collision outcomes over all 10,439 scenes. “Cheap only” denotes an at-fault collision from the lower-compute configuration but not the higher-compute configuration; “expensive only” denotes the converse.
Inference-time compute allocation decides where, when, and how much computation models spend during prediction to improve quality under resource limits.
Vision-Language-Action (VLA) models promise to bring end-to-end reasoning to autonomous driving, but their computational cost remains far too high for real-time control. The core challenge is structural: VLA inference is not a single bottleneck but a cascade of four. Visual encoding wastes compute on overlapping video frames; language-model prefill recomputes context that could be carried over from the previous timestep; reasoning tokens are generated serially despite low entropy; and flow-matching denoising applies uniform compute to a non-uniform velocity field. Addressing any one stage in isolation leaves the others untouched. We propose FlashDrive, an algorithm-system co-design framework that targets all four stages simultaneously. Our key insight is that each bottleneck admits a distinct, lightweight algorithmic shortcut: temporal overlap enables streaming KV-cache reuse across frames; the low per-token entropy and strong intra-block correlations of driving-domain reasoning make a non-autoregressive diffusion drafter highly effective for speculative decoding; and the velocity field's structure---sharp at the endpoints, flat in the middle---permits adaptive step caching that concentrates compute where it matters. Layered on system-level CUDA Graph compilation and kernel fusion, these techniques compound. Applied to Alpamayo 1.5-10B with W4A8 quantization, FlashDrive reduces end-to-end latency from 717ms to 151ms (4.7x) while leaving accuracy essentially unchanged: [email protected] shifts by only 0.08m, minADE1 improves, and closed-loop collision and off-road rates improve in simulation. By raising a 10B-parameter reasoning VLA from 1.4Hz to 6.6Hz on a single GPU, FlashDrive moves end-to-end autonomous driving substantially closer to real-time deployment.
Zekai Li, Yihao Liang, Hongfei Zhang +3
1UC San Diego · 2Princeton · 3Independent Researcher
Vision-Language-Action (VLA) models offer a promising paradigm for autonomous driving. However, the massive parameter scale of VLAs strains onboard computational resources during real-time inference. This directly increases energy consumption and limits vehicle battery range, making it critical to minimize computational overhead without compromising driving capability and explainability. Current research on efficient VLA-based driving focuses on sequential adaptation (e.g., Chain-of-Thought and token adjustment), leaving structural adaptation underexplored despite parameter redundancy across dynamic driving scenes. To address this gap, we propose Scene-Adaptive VLA, an efficient framework leveraging dynamic layer routing to adjust active parameters based on real-time scenes. Inspired by the temporal continuity of scenes and model predictive uncertainty, our approach characterizes model-perceived scene complexity via frame-level temporal scene variations and decision deviations. To uncover latent scene-to-parameter correspondences, we encode this characterization into scene-aware tokens, alongside two specialized queries. A budget-conditioned layer routing mechanism evaluates these refined queries to allocate a budget that determines the active parameter ratio and subsequently selects appropriate layer combinations. The entire framework is optimized end-to-end. Extensive evaluations on the Bench2Drive benchmark show that when adapted to a state-of-the-art VLA model, our approach significantly reduces inference-time computational overhead by 53.5% with negligible degradation in driving and language capabilities. Code is available at https:github.com/yang-luvu/Scene-Adaptive-VLA.
Yujia Yang, Quan Yuan, Jiawei Fan +5
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications
Large language models bring instruction following and scene reasoning to end-to-end driving, but their inference latency collides with the control rate a vehicle requires. Existing closed-loop agents hide this gap by invoking the model on alternate simulation ticks and replaying the previous command in between, so half of all control outputs ignore the newest observations. We present a fast-slow architecture that removes this compromise. A frozen 7B vision-language backbone acts as the slow system, digesting navigation instructions and visual history at low frequency while exposing its per-layer key-value cache as a standing representation of the scene. A lightweight action expert acts as the fast system, attending to this cache and to the current camera frame at every simulation tick to regress waypoints in a single forward pass. Since the cache lags behind the world at deployment, we train the expert under randomized staleness, aligning training with asynchronous execution. On LangAuto-Short routes in CARLA, our system produces fresh control at every 50 ms simulation tick and lifts route completion from 37.0 to 94.0 over the frame-skipping baseline. A frame-skip ablation with the same expert separates the two factors at work: the expert raises the driving score on its own, while per-tick freshness raises completion from 82.1 to 94.0 and cuts red-light violations by a third. Trained on a single town, the expert transfers zero-shot to two unseen towns, holding 84-94% route completion where the baseline reaches 31-41%. It reduces open-loop waypoint error by nearly a factor of four compared to the backbone's own action head, at a per-tick model cost of 32 ms that is independent of history length on a single consumer GPU.
Yun Li, Jiachen Gong, Simon Thompson +7
The University of Tokyo, Tokyo, Japan · TIER IV, Inc., Tokyo, Japan · Huawei