Deploying vision perception models in autonomous vehicles requires that we prioritize inference speeds, resulting in a model with shallower architectures and lesser model parameters (i.e., more pruned). Such small models do not generalize well, which could result in poor performance when encountered with novel scenarios. We propose a system that overcomes this by continuously retraining the vision models on the cloud with data uploaded by vehicles. We leverage the abundant compute resources, including machine learning accelerators, of the cloud to run a highly-accurate oracle model that will guide the retraining process of the on-vehicle model. This newly trained model is transmitted to the vehicle over the network and is utilized by the vehicle for perceptions, leading to improved inference accuracy over time.
Figures & tables
Fig. 1: A scenario where the vision model deployed in the vehicle fails to detect multiple important objects. The purple bounding boxes correspond to the detection of the larger Faster R-CNN model, and the yellow bounding boxes correspond to the smaller SSD-MobileNet model.
Fig. 2: AdaptAV continuously updates the vision model running on the vehicle. The image stream used by the vision model is compressed and uploaded to the cloud. A cloud instance with powerful compute with machine learning accelerators uses the data to retrain the model. The retrained model is then deployed back to the vehicle, closing the loop.
Class
Faster R-CNN
SSD-Mobilenet
person
0.45
0.17
car
0.72
0.49
bicycle
0.41
0.18
sign
0.48
0.09
Average
0.52
0.23
TABLE I: Mean average precision for oracle model and device model
Fig. 3: A model is likely to overfit when using frames captured/sampled at a high rate. Hence, a frame sampling is necessary.
Fig. 4: Density-based frame sampling. During each sampling period, the one that has the largest number of detected objects is selected.
Fig. 5: Event-based frame sampling. An event is defined by the appearance of a new object or the disappearance of an existing object.
Fig. 6: The validation loss during the continuous retraining of the SSD-Mobilenet model. There are a total of 4 retraining windows.
Fig. 7: Model accuracy (AP for the car class) v.s. retraining time for different hyperparameter configurations. The number of windows refers to the number of retraining windows into which the BDD100K training set is divided equally. Each line is cut at the point where the y-axis value converges.
Fig. 8: The impacts of different sampling approaches on the validation loss.
Fig. 9: The blue-colored vehicle on the left of the scene is first detected by the Oracle model at Frame 44. Thus, it is the only frame sampled by the event-based method. The density-based method samples three frames from these three windows, each with a length of five, although the scenes are similar.
Fig. 10: AdaptAV implemented on a small autonomous shuttle interacting with a public cloud platform. The vehicle uses the NVIDIA Jetson Orin as its computing platform and uses a ZED2 camera inputs for its vision tasks.
Fig. 11: Validation loss curve for the object detection model running in the autonomous shuttle.
Cloud-hosted vision-language models (VLMs) offer greater contextual reasoning capabilities than smaller onboard models, but frequent visual uploads increase communication overhead and add network and inference latency to tactical decisions. We present a risk-adaptive edge-cloud architecture in which onboard traffic assessment determines when cloud reasoning is requested. An onboard VLM and a lightweight detector capture temporal traffic conditions and path-relative hazards for conservative local response and selective cloud access. The cloud model provides tactical advice, while validation, vehicle control, and automatic emergency braking remain local. In CARLA experiments, our method matched the task success rate of periodic cloud access while reducing cloud requests by 54.1% and recording fewer automatic emergency braking (AEB) activations. In a delayed-roadwork ablation, semantic events triggered requests before the next scheduled audit. Across three emulated network profiles, the method continued to reduce cloud traffic, although lane changes took longer than with periodic access. Onboard traffic assessment therefore served as a practical trigger for selective VLM inference in these experiments.
Meng Ma, Shuyang Li, Naigang Wang +1
Rensselaer Polytechnic Institute, Troy, NY, USA · IBM T. J. Watson Research Center, USA
Vision-Language-Action (VLA) models promise to bring end-to-end reasoning to autonomous driving, but their computational cost remains far too high for real-time control. The core challenge is structural: VLA inference is not a single bottleneck but a cascade of four. Visual encoding wastes compute on overlapping video frames; language-model prefill recomputes context that could be carried over from the previous timestep; reasoning tokens are generated serially despite low entropy; and flow-matching denoising applies uniform compute to a non-uniform velocity field. Addressing any one stage in isolation leaves the others untouched. We propose FlashDrive, an algorithm-system co-design framework that targets all four stages simultaneously. Our key insight is that each bottleneck admits a distinct, lightweight algorithmic shortcut: temporal overlap enables streaming KV-cache reuse across frames; the low per-token entropy and strong intra-block correlations of driving-domain reasoning make a non-autoregressive diffusion drafter highly effective for speculative decoding; and the velocity field's structure---sharp at the endpoints, flat in the middle---permits adaptive step caching that concentrates compute where it matters. Layered on system-level CUDA Graph compilation and kernel fusion, these techniques compound. Applied to Alpamayo 1.5-10B with W4A8 quantization, FlashDrive reduces end-to-end latency from 717ms to 151ms (4.7x) while leaving accuracy essentially unchanged: [email protected] shifts by only 0.08m, minADE1 improves, and closed-loop collision and off-road rates improve in simulation. By raising a 10B-parameter reasoning VLA from 1.4Hz to 6.6Hz on a single GPU, FlashDrive moves end-to-end autonomous driving substantially closer to real-time deployment.
Zekai Li, Yihao Liang, Hongfei Zhang +3
1UC San Diego · 2Princeton · 3Independent Researcher
Autonomous vehicles operate in dynamic, ever-changing environments where new scenarios and edge cases constantly emerge. As a result, static learning models are inadequate for ensuring safe and reliable operation. Continuous learning is essential for adapting to these evolving conditions and maintaining robust performance across diverse real-world settings. However, autonomous vehicles generate massive streams of visual data during operation, and existing continuous learning approaches typically rely on heuristic sampling methods that fail to capture temporal dynamics, often overlooking critical learning opportunities or selecting redundant frames. In this paper, we introduce FrameScope, a temporal data valuation framework for continuous learning in autonomous vehicles. FrameScope extends neural tangent kernel theory to temporal domains, enabling principled valuation of streaming visual data. Unlike cloud-centric methods that transmit all video data for processing, our approach performs principled, local frame selection on the vehicle and queries a cloud-based oracle model only for labels of those high-value frames. Extensive experiments across multiple domain shifts show that FrameScope consistently outperforms existing methods, achieving higher sample efficiency and significantly reducing catastrophic forgetting in autonomous vehicle perception. By valuing data on the vehicle and querying only labels for selected frames, FrameScope reduces bandwidth requirements, enabling scalable operation with a lightweight cloud labeling service.
Yuheng Zhu, Man-Ki Yoon
Department of Computer Science North Carolina State University Raleigh, NC, USA