Deploying vision perception models in autonomous vehicles requires that we prioritize inference speeds, resulting in a model with shallower architectures and lesser model parameters (i.e., more pruned). Such small models do not generalize well, which could result in poor performance when encountered with novel scenarios. We propose a system that overcomes this by continuously retraining the vision models on the cloud with data uploaded by vehicles. We leverage the abundant compute resources, including machine learning accelerators, of the cloud to run a highly-accurate oracle model that will guide the retraining process of the on-vehicle model. This newly trained model is transmitted to the vehicle over the network and is utilized by the vehicle for perceptions, leading to improved inference accuracy over time.
Figures & tables
Fig. 1: A scenario where the vision model deployed in the vehicle fails to detect multiple important objects. The purple bounding boxes correspond to the detection of the larger Faster R-CNN model, and the yellow bounding boxes correspond to the smaller SSD-MobileNet model.
Fig. 2: AdaptAV continuously updates the vision model running on the vehicle. The image stream used by the vision model is compressed and uploaded to the cloud. A cloud instance with powerful compute with machine learning accelerators uses the data to retrain the model. The retrained model is then deployed back to the vehicle, closing the loop.
Class
Faster R-CNN
SSD-Mobilenet
person
0.45
0.17
car
0.72
0.49
bicycle
0.41
0.18
sign
0.48
0.09
Average
0.52
0.23
TABLE I: Mean average precision for oracle model and device model
Fig. 3: A model is likely to overfit when using frames captured/sampled at a high rate. Hence, a frame sampling is necessary.
Fig. 4: Density-based frame sampling. During each sampling period, the one that has the largest number of detected objects is selected.
Fig. 5: Event-based frame sampling. An event is defined by the appearance of a new object or the disappearance of an existing object.
Fig. 6: The validation loss during the continuous retraining of the SSD-Mobilenet model. There are a total of 4 retraining windows.
Fig. 7: Model accuracy (AP for the car class) v.s. retraining time for different hyperparameter configurations. The number of windows refers to the number of retraining windows into which the BDD100K training set is divided equally. Each line is cut at the point where the y-axis value converges.
Fig. 8: The impacts of different sampling approaches on the validation loss.
Fig. 9: The blue-colored vehicle on the left of the scene is first detected by the Oracle model at Frame 44. Thus, it is the only frame sampled by the event-based method. The density-based method samples three frames from these three windows, each with a length of five, although the scenes are similar.
Fig. 10: AdaptAV implemented on a small autonomous shuttle interacting with a public cloud platform. The vehicle uses the NVIDIA Jetson Orin as its computing platform and uses a ZED2 camera inputs for its vision tasks.
Fig. 11: Validation loss curve for the object detection model running in the autonomous shuttle.