Accurate cone localization in 3D space is essential in autonomous racing for precise navigation around the track. Approaches that rely on traditional computer vision algorithms are sensitive to environmental variations, and neural networks are often trained on limited data and are infeasible to run in real time. We present a UNet-based neural network for keypoint detection on cones, leveraging the largest custom-labeled dataset we have assembled. Our approach enables accurate cone position estimation and the potential for color prediction. Our model achieves substantial improvements in keypoint accuracy over conventional methods. Furthermore, we leverage our predicted keypoints in the perception pipeline and evaluate the end-to-end autonomous system. Our results show high-quality performance across all metrics, highlighting the effectiveness of this approach and its potential for adoption in competitive autonomous racing systems.
Figures & tables
Figure 1: Overview of the camera pipeline. Different cone position estimation methods are performed in parallel and combined with an Extended Kalman Filter. Red and Green represent system inputs and outputs respectively, with blue representing pipeline sub-routines.
Figure 2: A ZED2 left camera frame with YOLOv8 bounding boxes. Solid bars above cone annotations represent the confidence of the YOLOv8 predictions.
Figure 3: General keypoint labels and their corresponding labels on a blue and yellow cone.
Figure 4: A high-level overview of the architecture used in our KPR model.
Figure 5: The training and validation loss values at each step plotted throughout model training.
Model
MSE
Root MSE
Norm ME
Std Dev
mAP
ResNet
6.3165
2.3458
0.0597
6.4299
0.42
UNet
3.4172
1.8486
0.0263
3.4550
0.83
Table 1: Metric Comparison of ResNet and UNet KPR NNs.
Figure 6: The mAP measures the distance between true and predicted keypoints indicated by a red line between points.
Figure 7: The predicted keypoints (in red) are inaccurate with respect to the ground truths (in green) due to the cone being partially cropped out by YOLOv8. The yellow line represents the disparity between the labels and inferences.
Figure 8: Confusion matrix showing integrated KPR pipeline detected classes against ground truth annotations on simulated data. Off-diagonal non-zero values indicate inaccuracies in the pipeline.
Figure 9: Birds-eye view displaying experimentally derived covariances for perception pipeline at varying distance bins. Blue points represent the error of network predictions, centered around the ground truth position. Red ellipses show covariances for each distance bin, with mean equal to the ground truth position.
Figure 10: The CPU and memory usage of the PC inside the car while running a ROS bag. Subfigure (a) shows the plot with no KPR method enabled. Here, the perception pipeline relies on YOLOv8 detection and estimation. The bottom figure (b) runs the pipeline with the KPR neural network enabled. The PC is under heavier computation in this scenario, so the CPU and memory usage become higher.
In autonomous racing, fast detection of other participants' movements is required to plan safe, collision-free trajectories with non-cooperative opponents. LiDAR detection is inherently slower and harder to deploy on edge devices than vision methods, causing delayed detections that limit object tracking performance during high-dynamic maneuvering. Utilizing monocular 3D detection enables an easy-to-deploy, low-latency detection of other participants on the racetrack. We present SPARK, a single-camera pose-estimation algorithm for autonomous racing using keypoint detection. It achieves long-range detection with high accuracy, exceeding the performance of state-of-the-art monocular camera detection algorithms while maintaining lower latency. By employing well-optimized YOLO models and leveraging the fixed geometry in the autonomous racing domain, the algorithm also exhibits low latency and resource usage. We evaluate the performance of our approach on real-world autonomous racing data and compare it to state-of-the-art LiDAR and camera detection algorithms. The source code is available at: https://github.com/TUMFTM/SPARK-camera-det
Dominic Ebner, Markus Lienkamp
Technical University of Munich, Germany; School of Engineering & Design, Department of Mobility Systems Engineering, Institute of Automotive Technology and Munich Institute of Robotics and Machine Intelligence (MIRMI)
Lane boundary detection is a critical component in autonomous driving systems and has been rigorously studied in regular driving scenarios. However, it is less explored in vehicle racing, where the car moves at higher speeds across more extreme road geometries. To study this problem, we introduce a new dataset for 3D lane detection in racing, featuring >250k images from multiple camera feeds and inertial measurements taken with a Lexus LC 500 driving on a closed circuit. With this dataset, we compare various approaches to 3D lane detection and propose modifications that permit frames to be processed at rates of almost 300Hz while retaining high predictive performance in the racing application. This facilitates a multi-camera ensemble approach that is validated on hardware. We show that sensing modalities such as inertial measurements can be leveraged for pre-integration to regress road geometries over both cameras and time, yielding improvements in key metrics. Compared to methods such as BevLaneDet, adding odometry and ensemble predictions improves the F1 score by 3 points and reduces near-vehicle mean absolute errors (MAEs) by >30%. We show F1 scores >0.9 and lateral MAEs of <0.18m in vehicle deployments.
Omoruyi Atekha, John Subosits, Marcus Greiff
Massachusetts Institute of Technology (MIT), Cambridge, MA, USA. · Toyota Research Institute (TRI), 94022 Los Altos, CA, USA.
This paper introduces RaCo, a lightweight neural network designed to learn robust and versatile keypoints suitable for a variety of 3D computer vision tasks. The model integrates three key components: the repeatable keypoint detector, a differentiable ranker to maximize matches with a limited number of keypoints, and a covariance estimator to quantify spatial uncertainty in metric scale. Trained on perspective image crops only, RaCo operates without the need for covisible image pairs. It achieves strong rotational robustness through extensive data augmentation, even without the use of computationally expensive equivariant network architectures. The method is evaluated on several challenging datasets, where it demonstrates state-of-the-art performance in keypoint repeatability and two-view matching, particularly under large in-plane rotations. Ultimately, RaCo provides an effective and simple strategy to independently estimate keypoint ranking and metric covariance without additional labels, detecting interpretable and repeatable interest points. The code is available at https://github.com/cvg/RaCo.
Abhiram Shenoi, Philipp Lindenberger, Paul-Edouard Sarlin +1
ETH Zurich · Google · Microsoft Mixed Reality & AI Lab