Infrastructure-based Monocular 3D Vehicle Localization Framework with Experimental Validation
Authors: Akos T. Kopeczi-Bocz, Tian Mi, Gabor Orosz, Denes Takacs
Organizations: Department of Applied Mechanics, Budapest University of Technology and Economics, Budapest, H-1111, Hungary · Department of Mechanical Engineering, University of Michigan, Ann Arbor, MI 48109, USA · Department of Mechanical Engineering, the Department of Civil and Environmental Engineering, and the Transportation Research Institute at the University of Michigan, Ann Arbor, MI 48109, USA
This paper presents a one-stage learning framework that maps monocular roadside-camera images directly to vehicle states in a ground-fixed coordinate frame. Unlike conventional approaches that first detect vehicles in the image plane and subsequently apply geometric post-processing, the proposed method leverages features from a pretrained object detector to jointly estimate each vehicle's ground-plane position, dimensions, and yaw angle. The framework therefore uses visual features not only for vehicle detection but also for direct spatial and orientation estimation. To support model training and evaluation, we develop a data-collection and label-generation pipeline based on synchronized video from a roadside camera and an unmanned aerial vehicle (UAV). Acting as a temporary top-view sensing platform, the UAV provides vehicle trajectories, dimensions, and orientations, which are transformed into the ground-fixed coordinate frame and temporally aligned with the roadside-camera images to generate ground-truth labels. The framework is evaluated using data collected during multiple experiments at the Mcity Test Facility. Results show that the proposed method can recover vehicle trajectories and orientations from monocular roadside imagery without a separate geometric post-processing stage, demonstrating its potential as a scalable approach to infrastructure-based perception at urban intersections.
Figures & tables
Fig. 1: Top view snapshot of the experimental setup at a given time instant. Colored lines indicate the trajectories of the five vehicles. The red camera icon indicates the location and orientation of the roadside camera.
Fig. 2: Comparison of raw and processed ground truth data for Vehicle 1 (truck) performing a turn. Panel (a) and (b) show the time evolution of the yaw angle and the speed, respectively. The gray curves represent the raw data while the purple curves show the data obtained after applying the Savitzky-Golay filter. The noise is more significant for the speed shown in panel (b) as it is obtained via numerical differentiation of the position.
Fig. 3: Filtered data presented for all five vehicles. Panel (a) and (b) show the time evolution of the yaw angle and the speed, respectively.
Fig. 4: Visualization of the homography transformation. A sample grid (red) is depicted on both the top view (left) and the roadside view (right) images. The arrows indicate the correspondence between specific grid points.
Fig. 5: Roadside image taken at the time presented in Fig. 1 with projected ground truth. The bounding boxes and past trajectories of the vehicles are overlaid on the image.
Fig. 6: Spatial filtering on the top view plane. The red region represents the area visible to the roadside camera. Vehicle trajectories are colored within the field of view and are shown in gray when outside it.
Fig. 7: Simplified visualization of the YOLOv5 (v6.0) structure. Layers with trainable weights are shown and numbered. Intermediate untrainable layers (like upsampling and concatenation) are not shown. The Conv stands for the CBS module, which is composed of a standard convolution, batch normalization [ 34 ] layer and SiLU [ 35 ] activation function. The C3 is a Cross Stage Partial [ 36 ] module with 3 convolutions. The SPPF [ 37 ] stands for fast Spatial Pyramid Pooling. The Detect modules are custom convolutional detection heads; the corresponding output grid sizes are shown above the modules and l denotes the output layer index.
Fig. 8: Box losses for the training ( Lboxtrain ) and validation ( Lboxval ) for different freezing configurations.
Fig. 9: Objectness losses for the training ( Lobjtrain ) and validation ( Lobjval ) for different freezing configurations.
Parameter
Value
Pretrained weights
yolov5s.pt
Epochs
30
Batch size
32
Image size
640 × 640 px
Optimizer
SGD
Learning rate
0.01
TABLE I: Key hyperparameters used throughout the training.
Freeze
Lboxtrain
Lboxval
Lobjtrain
Lobjval
No Freeze
0.0100
1.9788
0.0039
0.0217
1-1
0.0105
1.9734
0.0039
0.0206
1-2
0.0097
1.8855
0.0039
0.0209
1-3
0.0098
1.9237
0.0040
0.0216
1-4
0.0104
2.1720
0.0039
0.0225
1-5
0.0100
2.0423
0.0039
0.0230
TABLE II: End loss values (at epoch 30) of the box and objectness losses for the training and validation datasets.
Fig. 10: Training and validation results on the base dataset. (a)-(b) Ground truth trajectories are shown in gray, and network predictions are shown using dark color. (c)-(f) Ground-truth yaw angle and speed are shown by light colors inside the field of view, while outside values are shown in gray. The network predictions are shown using dark colors.
Fig. 11: In-context test results on a held-out run from the same experimental scenarios. (a)-(b) Ground truth trajectories are shown in gray, and network predictions are shown using dark color. (c)-(f) Ground-truth yaw angle and speed are shown by light colors inside the field of view, while outside values are shown in gray. The network predictions are shown using dark colors.
Fig. 12: Out-of-context test results on an unseen experiment of an unseen scenario. (a)-(b) Ground truth trajectories are shown in gray, and network predictions are shown using dark color. (c)-(f) Ground-truth yaw angle and speed are shown by light colors inside the field of view, while outside values are shown in gray. The network predictions are shown using dark colors.
Accurate vehicle localization from monocular roadside surveillance cameras is important for intelligent transportation systems, traffic monitoring, and traffic conflict analysis. Standard approaches often estimate vehicle position from the center of the detector bounding box, which can produce large errors due to perspective distortion and parallax, especially for elevated cameras and large vehicles. This paper proposes a two-stage geometry-aware localization pipeline that estimates the projection of the vehicle footprint onto the road plane. First, vehicles are detected using a YOLO26-based detector. Second, a dedicated ResNet34 regression network predicts four corner points corresponding to the projected vehicle base. The final position is computed as the geometric center of the predicted quadrilateral. The method was trained on synthetic data generated in CARLA and fine-tuned on real-world roadside imagery from DAIR-V2X. Experiments on synthetic and real data showed clear improvements over naive bounding-box-center localization. On DAIR-V2X, the mean image-space localization error decreased from 31.77 px to 15.30 px, a 51.8% improvement, while the median error decreased to 4.29 px. Median ground-plane error for medium-range vehicles decreased from 5.52 m to 0.90 m, and for far-range vehicles from 8.67 m to 1.84 m. The results also show that contextual information surrounding the detector bounding box is important for geometric localization. The largest gains were observed for distant vehicles and geometrically challenging cases affected by strong perspective distortion and parallax.
Jan Gawroński, Witold Czajewski
Warsaw University of Technology · Faculty of Electrical Engineering · Warsaw, Poland
Infrastructure-based localization enhances road safety and traffic management by providing state estimates of road users. Development is hindered by fragmented, application-specific stacks that tightly couple perception, tracking, and middleware. We introduce Ufil, a Unified Framework for Infrastructure-Based Localization with a standardized object model and reusable multi-object tracking components. Ufil offers interfaces and reference implementations for prediction, detection, association, state update, and track management, allowing researchers to improve components without reimplementing the pipeline. Ufil is open-source C++/ROS 2 software with documentation and executable examples. We demonstrate Ufil by integrating three heterogeneous data sources into a single localization pipeline combining (i) vehicle onboard units broadcasting ETSI ITS-G5 Cooperative Awareness Messages, (ii) a lidar-based roadside sensor node, and (iii) an in-road sensitive surface layer. The pipeline runs unchanged in the CARLA simulator and a small-scale CAV testbed, demonstrating Ufil's scale-independent execution model. In a three-lane highway scenario with 423 and 355 vehicles in simulation and testbed, respectively, the fused system achieves lane-level lateral accuracy with mean lateral position RMSEs of 0.31 m in CARLA and 0.29 m in the CPM Lab, and mean absolute orientation errors around 2.2°. Median end-to-end latencies from sensing to fused output remain below 100 ms across all modalities in both environments.
Simon Schäfer, Lucas Hegerath, Marius Molz +2
The authors are with the Chair of Embedded Software, RWTH Aachen University · B. Alrifaee is with the Department of Aerospace Engineering, University of the Bundeswehr Munich, Germany
Accurate inter-vehicle distance estimation is a cornerstone of Advanced Driver Assistance Systems (ADAS) and autonomous driving. While LiDAR and radar provide high precision, their high cost prohibits widespread adoption in mass-market vehicles. Monocular camera-based estimation offers a low-cost alternative but suffers from fundamental scale ambiguity. Recent deep learning methods for monocular depth achieve impressive results yet require expensive supervised training, suffer from domain shift, and produce predictions that are difficult to certify for safety-critical deployment. This paper presents a framework that exploits the standardized typography of United States license plates as passive fiducial markers for metric ranging, resolving scale ambiguity through explicit geometric priors without any training data or active illumination. First, a four-method parallel plate detector achieves robust plate reading across the full automotive lighting range. Second, a three-stage state identification engine fusing optical character recognition text matching, multi-design color scoring, and a lightweight neural network classifier provides robust identification across all ambient conditions. Third, hybrid depth fusion with inverse-variance weighting and online scale alignment, combined with a one-dimensional constant-velocity Kalman filter, delivers smoothed distance, relative velocity, and time-to-collision for collision warning. Baseline validation on a controlled static dataset reproduces a 2.3% coefficient of variation in character height measurements and a 36% reduction in distance-estimate variance compared with plate-width methods from prior work.
Manognya Lokesh Reddy, Zheng Liu
Department of Computer and Information Science University of Michigan-Dearborn Dearborn, Michigan, USA · Department of Industrial and Manufacturing Systems Engineering University of Michigan-Dearborn Dearborn, Michigan, USA