Infrastructure-based Monocular 3D Vehicle Localization Framework with Experimental Validation

Akos T. Kopeczi-Bocz; Tian Mi; Gabor Orosz; Denes Takacs

cs.CVSep 1, 2026

Infrastructure-based Monocular 3D Vehicle Localization Framework with Experimental Validation

Authors: Akos T. Kopeczi-Bocz, Tian Mi, Gabor Orosz, Denes Takacs

Organizations: Department of Applied Mechanics, Budapest University of Technology and Economics, Budapest, H-1111, Hungary · Department of Mechanical Engineering, University of Michigan, Ann Arbor, MI 48109, USA · Department of Mechanical Engineering, the Department of Civil and Environmental Engineering, and the Transportation Research Institute at the University of Michigan, Ann Arbor, MI 48109, USA

Abstract

This paper presents a one-stage learning framework that maps monocular roadside-camera images directly to vehicle states in a ground-fixed coordinate frame. Unlike conventional approaches that first detect vehicles in the image plane and subsequently apply geometric post-processing, the proposed method leverages features from a pretrained object detector to jointly estimate each vehicle's ground-plane position, dimensions, and yaw angle. The framework therefore uses visual features not only for vehicle detection but also for direct spatial and orientation estimation. To support model training and evaluation, we develop a data-collection and label-generation pipeline based on synchronized video from a roadside camera and an unmanned aerial vehicle (UAV). Acting as a temporary top-view sensing platform, the UAV provides vehicle trajectories, dimensions, and orientations, which are transformed into the ground-fixed coordinate frame and temporally aligned with the roadside-camera images to generate ground-truth labels. The framework is evaluated using data collected during multiple experiments at the Mcity Test Facility. Results show that the proposed method can recover vehicle trajectories and orientations from monocular roadside imagery without a separate geometric post-processing stage, demonstrating its potential as a scalable approach to infrastructure-based perception at urban intersections.

Figures & tables

Fig. 1: Top view snapshot of the experimental setup at a given time instant. Colored lines indicate the trajectories of the five vehicles. The red camera icon indicates the location and orientation of the roadside camera.

Fig. 2: Comparison of raw and processed ground truth data for Vehicle 1 (truck) performing a turn. Panel (a) and (b) show the time evolution of the yaw angle and the speed, respectively. The gray curves represent the raw data while the purple curves show the data obtained after applying the Savitzky-Golay filter. The noise is more significant for the speed shown in panel (b) as it is obtained via numerical differentiation of the position.

Fig. 3: Filtered data presented for all five vehicles. Panel (a) and (b) show the time evolution of the yaw angle and the speed, respectively.

Fig. 4: Visualization of the homography transformation. A sample grid (red) is depicted on both the top view (left) and the roadside view (right) images. The arrows indicate the correspondence between specific grid points.

Fig. 5: Roadside image taken at the time presented in Fig. 1 with projected ground truth. The bounding boxes and past trajectories of the vehicles are overlaid on the image.

Fig. 6: Spatial filtering on the top view plane. The red region represents the area visible to the roadside camera. Vehicle trajectories are colored within the field of view and are shown in gray when outside it.

Fig. 7: Simplified visualization of the YOLOv5 (v6.0) structure. Layers with trainable weights are shown and numbered. Intermediate untrainable layers (like upsampling and concatenation) are not shown. The Conv stands for the CBS module, which is composed of a standard convolution, batch normalization [ 34 ] layer and SiLU [ 35 ] activation function. The C3 is a Cross Stage Partial [ 36 ] module with 3 convolutions. The SPPF [ 37 ] stands for fast Spatial Pyramid Pooling. The Detect modules are custom convolutional detection heads; the corresponding output grid sizes are shown above the modules and $l$ denotes the output layer index.

Fig. 8: Box losses for the training (

\mathcal{L}_{\text{box}}^{\text{train}}

) and validation (

\mathcal{L}_{\text{box}}^{\text{val}}

) for different freezing configurations.

Fig. 9: Objectness losses for the training (

\mathcal{L}_{\text{obj}}^{\text{train}}

) and validation (

\mathcal{L}_{\text{obj}}^{\text{val}}

) for different freezing configurations.

Parameter	Value
Pretrained weights	yolov5s.pt
Epochs	30
Batch size	32
Image size	640 $\times$ 640 px
Optimizer	SGD
Learning rate	0.01

TABLE I: Key hyperparameters used throughout the training.

Freeze	$\mathcal{L}_{\text{box}}^{\text{train}}$	$\mathcal{L}_{\text{box}}^{\text{val}}$	$\mathcal{L}_{\text{obj}}^{\text{train}}$	$\mathcal{L}_{\text{obj}}^{\text{val}}$
No Freeze	0.0100	1.9788	0.0039	0.0217
1-1	0.0105	1.9734	0.0039	0.0206
1-2	0.0097	1.8855	0.0039	0.0209
1-3	0.0098	1.9237	0.0040	0.0216
1-4	0.0104	2.1720	0.0039	0.0225
1-5	0.0100	2.0423	0.0039	0.0230

TABLE II: End loss values (at epoch 30) of the box and objectness losses for the training and validation datasets.

Fig. 10: Training and validation results on the base dataset. (a)-(b) Ground truth trajectories are shown in gray, and network predictions are shown using dark color. (c)-(f) Ground-truth yaw angle and speed are shown by light colors inside the field of view, while outside values are shown in gray. The network predictions are shown using dark colors.

Fig. 11: In-context test results on a held-out run from the same experimental scenarios. (a)-(b) Ground truth trajectories are shown in gray, and network predictions are shown using dark color. (c)-(f) Ground-truth yaw angle and speed are shown by light colors inside the field of view, while outside values are shown in gray. The network predictions are shown using dark colors.

Fig. 12: Out-of-context test results on an unseen experiment of an unseen scenario. (a)-(b) Ground truth trajectories are shown in gray, and network predictions are shown using dark color. (c)-(f) Ground-truth yaw angle and speed are shown by light colors inside the field of view, while outside values are shown in gray. The network predictions are shown using dark colors.

Autonomous Driving Perception

Autonomous driving perception enables vehicles to detect and interpret road users, drivable space, and hazards using onboard and shared sensors.

See all 379 papers

Accurate Localization of Road Traffic Objects on the Road Plane Using Surveillance Camera Imagery
Aug 6, 2026Jan Gawroński, Witold CzajewskiKeypoint Detection Video Surveillance
Warsaw University of Technology · Faculty of Electrical Engineering · Warsaw, Poland
Ufil: A Unified Framework for Infrastructure-based Localization
Apr 23, 2026Simon Schäfer, Lucas Hegerath, Marius Molz +2Intelligent Transportation Systems Robot Localization
The authors are with the Chair of Embedded Software, RWTH Aachen University · B. Alrifaee is with the Department of Aerospace Engineering, University of the Bundeswehr Munich, Germany
Physics-Grounded Monocular Vehicle Distance Estimation Using Standardized License Plate Typography
Apr 14, 2026Manognya Lokesh Reddy, Zheng LiuCollision Avoidance Metric Depth Estimation
Department of Computer and Information Science University of Michigan-Dearborn Dearborn, Michigan, USA · Department of Industrial and Manufacturing Systems Engineering University of Michigan-Dearborn Dearborn, Michigan, USA

Aug 6, 2026J/K move · Enter open · S save

Accurate Localization of Road Traffic Objects on the Road Plane Using Surveillance Camera Imagery

Jan Gawroński, Witold Czajewski

Warsaw University of Technology · Faculty of Electrical Engineering · Warsaw, Poland

Keypoint Detection Video Surveillance

Open paper AskPDF arXiv

Accurate vehicle localization from monocular roadside surveillance cameras is important for intelligent transportation systems, traffic monitoring, and traffic conflict analysis. Standard approaches often estimate vehicle position from the center of the detector bounding box, which can produce large errors due to perspective distortion and parallax, especially for elevated cameras and large vehicles. This paper proposes a two-stage geometry-aware localization pipeline that estimates the projection of the vehicle footprint onto the road plane. First, vehicles are detected using a YOLO26-based detector. Second, a dedicated ResNet34 regression network predicts four corner points corresponding to the projected vehicle base. The final position is computed as the geometric center of the predicted quadrilateral. The method was trained on synthetic data generated in CARLA and fine-tuned on real-world roadside imagery from DAIR-V2X. Experiments on synthetic and real data showed clear improvements over naive bounding-box-center localization. On DAIR-V2X, the mean image-space localization error decreased from 31.77 px to 15.30 px, a 51.8% improvement, while the median error decreased to 4.29 px. Median ground-plane error for medium-range vehicles decreased from 5.52 m to 0.90 m, and for far-range vehicles from 8.67 m to 1.84 m. The results also show that contextual information surrounding the detector bounding box is important for geometric localization. The largest gains were observed for distant vehicles and geometrically challenging cases affected by strong perspective distortion and parallax.

Abstract

Autonomous Driving Perception

Explore similar work

Accurate Localization of Road Traffic Objects on the Road Plane Using Surveillance Camera Imagery

Ufil: A Unified Framework for Infrastructure-based Localization

Physics-Grounded Monocular Vehicle Distance Estimation Using Standardized License Plate Typography