Resource-Efficient 3D Shape and Orientation Estimation of Road Users using Roadside RGB Camera and 3D Radar

Shiva Agrawal; Savankumar Bhanderi; Gordon Elger

cs.CVOct 13, 2026

Resource-Efficient 3D Shape and Orientation Estimation of Road Users using Roadside RGB Camera and 3D Radar

Authors: Shiva Agrawal, Savankumar Bhanderi, Gordon Elger

Organizations: Institute for Innovative Mobility (IIMo) Technische Hochschule Ingolstadt, Germany

Abstract

End-to-end deep learning pipelines for camera-radar 3D object detection are well established in driver assistance systems and autonomous vehicles; however, adapting them to roadside-mounted infrastructure presents significant challenges. On vehicles, sensor heights, pitch angles, and relative geometries remain largely uniform, allowing end-to-end models to generalize effectively. On the contrary, roadside sensor installations vary drastically in height, tilt, and field of view across locations, rendering end-to-end models prone to failure and difficult to adapt without massive location-specific datasets. Hence, instead of designing a complex end-to-end pipeline for direct 3D bounding box estimation, a low-cost, resource-efficient two-stage approach is proposed for real-time 3D shape and orientation estimation. The first stage utilizes any off-the-shelf, state-of-the-art method to extract 2D camera bounding boxes and associated 3D radar points for each road user. The proposed second stage then processes these inputs to construct 3D oriented bounding boxes where radar data determines 3D position, predefined templates define size for each road user, and a tiny neural network predicts a site-specific and optimized yaw angle. The algorithm is successfully validated at a curved road with a junction for pedestrians, bicycles, and cars, where it achieves 25 frames per second on an NVIDIA Jetson ORIN for real-time applications.

Figures & tables

Fig. 1 : Sensor setup and coordinate system (left) [ 30 ] , with corresponding coordinate transformations (right). $K$ denotes the camera intrinsic matrix; $T_{RC},T_{LC},T_{RL},$ and $T_{LG}$ represent the radar-to-camera, lidar-to-camera, radar-to-lidar, and lidar-to-ground extrinsic transformation matrices, respectively; $P$ and $\tilde{P}$ are the standard and homogeneous projection matrices.

Fig. 2 : Overview of the two-stage roadside 3D object detection framework. Inputs comprise a monocular image frame $\mathbf{I}$ and a sparse 3D radar point cloud frame $\mathbf{P}$ . Stage 1 yields a set of $N$ road user instances $\mathbf{D}=\{(\mathbf{b}_{i},c_{i},s_{i},\mathbf{P}_{i})\}_{i=1}^{N}$ , where $\mathbf{b}_{i}$ is the 2D bounding box, $c_{i}$ is the object category, $s_{i}$ is the detection confidence score, and $\mathbf{P}_{i}$ is the associated radar point subset. Stage 2 estimates the 3D Oriented Bounding Box (OBB) state vectors $\{\mathbf{\Theta}_{i}\}_{i=1}^{N}$ for all detected instances.

Fig. 3 : A lightweight neural network architecture of YawPredictionNet to estimate yaw angle ( $\psi$ ) for bicycles. It illustrates early feature extraction across radar and camera streams, feature concatenation, and continuous regression ( $\sin\psi,\cos\psi$ ).

Exp.	Parameters varied			Errors (mean, std)
Exp.	LSF	WSF	HSF	Length	Width	Height
1	$1.25$	$1.25$	$1$	$(0.27,0.10)$	$(0.07,0.06)$	$(0.20,0.11)$
2	$1.00$	$1.25$	$1$	$(0.44,0.10)$	$(0.07,0.06)$	$(0.20,0.11)$
3	$1.50$	$1.25$	$1$	$(0.13,0.09)$	$(0.07,0.06)$	$(0.20,0.11)$
4	$\mathbf{1.75}$	$\mathbf{1.25}$	$\mathbf{1}$	$\mathbf{(0.11,0.09)}$	$\mathbf{(0.07,0.06)}$	$\mathbf{(0.20,0.11)}$
5	$2.00$	$1.25$	$1$	$(0.23,0.14)$	$(0.07,0.06)$	$(0.20,0.11)$

TABLE I : Ablation study of parameter variation on dimension errors for Pedestrians. LSF: Length Scaling Factor, WSF: Width Scaling Factor, HSF: Height Scaling Factor

Exp.	Parameters Varied			Errors (mean, std)
Exp.	Length (m)	Width (m)	HSF	Length	Width	Height
	min / max	min / max
1	$1.8/2.2$	$0.7/1.0$	$1.00$	$(0.28,0.12)$	$(0.07,0.05)$	$(0.10,0.08)$
2	$1.6/2.2$	$0.7/1.0$	$1.00$	$(0.47,0.12)$	$(0.07,0.05)$	$(0.10,0.08)$
3	$2.0/2.2$	$0.7/1.0$	$1.00$	$(0.10,0.09)$	$(0.07,0.05)$	$(0.10,0.08)$
4	$2.0/2.4$	$0.7/1.0$	$1.00$	$(0.10,0.09)$	$(0.07,0.05)$	$(0.10,0.08)$

TABLE II : Ablation study of parameter variation on dimension errors for bicycles. HSF: Height Scaling Factor

Exp.	Parameters Varied			Errors (mean, std)
Exp.	Length (m)	Width (m)	HSF	$x_{c}$ (m)	Length (m)	$y_{c}$ (m)	Width (m)	$z_{c}$ (m)	Height (m)
	min / max	min / max
1	$3.8/5.5$	$1.7/2.2$	$1.00$	$(0.40,0.39)$	$(0.64,0.22)$	$(0.54,0.42)$	$(0.24,0.08)$	$(0.22,0.06)$	$(0.08,0.09)$
2	$4.0/5.5$	$1.7/2.2$	$1.00$	$(0.40,0.38)$	$(0.48,0.22)$	$(0.54,0.42)$	$(0.24,0.08)$	$(0.22,0.06)$	$(0.08,0.09)$
3	$4.2/5.5$	$1.7/2.2$	$1.00$	$(0.39,0.33)$	$(0.33,0.21)$	$(0.54,0.42)$	$(0.24,0.08)$	$(0.22,0.06)$	$(0.08,0.09)$
4	$4.4/5.5$	$1.7/2.2$	$1.00$	$(0.38,0.29)$	$(0.21,0.06)$	$(0.54,0.42)$	$(0.24,0.08)$	$(0.22,0.06)$	$(0.08,0.09)$

TABLE III: Ablation study of parameter variation on car dimension and centroid errors. HSF: Height Scaling Factor

Class	Method	$x_{c}$ (m)	$y_{c}$ (m)	$z_{c}$ (m)	Length (m)	Width (m)	Height (m)	Yaw ( ∘ )
Person	B1	$(1.10,0.89)$	$(0.37,0.44)$	$(0.21,0.12)$	$(0.40,0.09)$	$(0.15,0.04)$	$(0.27,0.19)$	–
	B2	$(1.10,0.89)$	$(0.37,0.44)$	$(0.21,0.12)$	$(0.40,0.09)$	$(0.15,0.04)$	$(0.27,0.19)$	–
	Ours	$\mathbf{(0.12,0.09)}$	$\mathbf{(0.09,0.07)}$	$\mathbf{(0.17,0.09)}$	$\mathbf{(0.11,0.09)}$	$\mathbf{(0.07,0.06)}$	$\mathbf{(0.20,0.11)}$	–
Bicycle	B1	$(2.42,2.43)$	$(0.53,0.76)$	$(0.14,0.09)$	$(0.28,0.11)$	$(0.05,0.04)$	$(0.16,0.16)$	–
	B2	$(2.42,2.43)$	$(0.53,0.76)$	$(0.14,0.09)$	$(0.28,0.11)$	$(0.05,0.04)$	$(0.16,0.16)$	$(15.04,16.27)$
	Ours	$\mathbf{(0.36,0.22)}$	$\mathbf{(0.21,0.16)}$	$\mathbf{(0.09,0.05)}$	$\mathbf{(0.10,0.09)}$	$\mathbf{(0.05,0.04)}$	$\mathbf{(0.10,0.08)}$	$\mathbf{(4.93,4.63)}$

TABLE IV : Comparison of bounding box estimation errors (mean, std) with baseline methods (B1: Baseline 1, B2: Baseline 2). Mean and standard deviation are calculated over all

500

validation frames.

Panel of Figure 8 — Fig. 4 : Output of 3D bounding box estimator. Fig. (a) and (b) shows the image plane with a 2D bounding box, class ID (1:Person, 2:Bicycle, 4:Car), associated radar points, and calculated Doppler velocity, and Fig. (c) and (d) show the corresponding estimated 3D oriented bounding box in the ground plane (green boxes are estimated, red boxes are ground truth, and white spheres are radar points). The red line in (c) and (d) shows the X-axis of the setup.

Camera-Radar Fusion

Camera-radar fusion combines visual semantics with radar’s range, velocity, and all-weather sensing for robust perception.

See all 49 papers

Sep 1, 2026

Infrastructure-based Monocular 3D Vehicle Localization Framework with Experimental Validation

This paper presents a one-stage learning framework that maps monocular roadside-camera images directly to vehicle states in a ground-fixed coordinate frame. Unlike conventional approaches that first detect vehicles in the image plane and subsequently apply geometric post-processing, the proposed method leverages features from a pretrained object detector to jointly estimate each vehicle's ground-plane position, dimensions, and yaw angle. The framework therefore uses visual features not only for vehicle detection but also for direct spatial and orientation estimation. To support model training and evaluation, we develop a data-collection and label-generation pipeline based on synchronized video from a roadside camera and an unmanned aerial vehicle (UAV). Acting as a temporary top-view sensing platform, the UAV provides vehicle trajectories, dimensions, and orientations, which are transformed into the ground-fixed coordinate frame and temporally aligned with the roadside-camera images to generate ground-truth labels. The framework is evaluated using data collected during multiple experiments at the Mcity Test Facility. Results show that the proposed method can recover vehicle trajectories and orientations from monocular roadside imagery without a separate geometric post-processing stage, demonstrating its potential as a scalable approach to infrastructure-based perception at urban intersections.

Akos T. Kopeczi-Bocz, Tian Mi, Gabor Orosz +1

Department of Applied Mechanics, Budapest University of Technology and Economics, Budapest, H-1111, Hungary · Department of Mechanical Engineering, University of Michigan, Ann Arbor, MI 48109, USA · Department of Mechanical Engineering, the Department of Civil and Environmental Engineering, and the Transportation Research Institute at the University of Michigan, Ann Arbor, MI 48109, USA

Autonomous Driving Perception Monocular 3D Object Detection

Ask PDF

Jun 30, 2026

Horizon3D: Sparse Radar-Camera Fusion for Long-Range 3D Perception in Autonomous Driving

Long-range 3D object detection is critical for safe autonomous driving at highway speeds, yet existing radar-camera fusion methods remain limited at extended ranges. BEV-based methods capture scene-level context but incur rapidly growing computation and often lose fine-grained object detail, while query-based methods are efficient but provide limited scene-level context. Temporal fusion further requires both multi-frame accumulation for sparse distant observations and object-level motion modeling for fast-moving objects. We propose Horizon3D, a sparse radar-camera fusion framework for long-range 3D object detection that combines Gaussian primitives with sparse BEV features. Horizon3D initializes Gaussian primitives at radar- and camera-estimated object keypoints using Keypoint-Guided Gaussian Initialization, refines them through Object-Centric Sparse Fusion, and splats them onto the BEV plane to fuse object-level detail with sparse radar BEV context. It further introduces Dual-Path Temporal Fusion, which aggregates temporal cues through a BEV path for scene-level accumulation and a Gaussian path for object-level motion propagation. Experiments on TruckScenes show that Horizon3D achieves state-of-the-art radar-camera 3D detection performance. On the validation set, it outperforms the previous best method by +3.0 NDS and +1.6 mAP while maintaining competitive inference speed.

Geonho Bang, Geunju Baek, Dongyoung Lee +2

Seoul National University, Seoul, Republic of Korea

Camera-Radar Fusion Autonomous Driving 3D Object Detection Multimodal Sensor Fusion

Ask PDF

Mar 12, 2026

R4Det: 4D Radar-Camera Fusion for High-Performance 3D Object Detection

4D radar-camera sensing configuration has gained increasing importance in autonomous driving. However, existing 3D object detection methods that fuse 4D Radar and camera data confront several challenges. First, their absolute depth estimation module is not robust and accurate enough, leading to inaccurate 3D localization. Second, the performance of their temporal fusion module will degrade dramatically or even fail when the ego vehicle's pose is missing or inaccurate. Third, for some small objects, the sparse radar point clouds may completely fail to reflect from their surfaces. In such cases, detection must rely solely on visual unimodal priors. To address these limitations, we propose R4Det, which enhances depth estimation quality via the Panoramic Depth Fusion module, enabling mutual reinforcement between absolute and relative depth. For temporal fusion, we design a Deformable Gated Temporal Fusion module that does not rely on the ego vehicle's pose. In addition, we built an Instance-Guided Dynamic Refinement module that extracts semantic prototypes from 2D instance guidance. Experiments show that R4Det achieves state-of-the-art 3D object detection results on the TJ4DRadSet and VoD datasets. The source code and models will be released at https://github.com/VDIGPKU/R4Det.

Zhongyu Xia, Yousen Tang, Yongtao Wang +2

Wangxuan Institute of Computer Technology, Peking University · EBTech Co. Ltd

3D Object Detection Camera-Radar Fusion Autonomous Driving Perception Multimodal Sensor Fusion Depth Estimation

Ask PDF

Abstract

Camera-Radar Fusion

Explore similar work

Infrastructure-based Monocular 3D Vehicle Localization Framework with Experimental Validation

Horizon3D: Sparse Radar-Camera Fusion for Long-Range 3D Perception in Autonomous Driving

R4Det: 4D Radar-Camera Fusion for High-Performance 3D Object Detection