Organizations: Department of Electrical and Computer Engineering, University of Rochester, Rochester, NY 14627, USA · Faculty of the Department of Electrical and Computer Engineering, University of Rochester, Rochester, NY 14627, USA
Amidst the rapid advancement of camera-based autonomous driving technology, effectiveness is often prioritized with limited attention to computational efficiency. To address this issue, this paper introduces LRHPerception, a real-time monocular perception package for autonomous driving that uses single-view camera video to interpret the surrounding environment. The proposed system combines the computational efficiency of end-to-end learning with the rich representational detail of local mapping methodologies. With significant improvements in object tracking and prediction, road segmentation, and depth estimation integrated into a unified framework, LRHPerception processes monocular image data into a five-channel tensor consisting of RGB, road segmentation, and pixel-level depth estimation, augmented with object detection and trajectory prediction. Experimental results demonstrate strong performance, achieving real-time processing at 29 FPS on a single GPU, representing a 555% speedup over the fastest mapping-based approach.
Figures & tables
Fig. 1 : Innovation and architecture blueprint a) Paradigm of end-to-end solution b) Paradigm of camera-fusion for local map solution c) Paradigm of our LRHPerception package, extracts essences from monocular camera for cost-info trade-off.
Fig. 2 : Granular Model Structure.1 Design of convolution decoder, object tracking, trajectory prediction, and depth estimation; magnify for details. BTAE mechanism in Algorithm 1. Remaining components are shown in Fig. 3.
Fig. 3 : Convolution Kernel & Transformation Visualization On the right, blue and green points represent Pt−1 and Pt . Purple and orange arrows denote displacement and rotational transform. Red dashed box is the predicted object position from KF.
Fig. 4 : Granular Model Structure.2 Design of road segmentation block, along with other components.
Fig. 5 : Results Visualization The quartet of images depicts the output of LRHPerception, with past trajectories delineated in blue and future trajectory predictions in red. The upper pair of images exemplifies two Success Cases , while the lower duo present one Failure Case .
Methods
Detect
MOTA ↑
IDF1 ↑
IDP ↑
Time(ms)
C-BYTE
YoloX
76.9%
81.2%
85.9%
31.0
OC(2023) [ 46 ]
YoloX
74.1%
77.8%
87.2 %
28.3
Byte(2022) [ 22 ]
YoloX
76.6%
79.3%
84.0%
27.1
BoT(2022) [ 24 ]
YoloX
76.8%
81.0%
85.7%
48.2
TABLE I : Object Tracking. Our model manifests a significant improvement over SOTA across all efficacy and efficiency metrics on MOT datasets, corresponding harmoniously with our structure- fundamentally predicated on achieving refined outputs for robustness. Hyperparameter θth and a are empirically determined as 0.9 and 210 .
Methods
MSE
CMSE
CFMSE
FPS ↑
Dataset: JAAD
(0.5/ 1.0/ 1.5s)
(1.5s)
(1.5s)
(8 / 12 / 24)
LRHP (Ours)
43/ 113/ 283
239
662
111 / 104/ 92.6
SGNet (2021)
82/ 328/ 1049
995
4076
2.8/ 2.8/ 2.7
Bitrap (2020)
93/ 378/ 1206
1105
4565
99.3/ 105/ 97.1
PIE_traj (2019)
110/ 399/ 1280
1183
4780
-/ -/ -
B-LSTM (2017)
159/ 539/ 1535
1447
5615
-/ -/ -
TABLE II : Trajectory Prediction. Our model exhibits significant enhancements in all aspects of efficacy and efficiency relative to SOTA methods, particularly pronounced as prediction time expands.
Methods
mIOU ↑
FPS ↑
Remarks
LRHP (Ours) ∗
88.9
55.0
∗ speed on dual
Road Segmentation (Yolo Backbone) ∗
88.4
96.4
tasks of detecti-
InternImage (2023) [ 36 ]
86.1
79.5
on and segmen-
MSeg (2020) [ 34 ]
77.6
65.7
tation
UJS-base (2021) [ 35 ]
80.5
-
UJS-refined (2021) [ 35 ]
88.3
-
TABLE III : Road Segmentation. Our model exhibits an appreciable acceleration and precision enhancement when operating an additional detection task with the YOLO backbone- a testament to the robustness and effectiveness of our architectural design.
Methods
Backbone
RMS
δ1↑
δ2↑
FPS ↑
LRHP (Ours)
Swin-m
0.229
0.966
0.996
42.0
Depth Estimate
Swin-L
0.216
0.975
0.997
13.3
VA-Depth (2023)
Swin-L
0.209
0.977
0.997
6.2
AdaBins (2021)
EffNet&Vit
0.236
0.964
0.995
1.7
BTS (2019)
DenseNet
0.280
0.955
0.993
20.1
ASTrans (2021)
Vit-B
0.269
0.963
0.995
-
TABLE IV : Depth Estimation. Our model showcases a significant acceleration in processing speed relative to SOTA, all the while maintaining comparable accuracy levels, thus underscoring the remarkable efficiency of our design.
Number of DualConv
Road-Seg (mIOU) ↑
Depth Estimation (RMS)
Time Difference ( Tn−Tn=1 )(ms)
n=1
88.5
0.223
0
n=3
88.5
0.219
0.268
n=2
88.9
0.220
0.049
TABLE V : Ablation study. We studied the effect of different numbers of Dual Convolutions in the C2f module on depth estimation and road segmentation.
Methods
Category
FPS ↑
LRHP(Ours)
Monocular
28.8
LRHP in series
Monocular
16.3
SOTA in series
Monocular
1.8
Uni-AD (2023) [ 13 ]
Multi-Cam Map
2.1
BEVerse-Tiny (2022) [ 14 ]
Multi-Cam Map
4.4
DETR3D (2021) [ 5 ]
Multi-Cam Map
2.0
TABLE VI : Entire Model. Our model witnesses an improvement of more than an order of magnitude over existing local mapping methods, a testament to the exceptional efficiency of our model. Note that Uni-AD’s planning module was removed for a fair comparison.
Autonomous driving scenes range from empty highways to dense intersections with dozens of interacting road users, yet current 3D detection models apply a fixed computation budget to every frame, wasting resources on simple scenes while lacking capacity for complex ones. Existing approaches compound this problem: Transformer-based interaction models scale quadratically with the number of detected objects, and frame-by-frame processing causes the system to immediately forget objects the moment they become occluded. We propose Enhanced HOPE, an adaptive perception architecture that measures the geometric complexity of each incoming LiDAR frame using an unsupervised statistical estimator and routes it through a shallow or deep processing path accordingly, requiring no manual scene labels. To keep interaction modeling efficient, we replace quadratic pairwise attention with a linear-time subspace-based network that groups nearby objects into clusters and processes them jointly. The computational savings from these two mechanisms free up resources for a persistent temporal memory module that retains previously detected objects and traffic rules across frames, enabling the system to recall occluded objects seconds after they disappear from view. On the nuScenes and CARLA benchmarks, Enhanced HOPE reduces latency by 38% on simple scenes with no accuracy loss, improves mean Average Precision by 2.7 points on rare long-tail scenarios, and tracks objects through occlusions lasting over 5 seconds, where all tested baselines fail.
Self-Supervised Monocular Depth Estimation (MDE) has garnered attention in recent years due to its independence from ground truth. However, most existing models are limited to a single scale and exhibit considerable performance degradation in complex driving environments. Networks specifically designed to handle dynamic traffic participants tend to be overly complex, hindering their deployment on resource-constrained automotive edge devices. To address these limitations and move towards robust driving perception, we propose FlexDepth, a scale-driven and flexible family of self-supervised MDE models tailored for challenging road scenarios. FlexDepth employs a two-stage static-dynamic decoupled training strategy, enabling the independent assessment of confidence for both static backgrounds and dynamic road objects. Furthermore, it introduces a meticulously designed Scale-Driven Decoder (SDD) to dynamically select components based on scale size, facilitating efficient feature fusion and the output of high-precision depth maps. Extensive experiments on standard driving benchmarks demonstrate that without any auxiliary information, our model achieves state-of-the-art performance across arbitrary scales with minimal computational overhead. Our smallest model, Flex-Nano, requires only 0.7 GFLOPs and achieves 37.6 FPS on mobile platforms, ensuring reliable real-time perception while maintaining excellent zero-shot generalization. Our source code is avalible: https://github.com/startnew/flexdepth
Zhaowen Zhu, Li Zhang, Yujie Chen +3
Hefei University of Technology, Hefei, China · Chengpin Home Tech, Nanjing, China · Differential Robotics, Hangzhou, China +1
This study analyses simulated and real-world implementations of depth-aware rover navigation, highlighting the transition from stereo vision to monocular depth estimation using edge AI. A Unity-based lunar terrain simulator with stereo cameras and OpenCV's StereoSGBM was used to generate disparity maps. A physical rover built on Raspberry Pi 4 employed UniDepthV2 for monocular metric depth estimation and YOLO12n for real-time object detection. While stereo vision yielded higher accuracy in simulation, the monocular approach proved more robust and cost-effective in real-world deployment, achieving 0.1 FPS for depth and 10 FPS for detection.
Lomash Relia, Jai G Singla, Amitabh +1
Department of Computer Engineering Devang Patel Institute of Advanced Technology and Research Anand, India · Space Applications Centre Indian Space Research Organization Ahmedabad, India