MM-BEV: Enhancing Timeliness by Computing Where and When it Matters
Authors: Liangkai Liu, Kang G. Shin
Organizations: Department of Computer Science, Texas Tech University, USA · Department of Computer Science and Engineering, University of Michigan, USA
Multimodal bird's-eye-view (BEV) perception combines LiDAR depth accuracy with dense camera semantics, but its high computational cost and imperfect sensing conditions make real-time deployment challenging. Existing methods largely compress individual detectors and overlook three opportunities: structured sparsity within camera and LiDAR inputs, timing misalignment between modalities, and the fact that many detected objects do not affect the planner's immediate action. We present MM-BEV, a real-time multimodal BEV system guided by a simple principle: compute where and when it matters. MM-BEV divides perception into mandatory work for safety-critical objects within braking distance of the ego vehicle and with short time-to-collision (TTC), and optional work for less urgent regions. It prioritizes mandatory work and reduces or sheds optional work under tight compute budgets. MM-BEV integrates four mechanisms: (1) a criticality-ranked temporal ROI selector based on motion-extrapolated detections from prior frames; (2) sparse, ROI-aware feature extraction using shared-shape camera crops at context-adaptive resolution and ROI-aware LiDAR voxelization; (3) a latency-aware coordinator that adapts LiDAR sweeps, image resolution, and keyframes according to scene dynamics and TTC; and (4) an asynchronous scheduler that decouples sensing from inference and skips stale frames. On nuScenes, MM-BEV reduces inference latency by 1.96x and end-to-end latency by 2.93x, with no loss in geometry-critical recall and only a 0.2 percentage-point drop in safety-critical recall. On a Clearpath Husky A300 equipped with an Ouster-128 LiDAR, BEV cameras, and a Jetson AGX Orin, MM-BEV further reduces mean latency by 2.11x, demonstrating its potential for real-world autonomous systems.
Figures & tables
Fig. 1: End-to-end pipeline for BEV perception.
Fig. 2: Object-presence probability heatmaps across the six nuScenes cameras. Dark areas indicate regions with low per-region probability Pˉ ; percentages (if shown) denote the share of regions with Pˉ<0.1 .
Fig. 3: mAP and inference latency with different LiDAR sweeps.
Fig. 4: Timestamp distributions of LiDAR (20 Hz) and BEV cameras (12 Hz) in nuScenes. Sampling-rate mismatch and asynchronous publication create inherent offsets ( ≳50 ms) even before bus/jitter effects.
Fig. 5: An example of camera–LiDAR misalignment. Two sync groups are formed: {T1cam,T2lidar} and {T2cam,T3lidar} with Δt1,2=350 ms and Δt2,3=150 ms. Objects move and FOV content changes across these offsets, leading to conflicting cross-modal evidence.
Fig. 6: Criticality analysis on nuScenes v1.0-mini ( 18,364 annotations). (a) Only 11% of objects are mandatory (safety-critical); 89% are optional. (b) Per-class mandatory fractions. (c) Critical objects concentrate at close range; optional ones span the full detection envelope. (d) In BEV, mandatory objects (red) cluster in the ego’s forward path while optional objects (blue) populate the periphery.
Fig. 7: MM-BEV system overview.
Fig. 8: Aggregation of LiDAR sweeps from a nuScenes scene without (a) and with (b) ego-motion compensation, above-ground returns only. Color encodes sweep age (red = current sweep, blue = oldest).
Fig. 9: ROI-aware voxelization and feature extraction.
Fig. 10: LiDAR latency vs. non-empty voxel count, profiled on the BEVFusion LiDAR-only branch over 84 runs ( 12 nuScenes scenes ×7 sub-sample rates). (a) Total LiDAR latency is linear in nv ( R2=0.99 ). (b) Voxelize and the sparse encoder scale with nv .
Fig. 11: ROS framework for MM-BEV .
Config
Sensors
Temporal ROI
Adaptive res.
Async sched.
BEVFusion [ liu2023bevfusion ]
L+C
×
×
×
Scheduler Only
L+C
×
×
✓
ROI
L+C
✓
×
×
ROI+LowRes
L+C
✓
fixed-low
×
ROI+Adapt+LowRes
L+C
✓
adaptive
×
MM-BEV
L+C
✓
adaptive
✓
TABLE I: Evaluated configurations.
Fig. 12: Per-frame inference latency split by keyframe and ROI-frame mode, for the LiDAR+Camera (fusion) and LiDAR-only detectors. Baseline must run full inference every cycle; MM-BEV runs fast ROI inference on most frames and full inference only on keyframes.
Fig. 13: Per-stage breakdown of inference time.
Fig. 14: Recall by criticality bucket. Geometry-critical recall is preserved exactly at 100% across all configurations; safety-critical recall drops at most 0.7 pp and all-object recall at most 2.3 pp, under the most aggressive compression.
Config
Per-frame lat. (ms)
All-obj R (%)
Geom-crit R (%)
Safety-crit R (%)
BEVFusion
202
79.9
100.0
94.9
ROI+Adapt+LowRes
95
78.4
100.0
95.2
MM-BEV
104
79.5
100.0
94.7
MM-BEV+LowRes
92
77.6
100.0
94.2
LiDAR Baseline
136
76.8
100.0
93.5
LiDAR MM-BEV
70
76.8
100.0
93.1
TABLE II: Accuracy vs. per-frame inference latency.
Fig. 15: Per-frame latency vs. recall for all ablation configurations (geom-critical preserved at 100% ; right panel shows safety-critical).
Fig. 17: End-to-end ROS pipeline results. Left : mean e2e latency per processed frame (bars show mean, error bars show ± 1 s.d., jitter shows all individual frames). Right : total frames processed per scene. The asynchronous scheduler is the dominant contributor to both gains.
Config
Frames/scene
E2E lat. mean (ms)
E2E lat. p95 (ms)
Speedup
BEVFusion
10
1006
1455
–
Scheduler Only
56
371
498
2.71×
MM-BEV
60
344
434
2.93×
TABLE III: End-to-end ROS pipeline summary.
Fig. 18: Deployment platform: Clearpath Husky A300 with Ouster OS1-128 LiDAR and e-consystems AR0234 FHD GMSL2 BEV-view camera, running MM-BEV on a NVIDIA Jetson AGX Orin.
Fig. 19: Representative deployment frame: six on-robot cameras (surrounding tiles) and the live BEV (center) with detected objects (orange) around the ego (green). MM-BEV runs on-device on the Jetson AGX Orin and detects the row of parked vehicles and the pedestrian crossing the robot’s forward path.
Fig. 20: Adaptive keyframe selection over a 240 s drive segment. (a) Robot speed from odometry. (b) Selected interval K (gray step); colored dots mark emitted keyframes by trigger reason. Mean Kˉ=2.83 across the full run.
The advancement of vision-only BEV (Bird's-Eye-View) perception is hindered by the fundamental trade-off between perception accuracy and deployment efficiency. We introduce Fast-BEV++, resolving this tension through two principles: Fast by Algorithm and Deployable by Design. By decomposing view transformation into a hardware-oriented Index-Gather-Reshape pipeline, Fast-BEV++ eliminates custom kernels while achieving no less than 3 times speedup over baseline methods. Empirically, Fast-BEV++ establishes a new state-of-the-art accuracy-speed trade-off on nuScenes, achieving 0.488 NDS while sustaining real-time inference at over 134 FPS. In particular, depth supervision yields consistent and tangible performance gains, maintaining the highest accuracy among comparable methods. The decomposed architecture enables seamless real-time deployment on production-level platforms, eliminating hardware constraints without loss of efficiency. Code and models are released on the linked project page.
Yuanpeng Chen, Hui Song, Sheng Yang +5
iMotion Automotive Technology (Suzhou) Co., Ltd · School of Data Science, Fudan University · Independent Researcher
Modern autonomous driving systems rely on bird's-eye-view (BEV) perception models that fuse camera and LiDAR inputs to detect objects in 3D space. These models are accurate, but they cannot be deployed through standard inference runtimes. The reason is an operator mismatch between dense convolutions (which runtimes handle well), sparse 3D convolutions (which runtimes cannot represent), and geometric scatter operations (which runtimes have no vocabulary for). Today, every sparse convolution library is CUDA-only and PyTorch-coupled, locking BEV deployment to a single vendor's hardware and a single execution framework. We present BEVPIPE, a framework for deploying multimodal BEV perception pipelines using portable GPU compute APIs and integrating them with production inference runtimes. BEVPIPE partitions the model into runtime-managed dense subgraphs and three external operator extensions (voxelizer, sparse encoder, BEV projector), connected through a shared GPU memory space. BEVPIPE achieves a 19.5x end-to-end speedup over conventional deployments while retaining 98.5% of reference mAP. We also showcase that BEVPIPE is portable across different GPU backends.
Efficient indoor LiDAR perception is challenging because mobile robots must understand cluttered three-dimensional environments under strict latency and memory constraints. Existing point-based and voxel-based methods often incur substantial computational overhead, whereas conventional bird's-eye-view (BEV) representations improve efficiency at the cost of discarding vertical geometric information. We present IndoorBEV, a lightweight LiDAR perception framework that mitigates this tradeoff through a height-aware BEV representation and geometry-conditioned feature fusion. IndoorBEV summarizes the vertical point distribution in each BEV cell using statistical height features and multi-frequency height encoding, allowing informative three-dimensional cues to be processed efficiently by two-dimensional convolutions. A lightweight encoder then integrates complementary geometric features with multi-scale local representations and compact global scene context. Decoupled dense prediction heads jointly produce semantic BEV maps and oriented object bounding boxes. IndoorBEV contains only 0.6M parameters and requires 2.3 MB of model storage. On an NVIDIA AGX Orin, it uses 21.52 MB of GPU memory per inference and achieves a mean latency of 169.6 ms under a 200 ms perception deadline, with a deadline miss ratio of 1.8%. Evaluations on simulated scenes, real-world robot scans, and an open-source indoor point-cloud dataset demonstrate a favorable tradeoff among perception accuracy, latency, and memory consumption. These results indicate that explicitly encoding vertical geometry within a compact BEV representation provides an effective approach to resource-efficient indoor LiDAR perception.