MM-BEV: Enhancing Timeliness by Computing Where and When it Matters
Authors: Liangkai Liu, Kang G. Shin
Organizations: Department of Computer Science, Texas Tech University, USA · Department of Computer Science and Engineering, University of Michigan, USA
Multimodal bird's-eye-view (BEV) perception combines LiDAR depth accuracy with dense camera semantics, but its high computational cost and imperfect sensing conditions make real-time deployment challenging. Existing methods largely compress individual detectors and overlook three opportunities: structured sparsity within camera and LiDAR inputs, timing misalignment between modalities, and the fact that many detected objects do not affect the planner's immediate action. We present MM-BEV, a real-time multimodal BEV system guided by a simple principle: compute where and when it matters. MM-BEV divides perception into mandatory work for safety-critical objects within braking distance of the ego vehicle and with short time-to-collision (TTC), and optional work for less urgent regions. It prioritizes mandatory work and reduces or sheds optional work under tight compute budgets. MM-BEV integrates four mechanisms: (1) a criticality-ranked temporal ROI selector based on motion-extrapolated detections from prior frames; (2) sparse, ROI-aware feature extraction using shared-shape camera crops at context-adaptive resolution and ROI-aware LiDAR voxelization; (3) a latency-aware coordinator that adapts LiDAR sweeps, image resolution, and keyframes according to scene dynamics and TTC; and (4) an asynchronous scheduler that decouples sensing from inference and skips stale frames. On nuScenes, MM-BEV reduces inference latency by 1.96x and end-to-end latency by 2.93x, with no loss in geometry-critical recall and only a 0.2 percentage-point drop in safety-critical recall. On a Clearpath Husky A300 equipped with an Ouster-128 LiDAR, BEV cameras, and a Jetson AGX Orin, MM-BEV further reduces mean latency by 2.11x, demonstrating its potential for real-world autonomous systems.
Figures & tables
Fig. 1: End-to-end pipeline for BEV perception.
Fig. 2: Object-presence probability heatmaps across the six nuScenes cameras. Dark areas indicate regions with low per-region probability Pˉ ; percentages (if shown) denote the share of regions with Pˉ<0.1 .
Fig. 3: mAP and inference latency with different LiDAR sweeps.
Fig. 4: Timestamp distributions of LiDAR (20 Hz) and BEV cameras (12 Hz) in nuScenes. Sampling-rate mismatch and asynchronous publication create inherent offsets ( ≳50 ms) even before bus/jitter effects.
Fig. 5: An example of camera–LiDAR misalignment. Two sync groups are formed: {T1cam,T2lidar} and {T2cam,T3lidar} with Δt1,2=350 ms and Δt2,3=150 ms. Objects move and FOV content changes across these offsets, leading to conflicting cross-modal evidence.
Fig. 6: Criticality analysis on nuScenes v1.0-mini ( 18,364 annotations). (a) Only 11% of objects are mandatory (safety-critical); 89% are optional. (b) Per-class mandatory fractions. (c) Critical objects concentrate at close range; optional ones span the full detection envelope. (d) In BEV, mandatory objects (red) cluster in the ego’s forward path while optional objects (blue) populate the periphery.
Fig. 7: MM-BEV system overview.
Fig. 8: Aggregation of LiDAR sweeps from a nuScenes scene without (a) and with (b) ego-motion compensation, above-ground returns only. Color encodes sweep age (red = current sweep, blue = oldest).
Fig. 9: ROI-aware voxelization and feature extraction.
Fig. 10: LiDAR latency vs. non-empty voxel count, profiled on the BEVFusion LiDAR-only branch over 84 runs ( 12 nuScenes scenes ×7 sub-sample rates). (a) Total LiDAR latency is linear in nv ( R2=0.99 ). (b) Voxelize and the sparse encoder scale with nv .
Fig. 11: ROS framework for MM-BEV .
Config
Sensors
Temporal ROI
Adaptive res.
Async sched.
BEVFusion [ liu2023bevfusion ]
L+C
×
×
×
Scheduler Only
L+C
×
×
✓
ROI
L+C
✓
×
×
ROI+LowRes
L+C
✓
fixed-low
×
ROI+Adapt+LowRes
L+C
✓
adaptive
×
MM-BEV
L+C
✓
adaptive
✓
TABLE I: Evaluated configurations.
Fig. 12: Per-frame inference latency split by keyframe and ROI-frame mode, for the LiDAR+Camera (fusion) and LiDAR-only detectors. Baseline must run full inference every cycle; MM-BEV runs fast ROI inference on most frames and full inference only on keyframes.
Fig. 13: Per-stage breakdown of inference time.
Fig. 14: Recall by criticality bucket. Geometry-critical recall is preserved exactly at 100% across all configurations; safety-critical recall drops at most 0.7 pp and all-object recall at most 2.3 pp, under the most aggressive compression.
Config
Per-frame lat. (ms)
All-obj R (%)
Geom-crit R (%)
Safety-crit R (%)
BEVFusion
202
79.9
100.0
94.9
ROI+Adapt+LowRes
95
78.4
100.0
95.2
MM-BEV
104
79.5
100.0
94.7
MM-BEV+LowRes
92
77.6
100.0
94.2
LiDAR Baseline
136
76.8
100.0
93.5
LiDAR MM-BEV
70
76.8
100.0
93.1
TABLE II: Accuracy vs. per-frame inference latency.
Fig. 15: Per-frame latency vs. recall for all ablation configurations (geom-critical preserved at 100% ; right panel shows safety-critical).
Fig. 17: End-to-end ROS pipeline results. Left : mean e2e latency per processed frame (bars show mean, error bars show ± 1 s.d., jitter shows all individual frames). Right : total frames processed per scene. The asynchronous scheduler is the dominant contributor to both gains.
Config
Frames/scene
E2E lat. mean (ms)
E2E lat. p95 (ms)
Speedup
BEVFusion
10
1006
1455
–
Scheduler Only
56
371
498
2.71×
MM-BEV
60
344
434
2.93×
TABLE III: End-to-end ROS pipeline summary.
Fig. 18: Deployment platform: Clearpath Husky A300 with Ouster OS1-128 LiDAR and e-consystems AR0234 FHD GMSL2 BEV-view camera, running MM-BEV on a NVIDIA Jetson AGX Orin.
Fig. 19: Representative deployment frame: six on-robot cameras (surrounding tiles) and the live BEV (center) with detected objects (orange) around the ego (green). MM-BEV runs on-device on the Jetson AGX Orin and detects the row of parked vehicles and the pedestrian crossing the robot’s forward path.
Fig. 20: Adaptive keyframe selection over a 240 s drive segment. (a) Robot speed from odometry. (b) Selected interval K (gray step); colored dots mark emitted keyframes by trigger reason. Mean Kˉ=2.83 across the full run.