Bird's-eye-view (BEV) representations are the dominant paradigm for 3D perception in autonomous driving, providing a unified spatial canvas where detection and segmentation features are geometrically registered to the same physical coordinate system. However, existing radar-camera fusion methods treat these tasks in isolation, missing the opportunity for cross-task feature sharing: object-level geometric cues from detection can sharpen segmentation, while dense road-layout context from segmentation can anchor detection. We propose CTAB (Cross-Task Attention Bridge), a bidirectional module that exchanges features between detection and segmentation branches via multi-scale deformable attention in shared BEV space. CTAB is integrated into a multi-task framework with an Instance Normalization-based segmentation decoder and learnable BEV upsampling to provide a more detailed BEV representation. On nuScenes, CTAB improves segmentation on 7 classes over the joint multi-task baseline at essentially neutral detection. On a 4-class subset (drivable area, pedestrian crossing, walkway, vehicle), our joint multi-task model achieves 51.0 mIoU-4 while simultaneously providing competitive 3D detection.
Figures & tables
Fig. 1: Overall architecture. Multi-view camera images and radar point clouds are processed by a backbone that combines an image backbone with a radar backbone to create a BEV fusion similar to RCBEVDet, yielding a shared BEV feature Fbev∈R256×128×128 . The detection path operates directly on Fbev in BEV coordinates; the segmentation decoder ( ★ ) extracts Fseg with Instance Normalization. CTAB ( ★ ) exchanges features bidirectionally via gated multi-scale deformable attention. CenterHead predicts 3D boxes from BEV features; BEV Upsampling ( ★ ) refines segmentation to 200×200 . HUW balances both objectives with learned task uncertainty weights.
Fig. 2: Detailed architecture of the CTAB module. Detection and segmentation features are projected to a shared d=128 space and flattened. Two parallel MSDA blocks perform bidirectional cross-attention: in Seg → Det, detection features serve as queries attending to segmentation values; in Det → Seg, the roles are reversed. Output convolutions project back to original dimensions. Confidence gates σ(g) , initialized at ≈0.12 , scale the cross-attention output before residual addition, preventing destabilization of pretrained features.
Method
Mod.
Task
NDS ↑
mAP ↑
mIoU-7 ↑
mIoU ∗ ↑
Single-task detection
CenterFusion ∗ [ 34 ]
R+C
Det
45.3
33.2
–
–
CRAFT ∗ [ 35 ]
R+C
Det
51.7
41.1
–
–
CRN ∗ [ 4 ] (R50)
R+C
Det
56.0
49.0
–
–
RCBEVDet ∗ [ 5 ] (R50)
R+C
Det
56.8
45.3
–
–
HyDRa ∗ [ 15 ] (R50)
R+C
Det
58.5
49.4
–
–
TABLE I: Radar-camera (R+C) methods on nuScenes val. ∗ : single-task model. ‡ : cross-task interaction. mIoU-7: our 7-class evaluation. mIoU-4: our 4-class subset (drivable, ped. crossing, walkway, vehicle). Seg mIoU ∗ from other works uses their own class definitions: a 2-class (vehicle, drivable), b 3-class (vehicle, drivable, lane), c 7-class (different class set).
Configuration
NDS ↑
mAP ↑
mIoU-7 ↑
mIoU-4 ↑
+Params
RCBEVDet ∗ (det only)
56.8
45.3
–
–
–
(A) MTL baseline
55.7
44.4
42.9
50.4
+1.63M
(B) MTL + CTAB
55.6
44.4
43.5
51.0
+2.21M
CTAB improvement
Δ (B − A)
−0.1
0.0
+0.6
+0.6
+0.58M
TABLE II: Multi-task ablation on nuScenes val. All models use our RCBEVDet-based MTL framework with ResNet-50, SegDecoderV2 (IN, 3 blocks), and BEV Upsampling to 200×200 .
Fig. 3: Confidence gate evolution during training (Exp B). Both gates are initialized at σ(−2.0)≈0.12 . The segmentation gate σ(gseg) (purple) rises faster than the detection gate σ(gdet) (blue), indicating that the segmentation branch benefits more from cross-task features. This asymmetry emerges entirely from learning—both gates share identical architecture and initialization.
Method
Drivable
Carpark
Ped. Cross.
Walkway
Stop Line
Divider
Vehicle
mIoU-7
(A) Baseline
72.7
38.6
34.7
44.5
26.1
33.7
49.8
42.9
(B) + CTAB
73.3
38.4
36.5
44.4
27.9
34.4
49.8
43.5
Δ
+0.6
−0.2
+1.8
−0.1
+1.8
+0.7
0.0
+0.6
TABLE III: Per-class BEV segmentation IoU (%) on nuScenes val at 200×200 resolution. IoU@max reported. Divider combines road and lane dividers.
Fig. 4: Per-class BEV segmentation IoU on nuScenes val. Grouped bars compare Baseline (blue) and CTAB (purple) across the seven map classes; dotted lines mark the overall mIoU-7. CTAB’s gains concentrate on thin and sparse classes—pedestrian crossing ( +1.8 ), stop line ( +1.8 ), and divider ( +0.7 )—while dense classes move by at most 0.2 pp in either direction. Vehicle IoU is unchanged at 49.8 in both models, reflecting that 3D box-derived GT masks are already well-captured by detection-branch features.
Fig. 5: Qualitative comparison on three nuScenes validation scenes. Each row shows a different scene. Column 1 : Front camera with projected radar points and 3D detection boxes. Column 2 : Ground Truth BEV segmentation. Columns 3–4 : Baseline and CTAB BEV segmentation with detected 3D boxes (colored by class) and radar point cloud (orange dots). Per-scene mIoU and the CTAB improvement ( Δ ) are shown.
Accurate 3D object detection for autonomous driving requires complementary sensors. Cameras provide dense semantics but unreliable depth, while millimeter-wave radar offers precise range and velocity measurements with sparse geometry. We propose MMF-BEV, a radar-camera BEV fusion framework that leverages deformable attention for cross-modal feature alignment on the View-of-Delft (VoD) 4D radar dataset [1]. MMF-BEV builds a BEVDepth [2] camera branch and a RadarBEVNet [3] radar branch, each enhanced with Deformable Self-Attention, and fuses them via a Deformable Cross-Attention module. We evaluate three configurations: camera-only, radar-only, and hybrid fusion. A sensor contribution analysis quantifies per-distance modality weighting, providing interpretable evidence of sensor complementarity. A two-stage training strategy - pre-training the camera branch with depth supervision, then jointly training radar and fusion modules stabilizes learning. Experiments on VoD show that MMF-BEV consistently outperforms unimodal baselines and achieves competitive results against prior fusion methods across all object classes in both the full annotated area and near-range Region of Interest.
Radar-camera BEV perception often suffers from degraded performance when evaluated across datasets, as changes in driving scenes, sensor configurations, and environmental conditions can alter both the input observations and the internal fused representations. This work studies this issue from the perspective of source-domain variation modeling, aiming to improve the robustness of BEV-based 3D detectors without relying on target-domain samples. We introduce a framework that characterizes visual scene variations in the frequency domain and uses them to synthesize diverse source-domain views. By comparing the resulting fused BEV representations, the framework further captures how image-level variations influence multi-modal BEV features. These variation patterns are then used to regularize the detector, encouraging the learned fusion space to remain stable under latent scene changes. The proposed method is applied only during training and leaves the inference pipeline unchanged. Experiments on cross-dataset radar-camera 3D detection between View-of-Delft and TJ4DRadSet demonstrate consistent improvements over multiple BEV fusion backbones, and the gains remain effective when a small amount of target-domain data is available.
Nowadays, an increasing number of works fuse LiDAR and RGB data in the bird's-eye view (BEV) space for 3D object detection in autonomous driving systems. However, existing methods suffer from over-reliance on the LiDAR branch, with insufficient exploration of RGB information. To tackle this issue, we propose Fusion4CA, which is built upon the classic BEVFusion framework and dedicated to fully exploiting visual input with plug-and-play components. Specifically, a contrastive alignment module is designed to calibrate image features with 3D geometry, and a camera auxiliary branch is introduced to mine RGB information sufficiently during training. For further performance enhancement, we leverage an off-the-shelf cognitive adapter to make the most of pretrained image weights, and integrate a standard coordinate attention module into the fusion stage as a supplementary boost. Experiments on the nuScenes dataset demonstrate that our method achieves 69.7% mAP with only 6 training epochs and a mere 3.48% increase in inference parameters, yielding a 1.2% improvement over the baseline which is fully trained for 20 epochs. Extensive experiments in a simulated lunar environment further validate the effectiveness and generalization of our method. Our code will be released through Fusion4CA.
Kang Luo, Xin Chen, Yangyi Xiao +1
IRMV Lab, the Department of Automation, Shanghai Jiao Tong University.