Radar-Camera BEV Multi-Task Learning with Cross-Task Attention Bridge for Joint 3D Detection and Segmentation

Ahmet İnanç; Özgür Erkent

cs.CVApr 14, 2026

Radar-Camera BEV Multi-Task Learning with Cross-Task Attention Bridge for Joint 3D Detection and Segmentation

Authors: Ahmet İnanç, Özgür Erkent

Organizations: Hacettepe University, Ankara, Turkey.

Abstract

Bird's-eye-view (BEV) representations are the dominant paradigm for 3D perception in autonomous driving, providing a unified spatial canvas where detection and segmentation features are geometrically registered to the same physical coordinate system. However, existing radar-camera fusion methods treat these tasks in isolation, missing the opportunity for cross-task feature sharing: object-level geometric cues from detection can sharpen segmentation, while dense road-layout context from segmentation can anchor detection. We propose CTAB (Cross-Task Attention Bridge), a bidirectional module that exchanges features between detection and segmentation branches via multi-scale deformable attention in shared BEV space. CTAB is integrated into a multi-task framework with an Instance Normalization-based segmentation decoder and learnable BEV upsampling to provide a more detailed BEV representation. On nuScenes, CTAB improves segmentation on 7 classes over the joint multi-task baseline at essentially neutral detection. On a 4-class subset (drivable area, pedestrian crossing, walkway, vehicle), our joint multi-task model achieves 51.0 mIoU-4 while simultaneously providing competitive 3D detection.

Figures & tables

Panel of Figure 1 — Fig. 1: Overall architecture. Multi-view camera images and radar point clouds are processed by a backbone that combines an image backbone with a radar backbone to create a BEV fusion similar to RCBEVDet, yielding a shared BEV feature $\mathbf{F}_{\text{bev}}\in\mathbb{R}^{256\times 128\times 128}$ . The detection path operates directly on $\mathbf{F}_{\text{bev}}$ in BEV coordinates; the segmentation decoder ( $\bigstar$ ) extracts $\mathbf{F}_{\text{seg}}$ with Instance Normalization. CTAB ( $\bigstar$ ) exchanges features bidirectionally via gated multi-scale deformable attention. CenterHead predicts 3D boxes from BEV features; BEV Upsampling ( $\bigstar$ ) refines segmentation to $200\times 200$ . HUW balances both objectives with learned task uncertainty weights.

Panel of Figure 2 — Fig. 2: Detailed architecture of the CTAB module. Detection and segmentation features are projected to a shared $d\!=\!128$ space and flattened. Two parallel MSDA blocks perform bidirectional cross-attention: in Seg $\to$ Det, detection features serve as queries attending to segmentation values; in Det $\to$ Seg, the roles are reversed. Output convolutions project back to original dimensions. Confidence gates $\sigma(g)$ , initialized at ${\approx}0.12$ , scale the cross-attention output before residual addition, preventing destabilization of pretrained features.

Single-task detection
Method	Mod.	Task	NDS $\uparrow$	mAP $\uparrow$	mIoU-7 $\uparrow$	mIoU ∗ $\uparrow$
CenterFusion ∗ [ 34 ]	R+C	Det	45.3	33.2	–	–
CRAFT ∗ [ 35 ]	R+C	Det	51.7	41.1	–	–
CRN ∗ [ 4 ] (R50)	R+C	Det	56.0	49.0	–	–
RCBEVDet ∗ [ 5 ] (R50)	R+C	Det	56.8	45.3	–	–
HyDRa ∗ [ 15 ] (R50)	R+C	Det	58.5	49.4	–	–

TABLE I: Radar-camera (R+C) methods on nuScenes val.

*

: single-task model.

\ddagger

: cross-task interaction. mIoU-7: our 7-class evaluation. mIoU-4: our 4-class subset (drivable, ped. crossing, walkway, vehicle). Seg mIoU ∗ from other works uses their own class definitions: a 2-class (vehicle, drivable), b 3-class (vehicle, drivable, lane), c 7-class (different class set).

CTAB improvement
Configuration	NDS $\uparrow$	mAP $\uparrow$	mIoU-7 $\uparrow$	mIoU-4 $\uparrow$	+Params
RCBEVDet ∗ (det only)	56.8	45.3	–	–	–
(A) MTL baseline	55.7	44.4	42.9	50.4	+1.63M
(B) MTL + CTAB	55.6	44.4	43.5	51.0	+2.21M
$\Delta$ (B $-$ A)	$-0.1$	$0.0$	+0.6	+0.6	+0.58M

TABLE II: Multi-task ablation on nuScenes val. All models use our RCBEVDet-based MTL framework with ResNet-50, SegDecoderV2 (IN, 3 blocks), and BEV Upsampling to

200\times 200

Fig. 3: Confidence gate evolution during training (Exp B). Both gates are initialized at $\sigma(-2.0)\approx 0.12$ . The segmentation gate $\sigma(g_{\text{seg}})$ (purple) rises faster than the detection gate $\sigma(g_{\text{det}})$ (blue), indicating that the segmentation branch benefits more from cross-task features. This asymmetry emerges entirely from learning—both gates share identical architecture and initialization.

Method	Drivable	Carpark	Ped. Cross.	Walkway	Stop Line	Divider	Vehicle	mIoU-7
(A) Baseline	72.7	38.6	34.7	44.5	26.1	33.7	49.8	42.9
(B) + CTAB	73.3	38.4	36.5	44.4	27.9	34.4	49.8	43.5
$\Delta$	+0.6	$-0.2$	+1.8	$-0.1$	+1.8	+0.7	$0.0$	+0.6

TABLE III: Per-class BEV segmentation IoU (%) on nuScenes val at

200\times 200

resolution. IoU@max reported. Divider combines road and lane dividers.

Fig. 4: Per-class BEV segmentation IoU on nuScenes val. Grouped bars compare Baseline (blue) and CTAB (purple) across the seven map classes; dotted lines mark the overall mIoU-7. CTAB’s gains concentrate on thin and sparse classes—pedestrian crossing ( $+1.8$ ), stop line ( $+1.8$ ), and divider ( $+0.7$ )—while dense classes move by at most $0.2$ pp in either direction. Vehicle IoU is unchanged at 49.8 in both models, reflecting that 3D box-derived GT masks are already well-captured by detection-branch features.

Fig. 5: Qualitative comparison on three nuScenes validation scenes. Each row shows a different scene. Column 1 : Front camera with projected radar points and 3D detection boxes. Column 2 : Ground Truth BEV segmentation. Columns 3–4 : Baseline and CTAB BEV segmentation with detected 3D boxes (colored by class) and radar point cloud (orange dots). Per-scene mIoU and the CTAB improvement ( $\Delta$ ) are shown.

Autonomous Driving Perception

Autonomous driving perception enables vehicles to detect and interpret road users, drivable space, and hazards using onboard and shared sensors.

See all 379 papers

Multi-Modal Sensor Fusion using Hybrid Attention for Autonomous Driving
Apr 6, 2026Mayank Mayank, Bharanidhar Duraisamy, Florian Geiß +1Camera-Radar Fusion 3D Object Detection
Research and Development, Mercedes-Benz AG, Germany · Department of Computer Science, University of Freiburg, Germany
Understanding Cross-Sensor Feature Variations for Generalizable 3D Perception
Jun 10, 2026Xin Qiu, Wenjie Liu, Fuyuan Ai +3Camera-Radar Fusion Multimodal Robustness
Zhejiang University
Fusion4CA: Boosting 3D Object Detection via Comprehensive Image Exploitation
Mar 5, 2026Kang Luo, Xin Chen, Yangyi Xiao +13D Object Detection Camera-LiDAR Fusion
IRMV Lab, the Department of Automation, Shanghai Jiao Tong University.

Apr 6, 2026J/K move · Enter open · S save

Multi-Modal Sensor Fusion using Hybrid Attention for Autonomous Driving

Mayank Mayank, Bharanidhar Duraisamy, Florian Geiß, Abhinav Valada

Research and Development, Mercedes-Benz AG, Germany · Department of Computer Science, University of Freiburg, Germany

Camera-Radar Fusion 3D Object Detection

Open paper AskPDF arXiv

Accurate 3D object detection for autonomous driving requires complementary sensors. Cameras provide dense semantics but unreliable depth, while millimeter-wave radar offers precise range and velocity measurements with sparse geometry. We propose MMF-BEV, a radar-camera BEV fusion framework that leverages deformable attention for cross-modal feature alignment on the View-of-Delft (VoD) 4D radar dataset [1]. MMF-BEV builds a BEVDepth [2] camera branch and a RadarBEVNet [3] radar branch, each enhanced with Deformable Self-Attention, and fuses them via a Deformable Cross-Attention module. We evaluate three configurations: camera-only, radar-only, and hybrid fusion. A sensor contribution analysis quantifies per-distance modality weighting, providing interpretable evidence of sensor complementarity. A two-stage training strategy - pre-training the camera branch with depth supervision, then jointly training radar and fusion modules stabilizes learning. Experiments on VoD show that MMF-BEV consistently outperforms unimodal baselines and achieves competitive results against prior fusion methods across all object classes in both the full annotated area and near-range Region of Interest.

Abstract

Autonomous Driving Perception

Explore similar work

Multi-Modal Sensor Fusion using Hybrid Attention for Autonomous Driving

Understanding Cross-Sensor Feature Variations for Generalizable 3D Perception

Fusion4CA: Boosting 3D Object Detection via Comprehensive Image Exploitation