TADP: Task-Aware Deformable Prediction for Single-Stage 3D Object Detection
Authors: Su Wang, Yaochen Li, Min Yang, Jiaohao Nie, Chang Liu, Yuehu Liu
Organizations: School of Software Engineering Xi’an Jiaotong University Xi’an, China · School of Electrical and Electronic Engineering Nanyang Technological University Singapore · CSSC Systems Engineering Research Institute Beijing, China
Most single-stage 3D object detectors complete different tasks with the same extracted features. Nevertheless, it is impossible to project features into a common space that is adaptive for all the tasks. We present a novel task-aware deformable prediction (TADP) method for single-stage 3D object detection to solve this problem. Firstly, a triple feature refinement aggregation module is designed to extract three-level features adaptively. Additionally, we design the multi-scale feature aggregation block to fuse multi-scale features in a scale-aware manner. Finally, the prediction of each task is deformed with the designed plug-and-play task-aware deformation head. It can percept the emphasis and interaction of each task. We also designed three different deformation modules. The experimental results demonstrate that the proposed deformation head shows good results on other detection methods. The experimental results on the KITTI dataset demonstrate that the car mAP is 80.91%, surpassing many state-of-the-art methods on the KITTI benchmark.
Figures & tables
Fig. 1: Visualization of detection in street scenes. It shows the different detection results of SECOND [ 5 ] and our TADP. We use red arrows to indicate the biased optimization of our method compared to SECOND detection.
Fig. 2: The figure shows the pipeline of our method TADP. (a) downsample and encode the point clouds. (b) show TFRA structure refine the three-scales features. (c) show the MSFA structure, scale maps Features and fused with SFA. (d) show the TADH, predict the deformation map, and deform the task prediction.
Fig. 3: The pipeline of TADH. The first branch of the features directly generates traditional task predictions through predict line. The second branch through P-Stack generates DMap. The upper features provide height attention. Finally, the deformation to the prediction result is generated through the Deform-line.
Fig. 4: The process and principle of PStack and DMap generating each task’s deformation. Below shows the raw point cloud and proposals. Above shows the process of task alignment
Fig. 5: Comparative experiments between different tasks and different deform modules. Wei. , dec. , add. , and null . We show how much the accuracy increases as a color chart. Based on the ‘ null ’ color.
Fig. 6: Our 3D detection results on the KITTI validation set are visualized. The ground truth is the green box, and the prediction box is the red box. The 3D detection box is projected to an RGB image.
Type
Method
Sens
AP3D(%)
Easy
Mod
Hard
2-stage
AVOD(2018) [ 21 ]
LIDAR+RGB
83.07
71.76
65.73
PI-RCNN(2020) [ 22 ]
LIDAR+RGB
84.37
74.82
70.03
PointRCNN(2019)
LIDAR
86.96
75.64
70.70
F-ConvNet(2019) [ 23 ]
LIDAR+RGB
87.36
76.39
66.69
UberATG-MMF(2019)
LIDAR+RGB
88.40
77.43
70.22
TABLE I: AP11-based comparation with SOTA on KITTI dataset benchmark for car class detection
TFRA
MSFA
TADH
TADH†
AP3D(%)
Easy
Mod
Hard
87.52
77.21
74.36
✓
88.27
78.41
75.89
✓
✓
89.26
79.37
76.76
✓
✓
✓
89.71
79.95
77.31
✓
✓
✓
90.03
80.62
78.64
TABLE II: Ablation experiments of TFRA and MSFA and TADH in TADP on KITTI validation dataset.
PointRCNN
Part-A2
MGAF-3DSSD
3D-CVF
Ours
time(ms)
643
80
80
85
40.53
mAP
77.67
79.7
80.51
80.79
80.91
TABLE III: Runtime (in milliseconds) and mAP (in AP11) compared to recent state-of-the-art secondary detectors
Stack_num
0
1
2
3
4
5
6
Improvement/ AP3D(%)
78.41
+0.11
+0.23
+0.38
+0.49
+0.58
+0.64
TABLE IV: Comparitive test of convolution layer parameter N in Percetion Stack
Method
AP3D (%)
Easy
Mod
Hard
SECOND
83,52
73.63
67.21
SECOND+DH
84.44
74.25
67.61
Improvement
+0.92
+0.62
+0.40
VoxelNet
79.62
65.97
59.71
VoxelNet+DH
80.78
66.82
60.35
TABLE V: Comparative test of car class detection after TADH insertion in some one-stage networks.
3D object detection in point cloud data remains a challenging task due to the sparsity and lack of global structure inherent in the input. In this work, we propose a novel Multi-Scale Attention (MSA) mechanism integrated into the 3DETR architecture to better capture both local geometry and global context. Our method introduces an upsampling operation that generates high-resolution feature maps, enabling the network to better detect smaller and semantically related objects. Experiments conducted on the ScanNetv2 dataset demonstrate that our 3DETR + MSA model improves detection performance, achieving a gain of almost 1% in mAP@25 and 4.78% in mAP@50 over the baseline. While applying MSA to the 3DETR-m variant shows limited improvement, our analysis reveals the importance of adapting the upsampling strategy for lightweight models. These results highlight the effectiveness of combining hierarchical feature extraction with attention mechanisms in enhancing 3D scene understanding.
Mustaqeem Khan, Aidana Nurakhmetova, Wail Gueaieb +1
College of Information Technology-United Arab Emirates University (UAEU), Al Ain, UAE · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · School of Elec. Eng. & Comp. Sc., University of Ottawa, Ottawa, ON, Canada
Single-stage fully sparse 3D object detectors rely on point clouds data to detect objects in autonomous driving scenarios. However, the sparsity and incompleteness of point clouds significantly limit the performance of 3D object detection. To address this issue, this paper proposes a point clouds completion method specifically designed for single-stage fully sparse detectors. The entire shape-prior-based completion process consists of two consecutive steps. In the first step, we design a novel Instance Selection module, which is capable of identifying point clouds corresponding to foreground objects even when the baseline model does not generate proposals, while effectively ignoring the point clouds of background regions. In the second step, we introduce a novel Alignment-Based Point Completion module, which aligns the point clouds of foreground objects with prototypes in terms of both their centers and orientations. Subsequently, points are selected from the prototype to fill in the missing parts of the foreground object. We evaluated our method on two single-stage fully sparse detectors using the KITTI dataset. The experimental results demonstrate that the proposed method significantly improves the detection performance, confirming its effectiveness and generalizability.
Kaizheng Wang, Mingqian Ji, Jian Yang +1
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China
Bird's-eye-view (BEV) representations are the dominant paradigm for 3D perception in autonomous driving, providing a unified spatial canvas where detection and segmentation features are geometrically registered to the same physical coordinate system. However, existing radar-camera fusion methods treat these tasks in isolation, missing the opportunity for cross-task feature sharing: object-level geometric cues from detection can sharpen segmentation, while dense road-layout context from segmentation can anchor detection. We propose CTAB (Cross-Task Attention Bridge), a bidirectional module that exchanges features between detection and segmentation branches via multi-scale deformable attention in shared BEV space. CTAB is integrated into a multi-task framework with an Instance Normalization-based segmentation decoder and learnable BEV upsampling to provide a more detailed BEV representation. On nuScenes, CTAB improves segmentation on 7 classes over the joint multi-task baseline at essentially neutral detection. On a 4-class subset (drivable area, pedestrian crossing, walkway, vehicle), our joint multi-task model achieves 51.0 mIoU-4 while simultaneously providing competitive 3D detection.