Organizations: Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, P.R.China · Intelligent Transportation Thrust of the Systems Hub, Hong Kong University of Science and Technology (GZ), Guangzhou, P.R.China · Amazon Alexa AI, Bellevue, Washington, United States
Monocular 3D object detection (M3OD) is intrinsically ill-posed, hence training a high-performance deep learning based M3OD model requires a humongous amount of labeled data with complicated visual variation from diverse scenes, variety of objects and camera poses.However, we observe that, due to strong human bias, the three independent entities, i.e., object, scene, and camera pose, are always tightly entangled when an image is captured to construct training data. More specifically, specific 3D objects are always captured in particular scenes with fixed camera poses, and hence lacks necessary diversity. Such tight entanglement induces the challenging issues of insufficient utilization and overfitting to uniform training data. To mitigate this, we propose an online object-scene-camera decomposition and recomposition data manipulation scheme to more efficiently exploit the training data. We first fully decompose training images into textured 3D object point models and background scenes in an efficient computation and storage manner. We then continuously recompose new training images in each epoch by inserting the 3D objects into the freespace of the background scenes, and rendering them with perturbed camera poses from textured 3D point representation. In this way, the refreshed training data in all epochs can cover the full spectrum of independent object, scene, and camera pose combinations. This scheme can serve as a plug-and-play component to boost M3OD models, working flexibly with both fully and sparsely supervised settings. In the sparsely-supervised setting, objects closest to the ego-camera for all instances are sparsely annotated. We then can flexibly increase the annotated objects to control annotation cost. For validation, our method is widely applied to five representative M3OD models and evaluated on both the KITTI and the more complicated Waymo datasets.
Figures & tables
Figure 1: Implementation of our data manipulation scheme on four released models under both fully and sparsely supervised settings. The x -axis represents the percentage of annotated objects used in our scheme out of all objects in the KITTI train split. The y -axis represents the moderate level of AP3D in the KITTI validation split. Our method significantly boosts the performance of the base models in both the fully (i.e., 100% annotation ratio) and the sparsely (i.e., 10%, 20%, and 50%) supervised settings. Our method with only 10% annotations even achieves on-par performance with the fully-supervised setting of the first three base models.
Figure 2: Illustration of our online object-scene-camera decomposition and recomposition data manipulation scheme. We first fully decompose training images into textured 3D object point models and background scenes in an efficient computation and storage manner. We then continuously recompose new training images in each epoch by inserting the 3D objects into the freespace of the background scenes, and rendering them with perturbed camera poses from textured 3D point representation. In this way, the refreshed training data in all epochs can cover the full spectrum of independent object, scene, and camera combinations.
Figure 3: Object point model rectification. The CAD model is intended to aid visualization.
Figure 4: Object position resampling. Using a 3D textured point representation, objects can be flexibly resampled to new positions.
Figure 5: Freespace generation process. For each row of the freespace map in the polar coordinates system, we scan each column to complete the empty area.
Figure 6: Illustration of the sparsely-supervised setting. Based on our scheme, only the objects closest to the camera will be labeled to reduce annotation costs.
Method
AP3Dtest
APBEVtest
AP3Dval
APBEVval
easy
mod
hard
easy
mod
hard
easy
mod
hard
easy
mod
hard
MonoDLE
17.23
12.23
10.29
24.79
18.89
16.00
19.48
15.21
13.57
26.03
21.10
18.59
+ours
25.55
18.04
15.45
34.51
24.82
20.86
26.57
20.42
17.34
36.37
26.74
23.07
improvement
+8.32
+5.81
+5.16
+9.72
+5.93
+4.86
+7.09
+5.21
+3.77
+10.34
+5.64
+4.48
GUPNet
22.26
15.02
13.12
30.29
21.19
18.20
21.95
16.94
14.43
29.57
23.49
20.40
+ours
28.60
19.35
16.35
38.18
26.51
23.01
26.91
20.45
17.45
35.17
26.22
22.84
Table 1: Comparison with four base models on the car category in the KITTI official test set and validation split.
AP3D
APBEV
Method
Venues
easy
mod
hard
easy
mod
hard
MonoDLE ( Ma et al., 2021 )
CVPR2021
17.23
12.23
10.29
24.79
18.89
16.00
CaDDN ( Reading et al., 2021 )
CVPR2021
19.17
13.41
11.46
27.94
18.91
17.19
MonoFlex ( Zhang et al., 2021 )
CVPR2021
19.94
13.89
12.07
28.23
19.75
16.89
GUPNet ( Lu et al., 2021 )
ICCV2021
22.26
15.02
13.12
30.29
21.19
18.20
MonoDistill ( Chong et al., 2022 )
ICLR2022
22.97
16.03
13.60
31.87
22.59
19.72
Table 2: Comparison with state-of-the-art methods on the car category in the KITTI official test set. Following the KITTI protocol, methods are ranked based on their performance under the moderate difficulty setting. The best results are in bold , and the best results of the baselines are underlined .
Method
Annotation
AP3D
APBEV
easy
mod
hard
easy
mod
hard
MonoDLE
Full
19.48
15.21
13.57
26.03
21.10
18.59
10%
6.67(-12.81)
4.67(-10.54)
3.61(-9.96)
10.10(-15.93)
6.91(-14.19)
5.54(-13.05)
10% w / ours
20.73(+1.25)
15.73(+0.52)
12.82(-0.75)
28.00(+1.97)
20.69(-0.41)
16.37(-2.22)
GUPNet
Full
21.95
16.94
14.43
29.57
23.49
20.40
10%
10.25(-11.70)
6.06(-10.88)
4.70(-9.73)
15.17(-14.40)
8.88(-14.61)
7.13(-13.27)
Table 3: Comparison of fully-supervised and sparsely-supervised results on the KITTI validation set. Our scheme with fewer annotations (especially 10% for the first three base models) achieves comparable performance to the fully-supervised training of the base models. When both use the same sparse annotations, our scheme significantly outperforms the base models.
AP3D
APH3D
IOU3D
Difficulty
Method
Vehicle
Pedestrian
Cyclist
Vehicle
Pedestrian
Cyclist
DID-M3D
2.64
1.49
1.05
2.63
1.36
1.02
+ours
3.73(+41%)
1.91(+28%)
2.78(+165%)
3.71(+41%)
1.72(+26%)
2.70(+165%)
MonoDETR
1.63
0.98
1.87
1.61
0.86
1.49
Level_1
+ours
2.52(+55%)
1.34(+37%)
2.14(+14%)
2.49(+55%)
1.2(+40%)
2.1(+41%)
DID-M3D
2.48
1.36
1.01
2.46
1.24
0.98
Table 4: Comparison of base models and our scheme on three categories in the Waymo-Mono Dataset.
Method
DID-M3D
MonoDETR
Annotation
Full
10%
10% w/ours
Full
10%
10% w /ours
AP3D
IOU3D@0.7
Level_1
2.64
0.71(-1.93)
2.69(+0.05)
1.63
0.77(-0.86)
1.53(-0.10)
Level_2
2.48
0.67(-1.81)
2.52(+0.04)
1.53
0.72(-0.81)
1.44(-0.09)
IOU3D@0.5
Level_1
11.83
4.37(-7.46)
11.24(-0.59)
8.03
4.51(-3.52)
7.47(-0.56)
Level_2
11.09
4.09(-7.00)
10.54(-0.55)
7.54
4.23(-3.31)
7.01(-0.53)
APH3D
IOU3D@0.7
Level_1
2.63
0.70(-1.93)
2.66(+0.03)
1.61
0.76(-0.85)
1.51(-0.10)
Table 5: Comparison of fully supervised and sparsely supervised (10% annotation ratio) settings on Waymo-Mono dataset (vehicle class). Our scheme with 10% annotation achieves comparable performance to the fully-supervised setting of the base models.
Setting
Difficulty
Method
AP3D
APH3D
0-30m
30-50m
50- ∞ m
0-30m
30-50m
50- ∞ m
fully-supervised
Level_1
DID-M3D
27.55
5.47
0.89
27.35
5.42
0.88
ours
33.68
6.53
0.75
33.41
6.47
0.74
Level_2
DID-M3D
27.45
5.27
0.78
27.25
5.22
0.76
ours
33.55
6.28
0.65
33.29
6.23
0.64
sparsely-supervised (10%)
Level_1
DID-M3D
11.84
1.41
0.18
11.64
1.38
0.17
Table 6: Comparison of both fully-supervised and sparsely supervised performance at different distances. The result is reported on DID-M3D with an IoU threshold of 0.5.
Supervision
Full
Sparse(10%)
Category
Method
mAP ↑
mATE ↓
mASE ↓
mAOE ↓
NDS* ↑
mAP ↑
mATE ↓
mASE ↓
mAOE ↓
NDS* ↑
Vehicle
PETR
0.497
0.704
0.156
0.126
0.584
0.074
1.250
0.228
0.764
0.205
+ours
0.508
0.684
0.153
0.114
0.596(+0.012)
0.435
0.769
0.162
0.205
0.528(+0.323)
Pedestrian
PETR
0.457
0.776
0.266
0.859
0.411
0.069
1.173
0.287
1.454
0.153
+ours
0.492
0.715
0.262
0.606
0.482(+0.071)
0.401
0.812
0.270
1.087
0.353(+0.200)
Cyclist
PETR
0.318
0.713
0.229
0.569
0.407
0.000
1.288
0.542
1.401
0.077
Table 7: Comparison of PETR ( Liu et al., 2022b ) and our scheme on Waymo-Ring Dataset.
Object-Scene Recomposition
Object-Scene Decomposition
Camera Pose Pertubation
AP3D
APBEV
easy
mod
hard
easy
mod
hard
-
-
-
15.89
12.76
10.57
23.14
18.17
15.31
✓
-
-
20.69
16.92
14.34
27.95
21.96
19.22
✓
✓
-
23.30
19.02
16.35
32.00
25.12
21.99
✓
✓
✓
26.57
20.42
17.34
36.37
26.74
23.07
Table 8: Ablation study of object-scene-camera decomposition and recomposition operations.
Decomposition Ratio(%)
AP3D
APBEV
easy
mod
hard
easy
mod
hard
0
15.89
12.76
10.57
23.14
18.17
15.31
25
18.24
14.96
12.64
24.31
19.88
17.35
50
19.92
15.71
14.01
27.31
21.80
18.94
100
22.44
17.97
15.28
31.94
24.59
21.70
Table 9: Ablation study of different decomposition ratios.
Counts of Recomposition
AP3D
APBEV
easy
mod
hard
easy
mod
hard
0
15.89
12.76
10.57
23.14
18.17
15.31
2
18.89
14.86
12.51
26.67
20.44
17.40
50
21.11
17.00
14.47
29.17
22.86
19.78
200
23.30
19.02
16.35
32.00
25.12
21.99
Table 10: Ablation study of different counts of recomposition per scene.
Method
Object-scene composition per scene
plain
≈ 4
+ours
≈ 700
Table 11: Ablation study of scene utilization before and after applying our scheme.
Figure 7: Visualization of recomposed data based on decomposed data on Waymo-Ring. Our scheme can generate diverse data with randomly sampled object-scene-camera combinations online. For each scene, the top row shows the raw scene, and the bottom row shows the recomposed scene. We circle some of the recomposed samples of each category in the figure with different color.
θ & α (°)
AP3D
APBEV
easy
mod
hard
easy
mod
hard
0
15.89
12.76
10.57
23.14
18.17
15.31
1
20.84
16.86
13.50
28.07
21.99
18.11
2
21.76
16.13
13.60
28.71
21.30
18.41
3
19.98
15.48
12.90
27.98
21.03
17.79
Table 12: Ablation study of different degrees of camera pose (pitch and roll) perturbation.
ϵz (m)
AP3D
APBEV
easy
mod
hard
easy
mod
hard
0
15.89
12.76
10.57
23.14
18.17
15.31
1
19.90
15.40
12.93
27.98
21.13
18.09
2
21.21
16.20
14.10
28.85
22.04
18.97
3
19.22
15.16
12.76
27.38
21.62
18.75
Table 13: Ablation study of different degrees of camera pose (translation) perturbation.
Figure 8: Qualitative comparison of detection results on the KITTI validation set for DID-M3D training with and without our scheme. For each scene, the left and right images show the results without and with our scheme, respectively. Ground truth boxes are shown in red, and predicted boxes are shown in green.
rempty
AP3D
APBEV
easy
mod
hard
easy
mod
hard
1.0
19.83
13.65
10.14
27.55
19.13
14.76
0.5
26.57
20.42
17.34
36.37
26.74
23.07
0
23.07
17.92
15.33
31.58
23.68
20.78
Table 14: Ablation study of different ratios of empty scenes per batch.
Model
AP3D/APBEV
easy
mod
hard
MonoDLE ( Ma et al., 2021 )
17.45 / 24.97
13.66 / 19.33
11.68 / 17.01
+EGC ( Lian et al., 2022 )
22.79 / 30.44
16.74 / 22.71
14.60 / 19.60
+MonoSample ( Qiao et al., 2024 )
19.71 / 26.66
15.82 / 21.14
13.58 / 18.36
+MonoPlace3D ( Parihar et al., 2025 )
22.49 / -
15.44 / -
12.89 / -
+ours
26.87 / 35.52
19.48 / 25.07
16.02 / 21.15
Table 15: Comparison with copy-paste strategies.
Method
FID ↓
LPIPS ↓
2D copy paste
9.977
0.098
ours
8.757
0.068
Table 16: Quantitative results of recomposed images.
Monocular 3D object detection has achieved impressive performance on densely annotated datasets. However, it struggles when only a fraction of objects are labeled due to the high cost of 3D annotation. This sparsely annotated setting is common in real-world scenarios where annotating every object is impractical. To address this, we propose a novel framework for sparsely annotated monocular 3D object detection with two key modules. First, we propose Road-Aware Patch Augmentation (RAPA), which leverages sparse annotations by augmenting segmented object patches onto road regions while preserving 3D geometric consistency. Second, we propose Prototype-Based Filtering (PBF), which generates high-quality pseudo-labels by filtering predictions through prototype similarity and depth uncertainty. It maintains global 2D RoI feature prototypes and selects pseudo-labels that are both feature-consistent with learned prototypes and have reliable depth estimates. Our training strategy combines geometry-preserving augmentation with prototype-guided pseudo-labeling to achieve robust detection under sparse supervision. Extensive experiments demonstrate the effectiveness of the proposed method. The source code is available at https://github.com/VisualAIKHU/MonoSAOD .
Metric 3D object detection is a core capability for embodied agents, yet most reliable systems lean on depth sensors, trading away cost, power, and integration simplicity. This motivates monocular 3D detection, which avoids additional constraints, yet it faces a major obstacle: from a single image, depth, and especially absolute scale, are underconstrained. As a result, the prevailing pattern of detecting in 2D and then predicting 3D attributes is often brittle, since modest range errors can dominate 3D localization, and the learned scale prior can fail when cameras, motion, or environments undergo domain shifts. To address this, we propose Map-Det3D, an online multi-view 3D object detection model that brings detection directly into a 3D space reconstructed from RGB. We map a short temporal window into multiple views and repurpose a feed-forward metric 3D reconstruction model as our geometric backbone while tuning its object-aware capabilities. Building on this representation, Map-Det3D directly predicts boxes in metric 3D space, without the widely used 2D-to-3D lifting. Experiments across different benchmarks show that this design supports strong online performance and robust transfer without adaptation, suggesting that training reconstruction priors for detection is a practical route to stable metric 3D detection from monocular video. Code and models are available at https://royyang0714.github.io/Map-Det3D.
Yung-Hsu Yang, Luigi Piccinelli, Samuel Rota Bulò +7
ETH Zürich, Switzerland · Meta Reality Labs Zürich, Switzerland · University of Bonn, Germany
Real-time monocular 3D object detection remains challenging due to severe depth ambiguity, viewpoint shifts, and the high computational cost of 3D reasoning. Existing approaches either rely on LiDAR or geometric priors to compensate for missing depth or sacrifice efficiency to achieve competitive accuracy. We introduce LeAD-M3D, a monocular 3D detector that achieves state-of-the-art accuracy and real-time inference without extra modalities. Our method is enabled by three key components. Asymmetric Augmentation Denoising Distillation (A2D2) transfers geometric knowledge from a clean-image teacher to a MixUp-noised student via a quality- and importance-weighted depth-feature loss, enabling stronger depth reasoning without LiDAR. 3D-aware Consistent Matching (CM3D) improves prediction-to-ground truth assignment by integrating 3D MGIoU into the matching score, yielding stable and precise supervision. Finally, Confidence-Gated 3D Inference (CGI3D) accelerates inference by restricting expensive 3D regression to confident regions. Together, these contributions set a new Pareto frontier for monocular 3D detection: LeAD-M3D achieves state-of-the-art accuracy on KITTI and Waymo, and the best reported car AP on Rope3D, while running up to 3.6× faster than prior high-accuracy models (e.g., MonoDiff). LeAD-M3D demonstrates that high fidelity and real-time monocular 3D detection is simultaneously attainable, without LiDAR, stereo, or strong geometric assumptions.
Johannes Meier, Jonathan Michel, Oussema Dhaouadi +7