Object-Scene-Camera Decomposition and Recomposition for Data-Efficient Monocular 3D Object Detection

Zhaonian Kuang; Rui Ding; Meng Yang; Xinhu Zheng; Gang Hua

cs.CVFeb 24, 2026

Object-Scene-Camera Decomposition and Recomposition for Data-Efficient Monocular 3D Object Detection

Authors: Zhaonian Kuang, Rui Ding, Meng Yang, Xinhu Zheng, Gang Hua

Organizations: Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, P.R.China · Intelligent Transportation Thrust of the Systems Hub, Hong Kong University of Science and Technology (GZ), Guangzhou, P.R.China · Amazon Alexa AI, Bellevue, Washington, United States

Abstract

Monocular 3D object detection (M3OD) is intrinsically ill-posed, hence training a high-performance deep learning based M3OD model requires a humongous amount of labeled data with complicated visual variation from diverse scenes, variety of objects and camera poses.However, we observe that, due to strong human bias, the three independent entities, i.e., object, scene, and camera pose, are always tightly entangled when an image is captured to construct training data. More specifically, specific 3D objects are always captured in particular scenes with fixed camera poses, and hence lacks necessary diversity. Such tight entanglement induces the challenging issues of insufficient utilization and overfitting to uniform training data. To mitigate this, we propose an online object-scene-camera decomposition and recomposition data manipulation scheme to more efficiently exploit the training data. We first fully decompose training images into textured 3D object point models and background scenes in an efficient computation and storage manner. We then continuously recompose new training images in each epoch by inserting the 3D objects into the freespace of the background scenes, and rendering them with perturbed camera poses from textured 3D point representation. In this way, the refreshed training data in all epochs can cover the full spectrum of independent object, scene, and camera pose combinations. This scheme can serve as a plug-and-play component to boost M3OD models, working flexibly with both fully and sparsely supervised settings. In the sparsely-supervised setting, objects closest to the ego-camera for all instances are sparsely annotated. We then can flexibly increase the annotated objects to control annotation cost. For validation, our method is widely applied to five representative M3OD models and evaluated on both the KITTI and the more complicated Waymo datasets.

Figures & tables

Figure 1: Implementation of our data manipulation scheme on four released models under both fully and sparsely supervised settings. The

x

-axis represents the percentage of annotated objects used in our scheme out of all objects in the KITTI train split. The

y

-axis represents the moderate level of

AP_{3D}

in the KITTI validation split. Our method significantly boosts the performance of the base models in both the fully (i.e., 100% annotation ratio) and the sparsely (i.e., 10%, 20%, and 50%) supervised settings. Our method with only 10% annotations even achieves on-par performance with the fully-supervised setting of the first three base models.

Figure 2: Illustration of our online object-scene-camera decomposition and recomposition data manipulation scheme. We first fully decompose training images into textured 3D object point models and background scenes in an efficient computation and storage manner. We then continuously recompose new training images in each epoch by inserting the 3D objects into the freespace of the background scenes, and rendering them with perturbed camera poses from textured 3D point representation. In this way, the refreshed training data in all epochs can cover the full spectrum of independent object, scene, and camera combinations.

Figure 3: Object point model rectification. The CAD model is intended to aid visualization.

Figure 4: Object position resampling. Using a 3D textured point representation, objects can be flexibly resampled to new positions.

Figure 5: Freespace generation process. For each row of the freespace map in the polar coordinates system, we scan each column to complete the empty area.

Figure 6: Illustration of the sparsely-supervised setting. Based on our scheme, only the objects closest to the camera will be labeled to reduce annotation costs.

Method	$AP_{3D}^{test}$			$AP_{BEV}^{test}$			$AP_{3D}^{val}$			$AP_{BEV}^{val}$
Method	easy	mod	hard	easy	mod	hard	easy	mod	hard	easy	mod	hard
MonoDLE	17.23	12.23	10.29	24.79	18.89	16.00	19.48	15.21	13.57	26.03	21.10	18.59
+ours	25.55	18.04	15.45	34.51	24.82	20.86	26.57	20.42	17.34	36.37	26.74	23.07
improvement	+8.32	+5.81	+5.16	+9.72	+5.93	+4.86	+7.09	+5.21	+3.77	+10.34	+5.64	+4.48
GUPNet	22.26	15.02	13.12	30.29	21.19	18.20	21.95	16.94	14.43	29.57	23.49	20.40
+ours	28.60	19.35	16.35	38.18	26.51	23.01	26.91	20.45	17.45	35.17	26.22	22.84

Table 1: Comparison with four base models on the car category in the KITTI official test set and validation split.

		$AP_{3D}$			$AP_{BEV}$
Method	Venues	easy	mod	hard	easy	mod	hard
MonoDLE ( Ma et al., 2021 )	CVPR2021	17.23	12.23	10.29	24.79	18.89	16.00
CaDDN ( Reading et al., 2021 )	CVPR2021	19.17	13.41	11.46	27.94	18.91	17.19
MonoFlex ( Zhang et al., 2021 )	CVPR2021	19.94	13.89	12.07	28.23	19.75	16.89
GUPNet ( Lu et al., 2021 )	ICCV2021	22.26	15.02	13.12	30.29	21.19	18.20
MonoDistill ( Chong et al., 2022 )	ICLR2022	22.97	16.03	13.60	31.87	22.59	19.72

Table 2: Comparison with state-of-the-art methods on the car category in the KITTI official test set. Following the KITTI protocol, methods are ranked based on their performance under the moderate difficulty setting. The best results are in bold , and the best results of the baselines are underlined .

Method	Annotation	$AP_{3D}$			$AP_{BEV}$
Method	Annotation	easy	mod	hard	easy	mod	hard
MonoDLE	Full	19.48	15.21	13.57	26.03	21.10	18.59
	10%	6.67(-12.81)	4.67(-10.54)	3.61(-9.96)	10.10(-15.93)	6.91(-14.19)	5.54(-13.05)
	10% w / ours	20.73(+1.25)	15.73(+0.52)	12.82(-0.75)	28.00(+1.97)	20.69(-0.41)	16.37(-2.22)
GUPNet	Full	21.95	16.94	14.43	29.57	23.49	20.40
	10%	10.25(-11.70)	6.06(-10.88)	4.70(-9.73)	15.17(-14.40)	8.88(-14.61)	7.13(-13.27)

Table 3: Comparison of fully-supervised and sparsely-supervised results on the KITTI validation set. Our scheme with fewer annotations (especially 10% for the first three base models) achieves comparable performance to the fully-supervised training of the base models. When both use the same sparse annotations, our scheme significantly outperforms the base models.

			$AP_{3D}$			$APH_{3D}$
$IOU_{3D}$	Difficulty	Method	Vehicle	Pedestrian	Cyclist	Vehicle	Pedestrian	Cyclist
		DID-M3D	2.64	1.49	1.05	2.63	1.36	1.02
		+ours	3.73(+41%)	1.91(+28%)	2.78(+165%)	3.71(+41%)	1.72(+26%)	2.70(+165%)
		MonoDETR	1.63	0.98	1.87	1.61	0.86	1.49
	Level_1	+ours	2.52(+55%)	1.34(+37%)	2.14(+14%)	2.49(+55%)	1.2(+40%)	2.1(+41%)
		DID-M3D	2.48	1.36	1.01	2.46	1.24	0.98

Table 4: Comparison of base models and our scheme on three categories in the Waymo-Mono Dataset.

Method			DID-M3D			MonoDETR
Annotation			Full	10%	10% w/ours	Full	10%	10% w /ours
$AP_{3D}$	$IOU_{3D}@0.7$	Level_1	2.64	0.71(-1.93)	2.69(+0.05)	1.63	0.77(-0.86)	1.53(-0.10)
	$IOU_{3D}@0.7$	Level_2	2.48	0.67(-1.81)	2.52(+0.04)	1.53	0.72(-0.81)	1.44(-0.09)
	$IOU_{3D}@0.5$	Level_1	11.83	4.37(-7.46)	11.24(-0.59)	8.03	4.51(-3.52)	7.47(-0.56)
	$IOU_{3D}@0.5$	Level_2	11.09	4.09(-7.00)	10.54(-0.55)	7.54	4.23(-3.31)	7.01(-0.53)
$APH_{3D}$	$IOU_{3D}@0.7$	Level_1	2.63	0.70(-1.93)	2.66(+0.03)	1.61	0.76(-0.85)	1.51(-0.10)

Table 5: Comparison of fully supervised and sparsely supervised (10% annotation ratio) settings on Waymo-Mono dataset (vehicle class). Our scheme with 10% annotation achieves comparable performance to the fully-supervised setting of the base models.

Setting	Difficulty	Method	$AP_{3D}$			$APH_{3D}$
Setting	Difficulty	Method	0-30m	30-50m	50- $\infty$ m	0-30m	30-50m	50- $\infty$ m
fully-supervised	Level_1	DID-M3D	27.55	5.47	0.89	27.35	5.42	0.88
	Level_1	ours	33.68	6.53	0.75	33.41	6.47	0.74
	Level_2	DID-M3D	27.45	5.27	0.78	27.25	5.22	0.76
	Level_2	ours	33.55	6.28	0.65	33.29	6.23	0.64
sparsely-supervised (10%)	Level_1	DID-M3D	11.84	1.41	0.18	11.64	1.38	0.17

Table 6: Comparison of both fully-supervised and sparsely supervised performance at different distances. The result is reported on DID-M3D with an IoU threshold of 0.5.

Supervision		Full					Sparse(10%)
Category	Method	mAP $\uparrow$	mATE $\downarrow$	mASE $\downarrow$	mAOE $\downarrow$	NDS* $\uparrow$	mAP $\uparrow$	mATE $\downarrow$	mASE $\downarrow$	mAOE $\downarrow$	NDS* $\uparrow$
Vehicle	PETR	0.497	0.704	0.156	0.126	0.584	0.074	1.250	0.228	0.764	0.205
Vehicle	+ours	0.508	0.684	0.153	0.114	0.596(+0.012)	0.435	0.769	0.162	0.205	0.528(+0.323)
Pedestrian	PETR	0.457	0.776	0.266	0.859	0.411	0.069	1.173	0.287	1.454	0.153
Pedestrian	+ours	0.492	0.715	0.262	0.606	0.482(+0.071)	0.401	0.812	0.270	1.087	0.353(+0.200)
Cyclist	PETR	0.318	0.713	0.229	0.569	0.407	0.000	1.288	0.542	1.401	0.077

Table 7: Comparison of PETR ( Liu et al., 2022b ) and our scheme on Waymo-Ring Dataset.

Object-Scene Recomposition	Object-Scene Decomposition	Camera Pose Pertubation	$\ AP_{3D}$			$\ AP_{BEV}$
Object-Scene Recomposition	Object-Scene Decomposition	Camera Pose Pertubation	easy	mod	hard	easy	mod	hard
-	-	-	15.89	12.76	10.57	23.14	18.17	15.31
✓	-	-	20.69	16.92	14.34	27.95	21.96	19.22
✓	✓	-	23.30	19.02	16.35	32.00	25.12	21.99
✓	✓	✓	26.57	20.42	17.34	36.37	26.74	23.07

Table 8: Ablation study of object-scene-camera decomposition and recomposition operations.

Decomposition Ratio(%)	$\ AP_{3D}$			$\ AP_{BEV}$
Decomposition Ratio(%)	easy	mod	hard	easy	mod	hard
0	15.89	12.76	10.57	23.14	18.17	15.31
25	18.24	14.96	12.64	24.31	19.88	17.35
50	19.92	15.71	14.01	27.31	21.80	18.94
100	22.44	17.97	15.28	31.94	24.59	21.70

Table 9: Ablation study of different decomposition ratios.

Counts of Recomposition	$\ AP_{3D}$			$\ AP_{BEV}$
Counts of Recomposition	easy	mod	hard	easy	mod	hard
0	15.89	12.76	10.57	23.14	18.17	15.31
2	18.89	14.86	12.51	26.67	20.44	17.40
50	21.11	17.00	14.47	29.17	22.86	19.78
200	23.30	19.02	16.35	32.00	25.12	21.99

Table 10: Ablation study of different counts of recomposition per scene.

Method	Object-scene composition per scene
plain	$\ \approx$ 4
+ours	$\ \approx$ 700

Table 11: Ablation study of scene utilization before and after applying our scheme.

Figure 7: Visualization of recomposed data based on decomposed data on Waymo-Ring. Our scheme can generate diverse data with randomly sampled object-scene-camera combinations online. For each scene, the top row shows the raw scene, and the bottom row shows the recomposed scene. We circle some of the recomposed samples of each category in the figure with different color.

$\theta$ & $\alpha$ (°)	$AP_{3D}$			$AP_{BEV}$
$\theta$ & $\alpha$ (°)	easy	mod	hard	easy	mod	hard
0	15.89	12.76	10.57	23.14	18.17	15.31
1	20.84	16.86	13.50	28.07	21.99	18.11
2	21.76	16.13	13.60	28.71	21.30	18.41
3	19.98	15.48	12.90	27.98	21.03	17.79

Table 12: Ablation study of different degrees of camera pose (pitch and roll) perturbation.

$\ \epsilon_{z}$ (m)	$\ AP_{3D}$			$\ AP_{BEV}$
$\ \epsilon_{z}$ (m)	easy	mod	hard	easy	mod	hard
0	15.89	12.76	10.57	23.14	18.17	15.31
1	19.90	15.40	12.93	27.98	21.13	18.09
2	21.21	16.20	14.10	28.85	22.04	18.97
3	19.22	15.16	12.76	27.38	21.62	18.75

Table 13: Ablation study of different degrees of camera pose (translation) perturbation.

Figure 8: Qualitative comparison of detection results on the KITTI validation set for DID-M3D training with and without our scheme. For each scene, the left and right images show the results without and with our scheme, respectively. Ground truth boxes are shown in red, and predicted boxes are shown in green.

$\ r_{empty}$	$\ AP_{3D}$			$\ AP_{BEV}$
$\ r_{empty}$	easy	mod	hard	easy	mod	hard
1.0	19.83	13.65	10.14	27.55	19.13	14.76
0.5	26.57	20.42	17.34	36.37	26.74	23.07
0	23.07	17.92	15.33	31.58	23.68	20.78

Table 14: Ablation study of different ratios of empty scenes per batch.

Model	$\ AP_{3D}/AP_{BEV}$
Model	easy	mod	hard
MonoDLE ( Ma et al., 2021 )	17.45 / 24.97	13.66 / 19.33	11.68 / 17.01
+EGC ( Lian et al., 2022 )	22.79 / 30.44	16.74 / 22.71	14.60 / 19.60
+MonoSample ( Qiao et al., 2024 )	19.71 / 26.66	15.82 / 21.14	13.58 / 18.36
+MonoPlace3D ( Parihar et al., 2025 )	22.49 / -	15.44 / -	12.89 / -
+ours	26.87 / 35.52	19.48 / 25.07	16.02 / 21.15

Table 15: Comparison with copy-paste strategies.

Method	FID $\downarrow$	LPIPS $\downarrow$
2D copy paste	9.977	0.098
ours	8.757	0.068

Table 16: Quantitative results of recomposed images.

Type	Action	Speed
Offline Data Manipulation	Scene Database Construction	3.5h
Offline Data Manipulation	Object Database Construction	1h
Online Data Manipulation	Object-Scene Recomposition	5fps
Online Data Manipulation	Camera Pose Perturbation	2500fps
Training	DID-M3D	24h
	DID-M3D+ours	40h

Table 17: Computational costs of our scheme.

Monocular 3D Object Detection

Monocular 3D object detection estimates objects’ positions, dimensions, and orientations in 3D from one camera, without depth sensors.

See all 22 papers

MonoSAOD: Monocular 3D Object Detection with Sparsely Annotated Label
Apr 2, 2026Junyoung Jung, Seokwon Kim, Jung Uk Kim3D Object Detection Monocular 3D Object Detection
Kyung Hee University
Map-Det3D: Metric Feed-Forward 3D Reconstruction Prior for Multi-view 3D Object Detection from Streaming Inputs
Aug 12, 2026Yung-Hsu Yang, Luigi Piccinelli, Samuel Rota Bulò +7Multi-View 3D Object Detection 3D Object Detection
ETH Zürich, Switzerland · Meta Reality Labs Zürich, Switzerland · University of Bonn, Germany
LeAD-M3D: Leveraging Asymmetric Distillation for Real-Time Monocular 3D Detection
Dec 5, 2025Johannes Meier, Jonathan Michel, Oussema Dhaouadi +7Efficient Object Detection Real-Time Object Detection
DeepScenario · ETH Zurich · TU Munich +5

Apr 2, 2026J/K move · Enter open · S save

MonoSAOD: Monocular 3D Object Detection with Sparsely Annotated Label

Junyoung Jung, Seokwon Kim, Jung Uk Kim

Kyung Hee University

3D Object Detection Monocular 3D Object Detection Pseudo-Labeling

Open paper AskPDF arXiv

Monocular 3D object detection has achieved impressive performance on densely annotated datasets. However, it struggles when only a fraction of objects are labeled due to the high cost of 3D annotation. This sparsely annotated setting is common in real-world scenarios where annotating every object is impractical. To address this, we propose a novel framework for sparsely annotated monocular 3D object detection with two key modules. First, we propose Road-Aware Patch Augmentation (RAPA), which leverages sparse annotations by augmenting segmented object patches onto road regions while preserving 3D geometric consistency. Second, we propose Prototype-Based Filtering (PBF), which generates high-quality pseudo-labels by filtering predictions through prototype similarity and depth uncertainty. It maintains global 2D RoI feature prototypes and selects pseudo-labels that are both feature-consistent with learned prototypes and have reliable depth estimates. Our training strategy combines geometry-preserving augmentation with prototype-guided pseudo-labeling to achieve robust detection under sparse supervision. Extensive experiments demonstrate the effectiveness of the proposed method. The source code is available at https://github.com/VisualAIKHU/MonoSAOD .

Abstract

Monocular 3D Object Detection

Explore similar work

MonoSAOD: Monocular 3D Object Detection with Sparsely Annotated Label

Map-Det3D: Metric Feed-Forward 3D Reconstruction Prior for Multi-view 3D Object Detection from Streaming Inputs

LeAD-M3D: Leveraging Asymmetric Distillation for Real-Time Monocular 3D Detection