Unified Unsupervised and Sparsely-Supervised 3D Object Detection by Semantic Pseudo-Labeling and Prototype Learning

Yushen He; Lei Zhao; Weidong Chen

cs.CVFeb 25, 2026

Unified Unsupervised and Sparsely-Supervised 3D Object Detection by Semantic Pseudo-Labeling and Prototype Learning

Authors: Yushen He, Lei Zhao, Weidong Chen

Organizations: School of Automation and Intelligent Sensing and Institute of Medical Robotics, Shanghai Jiao Tong University, and Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai 200240, China.

Abstract

3D object detection is essential for autonomous driving and robotic perception, yet its reliance on large-scale manually annotated data limits scalability and adaptability. To reduce annotation dependency, unsupervised and sparsely-supervised paradigms have emerged. However, they face intertwined challenges: low-quality pseudo-labels, unstable feature mining, and a lack of a unified training framework. This paper proposes SPL, a unified training framework for both unsupervised and sparsely-supervised 3D object detection via Semantic Pseudo-labeling and prototype Learning. SPL first generates high-quality pseudo-labels by integrating image semantics, point cloud geometry, and temporal cues, producing both 3D bounding boxes for dense objects and 3D point labels for sparse ones. These pseudo-labels are not used directly but as probabilistic priors within a novel, multi-stage prototype learning strategy. This strategy stabilizes feature representation learning through memory-based initialization and momentum-based prototype updating, effectively mining features from both labeled and unlabeled data. Extensive experiments on KITTI and nuScenes datasets demonstrate that SPL significantly outperforms state-of-the-art methods in both settings. Our work provides a robust and generalizable solution for learning 3D object detectors with minimal or no manual annotations. Our code is available at https://github.com/TossherO/SPL.

Figures & tables

Fig. 1: Comparison between representative unsupervised and sparsely-supervised 3D object detection pipelines and our unified SPL framework. Existing methods usually emphasize either pseudo-label generation or sparse-supervision feature learning, while SPL explicitly integrates semantic pseudo-labeling and prototype-based feature mining in a single training framework.

Fig. 2: Comparison of contrastive learning strategies under sparse 3D supervision. The figure contrasts in-batch contrast, memory-queue contrast, and prototype-based contrast, and highlights their different trade-offs in sample diversity, feature consistency, and training stability.

Fig. 3: Overview of the proposed 3D pseudo-label generation pipeline. Starting from synchronized LiDAR frames and RGB images, the pipeline performs data preprocessing, 3D point label generation, and 3D Bbox label generation with temporal refinement, producing both high-quality 3D Bbox pseudo labels and 3D point pseudo labels for sparse objects.

Fig. 4: Overview of the prototype-based training strategy in SPL. The framework unifies supervision into GT supervision labels and pseudo labels, mines foreground/background features from BEV representations, and optimizes detector training with heatmap supervision and intra/inter-class contrastive objectives while updating prototypes with momentum.

Fig. 5: Feature mining process that fuses prototype similarity with pseudo-heatmap priors. Candidate regions are first selected by high prototype similarity and then cross-validated with pseudo labels to form reliable mined positives and masked ambiguous regions, which are subsequently used to construct contrastive foreground/background feature sets.

Fig. 6: Overview of the three-stage training pipeline used to stabilize prototype learning. Stage 1 performs memory-based contrastive pretraining for robust prototype initialization, Stage 2 refines prototypes with GT-only supervision, and Stage 3 enables the full SPL strategy with pseudo-label-guided feature mining.

Annotation Rate	Method	Car AP @ 3D-IoU 0.7			Pedestrian AP @ 3D-IoU 0.5			Cyclist AP @ 3D-IoU 0.5
Annotation Rate	Method	Easy	Mod.	Hard	Easy	Mod.	Hard	Easy	Mod.	Hard
100%	Voxel-RCNN	92.3	84.9	82.6	69.6	63.0	58.6	88.7	72.5	68.2
2%	Voxel-RCNN	70.5	54.9	44.8	42.6	38.5	32.1	73.3	47.8	43.2
	CoIn	89.1	70.2	55.6	50.8	45.2	39.6	80.2	52.3	48.6
	CoIn++	92.0	79.5	71.5	46.7	36.1	31.2	82.0	58.4	54.6
	SP3D	91.3	80.5	74.0	67.4	58.7	50.9	92.5	73.1	68.3

TABLE I: Comparison on Sparsely-Supervised 3D Object Detection on KITTI val set.

Annotation Rate	Method	mAP	NDS	Car	Truck	C.V.	Bus	Trailer	Barrier	Motor.	Bike	Ped.	T.C.
100%	CenterPoint	56.11	64.61	84.63	52.87	16.40	66.85	36.85	65.79	53.97	35.39	83.46	64.91
10%	CenterPoint	8.09	25.77	24.62	2.84	0.00	15.66	0.00	4.07	3.33	0.29	25.11	4.96
	CoIn	12.47	33.79	38.70	6.85	0.00	20.67	7.81	11.51	2.85	3.36	34.85	8.50
	SPL (Ours)	38.70	48.96	72.41	42.03	21.02	59.11	23.29	47.45	3.94	0.00	70.51	47.22

TABLE II: Comparison on Sparsely-Supervised 3D Object Detection on nuScenes val set.

Train Set	Method	Car AP @ 3D-IoU 0.5			Pedestrian AP @ 3D-IoU 0.5			Cyclist AP @ 3D-IoU 0.5
Train Set	Method	Easy	Mod.	Hard	Easy	Mod.	Hard	Easy	Mod.	Hard
Waymo	MODEST	47.6	33.4	30.6	1.3	2.2	2.3	0.1	0.0	0.0
	OYSTER	65.3	54.8	43.6	3.0	3.0	3.0	1.7	1.8	1.9
	CPD	90.9	81.0	79.8	17.1	15.2	14.2	11.1	7.3	6.5
	Motal	96.2	87.6	85.8	37.9	33.4	31.1	56.3	37.8	35.5
KITTI	OYSTER	43.7	34.5	31.2	0.0	0.0	0.0	0.0	0.0	0.0

TABLE III: Comparison on Unsupervised 3D Object Detection on KITTI val set.

Method	Detector	mAP	NDS	Vehicle	Pedestrian	Cyclist
UNION	CenterPoint	25.1	24.4	31.0	44.2	0.0
AnnofreeOD	Voxel-NeXt	34.4	36.6	44.1	51.1	7.9
SPL (Ours)	CenterPoint	38.3	40.6	48.1	59.3	7.2

TABLE IV: Comparison on Unsupervised 3D Object Detection on nuScenes val set.

Pseudo Label Generation Method		Precision			Recall
Pseudo Label Generation Method		Car	Pedestrian	Cyclist	Car	Pedestrian	Cyclist
LISO		61.4	40.3	54.7	24.7	8.1	17.4
UNION		84.5	67.1	64.5	54.1	43.7	41.8
SPL	(a) without 3D point pseudo labels	93.4	84.5	90.7	34.7	21.5	28.6
	(b) with 3D point pseudo labels	82.6	64.3	76.1	67.4	54.9	61.7
	(c) without object points refinement	70.6	57.6	67.8	59.3	47.1	53.7

TABLE V: Ablation studies on Pseudo Label Generation Strategy on KITTI.

Pseudo heatmap	Prototype sim score map	Contrastive loss	Car (AP @ 3D-IoU 0.7)	Pedestrian (AP @ 3D-IoU 0.5)	Cyclist (AP @ 3D-IoU 0.5)
			60.2	38.4	52.1
✓			74.1	47.6	61.3
	✓		71.7	45.7	57.0
	✓	✓	77.9	52.7	69.2
✓	✓	✓	85.6	64.1	81.2

TABLE VI: Ablation studies on Prototype-Based Training Strategy on KITTI val set.

Stage 1	Stage 2	Stage 3	Car (AP @ 3D-IoU 0.7)	Pedestrian (AP @ 3D-IoU 0.5)	Cyclist (AP @ 3D-IoU 0.5)
		✓	81.3	60.4	77.9
	✓	✓	82.9	61.7	79.1
✓		✓	83.5	62.8	80.7
✓	✓	✓	85.6	64.1	81.2

TABLE VII: Ablation studies on Multi-Stage Training Pipeline on KITTI val set.

	Parameter Description	Value
Data Preprocessing	Number of LiDAR frames aggregated in one process	KITTI: 1, nuScenes: 11
3D Point Label Generation	Kernel size when dilating the 2D object masks	$3\times 3$
	Number of iterations when dilating the 2D object masks	$\sqrt{area_{mask}/20}$ , $area_{mask}$ is the pixel area of the mask
	The prior height ranges $[h_{c,min},h_{c,max}]$ for different classes	Pedestrian and Cyclist: 1.0-2.0 m, Car: 1.0-3.0 m, Bus: 1.5-5.0 m, Truck: 1.5-5.0 m
	Parameter $\epsilon$ of DBSCAN	Vehicle: 0.5 m, Pedestrian and Cyclist: 0.3 m
	Parameter min_samples of DBSCAN	4

TABLE VIII: The detailed parameters for Pseudo-Label Generation.

	Parameter Description	Value
Prototype Learning	Number of prototypes per class $K$	5
	Prototype feature dimension $D$	64
	Memory bank size in Stage 1	1000
	Similarity threshold for mined features ( $\tau_{s}$ )	0.9
	Prototype momentum coefficient ( $\alpha$ )	0.9
Prototype Update	Maximum number of updates per prototype in one iteration	KITTI sparse: 50, KITTI unsup: 50 nuScenes sparse: 50, nuScenes unsup: 100

TABLE IX: The detailed parameters for SPL Network and Training.

Fig. 7: The visualization of ground truth annotations, pseudo labels, and detection results of our SPL method on KITTI dataset. (Part 1)

Fig. 8: The visualization of ground truth annotations, pseudo labels, and detection results of our SPL method on KITTI dataset. (Part 2)

3D Object Detection

3D object detection identifies objects and estimates their locations, sizes, and orientations from images, point clouds, radar, or fused sensors.

See all 176 papers

MonoSAOD: Monocular 3D Object Detection with Sparsely Annotated Label
Apr 2, 2026Junyoung Jung, Seokwon Kim, Jung Uk Kim3D Object Detection Monocular 3D Object Detection
Kyung Hee University
Sparse3DTrack: Monocular 3D Object Tracking Using Sparse Supervision
Mar 18, 2026Nikhil Gosala, B. Ravi Kiran, Senthil Yogamani +1Visual Object Tracking 3D MOT
University of Freiburg, Germany · Qualcomm SARL France · Automated Driving, Qualcomm Technologies, Inc., USA
GhostPoint: Self-Supervised Representation Learning by Hallucinating Occluded LiDAR Structure
Aug 14, 2026Mohamed Abdelsamad, Bin Yang, Michael Ulrich +4Self-Supervised Point Cloud Learning Lidar-Based 3D Object Detection
Bosch Center for AI · University of Freiburg · University of Luebeck

Apr 2, 2026J/K move · Enter open · S save

MonoSAOD: Monocular 3D Object Detection with Sparsely Annotated Label

Junyoung Jung, Seokwon Kim, Jung Uk Kim

Kyung Hee University

3D Object Detection Monocular 3D Object Detection Pseudo-Labeling

Open paper AskPDF arXiv

Monocular 3D object detection has achieved impressive performance on densely annotated datasets. However, it struggles when only a fraction of objects are labeled due to the high cost of 3D annotation. This sparsely annotated setting is common in real-world scenarios where annotating every object is impractical. To address this, we propose a novel framework for sparsely annotated monocular 3D object detection with two key modules. First, we propose Road-Aware Patch Augmentation (RAPA), which leverages sparse annotations by augmenting segmented object patches onto road regions while preserving 3D geometric consistency. Second, we propose Prototype-Based Filtering (PBF), which generates high-quality pseudo-labels by filtering predictions through prototype similarity and depth uncertainty. It maintains global 2D RoI feature prototypes and selects pseudo-labels that are both feature-consistent with learned prototypes and have reliable depth estimates. Our training strategy combines geometry-preserving augmentation with prototype-guided pseudo-labeling to achieve robust detection under sparse supervision. Extensive experiments demonstrate the effectiveness of the proposed method. The source code is available at https://github.com/VisualAIKHU/MonoSAOD .

Abstract

3D Object Detection

Explore similar work

MonoSAOD: Monocular 3D Object Detection with Sparsely Annotated Label

Sparse3DTrack: Monocular 3D Object Tracking Using Sparse Supervision

GhostPoint: Self-Supervised Representation Learning by Hallucinating Occluded LiDAR Structure