3D object detection from LiDAR point clouds is a core problem in autonomous driving. Recent advances in self-supervised learning (SSL) enable scalable pretraining and transfers well to per-point tasks such as semantic and panoptic segmentation, but transfer to 3D detection remains weaker. We analyze recent SSL methods and find that most objectives are defined only on measured LiDAR returns from visible surfaces, leaving occluded and unobserved regions unconstrained. This visible-surface bias can be sufficient for point-wise prediction, but 3D detection requires robustness to missing structure. To address this gap, we propose GhostPoint, an SSL framework that hallucinates latent features in local neighborhoods around discovered instances, generated via a novel instance voxel dilation. In GhostPoint, an encoder processes observed returns, and an additional predictor infers neighborhood representations from observed context. In addition to standard encoder-level supervision, we introduce a predictor-level supervision scheme on sampled voxels from generated neighborhoods. Specifically, observed (visible/masked) voxels match teacher-encoder targets, while unobserved voxels match teacher-predictor hallucinations. This design encourages the learned representation to explicitly model structure beyond observed returns. Extensive evaluations on nuScenes and Waymo demonstrate that our method achieves state-of-the-art performance, consistently improving downstream 3D detection, especially under sparse scans and limited labels.
Figures & tables
Figure 1: GhostPoint learns beyond visible LiDAR surfaces. Raw scans observe only sparse object returns, causing discovered pseudo-instance centers to be biased toward visible surfaces. From these pseudo instances, GhostPoint samples adjacent unobserved regions and hallucinates their latent representations, producing features whose centers better align with the ground-truth object centers, marked by yellow crosses, while recovering object-level structure as illustrated by the PCA visualization.
Method
Sem. Seg
Obj. Det
mIoU
mAP
NDS
PTv3 (sup.)
80.3
63.8
68.4
DOS [ 2 ]
79.2
55.4
61.6
PointINS [ 32 ]
80.0
56.7
62.5
PointINS + Box Reg.
-
56.2
62.4
Table 1: Empirical gap of existing SSL approaches between semantic segmentation and object detection.
Figure 2: Overview of GhostPoint : A scan is augmented into two views. The student encodes a masked version of each view Fθ , while the teacher encoder Fθ~ (EMA) processes the unmasked views. Teacher features are used to discover instances. Discovered instances drive Neighborhood Sampling, which dilates their occupancy to form a neighborhood voxel set Q (visible, masked, and unobserved voxels). Teacher and student predictors Gθ~ and Gθ operate on tokens over Q : the teacher hallucinates only at no-return voxels (overwriting occupied voxels with encoder features), while the student predicts over all non-visible voxels Q∖Qvis . Softmap and offset heads produce targets and predictions at two levels: visible voxels use encoder-level features, while the second-level supervision on non-visible voxels is described in Section 3.2 .
Figure 3: Neighborhood Sampling. Teacher encoder features are passed to an offset head to predict center-offsets and cluster non-ground points into instance proposals. We drop proposals fully masked in the student view (visibility filtering), then dilate each remaining instance occupancy (kernel size ks>1 ) to form a neighborhood voxel set Q containing visible, masked, and newly activated no-return voxels. Q defines predictor queries and Softmap/offset supervision’s scope for predictor-level distillation.
Method
nuScenes
Waymo (L2 mAP)
mAP
NDS
Vehicle
Pedestrian
Cyclist
mAP
PTv3 (sup.)
63.8
68.4
69.5
67.5
66.7
67.9
PSA [ 22 ] (prob.)
41.3
52.8
45.6
42.4
40.8
42.4
SONATA [ 28 ] (prob.)
44.6
55.0
51.2
46.2
44.9
47.4
NOMAE [ 1 ] (prob.)
53.5
60.1
55.3
51.5
50.3
52.4
DOS [ 2 ] (prob.)
55.4
61.6
59.4
56.6
55.2
57.1
Table 2: Main results of SSL pretraining for LiDAR 3D object detection with PTv3 as backbone. probe : freeze the backbone Fθ and train the remaining detector components from scratch; ft : end-to-end fine-tuning. Best results in each setting are in bold .
Figure 4: Qualitative results on the nuScenes dataset [ 4 ] . All self-supervised models are evaluated under the decoder-probing protocol. Green boxes denote ground-truth annotations and red boxes denote predictions. GhostPoint produces predictions that align more closely with ground-truth boxes in both center localization and spatial extent, particularly for partially observed and occluded objects at mid-to-long range.
Figure 7
Model
mAP
NDS
PointINS
56.7
62.5
+ No hallucination
57.3
63.1
+ Zero targets
57.6
63.3
+ Swap masked → hallucinated
59.4
64.1
GhostPoint (ours)
59.5
64.2
Table 4: Disentangling hallucination from regularization on nuScenes (decoder probing).
Figure 6: Visualizations of original and hallucinated point clouds. Ground-truth bounding boxes are shown for reference. (a)/(b): pseudo instances obtained by clustering predicted offsets; (c)/(d): PCA projections of point features.
Table 10
Method
0.1%
1%
Vehicle
Pedestrian
Cyclist
mAP
Vehicle
Pedestrian
Cyclist
mAP
PTv3 (sup.)
32.1
15.2
12.6
20.0
42.1
27.2
33.5
34.3
NOMAE [ 1 ]
34.8
20.9
18.8
24.8
43.6
29.8
35.0
36.1
DOS [ 2 ]
38.3
23.2
20.0
27.2
45.1
33.2
36.5
38.3
PointINS [ 32 ]
39.0
22.8
19.6
27.1
46.5
34.3
37.2
39.3
GhostPoint
40.5
23.5
21.6
28.5
47.2
36.8
39.0
41.0
Table 7: Label efficiency results on Waymo.
Table 12
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Config
Value
Optimizer
AdamW
Scheduler
Cosine annealing
Learning rate
2e-4
Weight decay
4e-2
Batch size
16
Mask Ratio
0.6
Appendix
Table 10: Pretraining settings of GhostPoint
Table 14Table 15
Figure 7: Sensitivity analysis on two key hyperparameters: the dilation kernel size ks , which controls the spatial extent of occupancy dilation around pseudo-instance proposals, and the neighbor count k , which determines how many observed encoder features are aggregated via KNN interpolation to initialize tokens at unobserved voxels.
LiDAR-based 3D object detectors typically rely on proposal heads with hand-crafted components like anchor assignment and non-maximum suppression (NMS), complicating training and limiting extensibility. We present AutoReg3D, an autoregressive 3D detector that casts detection as sequence generation. Given point-cloud features, AutoReg3D emits objects in a range-causal (near-to-far) order and encodes each object as a short, discrete-token sequence consisting of its center, size, orientation, velocity, and class. This near-to-far ordering mirrors LiDAR geometry--near objects occlude far ones but not vice versa--enabling straightforward teacher forcing during training and autoregressive decoding at test time. AutoReg3D is compatible across diverse point-cloud or backbones and attains competitive nuScenes performance without anchors or NMS. Beyond parity, the sequential formulation unlocks language-model advances for 3D perception, including GRPO-style reinforcement learning for task-aligned objectives. These results position autoregressive decoding as a viable, flexible alternative for LiDAR-based detection and open a path to importing modern sequence-modeling tools into 3D perception.
Zanming Huang, Jinsu Yoo, Sooyoung Jeon +6
The Ohio State University · Cornell University · Boston University +1
Three-dimensional object detection for autonomous driving is dominated by detectors trained on large corpora of human-annotated 3D boxes. Such a detector learns a fixed category list, and everything outside it is invisible. This paper asks whether the task can be solved training-free and open-vocabulary. A promptable segmentation model (SAM3), queried with class names as text prompts, supplies instance masks in the vehicle's six surround-view cameras, and the masks are turned into metric 3D boxes using the geometry of the scene. The core is a controlled three-stage comparison on nuScenes in which 2D detection is held fixed and only the source of 3D geometry changes. Geometry predicted from images alone reaches 0.183 mean average precision (mAP) under the official protocol; fitting boxes from raw LiDAR points inside the same masks with training-free rules reaches 0.298 mAP / 0.348 nuScenes detection score (NDS) at zero labeling cost; borrowing supervised box geometry at inference time lifts the same detections to 0.413 mAP / 0.555 NDS, which locates the pipeline's largest deficit in measurement precision rather than 2D detection, while class confusion and confidence calibration survive that substitution. Reversing the direction, a three-state camera-witness rule built from the same masks improves a supervised LiDAR-only detector from 0.596 to 0.630 mAP, roughly half the gain of fully supervised camera fusion, with no training. A coverage analysis shows that SAM3 finds 84% of in-range objects with a correctly named mask; the classes that fail in the official metric are misnamed or geometrically unforgiving, not unseen.
Ömer Faruk Deniz, Mustafa Taha Koçyiğit
Institute of Data Science and Artificial Intelligence, Boğaziçi University, South Campus, Bebek, 34342, Istanbul, Türkiye
3D object detection is essential for autonomous driving and robotic perception, yet its reliance on large-scale manually annotated data limits scalability and adaptability. To reduce annotation dependency, unsupervised and sparsely-supervised paradigms have emerged. However, they face intertwined challenges: low-quality pseudo-labels, unstable feature mining, and a lack of a unified training framework. This paper proposes SPL, a unified training framework for both unsupervised and sparsely-supervised 3D object detection via Semantic Pseudo-labeling and prototype Learning. SPL first generates high-quality pseudo-labels by integrating image semantics, point cloud geometry, and temporal cues, producing both 3D bounding boxes for dense objects and 3D point labels for sparse ones. These pseudo-labels are not used directly but as probabilistic priors within a novel, multi-stage prototype learning strategy. This strategy stabilizes feature representation learning through memory-based initialization and momentum-based prototype updating, effectively mining features from both labeled and unlabeled data. Extensive experiments on KITTI and nuScenes datasets demonstrate that SPL significantly outperforms state-of-the-art methods in both settings. Our work provides a robust and generalizable solution for learning 3D object detectors with minimal or no manual annotations. Our code is available at https://github.com/TossherO/SPL.
Yushen He, Lei Zhao, Weidong Chen
School of Automation and Intelligent Sensing and Institute of Medical Robotics, Shanghai Jiao Tong University, and Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai 200240, China.