3D object detection from LiDAR point clouds is a core problem in autonomous driving. Recent advances in self-supervised learning (SSL) enable scalable pretraining and transfers well to per-point tasks such as semantic and panoptic segmentation, but transfer to 3D detection remains weaker. We analyze recent SSL methods and find that most objectives are defined only on measured LiDAR returns from visible surfaces, leaving occluded and unobserved regions unconstrained. This visible-surface bias can be sufficient for point-wise prediction, but 3D detection requires robustness to missing structure. To address this gap, we propose GhostPoint, an SSL framework that hallucinates latent features in local neighborhoods around discovered instances, generated via a novel instance voxel dilation. In GhostPoint, an encoder processes observed returns, and an additional predictor infers neighborhood representations from observed context. In addition to standard encoder-level supervision, we introduce a predictor-level supervision scheme on sampled voxels from generated neighborhoods. Specifically, observed (visible/masked) voxels match teacher-encoder targets, while unobserved voxels match teacher-predictor hallucinations. This design encourages the learned representation to explicitly model structure beyond observed returns. Extensive evaluations on nuScenes and Waymo demonstrate that our method achieves state-of-the-art performance, consistently improving downstream 3D detection, especially under sparse scans and limited labels.
Figures & tables
Figure 1: GhostPoint learns beyond visible LiDAR surfaces. Raw scans observe only sparse object returns, causing discovered pseudo-instance centers to be biased toward visible surfaces. From these pseudo instances, GhostPoint samples adjacent unobserved regions and hallucinates their latent representations, producing features whose centers better align with the ground-truth object centers, marked by yellow crosses, while recovering object-level structure as illustrated by the PCA visualization.
Method
Sem. Seg
Obj. Det
mIoU
mAP
NDS
PTv3 (sup.)
80.3
63.8
68.4
DOS [ 2 ]
79.2
55.4
61.6
PointINS [ 32 ]
80.0
56.7
62.5
PointINS + Box Reg.
-
56.2
62.4
Table 1: Empirical gap of existing SSL approaches between semantic segmentation and object detection.
Figure 2: Overview of GhostPoint : A scan is augmented into two views. The student encodes a masked version of each view Fθ , while the teacher encoder Fθ~ (EMA) processes the unmasked views. Teacher features are used to discover instances. Discovered instances drive Neighborhood Sampling, which dilates their occupancy to form a neighborhood voxel set Q (visible, masked, and unobserved voxels). Teacher and student predictors Gθ~ and Gθ operate on tokens over Q : the teacher hallucinates only at no-return voxels (overwriting occupied voxels with encoder features), while the student predicts over all non-visible voxels Q∖Qvis . Softmap and offset heads produce targets and predictions at two levels: visible voxels use encoder-level features, while the second-level supervision on non-visible voxels is described in Section 3.2 .
Figure 3: Neighborhood Sampling. Teacher encoder features are passed to an offset head to predict center-offsets and cluster non-ground points into instance proposals. We drop proposals fully masked in the student view (visibility filtering), then dilate each remaining instance occupancy (kernel size ks>1 ) to form a neighborhood voxel set Q containing visible, masked, and newly activated no-return voxels. Q defines predictor queries and Softmap/offset supervision’s scope for predictor-level distillation.
Method
nuScenes
Waymo (L2 mAP)
mAP
NDS
Vehicle
Pedestrian
Cyclist
mAP
PTv3 (sup.)
63.8
68.4
69.5
67.5
66.7
67.9
PSA [ 22 ] (prob.)
41.3
52.8
45.6
42.4
40.8
42.4
SONATA [ 28 ] (prob.)
44.6
55.0
51.2
46.2
44.9
47.4
NOMAE [ 1 ] (prob.)
53.5
60.1
55.3
51.5
50.3
52.4
DOS [ 2 ] (prob.)
55.4
61.6
59.4
56.6
55.2
57.1
Table 2: Main results of SSL pretraining for LiDAR 3D object detection with PTv3 as backbone. probe : freeze the backbone Fθ and train the remaining detector components from scratch; ft : end-to-end fine-tuning. Best results in each setting are in bold .
Figure 4: Qualitative results on the nuScenes dataset [ 4 ] . All self-supervised models are evaluated under the decoder-probing protocol. Green boxes denote ground-truth annotations and red boxes denote predictions. GhostPoint produces predictions that align more closely with ground-truth boxes in both center localization and spatial extent, particularly for partially observed and occluded objects at mid-to-long range.
Figure 7
Model
mAP
NDS
PointINS
56.7
62.5
+ No hallucination
57.3
63.1
+ Zero targets
57.6
63.3
+ Swap masked → hallucinated
59.4
64.1
GhostPoint (ours)
59.5
64.2
Table 4: Disentangling hallucination from regularization on nuScenes (decoder probing).
Figure 6: Visualizations of original and hallucinated point clouds. Ground-truth bounding boxes are shown for reference. (a)/(b): pseudo instances obtained by clustering predicted offsets; (c)/(d): PCA projections of point features.
Table 10
Method
0.1%
1%
Vehicle
Pedestrian
Cyclist
mAP
Vehicle
Pedestrian
Cyclist
mAP
PTv3 (sup.)
32.1
15.2
12.6
20.0
42.1
27.2
33.5
34.3
NOMAE [ 1 ]
34.8
20.9
18.8
24.8
43.6
29.8
35.0
36.1
DOS [ 2 ]
38.3
23.2
20.0
27.2
45.1
33.2
36.5
38.3
PointINS [ 32 ]
39.0
22.8
19.6
27.1
46.5
34.3
37.2
39.3
GhostPoint
40.5
23.5
21.6
28.5
47.2
36.8
39.0
41.0
Table 7: Label efficiency results on Waymo.
Table 12
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Config
Value
Optimizer
AdamW
Scheduler
Cosine annealing
Learning rate
2e-4
Weight decay
4e-2
Batch size
16
Mask Ratio
0.6
Appendix
Table 10: Pretraining settings of GhostPoint
Table 14Table 15
Figure 7: Sensitivity analysis on two key hyperparameters: the dilation kernel size ks , which controls the spatial extent of occupancy dilation around pseudo-instance proposals, and the neighbor count k , which determines how many observed encoder features are aggregated via KNN interpolation to initialize tokens at unobserved voxels.
School of Automation and Intelligent Sensing and Institute of Medical Robotics, Shanghai Jiao Tong University, and Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai 200240, China.