Emergent 3D Instance Segmentation from Self-Supervised Point Transformers
Authors: Ted Lentsch, Santiago Montiel-Marín, Holger Caesar, Julian F. P. Kooij
Organizations: Department of Cognitive Robotics, Delft University of Technology, The Netherlands · Department of Electronics, University of Alcalá, Spain
Unsupervised 3D instance segmentation of outdoor LiDAR scans has traditionally relied on handcrafted geometric priors such as density-based clustering, motion cues, or projected 2D detections. In this work, we investigate whether a frozen, self-supervised point transformer already contains the structural information required to isolate object instances without any handcrafted geometric prior. Using this transformer purely as a feature extractor, we probe its internal representations across the SemanticKITTI, nuScenes, and Waymo Perception datasets. Our analysis yields four core insights: (1) the instance signal concentrates in the attention queries and keys rather than in the values or final output features; (2) output features semantically collapse, merging adjacent same-class objects that the queries and keys keep distinct; (3) this instance signal is bimodal in depth, strongest at the shallowest and deepest encoder stages; and (4) this signal is driven predominantly by the rotary position encoding (RoPE), whose removal collapses its advantage. We put these findings into our method TokenGraph3D, a training-free segmenter that groups points via connected components on a key-similarity graph, using neither density-based clustering nor proximity priors. Under identical prior-free conditions, we substantially outperform output-feature baselines, making the emergent 3D instance structure visible.
Figures & tables
Figure 1: Emergent 3D instance segmentation from a frozen, self-supervised point transformer. (a) Outdoor LiDAR scan after ground removal via TerraSeg [ 12 ] . (b) Cosine similarity of every point’s attention key to the key of a single seed point (blue dot). (c) TokenGraph3D instance segmentation: connected components on the key-similarity graph, without density or proximity priors, recover the individual objects.
Figure 2: TokenGraph3D. A frozen, self-supervised point transformer (Utonia [ 33 ] ) provides per-point representations, i.e . internal attention keys or output features, and TerraSeg [ 12 ] removes the ground. Instances are the connected components of a cosine-affinity graph over the non-ground points, with no density and no proximity prior.
Figure 3: Representation probe [Finding 1]. For each encoder stage, we compare the query, key, and value against the output feature, on all three datasets. Across every stage and dataset the query and key carry a strong, nearly identical instance signal, the value is weak, and the output feature falls below the attention queries and keys.
Figure 4: The output feature’s seed-IoU collapses on large objects [Finding 2]. Mean seed-IoU versus object size (number of non-ground points), for the output feature and the enc4 attention key, each at its own best cosine threshold. As objects grow the output feature loses performance, whether by merging neighboring objects or capturing only a fragment, while the attention key remains discriminative on all three datasets. The effect is strongest for the relatively sparse LiDAR of the nuScenes [ 3 ] dataset.
SemanticKITTI
nuScenes
Waymo Perception
Node feature
τ
Sassoc
#clust.
Sassoc
#clust.
Sassoc
#clust.
enc0 Key
0.99
0.713
723.6
0.534
1363.5
0.650
532.9
enc3 Key
0.95
0.571
864.1
0.467
539.2
0.533
1022.4
enc4 Key
0.95
0.726
434.6
0.517
465.9
0.646
545.0
Pre-RoPE
0.98
0.236
1487.3
0.141
255.3
0.191
2143.3
enc4 Query
0.97
0.680
479.2
0.485
377.3
0.627
573.6
Table 1: Prior-free instance segmentation. Association score Sassoc (higher is better) and average predicted clusters per scan (#clust.) on each validation split, under the shared prior-free pipeline (Eq. ( 1 )). We list the two peak stages, enc0 and enc4 ; the intermediate stages are weaker (Finding 3). Best in bold , second-best underlined .
Method
Heuristics and Priors
Metrics
No 2D
No temporal
No density
Sassoc
#clusters
TerraSeg [ 12 ] + DBSCAN [ 6 ]
✓
✓
✗
0.708
500.1
TerraSeg [ 12 ] + HDBSCAN [ 15 ]
✓
✓
✗
0.689
436.6
3DUIS [ 17 ]
✓
✓
✗
0.550 †
-
TARL-Seg [ 18 ]
✓
✗
✗
0.668 †
-
4D-Seg [ 21 ]
✓
✗
✗
0.667 †
-
Table 2: Geometric prior on SemanticKITTI validation split. We report the association score Sassoc and the average number of predicted clusters per scan, using the same pipeline as Tab. 1 . † Taken from [ 21 ] . Best in bold , second-best is underlined .
Method
Heuristics and Priors
Metrics
No 2D
No temporal
No density
Sassoc
#clusters
TerraSeg [ 12 ] + DBSCAN [ 6 ]
✓
✓
✗
0.544
1008.5
TerraSeg [ 12 ] + HDBSCAN [ 15 ]
✓
✓
✗
0.374
244.0
TARL-Seg [ 18 ]
✓
✗
✗
0.189 †
-
4D-Seg [ 21 ]
✓
✗
✗
0.287 †
-
UNIT [ 21 ]
✓
✗
✗
0.390 †
-
Table 3: Geometric prior on nuScenes validation split. We report the association score Sassoc and the average number of predicted clusters per scan, using the same pipeline as Tab. 1 . † Taken from [ 21 ] . Best in bold , second-best is underlined .
Method
Heuristics and Priors
Metrics
No 2D
No temporal
No density
Sassoc
#clusters
TerraSeg [ 12 ] + DBSCAN [ 6 ]
✓
✓
✗
0.636
401.9
TerraSeg [ 12 ] + HDBSCAN [ 15 ]
✓
✓
✗
0.648
537.4
TokenGraph3D (ours)
✓
✓
✓
0.646
545.0
+ proximity (Key)
✓
✓
✗
0.668
491.4
+ proximity (Feat)
✓
✓
✗
0.662
431.3
Table 4: Geometric prior on Waymo Perception validation split. We report the association score Sassoc and the average number of predicted clusters per scan, using the same pipeline as Tab. 1 . Best in bold , second-best is underlined .
Bosch Center for AI, Stuttgart, Germany · Institute for Neuro- and Bioinformatics, University of Lübeck, Lübeck, Germany · Robert Learning Lab, University of Freiburg, Freiburg, Germany