In-Field 3D Wheat Head Instance Segmentation From TLS Point Clouds Using Deep Learning Without Manual Labels

Tomislav Medic; Liangliang Nan

cs.CVMar 15, 2026

In-Field 3D Wheat Head Instance Segmentation From TLS Point Clouds Using Deep Learning Without Manual Labels

Authors: Tomislav Medic, Liangliang Nan

Organizations: Institute of Geodesy and Photogrammetry, ETH Zurich, Zurich, Switzerland · Urban Data Science Section, Delft University of Technology, Delft, Netherlands

Abstract

3D instance segmentation for laser scanning (LiDAR) point clouds remains a challenge in many remote sensing-related domains. Successful solutions typically rely on supervised deep learning and manual annotations, and consequently focus on objects that can be well delineated through visual inspection and manual labeling of point clouds. However, for tasks with more complex and cluttered scenes, such as in-field plant phenotyping in agriculture, such approaches are often infeasible. In this study, we tackle the task of in-field wheat head instance segmentation directly from terrestrial laser scanning (TLS) point clouds. To address the problem and circumvent the need for manual annotations, we propose a novel two-stage pipeline. To obtain the initial 3D instance proposals, the first stage uses 3D-to-2D multi-view projections, the Grounded SAM pipeline for zero-shot 2D object-centric segmentation, and multi-view label fusion. The second stage uses these initial proposals as noisy pseudo-labels to train a supervised 3D panoptic-style segmentation neural network. Our results demonstrate the feasibility of the proposed approach and show performance improvementsrelative to Wheat3DGS, a recent alternative solution for in-field wheat head instance segmentation without manual 3D annotations based on multi-view RGB images and 3D Gaussian Splatting, showcasing TLS as a competitive sensing alternative. Moreover, the results show that both stages of the proposed pipeline can deliver usable 3D instance segmentation without manual annotations, indicating promising, low-effort transferability to other comparable TLS-based point cloud segmentation tasks.

Figures & tables

Figure 1: Flowchart of the implemented multi-view projection-based 3D instance segmentation (white: I/O, blue: data processing modules).

Figure 2: Examples of intensity (top) and range (bottom) images generated by the per-scan 3D-to-2D projection module, with a few examples of 2D instance masks generated by the 2D instance segmentation module. For clarity, only a small subset of the masks is shown.

Figure 3: Overview of experiment setup and data collection. Orange rectangles: (1) field phenotyping platform with a camera rig, (2) photogrammetric target (coded marker), (3) reference sphere, (4) TLS. Black rectangles: 7 wheat plots covered within a measurement campaign, from left to right - 5 training, 1 validation, 1 test / hold-out plot.

Figure 4: Sparse reference 3D instance annotations (red) overlaid over $P_{t}$ point cloud of the test / hold-out plot.

Metric	W3DGS	Stage-I	Stage-II	I+II
TP	83	267	296	348
FP	228	109	103	165
FN	375	191	162	110
P	0.27	0.71	0.74	0.68
R	0.18	0.58	0.64	0.76
F1	0.22	0.64	0.69	0.72

Table 1: 3D instance segmentation evaluation metrics for Wheat3DGS (W3DGS) and our pipeline; separately for Stage-I, -II and combined I+II. Legend: TP - true positives, FP - false positives, FN - false negatives, P - precision, R - recall, F1 - F1 score, CE - counting error, RCE - relative counting error, USR - under segmentation ratio in percent.

Figure 5: Detected 3D wheat head instances (random color per instance alignment) overlaid over $P_{t}$ point cloud of the hold-out (test) plot. A few side views are shown in Appendix A.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure A.1: Additional side-view of the detected 3D wheat head instances (random color per instance alignment) overlaid over $P_{t}$ point cloud of the hold-out (test) plot; orange rectangles (1-5) marking regions with visually perceivable differences in instance segmentation performance between the realizations.

Instance Segmentation

Instance segmentation identifies each object and assigns it a separate pixel-level mask in images or video.

See all 159 papers

Emergent 3D Instance Segmentation from Self-Supervised Point Transformers
Aug 16, 2026Ted Lentsch, Santiago Montiel-Marín, Holger Caesar +1Self-Supervised Point Cloud Learning 3D Instance Segmentation
Department of Cognitive Robotics, Delft University of Technology, The Netherlands · Department of Electronics, University of Alcalá, Spain
Zero-Shot 3D Plant Organ Segmentation with SAM3 and Semantic NeRFs
Sep 7, 2026Andreas Gilson, Laura Hennig, Peter PietrzykOpen-Vocabulary 3D Segmentation 3D Semantic Segmentation
Fraunhofer Institute for Integrated Circuits (IIS), Fürth, Germany · Otto-Friedrich-Universität Bamberg, Germany · Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany
SelectAnyTree: A Promptable Instance Segmentation Model for 3D Forest LiDAR Point Clouds
Jun 25, 2026Trung Thanh Nguyen, Daniel Lusk, Kilian Gerberding +103D Point Cloud Segmentation 3D Instance Segmentation
Nagoya University, Japan · University of Freiburg, Germany · University of California, Los Angeles, USA +5

Aug 16, 2026J/K move · Enter open · S save

Emergent 3D Instance Segmentation from Self-Supervised Point Transformers

Ted Lentsch, Santiago Montiel-Marín, Holger Caesar, Julian F. P. Kooij

Department of Cognitive Robotics, Delft University of Technology, The Netherlands · Department of Electronics, University of Alcalá, Spain

Self-Supervised Point Cloud Learning 3D Instance Segmentation Point Cloud Learning 3D ViTs

Open paper AskPDF arXiv

Unsupervised 3D instance segmentation of outdoor LiDAR scans has traditionally relied on handcrafted geometric priors such as density-based clustering, motion cues, or projected 2D detections. In this work, we investigate whether a frozen, self-supervised point transformer already contains the structural information required to isolate object instances without any handcrafted geometric prior. Using this transformer purely as a feature extractor, we probe its internal representations across the SemanticKITTI, nuScenes, and Waymo Perception datasets. Our analysis yields four core insights: (1) the instance signal concentrates in the attention queries and keys rather than in the values or final output features; (2) output features semantically collapse, merging adjacent same-class objects that the queries and keys keep distinct; (3) this instance signal is bimodal in depth, strongest at the shallowest and deepest encoder stages; and (4) this signal is driven predominantly by the rotary position encoding (RoPE), whose removal collapses its advantage. We put these findings into our method TokenGraph3D, a training-free segmenter that groups points via connected components on a key-similarity graph, using neither density-based clustering nor proximity priors. Under identical prior-free conditions, we substantially outperform output-feature baselines, making the emergent 3D instance structure visible.

Abstract

Instance Segmentation

Explore similar work

Emergent 3D Instance Segmentation from Self-Supervised Point Transformers

Zero-Shot 3D Plant Organ Segmentation with SAM3 and Semantic NeRFs

SelectAnyTree: A Promptable Instance Segmentation Model for 3D Forest LiDAR Point Clouds