Organizations: Dept. of Electrical and Computer Engineering, Univ. of California, Riverside, 900 University Avenue, Riverside, CA 92521, USA · Dept. of Environmental Sciences, Univ. of California, Riverside, 900 University Avenue, Riverside, CA 92521, USA · Gallo, 600 Yosemite Blvd, Modesto, CA 95354, USA
The dynamic and heterogeneous nature of agricultural fields presents significant challenges for object detection and localization, particularly for autonomous mobile robots that are tasked with surveying previously unseen unstructured environments. Concurrently, there is a growing need for real-time detection systems that do not depend on large-scale manually labeled real-world datasets. In this work, we introduce a comprehensive annotation-to-detection framework designed to train a robust multi-modal detector using limited and partially labeled training data. The proposed methodology incorporates cross-modal annotation transfer and an early-stage sensor fusion pipeline, which, in conjunction with a multi-stage detection architecture, effectively trains and enhances the system's multi-modal detection capabilities. The effectiveness of the framework was demonstrated through vine trunk detection in novel vineyard settings that featured diverse lighting conditions and varying crop densities to validate performance. When integrated with a customized multi-modal LiDAR and Odometry Mapping (LOAM) algorithm and a tree association module, the system demonstrated high-performance trunk localization, successfully identifying over 70% of trees in a single traversal with a mean distance error of less than 0.37m. The results reveal that by leveraging multi-modal, incremental-stage annotation and training, the proposed framework achieves robust detection performance regardless of limited starting annotations, showcasing its potential for real-world and near-ground agricultural applications.
Figures & tables
Fig. 1 : Our annotation-to-detection framework deployed onboard for multi-modal vine trunk detection. Integration with a customized LOAM framework enables concurrent vine trunk localization and the generation of attribute-rich sparse maps.
Fig. 2 : Our proposed annotation-to-detection framework for multi-modal vine trunk detection. The annotation pipeline initiates a tree trunk dataset in the form of annotation masks Mtm by combining multi-modal input ( Itv , Itn , Itτ ) and available spatial information during the robot’s operation. The detection pipeline is coupled with the annotated (pseudo-labeled) dataset to iteratively refine and augment the available data, towards training a robust multi-modal vine Vtm detector.
Fig. 3 : Generated semantic masks Mtτ , based on a thermal image Itτ input, by using SAM [ 14 ] and our shape-based mask filter.
Fig. 4 : Mask association and pseudo-labeling . By employing the interpolated point clouds Ptm and filtered masks Mtm , the vines are spatially associated to determine if they represent the same object across different modalities.
Fig. 5 : Evolution of vine trunk detection. By transitioning from the DS model, trained on the partially annotated dataset (b), to DT model, which incorporates the enriched pseudo-labeled dataset (d), the detector achieves superior performance across all input modalities.
Metrics
DS
DS+
DT
Precision
0.02
0.16
0.83
Recall
0.14
0.35
0.53
mAP 50
0.17
0.37
0.55
mAP 50:95
0.08
0.19
0.26
TABLE I : Evaluation Stages of Detector D Aiming Low-False-Positive Vine Trunk Detection
Experiment
Single-Row with 10 Trees
Dual-Row with 7×7 Trees
Metrics
1st−5th Tree
5th−10th Tree
Total
Total
L2 Distance Error
0.32 m
0.24 m
0.28 m
0.37 m
MAE <0.5
0.05 m
0.11 m
0.09 m
0.06 m
RMSE <0.5
0.05 m
0.12 m
0.10 m
0.07 m
Recall <0.5
60%
80%
80%
71%
Total Tree Detections
4 out of 5
4 out of 5
8 out of 10
10 out of 14
TABLE II : Evaluation of Vine Trunk Localization
Fig. 6 : Field assessment of vine detection and localization. Our system accurately detects and localizes tree trunks in a single pass across both single-row (10 trees, left) and dual-row ( 7×7 trees, right) configurations. By incorporating multi-modal information into the LOAM registration process, the system generates feature-rich sparse maps − such as the illustrated thermal visualizations − simultaneously with real-time vine trunk detection.
Reliable long-term deployment of autonomous robots in agricultural environments remains challenging due to perceptual aliasing, seasonal variability, and the dynamic nature of crop canopies. Vineyards, characterized by repetitive row structures and significant visual changes across phenological stages, represent a pivotal field challenge, limiting the robustness of conventional feature-based localization and mapping approaches. This paper introduces VinePT-Map, a semantic mapping framework that leverages vine trunks and support poles as persistent structural landmarks to enable season-agnostic and resilient robot localization. The proposed method formulates the mapping problem as a factor graph, integrating GPS, IMU, and RGB-D observations through robust geometrical constraints that exploit vineyard structure. An efficient perception pipeline based on instance segmentation and tracking, combined with a clustering filter for outlier rejection and pose refinement, enables accurate landmark detection using low-cost sensors and onboard computation. To validate the pipeline, we present a multi-season dataset for trunk and pole segmentation and tracking. Extensive field experiments conducted across diverse seasons demonstrate the robustness and accuracy of the proposed approach, highlighting its suitability for long-term autonomous operation in agricultural environments.
Giorgio Audrito, Mauro Martini, Alessandro Navone +2
Department of Electronics and Telecommunications, Politecnico di Torino, 10129, Torino, Italy.
While autonomous rovers have become indispensable to precision farming, achieving consistent operational safety remains a critical challenge. Conventional safety sensors, such as LiDAR, fail to detect obstacles positioned below the plant canopy, posing a significant risk. While camera-based supervised learning methods can detect common objects, they perform poorly when faced with obstacles that were not present in their training data. Actual unsupervised anomaly detection offers a solution by learning the normal visual patterns of an environment, but often fails for the dynamic scenes captured by a moving rover.\ This paper introduces Video Memory Transformers for Anomaly Detection (VMTAD), a fully unsupervised method designed for real-time obstacle detection in dynamic agricultural scenes. VMTAD utilizes a transformer-driven architecture augmented with a dedicated memory module. This memory module leverages temporal context by processing encoded representations of preceding frames. This approach enables the system to effectively address the dynamic context caused by the robot's movement. The model is trained using only images that represent normal operation, requiring no data labels.\ VMTAD was rigorously evaluated on the 'Grillion' agricultural rover. On a challenging rapeseed dataset, VMTAD achieved state-of-the-art performance, reaching a 0.973 detection and 0.997 segmentation Area Under the Receiver Operating Characteristic curve. A lightweight variant provides an optimal balance of high accuracy and real-time inference (14 ms), which is critical for safety, as confirmed by our analysis of the rover's total stopping distance.
Beyond the immediate biophysical benefits, urban trees play a foundational role in environmental sustainability and disaster mitigation. Precise mapping of urban trees is essential for environmental monitoring, post-disaster assessment, and strengthening policy. However, the transition from traditional, labor-intensive field surveys to scalable automated systems remains limited by high annotation costs and poor generalization across diverse urban scenarios. This study introduces a multimodal framework that integrates high-resolution satellite imagery with ground-level Google Street View to enable scalable and detailed urban tree detection under limited-annotation conditions. The framework first leverages satellite imagery to localize tree candidates and then retrieves targeted ground-level views for detailed detection, significantly reducing inefficient street-level sampling. To address the annotation bottleneck, domain adaptation is used to transfer knowledge from an existing annotated dataset to a new region of interest. To further minimize human effort, we evaluated three learning strategies: semi-supervised learning, active learning, and a hybrid approach combining both, using a transformer-based detection model. The hybrid strategy achieved the best performance with an F1-score of 0.90, representing a 12% improvement over the baseline model. In contrast, semi-supervised learning exhibited progressive performance degradation due to confirmation bias in pseudo-labeling, while active learning steadily improved results through targeted human intervention to label uncertain or incorrect predictions. Error analysis further showed that active and hybrid strategies reduced both false positives and false negatives. Our findings highlight the importance of a multimodal approach and guided annotation for scalable, annotation-efficient urban tree mapping to strengthen sustainable city planning.
In Seon Kim, Ali Moghimi
Department of Computer Science, University of California, Davis, , Davis, 95616, CA, USA · Department of Biological and Agricultural Engineering, University of California, Davis, Davis, CA, 95616, USA