Vision-Language Semantic Grounding for Multi-Domain Crop-Weed Segmentation
Authors: Nazia Hossain, Xintong Jiang, Yu Tian, Philippe Seguin, O. Grant Clark, Shangpeng Sun
Organizations: Department of Bioresource Engineering, McGill University, Sainte-Anne-de-Bellevue, QC, Canada · Department of Plant Science, McGill University, Sainte-Anne-de-Bellevue, QC, Canada
Fine-grained crop-weed segmentation is essential for enabling targeted herbicide application in precision agriculture. However, existing deep learning models struggle to generalize across heterogeneous agricultural environments due to reliance on dataset-specific visual features. We propose Vision-Language Weed Segmentation (VL-WS), a novel framework that addresses this limitation by grounding pixel-level segmentation in semantically aligned, domain-invariant representations. Our architecture employs a dual-encoder design, where frozen Contrastive Language-Image Pretraining (CLIP) embeddings and task-specific spatial features are fused and modulated via Feature-wise Linear Modulation (FiLM) layers conditioned on natural language captions. This design enables image level textual descriptions to guide channel-wise feature refinement while preserving fine-grained spatial localization. Unlike prior works restricted to training and evaluation on single-source datasets, VL-WS is trained on a unified corpus that includes close-range ground imagery (robotic platforms) and high-altitude UAV imagery, covering diverse crop types, weed species, growth stages, and sensing conditions. Experimental results across four benchmark datasets demonstrate the effectiveness of our framework, with VL-WS achieving a mean Dice score of 91.64% and outperforming the CNN baseline by 4.98%. The largest gains occur on the most challenging weed class, where VL-WS attains 80.45% Dice score compared to 65.03% for the best baseline, representing a 15.42% improvement. VL-WS further maintains stable weed segmentation performance under limited target-domain supervision, indicating improved generalization and data efficiency. These findings highlight the potential of vision-language alignment to enable scalable, label-efficient segmentation models deployable across diverse real-world agricultural domains.
Figures & tables
Figure 1: Image-text cosine similarity scores produced by a frozen CLIP model for soybean field images with different levels of weed presence. For crop-dominant scenes, image embeddings show higher similarity to the soybean prompt, while weed-dense scenes exhibit higher similarity to the weed prompt. This shift in similarity demonstrates that CLIP implicitly captures agronomic scene semantics without task-specific fine-tuning.
Figure 2: Overview of the proposed Vision-Language Weed Segmentation (VL-WS) framework. Dense spatial features extracted by a task-specific visual backbone are concatenated with global image embeddings from a pretrained CLIP encoder. The fused features are modulated by natural-language captions through Feature-wise Linear Modulation (FiLM), enabling text-conditioned channel adaptation for pixel-level crop-weed segmentation.
Figure 3: Study area and UAV orthomosaic of the experimental field. The figure illustrates the geographic context of the study site. The top-left panel shows the location of Sainte-Anne-de-Bellevue within the Province of Quebec, Canada, where the study area is located. The top-right panel presents an aerial view of the surrounding agricultural landscape, highlighting the specific experimental plot (outlined in red; 88 m × 18 m). The bottom panel displays the high-resolution UAV orthomosaic acquired using a DJI Mavic 3 Multispectral (M3M) at 5 m altitude, with a spatial scale bar provided for reference.
Figure 4: Representative RGB image tiles and corresponding ground-truth segmentation masks from the four weed segmentation datasets used in this study: UAV Soybean, PhenoBench, GrowingSoy, and ROSE. For each dataset, paired image-mask examples are shown, with RGB images on the left and manually annotated masks on the right. Mask colors denote background (black), crop (green), and weed (red). The datasets span diverse crop types, weed densities, and imaging conditions,, including aerial and ground-based imagery.
Figure 5: Example RGB image tiles paired with agronomy-aware natural language captions generated for each dataset (UAV Soybean, PhenoBench, GrowingSoy, and ROSE). Captions are produced using a standardized template and describe crop and weed presence, coarse spatial layout, and salient visual attributes. These image-caption pairs serve as multimodal inputs for training and evaluating the proposed vision-language segmentation framework.
Figure 6: Detailed architecture of the proposed vision-language weed segmentation framework.
Method
Weed
Crop
Bg
Avg
UNet
61.45
95.73
99.56
85.58
PSPNet
63.47
93.04
99.08
85.19
DeepLabv3+
65.03
95.48
99.47
86.66
VL-WS
80.45
95.23
99.24
91.64
Table 1: Aggregated segmentation performance (Dice %) across all datasets.
Method
UAV Soybean
PhenoBench
GrowingSoy
ROSE
Weed
Crop
Bg
Weed
Crop
Bg
Weed
Crop
Bg
Weed
Crop
Bg
UNet
79.02
98.12
96.73
58.27
96.57
99.70
75.84
92.62
98.80
43.97
88.25
97.11
PSPNet
69.34
97.28
93.98
59.07
93.29
99.22
76.42
91.22
98.35
62.53
90.69
96.75
Deeplabv3+
74.48
98.12
95.53
60.33
96.10
99.62
75.30
92.51
98.65
65.19
90.92
96.85
VL-WS
80.40
98.21
96.47
77.57
95.84
99.49
86.09
92.98
98.66
75.66
92.49
97.30
Table 2: Dataset-wise segmentation performance (Dice %) of the proposed model and baseline methods across four agricultural datasets.
Target Dataset
Source Datasets
Class
Target-Domain Training Data
10%
20%
50%
100%
UAV Soybean
PhenoBench + GrowingSoy + ROSE
Weed
68.63
74.63
79.95
80.40
Crop
96.92
97.31
98.06
98.21
Background
94.80
95.95
96.37
96.47
Mean
86.78
89.29
91.46
91.69
PhenoBench
UAV Soybean + GrowingSoy + ROSE
Weed
69.94
73.66
75.71
77.57
Table 3: Domain adaptation performance under limited target-domain supervision. Class-wise Dice (%) scores obtained by training on three full source datasets and varying fractions of labeled target-domain data.
Figure 7: Qualitative comparison of segmentation results across datasets for the proposed model and baseline methods. Segmentation masks show crop in green, weed in red, and background in black.
Figure 8: Qualitative segmentation results of the proposed VL-WS model across four agricultural datasets, demonstrating robust weed detection under varying lighting conditions, background textures, crop growth stages, and sensing platforms, including UAV and ground-based imagery. Predictions are overlaid on input images with green indicating crop, red indicating weed, and semi-transparent regions showing background.
λVL
Weed
Crop
Bg
Average
0.01
79.93
95.11
99.22
91.42
0.02
80.45
95.23
99.24
91.64
0.03
80.00
95.23
99.25
91.49
0.05
80.41
95.33
99.25
91.66
0.10
80.16
95.26
99.25
91.55
Table 4: Ablation study on the vision-language contrastive loss weight ( λVL ). Class-wise and mean Dice (%) scores are reported across the multi-dataset benchmark.
Figure 9: Field-scale weed segmentation results and zoomed predictions. (a) RGB orthomosaic of the soybean field. (b) Predicted weed distribution map with green indicating crop, red indicating weed, and black indicating background. (c)–-(f) Zoomed predictions from four field regions (top row) with corresponding RGB imagery (bottom row).
Figure 10: Qualitative error analysis of crop-weed segmentation predictions. Predicted segmentation masks are shown, with yellow boxes highlighting regions of inconsistency between model predictions and the ground truth, illustrating common failure cases under challenging field conditions.
Figure 11: Cross-dataset cosine similarity of deep feature embeddings from pretrained CLIP and ResNet encoders. Heatmaps showing pairwise cosine similarity between feature embeddings of image tiles drawn from four weed segmentation datasets: UAV Soybean, Phenobench, GrowingSoy, and ROSE. (Left) CLIP encoder features exhibit high within-dataset and substantial cross-dataset similarity, reflecting semantically rich and more dataset-invariant representations. (Right) Features extracted from a ResNet encoder (pretrained on ImageNet) display strong clustering within datasets but markedly lower similarity across datasets, indicating limited generalizability. The cross-dataset feature alignment achieved by the CLIP encoder supports improved robustness in multi-domain weed segmentation tasks.
Understanding field imagery such as detecting plants and distinguishing individual crop and weed instances is a central challenge in precision agriculture. Despite progress in vision-language tasks like captioning and visual question answering, Visual Grounding (VG), localising language-referred objects, remains unexplored in agriculture. A key reason is the lack of suitable benchmark datasets for evaluating grounding models in field conditions, where many plants look highly similar, appear at multiple scales, and the referred target may be absent from the image. To address these limitations, we introduce gRef-CW, the first dataset designed for generalised visual grounding in agriculture, including negative expressions. Benchmarking current state-of-the-art grounding models on gRef-CW reveals a substantial domain gap, highlighting their inability to ground instances of crops and weeds. Motivated by these findings, we introduce Weed-VG, a modular framework that incorporates multi-label hierarchical relevance scoring and interpolation-driven regression. Weed-VG advances instance-level visual grounding and provides a clear baseline for developing VG methods in precision agriculture. Code will be released upon acceptance.
College of Science and Engineering, James Cook University, Townsville, QLD, Australia · Centre for AI and Data Science Innovation, James Cook University, Townsville, QLD, Australia
Developing robust models for precision vegetable weeding is currently constrained by the scarcity of large-scale, annotated weed-crop datasets. To address this limitation, this study proposes a foundational crop-weed detection model by integrating heterogeneous datasets and leveraging self-supervised learning. A total of 618,642 crop-weed images were initially collected and subsequently refined to 199,388 filtered images for fine-tuning a DINOv3 vision transformer (ViT-small) through a sequential curation strategy. The fine-tuned DINOv3 backbone was then integrated into YOLO26, serving either as a primary backbone or part of a dual-backbone architecture. A feature alignment loss was introduced in the dual backbone framework to enhance feature fusion with minimal computational overhead. Experimental results show that the proposed DINOv3-finetuned ViT-small-based YOLO26-large achieved up to a +5.4% mAP50 gain on in-domain images collected in the 2025 season. Moreover, it demonstrated strong cross-domain generalization with mAP50 improvements of +14.0% on the 2021-2023 season dataset and +11.9% on the 2024 season dataset, compared to the standard YOLO26-large. Although the DINOv3-YOLO26-large model has 45.6% more parameters and a 2.9x increase in inference latency, it maintains real-time performance at ~28.5 frames per second (fps). The curated dataset and software programs developed in this study will be made publicly available.
Boyang Deng, Yuzhen Lu
Department of Biosystems & Agricultural Engineering, Michigan State University East Lansing, MI 48824
Weed detection is an important component of precision agriculture, enabling site-specific weed management and reducing unnecessary herbicide use. Although deep learning methods have achieved strong results for crop and weed detection, many studies rely on single-dataset evaluation, making it difficult to assess robustness across different agricultural domains. This paper presents a multi-dataset benchmark of deep object detectors for weed detection in precision agriculture, with a focused evaluation of YOLO26 models. We evaluate nano, small, and medium variants on seven public weed-detection datasets covering different crops, weed species, field conditions, acquisition setups, and annotation protocols. The models are compared in terms of detection accuracy, model complexity, inference latency, FPS, and model size. In addition to in-dataset evaluation, we investigate cross-domain generalization using a unified one-class weed setup and evaluate multi-source training using the combined training subsets from all datasets. The results show that YOLO26 achieves strong in-dataset performance, with YOLO26m obtaining the highest average accuracy and YOLO26s providing the best practical accuracy-efficiency trade-off. However, cross-domain performance decreases substantially, with YOLO26s dropping from an average in-domain mAP50:95 of 0.603 to 0.148 in the off-domain setting. Multi-source training improves performance on several datasets, but does not fully eliminate domain shift. Overall, the benchmark highlights the importance of dataset diversity, domain similarity, and target-domain adaptation for robust weed detection in real-world precision agriculture applications.