Vision-Language Semantic Grounding for Multi-Domain Crop-Weed Segmentation

Nazia Hossain; Xintong Jiang; Yu Tian; Philippe Seguin; O. Grant Clark; Shangpeng Sun

cs.CVFeb 27, 2026

Vision-Language Semantic Grounding for Multi-Domain Crop-Weed Segmentation

Authors: Nazia Hossain, Xintong Jiang, Yu Tian, Philippe Seguin, O. Grant Clark, Shangpeng Sun

Organizations: Department of Bioresource Engineering, McGill University, Sainte-Anne-de-Bellevue, QC, Canada · Department of Plant Science, McGill University, Sainte-Anne-de-Bellevue, QC, Canada

Abstract

Fine-grained crop-weed segmentation is essential for enabling targeted herbicide application in precision agriculture. However, existing deep learning models struggle to generalize across heterogeneous agricultural environments due to reliance on dataset-specific visual features. We propose Vision-Language Weed Segmentation (VL-WS), a novel framework that addresses this limitation by grounding pixel-level segmentation in semantically aligned, domain-invariant representations. Our architecture employs a dual-encoder design, where frozen Contrastive Language-Image Pretraining (CLIP) embeddings and task-specific spatial features are fused and modulated via Feature-wise Linear Modulation (FiLM) layers conditioned on natural language captions. This design enables image level textual descriptions to guide channel-wise feature refinement while preserving fine-grained spatial localization. Unlike prior works restricted to training and evaluation on single-source datasets, VL-WS is trained on a unified corpus that includes close-range ground imagery (robotic platforms) and high-altitude UAV imagery, covering diverse crop types, weed species, growth stages, and sensing conditions. Experimental results across four benchmark datasets demonstrate the effectiveness of our framework, with VL-WS achieving a mean Dice score of 91.64% and outperforming the CNN baseline by 4.98%. The largest gains occur on the most challenging weed class, where VL-WS attains 80.45% Dice score compared to 65.03% for the best baseline, representing a 15.42% improvement. VL-WS further maintains stable weed segmentation performance under limited target-domain supervision, indicating improved generalization and data efficiency. These findings highlight the potential of vision-language alignment to enable scalable, label-efficient segmentation models deployable across diverse real-world agricultural domains.

Figures & tables

Figure 1: Image-text cosine similarity scores produced by a frozen CLIP model for soybean field images with different levels of weed presence. For crop-dominant scenes, image embeddings show higher similarity to the soybean prompt, while weed-dense scenes exhibit higher similarity to the weed prompt. This shift in similarity demonstrates that CLIP implicitly captures agronomic scene semantics without task-specific fine-tuning.

Figure 2: Overview of the proposed Vision-Language Weed Segmentation (VL-WS) framework. Dense spatial features extracted by a task-specific visual backbone are concatenated with global image embeddings from a pretrained CLIP encoder. The fused features are modulated by natural-language captions through Feature-wise Linear Modulation (FiLM), enabling text-conditioned channel adaptation for pixel-level crop-weed segmentation.

Figure 3: Study area and UAV orthomosaic of the experimental field. The figure illustrates the geographic context of the study site. The top-left panel shows the location of Sainte-Anne-de-Bellevue within the Province of Quebec, Canada, where the study area is located. The top-right panel presents an aerial view of the surrounding agricultural landscape, highlighting the specific experimental plot (outlined in red; 88 m $\times$ 18 m). The bottom panel displays the high-resolution UAV orthomosaic acquired using a DJI Mavic 3 Multispectral (M3M) at 5 m altitude, with a spatial scale bar provided for reference.

Figure 4: Representative RGB image tiles and corresponding ground-truth segmentation masks from the four weed segmentation datasets used in this study: UAV Soybean, PhenoBench, GrowingSoy, and ROSE. For each dataset, paired image-mask examples are shown, with RGB images on the left and manually annotated masks on the right. Mask colors denote background (black), crop (green), and weed (red). The datasets span diverse crop types, weed densities, and imaging conditions,, including aerial and ground-based imagery.

Figure 5: Example RGB image tiles paired with agronomy-aware natural language captions generated for each dataset (UAV Soybean, PhenoBench, GrowingSoy, and ROSE). Captions are produced using a standardized template and describe crop and weed presence, coarse spatial layout, and salient visual attributes. These image-caption pairs serve as multimodal inputs for training and evaluating the proposed vision-language segmentation framework.

Figure 6: Detailed architecture of the proposed vision-language weed segmentation framework.

Method	Weed	Crop	Bg	Avg
UNet	61.45	95.73	99.56	85.58
PSPNet	63.47	93.04	99.08	85.19
DeepLabv3+	65.03	95.48	99.47	86.66
VL-WS	80.45	95.23	99.24	91.64

Table 1: Aggregated segmentation performance (Dice %) across all datasets.

Method	UAV Soybean			PhenoBench			GrowingSoy			ROSE
	Weed	Crop	Bg	Weed	Crop	Bg	Weed	Crop	Bg	Weed	Crop	Bg
UNet	79.02	98.12	96.73	58.27	96.57	99.70	75.84	92.62	98.80	43.97	88.25	97.11
PSPNet	69.34	97.28	93.98	59.07	93.29	99.22	76.42	91.22	98.35	62.53	90.69	96.75
Deeplabv3+	74.48	98.12	95.53	60.33	96.10	99.62	75.30	92.51	98.65	65.19	90.92	96.85
VL-WS	80.40	98.21	96.47	77.57	95.84	99.49	86.09	92.98	98.66	75.66	92.49	97.30

Table 2: Dataset-wise segmentation performance (Dice %) of the proposed model and baseline methods across four agricultural datasets.

Target Dataset	Source Datasets	Class	Target-Domain Training Data
Target Dataset	Source Datasets	Class	10%	20%	50%	100%
UAV Soybean	PhenoBench + GrowingSoy + ROSE	Weed	68.63	74.63	79.95	80.40
		Crop	96.92	97.31	98.06	98.21
		Background	94.80	95.95	96.37	96.47
		Mean	86.78	89.29	91.46	91.69
PhenoBench	UAV Soybean + GrowingSoy + ROSE	Weed	69.94	73.66	75.71	77.57

Table 3: Domain adaptation performance under limited target-domain supervision. Class-wise Dice (%) scores obtained by training on three full source datasets and varying fractions of labeled target-domain data.

Figure 7: Qualitative comparison of segmentation results across datasets for the proposed model and baseline methods. Segmentation masks show crop in green, weed in red, and background in black.

Figure 8: Qualitative segmentation results of the proposed VL-WS model across four agricultural datasets, demonstrating robust weed detection under varying lighting conditions, background textures, crop growth stages, and sensing platforms, including UAV and ground-based imagery. Predictions are overlaid on input images with green indicating crop, red indicating weed, and semi-transparent regions showing background.

$\lambda_{\text{VL}}$	Weed	Crop	Bg	Average
0.01	79.93	95.11	99.22	91.42
0.02	80.45	95.23	99.24	91.64
0.03	80.00	95.23	99.25	91.49
0.05	80.41	95.33	99.25	91.66
0.10	80.16	95.26	99.25	91.55

Table 4: Ablation study on the vision-language contrastive loss weight (

\lambda_{\mathrm{VL}}

). Class-wise and mean Dice (%) scores are reported across the multi-dataset benchmark.

Figure 9: Field-scale weed segmentation results and zoomed predictions. (a) RGB orthomosaic of the soybean field. (b) Predicted weed distribution map with green indicating crop, red indicating weed, and black indicating background. (c)–-(f) Zoomed predictions from four field regions (top row) with corresponding RGB imagery (bottom row).

Figure 10: Qualitative error analysis of crop-weed segmentation predictions. Predicted segmentation masks are shown, with yellow boxes highlighting regions of inconsistency between model predictions and the ground truth, illustrating common failure cases under challenging field conditions.

Figure 11: Cross-dataset cosine similarity of deep feature embeddings from pretrained CLIP and ResNet encoders. Heatmaps showing pairwise cosine similarity between feature embeddings of image tiles drawn from four weed segmentation datasets: UAV Soybean, Phenobench, GrowingSoy, and ROSE. (Left) CLIP encoder features exhibit high within-dataset and substantial cross-dataset similarity, reflecting semantically rich and more dataset-invariant representations. (Right) Features extracted from a ResNet encoder (pretrained on ImageNet) display strong clustering within datasets but markedly lower similarity across datasets, indicating limited generalizability. The cross-dataset feature alignment achieved by the CLIP encoder supports improved robustness in multi-domain weed segmentation tasks.

Domain Generalization for Semantic Segmentation

See all 14 papers

Multi-label Instance-level Generalised Visual Grounding in Agriculture
Mar 5, 2026Mohammadreza Haghighat, Alzayat Saleh, Mostafa Rahimi AzghadiVisual Grounding Agricultural Image Analysis
College of Science and Engineering, James Cook University, Townsville, QLD, Australia · Centre for AI and Data Science Innovation, James Cook University, Townsville, QLD, Australia
DINOv3 Meets YOLO26 for Weed Detection in Vegetable Crops
Feb 25, 2026Boyang Deng, Yuzhen LuWeed Detection Precision Agriculture
Department of Biosystems & Agricultural Engineering, Michigan State University East Lansing, MI 48824
A Multi-Dataset Benchmark of YOLO-Based Weed Detection in Precision Agriculture
Sep 27, 2026Hristina Zdraveska, Vlatko Spasev, Ivica Dimitrovski +2Multi-Source Domain Adaptation Weed Detection
Faculty of Computer Science and Engineering, University Ss Cyril and Methodius, Rudzer Boshkovikj 16, P.O. 393, Skopje, 1000, North Macedonia

Mar 5, 2026J/K move · Enter open · S save

Multi-label Instance-level Generalised Visual Grounding in Agriculture

Mohammadreza Haghighat, Alzayat Saleh, Mostafa Rahimi Azghadi

College of Science and Engineering, James Cook University, Townsville, QLD, Australia · Centre for AI and Data Science Innovation, James Cook University, Townsville, QLD, Australia

Visual Grounding Agricultural Image Analysis Precision Agriculture

Open paper AskPDF arXiv

Understanding field imagery such as detecting plants and distinguishing individual crop and weed instances is a central challenge in precision agriculture. Despite progress in vision-language tasks like captioning and visual question answering, Visual Grounding (VG), localising language-referred objects, remains unexplored in agriculture. A key reason is the lack of suitable benchmark datasets for evaluating grounding models in field conditions, where many plants look highly similar, appear at multiple scales, and the referred target may be absent from the image. To address these limitations, we introduce gRef-CW, the first dataset designed for generalised visual grounding in agriculture, including negative expressions. Benchmarking current state-of-the-art grounding models on gRef-CW reveals a substantial domain gap, highlighting their inability to ground instances of crops and weeds. Motivated by these findings, we introduce Weed-VG, a modular framework that incorporates multi-label hierarchical relevance scoring and interpolation-driven regression. Weed-VG advances instance-level visual grounding and provides a clear baseline for developing VG methods in precision agriculture. Code will be released upon acceptance.

Abstract

Domain Generalization for Semantic Segmentation

Explore similar work

Multi-label Instance-level Generalised Visual Grounding in Agriculture

DINOv3 Meets YOLO26 for Weed Detection in Vegetable Crops

A Multi-Dataset Benchmark of YOLO-Based Weed Detection in Precision Agriculture