cs.CVMar 5, 2026
SaveMulti-label Instance-level Generalised Visual Grounding in Agriculture
Organizations: College of Science and Engineering, James Cook University, Townsville, QLD, Australia · Centre for AI and Data Science Innovation, James Cook University, Townsville, QLD, Australia
Abstract
Understanding field imagery such as detecting plants and distinguishing individual crop and weed instances is a central challenge in precision agriculture. Despite progress in vision-language tasks like captioning and visual question answering, Visual Grounding (VG), localising language-referred objects, remains unexplored in agriculture. A key reason is the lack of suitable benchmark datasets for evaluating grounding models in field conditions, where many plants look highly similar, appear at multiple scales, and the referred target may be absent from the image. To address these limitations, we introduce gRef-CW, the first dataset designed for generalised visual grounding in agriculture, including negative expressions. Benchmarking current state-of-the-art grounding models on gRef-CW reveals a substantial domain gap, highlighting their inability to ground instances of crops and weeds. Motivated by these findings, we introduce Weed-VG, a modular framework that incorporates multi-label hierarchical relevance scoring and interpolation-driven regression. Weed-VG advances instance-level visual grounding and provides a clear baseline for developing VG methods in precision agriculture. Code will be released upon acceptance.
Figures & tables
Figure 1: Overview of the Weed-VG Framework for Precision Agriculture. This figure summarises the paper’s main contributions for tackling the challenges of visual grounding (VG) in agricultural imagery. It presents gRef-CW, a generalised Referring-expression dataset for Crop and Weed visual grounding, comprising 8,000+ high-resolution field images and 82,000 annotations at both the image level and instance level. The figure also highlights Hierarchical Relevance Scoring (HRS), a modular method that enables existence-aware instance grounding, first determining whether the referred instance is present and then localising it.
Figure 2: gRef-CW Data Collection and Annotation Pipeline. The dataset is constructed through a four-stage process: (1) Filtering out instances unrecognisable by humans and categorising images as single, mixed, or no-target from the selected subset; (2) Extracting instance attributes (e.g. size, category, and position) (3) Composing attributes into natural language templates to generate positive instance-level referring expressions; negative image-level sentences include the non-existence and (4) Replacing categories or swapping attributes to create test sentences.
| Dataset | General Statistics | Linguistics | Visual Properties | ||||||
| Images | Instances | Anns | Cats | Avg. | Neg. | Inst. | Norm. | Img. | |
| Len. | Expr. | Size | Size | Size | |||||
| RefCOCO [ 60 ] | 19,994 | 50,000 | 142,210 | 80 | 3.49 | ✗ | 230–640 | 0.17–1.0 | 640 |
| RefCOCO [ 60 ] | 19,992 | 49,856 | 141,564 | 80 | 3.58 | ✗ | 230–640 | 0.17–1.0 | 640 |
| RefCOCOg [ 42 ] | 26,711 | 54,822 | 104,560 | 80 | 8.46 | ✗ | 277–640 | 0.22–1.0 | 640 |
| ReferItGame [ 26 ] | 19,894 | 96,654 | 130,525 | 238 | 3.45 | ✗ | 360–480 | < 0.3 † | 480 |
Table 1: Statistical comparison of classical referring expression datasets with gRef-CW. Instance and image dimensions are shown as square-root ranges. (Avg. Len. = average annotation length in words; Neg. Expr. = includes negative expressions.) This table contextualises the scale and complexity of gRef-CW rather than direct comparison of the annotation generation process, since our annotations are template-based, unlike the listed datasets.
Figure 3: Detailed instance distribution of the gRef-CW across splits. The figure displays (a) the count of instances stratified by scale (from Tiny to Large) and (b) the number of images categorised by scene density. Annotations within the bars indicate the percentage of Crop (C) and Weed (W) for each subset.
Figure 4: Architecture of the Weed-VG framework. It extends a grounding model with an HRS module that fuses visual and textual features via Multi-Head Cross-Attention and an FFN. HRS decomposes relevance into two levels: (Level 0) global existence detection, which predicts whether the referred object appears in the image, and (Level 1) instance relevance, which ranks region proposals by integrating sentence-level and word-level similarities. A constraint enforces logical consistency by conditioning instance localisation on global existence.
| Model | Validation | Test | |||||||
| Top-1 | Top-5 | [email protected] | mIoU | Top-1 | Top-5 | [email protected] | mIoU | Neg-Acc | |
| MDETR [ 25 ] | 10.20 | 12.81 | 7.82 | 52.35 | 10.16 | 12.97 | 7.78 | 54.19 | 3.32 |
| GDINO-T [ 36 ] | 12.89 | 33.04 | 21.50 | 18.93 | 11.92 | 31.99 | 20.34 | 17.44 | 2.88 |
| GDINO-L [ 36 ] | 19.74 | 42.58 | 28.16 | 23.08 | 20.38 | 43.49 | 28.73 | 23.68 | 7.52 |
| SAM3 [ 5 ] | 33.85 | 66.21 | 46.87 | 31.98 | 34.88 | 66.80 | 46.65 | 32.76 | 25.53 |
| Weed-VG (Ours) | 63.01 | 81.64 | 55.71 | 58.12 | 62.42 | 82.45 | 55.44 | 57.25 | 78.35 |
Table 2: Main results on the gRef-CW. We report Top-1, Top-5, [email protected], and mIoU on both the Validation and Test sets. Negative Accuracy (Neg-Acc) is evaluated on the instance-level negative sentences provided in the Test set.
| Method | Set | Crop (Top-1 | mIoU) | Weed (Top-1 | mIoU) | ||||||
| Tiny | Small | Med | Large | Tiny | Small | Med | Large | ||
| MDETR [ 25 ] | Val | — † | 4.17 | 22.84 | 10.28 | 52.31 | 11.29 | 60.29 | — † | 2.08 | 19.27 | 10.12 | 50.80 | 12.91 | 69.14 |
| Test | — † | 3.31 | 21.18 | 10.34 | 53.56 | 11.71 | 63.40 | — † | 2.79 | 17.61 | 9.68 | 52.55 | 13.09 | 71.48 | |
| GDINO-T [ 36 ] | Val | 0.68 | 0.74 | 11.83 | 14.87 | 23.36 | 35.38 | 35.98 | 60.04 | 0.00 | 0.26 | 6.50 | 9.03 | 18.29 | 28.72 | 43.18 | 73.82 |
| Test | 0.00 | 0.05 | 7.56 | 9.82 | 23.75 | 34.86 | 31.64 | 53.53 | 0.31 | 0.72 | 6.45 | 8.75 | 20.79 | 31.48 | 43.32 | 68.73 | |
| GDINO-L [ 36 ] | Val | 0.70 | 2.21 | 22.48 | 23.24 | 41.52 | 50.46 | 55.99 | 70.63 | 0.75 | 1.29 | 12.44 | 13.85 | 37.15 | 45.26 | 63.53 | 79.55 |
Table 3: Detailed performance breakdown by instance scale on the gRef-CW dataset. We report Top-1 accuracy and mIoU for crops and weeds separately on both Validation and Test sets. Best results are in bold, second-best are underlined.
| Scene Density | Crop ([email protected] | mIoU) | Weed ([email protected] | mIoU) | |||||||||
| MDETR [ 25 ] | GDINO-T [ 36 ] | GDINO-L [ 36 ] | SAM3 [ 5 ] | Weed-VG (Ours) | MDETR [ 25 ] | GDINO-T [ 36 ] | GDINO-L [ 36 ] | SAM3 [ 5 ] | Weed-VG (Ours) | ||
| 1–10 Instances | |||||||||||
| Val | 22.03 | 54.84 | 58.12 | 29.61 | 68.53 | 37.80 | 85.48 | 44.72 | 84.26 | 66.80 | 10.14 | 58.46 | 30.41 | 24.29 | 39.55 | 26.80 | 69.35 | 37.67 | 69.73 | 58.75 | |
| Test | 21.94 | 57.76 | 51.09 | 24.37 | 67.93 | 38.67 | 84.77 | 49.58 | 82.85 | 63.92 | 10.83 | 59.77 | 31.17 | 25.83 | 39.88 | 27.65 | 69.37 | 37.63 | 69.02 | 56.36 | |
| 11–20 Instances | |||||||||||
| Val | 14.85 | 48.33 | 40.86 | 18.37 | 51.43 | 29.00 | 67.70 | 42.07 | 73.63 | 62.56 | 3.75 | 46.43 | 12.00 | 10.56 | 18.68 | 12.33 | 41.23 | 23.85 | 53.77 | 53.36 | |
Table 4: Scalability with respect to scene density on the gRef-CW dataset. We report [email protected] and mIoU for crops and weeds on both Validation and Test sets, grouped by number of instances in the scene. Best results are in bold, second-best are underlined.
| Method | Neg-Acc | |||
| Replace | Swap | Avg. | ||
| Category | Size | Position | ||
| MDETR [ 25 ] | 3.26 | 4.91 | 54.55 | 3.32 |
| GDINO-T [ 36 ] | 14.62 | 11.32 | 16.82 | 2.88 |
| GDINO-L [ 36 ] | 12.84 | 13.36 | 42.42 | 7.52 |
| SAM3 [ 5 ] | 41.02 | 34.03 | 60.75 | 25.53 |
Table 5: Negative Accuracy on the gRef-CW Test set. We report Neg-Acc for Replace and Swap changes, together with the weighted average across categories. Best results are in bold, second-best are underlined.
Figure 5: Qualitative results of Weed-VG on positive and negative referring expressions in gRef-CW across challenging scales (tiny–medium).
| Method | Top-1 | Top-5 | [email protected] | mIoU | Neg-Acc |
| Sentence-only | 49.29 | 74.06 | 50.53 | 49.35 | 74.12 |
| Word-only | 36.19 | 67.61 | 44.20 | 39.22 | 71.55 |
| w/o Query Projection | 33.20 | 58.02 | 38.43 | 35.39 | 69.83 |
| w/o Hierarchical Constraint | 59.87 | 80.57 | 53.87 | 55.83 | 41.60 |
| w/o InterpIoU | 49.15 | 73.28 | 44.61 | 47.72 | 75.88 |
| Full Model | 62.42 | 82.45 | 55.44 | 57.25 | 78.35 |
Table 6: Ablation study of the proposed framework. We evaluate the impact of different text processing strategies (Sentence-only vs. Word-only), removal of the Query Projection, removal of the Hierarchical Constraint, and the impact of the InterpIoU coupled with the matching strategy. All metrics are reported on the Test set.
More in this topic
Visual Grounding
Visual grounding connects language to the image regions, objects, or pixels that support a description, question, or action.
See all 144 papers