cs.CVOct 3, 2026
SaveAgroGround: Multi-Granularity Grounded Recognition in Agriculture
Organizations: Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, United Arab Emirates
Abstract
Agricultural visual models are typically evaluated for either recognition or localization, but reliable diagnosis requires identifying what is present and localizing the evidence. Agricultural visual question answering (VQA) datasets carry rich semantic labels but rarely link them to image regions, and adding such annotations by hand is costly at scale. We introduce AgroGround, a large-scale dataset for grounded agricultural recognition: identifying plant diseases and other agricultural targets and localizing their image regions. An automated pipeline converts the labels of eight agricultural VQA datasets into annotations for disease lesions and whole objects, producing 794,850 instruction examples. Healthy images provide negative supervision for disease queries, teaching the model to return empty predictions. We fine-tune a shared vision-language model on known-target grounding instructions combined with instructions requiring both recognition and localization. We evaluate predicted identities, regions, joint correctness, and healthy-image abstention on 1,480 human-verified images disjoint from all training data. Grounding-only fine-tuning reduces recognition accuracy from 51.8% to 29.1%, while adding recognition-and-localization instructions raises it to 72.6%. With images and annotations held fixed, combining the two formats raises joint accuracy from 19.2% to 43.3% at comparable grounding. Healthy negatives raise abstention on healthy images to 95.0%, and reinforcement learning improves lesion-level grounding. The resulting 2B model exceeds its annotation teacher in grounding F1 on our benchmark and on the external PlantSeg test set. AgroGround establishes a benchmark for grounded agricultural recognition, measuring joint correctness of identity and localization along with abstention on healthy images. The code is available at https://github.com/AB-Abdulla/AgroGround.
Figures & tables
Figure 1: Data construction in AgroGround. Left: target labels and host crops are extracted from the answers of eight agricultural VQA datasets, and samples that name a visually localizable target are retained as groundable (§ 3.2 ). (A) For each groundable sample, a crop-aware prompt drives GroundingDINO ( Liu et al., 2024a ) . SAM2 ( Ravi et al., 2025 ) masks and geometry checks filter the proposals, and the retained boxes are assigned to lesion-level or object-level instructions. (B) Healthy images, excluding benchmark images, provide negatives in both instruction formats with an empty target (§ 4.3 ). (C) Held-out candidates are audited against training positives and negatives by file identity and perceptual hash, and detector pre-annotations (orange) are corrected into human-verified boxes (green) (§ 3.3 ). All images, prompts, and boxes shown are real. The running example is a benchmark image of cashew anthracnose, and its automatic boxes are shown for illustration only and were not used for training.
Figure 2: Overview of the AgroGround model and training. Top: a shared vision-language model takes an image and an instruction, either known-target grounding or recognition and localization, and outputs the target name with boxes at the requested granularity, or an empty list when the queried target is absent. (A) Supervised fine-tuning mixes the two instruction formats with healthy negatives whose target is an empty list (§ 4.2 – 4.3 ). (B) Reinforcement learning starts from the supervised model and optimizes a grounded reward for localization, recall, recognition, abstention, and valid format (§ 4.4 ). Example outputs are predictions of the supervised 2B model on benchmark images.
| Training | Benchmark | |
|---|---|---|
| Annotation | automatic | human-verified |
| Image files | 171,580 | 1,480 |
| Lesion level | 391,758 ex. | 508 img. |
| Object level | 403,092 ex. | 467 img. |
| Healthy images | 43,000 | 505 |
| Disease targets | 97.6% | 62.3% |
Table 1: Training corpus and benchmark. Percentages are over groundable samples (training) and positive images (benchmark).
Figure 3: Grounding-only training echoes the first-listed candidate, while recognition training with randomized answer positions yields a uniform distribution. The model decides from image content, not prompt order.
| Localization | Recognition | Abstention | External | ||||
| Model | F1 [95% CI] | Lesion F1 | Acc. (%) | Joint (%) | Healthy (%) | False (%) | PlantSeg F1 |
| GroundingDINO ( Liu et al., 2024a ) | |||||||
| W. raw prompts | 0.401 [0.368, 0.434] | 0.323 | 27.6 † | 17.5 † | 0.0 | 0.1 | 0.182 |
| W. taxonomy prompts | 0.395 [0.374, 0.416] | 0.358 | – | – | 0.0 | 0.0 | 0.183 |
| W. annotator configuration | 0.392 [0.363, 0.425] | 0.282 | – | – | 20.2 | 0.5 | 0.161 |
| VisionReasoner-7B ( Liu et al., 2026 ) | 0.369 [0.340, 0.401] | 0.302 | 40.6 † | 26.8 † | 0.2 | 0.5 | – |
Table 2: Comparison with existing methods. Localization and abstention use the disease-given protocol, and recognition uses the candidate protocol. † Protocol adaptations in Appendix G . –: not supported or not evaluated. Complete results are in Appendix H .
| Configuration | F1 | Lesion F1 | Recog. (%) | Pos.-1 (%) | Joint (%) | Healthy abst. (%) | False abst. (%) |
|---|---|---|---|---|---|---|---|
| (a) Training components | |||||||
| Zero-shot Qwen3-VL-2B | 0.320 | 0.205 | 51.8 | 33.7 | 34.5 | 1.2 | 0.0 |
| Grounding only | 0.426 | 0.366 | 29.1 | 88.8 | 19.2 | 0.0 | 0.0 |
| Mixed | 0.435 | 0.377 | 72.6 | 26.7 | 43.3 | 0.2 | 0.0 |
| Mixed + healthy (6%) | 0.433 | – | 73.0 | – | 44.3 | 93.3 | 5.2 |
| Mixed + healthy (12%) | 0.440 | 0.392 | 74.7 | 25.1 | 44.8 | 95.0 | 4.6 |
Table 3: Ablation on the 2B model. (a) Supervised configurations are trained from the base model. RL starts from mixed + healthy (12%). (b) One-epoch RL variants. The main RL model uses step 170 (about 1.2 epochs). Pos.-1: share of in-list answers that select the first candidate.
Figure 4: Disease-given grounding on two dense benchmark images (rows 1 and 6 of Figure 7 , seed 3). Numbers are box counts.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Samples ingested | Groundable | Primary domain |
| CDDM ( Liu et al., 2024b ) | 1,060,294 | 513,650 | Crop disease QA |
| LeafNet ( Nguyen Quoc et al., 2026 ) | 121,337 | 81,747 | Leaf disease |
| MIRAGE ( Dongre et al., 2025 ) | 40,889 | 10,560 | Expert consultations |
| AgroMind ( Li et al., 2025 ) | 28,482 | 1,664 | Agricultural remote sensing |
| LeafBench ( Nguyen Quoc et al., 2026 ) | 13,950 | 4,140 | Leaf disease VQA |
| AgroCoT ( Wen et al., 2025 ) | 4,535 | 318 | Chain-of-thought reasoning |
Table 4: Source datasets and cleaning outcome. Groundable and non-groundable (659,116) partitions sum exactly to the ingested total. Sample counts are of the releases we ingested, which differ from the source papers’ headline figures in four cases: LeafNet is the public enalis/LeafNet training split (121,337 rows) rather than the full 186,000-image dataset. AgroCoT is the v1 release (4,535 samples, then named AgriCoT, with v2 onward reporting 4,759). AgMMU is the released evaluation file as distributed (772 entries, 770 with distinct identifiers) against the 746 questions reported in the paper, and the CDDM figure is our own count of its released question-answer files, the paper stating “137,000 images” and “1 million” question-answer pairs. Two sources carry partial box-level ground truth: AgroBench provides bounding boxes for its 609 weed-identification images to mark which weed each question asks about, and a boundary-analysis subset of AgroCoT embeds box coordinates as text inside answer strings. We use neither as supervision. Every training box comes from the annotation pipeline.
| Kind | Samples | Share | Description |
|---|---|---|---|
| Disease | 600,905 | 97.6% | Named diseases with visual symptoms |
| Object | 7,278 | 1.2% | Identifiable agricultural objects/organisms |
| Pest | 4,103 | 0.7% | Insect and invertebrate pests |
| Symptom description | 2,543 | 0.4% | Descriptive symptom queries |
| Weed | 609 | 0.1% | Weed species |
| Stress condition | 47 | 0.1% | Abiotic stress |
Table 5: Groundable samples by kind.
| Round | Images | Proposals | Kept ( 0.9) | Adjusted | Deleted | Final boxes | Annotator-drawn | IoU (matched) |
|---|---|---|---|---|---|---|---|---|
| 1 | 213 | 356 | 51.4% | 6.5% | 42.1% | 791 | 74.0% | 0.963 |
| 2 | 106 | 411 | 68.6% | 0.5% | 30.9% | 674 | 57.9% | 0.997 |
| 3 | 218 | 448 | 72.8% | 2.5% | 24.8% | 791 | 57.4% | 0.991 |
| 4 | 438 | 623 | 69.8% | 1.9% | 28.3% | 1,207 | 63.0% | 0.991 |
| All | 975 | 1,838 | 66.7% | 2.6% | 30.7% | 3,463 | 63.2% | 0.988 |
Table 6: Pre-annotation anchoring by round: how the annotator treated the detector’s editable proposals, and how much of the final ground truth is annotator-drawn.
| Model | Micro F1 | Macro F1 | Dense F1 | Non-dense F1 | [email protected] |
|---|---|---|---|---|---|
| GroundingDINO (raw) | 0.401 | 0.577 | 0.253 | 0.553 | 0.311 |
| GroundingDINO (taxonomy) | 0.395 | 0.475 | 0.315 | 0.459 | 0.375 |
| VisionReasoner-7B | 0.369 | 0.595 | 0.219 | 0.542 | 0.224 |
| Seg-Zero-7B | 0.247 | 0.504 | 0.012 | 0.519 | 0.206 |
| Qwen3-VL-2B (zero-shot) | 0.320 | 0.584 | 0.076 | 0.602 | 0.239 |
| Qwen3-VL-8B (zero-shot) | 0.380 | 0.615 | 0.160 | 0.643 | 0.270 |
Table 7: Metric variants, disease-given protocol. Dense = the 180 benchmark images with 5 boxes. Macro F1 averages per image over the 975 positives.
| Model | [email protected] [95% CI] | R | P | RIoU | Fine F1 | Coarse F1 | Abst. | False abst. |
|---|---|---|---|---|---|---|---|---|
| GroundingDINO (raw prompts) | 0.401 [0.368, 0.434] | 0.333 | 0.503 | 0.594 | 0.323 | 0.608 | 0.0% | 0.1% |
| GroundingDINO (annotation-time taxonomy prompts) | 0.395 [0.374, 0.416] | 0.411 | 0.380 | 0.537 | 0.358 | 0.476 | 0.0% | 0.0% |
| GroundingDINO + annotation-pipeline filters (annotator config) | 0.392 [0.363, 0.425] | 0.295 | 0.584 | 0.586 | 0.282 | 0.700 | 20.2% | 0.5% |
| VisionReasoner-7B ( Liu et al., 2026 ) | 0.369 [0.340, 0.401] | 0.314 | 0.448 | 0.618 | 0.302 | 0.563 | 0.2% | 0.5% |
| Seg-Zero-7B ( Liu et al., 2025a ) | 0.247 [0.223, 0.276] | 0.159 | 0.563 | 0.557 | 0.124 | 0.604 | 0.0% | 0.0% |
| Qwen3-VL-2B (zero-shot) | 0.320 [0.288, 0.357] | 0.220 | 0.584 | 0.614 | 0.205 | 0.661 | 1.2% | 0.0% |
Table 8: Grounding with the disease given, on 1,480 human-verified images (975 positives, 505 healthy). RIoU = region-level union IoU. Abst. / false abst. = empty output on healthy / diseased images. Existence models’ abstention uses a crop-matched disease query per healthy image. The AgroGround-2B rows are the seed-42 run, the best of three seeds on F1, recognition, joint accuracy, and abstention (Appendix N ). GroundingDINO is scored with raw names, with the taxonomy prompts that produced the training labels, and with the annotation pipeline’s filters.
| Model | Recognition [95% CI] | Pos.-1 share | Joint [95% CI] | Abst. | False abst. |
|---|---|---|---|---|---|
| GroundingDINO (best candidate) | 27.6% [24.8, 30.4] | 26.6% | 17.5% [15.4, 19.9] | 0.0% | 0.0% |
| Qwen3-VL-2B (zero-shot) | 51.8% [48.4, 54.8] | 33.7% | 34.5% [31.5, 37.4] | 0.0% | 3.0% |
| Qwen3-VL-8B (zero-shot) | 64.4% [61.1, 67.1] | 26.4% | 44.6% [41.5, 47.9] | 31.1% | 2.6% |
| AgroGround-2B, grounding-only | 29.1% [26.3, 32.1] | 88.8% | 19.2% [16.8, 21.7] | 0.0% | 1.8% |
| AgroGround-2B, +recognition format | 72.6% [70.0, 75.4] | 26.7% | 43.3% [40.2, 46.4] | 0.0% | 0.0% |
| AgroGround-2B, +existence 12% | 74.7% [71.9, 77.5] | 25.1% | 44.8% [41.7, 48.0] | 89.1% | 0.9% |
Table 9: Grounded disease recognition on 1,480 images (975 positives, 505 healthy), candidate protocol. Pos.-1 = share of in-list picks that are the first-listed candidate (uniform = 25%). Joint = correct disease and region IoU 0.5. VisionReasoner’s adapted score is reported in Table 2 and Appendix G .
| Model | LeafNet (342) | MIRAGE (289) | AgroBench (174) | CDDM (80) | AgMMU (35) | AgroMind (33) | LeafBench (20) |
|---|---|---|---|---|---|---|---|
| GroundingDINO | 20.2 / 0.394 | 32.9 / 0.585 | 31.0 / 0.270 | 33.8 / 0.339 | 31.4 / 0.333 | 27.3 / 0.598 | 20.0 / 0.278 |
| GroundingDINO (taxonomy prompts) | – / 0.402 | – / 0.453 | – / 0.327 | – / 0.392 | – / 0.345 | – / 0.441 | – / 0.329 |
| Qwen3-VL-8B (zero-shot) | 53.8 / 0.352 | 84.4 / 0.598 | 65.5 / 0.247 | 52.5 / 0.418 | 48.6 / 0.362 | 57.6 / 0.396 | 35.0 / 0.341 |
| AgroGround-2B, +recognition | 85.1 / 0.439 | 69.6 / 0.593 | 50.0 / 0.293 | 98.8 / 0.501 | 45.7 / 0.365 | 72.7 / 0.477 | 50.0 / 0.356 |
| AgroGround-2B, +existence 12% | 85.4 / 0.443 | 74.7 / 0.593 | 48.9 / 0.311 | 100.0 / 0.495 | 62.9 / 0.252 | 66.7 / 0.471 | 55.0 / 0.458 |
| AgroGround-2B, +GRPO | 80.4 / 0.497 | 65.1 / 0.570 | 48.3 / 0.344 | 95.0 / 0.539 | 51.4 / 0.369 | 66.7 / 0.552 | 60.0 / 0.580 |
Table 10: Per-source recognition accuracy (%, candidate protocol) / disease-given [email protected] on the benchmark positives (number of positives in parentheses, AgroCoT’s 2 positives omitted). The fine-tuned 2B models dominate on in-domain leaf sources, while the zero-shot 8B base is stronger at recognition on field sources. The fine-tuned 8B combines both on MIRAGE but not on AgroBench (recognition only shown for it, grounding only for the taxonomy-prompt GroundingDINO).
| Configuration | [email protected] [CI] | R | Fine F1 | Boxes (dense) | Abst. given / cand. | False abst. | Recog. | Joint |
|---|---|---|---|---|---|---|---|---|
| SFT initialization (+existence 12%) | 0.440 [0.409, 0.474] | 0.320 | 0.392 | 3.62 | 95.0 / 89.1 | 4.6% | 74.7% | 44.8% |
| Main reward, dense-oversampled rollouts, step 170 (reported) | 0.477 [0.448, 0.509] | 0.362 | 0.439 | 4.27 | 87.7 / 88.5 | 1.0% | 69.2% | 43.1% |
| Main reward, final checkpoint (2 epochs, step 292) | 0.467 [0.436, 0.497] | 0.347 | 0.440 | 4.02 | 89.3 / 95.2 | 6.5% | 67.2% | 39.0% |
| Recognition-weighted reward (0.30/0.10/0.35), 1 epoch | 0.480 [0.452, 0.514] | 0.367 | 0.436 | 4.41 | 84.2 / 87.1 | 3.0% | 70.2% | 44.1% |
| Main reward, uniform rollout mix, 1 epoch | 0.440 [0.413, 0.473] | 0.319 | 0.409 | 3.58 | 88.1 / 90.1 | 4.9% | 70.2% | 40.9% |
Table 11: RL configurations on the benchmark. Validation-reward curves: main run 0.845 0.879 (30) 0.867 (170) 0.859 (292), recognition-weighted 0.858 0.885 (30) 0.883 (146), uniform-mix 0.853 (146). Both surviving checkpoints of the main run (170 and 292) are shown. Dense oversampling drives the localization gain, the recognition cost appears in every configuration, and the final checkpoint is lower on F1, recall, recognition, and joint accuracy and higher on false abstention.
| Negatives | Abst. (given) [CI] | Abst. (cand.) | False abst. (given / cand.) | R | F1 | RIoU | Recog. | Joint |
|---|---|---|---|---|---|---|---|---|
| 0% | 0.2% [0.0, 0.6] | 0.0% | 0.0% / 0.0% | 0.311 | 0.435 | 0.617 | 72.6% | 43.3% |
| 6% | 93.3% [90.7, 95.5] | 89.5% | 5.2% / 0.5% | 0.310 | 0.433 | 0.585 | 73.0% | 44.3% |
| 12% | 95.0% [93.1, 96.9] | 89.1% | 4.6% / 0.9% | 0.320 | 0.440 | 0.578 | 74.7% | 44.8% |
Table 12: Existence-aware supervision (505 healthy images). Negatives buy near-complete abstention at a small false-abstention cost. Differences between 6% and 12% are within seed variance.
| Training data (grounding-only recipe) | [email protected] [CI] | R | P | RIoU | Fine F1 | Coarse F1 |
|---|---|---|---|---|---|---|
| PlantSeg expert boxes, 9,120 ex., 3 ep. | 0.376 [0.345, 0.407] | 0.275 | 0.595 | 0.688 | 0.282 | 0.691 |
| AgroGround pseudo-boxes, 9,120 ex., 3 ep. | 0.372 [0.340, 0.409] | 0.249 | 0.734 | 0.639 | 0.270 | 0.682 |
| AgroGround pseudo-boxes, 10K ex., 1 ep. | 0.400 [0.367, 0.439] | 0.275 | 0.734 | 0.647 | 0.306 | 0.698 |
| AgroGround pseudo-boxes, 100K ex., 1 ep. | 0.388 [0.358, 0.424] | 0.258 | 0.784 | 0.635 | 0.299 | 0.660 |
| AgroGround pseudo-boxes, 635,977 ex., 1 ep. | 0.426 [0.394, 0.459] | 0.296 | 0.760 | 0.622 | 0.366 | 0.619 |
Table 13: Expert labels vs. pseudo-labels at equal size, and scaling, on the benchmark (disease-given protocol). Paired differences: pseudo minus expert at equal size 0.004 [ 0.024, +0.017], full corpus minus equal-size subset +0.054 [+0.003, +0.098]. Subset runs are single-seed.
Figure 5: The same data as the table above: [email protected] with 95% CIs and lesion-level F1 against training data. Scale helps mainly at lesion level.
| Model | [email protected] [CI] | R | P | RIoU | Fine F1 | Coarse F1 |
|---|---|---|---|---|---|---|
| GroundingDINO (raw prompts) | 0.182 [0.172, 0.194] | 0.140 | 0.260 | 0.456 | 0.147 | 0.617 |
| GroundingDINO (taxonomy prompts) | 0.183 [0.173, 0.194] | 0.144 | 0.250 | 0.446 | 0.149 | 0.610 |
| GroundingDINO + annotation-pipeline filters (annotator config) | 0.161 [0.150, 0.173] | 0.104 | 0.351 | 0.371 | 0.133 | 0.655 |
| Qwen3-VL-2B (zero-shot) | 0.164 [0.152, 0.177] | 0.104 | 0.389 | 0.497 | 0.118 | 0.892 |
| Qwen3-VL-8B (zero-shot) | 0.215 [0.201, 0.231] | 0.140 | 0.463 | 0.547 | 0.173 | 0.899 |
| AgroGround-2B, grounding-only | 0.231 [0.219, 0.245] | 0.153 | 0.480 | 0.423 | 0.189 | 0.940 |
Table 14: PlantSeg test set, 2,165 images (1,750 lesion-level, 415 object-level), disease-given protocol with PlantSeg’s disease names. The released test split contains 2,295 images. 8 have empty masks and 3 have no connected component covering at least 0.1% of the image, leaving 2,284 usable, of which 119 are perceptual-hash near-duplicates of our training files and are removed. Masks are matched to images by file name and converted to component boxes. Taxonomy prompts were used for the 363 test images whose PlantSeg disease name maps to a taxonomy entry. The remaining 1,802 fall back to the raw name, so the taxonomy-based rows differ little from the raw-prompt row. Paired differences: RL minus its initialization +0.051 [+0.042, +0.059], RL minus the annotator configuration +0.121 [+0.109, +0.133], RL minus the PlantSeg-trained model +0.031 [+0.018, +0.042], RL minus the taxonomy-prompt teacher +0.099 [+0.089, +0.110], 8B minus 2B +0.006 [ 0.004, +0.015], and PlantSeg-trained minus equal-size pseudo +0.063 [+0.053, +0.075]. Object-level F1 is near-trivial here because many masks cover the whole plant. The grounding-only and +existence models tie at F1 0.231 with the same bootstrap interval. The tie is genuine, and they differ in recall and precision (0.153/0.480 vs. 0.161/0.412).
Figure 6: The tasks AgroGround trains and evaluates, one column per task, shown with the outputs of the main supervised model (AgroGround-2B, mixed + healthy) on benchmark cashew leaves. Boxes are the model’s predictions, and prompts are the evaluation prompts, abbreviated in the third panel.
Figure 7: Disease-given grounding on six benchmark images selected with a fixed seed (3) under stated rules: two dense lesion-level images where the RL model’s region IoU exceeds 0.5, two with 2–4 lesions, one object-level image, and one dense failure (at least 8 lesions, RL region IoU below 0.35). Columns: human ground truth (green), GroundingDINO with raw prompts (orange), the supervised AgroGround-2B (blue), and AgroGround-2B after GRPO (red). Numbers are box counts. The detector over-covers with leaf-spanning boxes (the flyspeck, early-blight, and anthracnose rows). The supervised model emits no box on the flyspeck apple, where RL recovers three, and on the 25-lesion anthracnose leaf both fine-tuned models under-enumerate (7 and 6 boxes).
Figure 8: Grounded disease recognition on benchmark images, selected with a fixed seed from cases where the +recognition model is correct, the grounding-only model is wrong, and the grounding-only answer is the first-listed candidate. The grounding-only model echoes that first candidate while localizing plausibly. The +recognition model names the target from the image. Two of the four examples name a species or a pest rather than a disease, reflecting the composition of the benchmark positives (Section 3.3 ).
| Source | Distribution | License (as stated by the source) | Our use |
|---|---|---|---|
| CDDM ( Liu et al., 2024b ) | GitHub (UnicomAI/UnicomBenchmark) | None stated a | training, evaluation |
| LeafNet / LeafBench ( Nguyen Quoc et al., 2026 ) | Hugging Face (gated) | Tagged CC BY 4.0 / MIT; non-commercial gate b | academic research use under the gate terms |
| MIRAGE ( Dongre et al., 2025 ) | GitHub, Hugging Face | CC BY-SA 4.0 | training, evaluation |
| AgroMind ( Li et al., 2025 ) | GitHub, Hugging Face | None stated c | training, evaluation |
| AgroCoT ( Wen et al., 2025 ) | Hugging Face | CC BY-SA 4.0 (dataset-card metadata) d | training, evaluation |
| AgroBench ( Shinoda et al., 2025 ) | Hugging Face (gated) | None stated e | training, evaluation |
Table 15: Source datasets, distribution, and licenses as stated by each source. No single license satisfies all nine datasets simultaneously: three state no license, three impose non-commercial terms, and three are share-alike. AgroGround therefore partitions its release by source (see below). No source image or source text is redistributed, and the model trained on PlantSeg is not released.
More in this topic
Agricultural Image Analysis
Agricultural image analysis uses computer vision and multimodal learning to monitor crops, livestock, produce quality, and field conditions.
See all 91 papers