From Scientific Crop Imagery to Agronomic Reasoning: A Multimodal LLM Benchmark for Plant Phenotyping

Yu Wu; Guangzeng Han; Ibra Niang Niang; Francia Ravelombola; Maiara Oliveira; Jason Davis; Dong Chen; Feng Lin

cs.CVApr 10, 2026

From Scientific Crop Imagery to Agronomic Reasoning: A Multimodal LLM Benchmark for Plant Phenotyping

Authors: Yu Wu, Guangzeng Han, Ibra Niang Niang, Francia Ravelombola, Maiara Oliveira, Jason Davis, Dong Chen, Feng Lin,

Organizations: Computer Science, University of Memphis, Memphis, 38111, TN, United States · Fisher Delta Research Extension and Education Center, University of Missouri, Portageville, 63873, MO, United States · University of Arkansas Division of Agriculture, Batesville, 72501, AR, United States · Agricultural & Biological Engineering, Mississippi State University, Starkville, 39762, MS, United States

Abstract

High-throughput, effective and comprehensive phenotyping is a critical prerequisite for crop improvement. While phenotypic analysis was traditionally performed manually, recent advances in multimodal foundation models, especially in vision-language models, have offered new opportunities for automated plant phenotyping. However, plant science remains particularly challenging for general-purpose VLMs because it requires domain-specific knowledge, fine-grained visual interpretation, and complex agronomic reasoning. We introduce PlantXpert, an evidence-grounded multimodal reasoning benchmark for plant phenotyping across four major crops: soybean, cotton, rice, and corn. Our benchmark provides a structured and reproducible framework for agronomic adaptation of VLMs, and enables controlled comparison between base models and their domain-adapted counterparts. We constructed a dataset comprising 545 digital images and 4,351 benchmark samples spanning key agronomic domains including disease, pest control, weed management and yield. The benchmark can assess diverse capabilities including visual expertise, quantitative reasoning, and multi-step agronomic reasoning. We trained and evaluated a total of 11 state-of-the-art VLMs. Fine-tuning on our dataset improves overall accuracy for all models, with many achieving around 70% accuracy, and improvements on 9 of 11 remaining statistically significant after Holm correction. We also identify diminishing returns from model scaling, uneven performance across crop subsets, and persistent weaknesses in quantitative reasoning and specialized image analysis. These limitations show that general multimodal competence alone is insufficient for reliable agronomic reasoning and demonstrate the value of domain-specific adaptation. We hope PlantXpert can serve as a foundation for advancing evidence-grounded multimodal reasoning in plant science.

Figures & tables

Study	Multimodal	Phenotypic Reasoning	Expert Annotation	Evidence-Grounded	Benchmark-Training Integration
AgriBench ( Zhou and Ryo, 2025 )	✓	$\times$	$\times$	$\times$	$\times$
AgEval ( Arshad et al., 2025 )	✓	✓	$\times$	$\times$	$\times$
AgroBench ( Shinoda et al., 2025 )	✓	✓	✓	$\times$	$\times$
AgriEval ( Yan et al., 2026 )	$\times$	✓	✓	$\times$	$\times$
PlantXpert (Ours)	✓	✓	✓	✓	✓

Table 1: Comparison of existing plant science benchmarks with our proposed framework across key capabilities.

Figure 1: An overview of our source retrieval benchmark construction pipeline.

Figure 2: Representative evidence-grounded multiple-choice data samples from PlantXpert. Each sample is annotated with a specific agricultural domain and primary reasoning challenge.

	Silver					Gold
Category	Soybean	Cotton	Rice	Corn	Total	Soybean	Cotton	Rice	Corn	Total
Domain
Plant Health & Pathology	1291	277	108	139	1815	91	63	22	32	208
Plant Biology & Physiology	149	73	90	82	394	6	18	27	20	71
Crop Science & Agronomy	122	47	114	112	395	7	11	29	30	77
Remote Sensing & Imaging	123	24	89	70	306	5	23	21	16	65

Table 2: Breakdown of domain and challenge types by crop in the Silver and Gold splits.

Model	Base					Tuned
Model	Soybean	Cotton	Rice	Corn	Total	Soybean	Cotton	Rice	Corn	Total	$\Delta$
Gemma3
Gemma3-4B	$56.56$	$55.47$	$43.31$	$52.76$	$51.98$	$68.85$	$63.28$	$56.69$	$66.14$	63.69 ∗∗	$11.71$
Gemma3-12B	$69.67$	$62.50$	$56.69$	$64.57$	$63.29$	$77.87$	$72.66$	$62.99$	$70.87$	71.03 ∗∗	$7.74$
Gemma3-27B	$71.31$	$67.19$	$58.27$	$70.87$	$66.87$	$80.33$	$75.78$	$66.14$	$71.65$	73.41 ∗∗	$6.55$
Qwen3-VL-Instruct

Table 3: Accuracy (%) on the subsets by plant (mean over 3 repeated inference runs) for the Base and Tuned models.

\Delta

represents the Base-to-Tuned improvement in overall accuracy, in percentage points.

Model	Base				Tuned
Model	Others	Quant.	Reasoning	Visual	Others	Quant.	Reasoning	Visual	Avg. $\Delta$
Gemma3
Gemma3-4B	$45.65$	$44.83$	$54.60$	$54.33$	$63.04$	$57.47$	$65.64$	$64.90$	$12.91$
Gemma3-12B	$71.74$	$40.23$	$68.10$	$67.31$	$76.09$	$62.07$	$73.01$	$72.12$	$8.98$
Gemma3-27B	$73.91$	$45.98$	$70.55$	$71.15$	$78.26$	$62.07$	$76.07$	$75.00$	$7.45$
Qwen3-VL-Instruct

Table 4: Accuracy (%) by challenge type (mean over 3 runs) for the Base and Tuned models. Avg.

\Delta

denotes the macro-average of the Tuned-minus-Base accuracy differences across the four challenge types, with each challenge weighted equally.

Model	Base						Tuned
Model	CropSci.	Entom.	Others	Bio.	Health	Remote	CropSci.	Entom.	Others	Bio.	Health	Remote	Avg. $\Delta$
Gemma3
Gemma3-4B	$51.95$	$48.78$	$42.86$	$56.34$	$53.37$	$50.77$	$66.23$	$51.22$	$61.90$	$69.01$	$66.83$	$53.85$	$10.83$
Gemma3-12B	$61.04$	$53.66$	$73.81$	$70.42$	$65.38$	$50.77$	$74.03$	$63.41$	$69.05$	$67.61$	$75.00$	$64.62$	$6.44$
Gemma3-27B	$64.94$	$63.41$	$76.19$	$66.20$	$70.19$	$55.38$	$72.73$	$65.85$	$76.19$	$74.65$	$79.81$	$55.38$	$4.72$
Qwen3-VL-Instruct

Table 5: Accuracy (%) by domain (mean over 3 repeated inference runs) for the Base and Tuned models. Avg.

\Delta

denotes the macro-average of the Tuned-minus-Base accuracy differences across the six domains, with each domain weighted equally.

Figure 3: Representative persistent error cases from the task-specific Tuned models.

Model	Gold	Gold-OOD	$\Delta$
Qwen3-VL-8B	76.39	71.43	-4.96
LLaVA-Next-8B	60.85	56.04	-4.81
Gemma-3-12B	71.03	70.33	-0.70

Table 6: Accuracy comparison between the original figure-disjoint Gold set ( Gold ) and the stricter source-paper-disjoint subset ( Gold-OOD ).

\Delta

denotes the change from Gold-Figure to Gold-Paper in percentage points.

Model	Base	Seed 1	Seed 2	Seed 3	Mean $\pm$ SD	$\Delta$
Qwen3-VL-2B	62.43	72.42	69.44	72.02	71.30 $\pm$ 1.62	+8.87
Qwen3-VL-8B	70.63	75.79	76.19	76.59	76.19 $\pm$ 0.40	+5.56
Qwen3-VL-30B	71.83	76.39	74.21	75.20	75.26 $\pm$ 1.09	+3.43

Table 7: Training-seed robustness on the Gold evaluation set. Each model is independently fine-tuned using three random seeds. Accuracy is reported in percent, and

\Delta

denotes the difference between the mean fine-tuned accuracy and the corresponding Base accuracy.

Model	Original (Tuned)	Image Mask	Image Shuffle	Option Shuffle
Qwen3-VL-8B	76.39	59.13 (-17.26)	56.75 (-19.64)	71.63 (-4.76)
LLaVA-Next-8B	60.85	56.94 (-3.91)	57.34 (-3.51)	58.53 (-2.32)
Gemma-3-12B	71.03	57.94 (-13.09)	53.77 (-17.26)	64.68 (-6.35)

Table 8: Visual-grounding and answer-bias ablations on the Gold set. Values are accuracy (%), with changes relative to the original evaluation shown in parentheses.

Model	MCQ Acc.	No Rationale	Mean Grounding	Grounded	Correct + Grounded
Qwen3-VL-8B	76.00%	78.00%	4.55	90.91%	18.00%
LLaVA-Next-8B	76.00%	44.00%	4.52	85.71%	44.00%
Gemma-3-12B	76.00%	12.00%	4.38	72.73%	64.00%

Table 9: Expert evaluation of rationale grounding on 50 randomly sampled Gold questions. No Rationale denotes responses assigned a grounding score of 1 by both experts. Mean Grounding and Grounded are calculated only among rationale-bearing responses, where Grounded denotes a mean expert score of at least 4. Correct + Grounded is calculated over the complete 50-question sample.

Training Data	Gold Accuracy (%)	$\Delta$ vs. Base (pp)
Base	70.63	–
GPT-5-mini	76.98	+6.35
Claude Sonnet 5.5	69.84	-0.79

Table 10: Synthetic-generator sensitivity of Qwen3-VL-8B. Both fine-tuning conditions use matched source figures, equal training-set size, and identical training settings.

Descriptor	Mean $\pm$ SD	OR	95% CI	$p$ -value
Question length (words)	$22.12\pm 6.07$	1.108	0.961–1.278	0.159
Figure resolution (MP)	$0.915\pm 0.833$	0.752	0.633–0.893	0.0012
Image entropy	$6.546\pm 1.217$	1.102	0.930–1.306	0.261

Table 11: Quantitative difficulty descriptors and their association with model correctness on the Gold set. Odds ratios (ORs) are obtained from a logistic GEE model using standardized predictors, model identity as a fixed effect, and scientific figure as the clustering unit.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Crop	Keywords	Articles	Figures
Soybean	Disease assessment, pest control, weed management, crop yield	1,685	305
Cotton	Disease assessment, pest control, weed management, crop yield	793	80
Rice	Disease assessment, pest control, weed management, crop yield	800	80
Corn	Disease assessment, pest control, weed management, crop yield	800	80

Appendix

Table 12: Summary of retrieved articles and figures.

Crop	Publication	Visual Evidence
Soybean	Development of an adaptive sprayer control system for UAV-based spot spraying in soybean.	UAV imagery
Soybean	A Data-Centric SPGT-ConvTransformer Framework for UAV-Based Soybean Disease Classification.	Disease imagery
Soybean	Texture- and threshold-based image segmentation for quantifying Asian soybean rust severity in digital images.	Leaf disease images
Soybean	Optimization and Experimental Evaluation of a Deep Learning-Based Target Spraying Device for Weed Control in Soybean Fields.	Field RGB images
Soybean	RFDAF-Net: a novel region-specific feature decoupling and adaptive fusion network for field soybean disease identification in precision agriculture.	Field disease images
Cotton	Detection and Precision Application Path Planning for Cotton Spider Mite Based on UAV Multispectral Remote Sensing.	UAV multispectral imagery

Appendix

Table 13: Candidate publications for future external validation.

Plant Phenotyping

Plant phenotyping uses imaging, sensing, and computational methods to measure plant traits, growth, health, and responses to the environment.

See all 68 papers

Mar 14, 2026

AgriChat: A Multimodal Large Language Model for Agriculture Image Understanding

The deployment of Multimodal Large Language Models (MLLMs) in agriculture is currently stalled by a critical trade-off: the existing literature lacks the large-scale agricultural datasets required for robust model development and evaluation, while current state-of-the-art models lack the verified domain expertise necessary to reason across diverse taxonomies. To address these challenges, we propose the Vision-to-Verified-Knowledge (V2VK) pipeline, a novel generative AI-driven annotation framework that integrates visual captioning with web-augmented scientific retrieval to autonomously generate the AgriMM benchmark, effectively eliminating biological hallucinations by grounding training data in verified phytopathological literature. The AgriMM benchmark contains over 3,000 agricultural classes and more than 607k VQAs spanning multiple tasks, including fine-grained plant species identification, plant disease symptom recognition, crop counting, and ripeness assessment. Leveraging this verifiable data, we present AgriChat, a specialized MLLM that presents broad knowledge across thousands of agricultural classes and provides detailed agricultural assessments with extensive explanations. Extensive evaluation across diverse tasks, datasets, and evaluation conditions reveals both the capabilities and limitations of current agricultural MLLMs, while demonstrating AgriChat's superior performance over other open-source models, including internal and external benchmarks. The results validate that preserving visual detail combined with web-verified knowledge constitutes a reliable pathway toward robust and trustworthy agricultural AI. The code and dataset are publicly available at https://github.com/boudiafA/AgriChat .

Abderrahmene Boudiaf, Irfan Hussain, Sajid Javed

Department of Computer Science, Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates

Agricultural Image Analysis Vision-Language Models Visual Question Answering LLM-Assisted Annotation Plant Disease Classification

Ask PDF

Mar 9, 2026

Using Vision Language Foundation Models to Generate Plant Simulation Configurations via In-Context Learning

This paper introduces a benchmark for evaluating whether vision-language models (VLMs) can generate plant simulation configurations from imagery using in-context learning. We study this benchmark for cowpea plot reconstruction for plant simulations, where the VLM needs to generate structured JSON configurations that include field and plant information. Open-source multimodal models from Gemma 4 and Qwen3.5 families are evaluated on a synthetic cowpea dataset with known JSON ground truth and on a real drone orthophoto dataset with field-collected JSON. Five in-context learning methods are used, from format restriction instruction to few-shot image examples with auxiliary grounding information. The results show that VLMs can generate valid JSON outputs, can generally estimate days after planting (DAP), plant counts, plant locations, sun angles, and leaf chlorophyll content, and can render approximate simulations of cowpea plots. Error metrics fluctuate across model families and often remain worse than dataset baselines, particularly when VLMs' pretrained knowledge dominates over weak visual evidence. These results position image-to-simulation JSON generation as a promising but currently challenging task, and establish a benchmark for studying how multimodal reasoning, prompt design, and the sim-to-real domain gap affect plant phenotyping tasks.

Heesup Yun, Isaac Kazuo Uyehara, Earl Ranario +4

University of California, Davis

VLM Evaluation Agricultural Image Analysis Plant Phenotyping Sim-to-Real Transfer In-Context Learning

Ask PDF

May 5, 2026

CropVLM: A Domain-Adapted Vision-Language Model for Open-Set Crop Analysis

High-throughput plant phenotyping, the quantitative measurement of observable plant traits, is critical for modern breeding but remains constrained by a "phenotyping bottleneck," where manual data collection is labor-intensive and prone to observer bias. Conventional closed-set computer vision systems fail to address this challenge, as they require extensive species-specific annotation and lack the flexibility to handle diverse breeding populations. To bridge this gap, we present CropVLM, a Vision-Language Model (VLM) adapted for the agricultural domain via Domain-Specific Semantic Alignment (DSSA). Trained on 52,987 manually selected image-caption pairs covering 37 species in natural field conditions, CropVLM effectively maps agronomic terminology to fine-grained visual features. We further introduce the Hybrid Open-Set Localization Network (HOS-Net), an architecture that integrates CropVLM to enable the detection of novel crops solely from natural language descriptions without retraining. By eliminating the reliance on species-specific training data, CropVLM provides a scalable solution for high-throughput phenotyping, accelerating genetic gain and facilitating large-scale biodiversity research essential for sustainable agriculture. The trained model weights and complete pipeline implementation are publicly available at: https://github.com/boudiafA/CropVLM. In comprehensive evaluations, CropVLM achieves 72.51% zero-shot classification accuracy, outperforming seven CLIP-style baselines. Our detection pipeline demonstrates superior zero-shot generalization to novel species, achieving 49.17 AP50 on our CVTCropDet benchmark and 50.73 AP50 on tropical fruit species, compared to 34.89 and 48.58 for the next-best method, respectively.

Abderrahmene Boudiaf, Sajd Javed

Department of Electrical Engineering and Computer Science, Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates

Open-Vocabulary Object Detection Agricultural Image Analysis Plant Phenotyping VLM Adaptation Zero-Shot Image Classification

Ask PDF

Abstract

Plant Phenotyping

Explore similar work

AgriChat: A Multimodal Large Language Model for Agriculture Image Understanding

Using Vision Language Foundation Models to Generate Plant Simulation Configurations via In-Context Learning

CropVLM: A Domain-Adapted Vision-Language Model for Open-Set Crop Analysis