From Scientific Crop Imagery to Agronomic Reasoning: A Multimodal LLM Benchmark for Plant Phenotyping

Yu Wu; Guangzeng Han; Ibra Niang Niang; Francia Ravelombola; Maiara Oliveira; Jason Davis; Dong Chen; Feng Lin

cs.CVApr 10, 2026

From Scientific Crop Imagery to Agronomic Reasoning: A Multimodal LLM Benchmark for Plant Phenotyping

Authors: Yu Wu, Guangzeng Han, Ibra Niang Niang, Francia Ravelombola, Maiara Oliveira, Jason Davis, Dong Chen, Feng Lin,

Organizations: Computer Science, University of Memphis, Memphis, 38111, TN, United States · Fisher Delta Research Extension and Education Center, University of Missouri, Portageville, 63873, MO, United States · University of Arkansas Division of Agriculture, Batesville, 72501, AR, United States · Agricultural & Biological Engineering, Mississippi State University, Starkville, 39762, MS, United States

Abstract

High-throughput, effective and comprehensive phenotyping is a critical prerequisite for crop improvement. While phenotypic analysis was traditionally performed manually, recent advances in multimodal foundation models, especially in vision-language models, have offered new opportunities for automated plant phenotyping. However, plant science remains particularly challenging for general-purpose VLMs because it requires domain-specific knowledge, fine-grained visual interpretation, and complex agronomic reasoning. We introduce PlantXpert, an evidence-grounded multimodal reasoning benchmark for plant phenotyping across four major crops: soybean, cotton, rice, and corn. Our benchmark provides a structured and reproducible framework for agronomic adaptation of VLMs, and enables controlled comparison between base models and their domain-adapted counterparts. We constructed a dataset comprising 545 digital images and 4,351 benchmark samples spanning key agronomic domains including disease, pest control, weed management and yield. The benchmark can assess diverse capabilities including visual expertise, quantitative reasoning, and multi-step agronomic reasoning. We trained and evaluated a total of 11 state-of-the-art VLMs. Fine-tuning on our dataset improves overall accuracy for all models, with many achieving around 70% accuracy, and improvements on 9 of 11 remaining statistically significant after Holm correction. We also identify diminishing returns from model scaling, uneven performance across crop subsets, and persistent weaknesses in quantitative reasoning and specialized image analysis. These limitations show that general multimodal competence alone is insufficient for reliable agronomic reasoning and demonstrate the value of domain-specific adaptation. We hope PlantXpert can serve as a foundation for advancing evidence-grounded multimodal reasoning in plant science.

Figures & tables

Study	Multimodal	Phenotypic Reasoning	Expert Annotation	Evidence-Grounded	Benchmark-Training Integration
AgriBench ( Zhou and Ryo, 2025 )	✓	$\times$	$\times$	$\times$	$\times$
AgEval ( Arshad et al., 2025 )	✓	✓	$\times$	$\times$	$\times$
AgroBench ( Shinoda et al., 2025 )	✓	✓	✓	$\times$	$\times$
AgriEval ( Yan et al., 2026 )	$\times$	✓	✓	$\times$	$\times$
PlantXpert (Ours)	✓	✓	✓	✓	✓

Table 1: Comparison of existing plant science benchmarks with our proposed framework across key capabilities.

Figure 1: An overview of our source retrieval benchmark construction pipeline.

Figure 2: Representative evidence-grounded multiple-choice data samples from PlantXpert. Each sample is annotated with a specific agricultural domain and primary reasoning challenge.

	Silver					Gold
Category	Soybean	Cotton	Rice	Corn	Total	Soybean	Cotton	Rice	Corn	Total
Domain
Plant Health & Pathology	1291	277	108	139	1815	91	63	22	32	208
Plant Biology & Physiology	149	73	90	82	394	6	18	27	20	71
Crop Science & Agronomy	122	47	114	112	395	7	11	29	30	77
Remote Sensing & Imaging	123	24	89	70	306	5	23	21	16	65

Table 2: Breakdown of domain and challenge types by crop in the Silver and Gold splits.

Model	Base					Tuned
Model	Soybean	Cotton	Rice	Corn	Total	Soybean	Cotton	Rice	Corn	Total	$\Delta$
Gemma3
Gemma3-4B	$56.56$	$55.47$	$43.31$	$52.76$	$51.98$	$68.85$	$63.28$	$56.69$	$66.14$	63.69 ∗∗	$11.71$
Gemma3-12B	$69.67$	$62.50$	$56.69$	$64.57$	$63.29$	$77.87$	$72.66$	$62.99$	$70.87$	71.03 ∗∗	$7.74$
Gemma3-27B	$71.31$	$67.19$	$58.27$	$70.87$	$66.87$	$80.33$	$75.78$	$66.14$	$71.65$	73.41 ∗∗	$6.55$
Qwen3-VL-Instruct

Table 3: Accuracy (%) on the subsets by plant (mean over 3 repeated inference runs) for the Base and Tuned models.

\Delta

represents the Base-to-Tuned improvement in overall accuracy, in percentage points.

Model	Base				Tuned
Model	Others	Quant.	Reasoning	Visual	Others	Quant.	Reasoning	Visual	Avg. $\Delta$
Gemma3
Gemma3-4B	$45.65$	$44.83$	$54.60$	$54.33$	$63.04$	$57.47$	$65.64$	$64.90$	$12.91$
Gemma3-12B	$71.74$	$40.23$	$68.10$	$67.31$	$76.09$	$62.07$	$73.01$	$72.12$	$8.98$
Gemma3-27B	$73.91$	$45.98$	$70.55$	$71.15$	$78.26$	$62.07$	$76.07$	$75.00$	$7.45$
Qwen3-VL-Instruct

Table 4: Accuracy (%) by challenge type (mean over 3 runs) for the Base and Tuned models. Avg.

\Delta

denotes the macro-average of the Tuned-minus-Base accuracy differences across the four challenge types, with each challenge weighted equally.

Model	Base						Tuned
Model	CropSci.	Entom.	Others	Bio.	Health	Remote	CropSci.	Entom.	Others	Bio.	Health	Remote	Avg. $\Delta$
Gemma3
Gemma3-4B	$51.95$	$48.78$	$42.86$	$56.34$	$53.37$	$50.77$	$66.23$	$51.22$	$61.90$	$69.01$	$66.83$	$53.85$	$10.83$
Gemma3-12B	$61.04$	$53.66$	$73.81$	$70.42$	$65.38$	$50.77$	$74.03$	$63.41$	$69.05$	$67.61$	$75.00$	$64.62$	$6.44$
Gemma3-27B	$64.94$	$63.41$	$76.19$	$66.20$	$70.19$	$55.38$	$72.73$	$65.85$	$76.19$	$74.65$	$79.81$	$55.38$	$4.72$
Qwen3-VL-Instruct

Table 5: Accuracy (%) by domain (mean over 3 repeated inference runs) for the Base and Tuned models. Avg.

\Delta

denotes the macro-average of the Tuned-minus-Base accuracy differences across the six domains, with each domain weighted equally.

Figure 3: Representative persistent error cases from the task-specific Tuned models.

Model	Gold	Gold-OOD	$\Delta$
Qwen3-VL-8B	76.39	71.43	-4.96
LLaVA-Next-8B	60.85	56.04	-4.81
Gemma-3-12B	71.03	70.33	-0.70

Table 6: Accuracy comparison between the original figure-disjoint Gold set ( Gold ) and the stricter source-paper-disjoint subset ( Gold-OOD ).

\Delta

denotes the change from Gold-Figure to Gold-Paper in percentage points.

Model	Base	Seed 1	Seed 2	Seed 3	Mean $\pm$ SD	$\Delta$
Qwen3-VL-2B	62.43	72.42	69.44	72.02	71.30 $\pm$ 1.62	+8.87
Qwen3-VL-8B	70.63	75.79	76.19	76.59	76.19 $\pm$ 0.40	+5.56
Qwen3-VL-30B	71.83	76.39	74.21	75.20	75.26 $\pm$ 1.09	+3.43

Table 7: Training-seed robustness on the Gold evaluation set. Each model is independently fine-tuned using three random seeds. Accuracy is reported in percent, and

\Delta

denotes the difference between the mean fine-tuned accuracy and the corresponding Base accuracy.

Model	Original (Tuned)	Image Mask	Image Shuffle	Option Shuffle
Qwen3-VL-8B	76.39	59.13 (-17.26)	56.75 (-19.64)	71.63 (-4.76)
LLaVA-Next-8B	60.85	56.94 (-3.91)	57.34 (-3.51)	58.53 (-2.32)
Gemma-3-12B	71.03	57.94 (-13.09)	53.77 (-17.26)	64.68 (-6.35)

Table 8: Visual-grounding and answer-bias ablations on the Gold set. Values are accuracy (%), with changes relative to the original evaluation shown in parentheses.

Model	MCQ Acc.	No Rationale	Mean Grounding	Grounded	Correct + Grounded
Qwen3-VL-8B	76.00%	78.00%	4.55	90.91%	18.00%
LLaVA-Next-8B	76.00%	44.00%	4.52	85.71%	44.00%
Gemma-3-12B	76.00%	12.00%	4.38	72.73%	64.00%

Table 9: Expert evaluation of rationale grounding on 50 randomly sampled Gold questions. No Rationale denotes responses assigned a grounding score of 1 by both experts. Mean Grounding and Grounded are calculated only among rationale-bearing responses, where Grounded denotes a mean expert score of at least 4. Correct + Grounded is calculated over the complete 50-question sample.

Training Data	Gold Accuracy (%)	$\Delta$ vs. Base (pp)
Base	70.63	–
GPT-5-mini	76.98	+6.35
Claude Sonnet 5.5	69.84	-0.79

Table 10: Synthetic-generator sensitivity of Qwen3-VL-8B. Both fine-tuning conditions use matched source figures, equal training-set size, and identical training settings.

Descriptor	Mean $\pm$ SD	OR	95% CI	$p$ -value
Question length (words)	$22.12\pm 6.07$	1.108	0.961–1.278	0.159
Figure resolution (MP)	$0.915\pm 0.833$	0.752	0.633–0.893	0.0012
Image entropy	$6.546\pm 1.217$	1.102	0.930–1.306	0.261

Table 11: Quantitative difficulty descriptors and their association with model correctness on the Gold set. Odds ratios (ORs) are obtained from a logistic GEE model using standardized predictors, model identity as a fixed effect, and scientific figure as the clustering unit.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Crop	Keywords	Articles	Figures
Soybean	Disease assessment, pest control, weed management, crop yield	1,685	305
Cotton	Disease assessment, pest control, weed management, crop yield	793	80
Rice	Disease assessment, pest control, weed management, crop yield	800	80
Corn	Disease assessment, pest control, weed management, crop yield	800	80

Appendix

Table 12: Summary of retrieved articles and figures.

Crop	Publication	Visual Evidence
Soybean	Development of an adaptive sprayer control system for UAV-based spot spraying in soybean.	UAV imagery
Soybean	A Data-Centric SPGT-ConvTransformer Framework for UAV-Based Soybean Disease Classification.	Disease imagery
Soybean	Texture- and threshold-based image segmentation for quantifying Asian soybean rust severity in digital images.	Leaf disease images
Soybean	Optimization and Experimental Evaluation of a Deep Learning-Based Target Spraying Device for Weed Control in Soybean Fields.	Field RGB images
Soybean	RFDAF-Net: a novel region-specific feature decoupling and adaptive fusion network for field soybean disease identification in precision agriculture.	Field disease images
Cotton	Detection and Precision Application Path Planning for Cotton Spider Mite Based on UAV Multispectral Remote Sensing.	UAV multispectral imagery

Appendix

Table 13: Candidate publications for future external validation.

Plant Phenotyping

Plant phenotyping uses imaging, sensing, and computational methods to measure plant traits, growth, health, and responses to the environment.

See all 68 papers

AgriChat: A Multimodal Large Language Model for Agriculture Image Understanding
Mar 14, 2026Abderrahmene Boudiaf, Irfan Hussain, Sajid JavedAgricultural Image Analysis Vision-Language Models
Department of Computer Science, Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates
Using Vision Language Foundation Models to Generate Plant Simulation Configurations via In-Context Learning
Mar 9, 2026Heesup Yun, Isaac Kazuo Uyehara, Earl Ranario +4VLM Evaluation Agricultural Image Analysis
University of California, Davis
CropVLM: A Domain-Adapted Vision-Language Model for Open-Set Crop Analysis
May 5, 2026Abderrahmene Boudiaf, Sajd JavedOpen-Vocabulary Object Detection Agricultural Image Analysis
Department of Electrical Engineering and Computer Science, Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates

Mar 14, 2026J/K move · Enter open · S save

AgriChat: A Multimodal Large Language Model for Agriculture Image Understanding

Abderrahmene Boudiaf, Irfan Hussain, Sajid Javed

Department of Computer Science, Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates

Agricultural Image Analysis Vision-Language Models Visual Question Answering LLM-Assisted Annotation Plant Disease Classification

Open paper AskPDF arXiv

The deployment of Multimodal Large Language Models (MLLMs) in agriculture is currently stalled by a critical trade-off: the existing literature lacks the large-scale agricultural datasets required for robust model development and evaluation, while current state-of-the-art models lack the verified domain expertise necessary to reason across diverse taxonomies. To address these challenges, we propose the Vision-to-Verified-Knowledge (V2VK) pipeline, a novel generative AI-driven annotation framework that integrates visual captioning with web-augmented scientific retrieval to autonomously generate the AgriMM benchmark, effectively eliminating biological hallucinations by grounding training data in verified phytopathological literature. The AgriMM benchmark contains over 3,000 agricultural classes and more than 607k VQAs spanning multiple tasks, including fine-grained plant species identification, plant disease symptom recognition, crop counting, and ripeness assessment. Leveraging this verifiable data, we present AgriChat, a specialized MLLM that presents broad knowledge across thousands of agricultural classes and provides detailed agricultural assessments with extensive explanations. Extensive evaluation across diverse tasks, datasets, and evaluation conditions reveals both the capabilities and limitations of current agricultural MLLMs, while demonstrating AgriChat's superior performance over other open-source models, including internal and external benchmarks. The results validate that preserving visual detail combined with web-verified knowledge constitutes a reliable pathway toward robust and trustworthy agricultural AI. The code and dataset are publicly available at https://github.com/boudiafA/AgriChat .

Abstract

Plant Phenotyping

Explore similar work

AgriChat: A Multimodal Large Language Model for Agriculture Image Understanding

Using Vision Language Foundation Models to Generate Plant Simulation Configurations via In-Context Learning

CropVLM: A Domain-Adapted Vision-Language Model for Open-Set Crop Analysis