AgroOmni: A Large-Scale Multi-view Agricultural Dataset for Cross-Scale Multimodal Reasoning

Jiarui Zhang; Junqi Hu; Zurong Mai; Yang Liu; Yuhang Chen; Shuohong Lou; Henglian Huang; Hong Cheng

cs.CVMar 15, 2026

AgroOmni: A Large-Scale Multi-view Agricultural Dataset for Cross-Scale Multimodal Reasoning

Authors: Jiarui Zhang, Junqi Hu, Zurong Mai, Yang Liu, Yuhang Chen, Shuohong Lou, Henglian Huang, Hong Cheng,

Organizations: Sun Yat-sen University · The Chinese University of Hong Kong · HuanTian Wisdom Technology Co., Ltd. · China Agricultural University ·

Abstract

Modern agricultural data is sourced from diverse platforms and spans multiple spatial scales, ranging from ground-level close-up photography to Unmanned Aerial Vehicle (UAV) aerial observation and satellite remote sensing imagery. Accordingly, agricultural multimodal reasoning demands robust cross-scale spatial understanding. However, due to the lack of multi-view agricultural benchmark datasets, existing multimodal large language models (MLLMs) exhibit severe ground-level bias, which leads to scale confusion then semantic collapse in agricultural perception tasks, such as misinterpreting farmland imagery as walls or floors. To address this, we introduce AgroOmni, a large-scale multi-view training corpus with 288K Visual Question Answering pairs covering 56 specialized task categories across 14 task types, designed to capture diverse scales in modern precision agriculture. Built on this dataset, we propose AgroNVILA, which achieves a new state-of-the-art of 62.32% on the AgroMind benchmark (+15.03% over GPT-5.2), effectively mitigating the multi-view cross-scale gap for holistic agricultural understanding. Diagnostic evaluations on AgMMU further reveal an inherent heterogeneity between macro-priors and micro-diagnostics through constrained zero-shot performance. Meanwhile, even minimal fine-tuning leads to a dramatic performance gain of AgroNVILA on AgMMU, strongly demonstrating its generalization capability empowered by AgroOmni. Full training scripts are publicly available at https://anonymous.4open.science/r/AgroOmni-6510.

Figures & tables

Figure 1: Overview Comparison of our framework with existing paradigms, and its comprehensive performance radar on the AgroMind benchmark (numbers represent the SOTA scores for each task)

Dataset	Images	SFT QA	Complex Reasoning	Spatial Grounding	Multi-View
AgroInstruct [ 17 ]	108K	70K	$\times$	$\times$	$\times$
Agri-LLaVA [ 28 ]	397K	6K*	$\times$	$\times$	$\times$
Agri-3M-VL [ 11 ]	1M	3M	✓	$\times$	$\times$
AGBASE [ 18 ]	57K	57K	✓	$\times$	$\times$
AgroOmni (Ours)	107K	288K	✓	✓	✓
*Agri-LLaVA claims 400K data, but only 6K high-quality dialogues are used for SFT.

Table 1: Comparison of existing agricultural vision-language instruction datasets. SFT QA refers to the number of complex instruction-tuning samples used for the fine-tuning stage. Grounding strictly indicates support for region-level geometric localization (e.g., bounding boxes). Multi-View denotes the inclusion of macroscopic perspectives (UAV/Satellite) alongside standard Ground imagery.

Figure 2: The curation pipeline of AgroOmni training set contains three stages, i.e., data collection and pre-processing, question generation, and quality control.

Figure 3: Comprehensive statistics of the AgroOmni. (a) Classification of 14 fine-grained agricultural tasks across four cognitive dimensions (b) Multi-view data scale and task distribution across UAV, Satellite, and Ground perspectives (c-d) Wordclouds for QA pairs, highlighting the high density of domain-specific agronomic terminology

Model	Spatial Perception			Object Understanding			Scene Understanding			Scene Reasoning				Overall
Model	SL	SR	BD	OC	PDD	GSR	SC	CO	AS	VPR	AR	CTR	PL	Overall
Human	19.15	15.00	40.83	57.60	46.17	47.22	35.49	28.88	39.46	12.33	25.62	34.07	13.50	33.15
Random	18.03	17.97	16.48	35.65	25.65	10.00	24.39	4.99	21.20	9.19	19.87	21.70	26.96	19.24
GPT-4o [ 26 ]	41.38	35.16	33.52	78.55	66.09	55.83	43.90	23.75	46.20	16.10	27.15	69.81	30.90	43.14
GPT-5.2	39.19	29.94	42.09	74.14	65.61	60.50	41.22	36.85	53.27	28.09	38.33	57.41	38.83	47.29
Gemini-1.5-Pro [ 27 ]	27.59	23.44	24.73	77.39	52.61	45.83	39.02	32.30	47.78	19.92	34.44	66.04	30.04	41.06

Table 2: Performance comparison on AgroMind benchmark. The overall score represents the average accuracy across all evaluation tasks. Bold and underlined values denote the best and second-best results respectively. Task abbreviations: AR (Anomaly Reasoning), AS (Area Statistics), BD (Boundary Detection), CO (Counting), CTR (Climate Type Reasoning), GSR (Growth Stage Recognition), OC (Object Classification), PDD (Pest/Disease Diagnosis), PL (Planning), SC (Scene Comparison), SL (Spatial Localization), SR (Spatial Relationship), and VPR (Visual Prompt Reasoning).

Method	Spatial Perception			Object Understanding			Scene Understanding			Scene Reasoning				Overall
Method	SL	SR	BD	OC	PDD	GSR	SC	CO	AS	AR	CTR	VPR	PL	Overall
SFT Baseline	51.23	30.64	56.76	84.79	90.09	83.33	38.81	47.01	69.40	78.30	68.97	52.45	44.90	61.61
+PPO [ 45 ]	50.74	30.87	59.73	84.81	90.61	83.67	42.06	45.95	70.03	77.71	69.52	55.12	42.63	61.96
+GRPO [ 46 ]	50.74	30.72	59.89	84.99	90.30	84.50	42.91	46.82	69.90	77.98	70.46	55.71	43.31	62.32
+DRPO [ 47 ]	50.88	31.03	59.40	84.87	90.35	84.17	43.27	46.82	69.87	78.11	70.36	55.37	43.19	62.26

Table 3: Effect of standard RL strategies on the AgroMind benchmark. We report accuracy (%) for each task and the overall average. The SFT Baseline serves as the starting point for alignment. Bold and underlined values denote the best and second-best results within each column respectively.

Proprietary Models
Model	Size	Disease	Insect/Pest	Species	Management	Symptom	Average
GPT-o4-mini [ 56 ]	-	77.9	85.4	90.3	93.8	84.3	86.5
Gemini 1.5 Pro [ 27 ]	-	76.2	81.1	82.8	88.1	76.9	82.4
Claude 3 Haiku	-	62.1	71.2	52.8	81.5	52.0	63.8
SOTA Open-sourced Models
LLaVA-1.5 [ 21 ]	7B	61.9	64.6	67.6	77.3	71.5	69.0

Table 4: Detailed performance comparison on the AgMMU [ 18 ] benchmark (MCQs). All scores are reported as accuracy (%). Bold and underlined values indicate the best and the second-best performance among all open-source and domain-specific models. By leveraging just 10% to 30% of AGBASE [ 18 ] , AgroNVILA(SFT and GRPO) demonstrates robust adaptation.

Panel of Figure 4 — Figure 4: Comprehensive Diagnostic Analysis on AgroMind. (a) Evaluation across different spatial perspectives demonstrates the effective mitigation of Ground-level bias. (b) The scaling trajectory exposes a strict representational plateau, where macro-topological tasks saturate rapidly despite increased data volume.

Figure 5: Scale Confusion and Semantic Collapse Case. Compared to current MLLMs, our AgroOmni dataset corrects the scale and perspective biases of generic MLLMs, enabling accurate agronomic interpretation of macro-scale agricultural scenes.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Reward Design Details
Single-choice / VQA	Reward = 1.0 for exact match, 0 otherwise.
Counting	Reward = $\max(0,1-\|\text{Ans}-\text{GT}\|/\max(\|\text{GT}\|,1))$ .
Multi-choice	Reward = geometric mean of set-IoU and recall.
Visual grounding / BBox	Reward = IoU(pred, GT).
Open-ended	Reward = ROUGE-L with length penalty.
Short-answer (e.g., growth-stage)	Reward = ordinal-distance and triplet-level matching.

Appendix

Table 5: Task-specific reward functions.

Config	$r_{\text{task}}$	$r_{\text{spatial}}$	$r_{\text{fmt}}$	Acc. (%)
Full	$\checkmark$	$\checkmark$	$\checkmark$	62.4
w/o $r_{\text{task}}$	$\times$	$\checkmark$	$\checkmark$	61.7
w/o $r_{\text{spatial}}$	$\checkmark$	$\times$	$\checkmark$	62.2
w/o $r_{\text{fmt}}$	$\checkmark$	$\checkmark$	$\times$	62.1

Appendix

Table 6: Ablation of reward components during policy optimization. Results are reported as accuracy (%) under the resource-efficient

G=4

configuration. Bold values indicate the best performance.

r_{\text{task}}

r_{\text{spatial}}

, and

r_{\text{fmt}}

denote task accuracy, spatial consistency, and format rewards, respectively.

Hyperparameter	Stage 1: SFT	Stage 2: GRPO & DRPO
Trainable Modules	LLM (LoRA), Projector	LLM (full), Projector
Frozen	Vision Encoder	Vision Encoder
LoRA Config ( $r,\alpha,\text{dropout}$ )	$128,256,0.05$	–
Global Batch Size	32	32
Per-Device Batch Size (prompts)	4	1
Group Size $G$ (rollouts per prompt)	–	8

Appendix

Table 7: Global Hyperparameters for the two-stage training pipeline.

Dataset	Year	Content	Resolution	Link
ACFR Orchard Fruit Dataset [ 41 ]	2016	pixel-level annotations for almond, apple, mango	308×202, 500×500	https://data.acfr.usyd.edu.au/ag/treecrops/2016-multifruit/
2018 AI Challenge Dataset	2018	Extensive crop leaf image collection featuring 3-level disease severity labels and 26 disease types across 9 crops.	Variable	https://aistudio.baidu.com/datasetdetail/76075
IP102 Dataset [ 30 ]	2019	A large-scale agricultural pest dataset containing VOC-format bounding boxes for 102 insect species.	Variable	https://github.com/xpwu95/IP102
Agriculture Vision Challenge [ 38 ]	2020	Aerial multi-spectral (RGB + NIR) imagery equipped with pixel-level binary masks for detecting six common field anomalies.	512×512	https://github.com/SHI-Labs/Agriculture-Vision
OilPalmUAV Dataset [ 37 ]	2021	High-resolution UAV drone imagery providing instance-level bounding box annotations for categorizing five distinct growth states of oil palms.	1024×1024	https://github.com/rs-dl/MOPAD
CropHarvest Dataset [ 43 ]	2021	Global remote sensing time-series combining satellite imagery and climate data with agricultural labels for multimodal crop classification.	896×832, 960×896	https://zenodo.org/records/5828893

Appendix

Table 8: Detailed description of datasets used in the AgroOmni

Panel of Figure 6 — Figure 6: Qualitative Result on Spatial Perception. When enclosing cultivated land, the baseline model (GPT-5.2) completely inverts the spatial mapping, incorrectly anchoring the target to the lower-left corner. AgroNVILA accurately grounds the semantic concept to the correct normalized spatial coordinates.

Panel of Figure 8 — Figure 8: Qualitative Result on Scene Understanding. For macro-area coverage estimation, the baseline model suffers from severe visual illusions, drastically overestimating the tree coverage ratio to $\sim$ 86%. AgroNVILA bypasses verbose hallucinations and accurately outputs the calibrated target area of 12,200-12,300 square meters.

Multimodal Reasoning

Multimodal reasoning studies how AI systems combine text with images, audio, video, and other signals to solve evidence-dependent tasks.

See all 496 papers

AgriScope: Pixel-Grounded Multimodal Understanding for Agricultural Images
Sep 17, 2026Abderrahmene Boudiaf, Mohamad Alanssari, Irfan Hussain +1Agricultural Image Analysis Multimodal Grounding
Department of Computer Science, Khalifa University of Science and Technology, P.O. Box 127788, Abu Dhabi, United Arab Emirates · Department of Mechanical and Nuclear Engineering, Khalifa University of Science and Technology, P.O. Box 127788, Abu Dhabi, United Arab Emirates
AgriChat: A Multimodal Large Language Model for Agriculture Image Understanding
Mar 14, 2026Abderrahmene Boudiaf, Irfan Hussain, Sajid JavedAgricultural Image Analysis Vision-Language Models
Department of Computer Science, Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates
AgroTools: A Benchmark for Tool-Augmented Multimodal Agents in Agriculture
May 21, 2026Zi Ye, Yibin Wen, Xiaoya Fan +10AI Agent Benchmarks Tool-Using Vision-Language Agents
1Sun Yat-Sen University · 2Southwest University · 3Northeastern University +4

Sep 17, 2026J/K move · Enter open · S save

AgriScope: Pixel-Grounded Multimodal Understanding for Agricultural Images

Abderrahmene Boudiaf, Mohamad Alanssari, Irfan Hussain, Sajid Javed

Department of Computer Science, Khalifa University of Science and Technology, P.O. Box 127788, Abu Dhabi, United Arab Emirates · Department of Mechanical and Nuclear Engineering, Khalifa University of Science and Technology, P.O. Box 127788, Abu Dhabi, United Arab Emirates

Agricultural Image Analysis Multimodal Grounding Referring Expression Segmentation Multimodal Instruction Tuning

Open paper AskPDF arXiv

Agricultural image understanding requires fine-grained recognition of plant diseases, pests, crop structures, and botanical species under complex real-world conditions. Despite recent advances in Multimodal Large Language Models (MLLMs), existing models remain limited to text-only outputs and lack pixel-level visual grounding capabilities. In this work, we introduce AgriScope, a unified pixel-grounded multimodal framework for agricultural image understanding. AgriScope jointly supports image-level, region-level, and pixel-level understanding within a unified framework, enabling tasks such as grounded caption generation, referring expression segmentation, and multi-turn multimodal interaction for agricultural imagery. AgriScope integrates biologically specialized semantic representations with dense spatial grounding through biological-semantic encoding, dense spatial representations, and pixel decoding. To support large-scale grounded learning, we introduce AgriGround, a large-scale pixel-grounded agricultural multimodal instruction-tuning dataset containing over 500K images and 11M instruction-following samples spanning plant disease analysis, crop and weed identification, insect pest recognition, and fine-grained botanical understanding. AgriGround is constructed through a multi-stage automatic annotation pipeline that integrates multimodal caption generation, phrase-level grounding, segmentation mask generation, and task-oriented instruction synthesis to produce densely grounded supervision. Extensive experiments across multiple agricultural vision-language tasks demonstrate the effectiveness of AgriScope in pixel-grounded multimodal understanding, establishing a strong benchmark for agricultural vision-language learning and visual grounding. The dataset and code will be made publicly available at (https://github.com/boudiafA/AgriScope)

Abstract

Multimodal Reasoning

Explore similar work

AgriScope: Pixel-Grounded Multimodal Understanding for Agricultural Images

AgriChat: A Multimodal Large Language Model for Agriculture Image Understanding

AgroTools: A Benchmark for Tool-Augmented Multimodal Agents in Agriculture