cs.CVAug 23, 2026
SaveFusing Perceptual Vision Experts with Multimodal LLMs for Explainable Plant Disease Diagnosis: From Benchmark Imagery to Real-World Robotic Field Validation
Organizations: Cornell University, Department of Biological and Environmental Engineering, Ithaca, 14850, NY, USA · University of the Peloponnese, Department of Informatics and Telecommunications, Tripoli, 22131, Greece · Agricultural University of Athens, Department of Agribusiness and Supply Chain Management, Athens, 11855, Greece
Abstract
Accurate field plant disease diagnosis requires reliable fusion of uncertain and conflicting perceptual evidence. We present the Hybrid Hierarchical Multi-Agent Framework (HMAF), combining decision-level fusion of EfficientNet-B3 and ConvNeXt-Tiny with semantic arbitration by open-weight multimodal large language models (MLLMs), Gemma 4 E4B and Qwen3.5 4B, using structured JSON evidence to generate explainable diagnoses, risk levels, treatment urgency, and financial exposure. (HMAF) is evaluated on 14,364 images (1,370 test images) across PlantDoc (2,922 images, 27 classes) and two non-public, continuously captured Cornell robot-acquired field datasets: Stage 2 (20 GB; 4,215 images) and Stage 4 (40 GB; 7,227 images), covering Early Blight, Late Blight, and Septoria Leaf Spot under uncontrolled field conditions. On PlantDoc, Gemma improves accuracy from 63.9% to 68.5%, achieving +7.6 points on the 41.7% CNN-conflict subset. Cornell accuracies reach 99.3% and 98.9%, with only 1.7-4.1% disagreement, demonstrating conflict-dependent MLLM utility. The critical-risk error of gemma is 0.14-0.5 points, whereas Qwen overflags by 3.5-14.4 points. These results establish MLLM arbitration as a promising, yet calibration-dependent, approach for explainable agricultural AI and robotic field decision support. Github Link: https://github.com/Applied-AI-Research-Lab/Explainable-AI-Plant-Disease-Detection
Figures & tables
Figure 1: Representative sample images from the three datasets used in this study. Top row: PlantDoc, internet-scraped “in-the-wild” photographs (27 classes). Middle and bottom rows: Cornell Stage 2 and Stage 4, continuously-captured, autonomous field-robot imagery (3 classes). The visual contrast between crowd-sourced internet photography and continuous field-video frames illustrates the acquisition-modality gap discussed throughout this section.
| Dataset | Images | Classes | Test | Acquisition | Setting |
| PlantDoc [ 6 , 26 ] | 2,922 | 27 | 252 | Internet-scraped photographs | Public, in-the-wild benchmark |
| Cornell Stage 2 (this work) | 4,215 | 3 | 403 | Autonomous field-robot video frames, 20 GB | Closed, real-world field data |
| Cornell Stage 4 (this work) | 7,227 | 3 | 715 | Autonomous field-robot video frames, 40 GB | Closed, real-world field data |
| Total | 14,364 | — | 1,370 | — | — |
Table 1: Overview of the three datasets used in this study. “Acquisition” describes how the images were captured, whereas “Setting” indicates whether the source is a public Internet-curated benchmark or a closed, real-world field dataset shared under a research collaboration.
| Class | Train | Test |
| Apple Scab Leaf | 83 | 10 |
| Apple Leaf | 79 | 9 |
| Apple Rust Leaf | 96 | 10 |
| Bell Pepper Leaf | 34 | 8 |
| Bell Pepper Leaf Spot | 74 | 9 |
| Blueberry Leaf | 106 | 11 |
Table 2: PlantDoc class distribution (27 classes used in experiments).
Figure 2: Demonstration of the customized Amiga robotic phenotyping platform used to collect the Cornell tomato-disease datasets under outdoor field conditions: (a) the robotic data-collection prototype operating during the earlier-season campaign, and (b) the same prototype operating within the denser tomato canopy during the later-season campaign. The platform incorporated five PhenoStereo imaging units, comprising ten RGB cameras and synchronized active strobe-lighting assemblies, to acquire multi-view canopy images while navigating along the crop rows.
| Dataset | Images | Classes | Setting | CNN Acc. |
| PlantVillage [ 2 ] | 54,306 | 38 | Lab, public | 99% |
| PlantDoc [ 6 ] | 2,598 | 27 | Field, public | 60–75% |
| PlantDoc v7 (this work) | 2,922 | 27 | Field, public | 63.9% |
| Cornell Stage 2 (this work) | 4,215 | 3 | Field, closed | 99.8% |
| Cornell Stage 4 (this work) | 7,227 | 3 | Field, closed | 98.3% |
Table 3: Comparison of plant disease image datasets relevant to this study.
Figure 3: H 2 MAF pipeline, applied uniformly across the three datasets. Two CNN experts (Phase 1) feed a structured JSON artefact (Phase 2) which, together with the original image, is processed by two MLLM reasoning agents (Phase 3) to produce structured XAI reports.
| Parameter | EfficientNet-B3 | ConvNeXt-Tiny |
| Input resolution | ||
| Pre-trained weights | ImageNet-1k | ImageNet-1k |
| Fine-tuning strategy | Partial fine-tuning (top 30%) | Full-network fine-tuning |
| Optimizer | AdamW | AdamW |
| Learning rate | ||
| Learning-rate schedule | Cosine annealing | Cosine annealing with 5-epoch warm-up |
Table 4: Fine-tuning hyperparameters for the two CNN perceptual experts, applied uniformly to PlantDoc and both Cornell datasets.
| Class | P | R | F1 | Gem | Qwn | |
| Apple Scab Leaf | 0.78 | 0.70 | 0.74 | 10 | 90% | 70% |
| Apple Leaf | 0.57 | 0.89 | 0.70 | 9 | 78% | 67% |
| Apple Rust Leaf | 1.00 | 0.60 | 0.75 | 10 | 60% | 70% |
| Bell Pepper Leaf | 0.80 | 0.50 | 0.62 | 8 | 62% | 62% |
| Bell Pepper Leaf Spot | 0.45 | 0.56 | 0.50 | 9 | 56% | 67% |
| Blueberry Leaf | 0.88 | 0.64 | 0.74 | 11 | 45% | 64% |
Table 5: Full per-class results on PlantDoc. CNX = ConvNeXt-Tiny (Precision / Recall / F1); Gem = Gemma 4 E4B (top-1 accuracy); Qwn = Qwen3.5 4B (top-1 accuracy). = test support. Best accuracy per class in bold.
| Model | Acc. | Macro F1 | W. F1 | Cov. |
| EfficientNet-B3 | 59.1% | 0.5857 | 0.5872 | 100% |
| ConvNeXt-Tiny | 63.9% | 0.6393 | 0.6479 | 100% |
| Gemma 4 E4B (MLLM) | 68.5% | 0.6843 | 0.6906 | 94.4% |
| Qwen3.5 4B (MLLM) | 67.2% | 0.6761 | 0.6788 | 96.8% |
Table 6: Classification performance on the PlantDoc test set (252 images, 27 classes). Coverage: fraction of test images for which the model produced a parseable prediction. Best results in bold .
| Study | Model(s) | Acc. | Full split? |
| Singh et al. [ 6 ] | VGG-16, ResNet-50 | 30–50% | Partial |
| Taneja et al. [ 31 ] | Custom CNN, VGG-16 | 60–70% | Partial |
| Shiyan et al. [ 32 ] | MobileNetV3, Eff-B0, DenseNet-121 | 60–75% | Yes |
| Fatma et al. [ 33 ] | YOLOv8, EfficientNet-B0 | 65–75% | Yes |
| Wojciuk et al. [ 34 ] | Fine-tuned CNNs (HPO) | 70–80% | No (balanced subset) |
| Hasan et al. [ 35 ] | YOLOv11n + ECA-NFNet | 75–80% | No (two-stage) |
Table 7: PlantDoc classification accuracy reported in recent literature. “Full split” indicates use of the official 27-class train/test split without class-balanced subsetting or external data.
| Model | Agree (147) | Disagree (105) |
| EfficientNet-B3 | 78.2% | 32.4% |
| ConvNeXt-Tiny | 78.2% | 43.8% |
| Gemma 4 E4B (MLLM) | 76.9% | 47.6% |
| Qwen3.5 4B (MLLM) | 75.9% | 51.4% |
Table 8: PlantDoc: accuracy conditioned on vision expert agreement. All four models are evaluated separately on the 147 consensus images and the 105 conflict images.
| Override type | Gemma 4 E4B | Qwen3.5 4B | ||
| Cases | Acc. | Cases | Acc. | |
| Agrees with both CNNs | 138 | 81.2% | 138 | 79.7% |
| Agrees w/ ConvNeXt only | 80 | 50.0% | 83 | 50.6% |
| Agrees w/ EfficientNet | 12 | 66.7% | 12 | 75.0% |
| Overrides both CNNs | 22 | 13.6% | 17 | 17.6% |
Table 9: PlantDoc: MLLM override behaviour and accuracy per override type.
| Outcome | Gemma | Qwen |
| MLLM improves over best CNN | 3 cases (1.2%) | 3 cases (1.2%) |
| MLLM hurts vs. best CNN | 35 cases (13.9%) | 32 cases (12.8%) |
| Neutral (same outcome) | 214 cases (84.9%) | 215 cases (86.0%) |
| Rescue rate (both CNNs wrong) | 5.3% (3/57) | 5.3% (3/57) |
| Damage rate (both CNNs right) | 2.6% (3/115) | 2.6% (3/115) |
Table 10: PlantDoc: per-image outcomes when comparing each MLLM against the best available CNN prediction.
| Confidence bucket | Images | ConvNeXt | Gemma |
| High ( 0.8) | 110 | 81.8% | 80.9% |
| Medium (0.5–0.8) | 94 | 57.4% | 57.4% |
| Low ( 0.5) | 48 | 35.4% | 41.7% |
Table 11: PlantDoc: accuracy conditioned on the maximum CNN confidence score, a proxy for image difficulty.
| Model | Critical | High | Medium | Low |
| Gemma 4 E4B | 25 (9.9%) | 99 (39.3%) | 87 (34.5%) | 41 (16.3%) |
| Qwen3.5 4B | 92 (36.5%) | 58 (23.0%) | 51 (20.2%) | 49 (19.4%) |
Table 12: PlantDoc: business risk level distribution assigned by the MLLM reasoning agents across 252 test images.
| Model | Acc. | Macro F1 | W. F1 | Cov. |
| EfficientNet-B3 | 98.5% | 0.9852 | 0.9852 | 100% |
| ConvNeXt-Tiny | 99.8% | 0.9974 | 0.9975 | 100% |
| Gemma 4 E4B (MLLM) | 99.3% | 0.9921 | 0.9925 | 100% |
| Qwen3.5 4B (MLLM) | 97.7% | 0.9763 | 0.9773 | 98.5% |
Table 13: Cornell Stage 2: classification performance (403 test images, 3 classes). Best results in bold .
| Class | P | R | F1 | |
| Early Blight | 1.00 | 1.00 | 1.00 | 147 |
| Late Blight | 0.99 | 1.00 | 1.00 | 141 |
| Septoria Leaf Spot | 1.00 | 0.99 | 1.00 | 115 |
Table 14: Cornell Stage 2: per-class precision/recall/F1 (ConvNeXt-Tiny) and Gemma accuracy. = test support.
| Override type | Gemma | Qwen | ||
| Cases | Acc. | Cases | Acc. | |
| Agrees with both CNNs | 396 | 100.0% | 387 | 100.0% |
| Agrees w/ ConvNeXt only | 5 | 80.0% | 1 | 100.0% |
| Agrees w/ EfficientNet | 2 | 0.0% | 2 | 0.0% |
| Overrides both CNNs | 0 | N/A | 8 | 0.0% |
Table 15: Cornell Stage 2: MLLM override behaviour and accuracy per override type.
| Model | Critical | High | Medium/Low/Unk. |
| Gemma 4 E4B | 139 (34.5%) | 78 (19.4%) | 186 (46.2%) / 0 / 0 |
| Qwen3.5 4B | 127 (31.5%) | 25 (6.2%) | 230 (57.1%) / 16 (4.0%) / 5 (1.2%) |
Table 16: Cornell Stage 2: business risk level distribution across 403 test images.
| Model | Acc. | Macro F1 | W. F1 | Cov. |
| EfficientNet-B3 | 96.8% | 0.9725 | 0.9679 | 100% |
| ConvNeXt-Tiny | 98.3% | 0.9856 | 0.9833 | 100% |
| Gemma 4 E4B (MLLM) | 98.9% | 0.9900 | 0.9888 | 100% |
| Qwen3.5 4B (MLLM) | 91.8% | 0.9108 | 0.9200 | 98.9% |
Table 17: Cornell Stage 4: classification performance (715 test images, 3 classes). Best results in bold .
| Class | P | R | F1 | |
| Early Blight | 0.99 | 0.97 | 0.98 | 329 |
| Late Blight | 1.00 | 1.00 | 1.00 | 146 |
| Septoria Leaf Spot | 0.96 | 0.99 | 0.98 | 240 |
Table 18: Cornell Stage 4: per-class precision/recall/F1 (ConvNeXt-Tiny). = test support.
| Override type | Gemma | Qwen | ||
| Cases | Acc. | Cases | Acc. | |
| Agrees with both CNNs | 686 | 99.6% | 634 | 99.8% |
| Agrees w/ ConvNeXt only | 19 | 89.5% | 13 | 76.9% |
| Agrees w/ EfficientNet | 9 | 77.8% | 7 | 71.4% |
| Overrides both CNNs | 1 | 0.0% | 54 | 1.9% |
Table 19: Cornell Stage 4: MLLM override behaviour and accuracy per override type.
| Model | Critical | High | Medium/Low/Unk. |
| Gemma 4 E4B | 147 (20.6%) | 229 (32.0%) | 339 (47.4%) / 0 / 0 |
| Qwen3.5 4B | 249 (34.8%) | 147 (20.6%) | 306 (42.8%) / 5 (0.7%) / 8 (1.1%) |
Table 20: Cornell Stage 4: business risk level distribution across 715 test images.
Figure 4: Top-1 accuracy of all four models across the three evaluation datasets. All models improve substantially from PlantDoc’s 27-class, internet-curated benchmark to the 3-class, real-world Cornell field datasets, with the CNN experts and Gemma remaining stable across both Cornell campaigns while Qwen shows greater variability.
| Dataset | Test | Agree | Disagree ( ) | Best CNN (conflict) | Best MLLM (conflict) |
| PlantDoc | 252 | 58.3% | 41.7% (105) | 43.8% (ConvNeXt) | 51.4% (Qwen), 7.6 pp |
| Cornell Stage 2 | 403 | 98.3% | 1.7% (7) | 85.7% (ConvNeXt) | 57.1% (Gemma), 28.6 pp |
| Cornell Stage 4 | 715 | 95.9% | 4.1% (29) | 69.0% (ConvNeXt) | 82.8% (Gemma), 13.8 pp |
Table 21: Cross-dataset synthesis: vision-expert agreement/disagreement rate and best-MLLM vs. best-CNN accuracy on the conflict-only subset.
Figure 5: (a) Vision-expert agreement/disagreement rate across the three datasets. (b) Best-MLLM minus best-CNN accuracy on the conflict-only image subset for each dataset, with sample size annotated. MLLM arbitration provides a substantial, positive gain when the conflict subset is reasonably sized (PlantDoc, ; Stage 4, ), but the tiny Stage 2 conflict subset ( ) is dominated by sampling noise and should not be interpreted as evidence against the framework.
Figure 6: Critical-risk assignment rate for Gemma 4 E4B and Qwen3.5 4B across the three datasets. Dashed lines indicate the true prevalence of the sole “high risk-aversion” disease class (Late Blight) in the two Cornell test sets, providing an objective calibration target. Gemma’s Critical rate tracks true prevalence almost exactly (0.14–0.5-point error); Qwen consistently over-flags (3.5–14.4-point error).
| Dataset | True prevalence | Gemma CRCE | Qwen CRCE |
| Cornell Stage 2 | 35.0% | 0.5 pp | 3.5 pp |
| Cornell Stage 4 | 20.4% | 0.14 pp | 14.4 pp |
Table 22: Critical-Risk Calibration Error (CRCE): absolute difference between assigned Critical-risk rate and true Late Blight prevalence, on the two Cornell datasets where ground truth is well defined.
Figure 7: Confusion matrix of ConvNeXt-Tiny predictions on the 27-class PlantDoc test set. Rows represent ground-truth classes, whereas columns represent predicted classes. Although most predictions are concentrated along the main diagonal, off-diagonal errors remain among visually similar disease categories, especially within the tomato classes. The matrix illustrates the difficulty of fine-grained disease recognition under heterogeneous backgrounds, illumination conditions, symptom appearances, and limited class-specific test support.
Figure 8: Confusion matrix of ConvNeXt-Tiny predictions on the Cornell Stage 2 test set. Rows denote ground-truth disease classes and columns denote predicted classes. The strongly diagonal distribution demonstrates high discrimination among Early Blight, Late Blight, and Septoria Leaf Spot under the Stage 2 field-acquisition conditions, with only a very small number of cross-class errors.
Figure 9: Confusion matrix of ConvNeXt-Tiny predictions on the Cornell Stage 4 test set. The matrix remains strongly concentrated along the main diagonal, indicating robust discrimination among the three tomato foliar diseases. The principal residual confusion occurs when a small number of Septoria Leaf Spot images are classified as Early Blight, reflecting the partially overlapping necrotic lesion characteristics of these diseases under natural field conditions.
More in this topic
Plant Disease Classification
Plant disease classification uses image and sensor data to identify crop diseases and support timely, targeted agricultural intervention.
See all 62 papers