cs.AIAug 28, 2026
SavePhenoIntel: A Lifecycle-Aligned Multi-Agent Web Application for Verified, Accessible Plant Phenotype Analysis
Organizations: VIT Bhopal University Bhopal, Madhya Pradesh, India · Centre of Studies in Resources Engineering Indian Institute of Technology Bombay Mumbai, Maharashtra, India
Abstract
Existing conversational plant-phenotyping platforms are difficult for plant scientists to use and lack the reliability scientific research demands: failed analyses are reported as valid measurements rather than flagged as missing, statistical tests run without checking assumptions, predictions carry no uncertainty estimate, and specialised hardware limits accessibility. We present PhenoIntel, a lifecycle-aligned multi-agent web platform that turns the full machine-learning workflow into a reliable, user-friendly phenotyping system. Nine specialised agents divide the analysis into stages, from image collection through model selection, inference, and reporting, rather than handing the whole task to one AI manager. Independent checks separate these stages, and every agent reads from and writes to one shared, fixed-structure record, so an inconsistent output from one stage is caught before it reaches the next. Uncertainty is matched to each model family, conformal prediction, detection-confidence spread, or Monte Carlo Dropout, rather than applied uniformly, and quality thresholds adapt to crop and task instead of one global cutoff. When no suitable model exists, PhenoIntel can propose, validate, and integrate a new one on its own. The model repository spans ten trained models across five crops and four imaging modalities. Classification models reach Macro F1 of 0.78-0.996; object-detection models reach 0.96 mAP@50 with a 54% reduction in counting error over an unoptimised baseline; and a temporal model reaches held-out Macro F1 of 0.7050. PhenoIntel runs in a browser on standard hardware, requiring no GPU, and a 1,200-test automated suite confirms complete pipeline execution. Every result carries calibrated uncertainty, validated statistics, and FAIR-compliant provenance, a combination existing conversational phenotyping tools do not offer.
Figures & tables
| Principle | Failure Category Addressed | Section |
| 1. Lifecycle-aligned agent decomposition | Monolithic orchestrator conflates all stages; non-deterministic routing; 30% tool-selection error rate | Section 4.2 |
| 2. Typed state contracts (shared session state) | Free-text inter-tool communication; fragile filename hand-off (different segmentation backends save under different naming conventions); silent breakage on message format changes | Section 4.5 |
| 3. Inter-stage verification with bounded recovery | Detection failure recorded as a plain zero; corrupted values propagate silently into statistics | Section 4.5 |
| 4. Per-model uncertainty quantification | Single-point phenotype estimates; no confidence intervals; MC Dropout misapplied to architectures that disable dropout at inference time | Section 4.4 |
| 5. Biological plausibility validation | Implausible values propagate unchecked; exclusive-or mask combination underestimates overlapping leaf area | Section 4.4 |
| 6. Sandboxed statistical execution | Model-generated code runs unsandboxed on local machine with container isolation disabled; no filesystem restriction, no timeout, no import allowlist | Section 4.6 |
Table 1: Seven architectural principles of PhenoIntel and the failure categories from prior monolithic phenotyping orchestrators each addresses.
| Crop | Task / Modality | Size | Source |
| Rice | Nitrogen severity / RGB close-range | 5,790 | Author-processed (Kaggle), kaggle.com/datasets/narendrensv/pheno-assistant |
| Wheat | Nutrient deficiency / UAV aerial | 3,000 | ( Yi et al.,, 2025 ) , DND-Diko-WWWR, github.com/jh-yi/DND-Diko-WWWR |
| Maize | Nutrient deficiency / RGB close-range | 17,000 | Author-processed (Kaggle), kaggle.com/datasets/narendrensv/pheno-assistant |
| Banana | Micronutrient deficiency / RGB | 1,500 | Public (Kaggle), kaggle.com/datasets/warcoder/nutrient-deficient-banana-plant-leaves |
| Coffee | Nutrient deficiency / RGB close-range | 800 | Author-processed CoLeaf (Kaggle), kaggle.com/datasets/narendrensv/coleaf |
| Rice (panicle) | Detection/counting / UAV aerial | 1,800 | Public (Kaggle), kaggle.com/datasets/rifat963/…bangladesh |
Table 2: Datasets used across the PhenoIntel model zoo. Sizes and splits describe the complete dataset used for fine-tuning and held-out evaluation; the smaller, separately curated case-study input sets are reported in Table 7 .
Figure 2: End-to-end methodology at a glance. Curation (Section 3 ) and background concepts (Section 4.1 ) are prerequisites shown outside this figure; every box here is described in full architectural detail from Section 4 onward.
| Agent (Stage) | Owns Exclusively | Failure Mode Addressed |
| Planning (0) | Structured session plan from an LLM, offline keyword fallback | No formal planning stage; ad-hoc per-turn routing |
| Data Collection (1) | Image validation, capture metadata, pixel calibration | Local-file/URL only; no calibration validation |
| Preprocessing (2) | Normalisation, channel-adapter projection, NaN checks | Corrupted tensors passed silently to inference |
| Feature Engineering (3) | ViT features, spectral indices, temporal assembly | No explicit feature stage; ad hoc prompts |
| Dataset Partitioning (4, Build) | Stratified split, leakage/near-duplicate/mislabel detection | No partitioning validation; leakage undetected |
| Model Selection (5) | Registry look-up, four-tier fallback | Text-based selection, 30% error rate |
Table 3: The nine lifecycle-aligned pipeline agents and the specific failure mode from existing monolithic orchestrators each one’s responsibility is designed to prevent.
| Source | What it provides | Example models |
| Built-in zoo | Small, hand-vetted set already used by the fixed pipeline; no download or matching step needed. | Fixed detection and classification backbones |
| timm | Pretrained convolutional and vision-transformer checkpoints spanning hundreds of architectures, fetched by name. | ConvNeXt, EfficientNet, ViT variants |
| Hugging Face Hub | Any compatible transformers / timm -hosted checkpoint, identified by repository name. | Community and research vision checkpoints |
| torchvision | Standard reference implementations bundled with PyTorch’s vision library. | ResNet, MobileNetV3 |
| Custom generation | A language model proposes a new architecture layer-by-layer when no external source fits; checked against the same structural rules and critique pass as any other candidate. | e.g. the CS5 single-stream ViT-S/16 five-head regression network |
Table 4: Backbone sources available to the Pipeline Adaptation Agent. All five paths compile through the same schema validation and verifier before a model is registered.
| Model Family | UQ Method | Interval Produced | Calibration Source |
| Classification | Conformal prediction (RAPS) ( Angelopoulos et al.,, 2021 ) | Prediction set guaranteed to contain the true class at a chosen coverage | Held-out calibration slice reserved during curation (Section 3.2 ) |
| Detection & zero-shot segmentation | Detection-confidence spread ( Celikkan et al.,, 2023 ) | Count/area interval width scaled by per-image detection confidence | Detection-confidence distribution across the image (no separate held-out set) |
| Temporal modelling | Monte Carlo Dropout ( ) ( Gal & Ghahramani,, 2016 ) + quantile regression heads ( Romano et al.,, 2019 ) | 95% interval from dropout spread; P10–P90 from quantile heads directly | Dropout kept active at inference; quantile heads trained with a monotonicity penalty |
Table 5: Uncertainty-quantification method dispatched per model family (Section 4.4 ), the interval it produces, and its calibration source.
| Checkpoint | Stage Boundary | Key Checks | Failure Action | Mutates State |
| V0 Pre-flight | Before the data-collection stage | Valid (crop, task, modality) triple; 1 image path; basic pixel content | Route to error before compute time is consumed | No |
| V1 Mass Integrity | After the data-collection stage | 1 validated image path; all files exist on disk; calibration mode valid; minimum resolution per modality | Route to recovery | No |
| V2 Normalisation | After Preprocessing | Preprocessed tensor count = image count; no tensor NaN; pixel values within expected normalised range | Route to recovery | No |
| V3 Feature Validity | After Feature Engineering | NDVI ; temporal array shape for the temporal model; spectral band ranges valid | Route to recovery | No |
| V4 Model Compatibility | After the model-selection stage | A model card was actually selected; modality match; checkpoint file exists (skipped for zero-shot/generated cards) | Route to recovery | No |
| V5 Post-Inference | After Inference | Detection status explicitly set, never left as "not run"; "no detections" pre-fills as missing (not a literal zero); a "success" status with empty output is a hard error | Missing-value pre-fill; Route to recovery | Yes |
Table 6: PhenoIntel verification checkpoints (V0–V7): stage, check summary, failure action, and whether the checkpoint can mutate state.
| Case Study | Mechanism Exercised | Min. Dataset |
| CS1 Wheat head detection | Tier-0 exact-match PREDICT; establishes baseline completion/F1 | 10 GWHD images |
| CS2 Arabidopsis leaf count | Tier-2 zero-shot segmentation fallback; manual calibration applied | 5 CVPPP2015 images |
| CS3 Tomato disease (BUILD) | Classification backbone, quality gate passed first attempt | 375 images, 5 classes |
| CS4 Cotton boll detection (BUILD) | Detection backbone, quality gate passed first attempt | 60 images |
| CS5 Lettuce multi-trait regression (BUILD) | ADAPT recovery: 3 inference failures generate-critique-verify loop | 112 RGB images |
| CS6 Satellite crop type (BUILD) | Degenerate -pixel case of the general temporal input format | 800 pixel series, 4 classes |
Table 7: Six cross-domain case studies. Group 1 exercises the PREDICT path (exact match or zero-shot fallback); Group 2 exercises BUILD (training a new model from an uploaded dataset).
Figure 7: PREDICT-workflow image-requirement notes generated automatically at model-matching time, as rendered in the web UI. CS1 resolves to an exact registered checkpoint, so no fallback notice is shown; CS2 has no registered checkpoint for Arabidopsis leaf counting, so the system surfaces the reduced-confidence, uncalibrated nature of the zero-shot path before the session proceeds.
Figure 9: Model-zoo cards auto-generated and registered at the end of each BUILD case study (CS3–CS6). Each records the model identifier, crop/task tags, headline metric, the dispatched uncertainty-quantification method, and the phenotype outputs the model exposes downstream: a provenance record a reviewer can inspect without touching the underlying code. The CS5 card is labelled auto_0_57b4b4ec : an architecture generated live by the Pipeline Adaptation Agent rather than drawn from the fixed model zoo.
| Model | Crop / Domain | Backbone | Origin | (eval) | Primary Metric | Secondary Metric / UQ | Reliability |
| Group 1, Close-Range Classification | |||||||
| 1. Rice N-severity classifier | Rice | Classification model | Manual | 5,790 (full-pool) | Macro F1 0.9721 a | ECE 0.0227 | Medium |
| 2. Wheat UAV classification model (TTA) | Wheat | Classification model | Manual | 1,896 (full-pool) | Macro F1 0.9052 a | ECE 0.1001 | Medium |
| 3. Maize classification model | Maize | Classification model | Manual | 17,627 (full-pool) | Macro F1 0.9920 a | ECE 0.1219 | Medium |
| 4. Banana classification model | Banana | Classification model | Manual | 1,500 (80/20 strat., seed = 42) | Macro F1 0.8555 | ECE 0.0560 | High |
| 5. Coffee classification model | Coffee | Classification model | Manual | 90 (held-out, seed = 42 split) | Macro F1 0.7811 | ECE 0.1418 | High |
Table 8: All 14 trained PhenoIntel checkpoints (10 model-zoo, manually trained offline; 4 case-study Build sessions run live through the web application), grouped by task family with a common column set. is the evaluation-pool size actually used for the reported figure, not the full dataset size. Reliability tiers ( High / Medium ) are defined in Section 6.5 .
| Metric | PhenoIntel |
| Reliability | |
| Pipeline completion rate (all 16 sessions: 10 model-zoo checkpoints + 6 case studies, Section 6.2 ) | 100.0% |
| Zero-from-detection failure rate (enforced at checkpoint V5, Section 4.5 ) | 0% (by design) |
| Unhandled exception / crash rate (0 crashes across all 16 sessions, Section 6.2 ) | 0.0% |
| Internal test-suite pass rate (1,128 unit + 72 integration tests, Section 6.9 ) | 1200/1200 (100%) |
| Phenotype Accuracy, CVPPP2015 (Zero-Shot, CS2) | |
Table 9: PhenoIntel evaluation summary across all evaluation dimensions and metrics. Rows marked a–d report two or more figures under distinct evaluation protocols (see footnotes below); all other rows report a single, unambiguous protocol.
| (a) Classification, Close-Range Deficiency / Disease Classifiers (Models 1–6) | ||
| 1. Rice N-severity, Confusion Matrix | 1. Rice N-severity, Reliability | 2. Wheat UAV, Confusion Matrix |
| 2. Wheat UAV, Reliability | 3. Maize, Confusion Matrix | 3. Maize, Reliability |
| 4. Banana, Confusion Matrix | 4. Banana, Reliability | 5. Coffee, Confusion Matrix |
| 5. Coffee, Reliability | 6. Tomato (CS3), Confusion Matrix | 6. Tomato (CS3), Reliability |
Figure 10: (a) Classification , Close-Range Deficiency/Disease Classifiers (Models 1–6): confusion matrix and reliability diagram per classification model (this page); continued over the following pages, with the full caption given after the final panel (e).
| (b) Detection (Models 7–9) and (c) Temporal (Model 10) Results | ||
| 7. Rice Panicle, Detections | 7. Rice Panicle, PR Curve | 8. Wheat Spike (GWHD), Detections |
| 8. Wheat Spike (GWHD), PR Curve | 9. Cotton Boll (CS4), Confusion Matrix | 9. Cotton Boll (CS4), PR Curve |
| 10. Crop-Type Head, Ensemble Comparison | 10. Crop-Type Head, Per-Fold F1 | 10. Satellite (CS6), Training Curve |
| 10. Satellite (CS6), Self-Eval Accuracy/F1 | 10. Satellite (CS6), Self-Eval Confidence Dist. | CS2 Arabidopsis, Pred. vs. True Leaf Count |
Figure 10 continued: (b) Detection models 7–9; (c) Temporal model 10.
| (d) Segmentation (CS2) and Regression (Models 11–12) Results | ||
| CS2 Arabidopsis, Per-Image Residuals | 11. Growth-Rate Head, Pred. vs. True | 11. Growth-Rate Head, Residuals |
| 12. Harvest-Timing Head, Pred. vs. True | 12. Harvest-Timing Head, Residuals | |
| (e) Multi-Trait Regression (Lettuce, CS5): Predicted vs. True | ||
| 13. Lettuce, Diameter, Pred. vs. True | 13. Lettuce, Dry Weight (Shoot), Pred. vs. True | 13. Lettuce, Fresh Weight (Shoot), Pred. vs. True |
| 13. Lettuce, Height, Pred. vs. True | 13. Lettuce, Leaf Area, Pred. vs. True | 13. Lettuce, Diameter, Residuals |
Figure 10 continued: (d) Segmentation (CS2) and Regression models 11–12; (e) Lettuce (CS5) predicted vs. true, per trait.
Figure 11: FAIR provenance package: 12 always-present files, checksum-indexed in the package manifest. Badges indicate which FAIR principle each file satisfies.
More in this topic
Plant Phenotyping
Plant phenotyping uses imaging, sensing, and computational methods to measure plant traits, growth, health, and responses to the environment.
See all 68 papers