A Lightweight Multimodal Vision-Language Framework for Early-Stage Anatomical Green Fruit Classification in Commercial Orchards
Authors: Ranjan Sapkota, William Bu, Chen Chen, Yunjun Xu, Manoj Karkee
Organizations: Cornell University, Department of Environmental and Biological Engineering, USA · Department of Computer Science, University of Central Florida, USA · Institute of Artificial Intelligence (IAI) & Department of Computer Science, University of Central Florida, USA · UCF Department of Mechanical and Aerospace Engineering, University of Central Florida, USA
Accurate identification of early-stage apple fruitlet anatomical structures, including the calyx, fruitlet body, and peduncle, is essential for robotic thinning, crop-load management, and other precision orchard operations. This study presents a lightweight multimodal vision-language framework that adapts TinyCLIP for fine-grained fruitlet anatomy classification in complex orchard environments. A dataset of 600 high-resolution RGB images collected from Scilate and Scifresh apple orchards was converted into 224 x 224 image patches and annotated for three anatomical classes. Domain-specific language prompts, such as ``a photo of a class,'' were used to guide multimodal alignment between orchard imagery and horticultural structures. A sliding-window inference strategy with a stride of 112 pixels aggregates patch-level predictions into spatial heatmaps, enabling interpretable whole-image localization of fruitlet components relevant to robotic thinning. Patch-level evaluation on an NVIDIA T4 GPU achieved F1-scores of 0.95 for calyx, 0.98 for fruitlet, and 0.85 for peduncle, with a macro-F1 score of 0.93. Deployment-oriented optimization using ONNX and TensorRT enabled efficient inference on NVIDIA Jetson hardware, preserved accuracy under INT8 quantization, and supported model sizes of approximately 127-137 MB with millisecond-level patch inference. These results demonstrate that lightweight vision-language models can provide interpretable and edge-deployable perception for automated fruitlet analysis and future robotic thinning systems. The source code and implementation details are publicly available at https://github.com/WilliamBu1/A-Lightweight-Vision-Language-Model-for-Early-Stage-Fruitlet-Classification-in-Apple-Orchards.
Figures & tables
Fig. 1: (a) Manual thinning of immature green fruitlets in a commercial apple orchard in Washington State. (b) Complex orchard scene with camouflaged early-stage fruitlets and their anatomical components (calyx, fruitlet, peduncle).
Fig. 2: Visual illustration of the orchard setting and imaging conditions. The left panel shows the high-density planting arrangement of Scifresh and Scilate cultivars, while the right panel highlights immature fruitlet clusters captured at varying distances and angles with an iPhone 14 Pro.
Fig. 3: Architecture showing the 224×224 orchard patch input, TinyCLIP vision encoder, text prompts, text encoder, image and text embeddings, cosine similarity module, sigmoid probability head, and multi-label outputs for calyx, fruitlet, and peduncle classification.
Fig. 4: (a) TinyCLIP affinity mimicking: the student distills CLIP’s multimodal geometry by matching image-to-text and text-to-image similarity distributions. (b) Weight inheritance strategy: masks select transferable parameters from a pre-trained CLIP model, enabling compact student initialization while discarding non-essential weights.
Fig. 5: Illustration of TinyCLIP’s multi-modal classification on an early-season orchard image. The left panel shows the original iPhone 14 Pro image captured in a commercial orchard, while the right panel presents representative 224×224 patches with corresponding ground-truth (“True”) and predicted (“Pred”) labels for fruitlet, calyx, and peduncle, demonstrating reliable fine-grained anatomical discrimination.
Fig. 6: Confusion matrices for TinyCLIP evaluated on the Jetson platform under (a) FP16 half-precision and (b) INT8 integer quantization. Diagonal strength indicates correct classification of calyx, fruitlet, peduncle, and negative patches, illustrating the effect of quantization on model discriminability.
Fig. 7: Per-class precision–recall curves for TinyCLIP under (a) FP16 and (b) INT8 inference. Each curve illustrates the trade-off between precision and recall for calyx, fruitlet, peduncle, and negative patches. Steeper, high-precision regions indicate stronger discriminability, while flatter curves (notably for peduncle) reflect the increased difficulty of detecting small, low-contrast anatomical structures in early-season orchard imagery.
Fig. 8: Per-class precision–recall curves for TinyCLIP under (a) FP16 and (b) INT8 inference. Curves illustrate threshold-dependent discriminability for calyx, fruitlet, peduncle, and negative patches. FP16 yields high precision across most recall levels, while INT8 shows expected reductions for fine-detail classes such as peduncle, demonstrating quantization effects on anatomical classification.
Fig. 9: Example Heatmaps of TinyCLIP based fruitlet parts (calyx, main fruitlet and peduncle) classification and localization
Class
IoU Mean
IoU Std. Dev.
Calyx
0.61
0.13
Fruitlet
0.67
0.10
Peduncle
0.49
0.14
TABLE III: Localization Performance via Intersection-over-Union (IoU)
Engine
Accuracy
Macro F1
Peduncle Recall
FPS
FP16
0.9115
0.9115
0.7500
96.33
INT8
0.8879
0.8881
0.6786
112.02
TABLE IV: Effect of Quantization on Classification Performance
Metric
Value
Engine Size
126.98 MB
Average Latency
10.38 ms
Throughput
96.33 FPS
Peak Memory (CUDA)
6452.11 MB
PyTorch Memory
9.28 MB
TABLE V: FP16 TensorRT Deployment Performance
Metric
Value
Engine Size
137.15 MB
Average Latency
8.93 ms
Throughput
112.02 FPS
Peak Memory (CUDA)
6519.40 MB
PyTorch Memory
9.28 MB
TABLE VI: INT8 TensorRT Deployment Performance
Fig. 10: Representative examples of correctly classified peduncle patches extracted from an early-season orchard image. The figure highlights the model’s ability to identify slender, low-contrast peduncles under challenging conditions including occlusion, variable illumination, and canopy clutter. These examples demonstrate the effectiveness of the lightweight TinyCLIP vision–language model in recognizing anatomically meaningful peduncle structures essential for autonomous robotic fruit thinning.
Fig. 11: Conceptual future roadmap for autonomous green-fruit thinning. A robotic arm equipped with a multimodal machine-vision camera and precision cutting end-effector identifies fruitlet anatomical structures—calyx, fruitlet body, and peduncle within a dense orchard canopy. The system illustrates how lightweight vision-language models can guide peduncle-targeted removal for safe, efficient, and scalable robotic thinning in commercial orchards.