cs.ROOct 7, 2026

IVG-UAV: An Intelligent Voice-Guided UAV System for Autonomous Ripe Fruit Harvesting with Vision-Based Classification and Adaptive Path Planning

Authors: Dinh Trung Duong

Organizations: School of Computing SUNY Binghamton University Binghamton, United States

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In tropical regions, their agricultural sectors remain highly dependent on manual labor for fruit harvesting. On large-scale farms, this dependency often results in significant labor cost and logistic complexities. This project presents the development and simulation of a voice-controlled Unmanned Aerial Vehicle (UAV) system designed to automate harvesting tasks in extensive plantations. The proposed system integrates speech recognition using Whisper [1] and LLM, computer vision-based ripeness classification, and adaptive path planning within a unified framework. The entire system is modeled and validated in a Gazebo simulation environment, allowing performance evaluation under controlled agricultural scenarios

Figures & tables

Fig. 1: UAV system will integrate voice command to automate the harvesting process. Using laser beam equipped with vision-based ripeness classification to recognize the satisfied objects and avoid obstacle during the harvesting process. Applying RRT* algorithm to find the shortest path aimed to the target
Fig. 2: Exploration and orchard mapping flow. The UAV performs exploration flight, the vision module detects trees, and the orchard map is constructed from the detected tree positions
Fig. 3: Gazebo environment conversion into probabilistic voxel-based 3D map using Octomap [ 5 ]
Fig. 4: The mission execution pipeline starts with voice commands, which are converted into mission instructions using Whisper and an LLM-based interpreter. The adaptive path planner generates navigation commands while handling obstacle avoidance. MAVSDK transmits control commands to the UAV, while the vision perception module continuously processes visual data during flight. The system runs on ROS2 within a Gazebo simulation environment.
Fig. 5: Multi-architecture LLM layers
Fig. 6: The ripe fruit is detected by detection module
Fig. 7: Overall evaluation overview of the proposed framework across 10 experimental trials.
MetricRate (%)
Voice Command Interpretation Accuracy70.0
Valid Task Plan Generation Rate85.7
Successful Navigation Rate100.0
Average Ripe Fruit Detection Rate63.1
TABLE I: Performance metrics of the proposed UAV harvesting framework based on 10 experimental trials
Fig. 8: Comparison of evaluation metrics for the proposed UAV harvesting framework.

Explore similar work

CardsList

  1. Robotic Strawberry Harvesting with Robust Vision and Deep Reinforcement Learning based Sim-to-Real Control

    May 22, 2026Al Bashir, Shao-Yang Chang, Partho Ghose +3Agricultural RoboticsSimulation-Based Reinforcement Learning

    Department of Biological & Agricultural Engineering, Texas A&M University, College Station, TX 77843, USA · Department of Biomechatronics Engineering, National Taiwan University, 10617, Taipei City, Taiwan · Texas A&M AgriLife Research, Texas A&M University System, Dallas, TX 75252, USA

  2. Vision-Based Obstacle Separation for Strawberry Harvesting in Clusters Using Hierarchical Reinforcement Learning

    Jul 15, 2026Teng Li, Hanfei Shi, Chunjiang Zhao +1Agricultural RoboticsObstacle Avoidance

    The Intelligent Equipment Research Center, Beijing Academy of Agriculture and Forestry Sciences, Beijing 100097, China. · School of Intelligence Science and Technology, University of Science and Technology Beijing, Beijing 100083, China. · Happy Elements Ltd., Beijing 100094, China.

  3. Real-Time Visual Intelligence on Low-Cost UAVs: A Modular Approach for Tracking, Scanning, and Navigation

    Jul 2, 2026Andrei-Marian Ungureanu, Stelian SpînuAutonomous DronesUnmanned Aerial Vehicles

    Military Technical Academy “Ferdinand I”, Bucharest, Romania