From Vision to Harvest: Benchmarking Vision-Language Models for Multi-Arm Robotic Fruit Harvesting

Vrishan Inukollu; Adyan Zaman; Anvi Kudaraya; Carlos Lazcano; Yuankai Zhu; Stavros Vougioukas; Xiaofan Yu

cs.ROSep 11, 2026

From Vision to Harvest: Benchmarking Vision-Language Models for Multi-Arm Robotic Fruit Harvesting

Authors: Vrishan Inukollu, Adyan Zaman, Anvi Kudaraya, Carlos Lazcano, Yuankai Zhu, Stavros Vougioukas, Xiaofan Yu

Organizations: University of California, Merced · University of California, Davis

Abstract

Multi-arm robotic harvesting offers a promising path to improve harvesting efficiency and reduce reliance on manual labor. However, practical deployment remains challenging because the system must generalize across diverse environments while efficiently coordinating multiple arms in a shared workspace. Existing methods often require substantial data collection in target environments or rely on simplifying assumptions that limit planning quality. In this work, we introduce the first comprehensive benchmark for evaluating pretrained Vision-Language Models (VLMs) on zero-shot multi-arm fruit harvesting planning. Our benchmark uses real-world apple and citrus orchard images and compares a VLM-based planning pipeline with a traditional perception-and-planning pipeline. The VLM pipeline directly generates harvesting sequences and waypoints for each arm, while a lightweight trajectory verifier checks for collisions. Our results show that frontier VLMs can generate effective multi-arm harvesting plans zero-shot, but a practical deployment remains limited by accurate 3D waypoint generation and collision-aware coordination. These results highlight both the promise and current limitations of pretrained VLMs for multi-arm robotic harvesting.

Figures & tables

Figure 1: Overview of our benchmark for evaluating zero-shot VLMs in multi-arm robotic fruit harvesting. An oracle planner provides an upper-bound reference for assessing VLMs as a generalizable and efficient alternative to intractable multi-arm planning.

Figure 2: The three-stage oracle pipeline, consisting of open-vocabulary detection and segmentation, 3D target localization and task allocation, and multi-arm trajectory optimization. Each stage is implemented using state-of-the-art works for the corresponding task.

Panel of Figure 3 — Figure 3: Prompt template used for zero-shot VLM-based multi-arm harvesting planning. The placeholders {FRUIT} , {N} , {DEPTH} , and {ZONE} are instantiated according to the crop, robot configuration, and experimental setting.

Category	Models
Closed-source	GPT-4o [ 23 ] , GPT-5.5-Pro [ 24 ] , Gemini 3 Flash [ 8 ] , Claude Sonnet 3.5 [ 1 ] , Claude Opus 4.7 [ 1 ]
Open-source	Qwen2-VL-7B [ 5 ] , GLM 4.6 [ 10 ]

Table 1: VLMs evaluated in our benchmark.

Figure 4: Qualitative comparison of VLM-generated trajectories against the oracle pipeline on a representative 6-fruit orchard scene using a 3-arm robot. (a) Input RGB orchard image. (b) 3D fruit localization from the oracle pipeline. (c) Oracle trajectory, reported with rectilinear trajectory distance. (d–i) VLM-generated trajectories for Claude Opus 4.7, Claude Sonnet 3.5, Gemini 3 Flash, GLM, GPT-4o, and Qwen2.5-VL-7B, respectively. Each panel reports its rectilinear trajectory distance. Red and grey fruit markers denote fruit included and not included, respectively, in the model’s harvesting plan.

Model	Dataset	Harvested Ratio (%) $\uparrow$			Detected	Collision	Traj.	Avg.
		$\delta=1.0$ m	$\delta=0.5$ m	$\delta=0.25$ m	Ratio (%) $\uparrow$	Ratio (%) $\downarrow$	Dist. (m) $\downarrow$	Tokens $\downarrow$
Optimal Planner ∗	Apple	100.0	100.0	100.0	100.0	100.0	0	12.4
Optimal Planner ∗	Citrus	100.0	100.0	100.0	100.0	100.0	0	5.5
Closed-source VLMs
GPT-5.5-Pro [ 24 ]	Apple	93.8	85.7	42.4	93.8	42.2	12.8	6364
GPT-5.5-Pro [ 24 ]	Citrus	99.3	60.7	6.7	89.8	37.4	13.7	8175

Table 2: Comprehensive performance comparison of VLM backends for multi-arm fruit-harvesting planning on the Apple and Citrus datasets. Metrics include Harvested Ratio across three spatial thresholds (

\delta

), collision ratio, trajectory distance, and average token usage. Results are benchmarked against the optimal planner. Best reported VLM result in each column is shown in Red text for apples and Orange text for citrus.

Prompt	Harvested Ratio (%)			Detected Ratio
	$\delta{=}1.0$ m	$\delta{=}0.5$ m	$\delta{=}0.25$ m	(%)
V1 – Full	93.3 $\pm$ 14.9	73.5 $\pm$ 22.4	36.9 $\pm$ 14.9	84.4 $\pm$ 8.4
V2 – w/o Scale	74.7 $\pm$ 35.4	47.9 $\pm$ 44.9	22.3 $\pm$ 24.8	80.8 $\pm$ 14.9
V3 – w/o Role	75.7 $\pm$ 33.6	43.2 $\pm$ 32.5	16.2 $\pm$ 16.4	75.9 $\pm$ 18.0
V4 – Minimal	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	45.3 $\pm$ 33.3
V5 – w/o Depth	96.9 $\pm$ 5.4	86.5 $\pm$ 23.4	49.0 $\pm$ 44.2	96.9 $\pm$ 5.4

Table 3: Prompt ablation summary, averaged across all five VLMs (mean

\pm

std). Each variant removes one component from the full prompt (V1). Best results are shown in bold .

Dataset	Fruit Count	$n$ (scenes)	Detection Ratio (%)	Harvested Ratio (%)
Apple	$<5$ apples	43	40.1	60.1
	$5$ – $10$ apples	86	38.1	53.5
	$>10$ apples	5	23.8	42.8
Citrus	$<5$ fruits	46	100.0	47.0
	$5$ – $10$ fruits	56	99.2	62.7
	$>10$ fruits	2	100.0	52.2

Table 4: Impact of fruit count on GPT-5.5-Pro performance with 2 arms fixed.

n

denotes the number of evaluated scenes in each group. Apple performance decreases as scene complexity increases, whereas Citrus detection remains near-perfect and harvested ratio varies across count groups.

Dataset	Arm Count	$n$ (scenes)	Detection Ratio (%)	Harvested Ratio (%)
Apple	2 arms	50	52.1	57.1
	3 arms	50	34.2	55.7
	4 arms	34	23.8	51.8
Citrus	2 arms	45	44.3	53.8
	3 arms	45	36.5	51.0
	4 arms	45	23.6	48.6

Table 5: Impact of arm count on GPT-5.5-Pro performance for scenes containing 5–10 apples.

n

denotes the number of evaluated scenes per configuration. Both detection and harvested ratios decrease as the number of arms increases.

Explore similar work

Visual-Language-Guided Task Planning for Horticultural Robots
Jan 17, 2026Jose Cuaran, Kendall Koe, Aditya Potnis +2Large Language Model-Based Robot Planning Vision-Language Grounding
The Siebel School of Computing and Data Science, University of Illinois, Urbana-Champaign · National Center for Supercomputing Applications at University of Illinois, Urbana-Champaign · The Department of Agricultural and Biological Engineering, University of Illinois, Urbana-Champaign
Robotic Strawberry Harvesting with Robust Vision and Deep Reinforcement Learning based Sim-to-Real Control
May 22, 2026Al Bashir, Shao-Yang Chang, Partho Ghose +3RL for Robotics Instance Segmentation
Department of Biological & Agricultural Engineering, Texas A&M University, College Station, TX 77843, USA · Department of Biomechatronics Engineering, National Taiwan University, 10617, Taipei City, Taiwan · Texas A&M AgriLife Research, Texas A&M University System, Dallas, TX 75252, USA
A Modular Dual-Arm Apple Harvesting Robot with Enhanced Field Performance
Jun 12, 2026Keyi Zhu, Kyle Lammers, Chaaran Arunachalam +3Bimanual Robotic Manipulation Agricultural Robotics
Department of Mechanical Engineering Michigan State University East Lansing, MI 48824, USA · Department of Electrical and Computer Engineering Michigan State University East Lansing, MI 48824, USA · United States Department of Agriculture Agricultural Research Service East Lansing, MI 48824, USA

Jan 17, 2026J/K move · Enter open · S save

Visual-Language-Guided Task Planning for Horticultural Robots

Jose Cuaran, Kendall Koe, Aditya Potnis, Naveen Kumar Uppalapati+1

The Siebel School of Computing and Data Science, University of Illinois, Urbana-Champaign · National Center for Supercomputing Applications at University of Illinois, Urbana-Champaign · The Department of Agricultural and Biological Engineering, University of Illinois, Urbana-Champaign

Large Language Model-Based Robot Planning Vision-Language Grounding Agricultural Robotics VLMs for Robotics Precision Agriculture

Open paper AskPDF arXiv

Crop monitoring is essential for precision agriculture, but current systems lack high-level reasoning. We introduce a novel, modular framework that uses a Vision Language Model (VLM) to guide robotic task planning by actively querying heterogeneous data sources, including enriched RGB camera feeds and 2D semantic occupancy maps, interleaved with robotic action primitives. We contribute a comprehensive benchmark for short- and long-horizon crop monitoring tasks in monoculture and polyculture environments. Our results show that while zero-shot VLMs perform robustly for short-horizon tasks (achieving 87% success, comparable to human experts), success drops significantly to under 10% for complex long-horizon, multi-target tasks. Despite this decline, task completion rates remain above 76% under noiseless conditions. Critically, the system degrades when relying on noisy semantic maps, demonstrating a key limitation in current VLM context grounding for sustained robotic operations. This work offers a deployable framework and critical insights into VLM capabilities and shortcomings for complex agricultural robotics.

Abstract

Explore similar work

Visual-Language-Guided Task Planning for Horticultural Robots

Robotic Strawberry Harvesting with Robust Vision and Deep Reinforcement Learning based Sim-to-Real Control

A Modular Dual-Arm Apple Harvesting Robot with Enhanced Field Performance