HarvestFlex: Strawberry Harvesting via Vision-Language-Action Policy Adaptation in the Wild

Ziyang Zhao; Shuheng Wang; Zhonghua Miao; Ya Xiong

cs.ROMar 6, 2026

HarvestFlex: Strawberry Harvesting via Vision-Language-Action Policy Adaptation in the Wild

Authors: Ziyang Zhao, Shuheng Wang, Zhonghua Miao, Ya Xiong

Organizations: The Intelligent Equipment Research Center, Beijing Academy of Agriculture and Forestry Sciences, Beijing 100097, China. · The School of Mechanical Electrical Engineering and Automation, Shanghai University, Shanghai 20044, China.

Abstract

This work presents the first study on transferring vision-language-action (VLA) policies to real greenhouse tabletop strawberry harvesting, a long-horizon, unstructured task challenged by occlusion and specular reflections. We built an end-to-end closed-loop system on the HarvestFlex platform using three-view RGB sensing (two fixed scene views plus a wrist-mounted view) and intentionally avoided depth clouds and explicit geometric calibration. We collected 3.71 h of VR teleoperated demonstrations (227 episodes) and fine-tuned pi_0, pi_0.5, and WALL-OSS with full fine-tuning and LoRA. Under a unified 50 trials real-greenhouse protocol and metrics spanning completion, pi_0.5 with full fine-tuning achieved success rate of 74.0% with 32.6 s/pick and damage rate of 4.1%. Asynchronous inference-control decoupling further improved performance over synchronous deployment. Results showed non-trivial closed-loop picking with fewer than four hours of real data, while remaining limited by close-range observability loss and contact-dynamics mismatch. A demonstration video is available at: https://youtu.be/bN8ZowZKPMI.

Figures & tables

Fig. 1: HarvestFlex end-to-end closed-loop harvesting pipeline: three-view RGB observations and robot state, conditioned on a language goal, were mapped by a VLA policy to 8-D actions for real-world execution.

Fig. 2: Real-world strawberry picking scene and camera configuration.

Panel of Figure 3 — Fig. 3: Example three-view observations in our dataset: (a) left scene camera (D455), (b) wrist camera (D405), and (c) right scene camera (D455).

Camera	Model	RGB (W $\times$ H) @ fps	Exposure / WB
Left scene	D455	1280 $\times$ 720 @ 30	Auto / Auto
Right scene	D455	1280 $\times$ 720 @ 30	Auto / Auto
Wrist	D405	640 $\times$ 480 @ 30	Auto / Auto

TABLE I: Camera acquisition settings and coverage.

Panel of Figure 5 — Fig. 4: VR teleoperation interface and single-operator control scheme. (a) Schematic of the third-person viewpoint used to illustrate the teleoperation setup. (b) First-person view from the operator’s headset recorded during demonstration collection.

Fig. 5: VR controller button map for one-person operation.

Factor	Category	Ratio (% episodes)
Illumination	Low light	18
	Normal	62
	Strong specular	20
Occlusion	Mild	60
	Moderate	25
	Heavy	15

TABLE II: Dataset coverage ratios by key factors (episode-level).

Method	$\pi_{0}$		$\pi_{0.5}$		wall-oss
Method	loss	time	loss	time	loss	time
lora	0.00241	39h52m12s	0.00329	43h18m04s	0.00211	42h46m29s
full	0.02702	42h47m53s	0.02981	45h16m38s	0.01758	47h32m51s

TABLE III: Loss and training time of LoRA vs. full fine-tuning on different models.

Fig. 6: Flow diagram showing the processing pipelines for synchronous (blocking, sequential) and asynchronous (non-blocking, concurrent) inference modes.

Fine-tuning	Metrics	PI0			wall-oss			PI05			Best
Fine-tuning	Metrics	2 (ep)	4 (ep)	6 (ep)	2 (ep)	4 (ep)	6 (ep)	2 (ep)	4 (ep)	6 (ep)
Full	SS (score)	20.4	46.6	72.6	26.4	52.4	78.8	30.8	58.4	82.6	82.6
	SR (%)	14.0	38.0	60.0	24.0	44.0	68.0	30.0	50.0	74.0	74.0
	Cycle time (s)	53.6	42.1	38.4	53.3	52.8	46.3	44.2	40.7	32.6	32.6
	DR (%)	5.2	4.1	4.2	3.5	4.0	3.9	4.4	3.8	4.1	3.5
LoRA	SS (score)	18.6	42.4	68.2	22.8	48.2	72.8	26.8	50.4	73.6	73.6

TABLE IV: Metrics for different models under different fine-tuning strategies and training levels; 2, 4, and 6 denote the number of training epochs.

Panel of Figure 11 — Fig. 7: Key frames showing representative successful, failed and retry cases.

Metric	Sync	Async
SS (score)	76.2	82.6
SR (%)	70.0	74.0
Cycle time (s)	45.7	32.6
DR (%)	3.8	4.1

TABLE V: Comparison of synchronous and asynchronous inference (

\pi_{0.5}

Metric	Left/Right	Left+Right	All
SS (score)	16.4	42.4	82.6
SR (%)	10.0	42.0	74.0
Cycle time (s)	75.3	61.3	32.6
DR (%)	5.5	4.3	4.1

TABLE VI: Impact of camera-view configurations on performance.

Metrics	Conv	VLA
SS (score)	91.8	82.6
SR (%)	89.0	74.0
Cycle time (s)	8.3	32.6
DR (%)	3.8	4.1

TABLE VII: Comparison between conventional (Conv) and VLA pipelines.

Explore similar work

Robotic Strawberry Harvesting with Robust Vision and Deep Reinforcement Learning based Sim-to-Real Control
May 22, 2026Al Bashir, Shao-Yang Chang, Partho Ghose +3RL for Robotics Instance Segmentation
Department of Biological & Agricultural Engineering, Texas A&M University, College Station, TX 77843, USA · Department of Biomechatronics Engineering, National Taiwan University, 10617, Taipei City, Taiwan · Texas A&M AgriLife Research, Texas A&M University System, Dallas, TX 75252, USA
Vision-Based Obstacle Separation for Strawberry Harvesting in Clusters Using Hierarchical Reinforcement Learning
Jul 15, 2026Teng Li, Hanfei Shi, Chunjiang Zhao +1Robotic Grasping Hierarchical RL
The Intelligent Equipment Research Center, Beijing Academy of Agriculture and Forestry Sciences, Beijing 100097, China. · School of Intelligence Science and Technology, University of Science and Technology Beijing, Beijing 100083, China. · Happy Elements Ltd., Beijing 100094, China.
Reinforcement Learning for the Full Strawberry Harvesting Process: Obstacle Separation, Detachment, and Placement
Jul 16, 2026Changyou Miao, Teng Li, Ya XiongRL for Robotics Long-Horizon Robotic Manipulation
The Intelligent Equipment Research Center, Beijing Academy of Agriculture and Forestry Sciences, Beijing 100097, China. · College of Engineering, Huazhong Agricultural University, Wuhan 430070, China

May 22, 2026J/K move · Enter open · S save

Robotic Strawberry Harvesting with Robust Vision and Deep Reinforcement Learning based Sim-to-Real Control

Al Bashir, Shao-Yang Chang, Partho Ghose, Prem Raj+2

Department of Biological & Agricultural Engineering, Texas A&M University, College Station, TX 77843, USA · Department of Biomechatronics Engineering, National Taiwan University, 10617, Taipei City, Taiwan · Texas A&M AgriLife Research, Texas A&M University System, Dallas, TX 75252, USA

RL for Robotics Instance Segmentation Agricultural Robotics

Open paper AskPDF arXiv

This study presents a closed-loop robotic strawberry harvesting system that combines a robust vision module, simulation-trained deep reinforcement learning (DRL) control, and ROS-based realrobot execution. For perception, we propose HRAttnEdge-YOLO26-seg, a modified YOLO26-seg architecture that incorporates a high-resolution P2 branch, segmentation-path attention, and edgesupervised prototype learning to improve instance segmentation in cluttered scenes. For control, we train a target-conditioned Proximal Policy Optimization (PPO) policy in Isaac Lab to produce smooth joint-position commands for a UR10e manipulator and deploy it on a UR10e robot for targetfruit reaching and harvesting. This simulation-based approach reduces hardware dependency, lowers development cost, and allows scalable policy training without exhaustive physical trials before real deployment. The proposed vision model demonstrated the highest overall performance among the evaluated methods. On both self-collected and public datasets, the model showed a 10 to 14% improvement in segmentation performance. In controlled in-house tests, the PPO controller produced stable and dynamically smoother motion than a inverse kinematics (IK)-based MoveIt baseline. In greenhouse trials, the proposed integrated system harvested 281 strawberries, achieving 96.6% reaching success, 91.3% grasp-and-pull success, and 84.3% overall harvesting success. These results illustrate that task-specific perception combined with simulation-trained PPO can serve as a practical and resource-efficient alternative to conventional planner-dependent reaching in manipulation, enabling reliable closed-loop robotic harvesting in complex agricultural environments.

Abstract

Explore similar work

Robotic Strawberry Harvesting with Robust Vision and Deep Reinforcement Learning based Sim-to-Real Control

Vision-Based Obstacle Separation for Strawberry Harvesting in Clusters Using Hierarchical Reinforcement Learning

Reinforcement Learning for the Full Strawberry Harvesting Process: Obstacle Separation, Detachment, and Placement