Organizations: Department of Electrical and Computer Engineering, The University of Memphis, Memphis, TN, USA · Department of Mechanical Engineering, Clemson University, Clemson, SC 29634 USA
Robotic manipulation in agricultural environments remains challenging due to variations in plant geometry, occlusion, and camera viewpoints. Pretrained vision-language-action (VLA) and generalist robot policies offer a promising approach to addressing these challenges by learning manipulation behaviors from multimodal observations and language instructions. However, adapting these policies to agricultural tasks requires domain-specific demonstrations, which are costly and time-consuming to collect on physical robots. In this work, we develop a simulation-to-real robot-learning framework for agricultural reaching and grasping that integrates task-specific demonstration collection and pretrained policy adaptation. A MuJoCo-based agricultural environment is developed with diverse tomato plant structures to support demonstration collection and controlled evaluation. Expert demonstrations are collected through joystick teleoperation, with a low-level controller used to execute robot actions. Two pretrained generalist robot policies, Octo and π0.5, are adapted using simulated demonstrations, followed by further fine-tuning on a physical Franka Research 3 manipulator using real-robot demonstrations. The adapted policies are evaluated under in-distribution, viewpoint/spatial-shift, and out-of-distribution conditions in simulation, followed by physical-robot experiments. The influence of proprioceptive information on policy performance is also investigated. Results show that π0.5 achieves higher task-performance scores than Octo and exhibits less degradation under distribution shifts. These findings demonstrate the potential of integrating pretrained generalist policies with task-specific demonstration learning for agricultural manipulation.
Figures & tables
Fig. 1: General structure of end-to-end robot learning: demonstrating the desired task to the robot, fine-tuning or training a policy, and deploying the policy for task execution.
Fig. 2: General structure of Vision Language Action models
Fig. 3: Nine different tomato plants are used to increase the variability and realism of the agricultural environment.
Fig. 4: Simulated mobile manipulator: a 7-DoF FR3 arm mounted on a Husky A200 base, with the placement and views of the primary and wrist cameras.
Fig. 5: Real-robot experimental setup used for demonstration collection and policy evaluation. The FR3 manipulator operates in front of an artificial tomato plant using a primary third-person camera and a wrist-mounted camera.
Fig. 6: Low-level controller structure.
Fig. 7: Task executions in simulation and on the physical robot. The first two rows show the progression of the reaching-and-grasping task, while the last two rows show the corresponding primary and wrist camera observations during execution.
Fig. 8: Octo and π0.5 performance when proprio is also included in observation based on three scenarios.
Fig. 9: Octo and π0.5 performance without proprioception as observation based on three scenarios.
Fig. 10: Average performance of Octo and π0.5 with and without proprioception as observation.
Fig. 11: Performace of Octo and π0.5 with and without proprioception as observation for real task.
Agricultural robotics develops autonomous machines that sense, navigate, and perform tasks such as monitoring, weeding, spraying, and harvesting crops.
We demonstrate that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training. We introduce Agent as Policy (AGP), which places task planning and execution under the agent's control. Given a task and a robot interface, the agent interprets visual evidence, writes executable programs, issues motion commands, and revises its actions in response to physical outcomes. This brings the agent's reasoning and programming capabilities into continuous interaction with the physical world. We study AGP across multiple real-world manipulation tasks spanning precision manipulation, dynamic motions, and deformable objects. These include assembly from human videos, block construction from goal images, dice flipping, targeted throwing, and bimanual towel folding. AGP achieves success rates of at least 80% in seven of eight task configurations and significantly outperforms previous agentic robot systems. We further study efficiency through task experience accumulation and find that reusing saved procedures and programs shortens execution time across repeated trials. These findings support a path for general-purpose agents to act as robot policies, extending their autonomy to physical manipulation through runtime reasoning, programming, and interaction.
Mengzhao Jia, Yang Lin, Xixin Zhang +3
University of Notre Dame · University of California San Diego · San Diego State University
Learning a generalist control policy for robotic manipulation typically relies on large-scale datasets. Given the high cost of real-world data collection, a practical alternative is to generate synthetic data through simulation. However, the resulting synthetic data often exhibits a significant gap from real-world distributions. While many prior studies have proposed algorithms to bridge the Sim-to-Real discrepancy, there remains a lack of principled research that grounds these methods in real-world manipulation tasks, particularly their performance on generalist policies such as Vision-Language-Action (VLA) models. In this study, we empirically examine the primary determinants of Sim-to-Real generalization across four dimensions: multi-level domain randomization, photorealistic rendering, physics-realistic modeling, and reinforcement learning updates. To support this study, we design a comprehensive evaluation protocol to quantify the real-world performance of manipulation tasks. The protocol accounts for key variations in background, lighting, distractors, object types, and spatial features. Through experiments involving over 10k real-world trials, we derive critical insights into Sim-to-Real transfer. To inform and advance future studies, we release both the robotic platforms and the evaluation protocol for public access to facilitate independent verification, thereby establishing a realistic and standardized benchmark for robotic manipulation policies.
Ruixing Jin, Zicheng Zhu, Ruixiang Ouyang +4
School of Data Science, The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute
Vision-based imitation learning has shown promise for robotic manipulation; however, its generalization remains limited in practical agricultural tasks. This limitation stems from scarce demonstration data and substantial visual domain gaps caused by i) crop-specific appearance diversity and ii) background variations. To address this limitation, we propose Dual-Region Augmentation for Imitation Learning (DRAIL), a region-aware augmentation framework designed for generalizable vision-based imitation learning in agricultural manipulation. DRAIL explicitly separates visual observations into task-relevant and task-irrelevant regions. The task-relevant region is augmented in a domain-knowledge-driven manner to preserve essential visual characteristics, while the task-irrelevant region is aggressively randomized to suppress spurious background correlations. By jointly handling both sources of visual variation, DRAIL promotes learning policies that rely on task-essential features rather than incidental visual cues. We evaluate DRAIL on diffusion policy-based visuomotor controllers through robot experiments on artificial vegetable harvesting and real lettuce defective leaf picking preparation tasks. The results show consistent improvements in success rates under unseen visual conditions compared to baseline methods. Further attention analysis and representation generalization metrics indicate that the learned policies rely more on task-essential visual features, resulting in enhanced robustness and generalization.
Shun Hattori, Hikaru Sasaki, Takumi Hachimine +2
Division of Information Science, Graduate School of Information Science, Nara Institute of Science and Technology (NAIST), Nara, Japan · DX · IT · Research & Development Center, TSUBAKIMOTO CHAIN CO., Kyoto, Japan