Organizations: Division of Information Science, Graduate School of Information Science, Nara Institute of Science and Technology (NAIST), Nara, Japan · DX · IT · Research & Development Center, TSUBAKIMOTO CHAIN CO., Kyoto, Japan
Vision-based imitation learning has shown promise for robotic manipulation; however, its generalization remains limited in practical agricultural tasks. This limitation stems from scarce demonstration data and substantial visual domain gaps caused by i) crop-specific appearance diversity and ii) background variations. To address this limitation, we propose Dual-Region Augmentation for Imitation Learning (DRAIL), a region-aware augmentation framework designed for generalizable vision-based imitation learning in agricultural manipulation. DRAIL explicitly separates visual observations into task-relevant and task-irrelevant regions. The task-relevant region is augmented in a domain-knowledge-driven manner to preserve essential visual characteristics, while the task-irrelevant region is aggressively randomized to suppress spurious background correlations. By jointly handling both sources of visual variation, DRAIL promotes learning policies that rely on task-essential features rather than incidental visual cues. We evaluate DRAIL on diffusion policy-based visuomotor controllers through robot experiments on artificial vegetable harvesting and real lettuce defective leaf picking preparation tasks. The results show consistent improvements in success rates under unseen visual conditions compared to baseline methods. Further attention analysis and representation generalization metrics indicate that the learned policies rely more on task-essential visual features, resulting in enhanced robustness and generalization.
Figures & tables
Fig. 1 : Overview of an agricultural manipulation and the importance of visual attention of visuomotor policies. (a) Robot automation system for a defective leaf removal task, the visuomotor policy must select appropriate actions from visual observations. (b) Fragile visuomotor policies tend to attend to various regions in the visual observation, whereas robust policies focus their attention on the defective leaf.
Fig. 2 : Overview of Dual-Region Augmentation for Imitation Learning (DRAIL). (a) shows task-relevant segmentation. A mask of the task-relevant region is initialized on the first frame of visual observations using SAM and then propagated to subsequent frames using XMem++, yielding per-frame masks. (b) shows task-relevant augmentation. The task-relevant region undergoes task-specific augmentation. (c) shows task-irrelevant randomization. The task-irrelevant region is perturbed using PixMix. The two streams are composited to form augmented data, repeated across all demonstration episodes.
Fig. 3 : Robot hardware used in experiments.
Fig. 4 : Task environments for the tomato harvesting task. (a) shows a demonstration environment. “G1” and “G2” mean two goals on the task. (b) shows a test environment. The test environment has distractors, and the color of the tomatoes changes.
Fig. 5 : Task environments for the carrot harvesting task (a) shows a demonstration environment. “G1” and “G2” mean two goals on the task. (b) shows a test environment. The test environment has distractors and another carrot on the pot.
Fig. 6 : Data augmentation in the tomato harvesting task. (a) shows visual observation. The red line indicates the borderline of task-relevant and task-irrelevant regions. (b) shows task-relevant augmented visual observation. (c) shows task-irrelevant augmented visual observation. (d) shows dual augmented visual observation.
Fig. 7 : Data augmentation in the carrot harvesting task. (a) shows visual observation. The red line indicates the borderline of task-relevant and task-irrelevant regions. (b) shows task-relevant augmented visual observation. (c) shows task-irrelevant augmented visual observation. (d) shows dual augmented visual observation.
Fig. 8 : Robot motion using policy learned by DRAIL and ablation methods in the tomato harvesting task
Fig. 9 : Robot motion using policy learned by DRAIL and ablation methods in the carrot harvesting task
Env.
Goal
DRAIL
DRAIL
DRAIL
DRAIL
w/o task-irr.
w/o task-rel.
w/o dual
Demo
G1
100%
100%
100%
100%
Demo
G2
100%
100%
100%
100%
Test
G1
100%
70%
0%
0%
Test
G2
100%
90%
0%
0%
TABLE I : Success rates in the tomato harvesting task. 10 trials were performed for each condition.
Fig. 10 : Visualization of the attention region of the learned policy using the saliency map in the tomato harvesting task. In the visual observation shown in (e), the red line highlights the important region that the policy should attend to for the task.
Fig. 11 : Visualization of the attention region of the learned policy by the saliency map in the carrot harvesting task. In the visual observation shown in (e), the red line highlights the important region that the policy should attend to for the task.
DRAIL
DRAIL
DRAIL
DRAIL
ARG
w/o task-irr.
w/o task-rel.
w/o dual
Mean
7.8×103
2.5×104†
2.4×106†
1.2×106†
STD
6.0×103
2.1×104
4.9×105
3.0×105
TABLE III : Comparison of ARGs of policies learned by each method in the tomato harvesting task. For each method, ARG was recomputed 20 times with different random samplings of the fixed network. Statistical significance was assessed using a paired t-test against DRAIL. “ † ” indicates a statistically significant difference with p<0.05 .
DRAIL
DRAIL
DRAIL
DRAIL
ARG
w/o task-irr.
w/o task-rel.
w/o dual
Mean
1.0×104
7.2×104†
1.1×105†
4.5×105†
STD
7.3×103
2.3×104
3.3×104
8.6×104
TABLE IV : Comparison of ARGs of policies learned by each method in the carrot harvesting task. For each method, ARG was recomputed 20 times with different random samplings of the fixed network. Statistical significance was assessed using a paired t-test against DRAIL. “ † ” indicates a statistically significant difference with p<0.05 .
Fig. 12 : Overview of the real lettuce defective leaf picking preparation task. Given a top-down wrist-camera observation, the policy drives the gripper toward the leaf base of the most defective leaf among the front-side leaves. The target leaf is outlined in red, and the desired grasp point is its leaf base.
Fig. 13 : Data augmentation in the real lettuce defective leaf picking preparation task. (a) shows visual observation. The red line indicates the borderline of task-relevant and task-irrelevant regions. (b) shows task-relevant augmented visual observation. (c) shows task-irrelevant augmented visual observation. (d) shows dual augmented visual observation.
Fig. 14 : Robot motion by learned policy with each method in the real lettuce defective leaf picking preparation task
DRAIL
DRAIL
DRAIL
DRAIL
w/o task-irr.
w/o task-rel.
w/o dual
Leaf
80%
67%
53%
47%
selection
Positional
73%
47%
40%
40%
alignment
TABLE V : Success rates in the real lettuce defective leaf picking preparation task. 15 trials were performed.
Fig. 15 : Visualization of the attention region of the learned policy by the saliency map in the real lettuce defective leaf picking preparation task. In the visual observation shown in (e), the red line highlights the important region that the policy should attend to for the task.
DRAIL
DRAIL
DRAIL
DRAIL
ARG
w/o task-irr.
w/o task-rel.
w/o dual
Mean
8.8×103
1.5×104
9.7×104†
1.7×106†
STD
5.8×103
1.3×104
1.9×104
8.9×105
TABLE VI : Comparison of ARGs of policies learned by each method in the real lettuce defective leaf picking preparation task. For each method, ARG was recomputed 20 times with different random samplings of the fixed network. Statistical significance was assessed using a paired t-test against DRAIL. “ † ” indicates a statistically significant difference with p<0.05 .
In vision-based behavior cloning (BC), conventional image augmentations such as Random Crop and Color Jitter often fall short under substantial visual domain shifts, including changes in shadows, distractors, and backgrounds. Superimposition-based augmentations, which blend in-domain and out-of-domain images, have shown promise for improving generalization in computer vision, but their suitability for BC remains uncertain because task-critical semantics, spatiotemporal relationships, and agent-target interactions must be preserved. To address this, we introduce RoboSaGA, a Saliency-Guided Augmentation method within the superimposition family tailored for vision-based BC. RoboSaGA dynamically adjusts augmentation intensity at the pixel level using policy-driven saliency, enabling aggressive augmentation in task-irrelevant regions while preserving task-critical information. It integrates seamlessly into existing architectures without requiring structural modifications or additional learning objectives. Experiments in both simulated and real-world settings show that RoboSaGA preserves in-domain performance while substantially improving robustness to visual domain shifts, including distractor and background changes, as well as lighting and shadow variations. Code is available at https://github.com/Zheyu-Zhuang/RoboSaGA.
Zheyu Zhuang, Ruiyu Wang, Nils Ingelhag +2
KTH Royal Institute of Technology · Aalto University
Enhancing the generalization of robotic learning in diverse unseen environments remains a fundamental challenge. Existing approaches often rely on large-scale pretraining, which is labor-intensive and time-consuming, or semantic data augmentation methods that assume flawless upstream object detection in real-world scenarios. In this work, we propose RoboAug, a novel generative data augmentation framework that reduces reliance on large-scale pretraining and perfect visual recognition by requiring only a single image with bounding box annotations for dataset construction. Leveraging this minimal supervision, RoboAug employs pretrained generative models for precise semantic augmentation and introduces a plug-and-play region-contrastive loss to guide attention toward task-relevant regions, thereby enhancing generalization and task success rates. Extensive real-world experiments on UR-5e, AgileX, and Tian Gong 2.0 demonstrate that RoboAug consistently outperforms state-of-the-art augmentation baselines under background, distractor, and lighting shifts. Our project is available at https://x-roboaug.github.io/.
Xinhua Wang, Kun Wu, Zhen Zhao +10
Beijing Innovation Center of Humanoid Robotics · Computation, Information and Technology, Technical University of Munich · City University of Hong Kong +3
Visual imitation learning is a promising approach to training robot manipulation policies capable of completing a wide variety of tasks. However, policies today remain brittle to viewpoint perturbations, making deployment in diverse environments a challenge. We present a controlled empirical study of which design choices allow visuomotor policies to generalize across viewpoints. We find that viewpoint generalization improves when dense visual tokens are retained and the action head participates in geometric reasoning. On a suite of simulated tasks that span a wide range of camera poses, we show that these design choices yield a policy that remains performant across viewpoints. As a practical consequence, a policy trained with these design choices also transfers zero-shot from simulation to the real world under random camera configurations.
Mino Nakura, Sriram Krishna, Yufei Wang +3
Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA.