Task-Relevant and Irrelevant Region-Aware Augmentation for Generalizable Vision-Based Imitation Learning in Agricultural Manipulation

Shun Hattori; Hikaru Sasaki; Takumi Hachimine; Yusuke Mizutani; Takamitsu Matsubara

cs.ROMar 5, 2026

Task-Relevant and Irrelevant Region-Aware Augmentation for Generalizable Vision-Based Imitation Learning in Agricultural Manipulation

Authors: Shun Hattori, Hikaru Sasaki, Takumi Hachimine, Yusuke Mizutani, Takamitsu Matsubara

Organizations: Division of Information Science, Graduate School of Information Science, Nara Institute of Science and Technology (NAIST), Nara, Japan · DX · IT · Research & Development Center, TSUBAKIMOTO CHAIN CO., Kyoto, Japan

Abstract

Vision-based imitation learning has shown promise for robotic manipulation; however, its generalization remains limited in practical agricultural tasks. This limitation stems from scarce demonstration data and substantial visual domain gaps caused by i) crop-specific appearance diversity and ii) background variations. To address this limitation, we propose Dual-Region Augmentation for Imitation Learning (DRAIL), a region-aware augmentation framework designed for generalizable vision-based imitation learning in agricultural manipulation. DRAIL explicitly separates visual observations into task-relevant and task-irrelevant regions. The task-relevant region is augmented in a domain-knowledge-driven manner to preserve essential visual characteristics, while the task-irrelevant region is aggressively randomized to suppress spurious background correlations. By jointly handling both sources of visual variation, DRAIL promotes learning policies that rely on task-essential features rather than incidental visual cues. We evaluate DRAIL on diffusion policy-based visuomotor controllers through robot experiments on artificial vegetable harvesting and real lettuce defective leaf picking preparation tasks. The results show consistent improvements in success rates under unseen visual conditions compared to baseline methods. Further attention analysis and representation generalization metrics indicate that the learned policies rely more on task-essential visual features, resulting in enhanced robustness and generalization.

Figures & tables

Panel of Figure 1 — Fig. 1 : Overview of an agricultural manipulation and the importance of visual attention of visuomotor policies. (a) Robot automation system for a defective leaf removal task, the visuomotor policy must select appropriate actions from visual observations. (b) Fragile visuomotor policies tend to attend to various regions in the visual observation, whereas robust policies focus their attention on the defective leaf.

Fig. 2 : Overview of Dual-Region Augmentation for Imitation Learning (DRAIL). (a) shows task-relevant segmentation. A mask of the task-relevant region is initialized on the first frame of visual observations using SAM and then propagated to subsequent frames using XMem++, yielding per-frame masks. (b) shows task-relevant augmentation. The task-relevant region undergoes task-specific augmentation. (c) shows task-irrelevant randomization. The task-irrelevant region is perturbed using PixMix. The two streams are composited to form augmented data, repeated across all demonstration episodes.

Panel of Figure 3 — Fig. 3 : Robot hardware used in experiments.

Panel of Figure 4 — Fig. 4 : Task environments for the tomato harvesting task. (a) shows a demonstration environment. “G1” and “G2” mean two goals on the task. (b) shows a test environment. The test environment has distractors, and the color of the tomatoes changes.

Panel of Figure 5 — Fig. 5 : Task environments for the carrot harvesting task (a) shows a demonstration environment. “G1” and “G2” mean two goals on the task. (b) shows a test environment. The test environment has distractors and another carrot on the pot.

Panel of Figure 6 — Fig. 6 : Data augmentation in the tomato harvesting task. (a) shows visual observation. The red line indicates the borderline of task-relevant and task-irrelevant regions. (b) shows task-relevant augmented visual observation. (c) shows task-irrelevant augmented visual observation. (d) shows dual augmented visual observation.

Panel of Figure 7 — Fig. 7 : Data augmentation in the carrot harvesting task. (a) shows visual observation. The red line indicates the borderline of task-relevant and task-irrelevant regions. (b) shows task-relevant augmented visual observation. (c) shows task-irrelevant augmented visual observation. (d) shows dual augmented visual observation.

Panel of Figure 8 — Fig. 8 : Robot motion using policy learned by DRAIL and ablation methods in the tomato harvesting task

Panel of Figure 9 — Fig. 9 : Robot motion using policy learned by DRAIL and ablation methods in the carrot harvesting task

Env.	Goal	DRAIL	DRAIL	DRAIL	DRAIL
			w/o task-irr.	w/o task-rel.	w/o dual
Demo	G1	100%	100%	100%	100%
Demo	G2	100%	100%	100%	100%
Test	G1	100%	70%	0%	0%
Test	G2	100%	90%	0%	0%

TABLE I : Success rates in the tomato harvesting task. 10 trials were performed for each condition.

Panel of Figure 11 — Fig. 10 : Visualization of the attention region of the learned policy using the saliency map in the tomato harvesting task. In the visual observation shown in (e), the red line highlights the important region that the policy should attend to for the task.

Panel of Figure 12 — Fig. 11 : Visualization of the attention region of the learned policy by the saliency map in the carrot harvesting task. In the visual observation shown in (e), the red line highlights the important region that the policy should attend to for the task.

	DRAIL	DRAIL	DRAIL	DRAIL
ARG		w/o task-irr.	w/o task-rel.	w/o dual
Mean	$\mathbf{7.8\times 10^{3}}$	$2.5\times 10^{4\dagger}$	$2.4\times 10^{6\dagger}$	$1.2\times 10^{6\dagger}$
STD	$\mathbf{6.0\times 10^{3}}$	$2.1\times 10^{4}$	$4.9\times 10^{5}$	$3.0\times 10^{5}$

TABLE III : Comparison of ARGs of policies learned by each method in the tomato harvesting task. For each method, ARG was recomputed 20 times with different random samplings of the fixed network. Statistical significance was assessed using a paired t-test against DRAIL. “

\dagger

” indicates a statistically significant difference with

p<0.05

	DRAIL	DRAIL	DRAIL	DRAIL
ARG		w/o task-irr.	w/o task-rel.	w/o dual
Mean	$\mathbf{1.0\times 10^{4}}$	$7.2\times 10^{4\dagger}$	$1.1\times 10^{5\dagger}$	$4.5\times 10^{5\dagger}$
STD	$\mathbf{7.3\times 10^{3}}$	$2.3\times 10^{4}$	$3.3\times 10^{4}$	$8.6\times 10^{4}$

TABLE IV : Comparison of ARGs of policies learned by each method in the carrot harvesting task. For each method, ARG was recomputed 20 times with different random samplings of the fixed network. Statistical significance was assessed using a paired t-test against DRAIL. “

\dagger

” indicates a statistically significant difference with

p<0.05

Fig. 12 : Overview of the real lettuce defective leaf picking preparation task. Given a top-down wrist-camera observation, the policy drives the gripper toward the leaf base of the most defective leaf among the front-side leaves. The target leaf is outlined in red, and the desired grasp point is its leaf base.

Panel of Figure 16 — Fig. 13 : Data augmentation in the real lettuce defective leaf picking preparation task. (a) shows visual observation. The red line indicates the borderline of task-relevant and task-irrelevant regions. (b) shows task-relevant augmented visual observation. (c) shows task-irrelevant augmented visual observation. (d) shows dual augmented visual observation.

Panel of Figure 17 — Fig. 14 : Robot motion by learned policy with each method in the real lettuce defective leaf picking preparation task

	DRAIL	DRAIL	DRAIL	DRAIL
		w/o task-irr.	w/o task-rel.	w/o dual
Leaf	80%	67%	53%	47%
selection	80%	67%	53%	47%
Positional	73%	47%	40%	40%
alignment	73%	47%	40%	40%

TABLE V : Success rates in the real lettuce defective leaf picking preparation task. 15 trials were performed.

Panel of Figure 19 — Fig. 15 : Visualization of the attention region of the learned policy by the saliency map in the real lettuce defective leaf picking preparation task. In the visual observation shown in (e), the red line highlights the important region that the policy should attend to for the task.

	DRAIL	DRAIL	DRAIL	DRAIL
ARG		w/o task-irr.	w/o task-rel.	w/o dual
Mean	$\mathbf{8.8\times 10^{3}}$	$1.5\times 10^{4}$	$9.7\times 10^{4\dagger}$	$1.7\times 10^{6\dagger}$
STD	$\mathbf{5.8\times 10^{3}}$	$1.3\times 10^{4}$	$1.9\times 10^{4}$	$8.9\times 10^{5}$

TABLE VI : Comparison of ARGs of policies learned by each method in the real lettuce defective leaf picking preparation task. For each method, ARG was recomputed 20 times with different random samplings of the fixed network. Statistical significance was assessed using a paired t-test against DRAIL. “

\dagger

” indicates a statistically significant difference with

p<0.05

Explore similar work

Enhancing Visual Domain Robustness in Behaviour Cloning via Saliency-Guided Augmentation
Aug 12, 2026Zheyu Zhuang, Ruiyu Wang, Nils Ingelhag +2Data Augmentation Domain Generalization
KTH Royal Institute of Technology · Aalto University
RoboAug: One Annotation to Hundreds of Scenes via Region-Contrastive Data Augmentation for Robotic Manipulation
Feb 15, 2026Xinhua Wang, Kun Wu, Zhen Zhao +10Synthetic Data Augmentation Generalization in Robotic Manipulation
Beijing Innovation Center of Humanoid Robotics · Computation, Information and Technology, Technical University of Munich · City University of Hong Kong +3
An Empirical Study on What Matters for Viewpoint-Generalizable Policies in Visual Imitation Learning
Sep 26, 2026Mino Nakura, Sriram Krishna, Yufei Wang +3Robot Imitation Learning Generalization in Robotic Manipulation
Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA.

Aug 12, 2026J/K move · Enter open · S save

Enhancing Visual Domain Robustness in Behaviour Cloning via Saliency-Guided Augmentation

Zheyu Zhuang, Ruiyu Wang, Nils Ingelhag, Ville Kyrki+1

KTH Royal Institute of Technology · Aalto University

Data Augmentation Domain Generalization Behavior Cloning Robot Policy Generalization

Open paper AskPDF arXiv

In vision-based behavior cloning (BC), conventional image augmentations such as Random Crop and Color Jitter often fall short under substantial visual domain shifts, including changes in shadows, distractors, and backgrounds. Superimposition-based augmentations, which blend in-domain and out-of-domain images, have shown promise for improving generalization in computer vision, but their suitability for BC remains uncertain because task-critical semantics, spatiotemporal relationships, and agent-target interactions must be preserved. To address this, we introduce RoboSaGA, a Saliency-Guided Augmentation method within the superimposition family tailored for vision-based BC. RoboSaGA dynamically adjusts augmentation intensity at the pixel level using policy-driven saliency, enabling aggressive augmentation in task-irrelevant regions while preserving task-critical information. It integrates seamlessly into existing architectures without requiring structural modifications or additional learning objectives. Experiments in both simulated and real-world settings show that RoboSaGA preserves in-domain performance while substantially improving robustness to visual domain shifts, including distractor and background changes, as well as lighting and shadow variations. Code is available at https://github.com/Zheyu-Zhuang/RoboSaGA.

Abstract

Explore similar work

Enhancing Visual Domain Robustness in Behaviour Cloning via Saliency-Guided Augmentation

RoboAug: One Annotation to Hundreds of Scenes via Region-Contrastive Data Augmentation for Robotic Manipulation

An Empirical Study on What Matters for Viewpoint-Generalizable Policies in Visual Imitation Learning