Targeted Modality Dropout for Real-Robot Manipulation Robust to Intermittent Vision Loss

Genki Shikada; Kazuki Osamura; Masaru Ide; Tetsuya Ogata; Kanata Suzuki

cs.ROOct 7, 2026

Targeted Modality Dropout for Real-Robot Manipulation Robust to Intermittent Vision Loss

Authors: Genki Shikada, Kazuki Osamura, Masaru Ide, Tetsuya Ogata, Kanata Suzuki

Organizations: Fujitsu Limited, Kanagawa 211-8588, Japan. · Faculty of Science and Engineering, Waseda University, Tokyo 169-8050, Japan. · National Institute of Advanced Industrial Science and Technology, Tokyo 100-8921, Japan.

Abstract

Imitation learning policies that integrate multiple sensory modalities are prone to overreliance on a dominant modality, such as vision, during training, which can disrupt policy execution when that modality is lost at inference time. In this paper, we introduce Targeted Modality Dropout (TMD), in which the dependence on each modality is estimated using attention and the most dominant modality is selectively dropped. This is combined with entropy regularization over the dependence distribution. Through real-robot evaluation using a bimanual manipulator, we show that under vision loss the success rate of the baseline policy drops substantially, whereas TMD sustains task execution. In contrast, a conventional dropout that selects the dropped modality at random, without the entropy regularization, fails on many tasks even without vision loss.

Figures & tables

Fig. 1: Overview of the problem and of the proposed method. Three modalities feed one policy. (a) Under ordinary training the dependence, estimated from attention, concentrates on vision. (b) TMD drops whichever modality is dominant during training, and the dependence balances. The bars are the estimated dependence at the end of training, with the vision bar left unfilled to mark what inference-time loss removes.

Fig. 2: Overview of the proposed method during training. Language and vision tokens are encoded by frozen pretrained encoders and integrated by a trainable adapter, while state tokens are projected and joined directly, so that the key/value memory of the action expert contains the tokens of all three modalities. Because the adapter attends across the language and vision tokens, each of its output tokens carries information from both, so the dependence on these two modalities is not measured on fully independent representations. The dependence $p_{t}$ drives two mechanisms: the tokens of $m^{\dagger}_{t-1}$ are zeroed in a subset of samples, and the entropy of $p_{t}$ enters the loss. In the step drawn here, vision is the target, so its token row is shaded and its tokens are dimmed. See Algorithm 1 for the order of operations.

Fig. 3: Top: modalities the policy receives from the bimanual manipulator CobotMagic: images from the head and the left- and right-hand cameras, taken at the moment when the slider of the fastener is grasped to close the zipper, and the motor currents. Bottom: six tasks used for real-world evaluation: bottle, close zipper, open zipper, towel folding, trash disposal, and unplug.

Fig. 4: Trajectory of the modality dependence

p_{t,m}

during training, for each condition. The three modalities are overlaid in each panel; curves are exponential moving averages. The dashed horizontal line marks the uniform level

1/M\simeq 0.33

, and the shaded band marks the interval in which the dropout probability

\rho_{t}

is nonzero (Baseline applies no dropout).

Fig. 5: Action delta at chunk arrival, in the normal setting and under vision loss, for each of the

30

trials per condition. Open and filled markers indicate the same trial, connected by a line, and horizontal bars are condition means. For both conditions, the action delta is larger when the chunk was generated under vision loss, with a smaller increase for TMD: the mean within-trial ratio of the two values, marked by

\times

, is

3.12

for Baseline versus

2.29

for TMD. Action delta is the sum over the two arms of the L2 norm of the change in the commanded action vector between adjacent steps; the action vector comprises the six joint angles and the gripper command and is reported in arbitrary units.

Task	Baseline	TMD	RMD
bottle	2/5	5/5	0/5
close zipper	3/5	0/5	0/5
open zipper	1/5	2/5	0/5
towel folding	4/5	3/5	1/5
trash disposal	1/5	2/5	0/5
unplug	3/5	3/5	3/5

TABLE I: Number of successes / number of trials on the real robot under the normal setting (

N=5

per condition).

Task	Baseline ( $\Delta$ )	TMD ( $\Delta$ )
bottle	1/5 ( $-1$ )	5/5 ( $0$ )
close zipper	0/5 ( $-3$ )	0/5 ( $0$ )
open zipper	0/5 ( $-1$ )	2/5 ( $0$ )
towel folding	1/5 ( $-3$ )	2/5 ( $-1$ )
trash disposal	0/5 ( $-1$ )	1/5 ( $-1$ )
unplug	0/5 ( $-3$ )	5/5 ( $+2$ )

TABLE II: Number of successes / number of trials on the real robot under vision loss (

N=5

per condition).

\Delta

is the change from the normal setting.

Explore similar work

Reinforcement Learning-Guided Retrieval with Soft Fusion for Robust Multimodal Imitation Learning under Missing Modalities
Jun 13, 2026Hassan Ismkhan, Hamid BouchahciaMissing-Modality Learning Multimodal Robustness
Bournemouth University Bournemouth, UK
GuidedAttention: Interpretable and Correctable Visual Attention for OOD-Robust Robot Manipulation via Imitation Learning
Jul 23, 2026Masaki Murooka, Ryoichi Nakajo, Keisuke Shirai +4Visual Attention Robot Skill Learning
CNRS-AIST JRL (Joint Robotics Laboratory), IRL, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8560, Japan · Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology (AIST), 2-3-26 Aomi, Koto-ku, Tokyo 135-0064, Japan
Sensing Which Modality Matters: Evidence-Gated Regularization for Robust VLA Policies
Sep 2, 2026Yue Yang, Diego Romeres, Chiori Hori +3Robustness of VLA Models Multimodal Sensor Fusion
Department of Computer Science, University of North Carolina at Chapel Hill · 2Mitsubishi Electric Research Laboratories (MERL)

Jun 13, 2026, cs.ROJ/K move · Enter open · S save

Reinforcement Learning-Guided Retrieval with Soft Fusion for Robust Multimodal Imitation Learning under Missing Modalities

Hassan Ismkhan, Hamid Bouchahcia

Bournemouth University Bournemouth, UK

Missing-Modality Learning Multimodal Robustness Imitation Learning Interactive Imitation Learning Robot Imitation Learning

Open paper AskPDF arXiv

Robotic systems perceive the world through multiple input modalities -- including visual camera streams and natural language instructions -- and must select appropriate actions based on these signals. However, assuming the permanent availability of all input devices is unrealistic, as sensors may fail, become occluded, or drop out entirely during deployment. Robust handling of such missing-modality scenarios is therefore essential for real-world robot operation. This paper introduces RL4IL, a reinforcement learning guided method for imitation learning that selects the most suitable action for a given observation by identifying the most relevant expert demonstrations from a training library. A reinforcement learning policy, trained via Proximal Policy Optimisation over Breadth-First Search candidate sets, ranks candidate demonstrations and a soft cross-attention fusion head aggregates their action signals to produce the final prediction. When a modality is missing at inference time, a dedicated per-modality RL retrieval policy identifies donor demonstrations from the training library, and a soft imputation head reconstructs the missing embedding via cross-attention over the top-ranked donors -- without requiring any retraining of the system. Experiments on three LIBERO benchmark suites demonstrate that RL4IL substantially outperforms state-of-the-art imitation learning methods under sensor dropout conditions, while requiring no policy network training. The code can be found at https://github.com/h-ismkhan/Reinforcement-Learning-via-kNN-for-Robotic-Learning-with-Missing-Camera

Abstract

Explore similar work

Reinforcement Learning-Guided Retrieval with Soft Fusion for Robust Multimodal Imitation Learning under Missing Modalities

GuidedAttention: Interpretable and Correctable Visual Attention for OOD-Robust Robot Manipulation via Imitation Learning

Sensing Which Modality Matters: Evidence-Gated Regularization for Robust VLA Policies