Cross-Modal Reinforcement Learning for Navigation with Degraded Depth Measurements

Omkar Sawant; Luca Zanatta; Grzegorz Malczyk; Kostas Alexis

cs.ROMar 23, 2026

Cross-Modal Reinforcement Learning for Navigation with Degraded Depth Measurements

Authors: Omkar Sawant, Luca Zanatta, Grzegorz Malczyk, Kostas Alexis

Organizations: Department of Engineering Cybernetics, Norwegian University of Science and Technology (NTNU), Norway

Abstract

This paper presents a cross-modal learning framework that exploits complementary information from depth and grayscale images for robust navigation. We introduce a Cross-Modal Wasserstein Autoencoder that learns shared latent representations by enforcing cross-modal consistency, enabling the system to infer depth-relevant features from grayscale observations when depth measurements are corrupted. The learned representations are integrated with a Reinforcement Learning-based policy for collision-free navigation in unstructured environments when depth sensors experience degradation due to adverse conditions such as poor lighting or reflective surfaces. Simulation and real-world experiments demonstrate that our approach maintains robust performance under significant depth degradation and successfully transfers to real environments.

Figures & tables

Fig. 1: Real-world navigation under depth sensor degradation with preserved grayscale image.

Fig. 2: Architecture of the Cross-Modal Weighted Autoencoder for joint depth–grayscale representation learning.

Fig. 3: Depth and grayscale images are encoded into a shared latent space via the cmwae . Later, the cross-modal representation is combined with state information and processed by the rl policy, which outputs the velocity actions for a quadrotor platform.

Fig. 4: Top-down view of the training environment at each curriculum level.

Corruption (%)	MSE ( $\times 10^{-3}$ )		SSIM ( $\times 10^{-2}$ )
	$S_{1}$ /	$S_{2}$	$S_{1}$ /	$S_{2}$
0	8.49 /	5.95	85.29 /	85.78
20	9.41 /	6.34	84.95 /	85.67
30	10.05 /	6.72	84.86 /	85.57
40	10.39 /	7.03	84.67 /	85.45
50	10.71 /	8.01	84.79 /	85.34

TABLE I: Average mse (

\times 10^{-3}

) and Average ssim (

\times 10^{-2}

) across corruption levels for CMWAE trained under two corruption schemes. Best values per metric are bolded.

Fig. 5: mse and ssim box plots for CMWAE trained under corruption scheme $S_{1}$

Fig. 6: mse and ssim box plots for CMWAE trained under corruption scheme $S_{2}$

Fig. 7: Example reconstructions produced by the encoder trained under corruption scheme $S_{2}$ in evaluation studies

Fig. 8: Top view of the trajectories of the policy across different obstacle configurations. Each column corresponds to a distinct environment with the overlaid path showing the robot’s motion from start (blue) to goal (red). The bottom row shows image captures corresponding to the points along each trajectory where the highest levels of depth corruption were encountered.

		DCE-RL / CMWAE-RL
Condition	Curr.	Success (%)	Timeout (%)	Crash (%)
Clean	0	99.95 / 91.05	0.0 / 0.43	0.05 / 8.53
	5	94.00 / 85.30	2.88 / 1.71	3.19 / 13.00
	10	91.02 / 78.10	3.74 / 2.32	5.25 / 19.58
	15	85.79 / 71.51	5.09 / 2.50	9.12 / 25.99
	20	74.84 / 66.93	9.51 / 2.88	15.65 / 30.19

TABLE II: RL Policy Evaluation Results. Best values per metric are bolded.

Env.	Time (s)	Path (m)	Avg. (%)	Min. (%)	Max. (%)
1	20.20	25.05	29.08	15.34	49.46
2	23.36	26.47	25.85	9.04	60.98
3	22.00	25.61	25.04	12.31	47.76
4	18.76	25.03	29.60	9.82	40.72
5	21.53	25.75	26.25	7.64	50.32
6	21.24	25.28	26.23	9.03	43.34

TABLE III: Experimental results across 7 environments, showing time to reach the goal, path length, and average (Avg.), minimum (Min.), and maximum (Max.) corruption of the depth image.

Robot Navigation

Robot navigation studies how robots perceive, localize, plan, and move safely toward goals in static or changing environments.

See all 554 papers

Cross-Modal Navigation with Multi-Agent Reinforcement Learning
May 7, 2026Shuo Liu, Xinzichen Li, Christopher AmatoCross-Modal Learning Multi-Agent Collaboration
Khoury College of Computer Sciences Northeastern University Boston, MA, 02120
REDACT: Robust Perceptive Locomotion under Unseen Visual Corruption
Sep 21, 2026Natapat Kirdwichai, Tobias Driskell-Poole, Andrei Sontea +3Image Corruption Robustness Robot Locomotion
The authors are with the Faculty of Engineering and Physical Sciences and the Faculty of Environmental and Life Sciences, University of Southampton, United Kingdom.
CReF: Cross-modal and Recurrent Fusion for Depth-conditioned Humanoid Locomotion
Mar 31, 2026Yuan Hao, Ruiqi Yu, Shixin Luo +3Humanoid Robot Locomotion Robot Locomotion
The authors are with Institute of Cyber-Systems and Control, Zhejiang University, Hangzhou 310027, China. · Guoteng Zhang is with the School of Control Science and Engineering, Shandong University, Jinan 250061, China.

May 7, 2026J/K move · Enter open · S save

Cross-Modal Navigation with Multi-Agent Reinforcement Learning

Shuo Liu, Xinzichen Li, Christopher Amato

Khoury College of Computer Sciences Northeastern University Boston, MA, 02120

Cross-Modal Learning Multi-Agent Collaboration Multimodal Agents Multi-Agent Reinforcement Learning

Open paper AskPDF arXiv

Robust embodied navigation relies on complementary sensory cues. However, high-quality and well-aligned multi-modal data is often difficult to obtain in practice. Training a monolithic model is also challenging as rich multi-modal inputs induce complex representations and substantially enlarge the policy space. Cross-modal collaboration among lightweight modality-specialized agents offers a scalable paradigm. It enables flexible deployment and parallel execution, while preserving the strength of each modality. In this paper, we propose CRONA, a Multi-Agent Reinforcement Learning (MARL) framework for Cross-Modal Navigation. CRONA improves collaboration by leveraging control-relevant auxiliary beliefs and a centralized multi-modal critic with global state. Experiments on visual-acoustic navigation tasks show that multi-agent methods significantly improve performance and efficiency over single-agent baselines. We find that homogeneous collaboration with limited modalities is sufficient for short-range navigation under salient cues; heterogeneous collaboration among agents with complementary modalities is generally efficient and effective; and navigation in large, complex environments requires both richer multi-modal perception and increased model capacity.

Abstract

Robot Navigation

Explore similar work

Cross-Modal Navigation with Multi-Agent Reinforcement Learning

REDACT: Robust Perceptive Locomotion under Unseen Visual Corruption

CReF: Cross-modal and Recurrent Fusion for Depth-conditioned Humanoid Locomotion