SenseWeather: Multimodal Sensing for Robust Robotic Perception under Extreme Conditions

Fucai Ke; Sukai Huang; Junyi Hao; Kal Backman; Naim Rastgoo; Gholamreza Haffari; Hamid Rezatofighi

cs.ROOct 8, 2026

SenseWeather: Multimodal Sensing for Robust Robotic Perception under Extreme Conditions

Authors: Fucai Ke, Sukai Huang, Junyi Hao, Kal Backman, Naim Rastgoo, Gholamreza Haffari, Hamid Rezatofighi

Organizations: Faculty of Information Technology, Monash University, Clayton, VIC 3800, Australia

Abstract

Robotic perception must remain reliable when extreme environmental conditions severely degrade the primary RGB observation. Although robotic platforms commonly carry complementary sensors such as depth and infrared, much of the modern perception stack remains RGB-centric, creating a mismatch between multimodal onboard sensing and pretrained downstream models. We study whether complementary sensing can be used upstream to recover a task-compatible RGB observation that preserves existing perception capabilities without modifying or retraining the downstream stack. To this end, we introduce SenseWeather, a controlled multimodal benchmark for robotic perception under severe rain, snow, dust, fog, and darkness, with registered RGB, depth, and infrared observations, clean references, and grounded high-level perception tasks. We further develop Fusion-to-RGB, with two parameter-efficient sensor-fusion realizations based on token-space fusion for a diffusion transformer and residual latent conditioning for an inpainting UNet. Across frozen open-vocabulary grounding and vision-language models, RGB+D+IR fusion improves all paired downstream comparisons over the corresponding RGB-only restorers, including AP50 gains across all ten detector comparisons. Importantly, these perception gains do not consistently follow conventional image-fidelity metrics, and increasing weather severity does not induce monotonic downstream degradation. These results show that complementary sensing can preserve reusable robotic perception under extreme conditions, while highlighting task-level perception robustness as an objective distinct from visual restoration quality.

Figures & tables

Fig. 1: SenseWeather studies Fusion-to-RGB as a task-compatible multimodal interface: complementary sensing recovers machine-readable RGB evidence for unchanged foundation models, even when RGB-only restoration remains ambiguous.

Environment	Representative operational content	Object instances	Coarse categories	Fine-grained categories
Island Base	Coastal terrain, fortifications, vehicles, rotorcraft, and port equipment	725	51	64
Offshore Dock 1	Dense containers, cranes, trucks, reachstackers, forklifts, and utilities	18,488	61	90
Offshore Dock 2	Ship-handling terminal, STS cranes, carriers, barriers, and service assets	1,059	52	65
Offshore Dock 3	Cargo storage, tanks, fences, trailers, trucks, and material handling	1,408	29	41
Oil Refinery	Process pipes, storage facilities, industrial frames, cranes, and vessels	815	16	21
All environments	Five environment-specific catalogs	22,495	175	247

TABLE I: Object inventory of the five SenseWeather environments before exclusion and visibility filtering. Each instance denotes one annotated object; coarse and fine-grained categories correspond to hierarchical labels (e.g., Architecture and Architecture-Hangar). The final row reports totals with categories deduplicated across environments.

Fig. 2: Fusion-to-RGB preserves one downstream interface while adapting fusion to the backbone. Qwen keeps the modalities as separate image-token segments; RealFill injects a learned depth–infrared residual into its RGB conditioning latent. The clear target appears only during training, and every downstream model remains frozen.

Fig. 3: RGB+D+IR fusion improves AP 50 in all ten paired detector comparisons and improves all twelve vision–language comparisons. The plot shows percentage-point gains over each RGB-only parent for AP 50 (top) and the VLM headline metrics (bottom).

Image domain	Grounding DINO	Florence-2	Falcon	OmDet-Turbo	YOLO-World
Degraded RGB	30.9	15.0	3.4	9.8	5.7
MAE [ 36 ]	11.5	4.9	4.4	2.8	1.0
Pix2Pix-Turbo [ 23 ]	28.4	13.2	4.0	8.1	4.4
RealFill [ 28 ]	26.3	9.3	2.0	8.2	3.6
Qwen-image [ 29 ]	35.7	18.3	4.0	12.6	6.6
Ideal RGB	42.1	24.2	4.0	14.1	7.5

TABLE II: Performance of RGB-only restoration baselines on open-vocabulary localization under extreme conditions. Each entry is AP 50 (%). Bold marks the best learned restoration for each detector. Florence-2 and Falcon assign tied confidence to retained boxes, so their AP should only be compared across image domains within the same detector.

Backbone	Input	Grounding DINO	Florence-2	Falcon	OmDet-Turbo	YOLO-World
RealFill [ 28 ]	RGB	26.3	9.3	2.0	8.2	3.6
	RGB+D	30.4	9.8	4.0	9.4	4.4
	RGB+IR	29.0	8.9	4.9	8.6	4.1
	RGB+D+IR	30.5	10.0	6.1	10.5	4.8
	$\Delta$	+4.2	+0.7	+4.1	+2.3	+1.2
Qwen-image [ 29 ]	RGB	35.7	18.3	4.0	12.6	6.6

TABLE III: Effect of complementary sensing in our Fusion-to-RGB framework on open-vocabulary localization. RGB denotes the RGB-only baseline for each backbone, while RGB+D, RGB+IR, and RGB+D+IR are sensor-conditioned Fusion-to-RGB variants. Each entry reports AP 50 (%) on the Primary Extreme-Condition Test Set.

\Delta

denotes RGB+D+IR minus the RGB baseline in percentage points.

Fig. 4: Weather intensity produces monotonic pixel degradation but not a monotonic detection response on the Controlled Intensity Study Set. Left: MAE increases within every weather, with much larger absolute effects for dust and fog. Right: median Spearman correlation between intensity and AP 50 across five detectors for each image domain; negative is the expected degradation direction. Ideal RGB is omitted because it is an input reference rather than a restoration method. Darkness is excluded because it was added only to the expanded primary evaluation and has no controlled intensity sweep.

	Qwen3-VL-2B			Qwen3-VL-4B
Image domain	Count	Presence	MCQ	Count	Presence	MCQ
Degraded RGB	45.7	73.8	56.5	30.4	74.2	65.8
MAE RGB	40.3	70.0	45.7	24.9	68.4	50.5
Pix2Pix-Turbo	51.8	70.6	59.7	33.5	71.0	62.9
RealFill RGB	45.7	69.6	60.4	33.5	72.2	62.9
Qwen RGB	50.5	67.1	66.1	41.5	72.2	71.9

TABLE IV: RGB-only restoration exhibits task- and scale-dependent utility for Qwen3-VL. All entries are percentages, and bold marks the best learned restoration in each column.

		Qwen3-VL-2B			Qwen3-VL-4B
Backbone	Input	Count	Presence	MCQ	Count	Presence	MCQ
RealFill [ 28 ]	RGB	45.7	69.6	60.4	33.5	72.2	62.9
	RGB+D	50.5	69.6	63.6	36.1	73.8	68.4
	RGB+IR	47.6	68.3	60.4	37.4	72.2	67.4
	RGB+D+IR	49.8	72.2	64.2	38.0	76.1	70.0
	$\Delta$	+4.1	+2.6	+3.8	+4.5	+3.9	+7.1

TABLE V: Effect of complementary sensing in our Fusion-to-RGB framework on frozen Qwen3-VL performance. RGB is the RGB-only baseline for each backbone; RGB+D, RGB+IR, and RGB+D+IR are our sensor-conditioned variants. Results use the Primary Extreme-Condition Test Set. The

\Delta

row reports RGB+D+IR minus RGB in percentage points.

Condition	Fog	Dust	Darkness	Rain	Snow
Mean gain (pp)	+7.11	+6.43	+4.33	+0.85	$-0.10$

TABLE VI: Condition-dependent gain of RGB+D+IR over RGB-only restoration. Values are mean percentage-point improvements across Grounding DINO and VQA for the Qwen and RealFill families.

Explore similar work

OctoSense: Self-Supervised Learning for Multimodal Robot Perception
Jun 25, 2026Anthony Bisulco, Jeremy Wang, Kostas Daniilidis +2Autonomous Driving Datasets Masked Autoencoders
1General, Robotics, Automation, Sensing and Perception (GRASP) Laboratory, University of Pennsylvania · Computer Science Department, Brown University
AURORA-KITTI: Any-Weather Depth Completion and Denoising
Mar 16, 2026Yiting Wang, Tim Brödermann, Hamed Haghighi +4Depth Completion Multimodal Robustness
Warwick Manufacturing Group, University of Warwick, Coventry, CV4 7AL, United Kingdom · School of Engineering and Materials Science, Queen Mary University of London, London, E1 4NS, United Kingdom · Computer Vision Laboratory, ETH Zürich, Zurich, 8057, Switzerland +1
DUALVISION: RGB-Infrared Multimodal Large Language Models for Robust Visual Reasoning
Apr 20, 2026Abrar Majeedi, Zhiyuan Ruan, Ziyi Zhao +3Cross-Modal Learning Visual Question Answering
University of Wisconsin-Madison · Amazon · Northeastern University

Jun 25, 2026J/K move · Enter open · S save

OctoSense: Self-Supervised Learning for Multimodal Robot Perception

Anthony Bisulco, Jeremy Wang, Kostas Daniilidis, Randall Balestriero+1

1General, Robotics, Automation, Sensing and Perception (GRASP) Laboratory, University of Pennsylvania · Computer Science Department, Brown University

Autonomous Driving Datasets Masked Autoencoders Robotic Perception Multimodal Learning Multimodal Sensor Fusion

Open paper AskPDF arXiv

We present OctoSense, an open-source sensor platform with stereo RGB and event cameras, LiDAR, a thermal camera, an inertial measurement unit, RTK-corrected global positioning system, and proprioception (CAN bus data from a car, and joint angles for a quadruped robot). The eponymous OctoSense dataset contains 59 hours of time-synchronized driving data across different types of environments at different times of the day, including situations with highly degraded sensors. We demonstrate multi-modal self-supervised learning using such real-world robotics data, where sensors have different representations, frequencies, latencies and noise. Our approach, a "late-fusion" masked autoencoder, (i) uses modality-specific tokenizers to account for different spatiotemporal characteristics of these sensors, and (ii) caches modality-specific tokens at inference time to process new measurements as they come. This architecture (i) is fast (6.68 ms and 112 ms on NVIDIA 5090 and Orin NX respectively, to compute the representation), (ii) performs better than existing image-only foundation models on tasks such as estimation of optical flow, depth, semantic segmentation, and ego-motion (translation, rotation, and steering angle), and (iii) predicts robustly at nighttime or in situations where sensory data is degraded. See our project page for links to the dataset, code, and supplementary videos: https://abisulco.com/octosense/.

Abstract

Explore similar work

OctoSense: Self-Supervised Learning for Multimodal Robot Perception

AURORA-KITTI: Any-Weather Depth Completion and Denoising

DUALVISION: RGB-Infrared Multimodal Large Language Models for Robust Visual Reasoning