VesperMUSE: Multi-Modal Fusion Vision Transformer for Image Semantic Segmentation Under Diverse Light Conditions

Raul Balmez

cs.CVOct 15, 2026

VesperMUSE: Multi-Modal Fusion Vision Transformer for Image Semantic Segmentation Under Diverse Light Conditions

Authors: Raul Balmez

Organizations: The University of Manchester, Department of Computer Science

Abstract

Semantic segmentation is critical for the reliable navigation and scene understanding of autonomous systems, yet its performance degrades severely in low-light environments due to noise, artifacts, colour distortion, and the loss of structural details. To address this challenge, this work introduces VesperMUSE, a unified multi-modal framework designed to achieve robust image semantic segmentation across diverse illumination conditions. The architecture addresses the light deficiency problem on multiple levels: proposing a novel physics-informed synthetic low-light adaptation algorithm, approaching Low-Light Image Enhancement (LLIE) through a thermal-guided paradigm, and finally aiming to recover the intrinsic scene details for semantic segmentation through a multi-modal fusion approach. Following enhancement, the framework dynamically integrates the primary RGB features with auxiliary thermal representations and surface normal maps highlighting geometric discontinuities and material boundaries. At the core of this integration is a highly scalable, modality-agnostic self-attention mechanism that extracts both global context and fine local features from an arbitrary number of auxiliary inputs. By combining and projecting these hierarchical multi-modal representations into a unified component, the attention module allows the network to selectively attend to relevant cross-modal cues without requiring any architectural modifications when different sensor configurations are used. Extensive evaluations demonstrate that VesperMUSE successfully recovers missing visual cues, providing a flexible, state-of-the-art (SOTA) solution for machine perception in extreme and unstructured real-world environments.

Figures & tables

Figure 1.1 : Mean Intersection over Union (mIoU) benchmarking results for semantic segmentation under three input configurations, well-lit, low-light, and enhanced low-light maps, on the PST900 [ 8 ] (left) and CART [ 25 ] (right) datasets, highlighting the impact of illumination conditions and enhancement strategies on semantic segmentation performance.

Figure 3.1 : An illustration of the real-world light distribution which is followed in the Real-world Distribution Matching stage

Figure 3.2 : Visual representation of the VesperMUSE LLIE module.

Figure 3.3 : Visual representation of the VesperMUSE Semantic Segmentation module.

Figure 3.4 : Visual representation of the VesperMUSE modality-agnostic attention mechanism.

Dataset	Quality Metrics		Image Statistics			Luminance (Y)
Dataset	SNR (dB) $\uparrow$	NMI $\uparrow$	Exp. $\downarrow$	Gamma $\approx 1$	CV (%) $\uparrow$	Min / Max	Mean	Std
LOL v1 [ 32 ]	9.59	0.46	0.13	1.03	69.54	2.33 / 82.58	15.49	10.77
LOL v2 real [ 33 ]	8.51	0.34	0.15	1.02	69.34	0.34 / 82.58	16.08	11.16
LOL v2 synthetic [ 33 ]	4.73	0.60	0.46	1.38	56.97	8.26 / 186.97	61.90	35.27
LOL Blur [ 35 ]	8.70	0.47	0.28	1.07	48.98	4.48 / 70.34	27.23	13.34
PST900 LL	10.71	0.31	0.29	1.00	51.96	3.38 / 29.59	12.86	6.68

Table 4.1: Quantitative comparison between real and low-light datasets across multiple statistical and physical metrics. The PST900 [ 8 ] /CART [ 25 ] synthetic table entries are paired images comprised from original RGB images from the two datasets and synthetically adapted low-light images.

Figure 4.1 : Illumination vs. signal integrity. Real low-light data retain high SNR, while prior synthetic methods introduce excessive noise and over-brightening. VesperMUSE preserves an accurate signal quality similar to that observable in true low-light conditions.

Figure 4.2 : PST900 [ 8 ] synthetic low-light adapted RGB and the original image.

Figure 4.3 : CART [ 25 ] synthetic low-light adapted RGB and the original image.

Method	PST900 LL		CART LL		Params (M) $\downarrow$
Method	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	Params (M) $\downarrow$
LYT-Net [ 47 ]	22.81	0.860	21.42	0.832	0.045
Retinexformer [ 50 ]	26.60	0.850	27.11	0.847	1.61
ISALux [ 51 ]	27.58	0.874	27.81	0.873	1.85
VesperMUSE LLIE	27.47	0.870	27.74	0.858	0.454

Table 4.2 : Quantitative comparison of LLIE methods on the synthetically low-light adapted PST900 LL and CART LL datasets. Best results are highlighted in bold .

Figure 4.4 : Qualitative comparison of VesperMUSE (VM) LLIE with SOTA methods enhancing the synthetically low-light adapted PST900 LL and CART LL datasets.

Method	Backbone	IoU per Class (%) $\uparrow$					mIoU (%) $\uparrow$
Method	Backbone	Background	Hand-Drill	Backpack	Fire-Ext.	Survivor	mIoU (%) $\uparrow$
EGFNet [ 77 ]	-	99.26	64.67	83.05	71.29	74.30	78.51
SA-Gate [ 76 ]	-	99.25	81.01	79.77	72.97	62.22	79.05
MMSMCNet 23 [ 78 ]	Mit-B3	-	62.40	89.20	73.30	74.70	79.80
CLNet-T 24 [ 79 ]	Mit-B4	-	68.50	87.90	72.70	75.70	80.80
DBCNet 23 [ 5 ]	ResNet101	-	77.20	82.70	73.00	76.70	81.80

Table 4.3 : Performance comparison with SOTA methods on the PST900 [ 8 ] dataset. Best results are highlighted in bold .

Model	Backbone	IoU per Class (%) $\uparrow$										mIoU (%) $\uparrow$
Model	Backbone	Bare Grd.	Rocky Terr.	Dev. Struct.	Road	Shrubs	Trees	Sky	Water	Vehicles	Person	mIoU (%) $\uparrow$
GMNet 21 [ 84 ]	ResNet152	81.50	88.80	83.80	60.30	70.80	79.50	92.00	97.40	54.80	17.80	72.70
EGFNet 22 [ 77 ]	ResNet101	80.30	85.20	56.30	60.50	69.90	78.40	92.80	97.40	57.20	17.10	69.50
EAEFNet 23 [ 11 ]	ResNet152	80.50	86.30	81.30	54.50	70.20	80.20	93.70	97.60	48.10	0.00	69.20
CMNext 23 [ 85 ]	Mit-B2	80.10	87.80	84.10	58.60	69.90	78.50	92.50	97.30	54.30	17.80	72.10
CMX 23 [ 9 ]	Mit-B2	82.40	90.30	85.00	60.40	71.00	79.90	93.60	97.60	57.90	22.20	74.00

Table 4.4 : Performance comparison with SOTA methods on the CART dataset [ 25 ] . Best results are highlighted in bold .

Figure 4.5 : Illustration of the per-class surface distribution on the PST900 [ 8 ] (left) and CART [ 25 ] (right) datasets. The percentage for each class is computed as the mean percentage of pixels covered across the samples where it appears (sample count is encoded as

n

Figure 4.6 : Qualitative comparison of VesperMUSE (VM) and the LLIE configuration (VM LLIE) with SOTA models on the PST900 dataset [ 8 ] .

Figure 4.7 : Qualitative comparison of VesperMUSE (VM) and the LLIE configuration (VM LLIE) with SOTA models on the CART dataset.

Model	Backbone	FLOPs (G) $\downarrow$	Params (M) $\downarrow$	mIoU (%) $\uparrow$
SGFNet 23 [ 81 ]	ResNet152	143.73	125.25	68.5
CMNeXt 23 [ 85 ]	Mit-B2	68.70	58.68	72.1
CMX 23 [ 9 ]	Mit-B2	67.20	66.57	74.0
CLNet-T 24 [ 79 ]	Mit-B4	217.85	130.84	73.7
CAINet 24 [ 86 ]	MobileNetV2	123.62	12.16	70.0
MCNet-T 25 [ 83 ]	ConvNeXt-B	278.85	199.79	73.7

Table 4.5 : Complexity comparison with an image resolution of

480\times 640

pixels. mIoU results are reported on the CART dataset [ 25 ] .

Primary Modality	Auxiliary Modalities	mIoU (%) $\uparrow$
Primary Modality	Auxiliary Modalities	PST900	CART
LLIE RGB	Thermal, Surface Norm.	78.18	59.57
	Thermal	74.53	58.56
	Surface Norm.	73.80	58.62
Thermal	Surface Norm., Ref., Illum.	60.60	50.76
Surface Norm.	LLIE RGB	67.70	55.90

Table 4.6 : Table analyzing the impact of different primary and auxiliary input modalities on the PST900 [ 8 ] and CART [ 25 ] datasets. Grouping by the primary backbone highlights the performance delta when injecting various auxiliary streams.

Method	Params (M) $\downarrow$	FLOPs (G) $\downarrow$	mIoU (%) $\uparrow$
Method	Params (M) $\downarrow$	FLOPs (G) $\downarrow$	PST900	CART
VesperMUSE LLIE	180	321	78.18	59.57
Concatenation-based Fusion	192	321	75.42	54.44
Convolution Decoder	179	326	73.51	53.20

Table 4.7 : Ablation study analyzing the impact of different fusion and decoder strategies on the PST900 [ 8 ] and CART [ 25 ] datasets, including parameter counts and computational complexity.

Figure 5.1 : Qualitative maps highlighting the noise and excessive degradation observed in few synthetically low-light adapted images.

Figure 5.2 : Histogram comparison between the two pairs of RGB and LLIE RGB maps showing the colour balance.

Figure 5.3 : Visual representation of the ratio between mIoU and the parameter count (M) of SOTA models.

Semantic Segmentation

Semantic segmentation assigns a class label to every image pixel, enabling detailed scene understanding across domains and sensors.

See all 416 papers

M2Retinexformer: Multi-Modal Retinexformer for Low-Light Image Enhancement
May 11, 2026Youssef Aboelwafa, Hicham G. Elmongui, Marwan TorkiImage Enhancement Low-Light Image Enhancement
Alexandria University, Egypt
IAF-Net: Illumination-Adaptive Fusion for Low-Light Urban Road Segmentation
May 29, 2026Bingtao Wang, Daojie Peng, Fulong Ma +2Autonomous Driving Perception Multimodal Fusion
The Shandong University · The Hong Kong University of Science and Technology (Guangzhou)
Multinex: Lightweight Low-light Image Enhancement via Multi-prior Retinex
Apr 11, 2026Alexandru Brateanu, Tingting Mu, Codruta Ancuti +1Low-Light Image Enhancement Image Enhancement
Department of Computer Science, University of Manchester, Manchester, United Kingdom · ETcTI, University Politehnica Timisoara, Timisoara, Romania · West University of Timisoara, Timisoara, Romania

May 11, 2026J/K move · Enter open · S save

M2Retinexformer: Multi-Modal Retinexformer for Low-Light Image Enhancement

Youssef Aboelwafa, Hicham G. Elmongui, Marwan Torki

Alexandria University, Egypt

Image Enhancement Low-Light Image Enhancement

Open paper AskPDF arXiv

Low-light image enhancement is challenging due to complex degradations, including amplified noise, artifacts, and color distortion. While Retinex-based deep learning methods have achieved promising results, they primarily rely on single-modality RGB information. We propose M2Retinexformer (Multi-Modal Retinexformer), a novel framework that extends Retinexformer by incorporating depth cues, luminance priors, and semantic features within a progressive refinement pipeline. Depth provides geometric context that is invariant to lighting variations, while luminance and semantic features offer explicit guidance on brightness distribution and scene understanding. Modalities are extracted at multiple scales and fused through cross-attention, with adaptive gating dynamically balancing illumination-guided self-attention and cross-attention based on the reliability of auxiliary cues. Evaluations on the LOL, SID, SMID, and SDSD benchmarks demonstrate overall improvements over Retinexformer and recent state-of-the-art methods. Code and pretrained weights are available at https://github.com/YoussefAboelwafa/M2Retinexformer

Abstract

Semantic Segmentation

Explore similar work

M2Retinexformer: Multi-Modal Retinexformer for Low-Light Image Enhancement

IAF-Net: Illumination-Adaptive Fusion for Low-Light Urban Road Segmentation

Multinex: Lightweight Low-light Image Enhancement via Multi-prior Retinex