Variance-Guided Spatial Attention Fusion for Robust End-to-End Driving under Asymmetric Sensor Degradation

Weizhi Tao; Zengwang Jin; Xiao Wang; Hailong Huang

cs.CVAug 25, 2026

Variance-Guided Spatial Attention Fusion for Robust End-to-End Driving under Asymmetric Sensor Degradation

Authors: Weizhi Tao, Zengwang Jin, Xiao Wang, Hailong Huang

Organizations: Department of Aeronautical and Aviation Engineering, The Hong Kong Polytechnic University, 999077, Hong Kong, China · School of Robotics, Anhui University, Hefei 230039, China; with the Anhui Provincial Engineering Research Center for Unmanned System and Intelligent Technology, Anhui University, Hefei 230039, China; and also with the Shenzhen Research Institute of Northwestern Polytechnical University, Shenzhen 518057, China · School of Robotics and Institute of Embodied Intelligence, Anhui University, Hefei 230039, China

Abstract

End-to-end multimodal driving has progressed rapidly by fusing camera and LiDAR streams. Existing pipelines remain fragile under asymmetric sensor degradation, where either an entire modality or only a localized region is corrupted while other regions remain useful. The key difficulty is not simply to add an uncertainty head, but to obtain dense reliability supervision, calibrate this reliability against physical fault severity, and use it before unreliable features bias the planner. We propose Variance-Guided Spatial Attention Fusion (VG-SAF), in which dense heteroscedastic reliability estimates act as interpretable spatial gates. The framework couples three components. First, a physically grounded augmentor simulates representative camera and LiDAR failures and emits a continuous spatial mask, providing dense supervision without additional annotation. Second, modality-specific experts predict per-pixel reliability scales through cross-branch dense distillation in log space, enforcing a monotone severity-to-scale response. Third, calibrated reliability maps drive a hybrid attention mechanism that suppresses unreliable cells with a local spatial gate and arbitrates between modalities through a cross-modal trust softmax. A Laplace uncertainty head emits a systemic waypoint uncertainty scale that signals severe or combined sensor degradation, including severities outside the training ranges. On the CARLA Longest6 benchmark, VG-SAF consistently improves closed-loop robustness over the baselines across camera-only, LiDAR-only, and joint degradation regimes, as measured by driving score, route completion, and infraction score.

Figures & tables

Fig. 1: Trust-everywhere fusion versus the proposed variance-guided spatial attention fusion (VG-SAF) under sensor corruption. (a) A cross-attention stage admits every token regardless of its sensor-side quality. The damaged region (dashed ellipse) propagates uniformly into the fused representation and biases the planner. (b) VG-SAF estimates a dense variance map for each modality. A local spatial gate $\exp(-\eta^{\mathrm{loc}}_{m}\bar{V}_{m})$ and a cross-modal trust softmax suppress the corrupted region while preserving the surviving modality.

Fig. 2: Two-stage training workflow of VG-SAF. Stage 1 (top) trains two independent perception experts under the per-modality augmentors $\mathcal{A}_{\mathrm{rgb}}$ and $\mathcal{A}_{\mathrm{lid}}$ and the perception loss $\mathcal{L}_{\mathrm{percep}}$ . Stage 2 (bottom) freezes the experts and learns the fusion stage, where dense reliability maps are aggregated into $\bar{V}_{m}$ and converted into a local gate and a trust softmax. Their product forms $\mathbf{A}_{m}$ , which gates each modality before waypoint decoding.

Model	$\mathrm{DS}\!\uparrow$	$\mathrm{RC}\!\uparrow$	$\mathrm{IS}\!\uparrow$
Camera corrupted (LiDAR clean)
Image-only	$18.4\pm 6.1$	$73.2\pm 8.7$	$0.25\pm 0.06$
Equal-Weight Fusion	$14.7\pm 5.8$	$68.5\pm 9.4$	$0.22\pm 0.05$
TransFuser	$32.1\pm 7.3$	$84.6\pm 6.2$	$0.40\pm 0.07$
VG-SAF (Ours)	$\mathbf{42.6\pm 5.9}$	$\mathbf{91.2\pm 4.1}$	$\mathbf{0.51\pm 0.06}$
LiDAR corrupted (camera clean)

TABLE I: Closed-loop Longest6 results under three sensor-corruption regimes (mean

\,\pm\,

std over three runs). Metrics are averaged at the route/run level; therefore, the product of the displayed mean RC and mean IS need not equal the displayed mean DS.

Fig. 3: Median waypoint

L_{1}

error (m) versus corruption severity

K

under four attention configurations and three corruption scenarios. For each setting, errors are first averaged over the four future waypoints of each frame and the median is then taken over all evaluated frames, routes, and runs. Lower is better.

Fig. 4: Fused per-modality attention magnitude

\bar{A}_{m}

versus corruption severity

K

under three scenarios. Here,

\bar{A}_{m}

is the spatial mean of

\mathbf{A}_{m}

, averaged over all evaluated frames, routes, and runs. The impaired modality is monotonically down-weighted while the intact modality absorbs the displaced mass. Data stop at

K\!=\!90\%

; the

100\%

tick is retained only for scale.

Modality/Mode	$K$ (%)	$\bar{V}$	$\bar{V}/\bar{V}_{\mathrm{clean}}$	$\Delta t$
Camera
signal_drop	100	37.26	37.5 $\times$	$-0.11$
local_noise	45	5.77	5.8 $\times$	$-0.01$
local_exposure_pulse	70	22.67	22.8 $\times$	$-0.09$
local_occlusion	59	11.75	11.8 $\times$	$-0.02$
night_lowlight	60	7.03	7.1 $\times$	$-0.01$

TABLE II: Per-mode reliability response in the upper severity tertile of each mode.

K

is the mean mask severity;

\bar{V}

is the pooled per-modality reliability scale averaged over all qualifying frames from 36 Longest6 routes and three runs;

\bar{V}/\bar{V}_{\mathrm{clean}}

is the clean-normalized ratio; and

\Delta t

is the corrupted modality’s trust shift relative to clean operation. Values above the training ranges in Tables S1–S2 are deliberate severity-extrapolation tests.

Fig. 5: Camera local_exposure_pulse fault at $K=48\%$ ; LiDAR clean. The dense reliability response raises the RGB-side uncertainty and suppresses unreliable local features while LiDAR remains trusted.

Fig. 6: LiDAR frustum_occlusion fault at $K=59\%$ ; camera clean. The BEV reliability scale peaks inside the missing wedge, down-weighting corrupted LiDAR cells and preserving the camera-supported trajectory.

Fig. 7: Closed-loop comparison under the night_lowlight corruption mode. (a) TransFuser baseline: the RGB semantic segmentation labels the gravel pavement and shoulder as drivable road; the ego exits the carriageway at the curve and the route fails (red cross). (b) VG-SAF: the RGB segmentation confines the road class to the carriageway and the route completes successfully (green check).

ID & Mode	Real-world analogue	Description	Scope	Training $K$
1 signal_drop	power loss / glare / snow	Frame-wide blackout, whiteout saturation, or luminance-correlated snow with chromatic jitter.	Global	$100\%$
2 local_noise	low-light ISO patches	2–6 Gaussian noise patches blending Poisson photon noise and additive Gaussian read-out noise.	Local	$10$ – $40\%$
3 local_exp *	specular flare / bloom	1–3 localized over-exposure spots with directional bloom and gamma–gain saturation.	Local	$10$ – $45\%$
4 local_occ *	lens debris / mud splash	2–4 alpha-blended elliptical occluders with edge darkening, mimicking foreign objects.	Local	$5$ – $30\%$
5 night_low *	dusk / night driving	Frame-wide gain reduction, non-linear gamma, warm shift, and contrast roll-off.	Global	$30$ – $65\%$
6 motion_blur	ego-motion / vibration	Directional convolution with a randomly oriented kernel, simulating ego-motion or shake.	Global	$23$ – $68\%$

TABLE S1: Camera fault modes implemented by the proposed augmentor.

ID & Mode	Real-world analogue	Description	Scope	Training $K$
1 signal_drop	jamming / spurious returns	Frame-wide jamming represented by spurious returns following range-conditioned sparsity.	Global	$100\%$
2 range_drop *	rain / fog / absorption	Distance-dependent point dropping with near-sensor false returns simulating dome-side backscatter.	Global	$20$ – $55\%$
3 frustum_occ *	mud / debris on dome	Angular wedge of complete blockage, populated with sparse near-sensor false returns.	Local	$8$ – $25\%$
4 local_speckle	airborne dust / insects	Sum of 2–5 oriented Gaussian blobs gated to empty cells, modulated by sparse Bernoulli patterns.	Local	$5$ – $20\%$
5 feature_noise	measurement / channel noise	Channel-wise Gaussian noise added to active voxels, modeling measurement or feature-channel disturbance.	Global	$20$ – $60\%$
* Shortened names: range_dropout (2), frustum_occlusion (3).

TABLE S2: LiDAR fault modes implemented by the proposed augmentor.

Fig. S1: Camera fault catalog. Panel (a) shows an unperturbed front-camera frame from the Longest6 validation split. Panels (b)–(i) display the eight trained corruption modes applied independently to (a). (b) signal_drop ; (c) local_noise ; (d) local_exposure_pulse ; (e) local_occlusion ; (f) night_lowlight ; (g) motion_blur ; (h) ghosting ; (i) color_shift . Modes (b), (f), (g), (h), (i) carry global masks; (c), (d), (e) carry local masks.

Fig. S2: LiDAR fault catalog. Panel (a) shows the unperturbed BEV histogram for the same frame as Fig. S1 (a). Panels (b)–(f) display the five trained LiDAR corruption modes. (b) signal_drop ; (c) range_dropout ; (d) frustum_occlusion ; (e) local_speckle ; (f) feature_noise . Modes (b), (c), (f) carry global masks; (d) and (e) carry local masks.

End-to-End Autonomous Driving

End-to-end autonomous driving learns to turn sensor observations and driving context into vehicle actions or trajectories.

See all 227 papers

Can BEV Perception Gracefully Degrade under Sensor Failures?
May 29, 2026Haifa Zhang, Yijing Wang, Haoyu Wang +2Missing-Modality Learning Multimodal Robustness
Tianjin Key Laboratory of Intelligent Unmanned Swarm Technology and System, School of Electrical and Information Engineering, Tianjin University, Tianjin, China · Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai, China
Mining Cross-Modal Perception Gaps in Autonomous Driving: When Camera and LiDAR Disagree
Oct 12, 2026Hongsen Cao, Mona Jaber, Shanxin Yuan +1Camera-LiDAR Fusion 3D Object Detection
School of Electronic Engineering and Computer Science, Queen Mary University of London, London, United Kingdom.
CRUISE: Vision-Language Model-Guided Uncertainty-Aware Cross-Modal Sensor Fusion for Robust Autonomous Driving
Aug 10, 2026Junyao Wang, Yulin Xu, Yu Li +2Multimodal Robustness Autonomous Driving Perception
Department of Computer Science, University of California, Irvine, 92697, USA · Department of Electrical Engineering and Computer Science, University of California, Irvine, 92697, USA · Department of Computer Science, the Department of Electrical Engineering and Computer Science, and the Department of Mechanical and Aerospace Engineering, University of California, Irvine, 92697, USA

May 29, 2026J/K move · Enter open · S save

Can BEV Perception Gracefully Degrade under Sensor Failures?

Haifa Zhang, Yijing Wang, Haoyu Wang, Zheng Li+1

Tianjin Key Laboratory of Intelligent Unmanned Swarm Technology and System, School of Electrical and Information Engineering, Tianjin University, Tianjin, China · Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai, China

Missing-Modality Learning Multimodal Robustness Camera-LiDAR Fusion Autonomous Driving Perception Bird's-Eye-View Representation

Open paper AskPDF arXiv

Despite the remarkable success of multi-modal bird's-eye view (BEV) perception in autonomous driving, current systems exhibit a critical vulnerability: existing fusion mechanisms are highly brittle to sensor corruptions, often causing catastrophic performance degradation. This vulnerability largely stems from the fact that standard fusion frameworks typically integrate multi-modal representations in a static manner, leading to a precipitous performance collapse under missing or corrupted modalities. In contrast, we show that graceful degradation is achievable through active modality reliability assessment. To this end, we present Grace-BEV, a lightweight and plug-and-play framework that enforces active reliability awareness during multi-modal fusion. Instead of relying on computationally expensive cross-modal interactions, Grace-BEV leverages the aligned BEV space to explicitly assess modality trustworthiness via a TrustGate Router and dynamically recalibrate feature integration using the FailSafe Fusion Block. Furthermore, we devise a Three-Phase Training strategy with Modality Dropout to prevent modality dominance and encourage balanced cross-modal learning under unreliable inputs. Extensive experiments on nuScenes-R and nuScenes-C show that Grace-BEV maintains robust performance across diverse corruption settings. Notably, under catastrophic LiDAR failures where standard baselines collapse to 0.0% mean Average Precision (mAP), Grace-BEV restores performance to as high as 34.7% mAP. Moreover, it improves clean accuracy by up to 1.4%, achieving a strong trade-off between robustness and efficiency.

Abstract

End-to-End Autonomous Driving

Explore similar work

Can BEV Perception Gracefully Degrade under Sensor Failures?

Mining Cross-Modal Perception Gaps in Autonomous Driving: When Camera and LiDAR Disagree

CRUISE: Vision-Language Model-Guided Uncertainty-Aware Cross-Modal Sensor Fusion for Robust Autonomous Driving