Layer-Specific Lipschitz Modulation for Fault-Tolerant Multimodal Representation Learning

Diyar Altinses; Andreas Schwung

cs.LGMar 26, 2026

Layer-Specific Lipschitz Modulation for Fault-Tolerant Multimodal Representation Learning

Authors: Diyar Altinses, Andreas Schwung

Organizations: Department of Automation Technology and Learning Systems, South Westphalia University of Applied Sciences, Soest, Germany

Abstract

Modern multimodal systems deployed in industrial and safety-critical environments must remain reliable under partial sensor failures, signal degradation, or cross-modal inconsistencies. This work introduces a mathematically grounded framework for fault-tolerant multimodal representation learning that unifies self-supervised anomaly detection and error correction within a single architecture. Building upon a theoretical analysis of perturbation propagation, we derive Lipschitz- and Jacobian-based criteria that determine whether a neural operator amplifies or attenuates localized faults. Guided by this theory, we propose a two-stage self-supervised training scheme: pre-training a multimodal convolutional autoencoder on clean data to preserve localized anomaly signals in the latent space, and expanding it with a learnable compute block composed of dense layers for correction and contrastive objectives for anomaly identification. Furthermore, we introduce layer-specific Lipschitz modulation and gradient clipping as principled mechanisms to control sensitivity across detection and correction modules. Experimental results on multimodal fault datasets demonstrate that the proposed approach improves both anomaly detection accuracy and reconstruction under sensor corruption. Overall, this framework bridges the gap between analytical robustness guarantees and practical fault-tolerant multimodal learning.

Figures & tables

Figure 1: Overview of the proposed fault-tolerant approach for robotic systems. The architecture includes encoding, correction with contrastive learning and detection modules, and decoding stages to reconstruct the original signal.

Panel of Figure 2 — Figure 2: Operational pipeline for robust industrial deployment. Heterogeneous inputs are encoded into a joint latent space. The representation is analyzed for faults: clean data bypasses the correction block for efficiency, while faulty data is rectified via the Lipschitz-controlled module before executing downstream tasks.

Panel of Figure 3 — Figure 3: Three image modality samples of the three distinct multimodal datasets [ 41 ] .

Panel of Figure 4 — Figure 4: Several randomly applied augmentation techniques on one random camera sample.

Model	Camera	Sensor	Combined
Training
MMAE	0.114 $\pm$ 0.063	0.815 $\pm$ 0.562	0.929 $\pm$ 0.584
MMVAE	0.137 $\pm$ 0.026	0.694 $\pm$ 0.435	0.831 $\pm$ 0.442
MMFR	0.129 $\pm$ 0.033	0.973 $\pm$ 0.500	1.102 $\pm$ 0.504
MMCF	0.113 $\pm$ 0.017	0.165 $\pm$ 0.077	0.279 $\pm$ 0.079
MMGAN	0.113 $\pm$ 0.025	0.158 $\pm$ 0.097	0.209 $\pm$ 0.101

Table 1: Train and test performances (in

10^{-2}

) of the failure correction using the mean over the final 250 batches of 10 trials. The reconstruction performance of the camera, the sensor, and both combined within the MuJoCo dataset.

Panel of Figure 5 — Figure 5: Visualization of the learned latent feature space using t-SNE. (a) Distribution of clean (purple) versus corrupted (yellow) samples before correction. (b) Distribution of clean (purple) versus corrected (yellow) samples.

Model	Camera	Sensor	Combined
Training
MMAE	0.131 $\pm$ 0.121	1.022 $\pm$ 1.203	1.153 $\pm$ 1.282
MMVAE	0.158 $\pm$ 0.093	0.961 $\pm$ 1.131	1.119 $\pm$ 1.180
MMFR	0.142 $\pm$ 0.081	0.927 $\pm$ 1.092	1.069 $\pm$ 1.133
MMCF	0.081 $\pm$ 0.022	0.602 $\pm$ 0.651	0.683 $\pm$ 0.643
MMGAN	0.074 $\pm$ 0.035	0.563 $\pm$ 0.997	0.637 $\pm$ 1.004

Table 2: Train and test performances (in

10^{-2}

) of the failure correction using the mean over the final 250 batches of 10 trials. The reconstruction performance of the camera, the sensor, and both combined within the single-robot welding station dataset.

Model	Camera	Sensor	Combined
Training
MMAE	0.031 $\pm$ 0.032	1.122 $\pm$ 1.241	1.153 $\pm$ 1.273
MMVAE	0.030 $\pm$ 0.011	0.962 $\pm$ 1.092	0.992 $\pm$ 1.103
MMFR	0.051 $\pm$ 0.041	1.121 $\pm$ 1.240	1.172 $\pm$ 1.281
MMCF	0.021 $\pm$ 0.010	0.872 $\pm$ 1.011	0.893 $\pm$ 1.021
MMGAN	0.078 $\pm$ 0.024	0.767 $\pm$ 0.562	0.844 $\pm$ 0.564

Table 3: Train and test performances (in

10^{-2}

) of the failure correction using the mean over the final 250 batches of 10 trials. The reconstruction performance of the camera (C), the sensor (S), and both combined (B) within the dual-robot welding station dataset.

Panel of Figure 6 — Figure 6: Training dynamics and spectral properties of MMSSL. (a) Both contrastive and similarity losses converge rapidly, with test losses (dashed) closely tracking training curves. (b) Reconstruction errors for both modalities decrease monotonically, confirming stable learning. (c) The anomaly detection loss stabilizes after an initial adaptation phase. (d) The spectral norm of the Corrector (green) is strictly bounded ( $<0.03$ ), ensuring a contractive mapping, while the Detector (red) maintains high sensitivity ( $>2.0$ ).

Class	Precision	Recall	F1-Score
Clean	0.929	0.999	0.963
Sensor Failure	0.997	0.839	0.911
Camera Failure	0.996	0.860	0.923
Macro Avg	0.974	0.900	0.933

Table 4: Fault detection performance aggregated over 250 batches. Metrics are macro-averaged by class.

Panel of Figure 7 — Figure 7: Training dynamics of the failure detector. (a) The model converges rapidly to a stable high-performance regime. (b) While clean states are identified immediately, the model progressively improves its sensitivity to sensor and camera faults, eventually plateauing at robust recall levels.

Figure 8: Normalized confusion matrix aggregated over 250 batches. The dominant diagonal confirms high classification accuracy. Off-diagonal errors are almost exclusively false negatives, identifying the detection threshold for subtle anomalies as the limiting factor.

Panel of Figure 9 — Figure 9: Latent error heatmaps for dense (a) and convolutional (b) encoders.

Figure 10: Log-histogram of latent error magnitudes. The convolutional encoder (orange) exhibits a sharp peak at zero, whereas the dense encoder (blue) shows a broad distribution, confirming global diffusion.

Modality	w/o Reg	MMSSL
	Training
Camera	0.127 $\pm$ 0.024	0.098 $\pm$ 0.023
Sensor	0.362 $\pm$ 0.349	0.307 $\pm$ 0.302
Combined	0.490 $\pm$ 0.357	0.405 $\pm$ 0.310
	Testing
Camera	0.147 $\pm$ 0.003	0.118 $\pm$ 0.005

Table 5: Train and test performances (in

10^{-2}

) of the proposed SSL approach with and without regularizer. Values represent the mean over the final 250 batches of 10 trials for camera, sensor, and both combined correction performance.

Figure 11: Dual-axis plot showing the Correction Loss (left y-axis, blue, scale $\times 10^{-3}$ ) and Detection Accuracy (right y-axis, orange, scale $\times 10^{-1}$ ) as a function of the $\lambda_{sim}/\lambda_{con}$ ratio (x-axis, logarithmic scale). The blue line with error bars represents the correction loss, and the orange line with square markers represents the detection accuracy.

Figure 12: Pareto trade-off between the unweighted self-supervised loss components. As the weighting ratio $\lambda_{sim}/\lambda_{con}$ increases, the similarity loss (blue) decreases, indicating better reconstruction, while the contrastive loss (red) rises, indicating reduced latent separability. The dashed lines illustrate the opposing linear trends of these competing objectives.

Figure 13: Dependence of detection performance on the loss weighting ratio. Both accuracy (blue) and F1 score (green) degrade as the similarity weight increases, confirming that strong contrastive regularization is essential for maintaining high discriminative power.

Explore similar work

Robust Fault Detection in Mechanical Multimodal Time Series via Self-Supervised Cross-Modal Reconstruction
Sep 14, 2026Magnus Munk Jensen, Dorte Hammershøi, Rafał Wiśniewski +1Multimodal Robustness Industrial Anomaly Detection
Automation and Control, Aalborg University, Aalborg, Denmark · Intelligent Maintenance and Operations Systems, EPFL, Lausanne, Switzerland
Adaptive Confidence Regularization for Multimodal Failure Detection
Mar 2, 2026Moru Liu, Hao Dong, Olga Fink +1Multimodal Robustness Multimodal Anomaly Detection
Technical University of Munich · ETH Zürich · EPFL +1
ModalImmune: Immunity Driven Unlearning via Self Destructive Training
Feb 18, 2026Rong Fu, WeiZhi Tang, Ziming Wang +6Multimodal Robustness Missing-Modality Learning
University of Macau · Zhejiang University · Renmin University of China +3

Sep 14, 2026J/K move · Enter open · S save

Robust Fault Detection in Mechanical Multimodal Time Series via Self-Supervised Cross-Modal Reconstruction

Magnus Munk Jensen, Dorte Hammershøi, Rafał Wiśniewski, Olga Fink

Automation and Control, Aalborg University, Aalborg, Denmark · Intelligent Maintenance and Operations Systems, EPFL, Lausanne, Switzerland

Multimodal Robustness Industrial Anomaly Detection Reconstruction-Based Anomaly Detection Multimodal Anomaly Detection Time-Series Anomaly Detection

Open paper AskPDF arXiv

Fault detection is essential in industrial systems, enabling early identification of abnormal behaviour and improving safety, reliability, and operational efficiency. Modern systems increasingly rely on heterogeneous sensing modalities that capture complementary aspects of the underlying physical process. However, existing data-driven anomaly detection methods often process each modality independently or use simple feature-level fusion, limiting their ability to exploit cross-modal relationships that characterize normal system behaviour. Their performance also commonly assumes similar training and deployment distributions, whereas real-world operation is affected by changing operating conditions, environmental influences, and system degradation that induce distribution shifts and reduce detection performance, especially in unseen regimes. In this work, we propose a multimodal anomaly detection framework based on cross-modal reconstruction of heterogeneous time-series sensor data. Rather than modeling each modality independently, the framework learns system dynamics by reconstructing each modality from the others, thereby exploiting complementary information across modalities. This integrates information across sensing channels without requiring explicit temporal alignment or identical sampling rates, while improving robustness to sensor noise, missing measurements, and modality-specific disturbances. To address distribution shifts during real-world deployment, anomalies are identified using cross-modal reconstruction error and an adaptive test-time thresholding mechanism that adjusts to changing operating conditions. Experiments on three industrial case studies show strong fault detection performance and substantially improved robustness under out-of-distribution conditions, with the largest gains observed in the most challenging operating regimes.

Abstract

Explore similar work

Robust Fault Detection in Mechanical Multimodal Time Series via Self-Supervised Cross-Modal Reconstruction

Adaptive Confidence Regularization for Multimodal Failure Detection

ModalImmune: Immunity Driven Unlearning via Self Destructive Training