cs.AIOct 3, 2026

Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness

Authors: Songyuan Sui, Zhen Tan, Mohan Zhang, Rana Muhammad Shahroz Khan, Xia Hu, Tianlong Chen

Organizations: Rice University · Stevens Institute of Technology · University of North Carolina at Chapel Hill

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Missing modality remains a longstanding challenge in multimodal learning. Existing methods typically address this issue through modality recovery or adaptive strategies. However, they overlook models' internal cross-modal dependencies formed during multimodal training, which later impair robustness. We systematically characterize a counterintuitive deployment-time failure mode: models trained on full modalities can underperform unimodal models when one modality is missing at inference time. This pattern appears across diverse architectures, such as fusion models, CLIP-style two-tower models, and vision-language models. We show that such degradation is closely associated with learned cross-modal dependencies in the principal parameter subspaces. Multimodal training induces structured rotations of these subspaces, particularly in cross-modal interaction layers. These rotations are associated with reduced task-aligned margins and larger task-aware representation harm under missing-modality inputs. We propose Geodesic Unlearning (GU), a lightweight parameter-editing method that leverages Grassmannian subspace geometry for structured subspace correction to improve missing-modality robustness. It rotates the principal input subspace toward a unimodal reference along a geodesic path. We prove that this correction minimizes the distance to the reference within a fixed subspace-distance budget. Experiments across architectures and datasets show that GU improves performance under missing-modality inference while preserving full-modality accuracy, outperforming strong missing-modality robustness baselines. These findings support a geometric view of deployment-time missing-modality degradation and suggest localized subspace editing as a practical route for robustness correction.

Figures & tables

Figure 1: Missing-modality degradation across architectures and datasets. F1 scores are reported for TI-TI, T-T, and TI-T settings. In 26 of the 27 settings, we observe TI-TI>T-T>TI-T\mathrm{TI\text{-}TI}>\mathrm{T\text{-}T}>\mathrm{TI\text{-}T} , revealing a recurring deployment-time missing-modality degradation pattern.
Model FamilyBackbone(s)Variants
FusionRoBERTa [ 33 ] + ResNet50 [ 28 ]Concat, Gate, Cross-Attention
Fusion (strong encoders)Qwen [ 62 ] + OpenCLIP [ 24 ]Concat, Gate, Cross-Attention
Two-towerOpenCLIPMean Pooling, Late Concat
VLMBLIP-2 [ 31 ]Classifier Head
Table 1: Representative model families used in our analysis. We cover distinct multimodal integration mechanisms, including explicit fusion architectures, aligned two-tower encoders, and VLMs.
Figure 2: Principal subspace rotation correlates with missing-modality degradation. (a) Within-family z-score normalization reveals a global positive association. (b) Raw within-family measurements show consistent positive associations without normalization.
Figure 3: Stage-wise distribution of task-aware representation harm. The final-error versus final-correct harm gap appears at cross-modal interaction stages and often persists near the classifier.
DatasetNull-tokenSMILFlex-MoEDyMoInterpolationGU (Ours)
HM57.6356.6047.5848.8258.5060.88\mathbf{60.88} (+2.38)
IU-XRay80.0079.2171.0279.1980.1481.41\mathbf{81.41} (+1.27)
SNLI-VE66.3572.2070.7461.3067.2974.41\mathbf{74.41} (+2.21)
Table 2: Main missing-image performance (TI-T, F1 ↑\uparrow ). Values are three-run means. Underline and bold denote the second-best and best results, and parentheses show gains over the second best.
Figure 4: Representation structure and ablation analysis. (a) On SNLI-VE, GU and layer interpolation preserve similar broad UMAP layouts. (b) On IU-XRay, target-layer selection has the largest effect on robustness recovery.
Family#VarTI-TTI-TIΔ\Delta TΔ\Delta TI
  Interp.GUInterp.GUInterp.GUInterp.GU
Fusion371.3473.86\mathbf{73.86}86.3386.37\mathbf{86.37}0.983.49\mathbf{3.49}−0.13-0.13−0.09\mathbf{-0.09}
Fusion (str.)372.3473.06\mathbf{73.06}75.1675.95\mathbf{75.95}0.961.69\mathbf{1.69}−1.13-1.13−0.34\mathbf{-0.34}
CLIP274.5074.68\mathbf{74.68}76.1376.70\mathbf{76.70}1.041.22\mathbf{1.22}−1.21-1.21−0.64\mathbf{-0.64}
VLM170.6570.65\mathbf{70.65}81.7281.73\mathbf{81.73}0.010.01\mathbf{0.01}0.00+0.01\mathbf{+0.01}
Table 3: Family-level robustness–preservation trade-off on SNLI-VE. ΔT\Delta T and ΔTI\Delta TI denote changes from the original TI model in TI-T and TI-TI, respectively.
ArchitectureMethod / PhaseWall-clock (s)Peak Mem. (GiB)ΔT\Delta TΔTI\Delta TI
Fusion CAPost-hoc LoRA1812.302.090+6.02-1.62
Dual CE2592.0313.954+13.57-0.94
T Reference1849.095.241––
GU Edit297.191.744––
GU (Total)2146.285.241+8.06-0.11
CLIP MPPost-hoc LoRA1910.394.423+1.03-3.66
Table 4: Offline cost and robustness–preservation trade-off on SNLI-VE. Including reference training, GU uses less time and peak GPU memory than Dual CE.
Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

FamilyDatasetVariantTI-TIT-TTI-TΔmiss\Delta_{\mathrm{miss}}
FusionHMConcat69.46±0.3269.46\pm 0.3268.99±0.2568.99\pm 0.2568.96±0.4468.96\pm 0.440.50±0.140.50\pm 0.14
HMGate67.98±0.2867.98\pm 0.2867.86±0.2367.86\pm 0.2367.57±0.3867.57\pm 0.380.41±0.120.41\pm 0.12
HMCross-Attn68.82±0.4868.82\pm 0.4868.41±0.4068.41\pm 0.4057.63±0.8857.63\pm 0.8811.19±0.4511.19\pm 0.45
IXConcat86.41±0.2286.41\pm 0.2281.94±0.1881.94\pm 0.1880.70±0.3880.70\pm 0.385.71±0.185.71\pm 0.18
IXGate82.79±0.2082.79\pm 0.2081.18±0.1781.18\pm 0.1780.76±0.3380.76\pm 0.332.03±0.152.03\pm 0.15
IXCross-Attn84.57±0.2884.57\pm 0.2880.97±0.2380.97\pm 0.2380.00±0.4780.00\pm 0.474.57±0.224.57\pm 0.22
Appendix
Table 5: F1 performance under three training–testing regimes. TI-T consistently underperforms T-T across model families and datasets.
VariantLayerΔharm\Delta_{\mathrm{harm}}Δviol\Delta_{\mathrm{viol}}
Concattext_proj_out0.51870.6170
image_proj_out0.00000.0000
fusion_mlp0_concat_input0.51870.6170
fusion_mlp3_output0.56170.7308
head_last_linear_input0.56170.7308
Gatetext_proj_out0.38810.5735
Appendix
Table 6: Layer-wise task-aware representational harm for fusion models on SNLI-VE. Harm gaps appear at cross-modal interaction stages and often persist near the classifier.
VariantLayerOverlap (In / Out)Mean Angle (In / Out)Fro. Dist.
Concattext_proj0.9666 / 0.96668.3333 / 8.31501.4884
image_proj0.8368 / 0.840919.2640 / 18.72822.5745
fusion_mlp00.9307 / 0.931411.4472 / 11.31101.8050
fusion_mlp30.9578 / 0.95734.2398 / 4.28951.3692
head_last_linear0.9991 / 1.00001.7072 / 0.00000.0511
Gatetext_proj0.9715 / 0.97147.7292 / 7.73251.4133
Appendix
Table 7: Layer-wise parameter subspace rotation for fusion models on SNLI-VE. Projection and cross-modal interaction layers generally exhibit larger rotations than the final classifier.
SplitHateful MemesSNLI-VEIU-XRay
Train8,500353,4822,928
Dev50011,898366
Test1,00011,937366
Appendix
Table 8: Dataset split statistics used in our experiments.
DatasetGUSMIL OuterSMIL InnerFlex-MoEDyMo
Hateful Memes3.0×10−53.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}1.0×10−31.0{\times}10^{-3}3.0×10−53.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}
SNLI-VE3.0×10−53.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}1.0×10−31.0{\times}10^{-3}3.0×10−53.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}
IU-XRay3.0×10−53.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}1.0×10−31.0{\times}10^{-3}3.0×10−53.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}
Appendix
Table 9: Learning rates for missing-modality baselines. Unspecified hyperparameters follow the official codebase defaults.
DatasetConcatCross-AttentionGate
Hateful Memes3.0×10−53.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}
SNLI-VE3.0×10−53.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}
IU-XRay3.0×10−53.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}
Appendix
Table 10: Learning rates for basic fusion models. A single global learning rate is used for all trainable parameters.
DatasetVariantText/ImageFusionBottleneck/Head
Hateful MemesConcat2.0×10−52.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}
Cross-Attention2.0×10−52.0{\times}10^{-5}2.0×10−52.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}
Gate1.0×10−51.0{\times}10^{-5}2.0×10−52.0{\times}10^{-5}2.0×10−52.0{\times}10^{-5}
SNLI-VEConcat2.0×10−52.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}
Cross-Attention1.0×10−51.0{\times}10^{-5}1.0×10−51.0{\times}10^{-5}1.0×10−51.0{\times}10^{-5}
Gate2.0×10−52.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}
Appendix
Table 11: Module-wise learning rates for fusion models with stronger encoders. Text/image denotes both text_encoder.proj and image_encoder.proj ; bottleneck/head denotes both modules.
DatasetVariantText/ImageFusionBottleneck/Head
Hateful MemesLate Concat1.0×10−51.0{\times}10^{-5}1.0×10−51.0{\times}10^{-5}1.0×10−51.0{\times}10^{-5}
Mean Pooling1.0×10−51.0{\times}10^{-5}–1.0×10−51.0{\times}10^{-5}
SNLI-VELate Concat1.0×10−51.0{\times}10^{-5}1.0×10−51.0{\times}10^{-5}1.0×10−51.0{\times}10^{-5}
Mean Pooling1.0×10−51.0{\times}10^{-5}–1.0×10−51.0{\times}10^{-5}
IU-XRayLate Concat1.0×10−51.0{\times}10^{-5}1.0×10−51.0{\times}10^{-5}1.0×10−51.0{\times}10^{-5}
Mean Pooling1.0×10−51.0{\times}10^{-5}–1.0×10−51.0{\times}10^{-5}
Appendix
Table 12: Module-wise learning rates for CLIP-style two-tower models. Text/image denotes both projection modules; bottleneck/head denotes both modules.
DatasetQ-FormerProjectionBottleneckHead
Hateful Memes5.0×10−65.0{\times}10^{-6}5.0×10−65.0{\times}10^{-6}2.0×10−52.0{\times}10^{-5}2.0×10−52.0{\times}10^{-5}
SNLI-VE1.0×10−51.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}
IU-XRay1.0×10−51.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}3.0×10−53.0{\times}10^{-5}
Appendix
Table 13: Module-wise learning rates for BLIP-2 based VLM models. Q-Former denotes both vlm_encoder.qformer_embeddings and vlm_encoder.backbone.qformer ; projection denotes both vlm_encoder.backbone.language_projection and vlm_encoder.proj .
Model FamilyVariantTarget Layer
Fusion / Fusion (strong)Concatfusion_mlp0
Fusion / Fusion (strong)Gategate_mlp0
Fusion / Fusion (strong)Cross-Attentionattn_v_from_t_in_proj
Two-towerLate Concatfusion_mlp0
Two-towerMean Poolingtext_proj
VLMBLIP-2 Classifier Headqformer_crossattn0_out
Appendix
Table 14: Target layers edited by Geodesic Unlearning. We use η=0.75\eta=0.75 and k=8k=8 for all configurations.
MethodHMIU-XRaySNLI-VE
Null-token57.63±0.8857.63\pm 0.8880.00±0.4780.00\pm 0.4766.35±0.3866.35\pm 0.38
SMIL56.60±0.7456.60\pm 0.7479.21±0.6879.21\pm 0.6872.20‾±0.52\underline{72.20}\pm 0.52
Flex-MoE47.58±1.3847.58\pm 1.3871.02±0.8271.02\pm 0.8270.74±0.6470.74\pm 0.64
DyMo48.82±0.9348.82\pm 0.9379.19±0.7779.19\pm 0.7761.30±0.7061.30\pm 0.70
Interpolation58.50‾±0.91\underline{58.50}\pm 0.9180.14‾±0.49\underline{80.14}\pm 0.4967.29±0.6167.29\pm 0.61
GU (Ours)60.88±0.73\mathbf{60.88}\pm 0.7381.41±0.63\mathbf{81.41}\pm 0.6374.41±0.77\mathbf{74.41}\pm 0.77
Appendix
Table 15: Supplementary results for Table 2 . We report mean ±\pm sample standard deviation over three independent runs. Underline and bold denote the second-best and best mean results, respectively.
MetricMethodFusionFusion (str.)CLIPVLM
TI-TInterp.71.34±0.4671.34\pm 0.4672.34±0.3872.34\pm 0.3874.50±0.4474.50\pm 0.4470.65±0.7970.65\pm 0.79
 GU73.86±0.56\mathbf{73.86}\pm 0.5673.06±0.43\mathbf{73.06}\pm 0.4374.68±0.52\mathbf{74.68}\pm 0.5270.65±0.68\mathbf{70.65}\pm 0.68
TI-TIInterp.86.33±0.1886.33\pm 0.1875.16±0.3775.16\pm 0.3776.13±0.3876.13\pm 0.3881.72±0.3081.72\pm 0.30
 GU86.37±0.14\mathbf{86.37}\pm 0.1475.95±0.31\mathbf{75.95}\pm 0.3176.70±0.31\mathbf{76.70}\pm 0.3181.73±0.18\mathbf{81.73}\pm 0.18
Δ\Delta TInterp.0.98±0.300.98\pm 0.300.96±0.220.96\pm 0.221.04±0.351.04\pm 0.350.01±0.100.01\pm 0.10
 GU3.49±0.26\mathbf{3.49}\pm 0.261.69±0.20\mathbf{1.69}\pm 0.201.22±0.33\mathbf{1.22}\pm 0.330.01±0.04\mathbf{0.01}\pm 0.04
Appendix
Table 16: Supplementary robustness–preservation results for Table 3 on SNLI-VE. Results are mean ±\pm sample standard deviation over three independent runs.
DatasetComparison10-seed paired Δ\Delta F1GU winsRaw ppHolm pp
HMGU −- Null+3.19±1.14+3.19\pm 1.149/100.0039060.011719
HMGU −- Interp.+2.41±0.77+2.41\pm 0.7710/100.0019530.011719
IXGU −- Null+1.51±1.71+1.51\pm 1.719/100.0332030.033203
IXGU −- Interp.+1.34±0.45+1.34\pm 0.4510/100.0019530.011719
SVGU −- Null+8.09±0.22+8.09\pm 0.2210/100.0019530.011719
SVGU −- SMIL+2.21±0.68+2.21\pm 0.6810/100.0019530.011719
Appendix
Table 17: Ten-seed paired tests for GU against Null and the strongest competing method in Table 2 . All six comparisons remain significant after Holm correction.
MethodTI-TCKA to orig.Error-margin gainCorr./regr. rate (%)
Orig. TI-T66.35±0.3866.35\pm 0.381.0001.000000/00/0
Interp.67.29±0.6167.29\pm 0.610.988±0.0010.988\pm 0.001+0.06±0.01+0.06\pm 0.0111.8±0.6 / 4.5±0.211.8\pm 0.6\,/\,4.5\pm 0.2
GU (Ours)74.41±0.7774.41\pm 0.770.991±0.0010.991\pm 0.001+0.30±0.02+0.30\pm 0.0234.0±1.3 / 5.1±0.334.0\pm 1.3\,/\,5.1\pm 0.3
Appendix
Table 18: Quantitative analysis of Figure 4 (a) on SNLI-VE Fusion Cross-Attention. Both edits preserve high representation similarity, while GU yields larger task-level improvements.
FamilyVariantTarget LayerMethodTI-TTI-TIΔT\Delta TΔTI\Delta TI
FusionConcatfusion_mlp0Interp.74.86±.6874.86\pm.6886.10±.1686.10\pm.161.82±.481.82\pm.48−0.12±.10-0.12\pm.10
 Concatfusion_mlp0GU75.11±.56\mathbf{75.11}\pm.5686.14±.12\mathbf{86.14}\pm.122.07±.32\mathbf{2.07}\pm.32−0.08±.05\mathbf{-0.08}\pm.05
 Gategate_mlp0Interp.71.88±.4771.88\pm.4786.40±.1886.40\pm.180.18±.240.18\pm.24−0.13±.11-0.13\pm.11
 Gategate_mlp0GU72.05±.35\mathbf{72.05}\pm.3586.44±.12\mathbf{86.44}\pm.120.35±.07\mathbf{0.35}\pm.07−0.09±.04\mathbf{-0.09}\pm.04
 Cross-Attnattn_v ←\leftarrow t_inInterp.67.29±.6167.29\pm.6186.50±.2486.50\pm.240.94±.290.94\pm.29−0.14±.15-0.14\pm.15
 Cross-Attnattn_v ←\leftarrow t_inGU74.41±.77\mathbf{74.41}\pm.7786.53±.17\mathbf{86.53}\pm.178.06±.40\mathbf{8.06}\pm.40−0.11±.08\mathbf{-0.11}\pm.08
Appendix
Table 19: Per-variant robustness–preservation results on SNLI-VE. GU generally provides stronger recovery with comparable or better full-modality preservation than interpolation.
MethodFusion ConcatCLIP Late ConcatCLIP Mean Pooling
Hateful Memes
Null-token68.96±0.44 / 69.46±0.3268.96\pm 0.44\,/\,69.46\pm 0.3266.58±0.41 / 67.31±0.2766.58\pm 0.41\,/\,67.31\pm 0.2766.87±0.24 / 67.14±0.2066.87\pm 0.24\,/\,67.14\pm 0.20
MoRA– / –65.64±0.28 / 65.39±0.8365.64\pm 0.28\,/\,65.39\pm 0.8360.45±6.17 / 64.98±0.8760.45\pm 6.17\,/\,64.98\pm 0.87
OGM-GE66.71±0.56 / 67.07±0.7866.71\pm 0.56\,/\,67.07\pm 0.7866.07±0.47 / 65.85±0.4166.07\pm 0.47\,/\,65.85\pm 0.4164.51±0.66 / 65.68±1.1064.51\pm 0.66\,/\,65.68\pm 1.10
GU69.01±0.24 / 69.32±0.3069.01\pm 0.24\,/\,69.32\pm 0.3067.03±0.17 / 67.26±0.3167.03\pm 0.17\,/\,67.26\pm 0.3166.96±0.34 / 67.09±0.2266.96\pm 0.34\,/\,67.09\pm 0.22
GU Δ\Delta TI−0.14-0.14−0.05-0.05−0.05-0.05
Appendix
Table 20: Cross-architecture comparison with MoRA and OGM-GE. GU achieves the highest mean TI-T in all nine evaluated settings. Entries report TI-T / TI-TI F1. Δ\Delta TI denotes GU’s mean TI-TI change relative to Null-token (GU minus Null-token).
ArchitectureMethodTI-TTI-TI
Fusion CANull-token66.35±0.3866.35\pm 0.3886.64±0.1086.64\pm 0.10
LoRA72.37±0.2572.37\pm 0.2585.02±0.3585.02\pm 0.35
Dual CE79.92±0.2379.92\pm 0.2385.70±0.2285.70\pm 0.22
GU74.41±0.7774.41\pm 0.7786.53±0.1786.53\pm 0.17
Dual CE+GU80.09±0.3080.09\pm 0.3086.51±0.1986.51\pm 0.19
CLIP MPNull-token74.15±0.1874.15\pm 0.1876.62±0.0676.62\pm 0.06
Appendix
Table 21: Comparison with post-hoc LoRA and Dual CE on SNLI-VE. Entries report F1 as mean ±\pm sample standard deviation over three independent runs.
 Angle onlyHarm onlyAngle+harm
Selected layerattn_t_from_v_out_projtext_projattn_v_from_t_in_proj
TI-T80.34±0.6680.34\pm 0.6680.72±0.4080.72\pm 0.4081.41±0.63\mathbf{81.41}\pm 0.63
TI-TI83.96±0.3783.96\pm 0.3784.21±0.3284.21\pm 0.3284.44±0.28\mathbf{84.44}\pm 0.28
ΔT\Delta T+0.34+0.34+0.72+0.72+1.41\mathbf{+1.41}
ΔTI\Delta TI−0.61-0.61−0.36-0.36−0.13\mathbf{-0.13}
Appendix
Table 22: Target-layer selection on IU-XRay Fusion Cross-Attention. Joint angle–harm selection yields the strongest recovery and preservation.
DatasetWeightLinearGU (Ours)
HM58.50±0.91 / 66.52±0.3358.50\pm 0.91\,/\,66.52\pm 0.3358.07±1.03 / 66.62±0.2158.07\pm 1.03\,/\,66.62\pm 0.2160.88±0.73 / 67.55±0.26\mathbf{60.88}\pm 0.73\,/\,\mathbf{67.55}\pm 0.26
IX80.14±0.49 / 82.67±0.5680.14\pm 0.49\,/\,82.67\pm 0.5679.83±0.75 / 82.39±0.3679.83\pm 0.75\,/\,82.39\pm 0.3681.41±0.63 / 84.44±0.28\mathbf{81.41}\pm 0.63\,/\,\mathbf{84.44}\pm 0.28
SV67.29±0.61 / 86.50±0.2467.29\pm 0.61\,/\,86.50\pm 0.2466.84±1.29 / 86.50±0.0766.84\pm 1.29\,/\,86.50\pm 0.0774.41±0.77 / 86.53±0.17\mathbf{74.41}\pm 0.77\,/\,\mathbf{86.53}\pm 0.17
Appendix
Table 23: Controlled comparison of editing strategies on Fusion Cross-Attention. GU achieves the highest mean TI-T and TI-TI across all three datasets. Entries report TI-T / TI-TI F1.
MethodTI-T Acc.TI-TI Acc.
T Refer. (T-T)47.17±0.3147.17\pm 0.31–
Shuffle-I–39.85±1.9739.85\pm 1.97
Orig. TI38.94±1.9238.94\pm 1.9250.79±0.7150.79\pm 0.71
Interp.39.80±1.2139.80\pm 1.2143.21±1.4443.21\pm 1.44
LoRA43.52±0.60\mathbf{43.52}\pm 0.6045.01±1.7245.01\pm 1.72
GU (Ours)41.07±1.7941.07\pm 1.7947.40±1.19\mathbf{47.40}\pm 1.19
Appendix
Table 24: A-OKVQA multiple-choice accuracy (%) over ten runs. GU provides the strongest full-modality preservation among the evaluated interventions. Bold marks the best result among Interpolation, LoRA, and GU.
MethodEval.HMIU-XRaySNLI-VE
Orig. TITI-TI68.82±0.4868.82\pm 0.4884.57±0.2884.57\pm 0.2886.64±0.1086.64\pm 0.10
I Refer.I-I66.58±0.1566.58\pm 0.1575.32±0.4175.32\pm 0.4168.74±0.0068.74\pm 0.00
Orig. TITI-I63.32±0.8863.32\pm 0.8873.58±0.8573.58\pm 0.8564.48±0.9764.48\pm 0.97
Interp.TI-I64.10±0.2464.10\pm 0.2473.70±0.7673.70\pm 0.7665.87±0.4665.87\pm 0.46
Interp.TI-TI68.30±0.2868.30\pm 0.2884.39±0.3684.39\pm 0.3686.01±0.1586.01\pm 0.15
GU (Ours)TI-I64.34±0.2064.34\pm 0.2074.42±0.2274.42\pm 0.2266.61±0.2766.61\pm 0.27
Appendix
Table 25: Missing-text evaluation with Fusion Cross-Attention. GU improves TI-I while largely preserving full-modality performance across all three datasets.

Explore similar work

CardsList
Jun 14, 2026

Unsupervised Learning for Missing Modalities in Multimodal Learning

This paper addresses the missing-modality challenge in multi-modal learning by introducing Unsupervised Learning for Missing Modalities in Multi-Modal Learning (UL4M4), a flexible framework that imputes missing feature embeddings in a task-independent manner before supervised prediction. We propose modality-specific normalization and a novel partial-modality distance metric to enable fair clustering of incomplete observations, capturing cross-modal structures while preserving scale-invariance across varying dimensionalities and modality counts. Cluster centers from this unsupervised stage guide an iterative greedy imputation process for any missing modalities during training or inference, supporting arbitrary numbers of modalities and arbitrary missing patterns per sample. The imputation module is lightweight, uses frozen encoders, and decouples from the downstream task, allowing easy integration with any fusion/prediction architecture. Extensive experiments under diverse and highly incomplete regimes demonstrate UL4M4's robustness, achieving, to the best of our knowledge, the first consistent F1-Micro scores above 0.7 on challenging missing configurations even when more than 50% of modality slots are missing. Results are also stable across cluster sizes and significantly outperform state-of-the-art baselines. Code is available here: https://github.com/h-ismkhan/Multimodal-Learning-with-Missing-Modalities-via-Unsupervised-Learning.

Hassan Ismkhan, Hamid Bouchahcia

Bournemouth University Bournemouth, UK

Missing-Modality LearningUnsupervised LearningLearning with Missing DataMultimodal Learning
AskPDF
Jul 8, 2026

General Incomplete Multimodal Learning via Dynamic Quality Perception

Multimodal learning robust to missing modalities is essential for real-world applications. Existing methods mainly focus on inter-modality missing, where entire modalities are absent, while overlooking intra-modality degradation, where modalities are present but severely corrupted. In practice, these two types of missing often coexist, making existing approaches ineffective. To address this limitation, we propose General Incomplete Multimodal Learning (GIML), a unified framework that simultaneously handles both inter-modality missing and intra-modality degradation through dynamic quality perception. Specifically, GIML models heterogeneous missing patterns as continuous modality information degradation, enabling degradation-aware adaptive fusion. To achieve reliable quality perception, we introduce a Noise-aware Quality Estimator that learns the mapping from corrupted features to noise intensity through controlled noise injection. Furthermore, we propose a Noise-Semantic Decoupled module that separates semantic information from noise interference. This improves robustness and generalization to unseen corruption patterns. Extensive experiments across datasets with diverse modality types demonstrate the effectiveness and generality of GIML. Code is available at: https://github.com/Yu-Five/GIML.

Xiangyu Meng, Shicai Wei

University of Electronic Science and Technology of China

Missing-Modality LearningMultimodal RobustnessLearning with Missing DataMultimodal Fusion
AskPDF
Feb 18, 2026

ModalImmune: Immunity Driven Unlearning via Self Destructive Training

Multimodal systems are vulnerable to partial or complete loss of input channels at deployment, which undermines reliability in real-world settings. This paper presents ModalImmune, a training framework that enforces modality immunity by intentionally and controllably collapsing selected modality information during training so the model learns joint representations that are robust to destructive modality influence. The framework combines a spectrum-adaptive collapse regularizer, an information-gain guided controller for targeted interventions, curvature-aware gradient masking to stabilize destructive updates, and a certified Neumann-truncated hyper-gradient procedure for automatic meta-parameter adaptation. Empirical evaluation on standard multimodal benchmarks demonstrates that ModalImmune improves resilience to modality removal and corruption while retaining convergence stability and reconstruction capacity.

Rong Fu, WeiZhi Tang, Ziming Wang +6

University of Macau · Zhejiang University · Renmin University of China +3

Multimodal RobustnessMissing-Modality LearningMultimodal Optimization
AskPDF