ModalImmune: Immunity Driven Unlearning via Self Destructive Training

Rong Fu; WeiZhi Tang; Ziming Wang; Jia Yee Tan; Zijian Zhang; Zhaolu Kang; Muge Qi; Shuning Zhang

cs.LGFeb 18, 2026

ModalImmune: Immunity Driven Unlearning via Self Destructive Training

Authors: Rong Fu, WeiZhi Tang, Ziming Wang, Jia Yee Tan, Zijian Zhang, Zhaolu Kang, Muge Qi, Shuning Zhang, +1 more

Organizations: University of Macau · Zhejiang University · Renmin University of China · University of Pennsylvania · Peking University · Tsinghua University

Abstract

Multimodal systems are vulnerable to partial or complete loss of input channels at deployment, which undermines reliability in real-world settings. This paper presents ModalImmune, a training framework that enforces modality immunity by intentionally and controllably collapsing selected modality information during training so the model learns joint representations that are robust to destructive modality influence. The framework combines a spectrum-adaptive collapse regularizer, an information-gain guided controller for targeted interventions, curvature-aware gradient masking to stabilize destructive updates, and a certified Neumann-truncated hyper-gradient procedure for automatic meta-parameter adaptation. Empirical evaluation on standard multimodal benchmarks demonstrates that ModalImmune improves resilience to modality removal and corruption while retaining convergence stability and reconstruction capacity.

Figures & tables

Figure 1: Overview of the ModalImmune framework, which treats modality destruction as an active causal intervention . The training strategy alternates between standard reconstruction and Self-Destructive Learning (SDL) built on three key components: Info-Drop Intervention (IDI) , where an EXP3.P bandit controller leverages an information-gain surrogate $\ell_{m}$ to adaptively select the target modality $m^{\star}$ ; Spectral Self-Collapse (SSC) , enforcing an irreversible directional loss through a spectrum-adaptive regularizer $L_{\mathrm{coll}}$ combined with a stable-rank penalty; Curvature-Gated Counter-Gradient (CGC) , which inspects empirical Fisher information to apply a masking multiplier $\rho$ that mitigates destabilizing gradient ascent. Meanwhile, meta-parameters $\xi=\{\lambda,\eta,\kappa\}$ are optimized via Bi-level Hyper-Gradient Descent (BHGD) using a certified Neumann-truncated estimator. This unified protocol ensures the Fusion Hub and Task Output $(C)$ remain robust under counterfactual interventions and adversarial modality collapse.

Figure 2: Training dynamics with explicit phase markers. The horizontal axis shows epochs from 0 to 50. The left vertical axis reports validation Acc2 and the right vertical axis reports MAE. Background shading separates Self-Destructive Learning (SDL) phases from reconstruction-only phases. Shaded bands indicate the 95% confidence interval over three independent seeds. The plot demonstrates that introducing SDL phases does not slow convergence nor induce overfitting on the validation set.

Figure 3: Quantified contribution of principal modules. Bars show absolute drops in validation Acc2 (percentage points) when the named module is ablated. The property-vector pathway produces the largest single decrease, indicating its central role in the model’s predictive power. This visual summary complements the ablation table by converting numeric differences into an immediately interpretable ranking.

Figure 4: BHGD hyperparameter trajectories versus grid-search baselines. Each subplot shows the online evolution of one meta-parameter across training epochs: collapse weight $\lambda$ , negative feedback scale $\kappa$ , and stable-rank penalty $\eta$ . Horizontal dashed lines mark best values found by offline grid search. An inset table compares wall-clock time for hyperparameter selection and demonstrates that BHGD attains comparable or superior meta-parameters while reducing tuning wall time by approximately 38%.

Figure 5: Certified Neumann truncation: error versus compute. The horizontal axis shows truncation depth $K$ from 1 to 10. The left vertical axis reports the relative hyper-gradient error norm $\lVert\widehat{\nabla}-\nabla\rVert/\lVert\nabla\rVert$ expressed in percent. The right vertical axis reports the average additional training time in seconds per hyper-update. A vertical dashed line indicates the online doubling stopping point (here $K=6$ ), at which point the relative error falls below 2% while compute remains practical. The plot supports the claim that the chosen truncation rule yields an efficient and accurate approximation.

Figure 6: Spectral collapse diagnostics. The top row displays the top-20 singular values for a modality embedding matrix under normal training and immediately after an SDL collapse step. Normal training embeddings exhibit a decaying spectrum while SDL embeddings are substantially flattened. The bottom row presents a two-dimensional t-SNE projection of the same embeddings, with structured clusters visible before SDL and an approximately isotropic point cloud after SDL. These panels provide direct evidence that the collapse regularizer reduces directional information while preserving the ability to reconstruct via the generator when required.

Figure 7: implementation across random seeds. The horizontal axis is epoch and the vertical axis is validation Acc2. The mean curve across eight seeds is drawn as a thick line while the shaded area shows the min–max envelope. Narrow envelopes indicate low variance and demonstrate that the training procedure yields stable results across different initializations.

Figure 8: Corruption robustness comparison. Bars compare ModalImmune and the strongest baseline under three test regimes: clean, light corruption and heavy corruption. The plotted metric is Acc2 on the validation set. ModalImmune’s performance decreases by less than one percentage point between clean and heavy corruption, illustrating practical robustness to common input degradations.

Method	CMU-MOSI					CMU-MOSEI
Method	Acc7	Acc2	F1	MAE $\downarrow$	Corr	Acc7	Acc2	F1	MAE $\downarrow$	Corr
HyCon [ 33 ]	46.6	85.2	85.1	0.741	0.779	52.8	85.4	85.6	0.554	0.751
UniMSE [ 34 ]	48.7	86.9	86.4	0.691	0.809	54.4	87.5	87.5	0.523	0.773
ConFEDE [ 35 ]	42.3	85.5	85.5	0.742	0.782	54.9	85.8	85.8	0.522	0.780
MGCL [ 36 ]	49.3	86.7	86.7	0.685	0.707	53.9	86.4	86.4	0.535	0.772
HyDiscGAN [ 37 ]	43.2	86.7	86.3	0.749	0.782	54.4	86.3	86.2	0.533	0.761

Table 1: CMU-MOSI [ 30 ] and CMU-MOSEI [ 31 ] full-modality comparison. Metrics: Acc7 (%), Acc2 (%), F1 (%), MAE (lower better), Corr. Best entries are bold . Mean

\pm

std over 3 runs.

Model	WA (%) $\uparrow$	UA (%) $\uparrow$
TwoStageFT [ 42 ]	74.9	76.1
AdaptiveMixup [ 43 ]	75.4	76.0
EmoAug [ 44 ]	72.7	73.8
MoMKE [ 45 ]	77.9	77.1
APIN [ 46 ]	77.8	78.2
IAM [ 47 ]	74.8	75.6

Table 2: IEMOCAP [ 32 ] full-modality comparison. WA: weighted accuracy; UA: unweighted accuracy. Best entries are bold . Mean

\pm

std over 3 runs.

Available	GCNet [ 50 ]	IMDer [ 51 ]	MoMKE [ 45 ]	LNLN [ 52 ]	EUAR [ 53 ]	CIDer [ 54 ]	RoHyDR [ 55 ]	ModalImmune (ours)
{t}	83.7/83.6/42.3	84.8/84.7/44.8	86.2/86.1/38.1	84.9/84.7/45.1	86.0/86.0/46.1	83.7/83.6/41.3	86.11/84.88/46.2	91.2/91.2/51.2
{v}	56.1/55.7/16.9	61.3/60.8/22.2	54.1/53.7/17.0	52.2/58.9/18.8	64.9/64.9/23.6	57.8/42.3/15.5	63.07/62.31/22.8	69.7/69.7/25.7
{a}	56.1/54.5/16.6	62.0/62.2/22.0	59.3/59.0/18.4	52.2/58.9/18.0	63.0/62.3/23.2	57.8/43.2/15.2	63.89/62.73/23.1	70.2/70.2/26.2
{t,v}	84.3/84.2/43.4	85.5/85.4/45.3	86.5/86.4/37.5	84.3/84.6/44.6	86.2/86.2/45.5	83.8/83.8/42.1	86.54/86.37/47.0	92.2/92.2/52.7
{t,a}	84.3/84.2/43.4	85.4/85.3/45.0	86.5/86.4/38.6	84.9/85.2/45.1	86.1/86.1/44.7	83.8/83.8/41.7	86.42/86.04/46.8	92.7/92.7/53.2
{v,a}	62.0/61.9/17.2	63.6/63.4/23.8	59.6/59.6/20.1	52.2/58.9/18.8	66.1/65.8/24.2	57.8/44.0/15.5	64.59/63.02/24.0	71.7/71.7/27.2

Table 3: Fixed missing-modality results on CMU-MOSI [ 30 ] . Each cell reports Acc2 / F1 / Acc7. Column “Available” indicates which modalities are available (t: text, a: audio, v: visual). Best results are bold . Mean

\pm

std over 3 runs.

Missing rate $\eta$	GCNet [ 50 ]	IMDer [ 51 ]	MoMKE [ 45 ]	LNLN [ 52 ]	EUAR [ 53 ]	CIDer [ 54 ]	RoHyDR [ 55 ]	ModalImmune (ours)
0.1	82.4/82.2/41.9	83.3/83.2/43.0	83.6/83.6/35.5	81.1/82.0/42.0	84.1/84.1/43.8	81.1/79.6/39.4	84.96/83.04/43.5	91.2/91.2/49.7
0.2	79.6/79.3/38.9	80.9/80.8/40.7	80.7/80.7/33.7	78.0/79.5/39.5	81.9/81.9/41.5	78.5/75.6/36.7	83.07/81.70/41.2	88.7/88.7/47.7
0.3	76.7/76.5/35.9	78.5/78.4/38.4	77.8/77.7/31.9	74.8/77.0/36.9	79.8/79.7/39.2	76.0/71.6/34.0	81.87/80.07/39.1	86.7/86.7/45.7
0.4	73.6/73.2/33.0	76.0/75.9/36.0	74.7/74.6/29.8	71.6/74.4/34.3	77.4/77.3/36.9	73.4/67.4/31.2	80.76/77.03/36.5	84.2/84.2/42.7
0.5	70.4/69.9/30.1	73.5/73.4/33.6	71.6/71.4/27.8	68.4/71.8/31.7	75.1/74.9/34.7	70.8/63.3/28.5	79.97/76.87/34.9	81.7/81.7/40.0
0.6	67.3/66.7/27.2	71.0/70.9/31.2	68.5/68.3/25.8	65.2/69.2/29.0	72.8/72.6/32.5	68.2/59.1/25.8	78.03/74.83/32.4	79.2/79.2/37.7

Table 4: Varying global missing rate

\eta

on CMU-MOSI [ 30 ] . Each cell reports Acc2 / F1 / Acc7. Best results are bold . Mean

\pm

std over 3 runs.

Variant	FIX (Acc2 / F1 / Acc7)	MR (Acc2 / F1 / Acc7)
w/o property embedding pathway	77.7 / 77.5 / 34.2	80.4 / 80.2 / 38.0
w/o reconstruction module	78.9 / 78.7 / 35.0	81.6 / 81.5 / 40.3
w/o mirror involution	79.2 / 79.0 / 35.3	81.8 / 81.7 / 40.5
w/o fusion module	80.3 / 80.2 / 36.5	82.7 / 82.6 / 41.6
w/o property pathway + reconstruction	76.6 / 76.4 / 33.4	79.3 / 79.1 / 37.3
w/o all modules	74.8 / 74.6 / 31.7	77.5 / 77.3 / 34.2

Table 5: Ablation study of ModalImmune on CMU-MOSI [ 30 ] . Each entry reports Acc2 / F1 / Acc7 for FIX and MR regimes. Best results are bold . Mean

\pm

std over 3 runs.

Condition	FIX	MR
With corruption	80.6 / 80.6 / 38.6	81.9 / 81.9 / 40.9
Clean (no corruption)	81.5 / 81.5 / 39.7	82.6 / 82.6 / 41.3

Table 6: Robustness to synthetic corruption on CMU-MOSI [ 30 ] (Acc2 / F1 / Acc7). Best results are bold . Mean

\pm

std over 3 runs.

Hyperparameter	Range	Acc2 (%)	Default	Significance
Collapse weight ( $\lambda$ )	[0.1, 1.0]	81.3–81.7	0.5	$p<0.01$
Negative feedback scale ( $\kappa$ )	[0.1, 1.0]	81.4–81.6	0.5	$p<0.01$
Stable-rank penalty ( $\eta$ )	[0.05, 0.2]	81.5–81.6	0.1	$p<0.01$
SDL probability ( $p_{\text{sdl}}$ )	[0.1, 0.3]	81.4–81.5	0.2	$p<0.01$
Fusion heads	[4, 16]	81.4–81.5	8	$p<0.01$
Property dimension	[64, 256]	81.4–81.6	128	$p<0.01$

Table 7: Hyperparameter sensitivity analysis of ModalImmune on CMU-MOSI. Each parameter is varied while keeping others at default values. Best results are bold . Mean

\pm

std over 3 runs.

Meta-parameter	Grid optimum	BHGD final	$\|\mathrm{Diff}\|$
Collapse weight $\lambda$	0.50	0.48	0.02
Negative-feedback scale $\kappa$	0.50	0.52	0.02
Stable-rank penalty $\eta$	0.10	0.07	0.03

Table 8: BHGD terminal estimates versus grid-search optima on CMU-MOSI. Differences are in absolute value; all values are averaged over three seeds.

Method	Acc2	F1	Acc7
RoHyDR [ 55 ]	76.3	76.2	36.7
ModalImmune (ours)	83.1	82.9	44.6

Table 9: Zero-shot robustness on CMU-MOSI. Training: {text, audio}; Testing: {text, visual}. Mean

\pm

std over 3 runs.

Metric	$\rho$	$p$ -value
Stable-rank ↓ vs. Accuracy ↑	0.81	$<0.01$

Table 10: Correlation between stable-rank reduction and missing-modality accuracy across epochs.

Method	Parameters (M)	Peak Memory (GB)	Latency (ms)
Baseline	24.80	6.9	13.9
ModalImmune	26.02 ( $\Delta$ 4.9%)	7.3 ( $\Delta$ 5.8%)	14.7 ( $\Delta$ 5.8%)

Table 11: Resource comparison on CMU-MOSI. “

\Delta

” denotes relative increase over baseline.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Constant	Meaning	Empirical value (mean of 3 runs)
$C$	Bernstein matrix concentration constant	$0.47$
$\sigma_{x}$	sub-Gaussian parameter of embedding $z$	$1.8$
$\beta$	Hessian spectral norm $\\|H\\|_{2}$	$6.1$
$\alpha$	inner learning rate	$5\times 10^{-3}$
$G$	gradient norm bound $\mathbb{E}\\|g_{0}\\|_{2}$	$4.2$
$V$	trace bound $\operatorname{tr}(\Sigma_{g_{0}})$	$190$

Appendix

Table 12: Empirical instantiation of ModalImmune theoretical constants on CMU-MOSI.

Explore similar work

Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness
Oct 3, 2026Songyuan Sui, Zhen Tan, Mohan Zhang +3Missing-Modality Learning Multimodal Robustness
Rice University · Stevens Institute of Technology · University of North Carolina at Chapel Hill
General Incomplete Multimodal Learning via Dynamic Quality Perception
Jul 8, 2026Xiangyu Meng, Shicai WeiMissing-Modality Learning Multimodal Robustness
University of Electronic Science and Technology of China
Unsupervised Learning for Missing Modalities in Multimodal Learning
Jun 14, 2026Hassan Ismkhan, Hamid BouchahciaMissing-Modality Learning Unsupervised Learning
Bournemouth University Bournemouth, UK

Oct 3, 2026J/K move · Enter open · S save

Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness

Songyuan Sui, Zhen Tan, Mohan Zhang, Rana Muhammad Shahroz Khan+2

Rice University · Stevens Institute of Technology · University of North Carolina at Chapel Hill

Missing-Modality Learning Multimodal Robustness

Open paper AskPDF arXiv

Missing modality remains a longstanding challenge in multimodal learning. Existing methods typically address this issue through modality recovery or adaptive strategies. However, they overlook models' internal cross-modal dependencies formed during multimodal training, which later impair robustness. We systematically characterize a counterintuitive deployment-time failure mode: models trained on full modalities can underperform unimodal models when one modality is missing at inference time. This pattern appears across diverse architectures, such as fusion models, CLIP-style two-tower models, and vision-language models. We show that such degradation is closely associated with learned cross-modal dependencies in the principal parameter subspaces. Multimodal training induces structured rotations of these subspaces, particularly in cross-modal interaction layers. These rotations are associated with reduced task-aligned margins and larger task-aware representation harm under missing-modality inputs. We propose Geodesic Unlearning (GU), a lightweight parameter-editing method that leverages Grassmannian subspace geometry for structured subspace correction to improve missing-modality robustness. It rotates the principal input subspace toward a unimodal reference along a geodesic path. We prove that this correction minimizes the distance to the reference within a fixed subspace-distance budget. Experiments across architectures and datasets show that GU improves performance under missing-modality inference while preserving full-modality accuracy, outperforming strong missing-modality robustness baselines. These findings support a geometric view of deployment-time missing-modality degradation and suggest localized subspace editing as a practical route for robustness correction.

Abstract

Explore similar work

Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness

General Incomplete Multimodal Learning via Dynamic Quality Perception

Unsupervised Learning for Missing Modalities in Multimodal Learning