Risk-Controlled Correction for Frozen Multimodal Models under Missing Modalities
Organizations: Hanoi University of Science and Technology, Hanoi, Vietnam · Victoria University of Wellington, Wellington, New Zealand
Abstract
A deployed multimodal predictor may face missing modalities after its weights can no longer be changed. This raises a question that training-time repair does not answer: what part of the resulting error is still correctable from the information that remains? We show that this part is characterized by the Bayes gap, the difference between the model's prediction and the Bayes prediction given the observed modalities. For cross-entropy and squared error, excess risk measures this gap in the geometry of the loss; ambiguity in the observed modalities cannot be removed. Thus, a badly degraded prediction is not necessarily a correctable one. Estimating the Bayes prediction normally requires labels. However, when a model observing more modalities is conditionally correct, averaging its outputs conditional on the remaining modalities recovers this target without retrieval labels; with the same fixed blend and neighbors, it is no worse in expected cross-entropy or Brier risk than averaging hard labels. Some such assumption is unavoidable: otherwise the target is not identifiable from unlabeled data alone. GUARD (Gated Utility-Aware Recalibration for Deployment) leaves the deployed predictor unchanged. It estimates a candidate correction by retrieval, learns from deployment-matched labeled data when correction is likely to help, and calibrates a threshold on a held-out split by conformal risk control. Under exchangeability, the marginal probability that GUARD applies a correction whose loss increase exceeds a prescribed tolerance is bounded by a preset level, regardless of the learned score. In controlled experiments, correction gain follows headroom rather than raw error; across nine frozen model-dataset pairs spanning six domains, GUARD retains most of blanket correction's gain within harm budgets that blanket correction often exceeds.
Figures & tables
| Method | Avg. | |||||||
|---|---|---|---|---|---|---|---|---|
| Published under their own training protocols; context, not a controlled comparison | ||||||||
| MCTN ( Pham et al., 2019 ) | 82.6/82.8 | 62.7/54.5 | 62.6/57.1 | 83.5/83.3 | 83.2/83.2 | 63.7/62.7 | 84.2/84.2 | 74.6/72.5 |
| MMIN ( Zhao et al., 2021 ) | 82.3/82.4 | 58.9/59.5 | 59.3/60.0 | 83.7/83.3 | 83.8/83.4 | 63.5/61.9 | 84.3/84.2 | 73.7/73.5 |
| GCNet ( Lian et al., 2023 ) | 83.0/83.2 | 60.2/60.3 | 61.9/61.6 | 84.3/84.4 | 84.3/84.4 | 64.1/57.2 | 85.2/85.1 | 74.7/73.7 |
| IMDer ( Wang et al., 2023b ) | 84.5/84.5 | 63.8/60.6 | 63.9/63.6 | 85.1/85.1 | 85.0/85.0 | 64.9/63.5 | 85.1/85.1 | 76.0/75.3 |
| DiCMoR ( Wang et al., 2023a ) | 84.2/84.3 | 62.9/60.4 | 63.6/63.6 | 85.0/84.9 | 84.9/84.9 | 65.2/64.4 | 85.1/85.1 | 75.8/75.4 |
| Benchmark (model) | Metric | Cond. | frozen | GUARD | Joint harm | Apply (%) |
|---|---|---|---|---|---|---|
| DrugBAN ( datasets) | AUROC | – | – | |||
| OPPORTUNITY (DeepConvLSTM) | wtd. F1 | – | – | |||
| CMU-MOSEI (CMAD) | binary acc. | – | – | |||
| IEMOCAP (MoMKE) | -class WA | – | – | |||
| AVE (AV-att) | -way acc. | – | – | |||
| NinaPro DB5 (sEMG CNN) | accuracy | – | – |
| Without tightening | With tightening | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Benchmark | Blanket | Conf. | LTT | Learned | Conf. | Random | LTT | GUARD | |
| CMU-MOSEI | / ∗ | / | / | / ∗ | / | / | / | / | / |
| IEMOCAP | / ∗ | / | / | / | / | / | / | / | / |
| OPPORTUNITY | / ∗ | / ∗ | / | / | / | / | / ∗ | / | / |
| AVE | / ∗ | / | / | / | / | / | / | / | / |
| NinaPro DB5 | / | / | / | / | / | / | / | / | / |
| richer-mask | gated gain | ||
|---|---|---|---|
| Dataset, split | accuracy | hard-label | cross-mask |
| Human, random | |||
| BindingDB, random | |||
| BioSNAP, random | |||
| OPPORTUNITY | |||
| BioSNAP, cluster | |||
| CMU-MOSEI | OPPORTUNITY | BioSNAP cluster | ||||||||||
| Retrieval | Probe | Retrieval | Probe | Retrieval | Probe | |||||||
| Labels | hard | cross | hard | cross | hard | cross | hard | cross | hard | cross | hard | cross |
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Meaning |
|---|---|
| the label | |
| modality masks; observes a subset of what observes | |
| everything about a sample the corrector may read at deployment | |
| the frozen model’s scores | |
| , | score-space loss and prediction loss, respectively |
| the prediction in mean-parameter form; for cross-entropy |
| Dataset | Frozen model | Anchor: published ours | Degradations |
|---|---|---|---|
| BioSNAP | DrugBAN | AUROC | protein , ; scaffold |
| BindingDB | DrugBAN | AUROC | protein , ; scaffold |
| Human | DrugBAN | — | protein , |
| OPPORTUNITY | DeepConvLSTM | wtd. F1 | four sensor subsets |
| CMU-MOSEI | CMAD | acc. within | audio, visual, or both |
| IEMOCAP | MoMKE | WA full; short sparse | audio only |
| Benchmark | Construction | ||||
| CMU-MOSEI | four disjoint quarters | ||||
| IEMOCAP | pool from other folds; fold split by video | ||||
| AVE | released pool; deployment split in thirds | ||||
| NinaPro | pool from other repetitions; rest in thirds | ||||
| OPPORTUNITY | random -window blocks | ||||
| PTB-XL | official patient folds |
| AUROC | Accuracy | ||||||
| Dataset | Split | frozen | GUARD | frozen | GUARD | Joint harm | Apply (%) |
| BioSNAP | random † | ||||||
| random ‡ | |||||||
| cluster † | |||||||
| BindingDB | random † | ||||||
| random ‡ | |||||||
| Weighted F1 | Accuracy | |||||
|---|---|---|---|---|---|---|
| Condition | frozen | GUARD | frozen | GUARD | Joint harm | Apply (%) |
| low-cost accels only | ||||||
| no IMU family | ||||||
| no shoe sensors | ||||||
| three sensors only | ||||||
| Dataset (model) | Available | frozen | GUARD | Joint harm | Apply (%) |
|---|---|---|---|---|---|
| CMU-MOSEI (CMAD) | audio visual | ||||
| binary accuracy | audio only | ||||
| visual only | |||||
| intact | |||||
| IEMOCAP (MoMKE) | audio only | ||||
| -class WA | text only |
| Blanket correction | GUARD | |||
|---|---|---|---|---|
| Benchmark | Conditions | joint harm | joint harm | |
| CMU-MOSEI | – | – | ||
| IEMOCAP | – | – | ||
| OPPORTUNITY | – | – | ||
| AVE | – | – | ||
| NinaPro DB5 | – | – | ||
| Cond. harm (%) | |||||
|---|---|---|---|---|---|
| Benchmark | mean | worst | Flip - | Flip + | |
| CMU-MOSEI | |||||
| IEMOCAP | |||||
| OPPORTUNITY | |||||
| AVE | |||||
| NinaPro DB5 | |||||
| Benchmark | Apply (%) | GUARD | Confidence | Agreement | Random | |
|---|---|---|---|---|---|---|
| CMU-MOSEI | / / | / / | / / | / / | / / | |
| IEMOCAP | / / | / / | / / | / / | / / | |
| OPPORTUNITY | / / | / / | / / | / / | / / | |
| AVE | / / | / / | / / | / / | / / | |
| NinaPro DB5 | / / | / / | / / | / / | / / | |
| PTB-XL | / / | / / | / / | / / | / / |
| Gated gain | Joint harm | Ungated harm | ||||
|---|---|---|---|---|---|---|
| Benchmark | Retrieval | Probe | Retrieval | Probe | Retrieval | Probe |
| IEMOCAP | ||||||
| AVE | ||||||
| NinaPro DB5 | ||||||
| PTB-XL | ||||||
| DrugBAN | ||||||
| Latency (ms) | State (MiB) | ||||||
|---|---|---|---|---|---|---|---|
| Benchmark | Retrieval | Probe | Retrieval | Probe | |||
| AVE | |||||||
| PTB-XL | |||||||
| IEMOCAP | |||||||
| NinaPro | |||||||
| Action score | AVE | NinaPro | PTB-XL | IEMOCAP | DrugBAN |
|---|---|---|---|---|---|
| Score family, all five features | |||||
| logistic (deployed) | / | / | / | / | / |
| decision tree, depth | / | / | / | / | / |
| random forest, trees | / | / | / | / | / |
| NN, | / | / | / | / | / |
| MLP, one hidden layer of | / | / | / | / | / |
| (a) CMU-MOSEI | |||||
|---|---|---|---|---|---|
| full | |||||
| Hard-label gain | |||||
| Cross-mask gain | |||||
| GUARD gain | |||||
| GUARD joint harm | |||||
| Flipped pool labels | |||||
| Retrieval | Probe | ||||
|---|---|---|---|---|---|
| Benchmark | Labels | hard | cross | hard | cross |
| CMU-MOSEI | / ( ) | / ( ) | / ( ) | / ( ) | |
| / ( ) | / ( ) | / ( ) | / ( ) | ||
| / ( ) | / ( ) | / ( ) | / ( ) | ||
| / ( ) | / ( ) | / ( ) | / ( ) | ||
| / ( ) | / ( ) | / ( ) | / ( ) | ||
| GUARD | LTT | ||||||
|---|---|---|---|---|---|---|---|
| Benchmark | Labels | gain | harm | apply (%) | gain | harm | apply (%) |
| PTB-XL | |||||||
| One threshold | One threshold, tightened | Per mask | Per mask, tightened | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Benchmark | gain | harm | worst | gain | harm | worst | gain | harm | worst | gain | harm | worst |
| CMU-MOSEI | ||||||||||||
| NinaPro DB5 | ||||||||||||
| IEMOCAP, | ||||||||||||
| IEMOCAP, | ||||||||||||
| Calibration only | Whole adapter | ||||
|---|---|---|---|---|---|
| Benchmark | Source to target | gain | harm | gain | harm |
| CMU-MOSEI | same condition | ||||
| richer to poorer | |||||
| poorer to richer | |||||
| IEMOCAP | same condition | ||||
| richer to poorer | |||||
| CMU-MOSEI | DrugBAN | |||||||
|---|---|---|---|---|---|---|---|---|
| Gain | Joint harm | Flip - | Flip + | Gain | Joint harm | Flip - | Flip + | |
| Configuration | Base acc | Loss gain | acc | ||
|---|---|---|---|---|---|
| Bias-dominant | |||||
| Mixed | |||||
| Ambiguity-dominant |
Missing-Modality Learning
Missing-modality learning develops multimodal models that remain useful when input streams are absent, corrupted, or available only during training.
See all 187 papers