Large video-language models (LVLMs) have shown remarkable performance on multimodal tasks like multimodal emotion recognition (ER) in the wild. ER is inherently multimodal, requiring a joint understanding of facial expressions, vocalizations, language, biosignals, and gestures. However, real-world deployment remains challenging: modalities may be missing or noisy at test time. Partial observations can be viewed as a distribution shift relative to the complete-modality distribution. SOTA TTA methods based on entropy minimization or perplexity reduction do not transfer to autoregressive LVLMs, while retrieval augmented generation (RAG) degrades when the observed modality is weak. Because no ground-truth supervision exists to verify individual updates, adaptation across this stream risks accumulating drift and degrading once the model departs from a reliable solution. An effective solution must therefore adapt to arbitrary missing-modality patterns and remain effective during continual adaptation. We address both jointly with Test-Time Self-Distillation (TTSD), a parameter-efficient framework in which a frozen teacher, trained on complete modalities, guides an adaptive low-rank student via self-distillation, updating only a negligible number of parameters. Stability is built into this same loop through Fisher-Anchored Restoration (FAR), which monitors Fisher information stability to detect convergence versus drift and restores the student toward the teacher's anchor when distributional shifts are identified. Our experiments on MELD, DFEW, and BAH under 0%-50% missing modalities show that this unified adaptation-restoration design consistently outperforms entropy-based adaptation, RAG, and perplexity-based generation over long adaptation horizons, where baselines without restoration progressively degrade while TTSD-FAR remains consistent.
Figures & tables
Figure 1: UMAP of the internal representations. The cross ( × ) represents the missing modality inputs, and the circle ( ∙ ) represents the full modality inputs.
Figure 2: Illustration of the proposed TTSD-FAR method. The solid green arrow shows the flow of information in the case of complete modalities, and the dashed red arrow shows the flow of information in the missing modality case. The solid green arrow shows both modalities, and the single dashed green arrow denotes the input ( x~ ) with masked input. In the missing case, the model bypasses adaptation and only infers with the student LoRA. In the complete case, the model uses complete modality features from the teacher LoRA and masked modality features from the student LoRA, and distillation loss Ldistill is calculated to match the student representation with the teacher representation.
Text Missing
Vision Missing
Method
Full
10%
20%
30%
40%
50%
10%
20%
30%
40%
50%
No Adaptation
0.6149
0.5926
0.5706
0.5390
0.5190
0.4785
0.6122
0.6025
0.5934
0.5825
0.5775
TENT (ICLR’21)
–
0.1650
0.1680
0.1765
0.1855
0.1655
0.1658
0.1725
0.1655
0.1838
0.1826
EATA (ICML’22)
–
0.2854
0.2690
0.2770
0.2736
0.2482
0.2950
0.2750
0.2745
0.2530
0.2382
RAG (ICCV’25)
–
0.5640
0.5248
0.5120
0.4845
0.4660
0.5925
0.5830
0.5634
0.5567
0.5534
Perplexity Gen.
–
0.5680
0.5460
0.5255
0.4938
0.4722
N/A
N/A
N/A
N/A
N/A
Table 1: Comparison of methods under progressively missing textual and visual modalities. We report F1 on the MELD dataset as the proportion of unavailable input increases. N/A: Perplexity Gen. is not applicable to the vision-missing case.
Method
Full
10%
20%
30%
40%
50%
No Adaptation
0.5549
0.5300
0.5078
0.4984
0.4837
0.4434
TENT
–
0.1548
0.1553
0.1665
0.1658
0.1640
EATA
–
0.2534
0.2778
0.2735
0.2845
0.2885
RAG
–
0.5340
0.5048
0.5020
0.4800
0.4468
TTSD-FAR (Ours)
–
0.5515
0.5300
0.5195
0.5015
0.4775
Table 2: Comparison of methods under progressively missing visual modality. We report F1 on the DFEW dataset as the proportion of unavailable visual input increases.
Method
Full
10%
20%
30%
40%
50%
Zero-Shot
0.7142
0.6835
0.6742
0.6526
0.6215
0.6023
TENT
–
0.4923
0.4834
0.4710
0.4572
0.4550
EATA
–
0.5630
0.5467
0.5360
0.5310
0.5250
RAG
–
0.6835
0.6730
0.6535
0.6215
0.5950
Perplexity Gen.
–
0.6870
0.6745
0.6535
0.6310
0.6030
TTSD-FAR (Ours)
–
0.7050
0.6925
0.6695
0.6405
0.6250
Table 3: Comparison of methods under progressively missing textual modality conditions for BAH dataset.
Figure 3: Effectiveness of the FAR module on the distillation results across different missing rates.
Figure 5: TTSD-FAR without teacher LoRA. Base weights provide teacher representations.
Method
Full
10%
20%
30%
40%
50%
No Adaptation
0.6341
0.5905
0.5798
0.5710
0.5662
0.5614
TENT
–
0.4910
0.4840
0.4733
0.4501
0.4453
EATA
–
0.5465
0.5367
0.5290
0.5245
0.5110
RAG
–
0.5770
0.5720
0.5540
0.5448
0.5250
Perplexity Gen.
–
0.5790
0.5695
0.5648
0.5488
0.5430
TTSD-FAR (Ours)
–
0.6185
0.6120
0.5955
0.5910
0.5870
Appendix
Table 4: Comparison of methods under progressively missing text conditions. Distilling from the Base model, i.e., no teacher LoRA. We report WF1 on the BAH dataset as the proportion of unavailable textual input increases.
Method
Full
10%
20%
30%
40%
50%
No Adaptation
0.5549
0.5530
0.5490
0.5365
0.5260
0.5235
TENT
–
0.1655
0.1725
0.1790
0.1935
0.1833
EATA
–
0.2635
0.2674
0.2835
0.2855
0.2943
RAG
–
0.5530
0.5425
0.5265
0.5265
0.5210
Perplexity Gen.
–
0.5535
0.5460
0.5350
0.5300
0.5210
TTSD-FAR (Ours)
–
0.5540
0.5510
0.5444
0.5376
0.5278
Appendix
Table 5: Comparison of methods under progressively missing textual modality conditions. F1 on the DFEW dataset as the proportion of missing textual input increases.
Method
Full
10%
20%
30%
40%
50%
No Adaptation
0.7142
0.7142
0.7095
0.7046
0.6950
0.6923
TENT
–
0.4834
0.4846
0.4790
0.4645
0.4754
EATA
–
0.5655
0.5428
0.5165
0.5095
0.5052
RAG
–
0.7100
0.7055
0.7059
0.7110
0.6975
TTSD-FAR (Ours)
–
0.7142
0.7125
0.7120
0.7125
0.7125
Appendix
Table 6: Comparison of methods under progressively missing visual modality. We report Avg-F1 on the BAH dataset as the proportion of unavailable visual input increases.
Figure 6: Impact of LoRA rank on distillation performance under varying missing-modality rates across three datasets.
Model
Method
Complete
10%
20%
30%
40%
50%
Video-LLaVA-7B
No Adaptation
0.6341
0.5905
0.5798
0.5710
0.5662
0.5614
TTSD-FAR (Ours)
0.6185
0.6120
0.5955
0.5910
0.5870
MobileVideoGPT-0.5B
No Adaptation
0.4889
0.4521
0.4413
0.4310
0.4218
0.4125
TTSD-FAR (Ours)
0.4685
0.4595
0.4510
0.4428
0.4385
Appendix
Table 7: F1 Score on the BAH dataset under progressively missing textual input for two models: Video-LLaVA-7B and MobileVideoGPT-0.5B.
Figure 7: Effect of τfmi on reactivation events. The three panels show the FMI trace at τfmi 1.0 (top), 2.0 (middle) and, 3.0 (bottom).
Figure 8: Timeline of FAR states ( active - anchored ) across the test stream.
Method
Full
10%
20%
30%
40%
50%
No-Adaptation
0.6341
0.5905
0.5798
0.5710
0.5662
0.5614
TTSD
–
0.6065
0.6039
0.5887
0.5816
0.5728
TTSD-LS
–
0.5910
0.5820
0.5730
0.5645
0.5560
TTSD-FAR (Ours)
–
0.6185
0.6120
0.5955
0.5910
0.5870
Appendix
Table 8: Ablation comparing TTSD-FAR against plain TTSD (no stopping criterion) and TTSD-LS (loss-plateau stopping criterion) under progressively missing text conditions. We report WF1 on the BAH dataset as the proportion of unavailable textual input increases.
Prompt Type
Prompt
Avg F1
Simple
Classify the emotion in the video as either Non-Ambivalent or Ambivalent . Respond with only one word.
0.2827
Definition 1
Definition: Ambivalence is the state of having contradictory or conflicting feelings or attitudes towards something or someone simultaneously. Classify the emotion in the video as either Non-Ambivalent or Ambivalent . Respond with only one word.
0.3326
Definition 2
Definition: Ambivalence and hesitancy is understood as the simultaneous experience of desires for change and against change. Classify the emotion in the video as either Non-Ambivalent or Ambivalent . Respond with only one word.
0.3772
Transcript + Def 1
Video transcript: {transcript} . Definition: Ambivalence is the state of having contradictory or conflicting feelings or attitudes towards something or someone simultaneously. Classify the emotion in the video as either Non-Ambivalent or Ambivalent . Respond with only one word.
0.6341
Transcript + Def 2
Video transcript: {transcript} . Definition: Ambivalence and hesitancy are understood as the simultaneous experience of desires for change and against change. Classify the emotion in the video as either Non-Ambivalent or Ambivalent . Respond with only one word.
0.3945
Appendix
Table 9: Summary of prompt variations for zero-shot inference with corresponding video-level F1 scores.
Figure 9: Illustration of the missing modality
Parameter
Value
Definition
βF
0.99
EMA decay parameter for the online Fisher diagonal
Vision-language models (VLMs) such as CLIP exhibit remarkable zero-shot capabilities, yet their performance frequently degrades sharply under unexpected test-time distribution shifts. While Test-Time Adaptation (TTA) offers a promising solution, continuously adapting VLMs over an unlabeled test stream presents fundamental challenges. Conventional top-1-centric updates often reinforce errors by corrupting the local semantic geometry among related classes, while iterative adaptation exacerbates progressive bias accumulation, ultimately driving the model toward mode collapse. To overcome these coupled vulnerabilities, we propose Local Margin Restoration (LMR), a lightweight, one-step TTA framework. At the sample level, our Protected Margin Restoration (PMR) objective recovers local semantic geometry by shielding plausible near-top candidates from external hard negatives. Concurrently, to combat stream-level degradation, we introduce a dual-stage stabilization mechanism, featuring an Adaptive Margin (AM) controller and Bias Correction (BC), to dynamically disrupt progressive bias accumulation and prevent mode collapse. Extensive experiments on CIFAR-C, ImageNet-C, and ImageNet variants demonstrate that LMR consistently outperforms state-of-the-art TTA baselines, proving exceptionally robust and efficient even in challenging low-batch test-time regimes. Our code is available at https://github.com/DennisHuangYan/LMR.
Yan Huang, Guowei Wang, Xu Wang +2
Guangzhou University Guangzhou, China · The Second Affiliated Hospital of Guangzhou University of Chinese Medicine Guangzhou, China · Jinan University Zhuhai, China +1
Test-time reinforcement learning enables vision-language models (VLMs) to adapt using unlabeled inputs. However, repeated sampling under fixed visual conditions can reinforce shared perceptual errors, while sequence-level rewards fail to isolate visual perception the foundational bottleneck that anchors multimodal reasoning risking the degradation of pre-trained reasoning capabilities. We propose TTRSD, a test-time reinforcement learning framework combining multi-view answer-level self-distillation with visual contrastive token selection. A shared policy aggregates teacher predictions across original, cropped, and downsampled views into an answer distribution. Student trajectories generated from the original image receive rewards based on the support for their final answers in this distribution. To allocate this feedback precisely toward perceptual bottlenecks, we compare the log-probabilities of the same sampled tokens under original and visually ablated inputs while holding their textual prefixes fixed, selecting visually sensitive positions for policy-gradient updates. TTRSD separates update direction, determined by group-relative advantages, from update position, determined by visual sensitivity, without requiring ground-truth labels, external verifiers, or a separate teacher. With only 20 unlabeled adaptation samples, TTRSD improves performance across seven benchmarks and three VLMs, raising InternVL3-2B's MMMU accuracy from 35.79% to 49.32%(+13.53%), demonstrating cross-dataset generalization while preserving inherent reasoning integrity.
Shuning Wang, Zhiheng Wu, Xun Zhou +8
Zhejiang University · Hong Kong University of Science and Technology · Harbin Institute of Technology +1
Vision-language models (VLMs) have achieved remarkable performance by leveraging complementary information from large-scale image-text pairs. However, missing-modality inputs are commonly encountered during real-world deployment, often leading to significant performance degradation. Existing methods primarily enhance model robustness by learning modality compensation strategies from source training data. However, their reliance on source training data makes them difficult to apply when original data are unavailable due to privacy, storage, or accessibility constraints, such as clinical applications and personalized AI services. This raises an important yet underexplored question: can VLMs be efficiently adapted at test time for visual recognition with missing modalities without accessing source training data? To this end, we propose Test-Time Logit Prompting (TLP), a lightweight source-free test-time adaptation framework for visual recognition with missing modalities. To address missing-induced prediction shifts, TLP optimizes logit prompts with uncertainty-aware adjustment and modality-complete consistency regularization, adaptively adjusting prediction confidence while preserving semantic consistency. Extensive experiments across diverse vision-language benchmarks demonstrate that TLP consistently enhances recognition performance under missing-modality scenarios, achieving up to 8% improvements while requiring only hundreds of tunable parameters and a few test-time optimization steps.
Taixi Chen, Nancy Guo
School of Computing State University of New York at Binghamton