TTSD-FAR: Test-Time Self-Distillation with Fisher-Anchored Restoration for Missing-Modality Emotion Recognition in LVLMs

Muhammad Haseeb Aslam; Alessandro Koerich; Marco Pedersoli; Ali Etemad; Eric Granger

cs.CVAug 18, 2026

TTSD-FAR: Test-Time Self-Distillation with Fisher-Anchored Restoration for Missing-Modality Emotion Recognition in LVLMs

Authors: Muhammad Haseeb Aslam, Alessandro Koerich, Marco Pedersoli, Ali Etemad, Eric Granger

Organizations: LIVIA, ETS Montreal, Canada · Aiim Lab, Queen’s University, Canada.

Abstract

Large video-language models (LVLMs) have shown remarkable performance on multimodal tasks like multimodal emotion recognition (ER) in the wild. ER is inherently multimodal, requiring a joint understanding of facial expressions, vocalizations, language, biosignals, and gestures. However, real-world deployment remains challenging: modalities may be missing or noisy at test time. Partial observations can be viewed as a distribution shift relative to the complete-modality distribution. SOTA TTA methods based on entropy minimization or perplexity reduction do not transfer to autoregressive LVLMs, while retrieval augmented generation (RAG) degrades when the observed modality is weak. Because no ground-truth supervision exists to verify individual updates, adaptation across this stream risks accumulating drift and degrading once the model departs from a reliable solution. An effective solution must therefore adapt to arbitrary missing-modality patterns and remain effective during continual adaptation. We address both jointly with Test-Time Self-Distillation (TTSD), a parameter-efficient framework in which a frozen teacher, trained on complete modalities, guides an adaptive low-rank student via self-distillation, updating only a negligible number of parameters. Stability is built into this same loop through Fisher-Anchored Restoration (FAR), which monitors Fisher information stability to detect convergence versus drift and restores the student toward the teacher's anchor when distributional shifts are identified. Our experiments on MELD, DFEW, and BAH under 0%-50% missing modalities show that this unified adaptation-restoration design consistently outperforms entropy-based adaptation, RAG, and perplexity-based generation over long adaptation horizons, where baselines without restoration progressively degrade while TTSD-FAR remains consistent.

Figures & tables

Figure 1: UMAP of the internal representations. The cross ( $\times$ ) represents the missing modality inputs, and the circle ( $\bullet$ ) represents the full modality inputs.

Figure 2: Illustration of the proposed TTSD-FAR method. The solid green arrow shows the flow of information in the case of complete modalities, and the dashed red arrow shows the flow of information in the missing modality case. The solid green arrow shows both modalities, and the single dashed green arrow denotes the input ( $\tilde{x}$ ) with masked input. In the missing case, the model bypasses adaptation and only infers with the student LoRA. In the complete case, the model uses complete modality features from the teacher LoRA and masked modality features from the student LoRA, and distillation loss $\mathcal{L}_{\text{distill}}$ is calculated to match the student representation with the teacher representation.

		Text Missing					Vision Missing
Method	Full	10%	20%	30%	40%	50%	10%	20%	30%	40%	50%
No Adaptation	0.6149	0.5926	0.5706	0.5390	0.5190	0.4785	0.6122	0.6025	0.5934	0.5825	0.5775
TENT (ICLR’21)	–	0.1650	0.1680	0.1765	0.1855	0.1655	0.1658	0.1725	0.1655	0.1838	0.1826
EATA (ICML’22)	–	0.2854	0.2690	0.2770	0.2736	0.2482	0.2950	0.2750	0.2745	0.2530	0.2382
RAG (ICCV’25)	–	0.5640	0.5248	0.5120	0.4845	0.4660	0.5925	0.5830	0.5634	0.5567	0.5534
Perplexity Gen.	–	0.5680	0.5460	0.5255	0.4938	0.4722	N/A	N/A	N/A	N/A	N/A

Table 1: Comparison of methods under progressively missing textual and visual modalities. We report F1 on the MELD dataset as the proportion of unavailable input increases. N/A: Perplexity Gen. is not applicable to the vision-missing case.

Method	Full	10%	20%	30%	40%	50%
No Adaptation	0.5549	0.5300	0.5078	0.4984	0.4837	0.4434
TENT	–	0.1548	0.1553	0.1665	0.1658	0.1640
EATA	–	0.2534	0.2778	0.2735	0.2845	0.2885
RAG	–	0.5340	0.5048	0.5020	0.4800	0.4468
TTSD-FAR (Ours)	–	0.5515	0.5300	0.5195	0.5015	0.4775

Table 2: Comparison of methods under progressively missing visual modality. We report F1 on the DFEW dataset as the proportion of unavailable visual input increases.

Method	Full	10%	20%	30%	40%	50%
Zero-Shot	0.7142	0.6835	0.6742	0.6526	0.6215	0.6023
TENT	–	0.4923	0.4834	0.4710	0.4572	0.4550
EATA	–	0.5630	0.5467	0.5360	0.5310	0.5250
RAG	–	0.6835	0.6730	0.6535	0.6215	0.5950
Perplexity Gen.	–	0.6870	0.6745	0.6535	0.6310	0.6030
TTSD-FAR (Ours)	–	0.7050	0.6925	0.6695	0.6405	0.6250

Table 3: Comparison of methods under progressively missing textual modality conditions for BAH dataset.

Figure 3: Effectiveness of the FAR module on the distillation results across different missing rates.

Figure 4: Fisher importance heatmaps (log 10 scale): TTSD-FAR (left), TTSD (middle), and difference map (right), confirming systematically higher importance under FAR.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: TTSD-FAR without teacher LoRA. Base weights provide teacher representations.

Method	Full	10%	20%	30%	40%	50%
No Adaptation	0.6341	0.5905	0.5798	0.5710	0.5662	0.5614
TENT	–	0.4910	0.4840	0.4733	0.4501	0.4453
EATA	–	0.5465	0.5367	0.5290	0.5245	0.5110
RAG	–	0.5770	0.5720	0.5540	0.5448	0.5250
Perplexity Gen.	–	0.5790	0.5695	0.5648	0.5488	0.5430
TTSD-FAR (Ours)	–	0.6185	0.6120	0.5955	0.5910	0.5870

Appendix

Table 4: Comparison of methods under progressively missing text conditions. Distilling from the Base model, i.e., no teacher LoRA. We report WF1 on the BAH dataset as the proportion of unavailable textual input increases.

Method	Full	10%	20%	30%	40%	50%
No Adaptation	0.5549	0.5530	0.5490	0.5365	0.5260	0.5235
TENT	–	0.1655	0.1725	0.1790	0.1935	0.1833
EATA	–	0.2635	0.2674	0.2835	0.2855	0.2943
RAG	–	0.5530	0.5425	0.5265	0.5265	0.5210
Perplexity Gen.	–	0.5535	0.5460	0.5350	0.5300	0.5210
TTSD-FAR (Ours)	–	0.5540	0.5510	0.5444	0.5376	0.5278

Appendix

Table 5: Comparison of methods under progressively missing textual modality conditions. F1 on the DFEW dataset as the proportion of missing textual input increases.

Method	Full	10%	20%	30%	40%	50%
No Adaptation	0.7142	0.7142	0.7095	0.7046	0.6950	0.6923
TENT	–	0.4834	0.4846	0.4790	0.4645	0.4754
EATA	–	0.5655	0.5428	0.5165	0.5095	0.5052
RAG	–	0.7100	0.7055	0.7059	0.7110	0.6975
TTSD-FAR (Ours)	–	0.7142	0.7125	0.7120	0.7125	0.7125

Appendix

Table 6: Comparison of methods under progressively missing visual modality. We report Avg-F1 on the BAH dataset as the proportion of unavailable visual input increases.

Figure 6: Impact of LoRA rank on distillation performance under varying missing-modality rates across three datasets.

Model	Method	Complete	10%	20%	30%	40%	50%
Video-LLaVA-7B	No Adaptation	0.6341	0.5905	0.5798	0.5710	0.5662	0.5614
Video-LLaVA-7B	TTSD-FAR (Ours)	0.6341	0.6185	0.6120	0.5955	0.5910	0.5870
MobileVideoGPT-0.5B	No Adaptation	0.4889	0.4521	0.4413	0.4310	0.4218	0.4125
MobileVideoGPT-0.5B	TTSD-FAR (Ours)	0.4889	0.4685	0.4595	0.4510	0.4428	0.4385

Appendix

Table 7: F1 Score on the BAH dataset under progressively missing textual input for two models: Video-LLaVA-7B and MobileVideoGPT-0.5B.

Figure 7: Effect of $\tau_{fmi}$ on reactivation events. The three panels show the FMI trace at $\tau_{fmi}$ 1.0 (top), 2.0 (middle) and, 3.0 (bottom).

Figure 8: Timeline of FAR states ( active - anchored ) across the test stream.

Method	Full	10%	20%	30%	40%	50%
No-Adaptation	0.6341	0.5905	0.5798	0.5710	0.5662	0.5614
TTSD	–	0.6065	0.6039	0.5887	0.5816	0.5728
TTSD-LS	–	0.5910	0.5820	0.5730	0.5645	0.5560
TTSD-FAR (Ours)	–	0.6185	0.6120	0.5955	0.5910	0.5870

Appendix

Table 8: Ablation comparing TTSD-FAR against plain TTSD (no stopping criterion) and TTSD-LS (loss-plateau stopping criterion) under progressively missing text conditions. We report WF1 on the BAH dataset as the proportion of unavailable textual input increases.

Prompt Type	Prompt	Avg F1
Simple	Classify the emotion in the video as either Non-Ambivalent or Ambivalent . Respond with only one word.	0.2827
Definition 1	Definition: Ambivalence is the state of having contradictory or conflicting feelings or attitudes towards something or someone simultaneously. Classify the emotion in the video as either Non-Ambivalent or Ambivalent . Respond with only one word.	0.3326
Definition 2	Definition: Ambivalence and hesitancy is understood as the simultaneous experience of desires for change and against change. Classify the emotion in the video as either Non-Ambivalent or Ambivalent . Respond with only one word.	0.3772
Transcript + Def 1	Video transcript: {transcript} . Definition: Ambivalence is the state of having contradictory or conflicting feelings or attitudes towards something or someone simultaneously. Classify the emotion in the video as either Non-Ambivalent or Ambivalent . Respond with only one word.	0.6341
Transcript + Def 2	Video transcript: {transcript} . Definition: Ambivalence and hesitancy are understood as the simultaneous experience of desires for change and against change. Classify the emotion in the video as either Non-Ambivalent or Ambivalent . Respond with only one word.	0.3945

Appendix

Table 9: Summary of prompt variations for zero-shot inference with corresponding video-level F1 scores.

Figure 9: Illustration of the missing modality

Parameter	Value	Definition
$\beta_{F}$	0.99	EMA decay parameter for the online Fisher diagonal
$\tau_{fc}$	0.02	Consolidation Index threshold
$K$	5	No. of consecutive adaptation steps required
$\tau_{fmi}$	2.0	Fisher Mismatch Index threshold
$\varepsilon$	$10^{-8}$	Numerical stability constant

Appendix

Table 10: FAR hyperparameters

Explore similar work

Local Margin Restoration for Test-Time Adaptation of Vision-Language Models
Aug 3, 2026Yan Huang, Guowei Wang, Xu Wang +2VLM Robustness Test-Time Adaptation
Guangzhou University Guangzhou, China · The Second Affiliated Hospital of Guangzhou University of Chinese Medicine Guangzhou, China · Jinan University Zhuhai, China +1
TTRSD: Test-Time Reinforcement Learning with Self-Distillation for Vision-Language Models
Sep 27, 2026Shuning Wang, Zhiheng Wu, Xun Zhou +8Test-Time Adaptation of VLMs VLM Distillation
Zhejiang University · Hong Kong University of Science and Technology · Harbin Institute of Technology +1
Test-Time Logit Prompting for Source-Free Missing Modality Adaptation
Sep 2, 2026Taixi Chen, Nancy GuoMissing-Modality Learning Test-Time Adaptation
School of Computing State University of New York at Binghamton

Aug 3, 2026J/K move · Enter open · S save

Local Margin Restoration for Test-Time Adaptation of Vision-Language Models

Yan Huang, Guowei Wang, Xu Wang, Kangjun Liu+1

Guangzhou University Guangzhou, China · The Second Affiliated Hospital of Guangzhou University of Chinese Medicine Guangzhou, China · Jinan University Zhuhai, China+1

VLM Robustness Test-Time Adaptation VLM Adaptation Test-Time Adaptation of VLMs

Open paper AskPDF arXiv

Vision-language models (VLMs) such as CLIP exhibit remarkable zero-shot capabilities, yet their performance frequently degrades sharply under unexpected test-time distribution shifts. While Test-Time Adaptation (TTA) offers a promising solution, continuously adapting VLMs over an unlabeled test stream presents fundamental challenges. Conventional top-1-centric updates often reinforce errors by corrupting the local semantic geometry among related classes, while iterative adaptation exacerbates progressive bias accumulation, ultimately driving the model toward mode collapse. To overcome these coupled vulnerabilities, we propose Local Margin Restoration (LMR), a lightweight, one-step TTA framework. At the sample level, our Protected Margin Restoration (PMR) objective recovers local semantic geometry by shielding plausible near-top candidates from external hard negatives. Concurrently, to combat stream-level degradation, we introduce a dual-stage stabilization mechanism, featuring an Adaptive Margin (AM) controller and Bias Correction (BC), to dynamically disrupt progressive bias accumulation and prevent mode collapse. Extensive experiments on CIFAR-C, ImageNet-C, and ImageNet variants demonstrate that LMR consistently outperforms state-of-the-art TTA baselines, proving exceptionally robust and efficient even in challenging low-batch test-time regimes. Our code is available at https://github.com/DennisHuangYan/LMR.

Abstract

Explore similar work

Local Margin Restoration for Test-Time Adaptation of Vision-Language Models

TTRSD: Test-Time Reinforcement Learning with Self-Distillation for Vision-Language Models

Test-Time Logit Prompting for Source-Free Missing Modality Adaptation