Test-Time Adaptation via Cache Personalization for Facial Expression Recognition in Videos
Organizations: LIVIA, Department of Systems Engineering, ´ETS Montreal, Canada · LIVIA, Department of Software and IT Engineering, ´ETS Montreal, Canada
Abstract
Facial expression recognition (FER) in videos requires model personalization to capture considerable variation across subjects. Vision-language models (VLMs) offer strong transfer through image-text alignment, but their performance can degrade under inter-subject distribution shifts. Test-time adaptation (TTA) can mitigate this challenge, yet most state-of-the-art methods rely on unsupervised parameter optimization, introducing computational overhead that limits real-world deployment. This paper introduces TTA through Cache Personalization (TTA-CaP), a gradient-free, cache-based method for cost-effective personalization of VLMs in video FER. Unlike prior cache-based TTA methods that rely solely on dynamic memories of test samples and may drift because of noisy pseudo-labels, TTA-CaP employs three complementary caches: a personalized static cache constructed through feature-statistics matching, a positive target cache containing reliable subject-specific samples, and a negative target cache containing low-confidence cases as negative evidence. A tri-gate mechanism prevents cache corruption by controlling updates according to temporal stability, confidence, and consistency with the personalized static cache. The caches jointly provide subject-matched positive and negative evidence for robust personalization. TTA-CaP further refines predictions through embedding fusion, supporting temporally stable video-level predictions. Experiments on BioVid, StressID, and BAH show that TTA-CaP outperforms state-of-the-art TTA methods under subject-specific and environmental shifts while maintaining low computational and memory overhead. Our code is publicly available at https://github.com/MasoumehSharafi/TTA-CaP.
Figures & tables
| Protocol | Method | BioVid | StressID | BAH | |||
| WAR | F1 | WAR | F1 | WAR | F1 | ||
| ZS | CLIP-ViT-B/32 (ICML’21) | 50.0 | 33.3 | 60.4 | 34.8 | 39.5 | 28.1 |
| CLIP-ViT-B/32 (ICML’21) | 69.7 | 66.6 | 67.0 | 44.5 | 60.4 | 39.8 | |
| EmoCLIP [ 8 ] (FG’24) | 67.7 | 63.4 | 63.5 | 35.9 | 56.2 | 36.5 | |
| X-CLIP [ 20 ] (ECCV’22) | 70.9 | 57.9 | 62.3 | 41.3 | 63.0 | 39.2 | |
| FT | Exp-CLIP [ 47 ] (WACV’25) | 70.2 | 66.7 | 63.1 | 44.5 | 62.2 | 38.5 |
| Method | Sub-1 | Sub-2 | Sub-3 | Sub-4 | Sub-5 | Sub-6 | Sub-7 | Sub-8 | Sub-9 | Sub-10 | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| TPT [ 29 ] (NeurIPS’22) | 92.0 | 51.5 | 40.0 | 49.6 | 79.0 | 87.0 | 79.5 | 43.4 | 99.0 | 54.0 | 67.5 |
| TDA [ 12 ] (CVPR’24) | 94.9 | 50.3 | 43.5 | 48.0 | 80.8 | 86.2 | 80.2 | 43.5 | 100.0 | 55.0 | 68.2 |
| DPE [ 45 ] (NeurIPS’24) | 90.4 | 51.5 | 43.5 | 73.1 | 72.0 | 83.6 | 79.8 | 48.5 | 100.0 | 54.4 | 69.6 |
| PromptAlign [ 1 ] (NeurIPS’23) | 94.2 | 60.1 | 42.5 | 61.5 | 83.5 | 90.5 | 80.2 | 43.5 | 100.0 | 60.0 | 71.6 |
| ReTA [ 17 ] (ACMMM’25) | 94.2 | 60.1 | 43.5 | 60.8 | 83.5 | 88.0 | 80.2 | 43.5 | 100.0 | 60.0 | 71.3 |
| MCP [ 5 ] (ICCV’25) | 94.5 | 62.0 | 43.5 | 64.0 | 86.0 | 92.0 | 82.0 | 44.0 | 100.0 | 62.0 | 73.0 |
| Method | Sub-1 | Sub-2 | Sub-3 | Sub-4 | Sub-5 | Sub-6 | Sub-7 | Sub-8 | Sub-9 | Sub-10 | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| TPT [ 29 ] (NeurIPS’22) | 70.7 | 43.6 | 86.0 | 83.1 | 36.4 | 78.7 | 41.1 | 50.0 | 40.0 | 50.0 | 57.9 |
| TDA [ 12 ] (CVPR’24) | 45.5 | 40.0 | 50.1 | 81.0 | 35.0 | 80.0 | 29.6 | 45.6 | 42.9 | 50.0 | 49.9 |
| DPE [ 45 ] (NeurIPS’24) | 41.0 | 40.6 | 49.0 | 81.0 | 49.8 | 80.0 | 56.0 | 46.0 | 42.9 | 56.6 | 54.2 |
| PromptAlign [ 1 ] (NeurIPS’23) | 41.3 | 45.0 | 45.6 | 82.3 | 48.1 | 80.0 | 44.0 | 58.0 | 42.9 | 45.5 | 53.2 |
| ReTA [ 17 ] (ACMMM’25) | 43.2 | 41.0 | 53.0 | 80.9 | 36.1 | 80.0 | 39.1 | 53.3 | 42.9 | 59.3 | 52.8 |
| MCP [ 5 ] (ICCV’25) | 55.0 | 44.5 | 60.0 | 78.3 | 43.6 | 79.1 | 52.2 | 61.2 | 44.0 | 65.0 | 58.2 |
| Personalization protocol | WAR (%) | F1 (%) | Time |
|---|---|---|---|
| Subject-level | 81.0 | 78.6 | |
| Video-level (online) | 80.8 | 78.0 |
| Method | WAR (%) | F1 (%) |
|---|---|---|
| TTA-CaP | 81.0 | 78.6 |
| w/o Positive Cache | 79.2 | 76.8 |
| w/o Negative Cache | 80.0 | 77.6 |
| w/o Tri-Gate | 78.7 | 76.0 |
| w/o Temporal Encoder | 80.6 | 78.1 |
| w/o Personalized Static Cache | 78.1 | 77.9 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Time (ms) | Memory (MB) | WAR (%) |
| TPT | 771.2 | 2800 | 71.1 |
| PromptAlign | 900.0 | 3100 | 75.3 |
| TDA | 97.8 | 2124 | 71.4 |
| DPE | 263.9 | 1700 | 73.1 |
| MCP | 145.0 | 1450 | 75.7 |
| ReTA | 174.0 | 2390 | 75.1 |
| Representation | Current-class | Non-current-class | |
| Pseudo-label: Stress | |||
| 0.61 | 0.47 | 0.14 | |
| 0.58 | 0.33 | 0.25 | |
| Pseudo-label: Neutral | |||
| 0.63 | 0.45 | 0.18 | |
| 0.60 | 0.31 | 0.29 | |
| Method | WAR (%) | F1 (%) |
| MCP | 75.7 | 73.0 |
| MCP + Temporal | 75.9 | 73.4 |
| MCP + Temporal + Personalized Cache | 77.6 | 75.3 |
| TTA-CaP | 81.0 | 78.6 |
| Backbone | Method | WAR (%) |
| RN101 | Source FT | 64.8 |
| TTA-CaP | 76.9 | |
| ViT-B/16 | Source FT | 66.1 |
| TTA-CaP | 79.6 | |
| ViT-B/32 | Source FT | 67.7 |
| TTA-CaP | 81.0 |
| Time (ms) | Memory (MB) | WAR (%) | |
| 1 | 104 | 870 | 76.2 |
| 2 | 107 | 885 | 78.2 |
| 3 | 110 | 904 | 81.0 |
| 4 | 113 | 921 | 81.0 |
| 5 | 116 | 938 | 81.0 |
| Window size | TTA-CaP w/o PSC | TTA-CaP |
| 16 | 41.1 | 41.0 |
| 24 | 42.0 | 43.5 |
| 32 | 42.2 | 43.8 |
| Method | Logit ECE | Embed ECE |
| TDA | 20.7 | 15.2 |
| ReTA | 19.2 | 14.0 |
| TTA-CaP | 17.8 | 12.3 |
| Weighting Strategy | WAR (%) |
| Conf.-weighted | 78.9 |
| Attention-based | 77.4 |
| Equal-weight | 81.0 |
| Prompt Template | WAR | F1 score |
| a photo of a [CLS] | 68.7 | 63.3 |
| a_photo_of_the_[CLS]_face | 66.5 | 59.3 |
| a_photo_of_one_[CLS]_face | 64.2 | 55.8 |
| a_close-up_photo_of_the_[CLS]_face | 68.2 | 62.6 |
| a_low_resolution_photo_of_a_[CLS]_face | 68.7 | 64.2 |
| a_good_photo_of_a_[CLS]_face | 69.2 | 65.4 |
| Cache construction | WAR (%) |
| Baseline (k-means, ) | 78.8 |
| Two-stage (k-means) | 79.3 |
| Two-stage (DBSCAN) | 79.5 |
| Personalized (k-means) | 79.6 |
| Personalized (DBSCAN) | 81.0 |
| Method | Sub-1 | Sub-2 | Sub-3 | Sub-4 | Sub-5 | Sub-6 | Sub-7 | Sub-8 | Sub-9 | Sub-10 | Avg. |
| TPT [ 29 ] (NeurIPS’22) | 40.6 | 35.7 | 43.7 | 45.9 | 45.6 | 43.1 | 35.0 | 34.7 | 40.7 | 32.9 | 39.7 |
| TDA [ 12 ] (CVPR’24) | 40.0 | 39.7 | 44.4 | 46.0 | 42.0 | 42.6 | 35.0 | 35.5 | 40.7 | 33.2 | 39.9 |
| DPE [ 45 ] (NeurIPS’24) | 34.8 | 37.1 | 44.2 | 46.0 | 45.3 | 43.0 | 37.2 | 32.5 | 41.0 | 33.2 | 39.4 |
| PromptAlign [ 1 ] (NeurIPS’23) | 25.0 | 37.0 | 46.9 | 51.0 | 46.1 | 43.5 | 41.0 | 32.5 | 41.0 | 33.2 | 39.7 |
| ReTA [ 17 ] (ACMMM’25) | 25.0 | 36.9 | 41.9 | 51.0 | 46.1 | 51.2 | 40.0 | 32.5 | 41.0 | 33.2 | 39.8 |
| MCP [ 5 ] (ICCV’25) | 38.5 | 37.8 | 44.5 | 48.0 | 45.5 | 45.0 | 39.0 | 33.5 | 40.7 | 28.5 | 40.1 |
| Method | Sub-1 | Sub-2 | Sub-3 | Sub-4 | Sub-5 | Sub-6 | Sub-7 | Sub-8 | Sub-9 | Sub-10 | Avg. |
| TPT | 91.9 0.12 | 53.0 0.31 | 50.0 0.00 | 61.5 0.36 | 79.0 0.19 | 88.9 0.14 | 76.0 0.12 | 50.0 0.00 | 98.9 0.05 | 62.0 0.28 | 71.1 |
| TDA | 93.0 0.10 | 52.6 0.29 | 50.0 0.00 | 63.9 0.32 | 79.5 0.17 | 87.5 0.16 | 76.3 0.18 | 50.0 0.00 | 100.0 0.00 | 61.6 0.26 | 71.4 |
| DPE | 91.7 0.13 | 55.0 0.27 | 50.0 0.00 | 85.0 0.18 | 73.0 0.24 | 84.7 0.20 | 80.0 0.16 | 50.0 0.00 | 100.0 0.00 | 61.8 0.20 | 73.1 |
| PromptAlign | 93.8 0.09 | 65.2 0.21 | 50.0 0.00 | 70.0 0.24 | 85.0 0.14 | 92.0 0.10 | 80.0 0.15 | 50.0 0.00 | 100.0 0.00 | 67.6 0.20 | 75.3 |
| ReTA | 93.8 0.08 | 65.2 0.19 | 50.0 0.00 | 69.5 0.22 | 85.0 0.13 | 90.2 0.12 | 80.0 0.14 | 50.0 0.00 | 100.0 0.00 | 67.6 0.18 | 75.1 |
| MCP [ 5 ] | 93.9 0.08 | 65.8 0.20 | 50.0 0.00 | 72.0 0.20 | 85.5 0.13 | 92.5 0.10 | 81.5 0.14 | 50.0 0.00 | 100.0 0.00 | 66.0 0.18 | 75.7 |
| Method | Sub-1 | Sub-2 | Sub-3 | Sub-4 | Sub-5 | Sub-6 | Sub-7 | Sub-8 | Sub-9 | Sub-10 | Avg. |
| TPT | 74.7 0.18 | 51.6 0.21 | 90.0 0.09 | 88.0 0.12 | 52.2 0.24 | 77.5 0.11 | 66.6 0.22 | 63.4 0.19 | 54.0 0.23 | 91.9 0.08 | 70.9 |
| TDA | 66.9 0.22 | 45.2 0.28 | 75.6 0.17 | 90.1 0.10 | 49.6 0.24 | 80.0 0.05 | 41.2 0.31 | 75.3 0.18 | 81.8 0.02 | 91.9 0.07 | 69.7 |
| DPE | 64.0 0.23 | 55.0 0.20 | 73.9 0.18 | 90.0 0.09 | 52.6 0.22 | 80.0 0.05 | 50.9 0.27 | 73.0 0.19 | 81.8 0.07 | 92.0 0.07 | 71.3 |
| PromptAlign | 69.9 0.20 | 61.6 0.17 | 78.6 0.14 | 90.9 0.08 | 51.2 0.21 | 80.0 0.15 | 60.4 0.19 | 81.3 0.13 | 81.8 0.06 | 90.9 0.09 | 74.6 |
| ReTA | 64.5 0.21 | 50.5 0.23 | 82.0 0.12 | 90.0 0.09 | 46.8 0.25 | 80.0 0.12 | 49.3 0.29 | 80.0 0.04 | 81.8 0.10 | 93.7 0.06 | 71.8 |
| MCP [ 5 ] | 72.5 0.19 | 61.0 0.18 | 81.5 0.13 | 90.5 0.09 | 55.0 0.21 | 80.0 0.08 | 65.0 0.20 | 78.0 0.14 | 81.8 0.06 | 89.7 0.07 | 75.5 |
| Method | Sub-1 | Sub-2 | Sub-3 | Sub-4 | Sub-5 | Sub-6 | Sub-7 | Sub-8 | Sub-9 | Sub-10 | Avg. |
| TPT | 62.0 0.18 | 53.7 0.19 | 73.9 0.14 | 80.0 0.11 | 81.2 0.10 | 71.0 0.16 | 58.0 0.17 | 55.0 0.18 | 69.0 0.15 | 53.0 0.20 | 65.6 |
| TDA | 58.0 0.20 | 53.7 0.18 | 75.0 0.13 | 80.3 0.10 | 79.0 0.12 | 71.6 0.15 | 58.0 0.16 | 55.9 0.17 | 69.0 0.14 | 51.7 0.22 | 65.2 |
| DPE | 58.6 0.19 | 59.6 0.16 | 79.6 0.10 | 80.5 0.10 | 83.4 0.09 | 67.0 0.18 | 59.0 0.15 | 54.0 0.19 | 71.9 0.12 | 54.0 0.18 | 66.7 |
| PromptAlign | 59.1 0.18 | 59.1 0.16 | 72.0 0.15 | 84.5 0.09 | 84.1 0.08 | 72.4 0.14 | 58.5 0.16 | 55.9 0.17 | 71.5 0.12 | 54.9 0.17 | 67.1 |
| ReTA | 61.4 0.17 | 60.0 0.15 | 72.0 0.14 | 84.0 0.09 | 84.1 0.08 | 72.4 0.13 | 58.5 0.15 | 55.9 0.16 | 72.0 0.11 | 56.0 0.16 | 67.6 |
| MCP [ 5 ] | 61.0 0.17 | 59.5 0.16 | 74.0 0.14 | 84.5 0.09 | 83.5 0.09 | 73.0 0.13 | 59.0 0.15 | 55.5 0.17 | 71.5 0.12 | 54.5 0.17 | 67.6 |
Test-Time Adaptation
Test-time adaptation updates or recalibrates pretrained models at inference to handle distribution shifts, often without target labels.
See all 322 papers