Self-Supervised Pretraining for Label-Efficient Facial Expression Recognition using Smart Glasses
Organizations: Università della Svizzera italiana, Lugano, Switzerland · Emteq Ltd., Brighton, United Kingdom · Ss. Cyril and Methodius University in Skopje, Skopje, North Macedonia
Abstract
Facial Expression Recognition (FER) on optomyographic (OMG) smart glasses such as OCOsense enables privacy-preserving, camera-free affect sensing for health and well-being. Most FER systems are fully supervised, i.e., they rely either on hand-crafted expert features with classical machine learning, or on end-to-end deep models trained from labelled data. Both depend on large quantities of labelled expressions, which are costly to collect and limit deployment in real-world, personalized settings. We investigate Self-Supervised Learning (SSL) as a route to label-efficient FER on these glasses. We pretrain a Vision Transformer with a masked-autoencoder objective on unlabelled OCOsense recordings and finetune it for five-class FER under person-independent cross-validation. Across label budgets from 5% to 100%, pretraining consistently improves performance; with only 5% of the labels, the pretrained model outperforms the from-scratch model trained with all labels. Our results show that SSL pretraining is a practical way to reduce labelling cost for smart eyewear FER, and we discuss the implications for future personalization and continual adaptation.
Figures & tables
| Data | Acc | F1 | AUC | MCC | ||||
|---|---|---|---|---|---|---|---|---|
| FS | Pretrain | FS | Pretrain | FS | Pretrain | FS | Pretrain | |
| 5% | ||||||||
| 10% | ||||||||
| 20% | ||||||||
| 40% | ||||||||
| 80% | ||||||||
| Input | Acc | F1 | AUC | MCC |
|---|---|---|---|---|
| Baseline | ||||
| IMU | ||||
| Euler | ||||
| Prox | ||||
| Nav | ||||
| IMU + Euler |
Facial Expression Recognition
Facial Expression Recognition studies how computers infer emotion and facial actions from images and video.
See all 85 papers