cs.AINov 3, 2026

Self-Supervised Pretraining for Label-Efficient Facial Expression Recognition using Smart Glasses

Authors: Zhengtao Gao, Francesco Bombassei De Bona, Leonardo Alchieri, Ivana Kiprijanovska, Simon Stankoski, Bojan Sofronievski, Hristijan Gjoreski, Marc Langheinrich, 

Organizations: Università della Svizzera italiana, Lugano, Switzerland · Emteq Ltd., Brighton, United Kingdom · Ss. Cyril and Methodius University in Skopje, Skopje, North Macedonia

Abstract

Facial Expression Recognition (FER) on optomyographic (OMG) smart glasses such as OCOsense enables privacy-preserving, camera-free affect sensing for health and well-being. Most FER systems are fully supervised, i.e., they rely either on hand-crafted expert features with classical machine learning, or on end-to-end deep models trained from labelled data. Both depend on large quantities of labelled expressions, which are costly to collect and limit deployment in real-world, personalized settings. We investigate Self-Supervised Learning (SSL) as a route to label-efficient FER on these glasses. We pretrain a Vision Transformer with a masked-autoencoder objective on unlabelled OCOsense recordings and finetune it for five-class FER under person-independent cross-validation. Across label budgets from 5% to 100%, pretraining consistently improves performance; with only 5% of the labels, the pretrained model outperforms the from-scratch model trained with all labels. Our results show that SSL pretraining is a practical way to reduce labelling cost for smart eyewear FER, and we discuss the implications for future personalization and continual adaptation.

Figures & tables

Explore similar work

May 19, 2026

LaCoVL-FER: Landmark-Guided Contrastive Learning Network with Vision-Language Enhancement for Facial Expression Recognition

Facial Expression Recognition (FER) in the wild requires models to identify subtle expression cues under large variations in pose, occlusion, illumination, and identity. Recent FER methods improve robustness by introducing visual attention, facial landmarks, or vision-language models as auxiliary priors. However, these priors are typically integrated in a static manner, failing to capture instance-specific facial variations, thereby resulting in severe attention redundancy and representation instability. To address this issue, we propose LaCoVL-FER, a landmark-guided contrastive learning network with vision-language enhancement for FER, which shifts FER from static prior injection to sample-adaptive prior refinement. Specifically, a Landmark-Guided Adaptive Encoder (LGAE) calibrates regional appearance features with landmark geometry through Bi-branch Gated Cross Attention (BGCA), suppressing noisy responses and producing expression-relevant representations. In parallel, a Vision-Language Enhancement Strategy (VLES) refines the generalizable visual features from a frozen CLIP image encoder into expression-specific visual representations. Based on them, an Expression-Conditioned Prompting (ECP) mechanism adapts fixed class-level textual prompts from the frozen CLIP text encoder into instance-aware textual representations. The resulting visual-textual representations are aligned as adaptive semantic priors to enhance robustness and generalization. Quantitative and qualitative experiments show that LaCoVL-FER outperforms state-of-the-art methods on RAF-DB, FERPlus, and AffectNet. The code is available at https://github.com/ylin06804/LaCoVL-FER.
Oct 8, 2026

SAGE: Efficient Protected Vision-Language Inference on Smart Glasses

Smart glasses are a promising platform for vision-language assistants, but private split inference remains expensive: protecting visual representations before transmission does not remove the cost of on-device vision encoding, wide protected tokens, or hardware-unaware execution. We present SAGE (Smart-glasses-Aware Guarded Encoding), a hardware-aware framework for efficient protected VLM inference on smart glasses. SAGE keeps raw images and unprotected visual tokens on the trusted device, reduces visual input cost, executes shape-specialized INT8 graphs, and compresses protected representations into compact 256-dimensional INT8 packets before transmission. Unlike efficiency methods optimized only for nominal model size, SAGE selects its operating point using downstream answer quality together with measured latency, memory, and thermal behavior on real Mentra Live glasses. On a matched Qwen workload, SAGE reduces glasses inference latency by 39.18%, device-sensor temperature rise by 21.91%, and protected packet size by 96.56%, while retaining 85.62% of the protected reference's relaxed utility and comparable prompted containment. Direct glasses-to-phone tests deliver all 192 packets and reduce median packet-plus-acknowledgment time from 71.3 to 10.0 ms. The compact protected interface also transfers across four additional VLM families. These results show that private wearable VLM inference must be optimized jointly across representation protection, communication, and real-device execution rather than through model compression alone.
Apr 10, 2026

ActFER: Agentic Facial Expression Recognition via Active Tool-Augmented Visual Reasoning

Recent advances in Multimodal Large Language Models (MLLMs) have created new opportunities for facial expression recognition (FER), moving it beyond pure label prediction toward reasoning-based affect understanding. However, existing MLLM-based FER methods still follow a passive paradigm: they rely on externally prepared facial inputs and perform single-pass reasoning over fixed visual evidence, without the capability for active facial perception. To address this limitation, we propose ActFER, an agentic framework that reformulates FER as active visual evidence acquisition followed by multimodal reasoning. Specifically, ActFER dynamically invokes tools for face detection and alignment, selectively zooms into informative local regions, and reasons over facial Action Units (AUs) and emotions through a visual Chain-of-Thought. To realize such behavior, we further develop Utility-Calibrated GRPO (UC-GRPO), a reinforcement learning algorithm tailored to agentic FER. UC-GRPO uses AU-grounded multi-level verifiable rewards to densify supervision, query-conditional contrastive utility estimation to enable sample-aware dynamic credit assignment for local inspection, and emotion-aware EMA calibration to reduce noisy utility estimates while capturing emotion-wise inspection tendencies. This algorithm enables ActFER to learn both when local inspection is beneficial and how to reason over the acquired evidence. Comprehensive experiments show that ActFER trained with UC-GRPO consistently outperforms passive MLLM-based FER baselines and substantially improves AU prediction accuracy.