Predicting affect in unconstrained environments remains a fundamental challenge in human-centered AI. While deep neural embeddings dominate contemporary approaches, they often lack interpretability and limit expert-driven refinement. We propose a novel framework that uses Language Models (LMs) as semantic context conditioners over handcrafted affect descriptors to model changes in Valence and Arousal. Our approach begins with interpretable facial geometry and acoustic features derived from structured domain knowledge. These features are transformed into symbolic natural-language descriptions encoding their affective implications. A pretrained LM processes these descriptions to generate semantic context embeddings that act as high-level priors over affective dynamics. Unlike end-to-end black-box pipelines, our framework preserves feature transparency while leveraging the contextual abstraction capabilities of LMs. We evaluate the proposed method on the Aff-Wild2 and SEWA datasets for affect change prediction. Experimental results show consistent improvements in accuracy for both Valence and Arousal compared to handcrafted-only and deep-embedding baselines. Our findings demonstrate that semantic conditioning enables interpretable affect modelling without sacrificing predictive performance, offering a transparent and computationally efficient alternative to fully end-to-end architectures
Figures & tables
Figure 1: Overview of the LaScA framework. Handcrafted facial and acoustic features are first extracted and filtered via data-driven salience estimation (Otsu’s threshold). Each feature is associated with a fixed affect-aware semantic description generated offline by a large language model, forming a deterministic semantic lexicon. Active feature descriptions are composed into structured templates and encoded using a frozen sentence transformer to produce semantic embeddings. These embeddings are fused with the original feature representations and used to construct preference pairs between temporal windows. A trainable MLP operates on the transformed pairwise representations to predict directional affect change (increase or decrease).
Model
Backbone
Pooling
Dim
all-mpnet-base-v2
MPNet
Mean
768
multi-qa-mpnet-base-dot-v1
QAMPNet
CLS
768
all-distilroberta-v1
DistilRoBERTa
Mean
768
all-MiniLM-L12-v2
MiniLM
Mean
384
multi-qa-distilbert-cos-v1
DistilBERT
Mean
768
Table 1: Pretrained sentence encoders evaluated in this work.
Visual
Backbone
Pooling
Dim
VGGFace2
CNN
Global Avg Pool
2048
SwinFace
Swin Transformer
Token Pooling
768
MAE-Face
Transformer (MAE)
CLS Token
768
Audio
Backbone
Pooling
Dim
Wav2Vec2
Wav2Vec2 Base
Mean over frames
768
MAE-Audio
Transformer (MAE)
CLS / Global Pool
512
Table 2: Pretrained baseline encoders used for benchmarking the proposed framework.
Arousal
Valence
Modality
Model
3s
5s
3s
5s
10%
20%
10%
20%
10%
20%
10%
20%
Visual
Features
0.65
0.59
0.69
0.61
0.60
0.60
0.67
0.63
MPNet
0.56
0.51
0.60
0.58
0.54
0.56
0.58
0.66
QAMPNet
0.56
0.51
0.60
0.60
0.56
0.56
0.61
0.67
DRoBERTa
0.58
0.50
0.62
0.64
0.56
0.53
0.59
0.65
Table 3: Aff-Wild2 accuracy of the LaScA framework across modalities, window sizes, thresholds and backbones. “Features” and “Sentence Transformers” denote feature-only and text-only models, respectively. The (F) suffix indicates multimodal fusion of text embeddings and features. Bold values represent the highest accuracy; underlined values are statistically on par with the best-performing models.
Arousal
Valence
Modality
Model
3s
5s
3s
5s
10%
20%
10%
20%
10%
20%
10%
20%
Visual
Features
0.51
0.50
0.50
0.50
0.52
0.50
0.50
0.50
MPNet
0.54
0.56
0.53
0.68
0.57
0.63
0.61
0.74
QAMPNet
0.58
0.57
0.55
0.67
0.58
0.68
0.68
0.70
DRoBERTa
0.55
0.62
0.53
0.61
0.60
0.69
0.67
0.69
Table 4: SEWA DB accuracy of the LaScA framework across modalities, window sizes, thresholds and backbones. “Features” and “Sentence Transformers” denote feature-only and text-only models, respectively. The (F) suffix indicates multimodal fusion of text embeddings and features. Bold values represent the highest accuracy; underlined values are statistically on par with the best-performing models.
Arousal
Valence
Modality
Model
3s
5s
3s
5s
10%
20%
10%
20%
10%
20%
10%
20%
Visual
VGGFace2
0.64
0.63
0.66
0.71
0.57
0.60
0.65
0.72
SwinFace
0.67
0.67
0.70
0.74
0.61
0.63
0.69
0.73
MAE-Face
0.65
0.64
0.71
0.72
0.58
0.61
0.67
0.71
LaScA (ours)
0.66
0.65
0.69
0.74
0.59
0.62
0.68
0.74
Table 5: Quantitative comparison of our method with state-of-the-art approaches for Aff-Wild2 prediction. Bold values indicate the best-performing model, while underlined values indicate models performing statistically on par.
Arousal
Valence
Modality
Model
3s
5s
3s
5s
10%
20%
10%
20%
10%
20%
10%
20%
Visual
VGGFace2
0.62
0.66
0.61
0.67
0.66
0.74
0.69
0.79
SwinFace
0.67
0.70
0.65
0.71
0.72
0.79
0.73
0.82
MAE-Face
0.64
0.68
0.66
0.70
0.69
0.77
0.71
0.81
LaScA (ours)
0.65
0.69
0.64
0.70
0.70
0.78
0.72
0.83
Table 6: Quantitative comparison of our method with state-of-the-art approaches for SEWA DB prediction. Bold values indicate the best-performing model, while underlined values indicate models performing statistically on par.
Arousal
Valence
Multimodal
LaScA
3s
5s
3s
5s
10%
20%
10%
20%
10%
20%
10%
20%
MPNet(F)
Feature
0.63
0.67
0.65
0.74
0.56
0.59
0.60
0.61
LLM (ours)
0.65
0.69
0.67
0.74
0.58
0.60
0.61
0.61
DRoBERTa(F)
Feature
0.64
0.68
0.66
0.74
0.56
0.59
0.60
0.61
LLM (ours)
0.65
0.70
0.67
0.74
0.59
0.60
0.61
0.63
Table 7: Quantitative comparison of feature-based and LLM-based Lexicon approach for Aff-Wild2. Results are reported across Arousal and Valence dimensions for different window sizes (3s, 5s) and selection thresholds (10%, 20%). Bold values indicate the best-performing model for each configuration.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Feature Name
Semantic Label
mfcc_0
high arousal energy
mfcc_1
dominant low tone
mfcc_2
neutral stability
mfcc_3
tense vocal focus
mfcc_4
clear controlled tone
mfcc_5
urgent brightness
Appendix
Table 8: Affect-aware semantic labels for MFCC acoustic features generated offline.
Feature Name
Semantic Label
Face_browDownLeft
left brow down focused irritation
Face_browDownRight
right brow down skeptical tension
Face_browInnerUp
inner brows up sad vulnerability
Face_browOuterUpLeft
left outer brow up curious surprise
Face_browOuterUpRight
right outer brow up questioning surprise
Face_cheekPuff
cheeks puff held frustration
Appendix
Table 9: Affect-aware semantic lexicon for facial blendshape features.
Ubiquitous Knowledge Processing Lab (UKP Lab), Technical University of Darmstadt · National Research Center for Applied Cybersecurity ATHENE, Germany · Psychiatric Control Systems Lab, Marburg University
Department of Computer Science, NTNU,Trondheim 7491, Norway · Kavli Institute for Systems Neuroscience, NTNU, Trondheim 7491, Norway · Microsoft, Trondheim 7012, Norway