Organizations: School of Electronics and Communication Engineering Sun Yat-sen University, Shenzhen Campus Shenzhen, China · Department of Network Intelligence Pengcheng Laboratory Shenzhen, China
Continuous emotion regression estimates moment-to-moment valence and arousal while a viewer watches a video. In familiar-video deployment, responses fron training participant-specific estimate, and prior-dominating fixed fusion tests whether physiology adds residual correction. In five-fold subject-held-out evaluation on 24was within 0.05 and 0.32 MAE of fusion in the internal and external evaluations, respectively. Source-explicit ablations showed that video identity and within-video tine accounted for most of the reduction, while EG-FNIRS gains were smaller and varied across participants and videos. These results identify the video-time prior as a strong, low-cost baseline and position EEG-fNIRS as an optional residual signal for familiar-video emotion regression.
Figures & tables
Figure 1. Method overview. (A) A new viewer contributes EEG–fNIRS at second t of a familiar video. (B) The video–time prior branch (blue) aggregates training-participant labels via a fold-wise cross-participant median and radius-three within-video smoothing to yield pˉv,t ; held-out labels are excluded. The EEG–fNIRS branch (orange) extracts one-second features from t−1 , t , and t+1 , applies graph encoders and bidirectional cross-modal attention, and regresses ms,v,t . (C) Prior-dominating fixed fusion with valence and arousal prior weights w=[0.99,0.92] produces y^s,v,t . A three-panel overview shows a new viewer watching a familiar video, the leakage-controlled video–time prior and participant-specific EEG–fNIRS branches, and their fixed fusion into a predicted valence–arousal trajectory.
Eval.
Target
Grid min. w∗/MAE
Near-min. range
Fixed w/Δ
Internal
Valence
.99 / 26.664
.93–1.00
.99 / +.000
Arousal
.92 / 31.365
.85–1.00
.92 / +.000
External
Valence
.83 / 23.748
.79–.86
.99 / +1.449
Arousal
.82 / 30.073
.77–.89
.92 / +.180
Table 1. Descriptive fusion-weight sensitivity on a 0.01 grid. The near-minimum range contains weights with target MAE no more than 0.10 above the grid minimum. Internal results use pooled out-of-fold predictions and are not nested selection estimates; external results are post hoc diagnostics that were not used to change the reported fixed weights. Δ is the fixed-weight MAE minus the grid-minimum MAE.
Variant
Participant phys.
Video key
Time key
Label lookup
Global constant
N
N
N
Y
Video identity
N
Y
N
Y
Video–time prior
N
Y
Y
Y
EEG–fNIRS branch
Y
N
N
N
Fixed fusion
Y
Y
Y
Y
Table 2. Information used by each source variant at inference. “Label lookup” means inference-time retrieval of a cached statistic derived only from labels in the corresponding training split; evaluated-participant labels never enter the statistic. Supervised models still use training targets. “Time key” means explicit lookup by within-video playback time, not temporal context inside the EEG–fNIRS branch.
Evaluation
Target
Global
Video ID
Video–time prior
EEG–fNIRS
Fusion
Internal
Overall
–
–
29.06
47.35
29.01
Valence
–
–
26.67
51.35
26.66
Arousal
–
–
31.46
43.35
31.37
External
Overall
44.33
33.36
28.04
42.75
27.72
Valence
47.46
32.56
25.34
45.29
25.20
Arousal
41.20
34.17
30.74
40.21
30.25
Table 3. Internal and external MAE on the original label scale. Lower is better; dashes mark source variants not reported for the internal evaluation.
Figure 2. Internal subject-held-out MAE. (a) Overall MAE for the EEG–fNIRS branch, video–time prior, and fixed fusion. (b) Additional MAE reduction from the prior to fusion by target. A two-panel figure compares overall internal MAE and the small additional reduction from the video–time prior to fixed fusion.
Figure 3. External fixed-fusion predicted-versus-observed densities. (a,b) Hexbin densities compare predictions with observed valence and arousal, respectively, for 6,143 one-second samples. The dashed diagonal marks exact prediction; dotted horizontal and vertical lines mark the neutral score of 128. Colour gives sample count on a logarithmic scale, and panel titles report MAE on the original [1,255] scale. Both panels show prediction-range compression toward the middle of the scale, particularly for arousal. Two hexbin plots compare fixed-fusion predictions with observed valence and arousal. Dashed diagonals mark exact prediction, dotted lines mark the neutral score of 128, and predicted arousal is concentrated mid-range.
Figure 4. External MAE and residual analysis. (a) Overall MAE for five source variants. (b,c) Participant- and video-level prior-minus-fusion increments; positive values favour fusion. (d) MAE over normalized video-time bins; this panel focuses on label-derived variants and fusion. A two-by-two four-panel figure shows external overall error, paired participant- and video-level fusion increments, and error over normalized video time. Panel d omits the EEG–fNIRS-only curve.
Figure 5. External video–time MAE. (a) Video–time prior and (b) fixed fusion across 15 familiar videos and ten equal bins of normalized playback time. Each cell equally pools valence and arousal absolute errors over all one-second samples assigned to that video–time cell. Both panels use the same fixed 0–65 MAE scale, with deep blue denoting lower error and orange–red denoting higher error. Overall MAE was 28.04 for the prior and 27.72 for fusion. Two side-by-side heatmaps compare video–time prior and fixed fusion MAE across 15 videos and ten normalized playback-time bins. Deep blue denotes lower error and orange–red denotes higher error; the strongest hotspot occurs in the latter half of video V1.
Continuous, second-by-second valence-arousal estimation from physiological signals is typically studied in a subject-dependent setting, where the model sees labeled data from the same person it is later evaluated on. We study the harder zero-shot cross-subject variant on a synchronized EEG-fNIRS dataset: predict raw-scale ([1, 255]) valence and arousal trajectories for subjects whose labels the model never observes, given only their unlabeled EEG/fNIRS recordings while watching the same video stimuli as a disjoint set of training subjects. We decompose the affect trajectory into a structure shared across subjects who watch the same stimuli and an individual structure estimated for each test subject from a label-free EEG marker (alpha-band cross-channel synchrony), which rescales the shared trajectory around the scale midpoint. We validate the per-subject calibration mechanism on four independent axes: leave-one-subject-out correlation between the marker and each subject's true optimal gain, a functional-form comparison against non-linear alternatives, a repeated leave-4-out component ablation isolating each part of the pipeline's contribution, and a ceiling analysis bounding the remaining headroom for per-subject scaling. On held-out subjects, the model reaches an overall MAE of 25.96 / 22.80 across two evaluation batches (valence 21.94 / 19.6, arousal 29.98 / 26.0), well below EEGNet and ASAC-Net baselines reported for the same subject-independent split (raw scale score 60.6 and 55.0 respectively). We further report a systematic negative-result search across model architectures, feature representations, and prediction targets that found no signal able to improve on the single alpha-synchrony marker.
Automatic emotion assessment can benefit from combining neural and behavioral signals, but many multimodal approaches rely on separate, modality-specific feature-extraction pipelines before fusion. This paper presents MUPA\textsuperscript{2}E, a unified perception framework that processes facial video and electroencephalography (EEG) through a single shared asymmetric-attention backbone. Facial video is represented through axis-folded frame tokens, while EEG is processed either as a raw multichannel waveform or projected into the spatial domain for multimodal fusion. The framework is evaluated on the DMER dataset under a stratified subject-independent protocol, comparing unimodal video, unimodal EEG, and fused video--EEG configurations with per-channel and merged EEG projections. Using the original recordings, with shorter trials zero-padded to match the longest duration, merged fusion at stride~30 achieves the highest validation performance and a test accuracy of 70.07%. Further analysis revealed that recording duration is unevenly distributed across the affective classes, making the padding pattern a potential classification cue. Controlling for this factor by cropping all recordings to a common duration of 20 seconds yielded a test accuracy of 62.71%, providing a stricter duration-controlled assessment of the framework in which differences in recording length are removed as a potential classification cue. These findings demonstrate the feasibility of processing structurally different neural and visual signals within a compact unified architecture while highlighting the importance of controlling duration-related cues in affective datasets.
Stefanos Gkikas, Eric Nichols, Christian Arzate Cruz +1
Mixed emotions represent a clinically relevant but still underexplored target for automatic emotion recognition. EEG provides millisecond-level access to neural activity, yet most EEG pipelines analyze the signal through a single temporal window, thereby fixing the temporal structure available to the model. This study introduces a multi-scale temporal framework for EEG-based emotion recognition. The EEG waveform is decomposed into windows of one or several durations, processed by a shared attention-based encoder, and integrated through a dynamic fusion module that assigns sample-specific weights across temporal scales. The framework is evaluated under a subject-independent protocol in binary and three-class settings, with the three-class task including the mixed affective category. The best results are 65.22% for the two-class task and 45.43% for the three-class task. Both are obtained with three-scale dynamic-fusion configurations and remain substantially above the full-signal baseline. The best-performing temporal scales differ between the two tasks. Dynamic fusion outperforms concatenation in the highest-scoring two-class configuration and slightly exceeds it in the highest-scoring three-class configuration, although these multi-scale settings require substantially more computation than the full-signal baseline.
Stefanos Gkikas, Yang Guo, Guangliang Li +3
Honda Research Institute Japan Wako City, Japan · Faculty of Information Science and Engineering Ocean University of China Qingdao, China · BioSIS (Biosensing & Intelligent Systems) Lab Centre for Intelligent Computing and Systems University of Canberra Canberra, Australia +1