cs.AIOct 2, 2026
SaveLow-Cost Video--Time Priors as a Strong Baseline for EEG--fNIRS Emotion Regression on Familiar Videos
Organizations: School of Electronics and Communication Engineering Sun Yat-sen University, Shenzhen Campus Shenzhen, China · Department of Network Intelligence Pengcheng Laboratory Shenzhen, China
Abstract
Continuous emotion regression estimates moment-to-moment valence and arousal while a viewer watches a video. In familiar-video deployment, responses fron training participant-specific estimate, and prior-dominating fixed fusion tests whether physiology adds residual correction. In five-fold subject-held-out evaluation on 24was within 0.05 and 0.32 MAE of fusion in the internal and external evaluations, respectively. Source-explicit ablations showed that video identity and within-video tine accounted for most of the reduction, while EG-FNIRS gains were smaller and varied across participants and videos. These results identify the video-time prior as a strong, low-cost baseline and position EEG-fNIRS as an optional residual signal for familiar-video emotion regression.
Figures & tables
Figure 1. Method overview. (A) A new viewer contributes EEG–fNIRS at second of a familiar video. (B) The video–time prior branch (blue) aggregates training-participant labels via a fold-wise cross-participant median and radius-three within-video smoothing to yield ; held-out labels are excluded. The EEG–fNIRS branch (orange) extracts one-second features from , , and , applies graph encoders and bidirectional cross-modal attention, and regresses . (C) Prior-dominating fixed fusion with valence and arousal prior weights produces . A three-panel overview shows a new viewer watching a familiar video, the leakage-controlled video–time prior and participant-specific EEG–fNIRS branches, and their fixed fusion into a predicted valence–arousal trajectory.
| Eval. | Target | Grid min. | Near-min. range | Fixed |
|---|---|---|---|---|
| Internal | Valence | .99 / 26.664 | .93–1.00 | .99 / +.000 |
| Arousal | .92 / 31.365 | .85–1.00 | .92 / +.000 | |
| External | Valence | .83 / 23.748 | .79–.86 | .99 / +1.449 |
| Arousal | .82 / 30.073 | .77–.89 | .92 / +.180 |
Table 1. Descriptive fusion-weight sensitivity on a 0.01 grid. The near-minimum range contains weights with target MAE no more than 0.10 above the grid minimum. Internal results use pooled out-of-fold predictions and are not nested selection estimates; external results are post hoc diagnostics that were not used to change the reported fixed weights. is the fixed-weight MAE minus the grid-minimum MAE.
| Variant | Participant phys. | Video key | Time key | Label lookup |
|---|---|---|---|---|
| Global constant | N | N | N | Y |
| Video identity | N | Y | N | Y |
| Video–time prior | N | Y | Y | Y |
| EEG–fNIRS branch | Y | N | N | N |
| Fixed fusion | Y | Y | Y | Y |
Table 2. Information used by each source variant at inference. “Label lookup” means inference-time retrieval of a cached statistic derived only from labels in the corresponding training split; evaluated-participant labels never enter the statistic. Supervised models still use training targets. “Time key” means explicit lookup by within-video playback time, not temporal context inside the EEG–fNIRS branch.
| Evaluation | Target | Global | Video ID | Video–time prior | EEG–fNIRS | Fusion |
|---|---|---|---|---|---|---|
| Internal | Overall | – | – | 29.06 | 47.35 | 29.01 |
| Valence | – | – | 26.67 | 51.35 | 26.66 | |
| Arousal | – | – | 31.46 | 43.35 | 31.37 | |
| External | Overall | 44.33 | 33.36 | 28.04 | 42.75 | 27.72 |
| Valence | 47.46 | 32.56 | 25.34 | 45.29 | 25.20 | |
| Arousal | 41.20 | 34.17 | 30.74 | 40.21 | 30.25 |
Table 3. Internal and external MAE on the original label scale. Lower is better; dashes mark source variants not reported for the internal evaluation.
Figure 2. Internal subject-held-out MAE. (a) Overall MAE for the EEG–fNIRS branch, video–time prior, and fixed fusion. (b) Additional MAE reduction from the prior to fusion by target. A two-panel figure compares overall internal MAE and the small additional reduction from the video–time prior to fixed fusion.
Figure 3. External fixed-fusion predicted-versus-observed densities. (a,b) Hexbin densities compare predictions with observed valence and arousal, respectively, for 6,143 one-second samples. The dashed diagonal marks exact prediction; dotted horizontal and vertical lines mark the neutral score of 128. Colour gives sample count on a logarithmic scale, and panel titles report MAE on the original scale. Both panels show prediction-range compression toward the middle of the scale, particularly for arousal. Two hexbin plots compare fixed-fusion predictions with observed valence and arousal. Dashed diagonals mark exact prediction, dotted lines mark the neutral score of 128, and predicted arousal is concentrated mid-range.
Figure 4. External MAE and residual analysis. (a) Overall MAE for five source variants. (b,c) Participant- and video-level prior-minus-fusion increments; positive values favour fusion. (d) MAE over normalized video-time bins; this panel focuses on label-derived variants and fusion. A two-by-two four-panel figure shows external overall error, paired participant- and video-level fusion increments, and error over normalized video time. Panel d omits the EEG–fNIRS-only curve.
Figure 5. External video–time MAE. (a) Video–time prior and (b) fixed fusion across 15 familiar videos and ten equal bins of normalized playback time. Each cell equally pools valence and arousal absolute errors over all one-second samples assigned to that video–time cell. Both panels use the same fixed 0–65 MAE scale, with deep blue denoting lower error and orange–red denoting higher error. Overall MAE was 28.04 for the prior and 27.72 for fusion. Two side-by-side heatmaps compare video–time prior and fixed fusion MAE across 15 videos and ten normalized playback-time bins. Deep blue denotes lower error and orange–red denotes higher error; the strongest hotspot occurs in the latter half of video V1.