A Multimodal Pre-trained Network for Integrated EEG-Video Seizure Detection

Tong Lu; Ke Xu; Zimo Zhang; Zitong Zhao; Danwei Weng; Ruiyu Wang; Miao Liu; Zizuo Zhang

cs.CVApr 29, 2026

A Multimodal Pre-trained Network for Integrated EEG-Video Seizure Detection

Authors: Tong Lu, Ke Xu, Zimo Zhang, Zitong Zhao, Danwei Weng, Ruiyu Wang, Miao Liu, Zizuo Zhang, +7 more

Organizations: Beijing Institute for Brain Research, Chinese Academy of Medical Sciences & Peking Union Medical College, Beijing, 102206, China · Chinese Institute for Brain Research, Beijing, Beijing, 102206, China · Department of Neurosurgery, SanBo Brain Hospital, Capital Medical University, Beijing, 100018, China · GenAns Biotechnology Co., Ltd, Beijing, 102206, China · Laboratory for Clinical Medicine, Capital Medical University, Beijing, 100069, China · Center of Epilepsy, Beijing Institute for Brain Disorders, Sanbo Brain Hospital, Capital Medical University, Beijing, 100018, China · Beijing Key Laboratory of Brain Science and Brain-Machine Interface, Beijing, 102206, China

Abstract

Reliable seizure detection in mouse models is essential for preclinical epilepsy research, yet manual review of synchronized video-EEG recordings is labor-intensive and single-modality systems fail for complementary reasons: video-based methods are easily confounded by benign behaviors, whereas EEG-based methods are vulnerable to ictal motion artifacts. We present EEGVFusion, a multimodal framework that combines self-supervised EEG representation learning, spatio-temporal video encoding, optimal-transport alignment, and bidirectional cross-attention to integrate neural and behavioral evidence. We also curate an expert-annotated dataset of synchronized EEG and video recordings comprising 93 sessions from 15 mice for training and evaluation. In the random-session split, EEGVFusion achieved a Balanced Accuracy of 0.9957 with perfect event sensitivity and an Event FAR of 0.6250 FP/h, indicating strong seizure detection performance with a low false-alarm burden. In a single held-out-subject evaluation with Subject 110 reserved for testing, EEGVFusion achieved a Balanced Accuracy of 0.9718 and reduced Event FAR from 2.7250 FP/h for the EEG-only counterpart to 0.4833 FP/h while preserving perfect event sensitivity. Targeted ablations further showed that EEG pre-training and OT alignment help reduce false alarms while preserving event sensitivity.

Explore similar work

Multimodal Pretraining for Generalizable EEG Representation Learning
Jul 23, 2026Targol Bakhtiarvand, Jugal Kalita, Adham AtyabiElectroencephalography Foundation Models Interictal Epileptiform Discharges
DiffEEG: A Self-Supervised Denoising Diffusion Model for Learning EEG Generic Representations
Jul 13, 2026Abdulkader Helwan, Lina Abou-Abbas, Hussein El Amouri +2Interictal Epileptiform Discharges Electroencephalography Spatial Super-Resolution
EEG-FuseFormer: A Transformer-Driven Feature Fusion Framework for Seizure Onset Prediction
Jun 1, 2026Vigneshwar Hariharan, Chithra Reghuvaran, Arlene John +4Interictal Epileptiform Discharges Fusion

Jul 23, 2026 · cs.AIJ/K move · Enter open · S save

Multimodal Pretraining for Generalizable EEG Representation Learning

Targol Bakhtiarvand, Jugal Kalita, Adham Atyabi

Department of Computer Science, University of Colorado Colorado Springs, Colorado Springs, CO, USA

Electroencephalography (EEG) models used for epilepsy are often limited to specific datasets and tasks. This limited approach can make it challenging to apply these models across different datasets or in various situations. However, recent studies in foundation models and self-supervised learning suggest that an adaptable EEG backbone could support a range of EEG related tasks. In this study, we have developed a multimodal EEG foundation model that combines a raw signal encoder based on the Mamba architecture, a Vision Transformer (ViT)-style encoder for time-frequency data, and a lightweight encoder for text, all within a shared embedding space. The pretraining process relies on several innovative techniques, such as masked modeling, cross-view contrastive alignment, and temporal consistency losses. These methods are designed to create rich, seizure-relevant representations without requiring labeled data. To assess the efficacy and generalization of our pretrained model, we fine-tuned it on the canonical CHB-MIT seizure detection benchmark and additional seizure detection datasets, and conducted extensive experiments comparing different model variants. On the standard CHB-MIT split, our best single model achieved an AUROC of 0.874, and an ensemble variant reached 0.878 AUROC, representing state-of-the-art performance on this benchmark. In addition to standard train-test splits, we evaluated performance under a leave-one-subject-out (LOSO) protocol, which is rarely reported in prior EEG seizure modeling work and highlights the difficulty of patient-independent seizure detection, with a mean LOSO balanced accuracy of 0.558 across 19 subjects. Across datasets and evaluation settings, our multimodal foundation model enabled robust seizure detection and straightforward adaptation to new seizure detection scenarios, while also supporting interpretable seizure localization.

Electroencephalography Foundation Models Interictal Epileptiform Discharges Representation Learning Multimodal Learning Vision Encoders

Open paper Ask about it PDF

Abstract

Explore similar work

Multimodal Pretraining for Generalizable EEG Representation Learning

DiffEEG: A Self-Supervised Denoising Diffusion Model for Learning EEG Generic Representations

EEG-FuseFormer: A Transformer-Driven Feature Fusion Framework for Seizure Onset Prediction