cs.AINov 3, 2026

Learning Behaviorally Relevant Dynamics in Multimodal Neural Recordings

Authors: Parastoo Azizeddin, Eray Erturk, Maryam M. Shanechi

Organizations: Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California · Thomas Lord Department of Computer Science, University of Southern California · Alfred E. Mann Department of Biomedical Engineering, University of Southern California · Neuroscience Graduate Program, University of Southern California

Abstract

Multimodal neural data can enable a more complete understanding of brain dynamics underlying behavior. Modalities such as spiking activity, local field potentials (LFPs), and behavior capture diverse spatiotemporal aspects of brain processes, yet prior neural-behavioral models that separate sources of variability rely on a single neural modality. By leveraging these complementary strengths, multimodal neural fusion can provide a unified, rich representation of brain-behavior processes and address the limitations of single-modality analyses. Here we develop BREM-NET, a nonlinear multimodal dynamical model that integrates behavior with multiple neural modalities having distinct statistical characteristics and temporal resolutions. BREM-NET performs multimodal neural fusion during inference while separating behaviorally relevant and residual neural dynamics via a structured learning framework. In a public multimodal non-human primate dataset with simultaneous spiking and local field potential recordings during a 2D reaching task, BREM-NET separates these two types of dynamics and learns them more accurately, as reflected in better behavior decoding, better neural prediction, and stronger latent alignment with behavior, and it retains these capabilities when neural modalities are asynchronous or suffer dropouts, which are major challenges in real-world recordings. This framework provides a new tool for studying behaviorally relevant neural computations across spatiotemporal scales of brain activity.

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. M3D-BFS: a Multi-stage Dynamic Fusion Strategy for Sample-Adaptive Multi-Modal Brain Network Analysis

    Apr 2, 2026Rui Dong, Xiaotong Zhang, Jiaxing Li +3Multimodal FusionBrain Network Analysis

  2. Scaling Neural Motor Decoding via Decoupled Behavioral Pretraining

    Nov 5, 2026Divyansha Lachi, Vinam Arora, Shivashriganesh P. Mahato +2Neural Decoding