Neural Decoding

Objective: Sparse Bayesian learning provides an effective framework to solve high-dimensional problems in brain signal decoding. However, conventional likelihoods regarding data distributions, such as Gaussian or Bernoulli, are potentially inadequate for handling the noisy recordings of brain activity. Hence, this work aims to formulate a robust sparse Bayesian learning framework to address noisy high-dimensional brain activity decoding. Methods: Motivated by the commendable robustness of the minimum error entropy learning criterion for addressing non-Gaussian signals, this study reformulated the sparse Bayesian learning framework under a generalized Bayesian paradigm, in which the model parameter is regulated with the minimum error entropy loss rather than a conventional likelihood function. Results: Our developed SBL-MEE algorithm was evaluated with two real-world brain decoding tasks of regression and classification scenarios, respectively. Experimental results demonstrated that our approach not only realizes superior brain decoding performance than existing methods, but also presents more physiologically interpretable decoder patterns. Conclusion: Although minimum error entropy is not constructed from an arbitrary probabilistic distribution, it is effective to establish noise-robust inference in sparse Bayesian learning method. Significance: This work provides a powerful tool to improve brain activity decoding capability, particularly regarding the noisy high-dimensional setting, thus promoting biomedical engineering applications such as brain-computer interface.

Brain encoding models predict time-resolved neural activity from computational representations of ongoing experience, providing a principled framework for testing how information is represented and transformed across cortical systems. Naturalistic audiovisual narratives are a particularly rich but challenging testbed for these models, requiring integration of multimodal inputs over long temporal horizons and generalization across individuals with substantial response variability. We introduce a multimodal sequence-to-sequence Transformer with a hybrid cross-subject parameterization that predicts cortex-wide parcel-wise fMRI time series autoregressively from visual, audio, language, and vision--language representations. We evaluate the approach on data from the Courtois NeuroMod project, where four deeply-sampled participants viewed six seasons of Friends and four feature-length films during fMRI. Sequence-to-sequence temporal modeling yields consistent improvements over single-frame prediction across cortical networks, with gains extending to novel stimuli. A hybrid architecture that pairs a shared stimulus encoder with lightweight subject-specific decoder components outperforms both fully shared and fully individual models, indicating complementary advantages of learning shared stimulus representations across subjects and fitting individual neural readouts. Finally, we show that in data-scarce settings, hybrid models can be personalized to new individuals with limited fMRI data, demonstrating that multi-subject pretraining serves as a strong inductive prior for building individual-specific encoding models. Together, these results indicate that combining multimodal sequence modeling with a hybrid cross-subject architecture offers a scalable framework for personalized brain encoding under naturalistic conditions.

Decoding language from the human brain remains a grand challenge for Brain-Computer Interfaces (BCIs). Current approaches typically rely on unimodal brain representations, neglecting the brain's inherently multimodal processing. Inspired by the brain's associative mechanisms, where viewing an image can evoke related sounds and linguistic representations, we propose a unified framework that leverages Multimodal Large Language Models (MLLMs) to align brain signals with a shared semantic space encompassing text, images, and audio. A router module dynamically selects and fuses modality-specific brain features according to the characteristics of each stimulus. Experiments on various fMRI datasets with textual, visual, and auditory stimuli demonstrate state-of-the-art performance, achieving an 8.48% improvement on the most commonly used benchmark. We further extend our framework to EEG and MEG data, demonstrating flexibility and robustness across varying temporal and spatial resolutions. To our knowledge, this is the first unified BCI architecture capable of robustly decoding multimodal brain activity across diverse brain signals and stimulus types, offering a flexible solution for real-world applications.

Patterns of microcircuitry in the cerebral cortex suggest that the brain has an array of repeated canonical computational units. However, neural representations are distributed, so the relevant computations may only be related indirectly to single-neuron transformations. It thus remains an open challenge how to define canonical distributed computations. Here we integrate normative and algorithmic theories of neural computation to present a mathematical framework for inferring canonical distributed computations from large-scale neural activity patterns. At the normative level, we hypothesize that the brain learns a structured internal model of its environment and its latent causes. At the algorithmic level, we propose that this inference process is a nonlinear message-passing algorithm on a graph-structured model of the world. Given a time series of neural activity during a perceptual inference task, our analysis framework finds (i) the neural representation of the relevant latent variables, (ii) interactions between these latent variables that define the brain's internal model of the world, and (iii) message functions that specify the inference algorithm. With enough data, key computational properties are identifiable due to constraints from the symmetries inherent in any canonical computation, up to a joint global transformation of all interactions and the message-passing functions. As a demonstration of this framework, we analyze artificial neural recordings generated by a simulated brain that implicitly implements an approximate inference algorithm. Given simulated inputs and neural activity we successfully recover the latent variables, their neural representation and dynamics, and canonical message functions that govern the dynamics. Analysis of these models reveals features of experimental design required to extract canonical computations from neural data.