BAT-CLIP: Trimodal Alignment of Brain, Audio and Text

Suhyun Kim; Jinmo Han; Danny Dongyeop Han; Ahhyun Lucy Lee; Jewoon Lee; Yonghyeon Gwon; Zach Paris; Chun Kee Chung

cs.SDSep 25, 2026

BAT-CLIP: Trimodal Alignment of Brain, Audio and Text

Authors: Suhyun Kim, Jinmo Han, Danny Dongyeop Han, Ahhyun Lucy Lee, Jewoon Lee, Yonghyeon Gwon, Zach Paris, Chun Kee Chung,

Organizations: Yonsei University, Seoul, Republic of Korea · Seoul National University, Seoul, Republic of Korea · Dartmouth College, Hanover, NH, USA

Abstract

Decoding and interpreting naturalistic speech from the brain increasingly relies on alignment to pretrained speech and language representation spaces. However, current CLIP-style brain-speech alignment ground neural activity to a single anchor modality-audio or text-despite the brain's inherently multimodal speech processing. This induces a trade-off: audio anchoring preserves temporal structure but weakens linguistic separability, while text anchoring captures semantics yet discards acoustic detail. We propose BAT-CLIP, the first CLIP-style trimodal alignment framework for iEEG that jointly aligns neural embeddings to both pretrained audio and text anchors in a shared, frozen audio-text manifold. On the naturalistic Podcast benchmark, BAT-CLIP yields more robust representations than bimodal CLIP baselines. We also highlight the importance of using self-supervised foundation models for CLIP training.

Figures & tables

Fig. 1 : Overview of BAT-CLIP . A trainable brain encoder $f_{\theta}$ , initialized from brain foundation model DIVER-1, maps neural windows to brain embeddings $z^{B}$ . Frozen AudioCLIP encoders provide audio anchors $z^{A}$ and text anchors $z^{T}$ . BAT-CLIP optimizes a trimodal contrastive objective $\mathcal{L}_{Total}$ to align neural embeddings with the joint audio–text representation space.

Method	Type	Content		Onset		POS		GPT Surprisal		Word Emb		Whisper
		AUC	$\Delta$	AUC	$\Delta$	AUC	$\Delta$	AUC	$\Delta$	AvgAUC	$\Delta$	Acc	$\Delta$
CNN(Baseline) †	Sup	0.5498	–	0.7519	–	0.5653	–	0.5019	–	0.6260	–	0.6634	–
BrainBERT	SSL	0.5505	+0.13%	0.6341	-15.67%	0.5325	-5.80%	0.5157	+2.75%	0.5533	-11.61%	0.5805	-12.49%
PopT	SSL	0.4950	-9.97%	0.4654	-38.10%	0.5031	-11.00%	0.5022	+0.06%	0.5002	-20.10%	0.5014	-24.42%
DIVER-1	SSL	0.5645	+2.67%	0.7729	+2.79%	0.5656	+0.05%	0.5155	+2.71%	0.6450	+3.04%	0.7022	+5.85%
BA-CLIP	Align	0.5840	+6.31%	0.6560	-12.61%	0.5673	+0.32%	0.5169	+3.02%	0.6468	+4.26%	0.7141	+7.75%

Table 1: Overall decoding performance (averaged over 9 subjects). We report AUC for Content , Onset , POS , GPT surprisal , and Word Emb decoding, and accuracy for Whisper .

\Delta

denotes relative gain over the supervised CNN baseline (baseline

\Delta

: “–”).

\dagger

indicates full fine-tuning (CNN); all other methods are evaluated via linear probing. BA-CLIP , BT-CLIP , and BAT-CLIP apply CLIP-style contrastive alignment on top of the pretrained DIVER-1 brain encoder, using brain–audio, brain–text, and trimodal brain–audio–text alignment, respectively. Overall, DIVER-1 is the strongest foundation-model baseline, CLIP alignment generally improves over DIVER-1, and BAT-CLIP achieves the best result among the aligned variants on five of six probe tasks. Bold and underline denote the best and second-best results within each column.

Method	Init.	Content		Onset		POS		GPT Surprisal		Word Emb		Whisper
		AUC	$\Delta$	AUC	$\Delta$	AUC	$\Delta$	AUC	$\Delta$	AvgAUC	$\Delta$	Acc	$\Delta$
BA-CLIP	Scratch	0.5671	+3.14%	0.7792	+3.75%	0.5624	-0.35%	0.5143	+2.55%	0.6059	-2.33%	0.6609	+0.05%
	DIVER-1	0.5840	+6.31%	0.6560	-12.61%	0.5673	+0.32%	0.5169	+3.02%	0.6468	+4.26%	0.7141	+7.75%
BT-CLIP	Scratch	0.5650	+2.85%	0.7822	+4.19%	0.5635	-0.18%	0.5120	+2.10%	0.6296	+1.52%	0.6607	-0.02%
	DIVER-1	0.5813	+5.82%	0.6562	-12.56%	0.5709	+0.97%	0.5203	+3.69%	0.6510	+4.94%	0.7157	+7.98%
BAT-CLIP	Scratch	0.5586	+1.70%	0.7600	+0.94%	0.5480	-2.94%	0.5119	+2.09%	0.6123	-1.32%	0.6463	-2.36%

Table 2: Effect of pretraining across methods. We compare CLIP-aligned models trained from scratch (Scratch) versus the same models initialized with pretrained DIVER-1 weights, across unimodal alignments ( BA-CLIP : brain–audio; BT-CLIP : brain–text) and trimodal alignment ( BAT-CLIP : brain–audio–text). As in Table 1 ,

\Delta

denotes relative gain over the supervised CNN baseline (baseline

\Delta

: “–”). Pretrained DIVER-1 initialization is key to realizing the gains from trimodal CLIP alignment; training from scratch largely fails to capitalize on brain–audio–text supervision.

Fig. 2 : Visualization of learned word-level representations. 2D t-SNE projection of brain embeddings extracted by various models for Sub03. Each point represents a single word-level embedding. The color of the point denotes the sentence to which the word belongs, and the shape indicates the speaker. Compared to the baselines, the proposed trimodal alignment model, BAT-CLIP , exhibits tighter clustering of words from the same sentence (shared colors), consistent with the sentence-level clustering results in Table 3 .

Method	Cosine Coherence ( $\uparrow$ )	Separation Ratio ( $\uparrow$ )
DIVER-1 (SSL)	0.0004	0.95
BA-CLIP	0.0789	1.09
BT-CLIP	0.0954	1.04
BAT-CLIP (Ours)	0.1572	1.23

Table 3: Quantitative clustering evaluation of word-level embeddings. We evaluate the structure of the learned embedding space for a representative subject (Sub03; Fig. 2 ) using sentence-level clustering metrics, where higher values indicate tighter grouping of word embeddings within the same sentence. The proposed trimodal alignment ( BAT-CLIP ) yields the strongest clustering compared to the unaligned foundation model (DIVER-1) and unimodal alternatives.

Cross-Modal Alignment

Cross-modal alignment maps representations from different data types into compatible spaces so models can compare, transfer, and combine information.

See all 433 papers

CAMEL-CLIP: Channel-aware Multimodal Electroencephalography-text Alignment for Generalizable Brain Foundation Models
Feb 27, 2026Hanseul Choi, Jinyeong Park, Seongwon Jin +2EEG Foundation Models Multimodal Pretraining
Department of Computer Science and Engineering, Incheon National University, Yeonsu-gu, 22012, Incheon, Republic of Korea · Department of Artificial Intelligence, Inha University, Michuhol-gu, 22212, Incheon, Republic of Korea · Center for Brain-Machine Interface, Incheon National University, Yeonsu-gu, 22012, Incheon, Republic of Korea
Decoding Naturalistic Speech from fMRI with Temporal Modeling and Multimodal Alignment
Oct 15, 2026Corentin Caris, Louis Jalouzot, Nona Rajabi +9Neural Decoding
Karavela · UNICOG, CNRS, INSERM, CEA, Paris-Saclay University · LSCP, EHESS, ENS, CNRS, PSL University +1
Semantic Alignment of EEG and Text: A Contrastive Learning Framework for Decoding Imagined Speech
Oct 14, 2026Shahryar G. Namdari, Francesco Iacomi, Riccardo Barbieri +1EEG Decoding EEG-to-Text Decoding
Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milano, 20133, Italy

Feb 27, 2026J/K move · Enter open · S save

CAMEL-CLIP: Channel-aware Multimodal Electroencephalography-text Alignment for Generalizable Brain Foundation Models

Hanseul Choi, Jinyeong Park, Seongwon Jin, Sungho Park+1

Department of Computer Science and Engineering, Incheon National University, Yeonsu-gu, 22012, Incheon, Republic of Korea · Department of Artificial Intelligence, Inha University, Michuhol-gu, 22212, Incheon, Republic of Korea · Center for Brain-Machine Interface, Incheon National University, Yeonsu-gu, 22012, Incheon, Republic of Korea

EEG Foundation Models Multimodal Pretraining Cross-Modal Alignment Multimodal Contrastive Learning Representation Alignment

Open paper AskPDF arXiv

Electroencephalography (EEG) foundation models have shown promise for learning generalizable representations, yet they remain sensitive to channel heterogeneity, such as changes in channel composition or ordering. We propose channel-aware multimodal EEG-text alignment contrastive language-image pretraining (CAMEL-CLIP), a contrastive EEG-text multimodal foundation model designed to be robust to heterogeneous channel configurations and widely applicable to diverse downstream tasks. CAMEL-CLIP introduces three key components: (1) channel attribute-based positional encoding, which identifies channels through semantic information; (2) dynamic channel projection, which generates variable-length embeddings by independently projecting each channel without feature compression; and (3) dual-level contrastive learning, which jointly performs channel-level and sample-level contrastive learning to capture both channel-specific and global signal characteristics. Experimental results demonstrate that CAMEL-CLIP achieves state-of-the-art performance under linear-probing and outperforms existing foundation models that rely on full-finetuning.

Abstract

Cross-Modal Alignment

Explore similar work

CAMEL-CLIP: Channel-aware Multimodal Electroencephalography-text Alignment for Generalizable Brain Foundation Models

Decoding Naturalistic Speech from fMRI with Temporal Modeling and Multimodal Alignment

Semantic Alignment of EEG and Text: A Contrastive Learning Framework for Decoding Imagined Speech