HyFI: Hyperbolic Feature Interpolation for Brain-Vision Alignment

Sangmin Jo; Wootaek Jeong; Da-Woon Heo; Yoohwan Hwang; Heung-Il Suk

cs.AIMar 24, 2026

HyFI: Hyperbolic Feature Interpolation for Brain-Vision Alignment

Authors: Sangmin Jo, Wootaek Jeong, Da-Woon Heo, Yoohwan Hwang, Heung-Il Suk

Organizations: Department of Artificial Intelligence, Korea University, Seoul, South Korea

Abstract

Recent progress in artificial intelligence has encouraged numerous attempts to understand and decode human visual system from brain signals. These prior works typically align neural activity independently with semantic and perceptual features extracted from images using pre-trained vision models. However, they fail to account for two key challenges: (1) the modality gap arising from the natural difference in the information level of representation between brain signals and images, and (2) the fact that semantic and perceptual features are highly entangled within neural activity. To address these issues, we utilize hyperbolic space, which is well-suited for considering differences in the amount of information and has the geometric property that geodesics between two points naturally bend toward the origin, where the representational capacity is lower. Leveraging these properties, we propose a novel framework, Hyperbolic Feature Interpolation (HyFI), which interpolates between semantic and perceptual visual features along hyperbolic geodesics. This enables both the fusion and compression of perceptual and semantic information, effectively reflecting the limited expressiveness of brain signals and the entangled nature of these features. As a result, it facilitates better alignment between brain and visual features. We demonstrate that HyFI achieves state-of-the-art performance in zero-shot brain-to-image retrieval, outperforming prior methods with Top-1 accuracy improvements of up to +17.3% on THINGS-EEG and +9.1% on THINGS-MEG.

Figures & tables

Figure 1: (a) The human visual system processes perceptual and semantic information, and some degradation occurs when neural activity is recorded. (b) Previous works aligned semantic and perceptual features through separate pathways, overlooking their entanglement in brain signals. (c) In contrast, hyperbolic interpolation merges perceptual and semantic features with lower complexity, enhancing alignment with brain signals.

Figure 2: (a) The semantic image $\mathbf{x}_{v}^{s}$ and perceptual image $\mathbf{x}_{v}^{p}$ are encoded by CLIP and projected via a linear layer, and then lifted onto the hyperboloid via the exponential map. Using a learned weight $t$ derived from the semantic image features, the two image features are interpolated on the hyperbolic manifold. Similarly, EEG inputs are encoded and projected onto the same hyperbolic space. Contrastive learning is then performed on the hyperboloid to bring paired EEG-image representations closer. (b) A schematic view of the hyperbolic embedding space. The interpolated representation $\hat{\mathbf{z}}_{v}$ lies along the geodesic between the semantic feature $\mathbf{z}_{v}^{s}$ and the perceptual feature $\mathbf{z}_{v}^{p}$ . Contrastive learning then pulls the EEG feature $\mathbf{z}_{b}$ toward the target $\hat{\mathbf{z}}_{v}$ .

Figure 3: Examples of image augmentations and retrieval results. The semantic image $\mathbf{x}_{v}^{s}$ and perceptual image $\mathbf{x}_{v}^{p}$ are generated via fovea blur and Gaussian blur, respectively. Retrieval results using CLIP embedding show that semantic queries return category-relevant matches (e.g., fruits), while perceptual queries retrieve images with similar low-level attributes such as color and shape.

Intra-subject: train and test on one subject
Method	Subject 1		Subject 2		Subject 3		Subject 4		Subject 5		Subject 6		Subject 7		Subject 8		Subject 9		Subject 10		Average
	T-1	T-5	T-1	T-5	T-1	T-5	T-1	T-5	T-1	T-5	T-1	T-5	T-1	T-5	T-1	T-5	T-1	T-5	T-1	T-5	T-1	T-5
BraVL	6.1	17.9	4.9	14.9	5.6	17.4	5.0	15.1	4.0	13.4	6.0	18.2	6.5	20.4	8.8	23.7	4.3	14.0	7.0	19.7	5.8	17.5
NICE	13.2	39.5	13.5	40.3	14.5	42.7	20.6	52.7	10.1	31.5	16.5	44.0	17.0	42.1	22.9	56.1	15.4	41.6	17.4	45.8	16.1	43.6
ATM-S	25.6	60.4	22.0	54.5	25.0	62.4	31.4	60.9	12.9	43.0	21.3	51.1	30.5	61.5	38.8	72.0	30.4	51.5	29.1	63.5	28.5	60.4
Cog-cap	31.4	79.7	31.4	77.8	38.2	85.7	40.4	85.8	24.4	66.3	34.8	78.8	34.7	81.0	48.1	88.6	37.4	79.4	35.6	79.3	35.6	80.2

Table 1: Top-1 (T-1) and top-5 (T-5) accuracy (

\%

) results in 200-way zero-shot brain-to-image retrieval on THINGS-EEG, reported for intra- and inter-subject settings. Bold and underline indicate the best and second-best results, respectively.

Intra-subject: train and test on one subject
Method	Subject 1		Subject 2		Subject 3		Subject 4		Average
	T-1	T-5	T-1	T-5	T-1	T-5	T-1	T-5	T-1	T-5
NICE	8.7	30.5	21.8	56.6	16.5	49.7	10.3	32.3	14.3	42.3
UBP	15.0	38.0	46.0	80.5	27.3	59.0	18.5	43.5	26.7	55.2
HyFI	17.6	40.1	63.8	91.1	38.0	76.9	23.7	50.2	35.8	64.6
Inter-subject: leave one subject out for test

Table 2: Top-1 and top-5 accuracy (

\%

) results in 200-way zero-shot brain-to-image retrieval on THINGS-MEG.

Interpolation	Hyperbolic	THINGS-EEG		THINGS-MEG
Interpolation	Hyperbolic	T-1	T-5	T-1	T-5
–	–	49.4	81.0	23.1	47.8
–	✓	54.3	82.5	28.8	55.1
✓	–	59.7	86.8	25.2	51.8
✓	✓	68.2	91.9	35.8	64.6

Table 3: Ablation study: effect of interpolation and hyperbolic space. The row with only interpolation checked refer to interpolation in CLIP space.

Method	Architecture	Model	THINGS-EEG		THINGS-MEG
Method	Architecture	Model	w/o HyFI	Ours	w/o HyFI	Ours
CLIP	CNN	RN50	49.4	68.2	23.1	35.8
	CNN	RN101	44.4	62.1	21.0	34.8
	ViT	ViT-B/16	36.0	41.6	18.6	23.3
		ViT-B/32	42.5	46.9	20.0	25.1
		ViT-L/14	30.1	34.1	13.0	21.0

Table 4: Top-1 image retrieval accuracy (

\%

) across visual encoders. The “w/o HyFI” represents alignment in CLIP space; MERU and HyCoCLIP use hyperbolic alignment.

Figure 4: Qualitative comparison of image retrieval results. Our method retrieves semantically and perceptually coherent images, while the previous method often suffers from color or semantic inconsistencies.

Brain Encoder	THINGS-EEG		THINGS-MEG
Brain Encoder	w/o HyFI	Ours	w/o HyFI	Ours
ShallowNet	36.6	50.5	15.1	23.2
EEGNet	36.2	51.6	19.5	26.4
TSConv	41.1	57.1	21.9	30.5
EEGProject	49.4	68.2	23.1	35.8

Table 5: Top-1 image retrieval accuracy (%) using different brain encoders and CLIP-RN50.

Figure 5: Distributions of embedding distances from the root in (a) CLIP space and (b) hyperbolic space. Interpolated image embeddings lie closer to the root in hyperbolic space, unlike in CLIP space.

Figure 6: Distribution of the interpolation coefficient $t$ and example images with low and high $t$ values.

Figure 7: Comparison of Top-3 retrieval results using (a) semantic and (b) perceptual images. Semantic queries tend to retrieve conceptually similar images (e.g., plants, fruits, animals). In contrast, perceptual queries retrieve images with shared low-level visual feature such as color and orientation.

Figure 8: Top-1 accuracy heatmap from a parameter search over perceptual and semantic visual augmentation configurations.

Model	Params	Emb dim	Semantic		Perceptual
			Norm	Std	Norm	Std
RN50	38.32 M	1024	233	0.05	277	0.06
RN101	56.26 M	512	282	0.09	325	0.11
ViT/B-16	86.19 M	512	1409	0.48	1404	0.48
ViT/B-32	87.85 M	512	1447	0.49	1473	0.51
ViT/L-14	303.97 M	768	2399	0.67	2535	0.71

Table 6: Comparison of vision encoders and feature statistics.

Semantic	Perceptual	Top-1 ACC	Top-5 ACC
Original	Fovea blur	53.8	83.7
	SAM	56.6	85.4
	Gaussian noise	61.4	89.8
	Low resolution	63.5	89.6
	Gaussian blur	65.3	91.4
Fovea blur	SAM	60.5	88.5

Table 7: Image retrieval accuracy (

\%

) with various semantic and perceptual augmentations.

Figure 9: Examples of different augmentations. SAM refer to SAM-based augmentation that segments the image and fills each region with its mean color. SAM refers to an augmentation that uses the SAM to segment the image and fills each region with its mean color.

Input feature	Top-1 ACC	Top-5 ACC
EEG	54.5	84.8
Perceptual	67.1	91.8
Original	67.6	91.8
Semantic	68.2	91.9

Table 8: Retrieval accuracy with different strategies for obtaining the interpolation coefficient

t

Method	Model	ShallowNet		EEGNet		TSConv		EEGProject
Method	Model	Base	Ours	Base	Ours	Base	Ours	Base	Ours
CLIP	RN50	36.6	50.5	36.2	51.6	41.1	57.1	49.4	68.2
	RN101	34.0	45.8	35.6	46.4	35.9	53.4	44.4	62.1
	ViT-B/16	28.3	36.7	30.7	35.1	31.4	40.5	36.0	41.6
	ViT-B/32	30.7	36.8	33.7	37.0	36.6	40.8	42.5	46.9
	ViT-L/14	19.8	22.8	19.1	27.0	20.9	27.4	30.1	34.1

Table 9: Top-1 accuracy (%) for various combinations of brain encoders and vision models on brain-to-image retrieval in the THING-EEG dataset. Base indicates performance using either contrastive alignment (CLIP) or hyperbolic contrastive alignment (MERU and HyCoCLIP), without our proposed interpolation method.

Figure 10: Feature distributions visualized on the Poincaré ball using HoronPCA. Interpolated visual features lie between perceptual and semantic features.

Figure 11: Top-5 image retrieval results on the THINGS-EEG dataset for successful and failure cases.

Figure 12: Top-5 image retrieval results on the THINGS-EEG dataset for successful and failure cases.

Figure 13: Image retrieval results for various VLMs

Cross-Modal Alignment

Cross-modal alignment maps representations from different data types into compatible spaces so models can compare, transfer, and combine information.

See all 440 papers

What Does the Brain See? Multiview Neural Representations to Demystify the Brain-Visual Alignment
Jun 24, 2026Salini Yadav, Taveena Lotey, Pravendra Singh +1Representation Learning EEG-Based Visual Decoding
Department of CSE IIT Roorkee, India · Department of CSE IIT (ISM) Dhanbad, India
Brain-Inspired Capture: Evidence-Driven Neuromimetic Perceptual Simulation for Visual Decoding
Apr 20, 2026Feixue Shao, Guangze Shi, Xueyu Liu +6Representation Learning Brain-Computer Interfaces
School of Mathematics, Taiyuan University of Technology, Shanxi 030024, China · College of Artificial Intelligence, Taiyuan University of Technology, Shanxi 030024, China · National Human Genetics Resource Center, National Research Institute for Family Planning, Beijing 100000, China +1
Aligning What EEG Can See: Structural Representations for Brain-Vision Matching
Mar 7, 2026Jingyi Tang, Shuai Jiang, Fei Su +1EEG-Based Visual Decoding Cross-Modal Alignment
Beijing University of Posts and Telecommunications, Beijing, 100876, China

Jun 24, 2026J/K move · Enter open · S save

What Does the Brain See? Multiview Neural Representations to Demystify the Brain-Visual Alignment

Salini Yadav, Taveena Lotey, Pravendra Singh, Partha Pratim Roy

Department of CSE IIT Roorkee, India · Department of CSE IIT (ISM) Dhanbad, India

Representation Learning EEG-Based Visual Decoding Cross-Modal Alignment Multi-View Learning

Open paper AskPDF arXiv

Zero-shot visual decoding from electroencephalography (EEG) aims to infer visual semantics from non-invasive neural recordings, but remains challenging due to the low signal-to-noise ratio, non-stationarity, and limited spatial resolution of EEG. Existing EEG-vision alignment methods often rely on holistic EEG embeddings, which can obscure the complementary temporal, spectral, and spatial structure underlying visual perception. We introduce a unified multiview EEG representation learning framework for aligning brain responses with visual semantic embeddings. Our method builds an EEG encoder that jointly models three complementary views: input-conditioned state-space temporal dynamics, learnable wavelet-based spectral decomposition for sample-adaptive frequency modeling, and attention-modulated graph learning for structured electrode interactions. The resulting multiview EEG embeddings are fused and aligned with pretrained visual representations in a shared semantic space using contrastive learning with EEG-specific regularization, enabling 200-way zero-shot visual classification. Experiments on THINGS-EEG benchmark show that our method achieves state-of-the-art performance, with 54.8% Top-1 and 85.6% Top-5 accuracy in the within-subject setting and 15.3% Top-1 and 45.4% Top-5 accuracy in the cross-subject setting. We further present the first systematic cross-session EEG-image decoding evaluation, achieving 40.8% Top-1 and 78.0% Top-5 accuracy. These results suggest that explicitly modeling multiview neural structure improves both semantic alignment and generalization in EEG-based visual decoding.

Abstract

Cross-Modal Alignment

Explore similar work

What Does the Brain See? Multiview Neural Representations to Demystify the Brain-Visual Alignment

Brain-Inspired Capture: Evidence-Driven Neuromimetic Perceptual Simulation for Visual Decoding

Aligning What EEG Can See: Structural Representations for Brain-Vision Matching