cs.CVMar 13, 2026
SaveCognitionCapturerPro: Towards High-Fidelity Visual Decoding from EEG/MEG via Multi-modal Information and Asymmetric Alignment
Organizations: Visual Information Processing Laboratory, School of Electronic Engineering, Xidian University, Xi’an 710071, China. · School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China
Abstract
Visual stimuli reconstruction from EEG remains challenging due to fidelity loss and representation shift. We propose CognitionCapturerPro, an enhanced framework that integrates EEG with multi-modal priors (images, text, depth, and edges) via collaborative training. Our core contributions include an uncertainty-weighted similarity scoring mechanism to quantify modality-specific fidelity and a fusion encoder for integrating shared representations. By employing a simplified alignment module and a pre-trained diffusion model, our method significantly outperforms the original CognitionCapturer on the THINGS-EEG dataset, improving Top-1 and Top-5 retrieval accuracy by 25.9% and 10.6%, respectively. Code is available at: https://github.com/XiaoZhangYES/CognitionCapturerPro.
Figures & tables
Fig. 1: Core challenges in EEG-based visual decoding. We categorize the misalignment between brain signals and visual stimuli into two aspects: (Left) Representational Shift , where the brain’s associative mechanisms introduce non-visual semantics that deviate from the pixel-level content; and (Right) Fidelity Loss , where selective attention and perceptual uncertainty result in incomplete or localized neural capture of the original image.
Fig. 2: Overview of CognitionCapturerPro: Figure (a) depicts the encoder training process, where multimodal data such as EEG and images are processed by independent encoders, integrated through a fusion encoder, and constrained by an improved contrastive loss. (b) depicts the SCM-Loss, which filters positive pairs using semantic labels and similarity to address one-to-many mappings. (c) depicts the STH-Align structure, which maps embeddings from various modalities into a unified image space via a shared backbone and multimodal projection heads. (d) depicts how the aligned multimodal embeddings are conditionally injected through the SDXL-Turbo model with a multi-branch IP-Adapter to reconstruct semantically consistent and high-fidelity images.
| Method | Subject 1 | Subject 2 | Subject 3 | Subject 4 | Subject 5 | Subject 6 | Subject 7 | Subject 8 | Subject 9 | Subject 10 | Avg | |||||||||||
| top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | |
| BraVL [ 4 ] | 6.1 | 17.9 | 4.9 | 14.9 | 5.6 | 17.4 | 5.0 | 15.1 | 4.0 | 13.4 | 6.0 | 18.2 | 6.5 | 20.4 | 8.8 | 23.7 | 4.3 | 14.0 | 7.0 | 19.7 | 5.8 | 17.5 |
| NICE [ 16 ] | 13.2 | 39.5 | 12.5 | 40.3 | 14.5 | 42.7 | 15.3 | 39.6 | 10.1 | 31.5 | 16.5 | 44.0 | 17.0 | 42.1 | 12.9 | 56.1 | 15.4 | 41.6 | 17.4 | 45.8 | 14.1 | 43.6 |
| NICE-S [ 16 ] | 13.3 | 40.2 | 13.1 | 36.1 | 15.3 | 42.6 | 20.9 | 52.0 | 9.8 | 34.4 | 14.2 | 42.4 | 17.9 | 43.6 | 28.2 | 50.2 | 14.4 | 38.7 | 16.0 | 42.8 | 16.7 | 41.7 |
| NICE-G [ 16 ] | 15.2 | 40.1 | 13.9 | 40.1 | 15.7 | 42.7 | 17.6 | 48.9 | 9.0 | 29.7 | 16.3 | 44.4 | 14.9 | 43.1 | 20.3 | 52.1 | 14.1 | 39.7 | 19.6 | 46.7 | 15.6 | 42.8 |
| MB2C [ 21 ] | 23.6 | 56.3 | 22.6 | 50.5 | 26.3 | 60.1 | 34.8 | 67.0 | 21.3 | 53.0 | 31.0 | 62.3 | 35.0 | 64.8 | 39.0 | 69.3 | 27.5 | 59.3 | 33.1 | 70.8 | 28.4 | 60.3 |
TABLE I: Zero-shot retrieval performance on the Things-EEG dataset. The table reports the Top-1 and Top-5 accuracies (%) for ten subjects and their average. CogCapPro (I), (T), (D), (E), and (F) denote the results using Image, Text, Depth, Edge, and multi-modal Fusion, respectively. Bold values indicate the best performance.
| Method | Subject 1 | Subject 2 | Subject 3 | Subject 4 | Avg | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | |
| NICE [ 16 ] | 9.6 | 27.8 | 18.5 | 47.8 | 14.2 | 41.6 | 9.0 | 26.6 | 12.8 | 36.0 |
| NICE-S [ 16 ] | 9.8 | 27.8 | 18.6 | 46.4 | 10.5 | 38.4 | 11.7 | 27.2 | 12.7 | 35.0 |
| NICE-G [ 16 ] | 8.7 | 30.5 | 21.8 | 56.6 | 16.5 | 49.7 | 10.3 | 32.3 | 14.3 | 42.3 |
| UBP [ 20 ] | 15.0 | 38.0 | 46.0 | 80.5 | 27.3 | 59.0 | 18.5 | 43.5 | 26.7 | 55.2 |
| ATS [ 19 ] | 15.5 | 42.0 | 61.5 | 88.0 | 33.0 | 69.5 | 19.0 | 43.5 | 32.3 | 62.4 |
TABLE II: Zero-shot retrieval performance on the Things-MEG dataset.
Fig. 3: Qualitative comparison of image reconstruction results on the Things-EEG dataset. The first row displays the ground-truth visual stimuli. Subsequent rows show reconstructions generated by CogCapPro, CogCap, and ATM, respectively.
| Method | Low-level | High-level | |||
|---|---|---|---|---|---|
| PixCorr | SSIM | Inception | CLIP | SwAV | |
| MindEye (fMRI) [ 37 ] | 0.322 | 0.431 | 0.954 | 0.930 | 0.344 |
| META (MEG) [ 2 ] | 0.090 | 0.341 | 0.703 | 0.811 | 0.567 |
| ATM (MEG) [ 15 ] | 0.104 | 0.340 | 0.619 | 0.603 | 0.651 |
| CogCap (EEG) [ 18 ] | 0.150 | 0.347 | 0.669 | 0.715 | 0.590 |
| ATM (EEG) [ 15 ] | 0.160 | 0.345 | 0.734 | 0.786 | 0.582 |
TABLE III: Image reconstruction performance. Metrics are divided into low-level and high-level features, where arrows indicate the direction of better performance.
| Baseline | UM | Loss | Mask | Avg (Fusion) | |
|---|---|---|---|---|---|
| top-1 (std) | top-5 (std) | ||||
| ✓ | ✗ | ✗ | ✗ | 51.8 (5.7) | 84.8 (4.8) |
| ✓ | ✓ | ✗ | ✗ | 54.7 (6.4) | 87.1 (3.9) |
| ✓ | ✓ | ✓ | ✗ | 60.7 (6.2) | 90.4 (3.2) |
| ✓ | ✓ | ✓ | ✓ | 61.2 (7.2) | 90.8 (4.4) |
TABLE IV: Ablation study of different modules on the baseline model.
| EEG Encoder | Image Encoder | Avg (Fusion) | |
|---|---|---|---|
| top-1 (std) | top-5 (std) | ||
| CogCap Encoder | Vit-H-14 | 56.0 (6.7) | 87.6 (5.3) |
| CogCap Encoder | RN50 | 61.2 (7.2) | 90.8 (4.4) |
| ShallowNet | RN50 | 37.0 (7.3) | 72.6 (8.2) |
| DeepNet | RN50 | 24.4 (4.6) | 56.8 (7.7) |
| TSConv | RN50 | 44.7 (7.5) | 79.9 (6.5) |
TABLE V: Performance comparison across different EEG and image encoders.
Fig. 4: Qualitative comparison of reconstruction results across different modality configurations and alignment strategies. From top to bottom: the original visual stimulus, results from all modalities, single modalities, and variants without our alignment module. Our full-modality approach effectively integrates complementary features to produce the most consistent reconstructions.
| Method | Low-level | High-level | |||
|---|---|---|---|---|---|
| PixCorr | SSIM | Inception | CLIP | SwAV | |
| Baseline Align | 0.097 | 0.192 | 0.481 | 0.505 | 0.760 |
| w/o Align Module | 0.103 | 0.215 | 0.516 | 0.483 | 0.751 |
| CogCapPro (Image) | 0.141 | 0.317 | 0.773 | 0.833 | 0.558 |
| CogCapPro (Depth) | 0.122 | 0.407 | 0.636 | 0.643 | 0.663 |
| CogCapPro (Edge) | 0.029 | 0.130 | 0.645 | 0.658 | 0.735 |
TABLE VI: Ablation study of modality combinations and alignment modules.
Fig. 5: Top-1 accuracy distribution across spatial and spectral dimensions. The top row illustrates the classification performance across different brain regions (Frontal, Temporal, Central, Parietal, and Occipital), while the bottom row shows the performance across five frequency bands (Delta, Theta, Alpha, Beta, and Gamma). Left and right columns represent results for the Things-EEG and Things-MEG datasets, respectively. “All Channels” denotes the integration of all spatial or spectral information.
Fig. 6: Semantic Analysis and Retrieval Visualization. (a) Similarity heatmaps of the fusion modality embeddings, sorted by semantic categories (left) and structural complexity from low to high (right). (b) Top-5 retrieval results for representative stimulus images.
Fig. 7: Visualization of model interpretability. (Top) Topographical maps of raw data and internal model layers (Attention/Convolution). (Middle) Grad-CAM heatmaps comparing CLIP and EEG-based attention across Image, Depth, and Edge modalities. (Bottom) Samples of reconstructed images.
Fig. 8: Visualization of reconstruction limitations. Rows from top to bottom display the original stimulus images and corresponding reconstructions generated using all modalities, image modality, depth modality, and edge modality, respectively.