Visual search is ubiquitous in daily life, from locating keys in a messy apartment to finding a tool in a cluttered workshop or an application icon on a desktop screen. While visual attention is to a large extent driven by image content, research in psychology indicates that user cognitive states have a significant influence on attention allocation. A promising way to access information about cognitive states is via Electroencephalography (EEG) recordings. Despite this potential, previous works in computational attention modeling have largely disregarded EEG, and related studies in psychology operate with abstract, simplistic stimuli, or even artificially guided search procedures. We bridge this gap with two contributions. First, we present EEGaze 140, the largest publicly available multimodal dataset of free visual search with realistic stimuli. The dataset contains synchronized EEG and eye tracking recordings from 36 participants across 140 naturalistic scenes, covering two scenarios: searching for desktop icons and finding tools in cluttered workshop scenes. Second, we introduce EEGazeNet, the first method for human attention prediction in visual search that integrates EEG signals. Our approach consists of a EEG guided refinement stage that adapts the output of a state of the art saliency prediction method to the user current cognitive state. In comprehensive evaluations on EEGaze 140, we show that EEGazeNet consistently outperforms all baselines and ablation conditions.
Figures & tables
Figure 1. We present EEGazeNet , the first method to integrate EEG signals into human attention prediction during visual search. The core element of EEGazeNet is a refinement approach that adapts outputs of a state-of-the-art saliency model to the user’s cognitive state assessed via EEG prior to the start of the search. teaser figure teaser figure
Figure 2. Examples of the desktop and workshop visual scenes we created for our study. stimuli stimuli
Reference
Available
Users
Size
Search
Stimuli
Brouwer et al. (2017)
✗
21
176
Guided
Abstract
Kastrati et al. (2021)
✓
356
21360
Guided
Abstract
Kamienkowski et al. (2012)
✗
17
209
Free
Abstract
Sharma et al. (2023)
✓
15
1800
Free
Realistic ∗
Kaunitz et al. (2014)
✗
12
2160
Free
Realistic ∗
EEGaze-140
✓
36
5040
Free
Realistic (Workshop and Desktop)
Table 1. Datasets for visual search task. Availability shows whether the dataset is publicly available or not. Size is a product of the total number of users and scenes shown to each user. ∗ denotes that only a single type of realistic visual stimuli is used, whereas our dataset comprises of two scene types: workshop and desktop scenes.
Figure 3. (a), (b) Distribution of number of objects in workshop and desktop scenes, highlighting the controlled complexity of desktops versus the variability and complexity in workshop scenarios. (c), (d) Search duration versus number of objects in workshop and desktop scenes, showing longer search times in workshops compared to desktops search, indicating greater familiarity with desktop setting. plots plots
Figure 4. Example saliency and binary fixation maps for Workshop (a–b) and Desktop (c–d) stimuli. saliency and binary fixation maps saliency and binary fixation maps
Figure 5. Our EEG-based saliency refinement framework generates an initial saliency map with SUM. EEG features are spatially projected via an MLP pipeline, fused with the SUM output through a channel-wise attention mechanism, refined with a VSSM network, and combined with the initial prediction through residual fusion to produce the final map. Outline res
Stimuli
Model
KL ↓
SIM ↑
NSS ↑
CC ↑
AUC ↑
Desktop
SUM (No Finetuning)
2.094 ± 0.100
0.240 ± 0.013
0.601 ± 0.051
0.165 ± 0.010
0.745 ± 0.021
SUM (Finetuning)
1.799 ± 0.110
0.323 ± 0.018
1.128 ± 0.095
0.322 ± 0.020
0.819 ± 0.013
EEGazeNet (SUM Frozen)
1.856 ± 0.142
0.303 ± 0.022
1.050 ± 0.137
0.302 ± 0.042
0.797 ± 0.020
EEGazeNet (Randomized EEG Descriptive Features)
2.031 ± 0.116
0.282 ± 0.020
1.057 ± 0.115
0.266 ± 0.027
0.818 ± 0.015
EEGazeNet (Randomized EEG Bandpower Ratios)
2.007 ± 0.115
0.282 ± 0.020
1.017 ± 0.115
0.262 ± 0.027
0.816 ± 0.015
EEGazeNet (EEG Bandpower Ratios)
1.766 ± 0.115
0.318 ± 0.020
1.203 ± 0.101
0.334 ± 0.025
0.829 ± 0.013
Table 2. Performance comparison on desktop and workshop stimuli for cross-user settings using previous EEG signals. Results are reported as mean ± 95% confidence intervals across users. Arrows indicate whether higher ( ↑ ) or lower ( ↓ ) values are better.
Stimuli
Model
KL ↓
SIM ↑
NSS ↑
CC ↑
AUC ↑
Desktop
SUM (No Finetuning)
2.201 ± 0.064
0.221 ± 0.010
0.636 ± 0.048
0.157 ± 0.011
0.741 ± 0.007
SUM (Finetuning)
2.127 ± 0.065
0.217 ± 0.011
0.921 ± 0.036
0.247 ± 0.012
0.761 ± 0.007
EEGazeNet (SUM Frozen)
1.965 ± 0.062
0.265 ± 0.012
1.009 ± 0.051
0.271 ± 0.014
0.784 ± 0.008
EEGazeNet (Randomized EEG Descriptive Features)
1.990 ± 0.066
0.250 ± 0.013
0.978 ± 0.044
0.270 ± 0.014
0.769 ± 0.007
EEGazeNet (Randomized EEG Bandpower Ratios)
1.993 ± 0.067
0.247 ± 0.013
0.966 ± 0.041
0.266 ± 0.015
0.764 ± 0.007
EEGazeNet (EEG Bandpower Ratios)
1.875 ± 0.062
0.280 ± 0.012
1.197 ± 0.041
0.328 ± 0.013
0.811 ± 0.007
Table 3. Performance comparison on desktop and workshop stimuli for within-user settings using previous EEG signals. Results are reported as mean ± 95% confidence intervals across users. Arrows indicate whether higher ( ↑ ) or lower ( ↓ ) values are better.
Figure 6. Cross-user attention map visualizations for desktop (first row) and workshop (second row) examples. Outline quant res quant
Recent EEG-to-image retrieval methods leverage pretrained vision encoders and foveation-inspired priors, but typically assume a fixed, center-focused view. This center bias conflicts with content-driven human attention, creating a geometric-semantic dissociation between visual features and EEG responses. We propose SIMON, a saliency-aware multi-view framework for zero-shot EEG-to-image retrieval. SIMON combines foreground segmentation and saliency prediction to select fixation centers via Saliency-Aware Sampling (SAS), then generates foveated views that emphasize informative object regions while suppressing background clutter. On THINGS-EEG, SIMON achieves state-of-the-art performance in both intra-subject and inter-subject settings, reaching an average Top-1 accuracy of 69.7% and 19.6%, respectively, consistently outperforming recent competitive baselines. Analyses across sampling granularity, EEG channel topology, and visual/brain encoder backbones further support the robustness of saliency-aware multi-view integration. Our code and models are publicly available at https://github.com/simonlink666/SIMON.
Visual stimuli reconstruction from EEG remains challenging due to fidelity loss and representation shift. We propose CognitionCapturerPro, an enhanced framework that integrates EEG with multi-modal priors (images, text, depth, and edges) via collaborative training. Our core contributions include an uncertainty-weighted similarity scoring mechanism to quantify modality-specific fidelity and a fusion encoder for integrating shared representations. By employing a simplified alignment module and a pre-trained diffusion model, our method significantly outperforms the original CognitionCapturer on the THINGS-EEG dataset, improving Top-1 and Top-5 retrieval accuracy by 25.9% and 10.6%, respectively. Code is available at: https://github.com/XiaoZhangYES/CognitionCapturerPro.
Kaifan Zhang, Lihuo He, Junjie Ke +4
Visual Information Processing Laboratory, School of Electronic Engineering, Xidian University, Xi’an 710071, China. · School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China
EEG-based visual decoding provides a non-invasive pathway for interpreting visual semantics. However, existing methods often overlook the perceptual asymmetry between foreground and background in complex scenes, leading to background interference and semantic misalignment. EEG signals also exhibit rapid temporal dynamics and nonstationary spatial patterns, making it difficult to capture the time-varying brain connectivity associated with focal visual attention. To address these limitations, we propose FSDBN, a unified framework for robust EEG-visual decoding. FSDBN introduces Semantic-Consistent Saliency Alignment to separate semantically relevant foreground regions from background noise under joint saliency and semantic constraints. It further employs Semantic-Prior Dynamic Gating Foreground Fusion to adaptively regulate the contributions of foreground and background features. In parallel, EEG signals are modeled as adaptive spatiotemporal brain networks whose functional connectivity dynamically reorganizes to capture neural responses to salient foregrounds. Experiments on zero-shot brain-to-image retrieval demonstrate that FSDBN achieves 69.0 percent top-1 accuracy and 92.2 percent top-5 accuracy, outperforming previous state-of-the-art methods. Code is available at https://github.com/LiuYiheng1/FSDBN-EEG.
Yiheng Liu, Chuhang Zheng, Peiliang Gong +3
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China · The International Joint Institute of Tianjin University, Fuzhou, Tianjin University, Tianjin, China