EEGaze: Multimodal Attention Prediction in Visual Search via EEG-Based Saliency Refinement

Mansi Sharma; Dinar Orazgaliyev; Antonio Krüger; Philipp Müller

cs.HCOct 26, 2026

EEGaze: Multimodal Attention Prediction in Visual Search via EEG-Based Saliency Refinement

Authors: Mansi Sharma, Dinar Orazgaliyev, Antonio Krüger, Philipp Müller

Organizations: Max-Planck-Institute for Intelligent Systems, Stuttgart, Germany · Saarland University and DFKI, Saarbrücken, Germany · Saarland University, Saarbrücken, Germany

Abstract

Visual search is ubiquitous in daily life, from locating keys in a messy apartment to finding a tool in a cluttered workshop or an application icon on a desktop screen. While visual attention is to a large extent driven by image content, research in psychology indicates that user cognitive states have a significant influence on attention allocation. A promising way to access information about cognitive states is via Electroencephalography (EEG) recordings. Despite this potential, previous works in computational attention modeling have largely disregarded EEG, and related studies in psychology operate with abstract, simplistic stimuli, or even artificially guided search procedures. We bridge this gap with two contributions. First, we present EEGaze 140, the largest publicly available multimodal dataset of free visual search with realistic stimuli. The dataset contains synchronized EEG and eye tracking recordings from 36 participants across 140 naturalistic scenes, covering two scenarios: searching for desktop icons and finding tools in cluttered workshop scenes. Second, we introduce EEGazeNet, the first method for human attention prediction in visual search that integrates EEG signals. Our approach consists of a EEG guided refinement stage that adapts the output of a state of the art saliency prediction method to the user current cognitive state. In comprehensive evaluations on EEGaze 140, we show that EEGazeNet consistently outperforms all baselines and ablation conditions.

Figures & tables

Figure 1. We present EEGazeNet , the first method to integrate EEG signals into human attention prediction during visual search. The core element of EEGazeNet is a refinement approach that adapts outputs of a state-of-the-art saliency model to the user’s cognitive state assessed via EEG prior to the start of the search. teaser figure teaser figure

Figure 2. Examples of the desktop and workshop visual scenes we created for our study. stimuli stimuli

Reference	Available	Users	Size	Search	Stimuli
Brouwer et al. (2017)	✗	21	176	Guided	Abstract
Kastrati et al. (2021)	✓	356	21360	Guided	Abstract
Kamienkowski et al. (2012)	✗	17	209	Free	Abstract
Sharma et al. (2023)	✓	15	1800	Free	Realistic ∗
Kaunitz et al. (2014)	✗	12	2160	Free	Realistic ∗
EEGaze-140	✓	36	5040	Free	Realistic (Workshop and Desktop)

Table 1. Datasets for visual search task. Availability shows whether the dataset is publicly available or not. Size is a product of the total number of users and scenes shown to each user. ∗ denotes that only a single type of realistic visual stimuli is used, whereas our dataset comprises of two scene types: workshop and desktop scenes.

Figure 3. (a), (b) Distribution of number of objects in workshop and desktop scenes, highlighting the controlled complexity of desktops versus the variability and complexity in workshop scenarios. (c), (d) Search duration versus number of objects in workshop and desktop scenes, showing longer search times in workshops compared to desktops search, indicating greater familiarity with desktop setting. plots plots

Figure 4. Example saliency and binary fixation maps for Workshop (a–b) and Desktop (c–d) stimuli. saliency and binary fixation maps saliency and binary fixation maps

Figure 5. Our EEG-based saliency refinement framework generates an initial saliency map with SUM. EEG features are spatially projected via an MLP pipeline, fused with the SUM output through a channel-wise attention mechanism, refined with a VSSM network, and combined with the initial prediction through residual fusion to produce the final map. Outline res

Stimuli	Model	KL $\downarrow$	SIM $\uparrow$	NSS $\uparrow$	CC $\uparrow$	AUC $\uparrow$
Desktop	SUM (No Finetuning)	2.094 $\pm$ 0.100	0.240 $\pm$ 0.013	0.601 $\pm$ 0.051	0.165 $\pm$ 0.010	0.745 $\pm$ 0.021
	SUM (Finetuning)	1.799 $\pm$ 0.110	0.323 $\pm$ 0.018	1.128 $\pm$ 0.095	0.322 $\pm$ 0.020	0.819 $\pm$ 0.013
	EEGazeNet (SUM Frozen)	1.856 $\pm$ 0.142	0.303 $\pm$ 0.022	1.050 $\pm$ 0.137	0.302 $\pm$ 0.042	0.797 $\pm$ 0.020
	EEGazeNet (Randomized EEG Descriptive Features)	2.031 $\pm$ 0.116	0.282 $\pm$ 0.020	1.057 $\pm$ 0.115	0.266 $\pm$ 0.027	0.818 $\pm$ 0.015
	EEGazeNet (Randomized EEG Bandpower Ratios)	2.007 $\pm$ 0.115	0.282 $\pm$ 0.020	1.017 $\pm$ 0.115	0.262 $\pm$ 0.027	0.816 $\pm$ 0.015
	EEGazeNet (EEG Bandpower Ratios)	1.766 $\pm$ 0.115	0.318 $\pm$ 0.020	1.203 $\pm$ 0.101	0.334 $\pm$ 0.025	0.829 $\pm$ 0.013

Table 2. Performance comparison on desktop and workshop stimuli for cross-user settings using previous EEG signals. Results are reported as mean

\pm

95% confidence intervals across users. Arrows indicate whether higher (

\uparrow

) or lower (

\downarrow

) values are better.

Stimuli	Model	KL $\downarrow$	SIM $\uparrow$	NSS $\uparrow$	CC $\uparrow$	AUC $\uparrow$
Desktop	SUM (No Finetuning)	2.201 $\pm$ 0.064	0.221 $\pm$ 0.010	0.636 $\pm$ 0.048	0.157 $\pm$ 0.011	0.741 $\pm$ 0.007
	SUM (Finetuning)	2.127 $\pm$ 0.065	0.217 $\pm$ 0.011	0.921 $\pm$ 0.036	0.247 $\pm$ 0.012	0.761 $\pm$ 0.007
	EEGazeNet (SUM Frozen)	1.965 $\pm$ 0.062	0.265 $\pm$ 0.012	1.009 $\pm$ 0.051	0.271 $\pm$ 0.014	0.784 $\pm$ 0.008
	EEGazeNet (Randomized EEG Descriptive Features)	1.990 $\pm$ 0.066	0.250 $\pm$ 0.013	0.978 $\pm$ 0.044	0.270 $\pm$ 0.014	0.769 $\pm$ 0.007
	EEGazeNet (Randomized EEG Bandpower Ratios)	1.993 $\pm$ 0.067	0.247 $\pm$ 0.013	0.966 $\pm$ 0.041	0.266 $\pm$ 0.015	0.764 $\pm$ 0.007
	EEGazeNet (EEG Bandpower Ratios)	1.875 $\pm$ 0.062	0.280 $\pm$ 0.012	1.197 $\pm$ 0.041	0.328 $\pm$ 0.013	0.811 $\pm$ 0.007

Table 3. Performance comparison on desktop and workshop stimuli for within-user settings using previous EEG signals. Results are reported as mean

\pm

95% confidence intervals across users. Arrows indicate whether higher (

\uparrow

) or lower (

\downarrow

) values are better.

Figure 6. Cross-user attention map visualizations for desktop (first row) and workshop (second row) examples. Outline quant res quant

Visual Attention

Visual attention studies how people and machines select relevant image regions over time for perception, action, and efficient computation.

See all 62 papers

SIMON: Saliency-aware Integrative Multi-view Object-centric Neural Decoding
May 1, 2026YuSheng Lin, Ji-Hwa Tsai, Chun-Shu WeiEEG-Based Visual Decoding Multi-View Learning
National Yang Ming Chiao Tung University, Taiwan
CognitionCapturerPro: Towards High-Fidelity Visual Decoding from EEG/MEG via Multi-modal Information and Asymmetric Alignment
Mar 13, 2026Kaifan Zhang, Lihuo He, Junjie Ke +4EEG-Based Visual Decoding Uncertainty-Aware Multimodal Fusion
Visual Information Processing Laboratory, School of Electronic Engineering, Xidian University, Xi’an 710071, China. · School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China
FSDBN: Foreground-Aware EEG-Visual Alignment via Dynamic Brain Networks
Jul 20, 2026Yiheng Liu, Chuhang Zheng, Peiliang Gong +3EEG-Based Visual Decoding Cross-Modal Alignment
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China · The International Joint Institute of Tianjin University, Fuzhou, Tianjin University, Tianjin, China

May 1, 2026J/K move · Enter open · S save

SIMON: Saliency-aware Integrative Multi-view Object-centric Neural Decoding

YuSheng Lin, Ji-Hwa Tsai, Chun-Shu Wei

National Yang Ming Chiao Tung University, Taiwan

EEG-Based Visual Decoding Multi-View Learning Object-Centric Representation Learning Visual Saliency

Open paper AskPDF arXiv

Recent EEG-to-image retrieval methods leverage pretrained vision encoders and foveation-inspired priors, but typically assume a fixed, center-focused view. This center bias conflicts with content-driven human attention, creating a geometric-semantic dissociation between visual features and EEG responses. We propose SIMON, a saliency-aware multi-view framework for zero-shot EEG-to-image retrieval. SIMON combines foreground segmentation and saliency prediction to select fixation centers via Saliency-Aware Sampling (SAS), then generates foveated views that emphasize informative object regions while suppressing background clutter. On THINGS-EEG, SIMON achieves state-of-the-art performance in both intra-subject and inter-subject settings, reaching an average Top-1 accuracy of 69.7% and 19.6%, respectively, consistently outperforming recent competitive baselines. Analyses across sampling granularity, EEG channel topology, and visual/brain encoder backbones further support the robustness of saliency-aware multi-view integration. Our code and models are publicly available at https://github.com/simonlink666/SIMON.

Abstract

Visual Attention

Explore similar work

SIMON: Saliency-aware Integrative Multi-view Object-centric Neural Decoding

CognitionCapturerPro: Towards High-Fidelity Visual Decoding from EEG/MEG via Multi-modal Information and Asymmetric Alignment

FSDBN: Foreground-Aware EEG-Visual Alignment via Dynamic Brain Networks