cs.LGApr 6, 2026

Estimating Central, Peripheral, and Temporal Visual Contributions to Human Decision Making in Atari Games

Authors: Henrik Krauss, Takehisa Yairi

Organizations: Department of Advanced Interdisciplinary Studies, The University of Tokyo, Tokyo, 153-8904, Japan. · Research Center for Advanced Science and Technology, The University of Tokyo, Tokyo, 153-8904, Japan.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We study how different visual information sources contribute to human decision making in dynamic visual environments. Using Atari-HEAD, a large-scale Atari gameplay dataset with synchronized eye-tracking, we introduce a controlled ablation framework as a means to reverse-engineer the contribution of peripheral visual information, explicit gaze information in the form of gaze maps, and past-state information from human behavior. We train action-prediction networks under six settings that selectively include or exclude these information sources. Across 20 games, peripheral information shows by far the strongest contribution, with median prediction-accuracy drops in the range of 35.27-43.90% when removed. Gaze information yields smaller drops of 2.11-2.76%, while past-state information shows a broader range of 1.52-15.51%, with the upper end likely more informative due to reduced peripheral-information leakage. To complement aggregate accuracies, we cluster states by true-action probabilities assigned by the different model configurations. This analysis identifies coarse behavioral regimes, including focus-dominated, periphery-dominated, and more contextual decision situations. These results suggest that human decision making in Atari depends strongly on information beyond the current focus of gaze, while the proposed framework provides a way to estimate such information-source contributions from behavior.

Figures & tables

Fig. 1: The controlled ablation framework performed in this study: Human gameplay and eye-tracking data from the Atari-HEAD dataset [ 1 ] are used to identify focus and periphery region, and construct gaze maps and past states. These information sources are included and excluded for the training of several action prediction networks to estimate their respective contributions.
Model1) Periphery2) Gaze3) Past stateNote
A❍❍❍All information
B❍❍✗comparable to AGIL [ 11 ]
C❍✗❍ 
D❍✗✗Standard IL
E✗❍❍ 
F✗❍✗ 
TABLE I: Model configurations of including and excluding different information sources as used in the controlled ablation study.
Fig. 2: Architecture of the human action prediction network with three options of including (I) peripheral information, (II) gaze information, and (III) past-state information.
Fig. 3: Validation action-prediction accuracies across games (left) and median relative performance drops with respect to model A, normalized by the A-common gap (right).
 ABCDEF
A+0.00+1.54+2.16+4.31+54.49+71.75
B-1.52+0.00-0.66+2.84+54.32+78.28
C-2.11+0.66+0.00+1.98+41.52+64.55
D-4.13-2.76-1.94+0.00+46.86+68.34
E-35.27-35.19-28.97-31.84+0.00+18.36
F-41.59-43.90-38.45-40.58-15.51+0.00
TABLE II: Median normalized pairwise accuracy differences (%) across games. Normalized by the column model’s accuracy difference with the common choice.
Fig. 4: Cluster composition across games (upper), mean true-action-probability profiles (middle), and namings (lower) for the five clusters.
Fig. 5: Mean silhouette scores per game and cluster, with an additional overall column.
Fig. 6: t-SNE visualization of the clustered six-model response space for all games (top) and per-game subsamples (bottom).
Fig. 7: Example states from the two games with the highest silhouette score for each cluster. Focus region of six visual degrees is shaded in blue.
Fig. 8: Single-subject comparison for DemonAttack and SpaceInvaders. Action-prediction accuracy across model types (top). Cluster composition (middle). Score statistics (bottom).
More in this topic

Human Behavior Prediction

Human behavior prediction models how people will act, decide, move, or feel from their histories, context, and observable signals.

See all 75 papers

Explore similar work

CardsList
Sep 29, 2026

GazeFlow: From Human Gaze Behavior to Generative Egocentric Gaze Prediction

Egocentric gaze prediction enables many downstream applications but remains challenging, as human gaze is inherently stochastic. This stochasticity is constrained by structured temporal dynamics alternating between fixations and saccades, top-down influences from tasks, and bottom-up visual saliency. Based on this observation, we introduce GazeFlow, a framework that directly models gaze as a joint distribution of temporal gaze positions conditioned upon both top-down and bottom-up information. In particular, GazeFlow uses conditional flow matching (CFM): a learned velocity field iteratively transports a Gaussian noise sample into a plausible gaze trajectory drawn from this joint distribution. The velocity field is conditioned on bottom-up visual features extracted by a video encoder and on top-down task information obtained by globally querying these features. On standard datasets, GazeFlow achieves state-of-the-art performance on per-frame metrics, and the generated trajectories align better with human gaze temporal dynamics.

Sheng Zhao, Weikai Lin, Yuhao Zhu

Department of Computer Science University of Rochester

Gaze EstimationEgocentric Video UnderstandingGaze Forecasting
AskPDF
Nov 16, 2025

Predicting upcoming visual features during eye movements yields scene representations aligned with human visual cortex

Natural scenes are complex arrangements of objects, surfaces, and backgrounds. For the brain's visual system to effectively operate, it needs to extract not only what objects are present, but also their spatial and semantic relations. We hypothesize that such structures may be learned, in a self-supervised fashion, by exploiting temporal regularities of natural active vision: each fixation reveals a glimpse that is related to the previous one via co-occurrence and saccade-conditioned spatial regularities. We instantiate this idea with Glimpse Prediction Networks (GPNs), recurrent models trained to predict the embedding of the next glimpse along human-like scanpaths. GPNs are shown to successfully extract complex scene information, including object co-occurrences and spatial object arrangements, and integrate information across glimpses. Importantly, GPN representations align strongly with human fMRI responses in mid and higher-level visual cortex and match, often outperform, alternative state-of-the-art ANN models, establishing next-glimpse-prediction as a biologically plausible route towards brain-aligned scene representations.

Sushrut Thorat, Adrien Doerig, Alexander Kroner +2

Institute of Cognitive Science, Osnabrück University, Osnabrück 49069, Germany. · Lossfunk, Bengaluru 560038, India. · Department of Education and Psychology, Freie Universität, Berlin 14195, Germany. +1

Predictive CodingVisual NeurosciencePredictive Representation LearningActive Visual PerceptionSelf-Supervised Visual Representation Learning
AskPDF
Oct 6, 2026

Knowing When to Trust a Prior: Reliability-Gated Cue Fusion for Video Gaze Prediction

Video gaze prediction is led by gaze-trained models, yet gaze-free priors carry signal those models have not absorbed, if one knows when to trust them. We propose FocusGate, a gated ensemble of gaze-free priors whose members may abstain. A per-frame gate reads three shape statistics of a defocus map and selects the frames on which the estimator is above chance on average, so rejected frames reduce to the base exactly, while midrank normalisation lets an all-zero prior abstain at zero parameters. Gated fusion is significantly positive on film, sports and web video, whereas unconditional fusion is harmful on sports and null on web. Added to four supervised predictors, the NTIRE 2026 champion among them, FocusGate improves all sixteen model-domain cells in shuffled AUC, fifteen significantly, one domain pre-registered and scored once, while adding only 1% to the champion's latency. Alone, it surpasses TASED-Net and UNISAL in shuffled AUC on film with a 16-frame causal mean.

Lichen Zhu, Yueqian Lin, Yiheng Wang +2

Duke University, Durham, NC, USA

Gaze EstimationSelective PredictionUncertainty-Aware Multimodal FusionGaze Forecasting
AskPDF