A Resource-Rational Principle for Modeling Visual Attention Control

Yunpeng Bai

cs.HCMar 2, 2026

A Resource-Rational Principle for Modeling Visual Attention Control

Authors: Yunpeng Bai

Organizations: National University of Singapore, Singapore

Abstract

Understanding how people allocate visual attention is central to Human-Computer Interaction (HCI), yet existing computational models of attention are often either descriptive, task-specific, or difficult to interpret. My dissertation develops a resource-rational, simulation-based framework for modeling visual attention as a sequential decision-making process under perceptual, memory, and time constraints. I formalize visual tasks, such as reading and multitasking, as bounded-optimal control problems using Partially Observable Markov Decision Processes, enabling eye-movement behaviors such as fixation and attention switching to emerge from rational adaptation rather than being hand-coded or purely data-driven. These models are instantiated in simulation environments spanning traditional text reading and reading-while-walking with smart glasses, where they reproduce classic empirical effects, explain observed trade-offs between comprehension and safety, and generate novel predictions under time pressure and interface variation. Collectively, this work contributes a unified computational account of visual attention, offering new tools for theory-driven and resource-efficient HCI design.

Figures & tables

Figure 1. The Heads-Up Multitasker: a hierarchical, resource-rational model of attention allocation during reading while walking. The model enables a simulated agent to coordinate reading on optical head-mounted displays (OHMDs) with safe locomotion by allocating visual attention under competing task demands. The top panel illustrates the hierarchical reinforcement learning architecture, which decomposes attention control into supervisory, task, and motor levels, allowing interpretable trade-offs between reading and safety to emerge from bounded-optimal control. The bottom panels show the simulated multitasking scenario in the external environment: (a) a third-person view of the agent walking in an environment containing hazards; (b) a first-person view in which the agent reads text presented on OHMDs; and (c) a first-person view in which the agent attends to environmental signage to support safe navigation.

Figure 2. Architecture of the resource-rational reading model. The model represents a reader as a sequential decision-making agent that allocates visual attention to maximize text comprehension under constraints of limited memory, visual acuity, and time. Because observations of the text are partial and noisy, the agent maintains probabilistic beliefs at three representational levels: lexical, sentence, and text, supported by corresponding memory systems (lexical store, short-term memory, and long-term gist memory). Decisions are organized hierarchically: a text-level controller guides sentence selection, sentence-level control determines word-level priorities, and word-level control governs eye-movement decisions over visible letters. Rewards combine comprehension utility with resource costs, allowing fixation patterns, skipping, and regressions to emerge as bounded-optimal strategies. This architecture links eye-movement control with memory-driven comprehension, providing an interpretable computational account of human reading behavior relevant to predictive and simulation-based HCI.

Figure 3. Reading behavior under different time constraints (30 s, 60 s, 90 s). The model and human readers adapt their visual attention strategies as available reading time changes, reflecting resource-rational control of comprehension. (a) Heatmaps. With increasing time (from left to right), both humans and the simulation allocate attention more broadly across the text and invest more fixation time, whereas under tight deadlines attention is concentrated on high-coverage regions. (b) Recall. Free-recall responses exhibit the same trade-off, with longer reading time yielding richer gist-level recall and more complete and detailed memory.

Explore similar work

Jul 10, 2026

TDSal: Task-Based Top-Down Saliency Prediction Model

Visual saliency aims to predict the regions of an image most likely to attract human visual attention. While most saliency models assume free-viewing conditions, human attention is often shaped by explicit task goals. In this work, we address task-driven saliency prediction by proposing a model that conditions visual attention on natural-language task descriptions. The model produces task-dependent saliency maps that reflect how attention shifts under different viewing intents. Through quantitative and qualitative analysis, we show that incorporating explicit task semantics enables more faithful modeling of goal-directed visual attention.

Can Mizrakli, Tolga K. Capin

Karlsruhe Institute of Technology, Germany · TED University, Türkiye

Visual Attention Salient Object Detection Visual Saliency

Ask PDF

Jan 13, 2025

Attention when you need

Paying attention improves performance, but attention is metabolically costly, so how should a resource-efficient agent allocate it? We study optimal allocation strategies using a normative model of a signal detection task in which attention comes at a cost. The model reveals that optimal attention is temporally structured in one of two patterns depending on task conditions: when attention costs penalize intense focus, optimal attention ramps up as evidence for the signal accumulates, but with less prohibitive attention costs, optimal attention fluctuates rhythmically. In cases with rhythmic attention, allocation frequency increases with task parameters such as reward magnitude for successful detection, signal brevity, and signal frequency. We argue that rhythmic attention emerges naturally in an agent whose belief updates are dominated by a stable temporal prior and not by the likelihood of new sensory observations. Our results characterize the conditions under which rhythmic vs. ramping attention is optimal and offer a normative account of attentional fluctuations observed in sustained attention tasks.

Grayson Matthew, Lokesh Boominathan, Yizhou Chen +2

Neuroscience Institute Carnegie Mellon University, Pittsburgh, PA 15213 · Neuroscience Baylor College of Medicine, Houston, TX 77030 · Neuroscience Institute and Department of Machine Learning Carnegie Mellon University, Pittsburgh, PA 15213

Temporal Attention

Ask PDF

Aug 31, 2026

Emergent Goal-Directed Attention in Large Vision-Language Models

Human observers prioritize visual information according to task goals. Most computational models of naturalistic viewing are gaze-trained for free viewing, leaving open whether goal-directed attention can emerge in systems without gaze supervision. We tested two off-the-shelf vision-language models (VLMs), Qwen3-VL-32B-Thinking and Gemma-4-26B-A4B-it, on 4,887 naturalistic scenes under visual-search and free-viewing instructions. Model predictions were compared with human fixations on the same images under corresponding tasks. Both models aligned more closely with human fixations under matching goals than under mismatched goals. This crossover persisted in target-absent scenes, where alignment could not be explained by simple visual grounding, and appeared in decoder-layer readouts. Furthermore, model-thinking traces were grounded in target semantics during search and in visual prominence during free viewing. These findings show that general-purpose VLMs can generate human-aligned, goal-directed spatial priorities without gaze-specific training, informing theories of goal-directed attention and offering scalable tools for predicting where people look across tasks.

Han Zhang

University of Michigan Transportation Research Institute, Ann Arbor, MI, USA

Visual Attention Large Vision-Language Models Visual Search VLM Evaluation

Ask PDF