From Visual Search to Movement Control: A Priority Field for Artificial Agents

Han Zhang; Zhong Cao

cs.ROSep 25, 2026

From Visual Search to Movement Control: A Priority Field for Artificial Agents

Authors: Han Zhang, Zhong Cao

Organizations: University of Michigan Transportation Research Institute · Civil and Environmental Engineering, University of Michigan

Abstract

Human spatial attention is widely conceptualized as being guided by a priority map that integrates perceptual salience, current goals, and past experiences. Here, we extend priority-based computation to movement control in artificial agents. We first introduce a lightweight model of visual search based on an integrated priority map. Trained on human saccades, it reproduced key behavioral patterns, including oculomotor suppression and history-driven selection. Extending the search model, we equipped an artificial agent with a priority field and evaluated its performance in a reach-avoid task that required reaching a goal destination while avoiding moving obstacles. Compared with alternative architectures, priority-field agents trained more efficiently and performed better in unseen, complex scenarios, even from simple demonstrations. Adding a simple memory mechanism also produced human-like, history-driven effects in anticipating the likely location of the upcoming goal. These findings suggest that priority-based computation may provide a promising foundation for movement control in artificial agents.

Figures & tables

Figure 1: Conceptual diagrams of priority-based models for visual search (A) and agent movement (B). In both cases, behavior is guided by a priority representation that integrates perceptual salience, the current goal, and selection history. This representation is a priority map defined over item locations in visual search, and a priority field defined over movement directions in agent movement.

Figure 2: The fitted model reproduced key patterns of visual search. (A) Oculomotor suppression in held-out participants and model predictions. (B) Model predictions for high- and low-salience distractors. (C) Target-location priming in held-out participants and model predictions.

Figure 3: Testing scenarios. The agent’s task is to reach the goal (green ring) while avoiding moving obstacles (blue circles). Panel A shows the demonstrated task used at training. Panel B shows one scalability test with 50 obstacles. Panel C shows another scalability test with obstacles moving at 4 $\times$ speed. Panel D shows the statistical learning test, where the goal is more likely to appear inside the left region. In all panels, the agent’s real-time priority field is shown as colored rays in each direction. The agent’s movement reflects the combined pull of high-priority (green) directions and push of low-priority (red) directions.

Figure 4: Comparison of the priority-field agent against two alternative models without explicit priority-based computation (MLP, transformer). (A) Training loss (mean squared error, log scale) over 500 epochs. (B) Performance on the demonstrated task (10 obstacles at the training speed): goals reached per minute (left; dashed line = scripted expert) and collisions per minute (right). (C) Density test: goals (top) and collisions (bottom) per minute as the number of obstacles increases from 10 to 50, with speed held at the training level. (D) Speed test: goals (top) and collisions (bottom) per minute as obstacle speed increases up to 4

\times

the training level, with the number of obstacles held at 10. In B–D, bars/points show the mean across 20 random seeds, and error bars/shaded bands indicate 95% bootstrap confidence intervals.

Figure 5: The history field produces anticipatory, history-driven movement. Anticipatory movement in the statistical-learning task, measured as the agent’s horizontal offset from the arena center at goal onset (negative = toward the high-probability left region). Goals were spatially biased for the first 90 trials (70% in the left region) and unbiased for the next 90 trials. “Agent with memory” is the priority-field agent with its history field enabled (strength $w_{H}=0.20$ , update rate $\eta_{H}=0.15$ ). “Agent without memory” is the same agent with the history field disabled ( $w_{H}=0$ ). The memory agent drifted toward the high-probability region during the biased block and returned to the center once goals became unbiased. Each dot represents the mean offset over 10-trial bins. Shaded bands are bootstrap 95% confidence intervals computed from 20 runs.

Vision-Based Robot Control

Vision-based robot control uses camera images to guide robots’ navigation and manipulation through closed-loop perception and action.

See all 221 papers

Why We Look Where We Look: Emergent Human-like Fixations of a Foveated Visual Language Model Maximizing Scene Understanding
May 18, 2026Shravan Murlidaran, Ziqi Wen, Sana Shehabi +1Visual Attention Large Vision-Language Models
Psychological & Brain Sciences, University of California, Santa Barbara & 93106, USA. · Computer Science, University of California, Santa Barbara & 93106, USA. · Electrical and Computer Engineering, University of California, Santa Barbara & 93106, USA.
Telogenesis: Goal Is All U Need
Mar 10, 2026Zhuoran Deng, Yizhi Zhang, Ziyi Zhang +1Uncertainty-Aware Exploration Attention Mechanisms
Independent Research
Emergent Goal-Directed Attention in Large Vision-Language Models
Aug 31, 2026Han ZhangVisual Attention Large Vision-Language Models
University of Michigan Transportation Research Institute, Ann Arbor, MI, USA

May 18, 2026J/K move · Enter open · S save

Why We Look Where We Look: Emergent Human-like Fixations of a Foveated Visual Language Model Maximizing Scene Understanding

Shravan Murlidaran, Ziqi Wen, Sana Shehabi, Miguel P. Eckstein

Psychological & Brain Sciences, University of California, Santa Barbara & 93106, USA. · Computer Science, University of California, Santa Barbara & 93106, USA. · Electrical and Computer Engineering, University of California, Santa Barbara & 93106, USA.

Visual Attention Large Vision-Language Models Active Visual Perception Foveated Vision Human Visual Perception

Open paper AskPDF arXiv

When humans view scenes without a specific task (free-viewing), they initially direct their eye movements toward the scene center and then fixate on people, text, objects being gazed at or grasped, and semantically meaningful regions. What these signature fixation patterns reflect and whether they optimize an underlying perceptual task remain unknown. We show that a computational agent with simulated foveation, trained to optimize scene comprehension, exhibits emergent human fixation signature patterns. In contrast, versions of the agent trained to search or classify scenes, or equipped with peripheral vision that was better or worse than human vision, predicted human fixation patterns less accurately. Thus, human free-viewing fixation patterns may emerge as a functional byproduct of optimizing scene comprehension under the biological constraints of foveated vision.

Abstract

Vision-Based Robot Control

Explore similar work

Why We Look Where We Look: Emergent Human-like Fixations of a Foveated Visual Language Model Maximizing Scene Understanding

Telogenesis: Goal Is All U Need

Emergent Goal-Directed Attention in Large Vision-Language Models