eess.IVApr 4, 2026

Cost-Efficient Multi-Scale Fovea for Semantic-Based Visual Search Attention

Authors: João Luzio, Alexandre Bernardino, Plinio Moreno

Organizations: Institute for Systems and Robotics, Instituto Superior Técnico, Lisbon, Portugal

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Semantics are one of the primary sources of top-down preattentive information. Modern deep object detectors excel at extracting such valuable semantic cues from complex visual scenes. However, the size of the visual input to be processed by these detectors can become a bottleneck, particularly in terms of time costs, affecting an artificial attention system's biological plausibility and real-time deployability. Inspired by classical exponential density roll-off topologies, we apply a new artificial foveation module to our novel attention prediction pipeline: the Semantic-based Bayesian Attention (SemBA) framework. We aim at reducing detection-related computational costs without compromising visual task accuracy, thereby making SemBA more biologically plausible. The proposed multi-scale pyramidal field-of-view retains maximum acuity at an innermost level, around a focal point, while gradually increasing distortion for outer levels to mimic peripheral uncertainty via downsampling. In this work we evaluate the performance of our novel Multi-Scale Fovea, incorporated into SemBA, on target-present visual search. We also compare it against other artificial foveal systems, and conduct ablation studies with different deep object detection models to assess the impact of the new topology in terms of computational costs. We experimentally demonstrate that including the new Multi-Scale Fovea module effectively reduces inherent processing costs while improving SemBA's scanpath prediction accuracy. Remarkably, we show that SemBA closely approximates human consistency while retaining the actual human fovea's proportions.

Figures & tables

Fig. 1: Illustration of our novel Multi-Scale Fovea mechanism. The proposed method consists of building a multi-resolution pyramid [ 12 ] , around a selected focal point, and then downsampling all levels to the size of the innermost layer, to mimic the eccentricity effect [ 8 ] . Object detections from outer levels tend to reflect the uncertainty that derives from such exponential pixel density reduction [ 13 ] . This technique facilitates the sequential extraction of semantic information [ 9 ] in a more cost-efficient and biologically plausible manner.
Fig. 2: Schematic representation of the Semantic-based Bayesian Attention framework, i.e. SemBA [ 10 ] , for attention prediction, applied to visual search. In this example, we apply the proposed Multi-Scale Fovea mechanism, that generates N=3N=3 layers around a focal point, starting at a base dimension of 256×256256\times 256 pixels and doubling in size between consecutive layers. All layers are then down-sampled to the base dimension to emulate peripheral distortion. Note that the artificial foveation block can be replaced by any other mechanism, such as FOVEA [ 17 ] or Laplacian Foveation [ 14 ] . The same follows for the deep object detection block, where although we apply YOLOv11 [ 23 ] any other modern detection model can be integrated, e.g. DETR [ 24 ] , RT-DETR [ 25 ] . Object detections are then filtered and fused on multiple grids of 20×3220\times 32 cells, to build semantic belief maps for all KK known classes. SemBA ’s active perception system then selects the map for a specific targeted class, which is cup in this example, and selects the next-best fixation point based on its current belief state. This process is repeated until a termination criterion is met, while applying inhibition of return (IOR [ 27 ] ) to prevent revisiting fixated locations.
Fig. 3: Comparison of the field-of-view topologies for each different foveal system used in this work: Full resolution image (baseline), Laplacian Foveation [ 14 ] , FOVEA (Magnification) [ 17 ] , and our Multi-Scale Fovea (in order, from left to right). Here, Multi-Scale Fovea ’s parameters are set such that the maximum acuity region, corresponding to the fovea [ 6 ] , comprises a region equivalent to around 2.0 \SIUnitSymbolDegree2.0\text{\,}\mathrm{\SIUnitSymbolDegree} of visual angle ( 64×6464\times 64 pixel patch in a 1050×16801050\times 1680 image). Note that SemBA processes each Multi-Scale Fovea layer separately. Here, to showcase the topology, we overlap all layers while preserving their resolution.
Fig. 4: Cumulative performances of humans (average), a random selection algorithm, and SemBA under different (a) object detectors, (b) foveal systems, and (c) Multi-Scale Fovea configurations, on target-present visual search. Regarding experiment (a) we apply our Multi-Scale Fovea on a 4×1604\times 160 configuration. We use the same configuration for experiment (b) when assessing the performance of the Multi-Scale Fovea . For the experiments (b) and (c) we use SemBA ×\times DETR.
 SemSS ↑\uparrowSemFED ↓\downarrowSS ↑\uparrowFED ↓\downarrow
Human Consistency0.4702.1440.4632.353
IVSN [ 31 ]0.3803.0340.3463.360
Gazeformer [ 32 ]0.4901.9280.5042.072
HAT [ 33 ]0.5401.5220.4682.063
CLIPgaze [ 34 ]0.5451.4890.4762.014
SemBA ×\times YOLOv11 ∗0.4482.1610.4142.572
TABLE I: Scanpath Prediction Model Evaluation in Visual Search
 SemSS ∗ ↑\uparrowSemFED ∗ ↓\downarrowPixels ↓\downarrow
Human Consistency0.4702.144-
Baseline (no Fovea)0.4302.025100 %100\text{\,}\mathrm{\%}
Laplacian Foveation [ 14 ]0.4162.488100 %100\text{\,}\mathrm{\%}
FOVEA (Magnification) [ 17 ]0.3842.769100 %100\text{\,}\mathrm{\%}
Multi-Scale Fovea 5×645\times 640.3513.7710.01 %0.01\text{\,}\mathrm{\%}
Multi-Scale Fovea 4×1284\times 1280.4602.4093.72 %3.72\text{\,}\mathrm{\%}
TABLE II: Ablation Study and Computational Cost Evaluation
More in this topic

Scanpath Prediction

See all 8 papers

Explore similar work

CardsList

  1. Efficient Semantic Understanding from Digital Foveation

    Sep 3, 2026Caterina Caccavella, Vittorio Fra, Andreas Ziegler +2Active Visual PerceptionSemantic Segmentation

    Zurich University of Applied Sciences (ZHAW), Wädenswil, Switzerland · ETH Zürich, Zürich, Switzerland · Politecnico di Torino, Turin, Italy +2

  2. Matched Outcomes, Divergent Gaze: How Foveated MLLMs Search Compared to Humans

    Aug 17, 2026Mohamed Amine Kerkouri, Marouane Tliba, Aladine Chetouani +2Visual SearchMultimodal Large Language Models

    F-initiatives, Paris, France · Université Sorbonne Paris Nord, villetaneuse, France · Northwestern University, Chicago, IL, USA +1

  3. CVSearch: Empowering Multimodal LLMs with Cognitive Visual Search for High-Resolution Image Perception

    May 22, 2026Liupeng Li, Haoqian Kang, Zhenyu Lu +4Efficient VLM InferenceLarge Vision-Language Models

    Harbin Institute of Technology, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China.