eess.IVApr 4, 2026

Cost-Efficient Multi-Scale Fovea for Semantic-Based Visual Search Attention

Authors: João Luzio, Alexandre Bernardino, Plinio Moreno

Organizations: Institute for Systems and Robotics, Instituto Superior Técnico, Lisbon, Portugal

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Semantics are one of the primary sources of top-down preattentive information. Modern deep object detectors excel at extracting such valuable semantic cues from complex visual scenes. However, the size of the visual input to be processed by these detectors can become a bottleneck, particularly in terms of time costs, affecting an artificial attention system's biological plausibility and real-time deployability. Inspired by classical exponential density roll-off topologies, we apply a new artificial foveation module to our novel attention prediction pipeline: the Semantic-based Bayesian Attention (SemBA) framework. We aim at reducing detection-related computational costs without compromising visual task accuracy, thereby making SemBA more biologically plausible. The proposed multi-scale pyramidal field-of-view retains maximum acuity at an innermost level, around a focal point, while gradually increasing distortion for outer levels to mimic peripheral uncertainty via downsampling. In this work we evaluate the performance of our novel Multi-Scale Fovea, incorporated into SemBA, on target-present visual search. We also compare it against other artificial foveal systems, and conduct ablation studies with different deep object detection models to assess the impact of the new topology in terms of computational costs. We experimentally demonstrate that including the new Multi-Scale Fovea module effectively reduces inherent processing costs while improving SemBA's scanpath prediction accuracy. Remarkably, we show that SemBA closely approximates human consistency while retaining the actual human fovea's proportions.

Figures & tables

Fig. 1: Illustration of our novel Multi-Scale Fovea mechanism. The proposed method consists of building a multi-resolution pyramid [ 12 ] , around a selected focal point, and then downsampling all levels to the size of the innermost layer, to mimic the eccentricity effect [ 8 ] . Object detections from outer levels tend to reflect the uncertainty that derives from such exponential pixel density reduction [ 13 ] . This technique facilitates the sequential extraction of semantic information [ 9 ] in a more cost-efficient and biologically plausible manner.
Fig. 2: Schematic representation of the Semantic-based Bayesian Attention framework, i.e. SemBA [ 10 ] , for attention prediction, applied to visual search. In this example, we apply the proposed Multi-Scale Fovea mechanism, that generates N=3N=3 layers around a focal point, starting at a base dimension of 256×256256\times 256 pixels and doubling in size between consecutive layers. All layers are then down-sampled to the base dimension to emulate peripheral distortion. Note that the artificial foveation block can be replaced by any other mechanism, such as FOVEA [ 17 ] or Laplacian Foveation [ 14 ] . The same follows for the deep object detection block, where although we apply YOLOv11 [ 23 ] any other modern detection model can be integrated, e.g. DETR [ 24 ] , RT-DETR [ 25 ] . Object detections are then filtered and fused on multiple grids of 20×3220\times 32 cells, to build semantic belief maps for all KK known classes. SemBA ’s active perception system then selects the map for a specific targeted class, which is cup in this example, and selects the next-best fixation point based on its current belief state. This process is repeated until a termination criterion is met, while applying inhibition of return (IOR [ 27 ] ) to prevent revisiting fixated locations.
Fig. 3: Comparison of the field-of-view topologies for each different foveal system used in this work: Full resolution image (baseline), Laplacian Foveation [ 14 ] , FOVEA (Magnification) [ 17 ] , and our Multi-Scale Fovea (in order, from left to right). Here, Multi-Scale Fovea ’s parameters are set such that the maximum acuity region, corresponding to the fovea [ 6 ] , comprises a region equivalent to around 2.0 \SIUnitSymbolDegree2.0\text{\,}\mathrm{\SIUnitSymbolDegree} of visual angle ( 64×6464\times 64 pixel patch in a 1050×16801050\times 1680 image). Note that SemBA processes each Multi-Scale Fovea layer separately. Here, to showcase the topology, we overlap all layers while preserving their resolution.
Fig. 4: Cumulative performances of humans (average), a random selection algorithm, and SemBA under different (a) object detectors, (b) foveal systems, and (c) Multi-Scale Fovea configurations, on target-present visual search. Regarding experiment (a) we apply our Multi-Scale Fovea on a 4×1604\times 160 configuration. We use the same configuration for experiment (b) when assessing the performance of the Multi-Scale Fovea . For the experiments (b) and (c) we use SemBA ×\times DETR.
 SemSS ↑\uparrowSemFED ↓\downarrowSS ↑\uparrowFED ↓\downarrow
Human Consistency0.4702.1440.4632.353
IVSN [ 31 ]0.3803.0340.3463.360
Gazeformer [ 32 ]0.4901.9280.5042.072
HAT [ 33 ]0.5401.5220.4682.063
CLIPgaze [ 34 ]0.5451.4890.4762.014
SemBA ×\times YOLOv11 ∗0.4482.1610.4142.572
TABLE I: Scanpath Prediction Model Evaluation in Visual Search
 SemSS ∗ ↑\uparrowSemFED ∗ ↓\downarrowPixels ↓\downarrow
Human Consistency0.4702.144-
Baseline (no Fovea)0.4302.025100 %100\text{\,}\mathrm{\%}
Laplacian Foveation [ 14 ]0.4162.488100 %100\text{\,}\mathrm{\%}
FOVEA (Magnification) [ 17 ]0.3842.769100 %100\text{\,}\mathrm{\%}
Multi-Scale Fovea 5×645\times 640.3513.7710.01 %0.01\text{\,}\mathrm{\%}
Multi-Scale Fovea 4×1284\times 1280.4602.4093.72 %3.72\text{\,}\mathrm{\%}
TABLE II: Ablation Study and Computational Cost Evaluation
More in this topic

Scanpath Prediction

See all 8 papers

Explore similar work

CardsList
Sep 3, 2026

Efficient Semantic Understanding from Digital Foveation

Dense semantic segmentation allocates computational resources uniformly across the entire image, regardless of scene complexity or task relevance. Inspired by biological vision, we investigate whether semantic understanding can be achieved more efficiently through digital foveated perception. We introduce a lightweight active-vision pipeline that combines saliency-driven fixation selection, high-resolution foveal observations, low-resolution contextual information, semantic accumulation, and adaptive computation. Beyond conventional dense prediction metrics, we use object-level evaluation to measure semantic understanding under sparse observations. On ADE20K-Object, a single foveated observation achieves 95.9% of the baseline Top-1 accuracy and 96.9% of the baseline Top-3 accuracy while requiring only 4.7% of the computational cost. At the scene level, semantic accumulation recovers 90.6% of the baseline object recall while using 58.6% of the computation. These results suggest that substantial semantic understanding can emerge from sparse observations when computation is allocated selectively, highlighting active vision as an efficient alternative to uniform dense processing and motivating evaluation protocols beyond conventional pixel-wise segmentation metrics.

Caterina Caccavella, Vittorio Fra, Andreas Ziegler +2

Zurich University of Applied Sciences (ZHAW), Wädenswil, Switzerland · ETH Zürich, Zürich, Switzerland · Politecnico di Torino, Turin, Italy +2

Active Visual PerceptionSemantic SegmentationAdaptive ComputationFoveated Vision
AskPDF
Aug 17, 2026

Matched Outcomes, Divergent Gaze: How Foveated MLLMs Search Compared to Humans

Human visual search is serial: the fovea must land on a candidate to confirm it, and those landings form a scanpath. Whether multimodal large language models (MLLMs), given the same foveated input, search as humans do bears on their use as models of human vision and on attention-alignment scores. We compare three general-purpose MLLMs with human eye-movement scanpaths on goal-directed search (COCO-Search18), driving each model fixation by fixation through an identical, human-matched foveated view and assessing it along three axes: the decision of target presence, the efficiency of reaching the target, and the gaze process itself. The axes dissociate. On the decision and on target acquisition the models match or exceed humans, detecting present targets near ceiling and reaching them on the first saccade more often than people do. The gaze process is not human. Under the human-matched condition, all three share one signature: low-entropy, large-amplitude, self-consistent scanpaths that agree with themselves far more closely than two humans agree with each other. That is consistent with a single-pass, non-serial architecture rather than a limit of acuity. Matched retinal input reproduces where humans look but not how the looking unfolds in time, and no degradation regime recovers human-like search at human-like success. The gap sits on a process axis that answer-alignment and saliency metrics do not measure. Because they miss it, such metrics cannot certify human-like vision, and zero-shot models suit outcome and spatial questions but not temporal, process-level ones.

Mohamed Amine Kerkouri, Marouane Tliba, Aladine Chetouani +2

F-initiatives, Paris, France · Université Sorbonne Paris Nord, villetaneuse, France · Northwestern University, Chicago, IL, USA +1

Visual SearchMultimodal Large Language ModelsVLM EvaluationFoveated Vision
AskPDF
May 22, 2026

CVSearch: Empowering Multimodal LLMs with Cognitive Visual Search for High-Resolution Image Perception

High-resolution (HR) image perception presents a key bottleneck for multimodal large language models (MLLMs). While visual search offers a promising solution, existing methods struggle with the trade-off between coverage and efficiency. Visual expert-assisted search is efficient but prone to blind spots when proposals fail, whereas scan-based search guarantees coverage at the cost of computational redundancy and semantic fragmentation. To address this dilemma, we introduce CVSearch, a training-free adaptive framework that dynamically schedules search strategies via an Assess-then-Search workflow. Specifically, CVSearch first invokes expert-assisted search when global information is insufficient, and only triggers a novel semantic-aware scanning mechanism upon failure. Distinct from rigid grid partitioning, this efficient scanning paradigm incorporates Semantic Guided Adaptive Patching to decompose images into semantically consistent regions, effectively mitigating object fragmentation. Furthermore, we devise a Dynamic Bottom-Up Search strategy driven by a Visual Complexity prior to enable efficient and precise iterative exploration of local details. Extensive experiments on HR benchmarks demonstrate that CVSearch achieves state-of-the-art accuracy while substantially improving search efficiency. Code is released at https://github.com/liliupeng28/ICML26-CVSearch.

Liupeng Li, Haoqian Kang, Zhenyu Lu +4

Harbin Institute of Technology, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China.

Efficient VLM InferenceLarge Vision-Language ModelsVisual Search
AskPDF