LLMind: Bio-inspired Training-free Adaptive Visual Representations for Vision-Language Models

Soumyaratna Debnath; Bui Duc Manh; Zinan Liu; Lin Wang

cs.CVMar 16, 2026

LLMind: Bio-inspired Training-free Adaptive Visual Representations for Vision-Language Models

Authors: Soumyaratna Debnath, Bui Duc Manh, Zinan Liu, Lin Wang

Organizations: Nanyang Technological University, Singapore

Abstract

Vision-Language Models (VLMs) typically assume a uniform spatial fidelity across the entire field of view of visual inputs, dedicating equal precision to even the uninformative regions. By contrast, human vision is neither uniform nor static; it is adaptive, selective, and resource-efficient. In light of this, we present the first systematic analysis of bio-inspired visual representation methods, providing insights for more efficient and adaptive VLMs. We propose LLMind (Looking Like the Mind), a novel training-free framework that mimics foveated encoding and cortical magnification in human vision to achieve adaptive, efficient representations for VLMs under tight pixel budgets. Our key idea is to explore a Bio-inspired Adaptive Sampling Strategy (BASS), enabling a Mobius-parameterized module that performs non-uniform sampling while preserving global scene structure. On top of BASS, we introduce closed-loop semantic feedback (CSF) via test-time adaptation to align perceptual saliency with textual information from the frozen VLM. We evaluate LLMind against uniform and other sampling baselines across diverse scene-level and region-guided visual question answering benchmarks. The results show dramatic gains, with average improvements of +20% on VQAv2, +38% on Seed-Bench, and +37% on A-OKVQA compared to uniform sampling under tight pixel budgets. More surprisingly, LLMind retains up to 82%, 92%, and 97% of the full-resolution performance using only 1%, 3%, and 5% of the pixels, respectively. Moreover, LLMind is lightweight, plug-and-play, and compatible with existing VLMs without requiring architectural changes.

Figures & tables

Figure 2: Qualitative comparison with Qwen2.5-VL on Seed-Bench at 5% pixel budget. LLMind adaptively allocates resolution to semantically important regions, preserving visual evidence critical for answering the question. Zoom in for a better view .

Figure 3: Overview of the proposed framework. Given the input image $I$ , the MLP network predicts the Möbius transformation coefficients, which are used by BASS (Sec. 3.1 ) to produce the sampled image $\hat{I}$ . The Perceptual loss $\mathcal{L}_{\text{img}}$ (Eq. 7 ) between $I$ and $\hat{I}$ flows through the network to optimize the MLP parameters. In parallel, a frozen VLM processes a set of questions $q$ to generate predicted answers $y_{\text{pred}}$ . These are compared with the ground-truth answers $y_{\text{gt}}$ to obtain the Semantic loss $\mathcal{L}_{\text{text}}$ (Eq. 8 ) for further guiding the optimization of the MLP parameters using SPSA (Eq. 9 ).

Figure 4: Illustration of the Bio-inspired Adaptive Sampling Strategy (BASS). Given an input image $I$ , the MLP predicts Möbius parameters $\theta$ to warp the image toward salient regions (Eq. 3 ). The warped image is uniformly sampled under pixel budget $B$ through $\mathcal{S}_{B}(\cdot)$ , and then reconstructed to its original resolution via an interpolation operator $\mathcal{I}(\cdot)$ . Finally, the inverse transformation (Eq. 5 ) restores the global spatial structure, yielding the adaptively sampled image $\hat{I}$ .

Dataset	Model	Accuracy (%)	Sampling Method	Accuracy Sampled			$\Delta$ Accuracy Gain (%)			Accuracy Retained
		Full Res.		(%)			over Uniform Sample			(% of Full Res.)
		Full Res.		1%	3%	5%	1%	3%	5%	1%	3%	5%
VQAv2	Qwen2.5-VL	86.96	Uniform Sampling	47.31	52.71	59.94	–	–	–	54.40	60.61	68.92
			Static Foveated	46.18	51.96	58.30	-2.38	-1.42	-2.73	53.10	59.75	67.04
			Sunflower Inspired	43.77	47.41	52.85	-7.48	-10.05	-11.82	50.33	54.51	60.77
			Radial Sampling	44.79	46.25	50.12	-5.32	-12.25	-16.38	51.50	53.18	57.63

Table 1: Quantitative comparison of bio-inspired and uniform sampling strategies on scene-level VQA benchmarks under tight-budget constraints (1%, 3%, 5%) . We report full-resolution accuracy, sampled accuracy at each budget, accuracy gain over uniform sampling, and accuracy retained relative to the full-resolution baseline.

Figure 5: Illustration of the compared sampling methods under 10% pixel budget. Zoom in for a better view .

Table 6

Figure 6: Category-wise performance on A-OKVQA at a 5% pixel budget with Qwen2.5-VL (4B). Zoom in for a better view .

Figure 7: Ablation study on A-OKVQA dataset with Qwen2.5-VL (4B) illustrating the impact of pixel budget and the CSF module on LLMind’s performance. Zoom in for a better view .

Foveated Vision

See all 16 papers

May 18, 2026

Starve to Perceive: Taming Lazy Perception in VLMs with Constrained Visual Bandwidth

Vision-Language Models (VLMs) deployed as situated agents in high-resolution visual environments require active perception -- the ability to dynamically decide where to look through operations like zooming, cropping, and panning. However, current training paradigms produce models that mimic the surface form of such operations without functionally depending on their outputs, a phenomenon we term lazy perception. We trace this to a fundamental learning asymmetry: when coarse global views combined with language priors suffice for moderate accuracy, the model has no incentive to learn harder multi-step visual search. If a model can succeed without actively looking, it will never learn to look. This motivates Starve to Perceive, a training paradigm that constrains visual bandwidth -- restricting each observation to a tight token budget so that no single view suffices for task completion, making active perception the only viable strategy. Despite requiring no auxiliary losses, reward shaping, or architectural changes -- serving as a minimal, plug-in modification to standard post-training pipelines -- models trained under perceptual starvation achieve substantial gains of 5% average relative improvement across diverse benchmarks.

Yuhuan Wu, Cong Wei, Fangzhen Lin +2

Hong Kong University of Science and Technology · University of Waterloo

Vision-Language Models VLM Adaptation Visual Search

Ask PDF

Feb 26, 2026

LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models

Multimodal Large Language Models (MLLMs) are shifting towards "Thinking with Images" by actively exploring image details. While effective, large-scale training is computationally expensive, which has spurred growing interest in lightweight, training-free solutions. However, existing training-free methods suffer from two flaws: perceptual redundancy from indiscriminate cropping, which increases computational cost and introduces noise; and a drift between semantic intent and spatial attention, which prevents accurate localization of user-focused regions. To address these challenges, we propose LookWise, a framework for adaptive visual reasoning. LookWise follows a two-stage pipeline: a confidence-based module decides when to look more carefully, and a semantic-guided localization module determines where to look. This design enables MLLMs to adaptively acquire fine-grained visual evidence without additional training. Experiments on fine-grained and high-resolution visual reasoning benchmarks show that LookWise consistently improves accuracy over strong baselines while achieving an approximately

4.0\times

inference speedup over the search-based method ZoomEye, demonstrating robust cross-model generalization.

Yuxiang Shen, Hailong Huang, Zhenkun Gao +6

the Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China, and also with the University of Science and Technology of China, Hefei 230026, China · Zhejiang University, Hangzhou 310027, China · East China Normal University, Shanghai 200241, China +3

VLM Reasoning Large Vision-Language Models Fine-Grained Visual Perception Efficient VLM Inference

Ask PDF

Mar 19, 2026

Tinted Frames: Question Framing Blinds Vision-Language Models

Vision-Language Models (VLMs) have been shown to be blind, often underutilizing their visual inputs even on tasks that require visual reasoning. In this work, we demonstrate that VLMs are selectively blind. They modulate the amount of attention applied to visual inputs based on linguistic framing even when alternative framings demand identical visual reasoning. Using visual attention as a probe, we quantify how framing alters both the amount and distribution of attention over the image. Constrained framings, such as multiple choice and yes/no, induce substantially lower attention to image context compared to open-ended, reduce focus on task-relevant regions, and shift attention towards uninformative tokens. We further demonstrate that this attention misallocation is the principal cause of degraded accuracy and cross-framing inconsistency. Building on this mechanistic insight, we introduce a lightweight prompt-tuning method using learnable tokens that encourages the robust, visually grounded attention patterns observed in open-ended settings, improving visual grounding and improving performance across framings.

Wan-Cyuan Fan, Jiayun Luo, Declan Kutscher +2

University of British Columbia · Vector Institute for AI · University of California, Berkeley

Vision-Language Models VLM Robustness Visual Evidence Sensitivity in VLMs Reasoning for VLMs Vision-Language Grounding

Ask PDF

Abstract

Foveated Vision

Explore similar work

Starve to Perceive: Taming Lazy Perception in VLMs with Constrained Visual Bandwidth

LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models

Tinted Frames: Question Framing Blinds Vision-Language Models