FAVE: Foveated Adaptive Visual Encoding for Efficient Fine-Grained Visual Understanding

Amitangshu Mukherjee; Kaushik Roy

cs.CVSep 3, 2026

FAVE: Foveated Adaptive Visual Encoding for Efficient Fine-Grained Visual Understanding

Authors: Amitangshu Mukherjee, Kaushik Roy

Organizations: Elmore Family School of Electrical and Computer Engineering Purdue University

Abstract

Fine-grained visual understanding depends on local detail, yet visual encoders face a trade-off between costly full-image high-resolution processing and compact global encoding that can weaken such evidence. Inspired by human active vision, we separate where to look from what to encode. We focus on the latter and introduce FAVE (Foveated Adaptive Visual Encoding), a lightweight variable-resolution ViT that encodes externally selected regions at high acuity while preserving native geometry. We first isolate this encoding problem using oracle ground-truth crops in a controlled small-object regime. On ImageNet objects with a native maximum side of 96 pixels, FAVE improves Top-1 by 9.4 points over a fixed-resolution ViT on the same crop window with 12.7 times lower FLOPs. Increasing global resolution or backbone capacity does not recover the same operating point. We then integrate FAVE as a complementary local branch in FastVLM. Its local tokens are combined with FastVLM's global visual tokens, while the original global pathway and language model remain frozen. With at most 16 additional local tokens, FAVE improves TextVQA by 1.60 points and achieves a 3.3 times controlled TTFT speedup over SmolVLM2-2.2B. On GQA attribute questions, it improves FastVLM-1.5B by 1.31 points, extending the benefit beyond text while narrowing the gap to FastVLM-7B. Together, these results show that selectively allocating high-acuity local capacity provides an efficient complement to broader global representations and model scaling for fine-grained understanding of small objects, text, and attributes.

Figures & tables

Figure 1: Motivation and overview of adaptive foveated visual processing. (a) Uniform high-resolution processing preserves fine-grained visual evidence but incurs high computational cost, whereas global downsampling is efficient but can discard locally important details. Selective high-resolution processing retains global context while allocating additional computation to salient local regions. (b) Our approach builds on active vision: a coarse peripheral representation preserves global scene context, candidate regions provide potential gaze targets, and selected regions receive adaptive high-acuity processing. The resulting global and foveated local representations support both adaptive visual recognition and multimodal reasoning. 2 2 2 Source photograph: Mendy Van Hoogdalem van Barneveld, Pexels. https://www.pexels.com/photo/photo-of-a-living-room-13675290/

Figure 2: FAVE: adaptive object-centric visual encoding for recognition. Starting from selected object-centric crops, FAVE preserves aspect ratio and aligns each crop to the 16-pixel patch grid, producing variable-resolution patch grids. Each crop yields $P_{i}=H_{i}W_{i}$ patch tokens and sequence length $N_{i}=P_{i}+1$ after prepending a CLS token. Variable-length sequences are processed either with standard padding to $N_{\max}$ or with Patch-n-Pack under token budget $T_{\max}$ , using block-diagonal self-attention to prevent cross-sample interaction. A shared ViT with factorized positional encoding produces one CLS representation per crop for recognition.

		Native bbox $\leq 96$			Native bbox $\leq 224$			General-2K
Model	Input	T1 $\uparrow$	Tok. $\downarrow$	G $\downarrow$	T1 $\uparrow$	Tok. $\downarrow$	G $\downarrow$	T1 $\uparrow$	Tok. $\downarrow$	G $\downarrow$
Fixed-Res ViT-T/16	Image	32.65	197.0	1.283	46.30	197.0	1.283	61.75	197.0	1.283
	Oracle crop	41.05	197.0	1.283	55.70	197.0	1.283	64.60	197.0	1.283
Adaptive ViT-T (Full)	Image	23.80	149.3	0.943	29.25	153.3	0.969	43.25	156.4	0.991
	Oracle crop	9.55	17.7	0.101	29.35	71.6	0.434	42.20	127.3	0.800
NaViT-T	Image	34.25	148.3	0.958	43.45	152.3	0.985	59.55	155.4	1.007

Table 1: Adaptive object recognition under fixed spatial selection. We compare full-image processing with GT-selected object regions across native object-scale cohorts. Ground-truth boxes fix spatial selection so that the object-centric rows isolate how the selected region is encoded. FAVE preserves crop geometry and adapts its spatial resolution and token count, whereas Fixed-Res ViT-T/16 resizes every selected crop to

224\times 224

. T1, Tok., and G denote Top-1 accuracy, mean visual tokens per image, and analytical GFLOPs per image. Bold indicates the best value within each cohort.

Native bbox $\leq 96$
Model	Policy	T1 $\uparrow$	Tok. $\downarrow$	G $\downarrow$
FAVE	Native	50.45	17.7	0.101
FAVE	Adaptive zoom	54.00	40.0	0.234
NaViT-T	LS cap	34.25	148.3	0.958
NaViT-T	$r=224$	34.20	192.3	1.277
Native bbox $\leq 224$

Table 2: Selective local magnification versus additional global resolution. FAVE allocates additional resolution to the GT-selected region using scale-dependent magnification, whereas NaViT increases resolution over the full image. T1, Tok., and G denote Top-1 accuracy, mean visual tokens per image, and analytical GFLOPs per image.

Model	Input	Params	T1 $\uparrow$	GFLOPs $\downarrow$
ViT-T/16	Full	5.72M	32.65	1.283
ViT-T/16	GT	5.72M	41.05	1.283
ViT-S/16	Full	22.05M	43.50	4.657
ViT-S/16	GT	22.05M	45.65	4.657
ViT-B/16	Full	86.57M	45.55	17.680
ViT-B/16	GT	86.57M	47.35	17.680

Table 3: Does increasing backbone capacity recover the small-object operating point? Fixed-resolution ViTs are evaluated on the full image ( Full ) and GT-selected crop ( GT ) on the native-bbox

\leq 96

cohort. FAVE-T uses adaptive native-geometry encoding of the same selected region.

Figure 3: FAVE for multimodal visual understanding. The full image is processed by the frozen FastVLM vision tower and projector to produce 256 global visual tokens, while up to $K$ externally selected regions are encoded by FAVE. In Stage 1, the FAVE ViT and learned reducer are trained jointly from scratch to compress each native-geometry crop from $P_{i}$ variable-length features to one $D_{\mathrm{FAVE}}$ -dimensional token. In Stage 2, the ViT is frozen, the reducer is fine-tuned, and a learned local projector maps the $K$ crop tokens into the $D_{\mathrm{LLM}}$ space. These local tokens are concatenated with the 256 FastVLM tokens and, together with the prompt, processed by the frozen Qwen2 language model. $D_{G}$ , $D_{\mathrm{FAVE}}$ , and $D_{\mathrm{LLM}}$ denote the global feature, FAVE feature, and language-model embedding widths, respectively.

Compact VLMs
Method	TextVQA $\uparrow$
SmolVLM-2.2B [ 31 ]	72.1 †
SmolVLM2-2.2B [ 31 ]	73.21 †
MobileVLM V2-1.7B [ 6 ]	52.1 †
MobileVLM V2-3B	57.5 †
DeepSeek-VL-1.3B [ 30 ]	55.8 †

Table 4: TextVQA accuracy. FAVE improves the reproduced FastVLM-1.5B baseline from 71.95 to 73.55 while retaining its efficient global pathway and adding sparse local visual evidence.

\dagger

denotes externally reported results.

\ddagger

denotes results under the unified LLaVA-1.5-7B evaluation reported by Q-Zoom [ 42 ] .

Compact VLM
Method	Vision enc. (ms) $\downarrow$	TTFT (ms) $\downarrow$
SmolVLM2-2.2B [ 31 ]	169.65	246.54
Efficient global + local encoding
FastVLM-1.5B [ 48 ]	29.54	59.00
+ FAVE (Mean)	$29.54+\textbf{9.25}=38.79$	68.03
+ FAVE ( $K{=}16$ , $M{=}1$ )	$29.54+\textbf{9.18}=38.72$	74.51

Table 5: TextVQA inference efficiency. We report vision-encoder latency and end-to-end time-to-first-token (TTFT). For FAVE, vision latency is decomposed into frozen FastViTHD global encoder and local FAVE encoder over all selected crops. FAVE uses a 224-pixel crop cap with Patch-n-Pack inference. TTFT is measured on an NVIDIA A40 with batch size 1 over a fixed 500-example subset; vision latency is measured separately on 50 examples.

Subset	FastVLM	FAVE	$\Delta$
OCRBench overall	54.5	55.8	+1.3
Scene Text VQA	66.5	80.0	+13.5
ESTVQA	58.0	80.0	+22.0
STVQA	40.0	66.0	+26.0
OCRVQA	94.0	96.0	+2.0
TextVQA	74.0	78.0	+4.0

Table 6: OCRBench transfer without OCRBench-specific training. Values are accuracy (%), and

\Delta

denotes percentage-point improvement. FAVE improves the overall OCRBench score and yields its largest gains on Scene Text-centric VQA, where localized scene text is most relevant.

Compact VLM
Method	Total params.	GQA Attribute $\uparrow$
SmolVLM2-2.2B [ 31 ]	2.247B	57.40
TinyLLaVA-Phi2-SigLIP [ 57 ]	3.1B	68.40
Fine-grained VLM
AttWarp (LLaVA-1.5-7B) [ 8 ]	$\sim$ 7B	69.30
ViCrop (LLaVA-1.5-7B) [ 56 ]	$\sim$ 7B	68.20

Table 7: Fine-grained attribute understanding on GQA. We report accuracy on questions labeled as attribute under the GQA semantic question type. FAVE improves FastVLM-1.5B from 70.73 to 72.04 while keeping the global visual pathway and language model frozen. It matches the substantially larger Mini-Gemini-HD-8B (72.02) and remains within 0.68 points of FastVLM-7B.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Qualitative GQA examples of local attribute and state recognition. FAVE correctly distinguishes the outlet and mug colors and recognizes that the plant pot is full rather than empty . The latter closely echoes the coarse-to-fine motivation of Figure 1 in the main paper: localized high-acuity processing recovers fine object-state evidence that can be weakened in the compact global representation.

Panel of Figure 5 — Figure 5: Qualitative GQA examples of fine-grained object-property recovery. FAVE correctly identifies the lamp as white and tall and the device on the mouse pad as small and silver , recovering the ground-truth answers in cases where FastVLM-1.5B fails.

Panel of Figure 6 — Figure 6: Qualitative GQA examples of spatial disambiguation and small-object attribute recovery. In the first example, the field is grassy but not white; the visible white regions belong to the players. FAVE correctly answers no , separating attributes of nearby objects from those of the queried field. In the second, FAVE correctly identifies the middle traffic light as green , recovering fine-grained evidence from a small object within a larger street scene.

Fine-Grained Visual Perception

Fine-grained visual perception studies how models detect, distinguish, and ground subtle visual details, attributes, parts, and changes.

See all 152 papers

Jul 27, 2026

MAViE: A Multi-scale Adaptive Vision Encoder for Fine-grained Visual Perception and Efficient Multimodal Reasoning

Vision-language models commonly project all tokens produced by a pretrained vision encoder into a large language model. However, final-layer features can discard text, local attributes, and spatial relationships, while high-resolution inputs substantially increase context length and inference latency. We introduce \method, a Multi-scale Adaptive Vision Encoder. \method uses position-dependent gates to fuse shallow, intermediate, and deep features from a vision Transformer, preserving global semantics while enhancing edges, text, and local structure. It then performs question-conditioned token routing according to question relevance, local information content, global semantics, and spatial coverage, with a token budget that adapts to image complexity. To mitigate compression loss, we further introduce full-to-compressed representation distillation and a spatial diversity regularizer. In an illustrative simulation under a unified 7B language-model framework, \method reduces the average number of SigLIP-SO400M visual tokens from 729 to 146 (approximately 80.0%) and improves the mean score on VQAv2, GQA, TextVQA, ScienceQA-IMG, and MMBench by 2.2 percentage points, while reducing single-image time to first token from 228,ms to 129,ms. We provide the full model design and evaluation protocol. All reported numbers currently serve only as placeholders for paper organization and experimental design; formal claims require real training runs, independent replications, and official benchmark evaluation.

Shaofei Lei

Efficient VLM Inference Multi-Scale Feature Fusion VLM Reasoning Visual Token Compression

Ask PDF

Sep 29, 2026

EviViT: Evidence-Adaptive Vision Transformers for Fine-Grained Perception

Fine-grained visual perception enables vision-language models to distinguish subtle attributes and ground their answers in visual evidence. In high-resolution scenes, processing the whole image at greater resolution spends visual tokens on irrelevant content, while isolated crops can lose the context needed to interpret the selected evidence. We introduce EviViT, a lightweight attachment that learns where a pretrained vision transformer should acquire detail. Human visual-search traces supervise a question-conditioned evidence density, which guides regional re-reading from the original pixels and the allocation of visual tokens. A sparse, coordinate-aware bridge then connects the regional features to the global scene, allowing the host to interpret precise evidence in context. Learned with the host backbone frozen, the attachment serves both the base model and compatible post-trained descendants without refitting. Experiments across nine hosts show consistent gains in average fine-grained accuracy. Matched-budget comparisons further show that EviViT outperforms global-only processing at every tested token ceiling while using fewer visual tokens.

Yaoxin Niu, Zhangquan Chen, Yang Zhang +5

Tsinghua University · Peng Cheng Laboratory · The Hong Kong University of Science and Technology +3

VLM Adaptation Efficient ViTs Fine-Grained Visual Perception

Ask PDF

Oct 6, 2026

Foveated Compression: Selective High-Resolution Preservation for Token-Efficient VLMs

Visual tokens are a major source of inference cost in vision-language models, yet simple image downsampling remains a surprisingly strong compression baseline. This raises a complementary question: under a fixed token budget, where should visual fidelity be preserved? We introduce Foveated Compression, which encodes a full-resolution image once and represents it with a mixture of native- and compressed-resolution visual tokens. A behaviorally self-distilled Foveated Merger compresses local visual tokens while preserving compatibility with their native counterparts, and a lightweight Foveated Selector chooses one of nine spatial cells to retain at native resolution using exhaustive budget-matched intervention supervision. At 11.11% visual tokens, uniform Foveated Compression shows no significant paired difference from iso-token downsampling. At 20.99%, the learned selector significantly outperforms random and fixed allocation, but remains below strong whole-image resizing, showing that localized fidelity is not universally preferable. A budget-matched region-choice oracle reaches 82.73 macro accuracy versus 69.61 for the learned selector, revealing substantial headroom within the same spatial action space. Matched probing further shows that signals predicting when compression breaks the answer are substantially more accessible after language-model computation than to the lightweight prefill-free selector. These results expose complementary bottlenecks in region selection and compressed-region fidelity.

Donghyun Han, Jangho Park, Yuseok Bae

ETRI, South Korea · Kyung Hee University, South Korea

Efficient VLM Inference Visual Token Pruning Visual Token Compression

Ask PDF

Abstract

Fine-Grained Visual Perception

Explore similar work

MAViE: A Multi-scale Adaptive Vision Encoder for Fine-grained Visual Perception and Efficient Multimodal Reasoning

EviViT: Evidence-Adaptive Vision Transformers for Fine-Grained Perception

Foveated Compression: Selective High-Resolution Preservation for Token-Efficient VLMs