FineViT: Progressively Unlocking Fine-Grained Perception with Dense Recaptions

Peisen Zhao; Xiaopeng Zhang; Mingxing Xu; Ruoyu Sun; Zewei Du; Dunzheng Wang; Guanghao Zheng; Haohang Xu

cs.CVMar 18, 2026

FineViT: Progressively Unlocking Fine-Grained Perception with Dense Recaptions

Authors: Peisen Zhao, Xiaopeng Zhang, Mingxing Xu, Ruoyu Sun, Zewei Du, Dunzheng Wang, Guanghao Zheng, Haohang Xu, +5 more

Organizations: Huawei Inc.

Abstract

While Multimodal Large Language Models (MLLMs) have experienced rapid advancements, their visual encoders frequently remain a performance bottleneck. Conventional CLIP-based encoders struggle with dense spatial tasks due to the loss of visual details caused by low-resolution pretraining and the reliance on noisy, coarse web-crawled image-text pairs. To overcome these limitations, we introduce FineViT, a novel vision encoder specifically designed to unlock fine-grained perception. By replacing coarse web data with dense recaptions, we systematically mitigate information loss through a progressive training paradigm.: first, the encoder is trained from scratch at a high native resolution on billions of global recaptioned image-text pairs, establishing a robust, detail rich semantic foundation. Subsequently, we further enhance its local perception through LLM alignment, utilizing our curated FineCap-450M dataset that comprises over $450$ million high quality local captions. Extensive experiments validate the effectiveness of the progressive strategy. FineViT achieves state-of-the-art zero-shot recognition and retrieval performance, especially in long-context retrieval, and consistently outperforms multimodal visual encoders such as SigLIP2 and Qwen-ViT when integrated into MLLMs. We hope FineViT could serve as a powerful new baseline for fine-grained visual perception.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Construction of rich QA training data from FineCap-450M, highlighting detailed local-level tasks. (b) FineViT performance boost in multimodal tasks enabled by FineCap-450M local-task pre-training.

Model	Depth	Patch Size	Hidden Size	Interm. Size	Heads	Activation	Pos. Embed
FineViT	28	14	1536	4608	16	SiLU	2D RoPE

Table 1: Architectural details of FineViT.

Figure 2: Illustration of FineViT training stages. Stage I utilizes a Masked Image Modeling (MIM) loss to establish foundational visual perception. Stage II implements a SigLIP loss via large-scale image-text contrastive learning to bridge the gap between visual features and semantic concepts. Finally, Stage III integrates a Large Language Model (LLM) at high resolution to train the model on global and local Question-Answering (QA) tasks, ensuring a robust sensitivity to fine-grained visual details.

Figure 3: The pipeline of data curation and recaption.

Global Caption		Local Caption			Rich-text OCR		Doc OCR		# total regions
# images	avg. token	# regions	avg. token	# categories	# regions	avg. token	# regions	avg. token	# total regions
63M	211.25	226M	56.58	631,252	142M	3.88	86M	4.33	454M

Table 2: Detailed information for FineCap-450M.

Training Stage	Stage I: MIM	Stage II: Contrastive Learning	Stage III: LLM Align
Samples Seen	1.8B	9.3B	0.5B
Batch Size	4,096	49,152	4,096
Resolution	$256\times 256$	within $448\times 448$	within $1000\times 1000$
Learning Rate	$1.0\times 10^{-3}$	$1.0\times 10^{-4}$	$1.0\times 10^{-5}$
Training Modules	ViT	ViT, text encoder	ViT, projector, LLM

Table 3: Overview of FineViT training details.

Model	Params	Zero-shot Classification			Zero-shot Retrieval
Model	Params	IN-1k val	IN v2	IN REAL	COCO T2I	COCO I2T	FLICKR T2I	FLICKR I2T
SigLIP-so400m/14 Zhai et al. (2023)	0.4B	83.2	77.1	87.5	52.0	70.2	80.5	93.5
SigLIP2-so400m/14 Tschannen et al. (2025)	0.4B	84.1	78.7	88.1	55.8	71.7	85.7	94.9
SigLIP2-g/16 Tschannen et al. (2025)	1B	85.0	79.8	88.5	56.1	72.8	86.0	95.4
Seed-ViT/14 Guo et al. (2025)	0.5B	83.6	77.6	-	-	-	-	-
FineViT/14	0.86B	84.2	75.5	88.7	60.7	80.7	84.8	96.7

Table 4: Comparison on Zero-shot Classification and Retrieval benchmarks.

Model	Params	Long-text Zero-shot Retrieval
Model	Params	DCI T2I	DCI I2T	IIW T2I	IIW I2T	Urban-1k T2I	Urban-1k I2T
SigLIP-so400m/14 Zhai et al. (2023)	0.4B	61.5	63.3	91.8	92.5	73.8	74.7
SigLIP2-so400m/14 Tschannen et al. (2025)	0.4B	66.8	66.9	95.6	94.8	75.6	78.1
LongCLIP-L/14 Zhang et al. (2024)	0.3B	63.8	56.2	96.1	94.1	84.0	81.1
FixCLIP-L/14 Wang et al. (2025a)	0.3B	74.2	72.0	98.2	97.9	96.3	93.7
FineViT/14	0.86B	84.8	83.4	99.7	99.8	99.1	98.9

Table 5: Comparison on Long text Zero-shot Retrieval benchmarks.

Figure 4: Illustration on long text zero-shot retrieval. Distinctive attributes within the long caption are highlighted in orange to help identify the correct image.

	Task	Benchmark	FineViT-VL	Qwen3-VL Bai et al. (2025a)	Intern3.5-VL Wang et al. (2025b)	Aquila-VL Gu et al. (2024)
General	General VQA	MMBench dev_en_v11	79.33	76.47	76.08	75.69
		MMStar	60.87	56.60	57.33	54.53
		HallusionBench	46.54	51.89	48.18	42.07
	Multimodal Reasoning	MMMU val	44.33	44.11	49.33	43.77
		MathVista mini	67.70	55.10	61.50	59.30
		MMVet	55.05	60.18	65.20	40.22

Table 6: Performance comparison across vision language models on various multimodal understanding benchmarks. Aquila-VL uses Qwen2.5 1.5B LLM, all others utilize Qwen3 1.7B LLM as language backbone. All results are evaluated using the standard VLMEvalKit Duan et al. (2024) framework and identical APIs.

	Task	Benchmark	LLM: Qwen3-1.7B		LLM: Qwen3-8B
	Task	Benchmark	FineViT	SigLIP2-naflex	FineViT	SigLIP2-naflex
General	General VQA	MMBench dev_en_v11	66.25	66.87	74.54	76.32
		MMStar	44.60	44.73	55.93	56.60
		HallusionBench	30.04	30.74	39.83	39.08
	Multimodal Reasoning	MMMU val	41.22	41.44	45.89	48.56
		MathVista mini	49.00	43.20	58.80	60.50

Table 7: Benchmarking multimodal performance of vision encoders and LLMs

	Task	Benchmark	Stage I: MIM	Stage II: CL	Stage III: Align
General	General VQA	MMBench dev_en_v11	62.15	67.18	69.97
		MMStar	43.73	46.00	49.20
		HallusionBench	27.48	32.45	34.13
	Multimodal Reasoning	MMMU val	37.44	40.33	41.44
		MathVista mini	44.60	48.20	52.70
		MMVet	27.80	36.65	35.83

Table 8: Multimodal performance of FineViT across different training stages.

Figure 5: Performance distribution: unfrozen vs frozen FineViT.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Rules	Data Source	Data Size
Stage I	image : short $\geqslant$ 224, remove duplicates	COCO [ 24 ] , CC3M [ 25 ] , CC12M [ 26 ] , YFCC15M [ 1 ] , LAION [ 3 ] , wukong [ 27 ] , datacomp [ 28 ] , COYO [ 29 ] , SBU [ 30 ] , Flickr30k [ 31 ]	1.8B images
Stage II	caption : remove repeated or null captions		1.56B image-text pairs
Stage III	image : short $\geqslant$ 448, ratio $\in$ [1/3, 3], Blurry, Dark, Saturation check region : bbox area $\geqslant$ $1\%$ , confidence scores $\geqslant$ 0.3, NMS with $\text{IoU}=0.7$ , label balance	previous stage data; additional OCR data from: FineVision [ 72 ] , LLaVA-OneVision [ 73 ] , Eagle 2 [ 40 ] , Infinity-MM [ 74 ] , LLaVA-OneVision-1.5-Instruct [ 38 ]	80M images 454M regions

Appendix

Table 9: Detailed information of the data at different FineViT training stages.

Figure 6: Resolution distribution of $1.56$ B images used at stage II.

Panel of Figure 8 — Figure 8: Token Length Distribution of of FineCap-450M.

Panel of Figure 9 — Figure 9: Relative bounding box area distribution of FineCap-450M.

Panel of Figure 10 — Figure 10: Bounding box aspect ratio distribution of FineCap-450M.

Panel of Figure 11 — Figure 11: Illustration of FineCap-450M. To prevent an excessive number of regions, we randomly sample a subset for better visualization.

Panel of Figure 12 — Figure 12: Illustration of FineCap-450M. To prevent an excessive number of regions, we randomly sampled a subset for visualization

Panel of Figure 13 — Figure 13: (a) Ratio of different type of data. (b)Multimodal data distribution.

Panel of Figure 14 — Figure 14: Long-text Zero-shot Retrieval. Distinctive attributes within the long caption are highlighted in orange to help identify the correct image.

Panel of Figure 15 — Figure 15: Long-text Zero-shot Retrieval. Distinctive attributes within the long caption are highlighted in orange to help identify the correct image.

Explore similar work

Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLM
Mar 9, 2026Junyuan Mao, Qiankun Li, Linghao Meng +5Large Vision-Language Models Fine-Grained Visual Perception
National University of Singapore · Nanyang Technological University
From Attenuation to Attention: Variational Information Flow Manipulation for Fine-Grained Visual Perception
Apr 14, 2026Jilong Zhu, Yang FengVision-Language Models Fine-Grained Visual Perception
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · University of Chinese Academy of Sciences, Beijing, China
LLaVA-UHD v4: What Makes Efficient Visual Encoding in MLLMs?
May 9, 2026Kechen Fang, Yihua Qin, Chongyi Wang +3Efficient VLM Inference Multimodal Large Language Models
Tsinghua University · ModelBest

Mar 9, 2026J/K move · Enter open · S save

Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLM

Junyuan Mao, Qiankun Li, Linghao Meng, Zhicheng He+4

National University of Singapore · Nanyang Technological University

Large Vision-Language Models Fine-Grained Visual Perception VLM Adaptation Visual Representation Learning Visual Token Compression

Open paper AskPDF arXiv

Recent advances in multimodal large language models largely rely on CLIP-based visual encoders, which emphasize global semantic alignment but struggle with fine-grained visual understanding. In contrast, DINOv3 provides strong pixel-level perception yet lacks coarse-grained semantic abstraction, leading to limited multi-granularity reasoning. To address this gap, we propose Granulon, a novel DINOv3-based MLLM with adaptive granularity augmentation. Granulon introduces a text-conditioned granularity Controller that dynamically adjusts the visual abstraction level according to the semantic scope of the textual input, and an Adaptive Token Aggregation module that performs granularity-guided pooling and relation-aware clustering to produce compact, semantically rich visual tokens. This design enables unified "pixel-to-fine-to-coarse" reasoning within a single forward pass. Extensive and interpretable experiments demonstrate that Granulon improves accuracy by ~30% and reduces hallucination by ~20%, outperforming all visual encoders under identical settings.

Abstract

Explore similar work

Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLM

From Attenuation to Attention: Variational Information Flow Manipulation for Fine-Grained Visual Perception

LLaVA-UHD v4: What Makes Efficient Visual Encoding in MLLMs?