VIVAS: Vitalizing Visual Perception in VLM Pre-training via Vision-language Unified Autoregressive Supervision

Zhehan Kan; Yubo Zhu; Xinghua Jiang; Zhixiang Wei; Shifeng Liu; Wei Tong; Sheng Zhong; Qingmin Liao

cs.CVAug 25, 2026

VIVAS: Vitalizing Visual Perception in VLM Pre-training via Vision-language Unified Autoregressive Supervision

Authors: Zhehan Kan, Yubo Zhu, Xinghua Jiang, Zhixiang Wei, Shifeng Liu, Wei Tong, Sheng Zhong, Qingmin Liao,

Organizations: Tsinghua University · Tencent Youtu Lab · Nanjing University

Abstract

While Vision-Language Models (VLMs) demonstrate strong capabilities, they continue to suffer from a critical limitation: insufficient fine-grained visual perception, which fundamentally limits their multimodal understanding. We attribute this bottleneck to text-dominant optimization biases during pre-training, which encourage the model to overlook fine-grained visual details, thereby limiting the capability of multimodal understanding. We investigate that overcoming this bottleneck requires two key elements: (1) a unified token space paradigm that ensures stable training dynamics, and (2) a modality-aligned dense visual supervision signal enriched with both structural granularity and semantic information to capture critical visual representations. Based on these insights, we propose VIVAS, a framework built upon the unified token space paradigm, which introduces a dense-structural-semantic vision tokenizer, which expands the textual vocabulary into a unified vision-language vocabulary by incorporating a visual vocabulary. During pretraining, VIVAS performs vision-language unified autoregressive supervision over both visual details and linguistic content, thereby enhancing visual perception to improve multimodal understanding. Trained end-to-end on 12.4T tokens, VIVAS achieves state-of-the-art performance across 7 tasks and 39 multimodal benchmarks.

Figures & tables

Figure 1: Left: Principal Component Analysis (PCA) visualizations of hidden states extracted from the output-layer vision tokens, VIVAS demonstrates significantly more fine-grained visual perception. Right: Comprehensive multimodal understanding evaluation of VIVAS against the SOTA Qwen3-VL (4B) and an ablated baseline without visual supervision. VIVAS yields substantial performance improvements.

Figure 2: Left : Previous text-dominant VLMs lack direct visual supervision. Middle : To introduce visual supervision, VIVAS formulates a unified token space, fundamentally avoiding the instability inherent in architectural separation approaches. Right : Within this unified space, we introduce the dense-structural-semantic visual supervision signal, starkly outperforming detrimental pixel-level or ineffective semantic-level signals.

Figure 3: Comparison of training loss between architectural separation and unified token space. Vision and text losses correspond to the supervised training objectives for the visual and textual modalities, respectively.

Methods	VisuLogic	$MMBench_{EN}$	RealWorldQA	MME	$CRPE_{EXIST}$	$CRPE_{RELATION}$	$CVBench_{2D}$	HallusionBench	TextVQA	DocVQA	RefCOCO ${}_{\text{val}}$	RefCOCOg ${}_{\text{val}}$
w/o Visual Supervision	22.2	53.0	43.0	48.2	65.7	57.0	51.0	55.3	52.7	60.2	62.1	59.4
Text-decoupled vocabularies	21.6	51.2	41.8	46.9	63.5	55.1	49.3	53.9	51.4	58.6	60.7	57.8
Vision-language semantic vocabularies	23.6	53.6	43.2	47.7	66.2	57.6	51.4	54.8	53.1	59.3	63.0	60.7
Ours	24.4	54.9	46.5	49.8	72.2	59.0	56.3	56.5	54.2	63.4	65.2	63.1

Table 1: Performance comparison of various visual supervision strategies. All models are pre-trained on a 0.5T-token corpus and evaluated under a 5-shot setting across diverse multimodal benchmarks.

Figure 4: Overview of VIVAS. Left : The dense structural–semantic vision tokenizer constructs a visual codebook by fusing semantic and geometric features through cross-attention. Right : During end-to-end pretraining, VIVAS performs unified autoregressive supervision over both the textual vocabulary and the vision vocabulary.

Pre-train VS	Post-train VS	VisuLogic	MMBench EN	CRPE ${}_{\text{relation}}$	HallusionBench
$\times$	$\times$	23.1	83.2	69.5	57.3
✓	$\times$	25.7	83.9	72.2	59.1
$\times$	✓	23.3	82.9	69.9	57.7
✓	✓	23.6	83.3	70.1	58.1

Table 2: Ablation study of applying visual supervision at different training stages.

Benchmarks	LLaVA-OV	Qwen3-VL	LLaVA-OV	InternVL-3.5	Qwen3-VL	VIVAS w/o	VIVAS
Benchmarks	1.5	instruct	1.5		instruct	Vision Sup.	(Ours)
Params	8B	8B	4B	4B	4B	4B	4B
General VQA
MMBench CN [ 24 ]	81.0	84.7	76.9	–	83.5	82.7	83.6
MMBench EN [ 24 ]	84.1	84.5	84.2	80.3	83.9	83.2	83.9
MMStar [ 3 ]	67.7	70.9	64.9	65.0	69.8	70.4	71.1

Table 3: Performance comparison between VIVAS and SOTA VLMs across multiple benchmarks. All scores adhere to official evaluation protocols. The results reproduced via our internal evaluation system are marked with ‘*’.

Codebook Size	VisuLogic	MMBench EN	CRPE ${}_{\text{relation}}$	HallusionBench	Utilization (%)
50k	23.6	53.5	57.3	55.6	98.3
100k	24.0	54.1	58.1	56.1	98.6
150k	24.4	54.9	59.0	56.5	97.7
200k	24.2	54.5	58.7	56.3	96.6

Table 4: Ablation study on the codebook size of the visual tokenizer.

Codebook Dim	VisuLogic	MMBench EN	CRPE ${}_{\text{relation}}$	HallusionBench	Utilization (%)
64	22.7	53.1	57.4	55.5	95.1
256	23.6	53.7	58.6	55.9	97.8
768	24.4	54.9	59.0	56.5	97.7
1024	23.9	54.6	58.6	56.4	94.3

Table 5: Ablation study on the codebook embedding dimension of the visual tokenizer.

Benchmark	Qwen3-VL-4B	VIVAS w/o VS	VIVAS	Gain
GEOBench-VLM [ 5 ]	40.21	44.64	46.37	+1.73
VLMsAreBiased [ 43 ]	21.31	18.33	22.16	+3.83
V* [ 47 ]	80.63	78.53	81.15	+2.62

Table 6: Out-of-distribution and fine-grained perception evaluation.

L1 Loss	VisuLogic	MMBench EN	CRPE ${}_{\text{relation}}$	HallusionBench
$\times$	24.4	54.9	59.0	56.5
✓	23.7	54.2	57.9	56.1

Table 7: Ablation study on adding L1 loss during codebook training.

Method	VisuLogic	CVBench2D	MME	RefCOCO ${}_{\text{val}}$	DocVQA	HallusionBench
VIVAS w/o VS	22.2	51.0	48.2	62.1	60.2	55.3
w/ combined visual encoder	22.6	50.7	48.5	60.9	60.5	54.6
VIVAS	24.4	56.3	49.8	65.2	63.4	56.5

Table 8: Comparison with a combined visual encoder baseline. The combined visual encoder baseline uses SigLIP2 and DINOv3 features with a learnable projector, but does not use visual-token supervision.

Panel of Figure 5 — Figure 5: Scaling analysis of VIVAS. (a) The evolution of average performance across 27 multimodal benchmarks as a function of training tokens. The results reveal steady and consistent capability improvements during training across different data regimes, indicating that the model effectively exploits large-scale data without showing signs of empirical saturation. (b) Examination of the mean evaluation error ( $1-\text{Score}$ ). The optimization dynamics in both Stage 2 and Stage 3 follow neural scaling laws, with scaling exponents estimated at $\alpha\approx 0.102$ and $\alpha\approx 0.079$ , respectively.

Vision-Language Pretraining

Vision-language pretraining learns visual representations by relating images, video, or other visual inputs to language.

See all 79 papers

UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm
Aug 25, 2026Zhehan Kan, Xinghua Jiang, Yubo Zhu +10Vision-Language Models Unified Multimodal Models
Tsinghua University · Tencent Youtu Lab · Nanjing University +1
DV-SFT: Direct Vision Supervision for Fine-Grained Visual Understanding
May 26, 2026Jianfei Zhao, Feng Zhang, Xin Sun +3Supervised Fine-Tuning Vision-Language Models
School of Computer Science and Technology, Beijing Institute of Technology · 2Zhongguancun Academy · 5Southeast Academy of Information Technology, Beijing Institute of Technology +2
VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors
Apr 2, 2026Haz Sameen Shahgir, Xiaofu Chen, Yu Fu +4VLM Reasoning Fine-Grained Visual Perception
University of California, Riverside 1 · MBZUAI2

Aug 25, 2026J/K move · Enter open · S save

UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm

Zhehan Kan, Xinghua Jiang, Yubo Zhu, Yanlin Liu+9

Tsinghua University · Tencent Youtu Lab · Nanjing University+1

Vision-Language Models Unified Multimodal Models Multimodal Large Language Models Multimodal Understanding Multimodal Pretraining Visual Tokenization

Open paper AskPDF arXiv

Despite remarkable advancements in multimodal large language models (MLLMs), their fine-grained visual understanding is constrained by a primary reliance on sparse textual supervision. Existing efforts to introduce visual supervision typically do so during post-training, when visual representations have already been largely fixed, causing such signals to act mainly as auxiliary constraints rather than as a primary force for shaping perceptual features. In this paper, we aim to fundamentally reshape the model's perceptual backbone by incorporating vision supervision directly into the pre-training stage. We observe that pixel-level image patches and textual tokens naturally coexist in a shared, raw high-dimensional space characterized by an inherent input symmetry. Leveraging this insight, we propose UVU, a novel vision-language unified autoregressive framework that eschews vector quantization. It uniquely employs continuous visual encoding for lossless representation of visual inputs and proposes a large-scale iterative hierarchical clustering algorithm to construct a pixel-level visual codebook, thereby extending the vocabulary for unified supervision and enabling autoregressive generation of pixel-level image tokens alongside textual tokens. UVU effectively synergizes pixel-level visual perception with semantic-level visual understanding, internalizing visual reconstruction capabilities and unlocking the facilitative role of visual supervision in enhancing understanding in the pre-training stage. Extensive experiments across multiple tasks demonstrate that MLLMs are capable of achieving superior multimodal understanding performance under the supervised learning paradigm of UVU.

Abstract

Vision-Language Pretraining

Explore similar work

UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm

DV-SFT: Direct Vision Supervision for Fine-Grained Visual Understanding

VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors