Despite remarkable advancements in multimodal large language models (MLLMs), their fine-grained visual understanding is constrained by a primary reliance on sparse textual supervision. Existing efforts to introduce visual supervision typically do so during post-training, when visual representations have already been largely fixed, causing such signals to act mainly as auxiliary constraints rather than as a primary force for shaping perceptual features. In this paper, we aim to fundamentally reshape the model's perceptual backbone by incorporating vision supervision directly into the pre-training stage. We observe that pixel-level image patches and textual tokens naturally coexist in a shared, raw high-dimensional space characterized by an inherent input symmetry. Leveraging this insight, we propose UVU, a novel vision-language unified autoregressive framework that eschews vector quantization. It uniquely employs continuous visual encoding for lossless representation of visual inputs and proposes a large-scale iterative hierarchical clustering algorithm to construct a pixel-level visual codebook, thereby extending the vocabulary for unified supervision and enabling autoregressive generation of pixel-level image tokens alongside textual tokens. UVU effectively synergizes pixel-level visual perception with semantic-level visual understanding, internalizing visual reconstruction capabilities and unlocking the facilitative role of visual supervision in enhancing understanding in the pre-training stage. Extensive experiments across multiple tasks demonstrate that MLLMs are capable of achieving superior multimodal understanding performance under the supervised learning paradigm of UVU.
Figures & tables
Figure 1: Comparison between traditional AR+VQ architectures ( left ), which suffer from visual input discretization, image-text loss gradient orthogonality, and external decoder reliance, and the proposed UVU framework ( right ), which employs continuous visual encoding and pixel-level visual codebooks to unify high-dimensional spaces, internalize visual reconstruction, and enhance multimodal understanding.
Figure 2: The angle between image and text loss gradients over training steps. The AR+VQ baseline (green) consistently exceeds 90°, indicating orthogonal or opposing directions that hinder multimodal alignment, whereas our UVU (blue) maintains angles below 90°, enabling synergistic optimization.
Figure 3: Illustration of the large-scale iterative hierarchical clustering algorithm for constructing a pixel-level visual codebook from image patches . Featuring initial K-means clustering, duplicate removal, uniform sampling, incremental supersampling, and iterative refinement with caching for high coverage and precision.
Figure 4: Illustration of the clustering visualization for four image token IDs. For each patch, the values on the right, from top to bottom, represent the image token ID and the distance to the cluster center, respectively.
Figure 5: Overview of the UVU vision-language unified autoregressive framework. Continuous visual features from SigLIP-2 are projected and integrated with textual embeddings for autoregressive next-token prediction in an LM decoder, generating pixel-level image tokens indexed via a clustered codebook for lossless reconstruction, merging text and image modalities.
Figure 6: Original images and their reconstructions by UVU. Each pair consists of the original image on the left and the reconstructed version on the right.
Figure 7: Comparison of visual attention heatmaps under three paradigms : from left to right, without visual supervision, AR + VQ, and UVU (Ours). The query is ”How many [objects] are in the image?” with the specific objects (giraffes or dogs) indicated on the left of each example.
School of Computer Science and Technology, Beijing Institute of Technology · 2Zhongguancun Academy · 5Southeast Academy of Information Technology, Beijing Institute of Technology +2