Large Vision-Language Models (LVLMs) use their vision encoders to translate images into representations for downstream reasoning, but the encoders often underperform in domain-specific visual tasks such as medical image diagnosis or fine-grained classification, where representation errors can cascade through the language model, leading to incorrect responses. Existing adaptation methods modify the continuous feature interface between encoder and language model through projector tuning or other parameter-efficient updates, which still couples the two components and requires re-alignment whenever the encoder changes. We introduce CRAFT (Codebook RegulAted Fine-Tuning), a lightweight method that fine-tunes the encoder using a discrete codebook that anchors visual representations to a stable token space, achieving domain adaptation without modifying other parts of the model. This decoupled design allows the adapted encoder to seamlessly boost the performance of LVLMs with different language architectures, as long as they share the same codebook. Empirically, CRAFT achieves an average gain of 13.51% across 10 domain-specific benchmarks such as VQARAD and PlantVillage, while preserving the LLM's linguistic capabilities and outperforming peer methods that operate on continuous tokens.
Figures & tables
Figure 1: Continuous vs. Discrete Adaptation. (a) In conventional continuous-space adaptation, fine-tuning the vision encoder shifts its feature distribution, requiring costly re-alignment with each language model. (b) CRAFT introduces a discrete interface that anchors visual features to a shared codebook, allowing a single adapted encoder to work seamlessly across language models of different architectures without additional re-training or alignment.
Figure 2: Examples from plant pathology [ 37 ] , medical imaging [ 19 ] , and abstract diagram understanding [ 34 ] are shown using a general continuous LVLM [ 25 ] , its PEFT-tuned variant, and our CRAFT model built on the discrete LVLM [ 51 ] . General LVLM often lacks visual grounding or domain-specific knowledge in under-represented domains ( e.g. , misidentifying plant diseases). PEFT improves task accuracy such as question answering, but its language output collapses into rigid responses. In contrast, CRAFT captures domain-specific visual cues ( e.g. , identifying lesions in medical images) while keeping alignment stable through the shared discrete token interface, allowing the model to produce both accurate decisions and coherent explanations. Correct and incorrect answers or explanations are marked in green and red , respectively.
Figure 3: Overview of the CRAFT framework. CRAFT adapts to a target domain by fine-tuning only the vision encoder, and its outputs are quantized into a shared discrete codebook. Training is guided by commitment and contrastive losses, with an additional surrogate LLM providing multimodal supervision. At inference, the adapted encoder can be used with any LLM that shares the same codebook.
Figure 4: Illustration of the Token Pruning Process. Most of the white background patches are mapped to the same codebook entry (ID 11745), whereas semantically meaningful objects such as the Chihuahua are represented by rarer token IDs (ID-5825). Token IDs that appear too often in the training set usually repeat information already captured by others. To reduce this redundancy, we regularize them by pruning a subset at test-time. Tokens that appear frequently during training receive lower rarity weights and are pruned more aggressively than those with higher rarity weights.
Figure 5: Accuracy of CRAFT encoder with VILA-U-7B backbone on various datasets versus the keep ratio. Each curve represents a different dataset. The keep ratio is the ratio between target budget M and image token number N ; 1.0 indicates no pruning. Performance is consistently reliable when the keep ratio is above 0.6.
Figure 6: Additional examples following Figure 2 of the main paper. The general VLM makes incorrect predictions because errors from the vision encoder mislead the LLM: it misinterprets the flower’s shape, it hallucinates on the dog image when the encoder fails to extract reliable visual information, and it makes an incorrect medical judgement when the encoder does not have the necessary contextual understanding and instead provides misleading mass-like features. The fine-tuned baseline suffers from catastrophic forgetting of its instruction-following ability and cannot provide explanations even when explicitly asked. Our method delivers accurate predictions while also preserving more precise reasoning capability across these tasks.
Figure 7: Images with the first-level residual quantization indices overlaid, following the VILA-U paper. We observe that homogeneous regions with redundant visual information often produce many repeated tokens sharing the same first-level indices, for example, 13035, 3793, and 8015 in the aircraft image; 11501 and 14085 in the flower image; and 9247 and 14225 in the car image. This repetition suggests that these areas can be pruned to give a more concise visual representation.
Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.