Large Vision-Language Models (LVLMs) use their vision encoders to translate images into representations for downstream reasoning, but the encoders often underperform in domain-specific visual tasks such as medical image diagnosis or fine-grained classification, where representation errors can cascade through the language model, leading to incorrect responses. Existing adaptation methods modify the continuous feature interface between encoder and language model through projector tuning or other parameter-efficient updates, which still couples the two components and requires re-alignment whenever the encoder changes. We introduce CRAFT (Codebook RegulAted Fine-Tuning), a lightweight method that fine-tunes the encoder using a discrete codebook that anchors visual representations to a stable token space, achieving domain adaptation without modifying other parts of the model. This decoupled design allows the adapted encoder to seamlessly boost the performance of LVLMs with different language architectures, as long as they share the same codebook. Empirically, CRAFT achieves an average gain of 13.51% across 10 domain-specific benchmarks such as VQARAD and PlantVillage, while preserving the LLM's linguistic capabilities and outperforming peer methods that operate on continuous tokens.
Figures & tables
Figure 1: Continuous vs. Discrete Adaptation. (a) In conventional continuous-space adaptation, fine-tuning the vision encoder shifts its feature distribution, requiring costly re-alignment with each language model. (b) CRAFT introduces a discrete interface that anchors visual features to a shared codebook, allowing a single adapted encoder to work seamlessly across language models of different architectures without additional re-training or alignment.
Figure 2: Examples from plant pathology [ 37 ] , medical imaging [ 19 ] , and abstract diagram understanding [ 34 ] are shown using a general continuous LVLM [ 25 ] , its PEFT-tuned variant, and our CRAFT model built on the discrete LVLM [ 51 ] . General LVLM often lacks visual grounding or domain-specific knowledge in under-represented domains ( e.g. , misidentifying plant diseases). PEFT improves task accuracy such as question answering, but its language output collapses into rigid responses. In contrast, CRAFT captures domain-specific visual cues ( e.g. , identifying lesions in medical images) while keeping alignment stable through the shared discrete token interface, allowing the model to produce both accurate decisions and coherent explanations. Correct and incorrect answers or explanations are marked in green and red , respectively.
Figure 3: Overview of the CRAFT framework. CRAFT adapts to a target domain by fine-tuning only the vision encoder, and its outputs are quantized into a shared discrete codebook. Training is guided by commitment and contrastive losses, with an additional surrogate LLM providing multimodal supervision. At inference, the adapted encoder can be used with any LLM that shares the same codebook.
Figure 4: Illustration of the Token Pruning Process. Most of the white background patches are mapped to the same codebook entry (ID 11745), whereas semantically meaningful objects such as the Chihuahua are represented by rarer token IDs (ID-5825). Token IDs that appear too often in the training set usually repeat information already captured by others. To reduce this redundancy, we regularize them by pruning a subset at test-time. Tokens that appear frequently during training receive lower rarity weights and are pruned more aggressively than those with higher rarity weights.
Figure 5: Accuracy of CRAFT encoder with VILA-U-7B backbone on various datasets versus the keep ratio. Each curve represents a different dataset. The keep ratio is the ratio between target budget M and image token number N ; 1.0 indicates no pruning. Performance is consistently reliable when the keep ratio is above 0.6.
Figure 6: Additional examples following Figure 2 of the main paper. The general VLM makes incorrect predictions because errors from the vision encoder mislead the LLM: it misinterprets the flower’s shape, it hallucinates on the dog image when the encoder fails to extract reliable visual information, and it makes an incorrect medical judgement when the encoder does not have the necessary contextual understanding and instead provides misleading mass-like features. The fine-tuned baseline suffers from catastrophic forgetting of its instruction-following ability and cannot provide explanations even when explicitly asked. Our method delivers accurate predictions while also preserving more precise reasoning capability across these tasks.
Figure 7: Images with the first-level residual quantization indices overlaid, following the VILA-U paper. We observe that homogeneous regions with redundant visual information often produce many repeated tokens sharing the same first-level indices, for example, 13035, 3793, and 8015 in the aircraft image; 11501 and 14085 in the flower image; and 9247 and 14225 in the car image. This repetition suggests that these areas can be pruned to give a more concise visual representation.
Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.
Large Vision Language Models (VLMs) excel at general visual reasoning but experience significant performance degradation when deployed in novel domains that exhibit substantial distribution shifts from their pretraining data. Existing domain adaptation methods rely on finetuning standard VLM components; however, depending on which components are updated, these approaches either limit the model's ability to learn domain-specific representations or cause catastrophic forgetting of previously acquired capabilities. We introduce Vision Contextualized Probing (VisCoP), a parameter-efficient adaptation framework that augments the VLM vision encoder with a compact set of learnable visual probes. By learning domain-specific visual representations through these probes while requiring only minimal updates to pretrained model components, VisCoP effectively adapts to new domains without sacrificing existing knowledge. We evaluate VisCoP across three challenging adaptation settings: cross-view (exocentric to egocentric), cross-modal (RGB to depth), and cross-task (human understanding to robot control). Across all scenarios, VisCoP consistently outperforms existing domain adaptation strategies, achieving superior target-domain performance while preserving the pretrained VLM's capabilities on the source domain. These results demonstrate that lightweight visual probing provides an effective and robust solution for adapting VLMs under substantial distribution shifts. Code, models, and evaluation protocols are available at https://github.com/dominickrei/VisCoP.
Dominick Reilly, Manish Kumar Govind, Le Xue +1
University of North Carolina at Charlotte · Elorian AI
Conventional fine-tuning on domain-specific datasets can inadvertently alter a model's pretrained multimodal priors, leading to reduced generalization. To address this, we propose Chain-of-Adaptation (CoA), an adaptation framework designed to integrate domain knowledge while maintaining the model's inherent reasoning and perceptual capabilities. CoA introduces a structured reasoning format that enhances domain alignment without sacrificing general multimodal competence by reinforcement learning. Experiments on standard surgical benchmarks, under both in-distribution and out-of-distribution settings, demonstrate that CoA achieves higher accuracy, stronger generalization, and more stable behavior than supervised fine-tuning. Furthermore, ablation studies confirm that CoA effectively preserves the model's core visual-language abilities, providing a reliable pathway for domain specialization in VLMs.
The integration of Large Language Model (LLM) blocks with Vision Transformers (ViTs) holds significant promise for vision-only tasks by leveraging the rich semantic knowledge and reasoning capabilities of LLMs. However, a fundamental challenge lies in the inherent modality mismatch between the text-centric pre-training of LLMs and the vision-centric training of ViTs. Direct fusion often fails to fully exploit the LLM's potential and suffers from unstable finetuning. Consequently, prior works typically keep LLM blocks frozen while learning only the vision components. To address these challenges, we introduce Language-Adapted Vision Enhancer (LAVIE), a novel framework that bridges this modality gap through a synergistic pre-training strategy. LAVIE co-adapts a ViT backbone and an LLM fusion block by (1) employing Masked Auto-Encoding (MAE) to pre-train the ViT for richer visual representations, and (2) concurrently training Low-Rank Adaptation (LoRA) layers within the LLM block using the same MAE objective. This joint optimization guides the ViT to produce LLM-aligned features and the LLM to effectively interpret visual information. We demonstrate through extensive experiments that LAVIE significantly improves performance in various downstream vision tasks, offering an effective and efficient way to enhance visual understanding using frozen LLM knowledge. Code is available at https://github.com/selimkuzucu/LAVIE
Selim Kuzucu, Muhammad Ferjad Naeem, Anna Kukleva +2
Max Planck Institute for Informatics, SIC · Google