Chain-of-Adaptation: Surgical Vision-Language Adaptation with Reinforcement Learning

Jiajie Li; Chenhui Xu; Meihuan Liu; Jinjun Xiong

cs.CVMar 20, 2026

Chain-of-Adaptation: Surgical Vision-Language Adaptation with Reinforcement Learning

Authors: Jiajie Li, Chenhui Xu, Meihuan Liu, Jinjun Xiong

Organizations: University at Buffalo Buffalo, NY, USA

Abstract

Conventional fine-tuning on domain-specific datasets can inadvertently alter a model's pretrained multimodal priors, leading to reduced generalization. To address this, we propose Chain-of-Adaptation (CoA), an adaptation framework designed to integrate domain knowledge while maintaining the model's inherent reasoning and perceptual capabilities. CoA introduces a structured reasoning format that enhances domain alignment without sacrificing general multimodal competence by reinforcement learning. Experiments on standard surgical benchmarks, under both in-distribution and out-of-distribution settings, demonstrate that CoA achieves higher accuracy, stronger generalization, and more stable behavior than supervised fine-tuning. Furthermore, ablation studies confirm that CoA effectively preserves the model's core visual-language abilities, providing a reliable pathway for domain specialization in VLMs.

Figures & tables

Figure 1: Chain-of-Adaptation Overview. Left: The training pipeline of the CoA. It features a evidence-orientated cold start that enrich model’s domain-specific concepts, and a RL-based training that encourages compliance with the CoA reasoning format and accurate final answers. Right: CoA performs adaptation from general to specialized domains through a four-stage reasoning format. <general description> gives plain descriptions that language models excel at. <evidence> explicitly collects mined information and connects it with domain knowledge. <thought> and <answer> further deepen the reasoning and draw the final conclusion.

Method	Endovis2018				CholecT50
	Precision	Recall	F 1	F ${}_{1}^{\text{cls}}$	Precision	Recall	F 1	F ${}_{1}^{\text{cls}}$
Base (Qwen3-VL-8B-Instruct)	54.4	45.6	48.5	42.0	26.0	65.1	35.3	20.7
+ SFT	65.9	69.8	65.7	43.2	60.0	62.2	58.7	15.3
+ Cold Start	58.0	64.2	59.3	45.5	26.8	62.1	36.2	20.4
+ Cold Start + SFT	63.6	64.3	62.0	45.6	64.8	65.8	62.4	15.4
+ Cold Start + RLVR (CoA)	80.6	89.8	83.7	58.0	57.6	78.4	64.4	20.2

Table 1: In-distribution Evaluation on Endovis2018 and CholecT50 (%)

Panel of Figure 5 — Figure 5: SFT vs GRPO on CholecT50 and Endovis2018.

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 545 papers

Decoupling Vision and Language: Codebook Anchored Visual Adaptation
Feb 23, 2026Jason Wu, Tianchen Zhao, Chang Liu +7VLM Adaptation Large Vision-Language Models
AWS · University of California, Los Angeles
VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models
Oct 15, 2025Dominick Reilly, Manish Kumar Govind, Le Xue +1Domain Adaptation VLM Adaptation
University of North Carolina at Charlotte · Elorian AI
Can LLM-Generated Text Empower Surgical Vision-Language Pre-training?
Apr 20, 2026Chengan Che, Chao Wang, Jiayuan Huang +2Surgical Video Understanding Medical VLMs
Visual Understanding Research Group, Department of Informatics, King’s College London, UK

Feb 23, 2026J/K move · Enter open · S save

Decoupling Vision and Language: Codebook Anchored Visual Adaptation

Jason Wu, Tianchen Zhao, Chang Liu, Jiarui Cai+6

AWS · University of California, Los Angeles

VLM Adaptation Large Vision-Language Models

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) use their vision encoders to translate images into representations for downstream reasoning, but the encoders often underperform in domain-specific visual tasks such as medical image diagnosis or fine-grained classification, where representation errors can cascade through the language model, leading to incorrect responses. Existing adaptation methods modify the continuous feature interface between encoder and language model through projector tuning or other parameter-efficient updates, which still couples the two components and requires re-alignment whenever the encoder changes. We introduce CRAFT (Codebook RegulAted Fine-Tuning), a lightweight method that fine-tunes the encoder using a discrete codebook that anchors visual representations to a stable token space, achieving domain adaptation without modifying other parts of the model. This decoupled design allows the adapted encoder to seamlessly boost the performance of LVLMs with different language architectures, as long as they share the same codebook. Empirically, CRAFT achieves an average gain of 13.51% across 10 domain-specific benchmarks such as VQARAD and PlantVillage, while preserving the LLM's linguistic capabilities and outperforming peer methods that operate on continuous tokens.

Abstract

VLM Adaptation

Explore similar work

Decoupling Vision and Language: Codebook Anchored Visual Adaptation

VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models

Can LLM-Generated Text Empower Surgical Vision-Language Pre-training?