From Generalist to Specialist: A Context-Fusion Framework for Endoscopic Polyp Reporting with a Frozen VLM

Ruijie Yang; Yan Zhu; Peiyao Fu; Siyuan Li; Te Luo; Zhihua Wang; Quanlin Li; Pinghong Zhou

cs.AIAug 16, 2026

From Generalist to Specialist: A Context-Fusion Framework for Endoscopic Polyp Reporting with a Frozen VLM

Authors: Ruijie Yang, Yan Zhu, Peiyao Fu, Siyuan Li, Te Luo, Zhihua Wang, Quanlin Li, Pinghong Zhou,

Organizations: Zhejiang University, Hangzhou, China · Shanghai Institute for Advanced Study, Zhejiang University, Shanghai, China · Shanghai Key Laboratory of MICCAI, Shanghai, China · Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China ·

Abstract

Reliable endoscopic polyp reporting requires integrating quantitative lesion sizing, standardized Paris classification, and clinically meaningful morphological description within a single record. General-purpose vision-language models (VLMs) offer a unified interface for image understanding and report generation. Existing specialization strategies, however, typically rely on task-specific models or model-weight adaptation, leaving unresolved how to introduce reliable specialist knowledge while preserving both this unified interface and the VLM's pretrained capabilities. We introduce a context-fusion framework that specializes a frozen general-purpose VLM through both implicit instruction context and explicit transduction context without modifying its pretrained weights. Specifically, a self-supervised polyp encoder retrieves related image-report pairs as explicit, query-specific evidence, while learned continuous specialist tokens provide implicit instruction context shared across cases. Experiments were conducted on 2,056 expert-annotated public endoscopic images. We compared the framework with general-purpose VLMs, task-specific predictors, and weight-adaptation methods to assess specialist performance, unified reporting, and adaptation efficiency. Across numerical, categorical, and report-generation metrics, the proposed framework substantially improved direct frozen-VLM inference and achieved the strongest overall performance among the evaluated methods. It added trainable parameters equal to only 0.006% of the frozen VLM's parameter count. When the top-1 retrieved case carried the correct target category, our framework corrected 70.5% of the errors made by a weight-adaptation baseline. These findings support the context-fusion framework as a lightweight and effective strategy for specialist adaptation of a frozen VLM.

Figures & tables

Figure 1: Workflow and motivation for unified endoscopic polyp report generation. (A) Conventional image-to-report generation embeds quantitative, categorical, and descriptive findings within a natural clinical sentence. We reorganize the same clinical content as a structured multi-task JSON report: diameter and Paris type can be evaluated directly, while the complete output is converted into a fixed clinical sentence for report-generation evaluation. (B) Direct general-VLM prompting lacks specialist case context, whereas separate task-specific models fragment reporting across independent predictors. (C) The proposed framework combines explicit transduction context and implicit instruction context around a frozen VLM, enabling all three reporting tasks through one unified interface.

Figure 2: Context-fusion framework for a frozen VLM. (A) Offline, a fixed self-supervised polyp encoder maps the training images to L2-normalized CLS representations $e_{1},\ldots,e_{N}$ , which are stored with their reports in the case bank; online, the same encoder retrieves the top- $K$ image–report pairs for a query. (B) The retrieved pairs provide explicit transduction context, while learned continuous specialist tokens provide implicit instruction context within the multimodal prompt. (C) The composed context is supplied to the frozen VLM to generate one structured report.

Figure 3: Dataset characteristics of the structured polyp-reporting subset. Each image is paired with diameter, Paris type, and a morphology description. These fields are evaluated with numerical, categorical, and language-generation metrics, respectively. The distribution panel summarizes Paris-type and diameter-bin counts across the training, validation, and test splits.

Section	Method	Diameter			Paris type				Report generation
		MAE $\downarrow$	ACC $>$ 0.5 $\uparrow$	ACC $>$ 1.0 $\uparrow$	ACC $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	BLEU-4 $\uparrow$	ROUGE-L $\uparrow$	CIDEr $\uparrow$
Val	GPT-5.5	0.354 (0.304–0.404)	0.684 (0.621–0.748)	0.665 (0.602–0.728)	0.296 (0.238–0.359)	0.448 (0.219–0.505)	0.265 (0.203–0.333)	0.237 (0.163–0.313)	0.386 (0.368–0.404)	0.651 (0.638–0.665)	0.063 (0.050–0.077)
	Claude Sonnet 5	0.356 (0.304–0.408)	0.767 (0.704–0.821)	0.728 (0.670–0.786)	0.388 (0.320–0.451)	0.358 (0.181–0.392)	0.177 (0.142–0.226)	0.172 (0.113–0.231)	0.420 (0.401–0.439)	0.686 (0.673–0.700)	0.080 (0.064–0.097)
	Gemini 3.5 Flash	0.314 (0.269–0.364)	0.816 (0.757–0.864)	0.728 (0.670–0.787)	0.403 (0.335–0.466)	0.280 (0.238–0.320)	0.235 (0.192–0.275)	0.227 (0.183–0.267)	0.443 (0.422–0.464)	0.689 (0.674–0.704)	0.086 (0.073–0.102)
	Qwen3.6 Plus	0.374 (0.335–0.417)	0.748 (0.689–0.806)	0.670 (0.602–0.733)	0.325 (0.262–0.388)	0.198 (0.179–0.221)	0.138 (0.112–0.169)	0.121 (0.095–0.149)	0.389 (0.374–0.406)	0.660 (0.649–0.671)	0.060 (0.051–0.069)
	Frozen VLM	0.569 (0.509–0.628)	0.534 (0.471–0.602)	0.408 (0.345–0.476)	0.214 (0.160–0.267)	0.240 (0.175–0.301)	0.200 (0.135–0.258)	0.149 (0.106–0.187)	0.350 (0.328–0.374)	0.619 (0.601–0.638)	0.065 (0.051–0.080)

Table 1: Comparison of general-purpose VLMs and the proposed method on the validation and test sets, with bootstrap 95% confidence intervals. Bold and underlined values denote the best and second-best distinct point estimates within each split and metric, respectively.

Method	Output scope	Val MAE $\downarrow$	Val ACC $\uparrow$	Test MAE $\downarrow$	Test ACC $\uparrow$
EndoFinder retrieval baseline	Task-specific	0.321	0.655	0.311	0.638
Supervised ViT Paris classifier	Single-task	–	0.670	–	0.657
Supervised ViT diameter regressor	Single-task	0.279	–	0.306	–
Ours	Unified report	0.155	0.772	0.200	0.698

Table 2: Comparison with retrieval-based and single-task predictors on validation and test sets. Lower MAE is better for diameter; higher ACC is better for Paris type. Bold and underlined values denote the best and second-best distinct point estimates, respectively. A dash indicates that a single-task predictor does not produce the other task’s output.

Section	Method	Diameter			Paris type				Report generation
		MAE $\downarrow$	ACC $>$ 0.5 $\uparrow$	ACC $>$ 1.0 $\uparrow$	ACC $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	BLEU-4 $\uparrow$	ROUGE-L $\uparrow$	CIDEr $\uparrow$
Val	Frozen VLM	0.569 (0.509–0.628)	0.534 (0.471–0.602)	0.408 (0.345–0.476)	0.214 (0.160–0.267)	0.240 (0.175–0.301)	0.200 (0.135–0.258)	0.149 (0.106–0.187)	0.350 (0.328–0.374)	0.619 (0.601–0.638)	0.065 (0.051–0.080)
	Finetune	0.208 (0.169–0.250)	0.908 (0.869–0.942)	0.908 (0.864–0.947)	0.718 (0.655–0.777)	0.751 (0.534–0.818)	0.753 (0.529–0.817)	0.740 (0.514–0.797)	0.720 (0.687–0.752)	0.856 (0.839–0.875)	0.453 (0.409–0.501)
	LoRA	0.195 (0.162–0.234)	0.879 (0.830–0.922)	0.893 (0.850–0.932)	0.699 (0.631–0.762)	0.758 (0.535–0.829)	0.725 (0.499–0.792)	0.727 (0.503–0.787)	0.689 (0.658–0.721)	0.837 (0.819–0.855)	0.404 (0.361–0.448)
	MM-RAG	0.216 (0.172–0.266)	0.859 (0.811–0.903)	0.888 (0.845–0.927)	0.641 (0.573–0.704)	0.689 (0.570–0.747)	0.678 (0.537–0.752)	0.662 (0.525–0.718)	0.694 (0.659–0.729)	0.840 (0.821–0.860)	0.434 (0.383–0.484)
	Prompt tuning	0.244 (0.203–0.294)	0.864 (0.820–0.908)	0.869 (0.820–0.913)	0.641 (0.568–0.704)	0.505 (0.396–0.609)	0.463 (0.391–0.541)	0.476 (0.391–0.551)	0.668 (0.633–0.700)	0.826 (0.807–0.844)	0.387 (0.342–0.432)

Table 3: Main comparison on validation and test sets with bootstrap 95% confidence intervals. Bold and underlined values denote the best and second-best distinct point estimates within each split and metric, respectively.

Figure 4: Effect of explicit transduction context and implicit instruction context. Direct denotes frozen-VLM inference without either context; the remaining settings use implicit instruction context, explicit transduction context, or both. The three rows evaluate diameter estimation, Paris-type classification, and report generation after converting structured JSON outputs into the fixed clinical sentence template. For display in panel A, MAE is transformed to $1/(1+\mathrm{MAE})$ , so all bars use a higher-is-better orientation. The combined setting yielded the highest overall values across the three metric groups.

Section	Method	Diameter			Paris type				Report generation
		MAE $\downarrow$	ACC $>$ 0.5 $\uparrow$	ACC $>$ 1.0 $\uparrow$	ACC $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	BLEU-4 $\uparrow$	ROUGE-L $\uparrow$	CIDEr $\uparrow$
Val	MM-RAG with random cases	0.365	0.806	0.786	0.369	0.415	0.272	0.216	0.502	0.730	0.143
	Prompt tuning + random MM-RAG	0.320	0.840	0.811	0.597	0.576	0.384	0.411	0.608	0.786	0.272
	MM-RAG	0.216	0.859	0.888	0.641	0.689	0.678	0.662	0.694	0.840	0.434
	Prompt tuning	0.244	0.864	0.869	0.641	0.505	0.463	0.476	0.668	0.826	0.387
	Ours	0.155	0.898	0.917	0.772	0.844	0.756	0.788	0.765	0.879	0.528

Table 4: Comparison of context configurations with five multimodal context cases. Random MM-RAG denotes MM-RAG supplied with randomly sampled training image–report pairs instead of EndoFinder-retrieved cases; the number and modalities of the context cases remain unchanged. Bold and underlined values denote the best and second-best distinct point estimates within each split and metric, respectively.

Figure 5: Parameter efficiency on the test set. Paris-type accuracy is plotted against the number of trainable parameters. Direct inference and MM-RAG leave the backbone unchanged and use no trainable parameters; Prompt tuning and Ours optimize only 0.524 M specialist-token parameters. The proposed combination attains the highest Paris-type accuracy with substantially fewer trainable parameters than LoRA and visual–language merger fine-tuning.

Section	$K$	Evidence source	Diameter			Paris type				Report generation			Evidence coverage
			MAE $\downarrow$	ACC $>$ 0.5 $\uparrow$	ACC $>$ 1.0 $\uparrow$	ACC $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	BLEU-4 $\uparrow$	ROUGE-L $\uparrow$	CIDEr $\uparrow$	Paris hit@ $K$ $\uparrow$	Size hit@ $K$ $\uparrow$
Val	1	Random cases	0.311	0.816	0.801	0.553	0.571	0.329	0.346	0.586	0.777	0.233	0.252	0.558
		EndoFinder	0.206	0.850	0.879	0.723	0.869	0.724	0.769	0.717	0.850	0.441	0.655	0.825
		Oracle	0.168	0.908	0.879	0.816	0.934	0.798	0.846	0.782	0.881	0.551	1.000	0.990
	3	Random cases	0.318	0.820	0.825	0.597	0.580	0.372	0.398	0.601	0.784	0.258	0.578	0.859
		EndoFinder	0.163	0.874	0.908	0.738	0.817	0.728	0.758	0.751	0.870	0.513	0.845	0.932

Table 5: Comparison of report-generation performance across context sizes and evidence sources using internal-context multimodal RAG. At each

K

, random, EndoFinder, and oracle evidence are evaluated with the same number of retrieved cases. Bold and underlined values denote the best and second-best distinct point estimates within each split,

K

, and metric, respectively.

Split

D

P

R

Diameter

Paris type

Morphology description

MAE

\downarrow

ACC

>

0.5

\uparrow

ACC

>

1.0

\uparrow

ACC

\uparrow

Precision

\uparrow

Recall

\uparrow

\uparrow

BLEU-4

\uparrow

ROUGE-L

\uparrow

CIDEr

\uparrow

Val

\checkmark

0.199

0.874

0.888

–

\checkmark

–

0.723

0.808

0.735

0.757

–

\checkmark

–

0.446

0.677

0.380

\checkmark

\checkmark

\checkmark

0.155

0.898

0.917

0.772

0.844

0.756

0.788

0.672

0.826

0.623

Test

\checkmark

0.225

0.903

0.888

–

Table 6: Single-task and complete-report generation with the proposed method. Check marks indicate the fields requested in each output schema. A dash denotes a metric that is not applicable because the corresponding field was not generated. For

R

and

DPR

, language metrics are computed from the same isolated morphology-description clause; diameter and Paris-type text is excluded from both predictions and references. Bold values indicate the better result between the single-task and complete-report settings for each split and metric.

Figure 6: Representative context-attribution cases. Each case column contains the query, five retrieved image–report pairs, and predictions from Direct, Prompt tuning, MM-RAG, and Ours. Token-conditioned overlays visualize the attribution of the generated Diameter(cm) , ParisType , and Description values to the query and evidence images; the six image maps within a case share one normalization scale. In the reports, blue denotes content that agrees with the reference, red denotes incorrect content, and orange denotes spurious content not supported by the reference report.

Figure 7: Representative case-study comparison on the test set. Each column shows one query case, and the rows compare target reports with direct outputs from general VLMs and the local frozen VLM, followed by the proposed context-fusion setting. Structured JSON predictions are converted into fixed clinical sentences for visual comparison. In model-generated reports, blue denotes content that agrees with the reference, red denotes incorrect content, and orange denotes spurious content not supported by the reference report. In these examples, fluent text can contain clinically important field errors, whereas the proposed setting corrects size or Paris-type content while retaining morphology descriptions.

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 545 papers

Aug 5, 2026

EndoVLM: An Endoscopy Vision-Language Pre-training Model via Anatomy-Guided Sparsity and Progressive Alignment

The development of foundation models (FMs) is crucial for advancing endoscopic image analysis. However, existing endoscopy FMs mainly rely on self-supervised learning from uni-modal images or videos, overlooking the rich semantic knowledge contained in clinical reports. Furthermore, effectively leveraging these records is hindered by a fundamental modality gap: structured anatomical descriptions are not naturally mapped to specific frames within the high-redundancy, uncurated visual streams. In this paper, we present EndoVLM, a novel vision-language FM pre-trained on over 348K endoscopic examinations, each pairing a clinical report with its corresponding image collection. An Anatomy-Guided Sparse Pooling mechanism utilizes textual descriptions as queries to drive sparse attention, efficiently aggregating semantically salient frames into anatomy-specific visual representations across redundant image-sets. Next, a Progressive Semantic-Aware Alignment strategy models clinical taxonomy (anatomy and pathological status) via structured soft targets, bridging the gap from global patient-level matching to fine-grained localized alignment. Finally, a Semantic-Concentrated Masked Autoencoder is applied exclusively to these semantic-rich frames, integrating low-level visual precision with robust high-level semantic representation. Extensive experiments across various downstream tasks demonstrate that EndoVLM outperforms existing foundation models and remains competitive with task-specific methods. Remarkably, EndoVLM also exhibits robust zero-shot generalization capabilities, highlighting its potential for broader clinical application.

Zhenyu Yi, Jianwei Xu, Yue Hu +6

1DAMO Academy, Alibaba Group · 3Shanghai Jiao Tong University · 4Hupan Lab +3

Medical Imaging Foundation Models Medical VLMs Vision-Language Alignment Medical Image Representation Learning

Ask PDF

Jul 30, 2026

A report-grounded vision-language foundation model for colonoscopy from 280000 routine reports

Vision-language models remain underused in colonoscopy despite the rich expert descriptions recorded in routine reports. These reports document lesion appearance, size and location but summarise entire procedures rather than caption individual frames, leaving clinical findings only weakly linked to the corresponding images. Here we develop EndoCLIP, a colonoscopy vision-language foundation model trained on 125,756 lesion-level image-text pairs progressively recovered from 280,476 routine colonoscopy records. Across lesion-level image-text retrieval, structured report generation and six multi-centre clinical classification tasks, EndoCLIP outperforms general-purpose and biomedical vision-language encoders in both zero-shot and linear-probe settings. On benign-versus-malignant classification, its linear probe approaches the performance of expert readers in a blinded study involving 12 endoscopists. These results suggest that recovering finding-to-frame correspondence can transform routine documentation into scalable supervision, enabling clinical targets to be specified in language rather than separately annotated for each task.

Jia Yu, Yan Zhu, Yili He +12

Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China. · Shanghai Collaborative Innovation Center of Endoscopy, Shanghai, China. · Shanghai Institute for Advanced Study of Zhejiang University, Shanghai, China. +5

Endoscopy Medical VLMs Medical Image Representation Learning Vision-Language Pretraining

Ask PDF

Jul 29, 2026

Towards Grounded GI Endoscopy VQA via Multi-Task Learning on Small VLMs

Gastrointestinal (GI) endoscopic image analysis has shifted from single-label classification toward visual question answering (VQA), where a model must answer free-form clinical questions about an image. While recent vision-language models (VLMs) achieve promising answer accuracy on this task, clinical adoption also requires the model's internal representations to reflect the visual evidence behind its answers. We propose a simple multi-task fine-tuning recipe that constructs auxiliary grounding and description tasks from an existing VQA dataset with minimal additional annotation: expert-annotated polyp masks are reused directly, while a GI-domain pretrained classifier with Grad-CAM localization provides weak supervision for finding categories that lack ground-truth masks. Three small VLM backbones are fine-tuned with low-rank adaptation under matched VQA-only and multi-task recipes on Kvasir-VQA-x1, and we show consistent accuracy gains together with improved implicit alignment between answer tokens and the relevant image region, evaluated on both in-distribution and out-of-distribution data.

Itbaan Safwan, Ramail Khan, Muhammad Annas Shaikh +1

Institute of Business and Administration, Karachi, Pakistan

Endoscopy Medical VQA Multi-Task Learning Medical VLMs Medical Image Grounding

Ask PDF

Abstract

VLM Adaptation

Explore similar work

EndoVLM: An Endoscopy Vision-Language Pre-training Model via Anatomy-Guided Sparsity and Progressive Alignment

A report-grounded vision-language foundation model for colonoscopy from 280000 routine reports

Towards Grounded GI Endoscopy VQA via Multi-Task Learning on Small VLMs