cs.CVMar 7, 2026

Deep Expert Injection for Anchoring Retinal VLMs with Domain-Specific Knowledge

Authors: Shuai Lu, Meng Wang, Jia Guo, Jiawei Du, Bo Liu, Shengzhu Yang, Weihang Zhang, Huazhu Fu, 

Organizations: Beijing Institute of Technology, Beijing, China · Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · National University of Singapore, Singapore · Tsinghua University, Beijing, China · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large Vision Language Models (LVLMs) show immense potential for automated ophthalmic diagnosis. However, their clinical deployment is severely hindered by lacking domain-specific knowledge. In this work, we identify two structural deficiencies hindering reliable medical reasoning: 1) the Perception Gap, where general-purpose visual encoders fail to resolve fine-grained pathological cues (e.g., microaneurysms); and 2) the Reasoning Gap, where sparse visual evidence is progressively overridden by massive language priors in deeper transformer layers, leading to ungrounded hallucinations. To bridge these gaps, we propose EyExIn, a data-efficient framework designed to anchor retinal VLMs with expert knowledge via a Deep Expert Injection mechanism. Our architecture employs an Expert-Aware Dual-Stream encoding strategy that decouples visual representation into a general stream for anatomical context and a specialized expert stream for pathological semantics. To ensure high-fidelity integration, we design a Semantic-Adaptive Gated Fusion module, which dynamically amplifies subtle lesion signals while filtering irrelevant background noise. Furthermore, we introduce Adaptive Deep Expert Injection to embed persistent "Vision Anchors" by integrating fused visual features as residual biases directly into intermediate LLM layers. This mechanism creates a visual shortcut that forces the reasoning stack to remain strictly grounded in visual evidence. Extensive experiments across four benchmarks demonstrate that our model consistently outperforms massive proprietary systems. EyExIn significantly enhances domain-specific knowledge embedding and achieves state-of-the-art precision in ophthalmic visual question answering, advancing the development of trustworthy ophthalmic AI.

Figures & tables

Figure 1: Architectural comparison between a standard VLM and the proposed EyExIn framework. (Top) Standard VLMs suffer from visual signal decay across deep LLM layers, leading to ungrounded reasoning. (Bottom) EyExIn introduces an expert-aware dual-stream encoder and Semantic-Adaptive Gated Fusion (GF). Crucially, Deep Expert Injection establishes persistent “Vision Anchors” in intermediate layers, ensuring evidence-grounded clinical outputs.
DatasetMethodClosed VQA (%)Open-ended VQA (%)
F1RecallPrecF1RecallPrec
TM4KQwen3-VL-Max (Prop.)7.829.1614.5811.3911.9732.58
ChatGPT-5.2 (Prop.)15.9321.5523.8326.7227.4444.20
Gemini3-Pro (Prop.)37.9948.1140.5347.7154.1753.25
LLaVA [ 15 ] (FT)39.8346.1344.8039.7143.0144.61
Qwen2.5-VL [ 2 ] (FT)55.7269.1251.2247.8456.0254.44
Table 1: Comparison of Macro Average metrics on multi-class Closed and Open-ended VQA tasks. Best results are highlighted in bold . Models are categorized as Proprietary (Prop.) general-domain LVLMs and Fine-Tuned (FT) open-source models.
MethodBLEU-1ROUGE-LMETEORBERT-F1
Qwen3-VL-Max0.36230.20820.32580.9352
ChatGPT-5.20.29420.20980.27260.9320
Gemini3-Pro0.25640.21300.30560.9286
LLaVA [ 15 ] (FT)0.38540.26880.36920.9387
Qwen2.5-VL [ 2 ] (FT)0.41230.30460.40760.9420
Our EyExIn (FT)0.45790.35520.43160.9466
Table 2: Comparison of text similarity metrics on the TM4K dataset .
Figure 2: Qualitative VQA comparison on two real-world clinical cases. Red text indicates severe missed diagnoses and hallucinated artifacts by Gemini, while green text highlights accurate, expert-aligned lesion identification and clinical metrics by EyExIn.
ModelExpertFusionInjectionF1RecallPrec
VariantStreamStrategyStrategy
Baseline (Qwen2.5-VL)---55.7269.1251.22
Variant 1✓Add-64.2381.3055.10
Variant 2✓Gated-71.4578.5067.65
Variant 3✓GatedDirect74.1282.1569.80
EyExIn✓GatedAdapt78.0782.4277.33
Table 3: Ablation study on the TM4K dataset (Closed VQA). We compare different strategies for integrating the expert stream: sequence addition (Add) vs. our semantic-adaptive gated fusion (Gated), and unconditional addition (Direct) vs. our adaptive routing (Adapt) for deep injection.
More in this topic

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 736 papers

Explore similar work

CardsList
Sep 26, 2026

EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding

Vision-language models (VLMs) have shown increasing potential for medical image understanding, yet their capabilities in ophthalmic imaging remain insufficiently characterized. Existing ophthalmic datasets are typically designed for individual diseases or specialized tasks, making it difficult to systematically evaluate whether VLMs can move beyond disease recognition toward comparative reasoning and fine-grained spatial grounding. We introduce EyeVQA, a unified visual question answering benchmark for comprehensive evaluation of ophthalmic VLMs. EyeVQA is constructed from 21 available ophthalmic datasets and contains 20,000 question-answer pairs spanning six disease groups and seven question types: Single-Choice, Multi-Select, Variable-Select, True-False, Ranking, Point Location, and Bounding Box. Gold answers are deterministically derived from source-provided diagnoses, severity grades, clinical findings, segmentation masks, bounding boxes, and anatomical landmarks, enabling reproducible evaluation without relying on model-generated annotations. Notably, 44.5% of the questions require reasoning across multiple images, extending evaluation beyond conventional single-image medical VQA. We benchmark fourteen representative general-purpose, scientific, and medically specialized VLMs under a unified zero-shot protocol. The best-performing model only achieves an overall score of 62.8, while substantial gaps remain in spatial grounding and cross-task generalization. These results highlight the limitations of current VLMs in comprehensive ophthalmic visual understanding and establish EyeVQA as a diagnostic benchmark for developing more reliable and spatially grounded ophthalmic multimodal models. The project page is available at https://github.com/PKUTHM/EyeVQA.

Gujie Shao, Zixun Xie, Xuechun Xing +7

VLM EvaluationMedical VQAVision-Language GroundingMedical Image BenchmarksOphthalmic Image AnalysisOphthalmology
AskPDF
May 21, 2026

Towards Clinically Interpretable Ophthalmic VQA via Spatially-Grounded Lesion Evidence

Visual Question Answering (VQA) holds great promise for clinical support, particularly in ophthalmology, where retinal fundus photography is essential for diagnosis. However, ophthalmic VQA benchmarks primarily emphasize answer accuracy, neglecting the explicit visual evidence necessary for clinical interpretability. In this work, we introduce FundusGround, a new benchmark for clinically interpretable ophthalmic VQA with spatially-grounded lesion evidence. Specifically, we propose a three-stage pipeline that collects 10,719 fundus images with 15,595 image-level meticulously annotated lesions. To ensure anatomical consistency and clinical validity, all lesions are spatially localized using the Early Treatment Diabetic Retinopathy Study (ETDRS) grid, enabling standardized mapping to nine clinically meaningful retinal regions. Built upon this structured lesion evidence, 72,706 questions are then generated spanning four formats: open-ended, closed-ended, single-choice, and multiple-choice. We further benchmark multiple general- and medical- large vision-language models using dual metrics for answer accuracy and lesion-level reasoning. The experiments demonstrate that incorporating lesion-level visual evidence consistently improves model performance and transparency, highlighting the necessity of explicit spatial grounding for reliable and explainable ophthalmic VQA.

Xingyue Wang, Bo Liu, Meng Wang +4

Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China · The Hong Kong Polytechnic University, Hong Kong S.A.R., China. · National University of Singapore, Singapore. +2

Medical VQAFundus ImagingMedical Image GroundingExplainable Medical Image AnalysisMedical Image BenchmarksOphthalmology
AskPDF
Aug 22, 2025

Bridging the Gap in Ophthalmic AI: MM-Retinal-Reason Dataset and OphthaReason Model toward Dynamic Multimodal Reasoning

Multimodal large language models (MLLMs) have recently demonstrated remarkable reasoning abilities under reinforcement learning (RL) paradigm. However, most existing multimodal medical reasoning models focus on basic reasoning, which refers to shallow inference based on visual feature matching. In contrast, real-world clinical diagnosis extends beyond basic reasoning, demanding complex reasoning that integrates heterogeneous clinical information (such as chief complaints and medical history) with multimodal medical imaging data. To bridge this gap, we introduce MM-Retinal-Reason, an ophthalmic multimodal dataset covering the full spectrum of perception and reasoning. Specifically, it is the first dataset in ophthalmology to encompass both basic and complex reasoning tasks with Chain-of-Thought (CoT) trajectories, aiming to enhance visual-centric reasoning and emulate realistic clinical decision-making. Building upon MM-Retinal-Reason, we propose OphthaReason, the first RL-enhanced ophthalmic multimodal reasoning model with step-by-step reasoning traces. To enable flexible adaptation to both basic and complex reasoning tasks, we further introduce Uncertainty-Aware Dynamic Thinking (UADT), which estimates sample-level uncertainty via entropy and dynamically modulates exploration depth through a shaped advantage mechanism. Comprehensive experiments demonstrate the effectiveness of our model on both basic and complex reasoning tasks, outperforming general-purpose MLLMs, medical MLLMs, RL-based medical MLLMs, and ophthalmic MLLMs by at least 15.47%. Project Page: link.

Ruiqi Wu, Yuang Yao, Na Su +11

School of Computer Science and Engineering, Southeast University, Nanjing, China · Department of Ophthalmology, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China · School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China

RL for VLMsMedical Image AnalysisMultimodal ReasoningOphthalmic Image AnalysisMedical VLMsMultimodal CoT Reasoning+1
AskPDF