LUX: A Lesion-Aware Graph-Conditioned Visual - Language Architecture for Explainable Endoscopic Captioning
Authors: Alexis Ivan Escamilla-Lopez, Gilberto Ochoa-Ruiz, Salvador Hinojosa, Sharib Ali
Organizations: School of Engineering and Sciences, Tecnologico de Monterrey, Monterrey, 64700, NL, Mexico · AI in Medicine and Surgery group, School of Computer Science, University of Leeds, Leeds, LS2 9JT, United Kingdom
The interpretation of endoscopic imagery in ulcerative colitis is complex and subjective, with variability in human assessment and subtle mucosal inflammation. Although deep learning has advanced automated analysis, most vision-language models rely on global visual embeddings that overlook the localized and relational nature of pathological evidence, limiting clinical reliability and interpretability. We introduce LUX (Lesion-aware Unified eXplainable captioning), a graph-conditioned vision-language architecture for explainable endoscopic image captioning. LUX constructs a lesion-centric scene graph from Grad-CAM and CBAM activation maps, representing pathological regions as nodes and encoding their spatial and clinical relationships. These graph embeddings are integrated into the cross-attention layers of a T5 decoder, enabling generated words to attend to specific lesion nodes rather than only to global image features. This provides direct alignment between linguistic content and pathological evidence, supporting token-level interpretability and relational reasoning. LUX outperforms strong baseline and state-of-the-art medical captioning models across BLEU, METEOR, ROUGE-L, and CIDEr, with particularly strong gains in CIDEr. It also reduces hallucinated clinical findings and improves lesion-level grounding through stronger correspondence between generated tokens and localized pathological regions.
Figures & tables
Figure 1: Comparison between conventional global visual-language captioning and the proposed lesion-aware graph-conditioned approach. (A) Conventional medical captioning compresses visual information into global embeddings, often producing weakly grounded descriptions and hallucinated findings. (B) In LUX , pathological regions are represented as lesion nodes within a relational graph that conditions decoder cross-attention, enabling token-level grounding and clinically traceable reasoning.
Figure 2: Illustration of the evidence-first reasoning pipeline in the proposed LUX framework. Raw endoscopic imagery is progressively transformed into lesion-level representations and structured through a scene–lesion graph that explicitly conditions language generation. This design emphasizes grounded multimodal reasoning rather than direct image-to-text prediction.
Figure 3: Evolution of medical vision–language captioning paradigms. Traditional approaches rely on global visual embeddings, whereas grounded methods associate generated descriptions with localized lesion evidence. Graph-conditioned architectures additionally capture spatial and semantic interactions among pathological findings. The proposed LUX framework unifies lesion-level grounding and graph-based relational reasoning through dual cross-attention, producing clinically interpretable captions supported by explicit pathological evidence.
MES
Clinical Description
Dominant Endoscopic Characteristics
0
Normal or inactive disease
Normal mucosa, intact vascular pattern, no friability, absence of bleeding
Table 1: Mayo Endoscopic Subscore (MES) definitions and dominant endoscopic characteristics used for ulcerative colitis severity assessment.
Figure 4: Conceptual comparison between image-level classification, global image captioning, and the proposed lesion-aware visual–language framework. Image-level classification produces a single severity label without visual or linguistic justification. Global image captioning generates descriptive text but lacks explicit lesion localization and relational context. In contrast, the proposed LUX framework constrains language generation through lesion-centric representations and graph-conditioned reasoning, ensuring that each generated term is grounded on explicit lesion evidence and inter-lesion relationships.
Figure 5: Detailed architecture of the CBAM–enhanced ResNet-50 visual encoder used in LUX . The left side illustrates the full ResNet-50 convolutional hierarchy ( conv1 through conv5_x ), including downsampling blocks and the final spatial feature map projection. On the right, the Convolutional Block Attention Module (CBAM) is decomposed into its two sequential components: (1) Channel Attention , which refines feature maps using global average pooling (GAP), global max pooling (GMP), and a shared MLP; and (2) Spatial Attention , which aggregates feature cues via a 7×7 convolution followed by a sigmoid mask. The resulting attention-refined feature map constitutes the visual embedding used for lesion–scene graph construction and T5 cross-attention decoding.
Figure 6: Scene–Lesion Graph decomposition and relational reasoning within the proposed LUX framework. (A) Attention-refined activation maps derived from the CBAM-enhanced visual encoder are transformed into lesion-centric node embeddings, where each node represents a localized pathological finding. (B) Lesion nodes are connected through spatial proximity and semantic similarity relations, forming a structured scene–lesion graph that models interactions among inflammatory regions prior to reasoning. Stacked GCN layers propagate contextual information across connected lesions, enabling the model to capture relational inflammatory patterns rather than isolated visual cues. (C) The resulting graph-aware embeddings are injected into the T5 decoder through dual cross-attention, allowing generated clinical tokens to remain explicitly grounded in lesion-level pathological evidence.
Figure 7: Component II: Scene–Lesion Graph Construction. Localized lesion evidence is transformed into a structured scene–lesion graph, where nodes represent pathological findings and edges encode spatial and clinical relations among lesions. This relational representation enables explicit lesion-centric reasoning prior to language generation.
Figure 8: Component III: Lesion-Aware Captioning. The scene–lesion graph conditions the T5 decoder through dual cross-attention, allowing generated tokens to attend jointly to visual embeddings and structured lesion relations, thereby grounding clinical captions in localized pathological evidence.
Figure 9: Lesion-aware captioning module (Component III). The T5 decoder receives two complementary information streams: (1) dense visual embeddings extracted from the CBAM–ResNet encoder, and (2) structured relational embeddings produced by the lesion–scene graph. Each decoder query attends jointly to both sources through dual cross-attention. A learnable gating parameter γt adaptively balances visual context and graph-derived lesion semantics at each decoding step, enabling the model to generate clinically grounded descriptions. The relational embedding aligns each generated word with the most relevant lesion node, supporting traceable and explainable caption generation.
Figure 10: Dataset methodology and integration strategy. UC-Caption is a captioned dataset providing paired endoscopic images, expert-authored clinical descriptions, and MES labels, enabling supervised learning of clinically grounded language. LIMUC is a non-captioned auxiliary dataset providing endoscopic images and MES labels. Since the public LIMUC dataset does not include lesion localization masks, a manually annotated subset of 400 images was created to supervise the localization branch during training, while the remaining images were used for encoder pretraining, lesion priors, and lesion-centric relational learning. Together, both datasets provide complementary supervision that couples linguistic grounding (UC-Caption) with large-scale lesion-aware visual evidence (LIMUC) within the proposed training pipeline.
Figure 11: Dataset statistics illustrating class imbalance and linguistic variability. (A) The LIMUC dataset exhibits predominance of remission and mild cases (MES 0–1), providing large-scale visual diversity. (B) The UC-Caption dataset shows a more balanced representation across moderate and severe classes, suitable for linguistic supervision. (C) Caption length distribution indicates clinically structured yet lexically rich descriptions, with a mean of ∼ 25 words per caption.
Figure 12: Clinical feature analysis of the UC-Caption dataset. (A) Mean feature severity demonstrates monotonic increase with MES, confirming clinical consistency. (B) Radar plot reveals progressive prevalence of bleeding, friability, and erythema in higher MES grades. (C) Spearman correlation analysis validates strong associations between MES scores and vascular/friability attributes, reflecting underlying disease pathology.
Category
Metric
Value Range
Purpose
Captioning accuracy
BLEU-1/2/3/4
[0,1]
N-gram precision at increasing sequence lengths
ROUGE-L
[0,1]
Longest common subsequence overlap between hypothesis and reference
METEOR
[0,1]
Synonym- and morphology-aware matching
CIDEr
[0,∼1.5]
Consensus similarity with reference captions (higher is better)
SPICE
[0,1]
Semantic correctness via scene-graph parsing
Clinical grounding
Factuality Score
[0,1]
Fraction of clinically valid statements
Table 2: Summary of evaluation metrics used for captioning quality, clinical grounding, lesion localization, and MES classification, including their valid value ranges and interpretative purpose.
Model
BLEU-4
METEOR
ROUGE-L
CIDEr
SPICE
ResNet+LSTM
0.21
0.18
0.29
0.47
0.11
ResNet+T5
0.28
0.22
0.34
0.66
0.15
CBAM-ResNet+T5
0.31
0.24
0.37
0.72
0.18
GCN(encoder)+T5
0.33
0.25
0.38
0.79
0.19
LUX (ours)
0.41
0.29
0.44
0.92
0.25
Table 3: Captioning performance on the UC-Caption test set (higher is better for all metrics).
Figure 13: Qualitative comparison between the proposed LUX framework and a conventional global-captioning baseline (ResNet+LSTM) across representative Mayo Endoscopic Subscore (MES) severity levels. Each example contrasts lesion-grounded caption generation against globally inferred descriptions lacking explicit pathological localization. The proposed framework produces clinically consistent descriptions aligned with localized inflammatory evidence and Grad-CAM activations, whereas the baseline frequently introduces hallucinated findings, incorrect severity estimation, or diffuse pathological interpretations unsupported by visible mucosal evidence. Highlighted terms indicate clinically relevant findings associated with vascular alterations, erosions, bleeding, friability, and ulceration.
Model
MES Acc.
BLEU-4
METEOR
ROUGE-L
CIDEr
SPICE
Halluc. (%)
Classification Models Adapted to Captioning
Takenaka et al. (2019) – CNN
78.1
0.08
0.06
0.17
0.11
0.03
32.4
Stidham et al. (2019) – DNN
79.4
0.10
0.07
0.18
0.15
0.04
29.8
Valencia et al. (2023) – ResNet50
77.8
0.12
0.08
0.20
0.18
0.05
27.6
Bossuyt et al. (2021) – Ensemble
80.2
0.13
0.09
0.22
0.21
0.06
25.1
Min et al. (2021) – UC-Transformer
81.5
0.15
0.10
0.24
0.27
0.07
21.3
Table 4: Unified evaluation of adapted classification models, captioning models, and foundation vision–language models on the UC dataset. Classification models were extended with a T5 caption decoder, captioning models were equipped with an MES classification head, and foundation VLMs were adapted to the UC captioning protocol using the same evaluation setting. LUX achieves the strongest joint performance across MES classification and caption generation, while substantially reducing hallucinated clinical findings through lesion-level grounding and graph-conditioned relational reasoning.
Configuration
BLEU-4
METEOR
CIDEr
SPICE
Halluc. (%)
Full LUX
0.41
0.29
0.92
0.25
5.3
w/o CBAM
0.36
0.26
0.81
0.21
9.8
w/o Lesion Graph
0.34
0.25
0.76
0.19
13.4
Encoder Graph Only
0.35
0.25
0.79
0.20
11.7
w/o Graph Alignment
0.37
0.27
0.84
0.22
10.9
w/o Pseudo Captions
0.38
0.27
0.86
0.23
9.1
Table 5: Architectural ablation study evaluating the contribution of major components within LUX . Higher is better for all metrics except hallucination rate.
Improved mucosal focus but incomplete pathology composition
GCN Encoder
Structured lesion context
Decoder partially unguided
Better semantic structure but inconsistent severity descriptions
Full LUX
Strong grounding + compositional reasoning
Slightly higher complexity
Most clinically faithful, severity-consistent, and explainable outputs
Table 6: Qualitative architectural progression across representative MES grades. Each component progressively improves pathological specificity, semantic compositionality, and hallucination suppression.
Figure 14: Qualitative comparison of generated captions across MES severity levels. Representative remission (MES 0), mild (MES 1), moderate (MES 2), and severe (MES 3) ulcerative colitis cases comparing baseline architectures against LUX . Baseline models frequently exhibit hallucinations, incomplete lesion descriptions, or severity misclassification, whereas LUX preserves stronger lesion grounding, semantic specificity, and severity-consistent clinical interpretation.
Model
Halluc. (%)
ResNet+T5
16.4
CBAM-ResNet+T5
12.8
GCN(encoder)+T5
10.7
LUX
5.3
Table 7: Clinical grounding and hallucination evaluation across progressively enhanced architectures. Lower hallucination rates indicate stronger consistency between generated descriptions and localized pathological evidence.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Model
Generated Caption (Typical Output Summary)
Clinical Omissions and Limitations
Classification Models Adapted to Captioning
Takenaka et al. (2019) – CNN
Short, templated statements describing generic inflammation with weak lesion specificity.
Highly generic phrasing; limited lesion vocabulary and poor compositionality. Frequent omission of key cues (erosions vs. ulcers, friability grading, vascular pattern qualifiers). Weak grounding and limited severity rationale.
Stidham et al. (2019) – DNN
Generic inflammatory description with minimal structured clinical detail.
Low clinical specificity and limited ability to localize/attribute findings. Severity statements often unsupported or ambiguous; tends to omit absence statements (e.g., no deep ulceration).
Valencia et al. (2023) – ResNet50
Brief description of abnormal mucosa and reduced vascular pattern, occasionally mentioning erythema.
Partial cue coverage but limited narrative structure. Typically omits friability/bleeding qualifiers and fails to link observations to MES reasoning.
Bossuyt et al. (2021) – Ensemble
More stable phrasing than single CNN/DNN, still largely symptom-centric.
Improved consistency but still lacks lesion-level granularity. Often merges distinct findings into broad terms (“inflamed mucosa”), limiting interpretability and MES traceability.
Min et al. (2021) – UC-Transformer
Mentions UC-related inflammatory cues more frequently than generic CNN baselines.
Better domain language, but captions remain shallow: incomplete lesion enumeration, limited absence statements, and weak evidence-to-score linkage.
Appendix
Table 8: Appendix qualitative analysis of all models reported in Table 4 . The table summarizes typical caption behavior and highlights clinically relevant omissions and limitations relative to the proposed LUX framework.
Lesion detection, symptom tracking, and visual explainability are central to real-world medical image analysis, yet current medical Vision-Language Models (VLMs) still lack mechanisms that translate their broad knowledge into clinically actionable outputs. To bridge this gap, we present MEDIC-AD, a clinically oriented VLM that strengthens these three capabilities through a stage-wise framework. First, learnable anomaly-aware tokens (<Ano>) encourage the model to focus on abnormal regions and build more discriminative lesion centered representations. Second, inter image difference tokens (<Diff>) explicitly encode temporal changes between studies, allowing the model to distinguish worsening, improvement, and stability in disease burden. Finally, a dedicated explainability stage trains the model to generate heatmaps that highlight lesion-related regions, offering clear visual evidence that is consistent with the model's reasoning. Through our staged design, MEDIC-AD steadily boosts performance across anomaly detection, symptom tracking, and anomaly segmentation, achieving state-of-the-art results compared with both closed source and medical-specialized baselines. Evaluations on real longitudinal clinical data collected from real hospital workflows further show that MEDIC-AD delivers stable predictions and clinically faithful explanations in practical patient-monitoring and decision-support workflows
Woohyeon Park, Jaeik Kim, Sunghwan Steve Cho +7
AIDAS Laboratory, Seoul National University · Samsung Changwon Hospital · Samsung Medical Center +1
Vision-language models remain underused in colonoscopy despite the rich expert descriptions recorded in routine reports. These reports document lesion appearance, size and location but summarise entire procedures rather than caption individual frames, leaving clinical findings only weakly linked to the corresponding images. Here we develop EndoCLIP, a colonoscopy vision-language foundation model trained on 125,756 lesion-level image-text pairs progressively recovered from 280,476 routine colonoscopy records. Across lesion-level image-text retrieval, structured report generation and six multi-centre clinical classification tasks, EndoCLIP outperforms general-purpose and biomedical vision-language encoders in both zero-shot and linear-probe settings. On benign-versus-malignant classification, its linear probe approaches the performance of expert readers in a blinded study involving 12 endoscopists. These results suggest that recovering finding-to-frame correspondence can transform routine documentation into scalable supervision, enabling clinical targets to be specified in language rather than separately annotated for each task.
Jia Yu, Yan Zhu, Yili He +12
Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China. · Shanghai Collaborative Innovation Center of Endoscopy, Shanghai, China. · Shanghai Institute for Advanced Study of Zhejiang University, Shanghai, China. +5
Multimodal Large Language Models (MLLMs) have demonstrated remarkable potential in medical image analysis. However, their application in gastrointestinal endoscopy is currently hindered by two critical limitations: the misalignment between general model reasoning and standardized clinical cognitive pathways, and the lack of causal association between visual features and diagnostic outcomes. In this paper, we propose a novel Clinical-Cognitive-Aligned (CogAlign) framework to address these challenges. First, we endow the model with rigorous clinical analytical capabilities by constructing the hierarchical clinical cognition dataset and employing Supervised Fine-Tuning (SFT). Unlike conventional approaches, this strategy internalizes the hierarchical diagnostic logic of experts, ranging from anatomical localization and morphological evaluation to microvascular analysis, directly into the model. Second, to eliminate visual bias, we provide a theoretical analysis demonstrating that standard supervised tuning inevitably converges to spurious background correlations. Guided by this insight, we propose a counterfactual-driven reinforcement learning strategy to enforce causal rectification. By generating counterfactual normal samples via lesion masking and optimizing through clinical-cognition-centric rewards, we constrain the model to strictly ground its diagnosis in causal lesion features. Extensive experiments demonstrate that our approach achieves State-of-the-Art (SoTA) performance across multiple benchmarks, significantly enhancing diagnostic accuracy in complex clinical scenarios.
Huan Zheng, Yucheng Zhou, Tianyi Yan +6
SKL-IOTSC, CIS, University of Macau · 2Shanghai Jiao Tong University · 3COWARobot Co. Ltd. +1