cs.CVAug 24, 2026
SaveLUX: A Lesion-Aware Graph-Conditioned Visual - Language Architecture for Explainable Endoscopic Captioning
Organizations: School of Engineering and Sciences, Tecnologico de Monterrey, Monterrey, 64700, NL, Mexico · AI in Medicine and Surgery group, School of Computer Science, University of Leeds, Leeds, LS2 9JT, United Kingdom
Abstract
The interpretation of endoscopic imagery in ulcerative colitis is complex and subjective, with variability in human assessment and subtle mucosal inflammation. Although deep learning has advanced automated analysis, most vision-language models rely on global visual embeddings that overlook the localized and relational nature of pathological evidence, limiting clinical reliability and interpretability. We introduce LUX (Lesion-aware Unified eXplainable captioning), a graph-conditioned vision-language architecture for explainable endoscopic image captioning. LUX constructs a lesion-centric scene graph from Grad-CAM and CBAM activation maps, representing pathological regions as nodes and encoding their spatial and clinical relationships. These graph embeddings are integrated into the cross-attention layers of a T5 decoder, enabling generated words to attend to specific lesion nodes rather than only to global image features. This provides direct alignment between linguistic content and pathological evidence, supporting token-level interpretability and relational reasoning. LUX outperforms strong baseline and state-of-the-art medical captioning models across BLEU, METEOR, ROUGE-L, and CIDEr, with particularly strong gains in CIDEr. It also reduces hallucinated clinical findings and improves lesion-level grounding through stronger correspondence between generated tokens and localized pathological regions.
Figures & tables
Figure 1: Comparison between conventional global visual-language captioning and the proposed lesion-aware graph-conditioned approach. (A) Conventional medical captioning compresses visual information into global embeddings, often producing weakly grounded descriptions and hallucinated findings. (B) In LUX , pathological regions are represented as lesion nodes within a relational graph that conditions decoder cross-attention, enabling token-level grounding and clinically traceable reasoning.
Figure 2: Illustration of the evidence-first reasoning pipeline in the proposed LUX framework. Raw endoscopic imagery is progressively transformed into lesion-level representations and structured through a scene–lesion graph that explicitly conditions language generation. This design emphasizes grounded multimodal reasoning rather than direct image-to-text prediction.
Figure 3: Evolution of medical vision–language captioning paradigms. Traditional approaches rely on global visual embeddings, whereas grounded methods associate generated descriptions with localized lesion evidence. Graph-conditioned architectures additionally capture spatial and semantic interactions among pathological findings. The proposed LUX framework unifies lesion-level grounding and graph-based relational reasoning through dual cross-attention, producing clinically interpretable captions supported by explicit pathological evidence.
| MES | Clinical Description | Dominant Endoscopic Characteristics |
| 0 | Normal or inactive disease | Normal mucosa, intact vascular pattern, no friability, absence of bleeding |
| 1 | Mild disease | Mild erythema, decreased vascular pattern visibility, minimal granularity, no spontaneous bleeding |
| 2 | Moderate disease | Marked erythema, absent vascular pattern, friability, erosions, contact bleeding |
| 3 | Severe disease | Spontaneous bleeding, ulceration, extensive friability, mucosal destruction |
Table 1: Mayo Endoscopic Subscore (MES) definitions and dominant endoscopic characteristics used for ulcerative colitis severity assessment.
Figure 4: Conceptual comparison between image-level classification, global image captioning, and the proposed lesion-aware visual–language framework. Image-level classification produces a single severity label without visual or linguistic justification. Global image captioning generates descriptive text but lacks explicit lesion localization and relational context. In contrast, the proposed LUX framework constrains language generation through lesion-centric representations and graph-conditioned reasoning, ensuring that each generated term is grounded on explicit lesion evidence and inter-lesion relationships.
Figure 5: Detailed architecture of the CBAM–enhanced ResNet-50 visual encoder used in LUX . The left side illustrates the full ResNet-50 convolutional hierarchy ( conv1 through conv5_x ), including downsampling blocks and the final spatial feature map projection. On the right, the Convolutional Block Attention Module (CBAM) is decomposed into its two sequential components: (1) Channel Attention , which refines feature maps using global average pooling (GAP), global max pooling (GMP), and a shared MLP; and (2) Spatial Attention , which aggregates feature cues via a convolution followed by a sigmoid mask. The resulting attention-refined feature map constitutes the visual embedding used for lesion–scene graph construction and T5 cross-attention decoding.
Figure 6: Scene–Lesion Graph decomposition and relational reasoning within the proposed LUX framework. (A) Attention-refined activation maps derived from the CBAM-enhanced visual encoder are transformed into lesion-centric node embeddings, where each node represents a localized pathological finding. (B) Lesion nodes are connected through spatial proximity and semantic similarity relations, forming a structured scene–lesion graph that models interactions among inflammatory regions prior to reasoning. Stacked GCN layers propagate contextual information across connected lesions, enabling the model to capture relational inflammatory patterns rather than isolated visual cues. (C) The resulting graph-aware embeddings are injected into the T5 decoder through dual cross-attention, allowing generated clinical tokens to remain explicitly grounded in lesion-level pathological evidence.
Figure 7: Component II: Scene–Lesion Graph Construction. Localized lesion evidence is transformed into a structured scene–lesion graph, where nodes represent pathological findings and edges encode spatial and clinical relations among lesions. This relational representation enables explicit lesion-centric reasoning prior to language generation.
Figure 8: Component III: Lesion-Aware Captioning. The scene–lesion graph conditions the T5 decoder through dual cross-attention, allowing generated tokens to attend jointly to visual embeddings and structured lesion relations, thereby grounding clinical captions in localized pathological evidence.
Figure 9: Lesion-aware captioning module (Component III). The T5 decoder receives two complementary information streams: (1) dense visual embeddings extracted from the CBAM–ResNet encoder, and (2) structured relational embeddings produced by the lesion–scene graph. Each decoder query attends jointly to both sources through dual cross-attention. A learnable gating parameter adaptively balances visual context and graph-derived lesion semantics at each decoding step, enabling the model to generate clinically grounded descriptions. The relational embedding aligns each generated word with the most relevant lesion node, supporting traceable and explainable caption generation.
Figure 10: Dataset methodology and integration strategy. UC-Caption is a captioned dataset providing paired endoscopic images, expert-authored clinical descriptions, and MES labels, enabling supervised learning of clinically grounded language. LIMUC is a non-captioned auxiliary dataset providing endoscopic images and MES labels. Since the public LIMUC dataset does not include lesion localization masks, a manually annotated subset of 400 images was created to supervise the localization branch during training, while the remaining images were used for encoder pretraining, lesion priors, and lesion-centric relational learning. Together, both datasets provide complementary supervision that couples linguistic grounding (UC-Caption) with large-scale lesion-aware visual evidence (LIMUC) within the proposed training pipeline.
Figure 11: Dataset statistics illustrating class imbalance and linguistic variability. (A) The LIMUC dataset exhibits predominance of remission and mild cases (MES 0–1), providing large-scale visual diversity. (B) The UC-Caption dataset shows a more balanced representation across moderate and severe classes, suitable for linguistic supervision. (C) Caption length distribution indicates clinically structured yet lexically rich descriptions, with a mean of 25 words per caption.
Figure 12: Clinical feature analysis of the UC-Caption dataset. (A) Mean feature severity demonstrates monotonic increase with MES, confirming clinical consistency. (B) Radar plot reveals progressive prevalence of bleeding, friability, and erythema in higher MES grades. (C) Spearman correlation analysis validates strong associations between MES scores and vascular/friability attributes, reflecting underlying disease pathology.
| Category | Metric | Value Range | Purpose |
| Captioning accuracy | BLEU-1/2/3/4 | N-gram precision at increasing sequence lengths | |
| ROUGE-L | Longest common subsequence overlap between hypothesis and reference | ||
| METEOR | Synonym- and morphology-aware matching | ||
| CIDEr | Consensus similarity with reference captions (higher is better) | ||
| SPICE | Semantic correctness via scene-graph parsing | ||
| Clinical grounding | Factuality Score | Fraction of clinically valid statements |
Table 2: Summary of evaluation metrics used for captioning quality, clinical grounding, lesion localization, and MES classification, including their valid value ranges and interpretative purpose.
| Model | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE |
| ResNet+LSTM | 0.21 | 0.18 | 0.29 | 0.47 | 0.11 |
| ResNet+T5 | 0.28 | 0.22 | 0.34 | 0.66 | 0.15 |
| CBAM-ResNet+T5 | 0.31 | 0.24 | 0.37 | 0.72 | 0.18 |
| GCN(encoder)+T5 | 0.33 | 0.25 | 0.38 | 0.79 | 0.19 |
| LUX (ours) | 0.41 | 0.29 | 0.44 | 0.92 | 0.25 |
Table 3: Captioning performance on the UC-Caption test set (higher is better for all metrics).
Figure 13: Qualitative comparison between the proposed LUX framework and a conventional global-captioning baseline (ResNet+LSTM) across representative Mayo Endoscopic Subscore (MES) severity levels. Each example contrasts lesion-grounded caption generation against globally inferred descriptions lacking explicit pathological localization. The proposed framework produces clinically consistent descriptions aligned with localized inflammatory evidence and Grad-CAM activations, whereas the baseline frequently introduces hallucinated findings, incorrect severity estimation, or diffuse pathological interpretations unsupported by visible mucosal evidence. Highlighted terms indicate clinically relevant findings associated with vascular alterations, erosions, bleeding, friability, and ulceration.
| Model | MES Acc. | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE | Halluc. (%) |
| Classification Models Adapted to Captioning | |||||||
| Takenaka et al. (2019) – CNN | 78.1 | 0.08 | 0.06 | 0.17 | 0.11 | 0.03 | 32.4 |
| Stidham et al. (2019) – DNN | 79.4 | 0.10 | 0.07 | 0.18 | 0.15 | 0.04 | 29.8 |
| Valencia et al. (2023) – ResNet50 | 77.8 | 0.12 | 0.08 | 0.20 | 0.18 | 0.05 | 27.6 |
| Bossuyt et al. (2021) – Ensemble | 80.2 | 0.13 | 0.09 | 0.22 | 0.21 | 0.06 | 25.1 |
| Min et al. (2021) – UC-Transformer | 81.5 | 0.15 | 0.10 | 0.24 | 0.27 | 0.07 | 21.3 |
Table 4: Unified evaluation of adapted classification models, captioning models, and foundation vision–language models on the UC dataset. Classification models were extended with a T5 caption decoder, captioning models were equipped with an MES classification head, and foundation VLMs were adapted to the UC captioning protocol using the same evaluation setting. LUX achieves the strongest joint performance across MES classification and caption generation, while substantially reducing hallucinated clinical findings through lesion-level grounding and graph-conditioned relational reasoning.
| Configuration | BLEU-4 | METEOR | CIDEr | SPICE | Halluc. (%) |
| Full LUX | 0.41 | 0.29 | 0.92 | 0.25 | 5.3 |
| w/o CBAM | 0.36 | 0.26 | 0.81 | 0.21 | 9.8 |
| w/o Lesion Graph | 0.34 | 0.25 | 0.76 | 0.19 | 13.4 |
| Encoder Graph Only | 0.35 | 0.25 | 0.79 | 0.20 | 11.7 |
| w/o Graph Alignment | 0.37 | 0.27 | 0.84 | 0.22 | 10.9 |
| w/o Pseudo Captions | 0.38 | 0.27 | 0.86 | 0.23 | 9.1 |
Table 5: Architectural ablation study evaluating the contribution of major components within LUX . Higher is better for all metrics except hallucination rate.
| Model Variant | Strengths | Weaknesses | Clinical Behavior |
| ResNet+T5 | Fluent language | Weak lesion specificity | Generic symptom descriptions, frequent under/over-calling |
| CBAM+T5 | Better localization | Limited relational reasoning | Improved mucosal focus but incomplete pathology composition |
| GCN Encoder | Structured lesion context | Decoder partially unguided | Better semantic structure but inconsistent severity descriptions |
| Full LUX | Strong grounding + compositional reasoning | Slightly higher complexity | Most clinically faithful, severity-consistent, and explainable outputs |
Table 6: Qualitative architectural progression across representative MES grades. Each component progressively improves pathological specificity, semantic compositionality, and hallucination suppression.
Figure 14: Qualitative comparison of generated captions across MES severity levels. Representative remission (MES 0), mild (MES 1), moderate (MES 2), and severe (MES 3) ulcerative colitis cases comparing baseline architectures against LUX . Baseline models frequently exhibit hallucinations, incomplete lesion descriptions, or severity misclassification, whereas LUX preserves stronger lesion grounding, semantic specificity, and severity-consistent clinical interpretation.
| Model | Halluc. (%) |
| ResNet+T5 | 16.4 |
| CBAM-ResNet+T5 | 12.8 |
| GCN(encoder)+T5 | 10.7 |
| LUX | 5.3 |
Table 7: Clinical grounding and hallucination evaluation across progressively enhanced architectures. Lower hallucination rates indicate stronger consistency between generated descriptions and localized pathological evidence.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Model | Generated Caption (Typical Output Summary) | Clinical Omissions and Limitations |
| Classification Models Adapted to Captioning | ||
| Takenaka et al. (2019) – CNN | Short, templated statements describing generic inflammation with weak lesion specificity. | Highly generic phrasing; limited lesion vocabulary and poor compositionality. Frequent omission of key cues (erosions vs. ulcers, friability grading, vascular pattern qualifiers). Weak grounding and limited severity rationale. |
| Stidham et al. (2019) – DNN | Generic inflammatory description with minimal structured clinical detail. | Low clinical specificity and limited ability to localize/attribute findings. Severity statements often unsupported or ambiguous; tends to omit absence statements (e.g., no deep ulceration). |
| Valencia et al. (2023) – ResNet50 | Brief description of abnormal mucosa and reduced vascular pattern, occasionally mentioning erythema. | Partial cue coverage but limited narrative structure. Typically omits friability/bleeding qualifiers and fails to link observations to MES reasoning. |
| Bossuyt et al. (2021) – Ensemble | More stable phrasing than single CNN/DNN, still largely symptom-centric. | Improved consistency but still lacks lesion-level granularity. Often merges distinct findings into broad terms (“inflamed mucosa”), limiting interpretability and MES traceability. |
| Min et al. (2021) – UC-Transformer | Mentions UC-related inflammatory cues more frequently than generic CNN baselines. | Better domain language, but captions remain shallow: incomplete lesion enumeration, limited absence statements, and weak evidence-to-score linkage. |
Table 8: Appendix qualitative analysis of all models reported in Table 4 . The table summarizes typical caption behavior and highlights clinically relevant omissions and limitations relative to the proposed LUX framework.
More in this topic
Image Captioning
Image captioning generates natural-language descriptions of visual content, with research focused on accuracy, coverage, grounding, and evaluation.
See all 96 papers