MultiAttenGastro: Multi-Dimensional Attention Augmentation for Gastrointestinal Endoscopy Classification

Sadhana Devarajan; Praveen Kumar Chandaliya; Dhruvin Jashvant Kumar Shah; Kishor Upla; Kiran Raja

cs.CVSep 4, 2026

MultiAttenGastro: Multi-Dimensional Attention Augmentation for Gastrointestinal Endoscopy Classification

Authors: Sadhana Devarajan, Praveen Kumar Chandaliya, Dhruvin Jashvant Kumar Shah, Kishor Upla, Kiran Raja

Organizations: Sardar Vallabhbhai National Institute of Technology, Surat, Gujarat, India · Swasthyam Gastro and Liver Hospital, Surat, Gujarat, India · Norwegian University of Science and Technology, Norway

Abstract

Automated gastrointestinal (GI) endoscopy classification requires models that generalize across diverse modalities and class distributions, often far from natural-image pretraining. We propose MultiAttenGastro, a plug-and-play attention framework with parallel 1-D channel, 2-D spatial, and 3-D contextual heads, and present the first systematic cross-dataset evaluation across eight CNN and transformer backbones on five public GI datasets (80 backbone--dataset runs). We find that attention effectiveness is not universal but tracks the representational gap between ImageNet features and the target distribution: MultiAttenGastro improves 6 of 8 backbones on Kvasir-Capsule (14-class WCE, large gap; best macro F1 98.33%), is uniformly negative on the small-gap Kvasir-v2 benchmark (0/8), and shows mixed outcomes on datasets with intermediate gap. Five-seed ablation on the strongest case (Kvasir-Capsule, ConvNeXt-Tiny) shows this improvement is directionally consistent, but not statistically decisive (paired $t$ : $p=0.47$ ; Wilcoxon: $p=0.63$ ), and that individual attention heads are not uniformly beneficial in isolation only their combination yields a positive mean effect. Centered Kernel Alignment (CKA) analysis links this pattern to representational redundancy: low inter-head CKA under large domain gaps coincides with the framework's only consistent gains, while high redundancy under small gaps coincides with its losses. We report these results, including the non-significant margins, as evidence for when and why multi-dimensional attention helps GI endoscopy classification, rather than as a claim that MultiAttenGastro is a strictly superior architectural choice.

Figures & tables

Method	Backbone	Dataset	#Cls	Acc.	Macro F1
Kvasir-v2
Al-Otaibi et al. [ 6 ]	EfficientNet-B1	Kvasir-v2	8	98.94%	–
Attallah et al. [ 23 ]	CNN fusion $+$ ML	Kvasir-v2	8	97.8%	–
MultiAttenGastro	ResNet50	Kvasir-v2	8	94.33%	94.32%
HyperKvasir
Li et al. [ 5 ]	17-CNN ensemble	HyperKvasir	10	95.30%	–

Table 1: Comparison with recent GI classification methods on overlapping datasets. Prior work typically reports accuracy on curated subsets, while our results include both accuracy and macro F1 under stratified multiseed evaluation. The RT-DETR entry is an object-detection task and not directly comparable to classification rows.

Figure 1: Architecture of the proposed MultiAttenGastro architecture for GI endoscopy classification. A pretrained CNN backbone first extracts deep feature maps, followed by a $1\times 1$ convolutional layer that projects the features to a 256-channel representation. The refined features are processed by three parallel attention branches: (i) channel-wise (1D), (ii) spatial (2D), and (iii) contextual (3D). The outputs of all attention heads are fused via element-wise addition ( $\oplus$ ), enabling complementary feature integration. The fused representation is subsequently aggregated using global average pooling (GAP) and passed to a fully connected layer for final class prediction.

Dataset	Modality	Train	Val	Test	Imbalance	Gap	Wins	Losses
Kvasir-Capsule [ 3 ]	WCE	9,299	1,641	2,354	3,434:1	Large	6/8	2/8
SEE-AI 1 1 1 https://www.kaggle.com/datasets/capsuleyolo/kyucapsule	WCE	12,943	2,769	2,769	Moderate	Small	4/8	4/8
Kvasir-v2 [ 3 ]	Gastroscopy	5,600	1,200	1,200	Balanced	Small	0/8	8/8
HyperKvasir	Gastroscopy	6,799	1,450	1,450	Moderate	Small–Med	3/8	5/8
GastroVision	Gastroscopy	7,785	1,662	1,662	Moderate	Small–Med	2/8	6/8

Table 2: Dataset overview with split sizes, domain gap, and MultiAttenGastro win/loss counts across the 8-backbone benchmark (Tables 3 , 4 ).

Backbone	Params(Base)	Params( $+$ MAG)	F1 Base(%)	F1 $+$ MAG( $\Delta$ %)
ResNet50	23.54M	24.25M	97.29	97.78 (+0.49)
DenseNet121	6.97M	7.43M	98.04	97.80 ( $-$ 0.24)
EfficientNet-B3	10.72M	11.30M	97.33	97.68 (+0.35)
Inception-v3	21.81M	22.52M	96.97	97.45 (+0.48)
MobileNet-v2	2.24M	2.77M	97.13	98.10 (+0.97)
MobileNet-v3-L	2.99M	3.43M	97.79	97.49 ( $-$ 0.30)

Table 3: Kvasir-Capsule (WCE, 14 classes, large domain gap), single-seed (seed=42). MultiAttenGastro improves 6 of 8 backbones. See Sect. 7 for multiseed re-evaluation of DenseNet121 and MobileNet-v3-L exceptions.

	Kvasir-v2		HyperKvasir		GastroVision		SEE-AI
Backbone	Base	$+$ MAG( $\Delta$ )	Base	$+$ MAG( $\Delta$ )	Base	$+$ MAG( $\Delta$ )	Base	$+$ MAG( $\Delta$ )
ResNet50	94.32	93.31( $-$ 1.01)	83.89	84.63(+0.74)	89.09	88.07( $-$ 1.02)	69.61	69.81(+0.20)
DenseNet121	93.20	92.79( $-$ 0.41)	83.75	83.48( $-$ 0.27)	88.48	89.00(+0.52)	75.04	66.33( $-$ 8.71)
EfficientNet-B3	93.74	91.87( $-$ 1.87)	83.82	83.00( $-$ 0.82)	88.71	87.82( $-$ 0.89)	74.19	74.24(+0.05)
Inception-v3	91.98	89.19( $-$ 2.79)	80.71	81.69(+0.98)	87.19	85.75( $-$ 1.44)	62.37	58.30( $-$ 4.07)
MobileNet-v2	93.08	92.48( $-$ 0.60)	84.94	81.49( $-$ 3.45)	90.45	86.63( $-$ 3.82)	70.44	66.30( $-$ 4.14)

Table 4: Backbone benchmark on the four remaining datasets (F1 Base(%) /

+

MAG(

\Delta

%)). Bold = best F1 in each dataset. Kvasir-v2: 0/8 wins (uniform degradation, small domain gap). HyperKvasir: 3/8 wins (small–medium gap). GastroVision: 2/8 wins (small–medium gap). SEE-AI: 4/8 wins (WCE, small effective gap).

Configuration	F1 $\mu$ $\pm$ $\sigma$ (%)	AUC(%)	$\Delta$ F1 vs. base(%)	paired $t$ ( $p$ )	Wilcoxon $p$
No Attention	98.096 $\pm$ 0.341	99.973	—	0.793 (0.472)	0.625
1D Channel	97.694 $\pm$ 0.598	—	$-$ 0.402	—	—
2D Spatial	97.894 $\pm$ 0.323	—	$-$ 0.202	—	—
3D Contextual	98.132 $\pm$ 0.127	—	+0.036	—	—
SE-Net [ 8 ]	97.990 $\pm$ 0.341	—	$-$ 0.106	2.394 (0.075)	0.125
CBAM [ 9 ]	98.098 $\pm$ 0.351	—	+0.002	0.919 (0.410)	0.438

Table 5: Head-level ablation and alternative-mechanism comparison on Kvasir-Capsule (ConvNeXt-Tiny), 5-seed evaluation (seeds: 7, 42, 99, 123, 2024). Top block: per-head ablation (1D and 2D individually trail baseline; only 3D and the full combination reach or exceed it). Bottom block: alternative attention mechanisms substituted for full MAG under the same protocol. Significance columns report paired tests (matched by seed) against Full MAG; none reach

p<0.05

Backbone	Baseline (min)	$+$ MAG (min)	Overhead (%)
ConvNeXt-Tiny	22.22	23.18	+4.3
MobileNet-v2	12.54	13.14	+4.8
DenseNet121	24.42	26.66	+9.2
MobileNet-v3-Large	12.88	14.26	+10.7

Table 6: Per-backbone training-time overhead of MultiAttenGastro relative to the no-attention baseline, mean over 5 seeds (7, 42, 99, 123, 2024), Kvasir-Capsule.

Dataset	Baseline F1 mean $\pm$ std (%)	Full MAG F1 mean $\pm$ std (%)	$\Delta$ F1 (%)
Kvasir-v2	92.73 $\pm$ 0.471	92.56 $\pm$ 0.508	$-$ 0.17
HyperKvasir	85.77 $\pm$ 0.933	86.25 $\pm$ 0.910	+0.48
SEE-AI	77.39 $\pm$ 2.622	76.62 $\pm$ 1.685	$-$ 0.77

Table 7: Multiseed evaluation of ConvNeXt-Tiny with and without MultiAttenGastro on Kvasir-v2, HyperKvasir, and SEE-AI (5 seeds each: 7, 42, 99, 123, 2024).

Panel of Figure 2 — Figure 2: Grad-CAM comparison of MultiAttenGastro across high- and low-domain-gap WCE datasets. Both figures use the same column order: Original $\mid$ Baseline (w/o MAG) $\mid$ MultiAttenGastro (+MAG) with ConvNeXt-Tiny.

Dataset	Domain Gap	Mean inter-head CKA	Faithfulness gain
Kvasir-Capsule	Large	0.577	+0.021
SEE-AI	Small	0.749	+0.008
HyperKvasir	Small-Med	0.858	$-$ 0.080
GastroVision	Small-Med	0.858	$-$ 0.052
Kvasir-v2	Small	0.864	$-$ 0.019

Table 8: Mean inter-head CKA (representational redundancy) and Grad-CAM faithfulness gain (insertion AUC, MAG vs. baseline) across five datasets, ConvNeXt-Tiny.

Dataset	CKA(frozen, fine-tuned)	Test $n$
SEE-AI	0.122	2000
Kvasir-Capsule	0.225	2000
HyperKvasir	0.420	1450
Kvasir-v2	0.459	1200
GastroVision	0.470	1662

Table 9: Linear CKA between frozen ImageNet and fine-tuned baseline ConvNeXt-Tiny features. Lower CKA indicates greater feature shift during fine-tuning.

Explore similar work

Representation-driven Endoscopic Visual Embedding Alignment for Latent Generation
Aug 7, 2026Francisco Caetano, Tim J. M. Jaspers, Haiko Middeljans +7Endoscopy Diffusion Transformer
Department of Electrical Engineering, ARIA Lab, Eindhoven University of Technology, Eindhoven, The Netherlands · Department of Gastroenterology and Hepatology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands
Towards Grounded GI Endoscopy VQA via Multi-Task Learning on Small VLMs
Jul 29, 2026Itbaan Safwan, Ramail Khan, Muhammad Annas Shaikh +1Endoscopy Medical VQA
Institute of Business and Administration, Karachi, Pakistan
Training-Time Optical Priors for Wireless Capsule Endoscopy Classification: Hemoglobin-Aware Input Fusion with Cross-Vendor Evaluation
May 14, 2026Chengshuai Yang, Lei Xing, Keyaan Zawad Alam +4Endoscopy Medical Image Classification
Department of Biomedical Engineering, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd, Dallas, 75390, TX, USA · Department of Radiation Oncology, Stanford University, Stanford, 94305, CA, USA · Coppell High School, Coppell, TX, USA +4

Aug 7, 2026J/K move · Enter open · S save

Representation-driven Endoscopic Visual Embedding Alignment for Latent Generation

Francisco Caetano, Tim J. M. Jaspers, Haiko Middeljans, Martijn R. Jong+6

Department of Electrical Engineering, ARIA Lab, Eindhoven University of Technology, Eindhoven, The Netherlands · Department of Gastroenterology and Hepatology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands

Endoscopy Diffusion Transformer Medical Image Synthesis Medical Image Representation Learning Diffusion Models

Open paper AskPDF arXiv

Developing foundation generative models for endoscopy is limited by the gap between natural and clinical images and the computational cost of training large Diffusion Transformers. Although representation alignment has improved efficiency in general computer vision, its role within the highly specialized endoscopic image space remains unclear. We introduce REVEAL (Representation-driven Endoscopic Visual Embedding Alignment), the largest generative foundation model for endoscopy to date, trained on GastroNet-5M (GN-5M), a multicenter dataset of 5 million endoscopic frames. Instead of depending on out-of-domain priors, REVEAL employs encoders pretrained directly on the endoscopic distribution to align diffusion latents with domain-specific visual features, preserving fine textures and intricate anatomical structures. Beyond image generation, REVEAL also serves as a powerful feature extractor; in multiple benchmarks, it delivers performance that is competitive with, and in several cases exceeds, endoscopic foundation models such as EndoViT and Endo-FM, specifically tuned for classification tasks, while demonstrating strong representation robustness under realistic imaging corruptions. REVEAL produces high-fidelity images and maintains robust structural coherence in latent-space edits such as inpainting and outpainting. This high-capacity backbone lowers the computational threshold for building specialized clinical tools, offering an open, versatile foundation for conditional synthesis, segmentation, and out-of-distribution detection in future intelligent gastroenterology systems.

Abstract

Explore similar work

Representation-driven Endoscopic Visual Embedding Alignment for Latent Generation

Towards Grounded GI Endoscopy VQA via Multi-Task Learning on Small VLMs

Training-Time Optical Priors for Wireless Capsule Endoscopy Classification: Hemoglobin-Aware Input Fusion with Cross-Vendor Evaluation