RARE disease detection from Capsule Endoscopic Videos based on Vision Transformers

X. Gao; C. Chien; G. Liu; A. Manullang

cs.CVMar 16, 2026

RARE disease detection from Capsule Endoscopic Videos based on Vision Transformers

Authors: X. Gao, C. Chien, G. Liu, A. Manullang

Organizations: Department of Computer Science, Middlesex University, London, UK

Abstract

This work is corresponding to the Gastro Competition for multi-label classification from capsule endoscopic videos (CEV). Deep learning network based on Transformers are fined-tune for this task. The based online mode is Google Vision Transformer (ViT) batch16 with 224 x 224 resolutions. In total, 17 labels are classified, which are mouth, esophagus, stomach, small intestine, colon, z-line, pylorus, ileocecal valve, active bleeding, angiectasia, blood, erosion, erythema, hematin, lymphangioectasis, polyp, and ulcer. For test dataset of three videos, the overall mAP @0.5 is 0.0205 whereas the overall mAP @0.95 is 0.0196.

Explore similar work

Anatomy-Guided Vision-Language Learning with Angular Prototype Separation for Multi-Label Video Capsule Endoscopy Classification Under Class Imbalance
Mar 18, 2026Podakanti Satyajith Chary, Nagarajan GanapathyMedical Imaging Temporal Video Understanding
Department of Engineering Science, IIT Hyderabad · Department of Biomedical Engineering, IIT Hyderabad
GALAR-TemporalNet v2: Anatomy-Guided Dual-Branch Temporal Classification with Bidirectional Mamba and Dual-Graph GCN for Video Capsule Endoscopy -- after competition results
May 21, 2026Jiye Won, Seangmin Lee, Soon Ki JungEndoscopy Medical Imaging
Kyungpook National University · School of Computer Science and Engineering, Kyungpook National University, Daegu, Republic of Korea
Divide-then-Diagnose: Weaving Clinician-Inspired Contexts for Ultra-Long Capsule Endoscopy Videos
Apr 23, 2026Bowen Liu, Li Yang, Shanshan Song +6Endoscopy Medical Imaging
1The Hong Kong University of Science and Technology · 2Renji Hospital, Shanghai Jiao Tong University School of Medicine

Mar 18, 2026J/K move · Enter open · S save

Anatomy-Guided Vision-Language Learning with Angular Prototype Separation for Multi-Label Video Capsule Endoscopy Classification Under Class Imbalance

Podakanti Satyajith Chary, Nagarajan Ganapathy

Department of Engineering Science, IIT Hyderabad · Department of Biomedical Engineering, IIT Hyderabad

Medical Imaging Temporal Video Understanding Class-Imbalanced Learning Prototype-Based Learning Medical VLMs

Open paper AskPDF arXiv

This work presents a multi-label temporal event detection framework for video capsule endoscopy (VCE) that addresses the extreme class imbalance inherent in the Galar dataset by combining two principal contributions: an Angular Separation Loss on class prototypes and a Biological State Machine temporal decoder. The backbone remains BiomedCLIP, a biomedical vision-language foundation model. Three consecutive frames are fused through a Local Differencing Attention module that amplifies transient pathological signals by suppressing static temporal redundancy. An Anatomy Context Head then conditions pathological predictions on soft anatomical activations, exploiting the known spatial co-occurrence structure of GI findings. Learnable text-feature prompts and prototype-based logit augmentation are trained alongside an Angular Separation Loss that penalizes off-diagonal cosine similarity between class prototypes, preventing the prototype collapse that afflicts rare classes under extreme imbalance. To counteract the skewed label distribution, the training regime combines asymmetric focal loss, inverse-frequency weighted sampling, temporal Mixup, Exponential Moving Average, and per-class threshold calibration. The Biological State Machine decoder replaces naive gap merging with a physiologically grounded forward-only state transition over anatomy labels, eliminating the fragmentation artefact that produced hundreds of spurious anatomy events per video in the prior approach and reducing per-video anatomy output to 2--3 clinically realistic events. On the held-out RARE-VISION test set comprising three NaviCam examinations (161,025 frames), the updated pipeline achieves an overall temporal [email protected] of 0.3597 and [email protected] of 0.3399, representing a relative improvement of 46% and 44% respectively over the prior submission, with total inference completed in approximately 21 minutes on a single GPU.

Abstract

Explore similar work

Anatomy-Guided Vision-Language Learning with Angular Prototype Separation for Multi-Label Video Capsule Endoscopy Classification Under Class Imbalance

GALAR-TemporalNet v2: Anatomy-Guided Dual-Branch Temporal Classification with Bidirectional Mamba and Dual-Graph GCN for Video Capsule Endoscopy -- after competition results

Divide-then-Diagnose: Weaving Clinician-Inspired Contexts for Ultra-Long Capsule Endoscopy Videos