This work presents a multi-label temporal event detection framework for video capsule endoscopy (VCE) that addresses the extreme class imbalance inherent in the Galar dataset by combining two principal contributions: an Angular Separation Loss on class prototypes and a Biological State Machine temporal decoder. The backbone remains BiomedCLIP, a biomedical vision-language foundation model. Three consecutive frames are fused through a Local Differencing Attention module that amplifies transient pathological signals by suppressing static temporal redundancy. An Anatomy Context Head then conditions pathological predictions on soft anatomical activations, exploiting the known spatial co-occurrence structure of GI findings. Learnable text-feature prompts and prototype-based logit augmentation are trained alongside an Angular Separation Loss that penalizes off-diagonal cosine similarity between class prototypes, preventing the prototype collapse that afflicts rare classes under extreme imbalance. To counteract the skewed label distribution, the training regime combines asymmetric focal loss, inverse-frequency weighted sampling, temporal Mixup, Exponential Moving Average, and per-class threshold calibration. The Biological State Machine decoder replaces naive gap merging with a physiologically grounded forward-only state transition over anatomy labels, eliminating the fragmentation artefact that produced hundreds of spurious anatomy events per video in the prior approach and reducing per-video anatomy output to 2--3 clinically realistic events. On the held-out RARE-VISION test set comprising three NaviCam examinations (161,025 frames), the updated pipeline achieves an overall temporal [email protected] of 0.3597 and [email protected] of 0.3399, representing a relative improvement of 46% and 44% respectively over the prior submission, with total inference completed in approximately 21 minutes on a single GPU.
Figures & tables
Figure 1: Overview of the proposed pipeline. Stage 1: Three consecutive VCE frames are processed using a BiomedCLIP ViT-B/16 backbone with LDA for temporal feature refinement. Stage 2: An Excitation Block and multi-head classifier jointly predict anatomy and pathology labels, enhanced using CoOp text features, learnable prototypes, and angular prototype augmentation. Stage 3: Training combines asymmetric focal loss, contrastive BCE, orthogonality regularization, Angular Separation Loss, EMA ( β=0.999 ), temporal Mixup ( α=0.3 ), and inverse-frequency weighted sampling for robust imbalanced learning. Stage 4: A BSM decoder performs temporal smoothing, forward-only GI transit enforcement, and event refinement to generate the final competition JSON with temporally localized multi-label VCE events.
Table 1: Per-video temporal mAP on the RARE-VISION test set (3 NaviCam examinations).
Label
AP
AUC
F1
Prec
Rec
Sup
mouth
0.543
0.947
0.625
0.517
0.789
19
esophagus
0.054
0.996
0.200
0.143
0.333
3
stomach
0.894
0.985
0.814
0.860
0.772
2,920
small intestine
0.893
0.970
0.850
0.803
0.903
14,689
colon
0.991
0.978
0.954
0.951
0.957
40,143
z-line
0.000
—
0.000
—
—
0
Table 2: Per-class validation metrics at the best checkpoint (epoch 3), after temperature calibration ( T∗=1.5 ) and per-class threshold optimization. “Sup” denotes the number of positive frames in the validation set.
Epoch
Train Loss
Val Loss
Val mAP
Macro F1
1
0.2850
0.0696
0.2373
0.2129
2
0.1757
0.0952
0.2421
0.2273
3
0.1457
0.1102
0.2438
0.2395
4
0.1241
0.1411
0.2388
0.2273
5
0.1135
0.1671
0.2337
0.2216
Table 3: Training dynamics over 5 epochs. Best checkpoint at epoch 3 is highlighted.
Video
Frames
Events
Class distribution
ukdd_navi_00051
44,878
22
stomach(1), blood(19), small int.(1), colon(1)
ukdd_navi_00068
53,220
23
stomach(1), blood(15), polyp(2), erosion(3), small int.(1), colon(1)
ukdd_navi_00076
62,927
24
small int.(1), blood(20), erosion(2), colon(1)
Table 4: Predicted event distribution per test video after BSM decoding.
Video Capsule Endoscopy (VCE) poses a challenging multi-label temporal classification problem, requiring simultaneous localization of 8 anatomical regions and detection of 9 pathological findings across tens of thousands of frames. We present GALAR-TemporalNet v2, a hierarchical temporal model that addresses three core challenges: extreme class imbalance, long-range temporal dependencies, and pathology--anatomy entanglement. Our architecture combines windowed self-attention for local modeling, a Dual-Graph GCN for global frame relationships, and Bidirectional Mamba for selective boundary context encoding. A novel anatomy prototype residual pathway decouples pathological deviation signals from normal organ appearance, and a frame-level GCN skip connection stabilizes training of visually confusable rare classes. The competition version, GALAR-TemporalNet, achieved an overall [email protected] of 0.2644 and [email protected] of 0.2353 on the RARE-VISION test set. Following the competition, the redesigned GALAR-TemporalNet v2 -- incorporating a restructured pathology branch, refined loss functions, and extended post-processing -- improved these results to [email protected] of 0.3409 and [email protected] of 0.3333.
Jiye Won, Seangmin Lee, Soon Ki Jung
Kyungpook National University · School of Computer Science and Engineering, Kyungpook National University, Daegu, Republic of Korea
The classification performance of deep neural networks relies strongly on access to large, accurately annotated datasets. In medical imaging, however, obtaining such datasets is particularly challenging since annotations must be provided by specialized physicians, which severely limits the pool of annotators. Furthermore, class boundaries can often be ambiguous or difficult to define which further complicates machine learning-based classification. In this paper, we want to address this problem and introduce a framework for mislabel detection in medical datasets. This is validated on the two largest, publicly available datasets for Video Capsule Endoscopy, an important imaging procedure for examining the gastrointestinal tract based on a video stream of lowresolution images. In addition, potentially mislabeled samples identified by our pipeline were reviewed and re-annotated by three experienced gastroenterologists. Our results show that the proposed framework successfully detects incorrectly labeled data and results in an improved anomaly detection performance after cleaning the datasets compared to current baselines.
Julia Werner, Julius Oexle, Oliver Bause +5
Embedded Systems University of Tübingen Tübingen, Germany · Department of Medicine I University Hospital Dresden TU Dresden, Germany
Capsule endoscopy event detection is challenging because diagnostically relevant findings are sparse, visually heterogeneous, and embedded in long, noisy video streams, while evaluation is performed at the event level rather than by frame accuracy alone. We therefore formulate the RARE-VISION task as a metric-aligned event detection problem instead of a purely frame-wise classification task. Our framework combines two complementary backbones, EndoFM-LV for local temporal context and DINOv3 ViT-L/16 for strong frame-level visual semantics, followed by a Diverse Head Ensemble, Validation-Guided Hierarchical Fusion, and Anatomy-Aware Temporal Event Decoding. The fusion stage uses validation-derived class-wise model weighting, backbone weighting, and probability calibration, while the decoding stage applies temporal smoothing, anatomical constraints, threshold refinement, and per-label event generation to produce stable event predictions. Validation ablations indicate that complementary backbones, validation-guided fusion, and anatomy-aware temporal decoding all contribute to event-level performance. On the official hidden test set, the proposed method achieved an overall temporal [email protected] of 0.3530 and temporal [email protected] of 0.3235.
Bo-Cheng Qiu, Yu-Fan Lin, Yu-Zhe Pien +4
National Cheng Kung University, Taiwan · National Yang Ming Chiao Tung University, Taiwan · NVIDIA, Taipei, Taiwan