Organizations: Institute of Medical Robotics, School of Biomedical Engineering, Shanghai Jiao Tong University · Institute of Medical Robotics, School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China. · Daping Hospital, Army Medical University · Department of Gastroenterology, Daping Hospital, Army Medical University (Third Military Medical University), Chongqing, 400042, China. · Sir Run Run Shaw Hospital, Zhejiang University School of Medicine · Department of Gastroenterology, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, 310016, China.
Gastrointestinal diseases impose a growing global health burden, and endoscopy is a primary tool for early diagnosis. However, routine endoscopic image interpretation still suffers from missed lesions and limited efficiency. Although AI-assisted diagnosis has shown promise, existing models often lack generalizability, adaptability, robustness, and scalability because of limited medical data, domain shift, and heterogeneous annotations. To address these challenges, we develop RATNet, a foundation model for gastrointestinal endoscopy imaging based on analogical reasoning. RATNet acquires and transfers knowledge from heterogeneous expert annotations across five gastrointestinal endoscopy datasets through a cyclic pre-training strategy. Its architecture consists of an encoder, a relevance-knowledge acquisition and transfer (RAT) module, a projector, and a multi-task head, and supports fine-tuning, linear probing, and zero-shot transfer. Evaluations show that RATNet outperforms existing foundation models, including GastroNet and GastroVision, across six scenarios: diagnosis of common gastrointestinal diseases, few-shot learning for rare diseases, zero-shot transfer to new medical sites, robustness under long-tailed disease distributions, adaptation to novel diseases, and privacy-preserving deployment via federated learning. Its advantage comes from an analogical reasoning mechanism that matches image-derived posterior knowledge to a learned prior knowledge base and transfers relative knowledge to guide diagnosis, improving generalization and resistance to bias. RATNet is open and cost-effective, supports automatic integration of heterogeneous annotations without manual label unification, and reduces data acquisition costs, making it a practical foundation for intelligent gastrointestinal diagnosis, especially in resource-limited settings.
Figures & tables
Figure 1: Workflow of the base model RATNet. This model utilizes gastrointestinal images obtained from medical endoscopy to provide diagnostic assistance. RATNet simulates the analogical reasoning process of physicians by first learning from multiple datasets to construct a prior knowledge base. It then estimates posterior knowledge from the current gastrointestinal images. By comparing and acquiring relevant knowledge from the prior knowledge base, the model transfers this knowledge to the domain of the posterior knowledge, enabling effective reasoning for diagnostic outcomes.
Figure 2: Relevance-knowledge acquisition and transfer network (RATNet) is a foundation model for gastrointestinal disease diagnosis. ( a ) RATNet is pre-trained cyclically using heterogeneous expert annotations from five gastrointestinal endoscopy datasets: CP-CHILD Wang et al. (2020) , LIMUC Polat et al. (2023) , HyperKvasir Borgli et al. (2020) , Daping, and Kvasir Pogorelov et al. (2017) . ( b ) The model comprises four pre-trained components: encoder, RAT module, projector, and multi-task head, adaptable via fine-tuning, linear probing, or zero-shot transfer. Fine-tuning trains the entire model with a new classifier on target data. Linear probing trains only a new classifier on pre-extracted embeddings. Zero-shot transfer aggregates predictions from all classification heads using domain similarity weights, requiring no additional training. ( c ) In federated learning, local RATNet models train at each site using cyclic training for student models and EMA for teacher models. After each round, student weights are averaged at a central server to create a global model, which is redistributed for iterative improvement.
Figure 3: Diagnostic performance for common gastrointestinal diseases on internal test sets. a. Comparison of five large pre-trained models in diagnosing gastrointestinal diseases across five internal test sets. The radar charts show the average scores for metrics such as AUC, MCC, F1, and AP across all diseases within each dataset, highlighting the overall diagnostic performance of each model. b. Comparison of the domain generalization ability of five large pre-trained models. We compared the AUC scores of the five models for Mayo endoscopic scoring across three internal test sets, including LIMUC Polat et al. (2023) , Daping, and HyperKvasir Borgli et al. (2020) , to assess their generalization ability. c. Two-dimensional t-distributed Stochastic Neighbor Embedding (t-SNE) Van der Maaten and Hinton (2008) visualization of prior and posterior knowledge acquired by RATNet from the pre-training dataset. This visualization was used to evaluate whether the model can correctly distinguish posterior knowledge from different tasks and successfully extract a representative prior knowledge base.
Figure 4: Evaluation of the few-shot learning capability for detecting rare conditions. The adenomatous, hyperplastic, and serrated classes from a colonoscopic dataset Mesejo et al. (2016) were used to simulate a low-data scenario for rare disease detection. The adaptability and robustness of five pre-trained models were evaluated under a k-shot learning setting. The box plot illustrates the distribution of AUC scores across 100 experimental runs, showing the median (center line), interquartile range (box boundaries at the 25th and 75th percentiles), whiskers (1.5 times the interquartile range), and outliers as individual points. P-values from two-sided independent t-tests are reported in the figure.
Figure 5: Performance on recognizing common gastrointestinal diseases in new settings without training. The generalizability and robustness of RATNet were evaluated via zero-shot transfer on two unseen datasets (PolypGen Ali et al. (2023) and Shaoyifu) and four disease categories previously observed during pre-training (Mayo1, Mayo2, Mayo3, and Polyp). Although these two datasets were not included in pre-training, RATNet had previously encountered the four disease categories in other datasets used for pre-training, specifically Mayo1, Mayo2, and Mayo3 from LIMUC Polat et al. (2023) , DAPING, and HyperKvasir Borgli et al. (2020) , and Polyp from CP-CHILD Wang et al. (2020) and Kvasir Pogorelov et al. (2017) . Therefore, predictions for the four diseases were obtained directly from the pre-trained RATNet head without further fine-tuning, which we refer to as zero-shot transfer. For comparison, two supervised models (GastroHUN Bravo et al. (2025a) and Gastrovision Jha et al. (2023) ) and two self-supervised models (GastroNet Boers et al. (2024) and SSL Bravo et al. (2025b) ) were evaluated under the same conditions. CI, confidence interval of the ROC curves.
Figure 6: Performance for long-tailed gastrointestinal diseases. We evaluated four pre-trained models (GastroHUN Bravo et al. (2025a) , GastroVision Jha et al. (2023) , GastroNet Boers et al. (2024) , and RATNet) on the unseen GastroVision dataset Jha et al. (2023) for diagnosing 22 gastrointestinal diseases with a long-tailed distribution. Each model and setup was tested over 10 independent runs. Performance is reported in terms of mean AUC scores with standard deviation error bars for ( a ) 10 head classes and ( b ) 12 tail classes. c. Performance of GastroHUN, GastroVision, GastroNet, and RATNet across all 22 diseases for various fractions of training data. To assess label efficiency, we compared the linear probing performance of the models under reduced data settings. For the results reported on each dataset, the appendix ’FT’ denotes results obtained via fine-tuning, while the appendix ’LP’ denotes results obtained via linear probing.
Figure 7: Illustration of how the embeddings for Kvasir-Capsule Smedsrud et al. (2021) dataset’s diseases evolve in t-SNE Van der Maaten and Hinton (2008) . a. We progressively enhanced RATNet by continually pre-training it on the Kvasir-Capsule diagnostic task, resulting in an upgraded model termed RATNet +KC . The t-SNE visualization illustrates the ability of RATNet +KC to capture discriminative prior knowledge for the new task. b. We demonstrate the evolution of feature separability across different disease categories as the model advances from the pre-trained RATNet to the incrementally learned RATNet +KC , and finally to the Fine-tuned RATNet +KC .
Figure 8: Illustration of the training pipeline of RATNet. RATNet is a teacher-student framework with multi-task heads designed to address heterogeneous label spaces across tasks. Specifically, each task is associated with an independent prediction head that maps shared representations to its own label set, while cyclical pretraining enables knowledge acquisition and transfer across tasks despite label inconsistency. In each iteration, the student scans the dataset task by task, and the relevance-knowledge acquisition and transfer module updates both the prior knowledge base and the posterior knowledge template. The student’s newly acquired knowledge is accumulated into the teacher through exponential moving average (EMA), allowing the teacher to guide the student on subsequent tasks. After pretraining, the knowledge accumulated in the teacher can be transferred to downstream target tasks.
Despite recent advancements in the field of medical image analysis with the use of pretrained foundation models, the issue of distribution shifts between cross-source images largely remains adamant. To circumvent that issue, investigators generally train a separate model for each source. However, this method becomes expensive when we fully fine-tune pretrained large models for a single dataset, as we must store multiple copies of those models. Thus, in this work, we propose using a low-rank adaptation (LoRA) module for fine-tuning downstream classification tasks. LoRAs learn lightweight task-specific low-rank matrices that perturb pretrained weights to optimize those downstream tasks. For gastrointestinal tract diseases, they exhibit significantly better results than end-to-end finetuning with improved parameter efficiency. Code is available at: github.com/sanjay931/peft-gi-recognition.
Sanjaya Poudel, Nikita Kunwor, Raj Simkhada +3
Auburn University · Tribhuvan University · The University of Texas at Austin +2
Multimodal Large Language Models (MLLMs) have demonstrated remarkable potential in medical image analysis. However, their application in gastrointestinal endoscopy is currently hindered by two critical limitations: the misalignment between general model reasoning and standardized clinical cognitive pathways, and the lack of causal association between visual features and diagnostic outcomes. In this paper, we propose a novel Clinical-Cognitive-Aligned (CogAlign) framework to address these challenges. First, we endow the model with rigorous clinical analytical capabilities by constructing the hierarchical clinical cognition dataset and employing Supervised Fine-Tuning (SFT). Unlike conventional approaches, this strategy internalizes the hierarchical diagnostic logic of experts, ranging from anatomical localization and morphological evaluation to microvascular analysis, directly into the model. Second, to eliminate visual bias, we provide a theoretical analysis demonstrating that standard supervised tuning inevitably converges to spurious background correlations. Guided by this insight, we propose a counterfactual-driven reinforcement learning strategy to enforce causal rectification. By generating counterfactual normal samples via lesion masking and optimizing through clinical-cognition-centric rewards, we constrain the model to strictly ground its diagnosis in causal lesion features. Extensive experiments demonstrate that our approach achieves State-of-the-Art (SoTA) performance across multiple benchmarks, significantly enhancing diagnostic accuracy in complex clinical scenarios.
Huan Zheng, Yucheng Zhou, Tianyi Yan +6
SKL-IOTSC, CIS, University of Macau · 2Shanghai Jiao Tong University · 3COWARobot Co. Ltd. +1
Developing foundation generative models for endoscopy is limited by the gap between natural and clinical images and the computational cost of training large Diffusion Transformers. Although representation alignment has improved efficiency in general computer vision, its role within the highly specialized endoscopic image space remains unclear. We introduce REVEAL (Representation-driven Endoscopic Visual Embedding Alignment), the largest generative foundation model for endoscopy to date, trained on GastroNet-5M (GN-5M), a multicenter dataset of 5 million endoscopic frames. Instead of depending on out-of-domain priors, REVEAL employs encoders pretrained directly on the endoscopic distribution to align diffusion latents with domain-specific visual features, preserving fine textures and intricate anatomical structures. Beyond image generation, REVEAL also serves as a powerful feature extractor; in multiple benchmarks, it delivers performance that is competitive with, and in several cases exceeds, endoscopic foundation models such as EndoViT and Endo-FM, specifically tuned for classification tasks, while demonstrating strong representation robustness under realistic imaging corruptions. REVEAL produces high-fidelity images and maintains robust structural coherence in latent-space edits such as inpainting and outpainting. This high-capacity backbone lowers the computational threshold for building specialized clinical tools, offering an open, versatile foundation for conditional synthesis, segmentation, and out-of-distribution detection in future intelligent gastroenterology systems.
Francisco Caetano, Tim J. M. Jaspers, Haiko Middeljans +7
Department of Electrical Engineering, ARIA Lab, Eindhoven University of Technology, Eindhoven, The Netherlands · Department of Gastroenterology and Hepatology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands