cs.CVApr 7, 2026

Analogical Reasoning as a Doctor: A Foundation Model for Gastrointestinal Endoscopy Diagnosis

Authors: Peixi Peng, Housheng Xie, Yanling Wei, Guangcong Ruan, Xiaoyang Zou, Qian Cao, Yongjian Nian, Guoyan Zheng

Organizations: Institute of Medical Robotics, School of Biomedical Engineering, Shanghai Jiao Tong University · Institute of Medical Robotics, School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China. · Daping Hospital, Army Medical University · Department of Gastroenterology, Daping Hospital, Army Medical University (Third Military Medical University), Chongqing, 400042, China. · Sir Run Run Shaw Hospital, Zhejiang University School of Medicine · Department of Gastroenterology, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, 310016, China.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Gastrointestinal diseases impose a growing global health burden, and endoscopy is a primary tool for early diagnosis. However, routine endoscopic image interpretation still suffers from missed lesions and limited efficiency. Although AI-assisted diagnosis has shown promise, existing models often lack generalizability, adaptability, robustness, and scalability because of limited medical data, domain shift, and heterogeneous annotations. To address these challenges, we develop RATNet, a foundation model for gastrointestinal endoscopy imaging based on analogical reasoning. RATNet acquires and transfers knowledge from heterogeneous expert annotations across five gastrointestinal endoscopy datasets through a cyclic pre-training strategy. Its architecture consists of an encoder, a relevance-knowledge acquisition and transfer (RAT) module, a projector, and a multi-task head, and supports fine-tuning, linear probing, and zero-shot transfer. Evaluations show that RATNet outperforms existing foundation models, including GastroNet and GastroVision, across six scenarios: diagnosis of common gastrointestinal diseases, few-shot learning for rare diseases, zero-shot transfer to new medical sites, robustness under long-tailed disease distributions, adaptation to novel diseases, and privacy-preserving deployment via federated learning. Its advantage comes from an analogical reasoning mechanism that matches image-derived posterior knowledge to a learned prior knowledge base and transfers relative knowledge to guide diagnosis, improving generalization and resistance to bias. RATNet is open and cost-effective, supports automatic integration of heterogeneous annotations without manual label unification, and reduces data acquisition costs, making it a practical foundation for intelligent gastrointestinal diagnosis, especially in resource-limited settings.

Figures & tables

Figure 1: Workflow of the base model RATNet. This model utilizes gastrointestinal images obtained from medical endoscopy to provide diagnostic assistance. RATNet simulates the analogical reasoning process of physicians by first learning from multiple datasets to construct a prior knowledge base. It then estimates posterior knowledge from the current gastrointestinal images. By comparing and acquiring relevant knowledge from the prior knowledge base, the model transfers this knowledge to the domain of the posterior knowledge, enabling effective reasoning for diagnostic outcomes.
Figure 2: Relevance-knowledge acquisition and transfer network (RATNet) is a foundation model for gastrointestinal disease diagnosis. ( a ) RATNet is pre-trained cyclically using heterogeneous expert annotations from five gastrointestinal endoscopy datasets: CP-CHILD Wang et al. (2020) , LIMUC Polat et al. (2023) , HyperKvasir Borgli et al. (2020) , Daping, and Kvasir Pogorelov et al. (2017) . ( b ) The model comprises four pre-trained components: encoder, RAT module, projector, and multi-task head, adaptable via fine-tuning, linear probing, or zero-shot transfer. Fine-tuning trains the entire model with a new classifier on target data. Linear probing trains only a new classifier on pre-extracted embeddings. Zero-shot transfer aggregates predictions from all classification heads using domain similarity weights, requiring no additional training. ( c ) In federated learning, local RATNet models train at each site using cyclic training for student models and EMA for teacher models. After each round, student weights are averaged at a central server to create a global model, which is redistributed for iterative improvement.
Figure 3: Diagnostic performance for common gastrointestinal diseases on internal test sets. a. Comparison of five large pre-trained models in diagnosing gastrointestinal diseases across five internal test sets. The radar charts show the average scores for metrics such as AUC, MCC, F1, and AP across all diseases within each dataset, highlighting the overall diagnostic performance of each model. b. Comparison of the domain generalization ability of five large pre-trained models. We compared the AUC scores of the five models for Mayo endoscopic scoring across three internal test sets, including LIMUC Polat et al. (2023) , Daping, and HyperKvasir Borgli et al. (2020) , to assess their generalization ability. c. Two-dimensional t-distributed Stochastic Neighbor Embedding (t-SNE) Van der Maaten and Hinton (2008) visualization of prior and posterior knowledge acquired by RATNet from the pre-training dataset. This visualization was used to evaluate whether the model can correctly distinguish posterior knowledge from different tasks and successfully extract a representative prior knowledge base.
Figure 4: Evaluation of the few-shot learning capability for detecting rare conditions. The adenomatous, hyperplastic, and serrated classes from a colonoscopic dataset Mesejo et al. (2016) were used to simulate a low-data scenario for rare disease detection. The adaptability and robustness of five pre-trained models were evaluated under a k-shot learning setting. The box plot illustrates the distribution of AUC scores across 100 experimental runs, showing the median (center line), interquartile range (box boundaries at the 25th and 75th percentiles), whiskers (1.5 times the interquartile range), and outliers as individual points. P-values from two-sided independent t-tests are reported in the figure.
Figure 5: Performance on recognizing common gastrointestinal diseases in new settings without training. The generalizability and robustness of RATNet were evaluated via zero-shot transfer on two unseen datasets (PolypGen Ali et al. (2023) and Shaoyifu) and four disease categories previously observed during pre-training (Mayo1, Mayo2, Mayo3, and Polyp). Although these two datasets were not included in pre-training, RATNet had previously encountered the four disease categories in other datasets used for pre-training, specifically Mayo1, Mayo2, and Mayo3 from LIMUC Polat et al. (2023) , DAPING, and HyperKvasir Borgli et al. (2020) , and Polyp from CP-CHILD Wang et al. (2020) and Kvasir Pogorelov et al. (2017) . Therefore, predictions for the four diseases were obtained directly from the pre-trained RATNet head without further fine-tuning, which we refer to as zero-shot transfer. For comparison, two supervised models (GastroHUN Bravo et al. (2025a) and Gastrovision Jha et al. (2023) ) and two self-supervised models (GastroNet Boers et al. (2024) and SSL Bravo et al. (2025b) ) were evaluated under the same conditions. CI, confidence interval of the ROC curves.
Figure 6: Performance for long-tailed gastrointestinal diseases. We evaluated four pre-trained models (GastroHUN Bravo et al. (2025a) , GastroVision Jha et al. (2023) , GastroNet Boers et al. (2024) , and RATNet) on the unseen GastroVision dataset Jha et al. (2023) for diagnosing 22 gastrointestinal diseases with a long-tailed distribution. Each model and setup was tested over 10 independent runs. Performance is reported in terms of mean AUC scores with standard deviation error bars for ( a ) 10 head classes and ( b ) 12 tail classes. c. Performance of GastroHUN, GastroVision, GastroNet, and RATNet across all 22 diseases for various fractions of training data. To assess label efficiency, we compared the linear probing performance of the models under reduced data settings. For the results reported on each dataset, the appendix ’FT’ denotes results obtained via fine-tuning, while the appendix ’LP’ denotes results obtained via linear probing.
Figure 7: Illustration of how the embeddings for Kvasir-Capsule Smedsrud et al. (2021) dataset’s diseases evolve in t-SNE Van der Maaten and Hinton (2008) . a. We progressively enhanced RATNet by continually pre-training it on the Kvasir-Capsule diagnostic task, resulting in an upgraded model termed RATNet +KC . The t-SNE visualization illustrates the ability of RATNet +KC to capture discriminative prior knowledge for the new task. b. We demonstrate the evolution of feature separability across different disease categories as the model advances from the pre-trained RATNet to the incrementally learned RATNet +KC , and finally to the Fine-tuned RATNet +KC .
Figure 8: Illustration of the training pipeline of RATNet. RATNet is a teacher-student framework with multi-task heads designed to address heterogeneous label spaces across tasks. Specifically, each task is associated with an independent prediction head that maps shared representations to its own label set, while cyclical pretraining enables knowledge acquisition and transfer across tasks despite label inconsistency. In each iteration, the student scans the dataset task by task, and the relevance-knowledge acquisition and transfer module updates both the prior knowledge base and the posterior knowledge template. The student’s newly acquired knowledge is accumulated into the teacher through exponential moving average (EMA), allowing the teacher to guide the student on subsequent tasks. After pretraining, the knowledge accumulated in the teacher can be transferred to downstream target tasks.

Explore similar work

CardsList

  1. Parameter Efficient Fine-tuning for Domain-specific Gastrointestinal Disease Recognition

    Apr 12, 2026Sanjaya Poudel, Nikita Kunwor, Raj Simkhada +3Vision Foundation Model AdaptationMedical Diagnosis

    Auburn University · Tribhuvan University · The University of Texas at Austin +2

  2. Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMs

    Mar 21, 2026Huan Zheng, Yucheng Zhou, Tianyi Yan +6EndoscopyMedical Diagnosis

    SKL-IOTSC, CIS, University of Macau · 2Shanghai Jiao Tong University · 3COWARobot Co. Ltd. +1

  3. Representation-driven Endoscopic Visual Embedding Alignment for Latent Generation

    Aug 7, 2026Francisco Caetano, Tim J. M. Jaspers, Haiko Middeljans +7EndoscopyDiffusion Transformer

    Department of Electrical Engineering, ARIA Lab, Eindhoven University of Technology, Eindhoven, The Netherlands · Department of Gastroenterology and Hepatology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands