Organizations: College of Biomedical Engineering, Fudan University, Shanghai 200433, China · Department of Endoscopy, Fudan University Shanghai Cancer Center, Shanghai 200032, China · Endoscopy Center and Endoscopy Research Institute, Zhongshan Hospital, Fudan University, Shanghai 200032, China · Shanghai Collaborative Innovation Center of Endoscopy, Shanghai 200032, China
Colonoscopy video generation delivers dynamic, information-rich data critical for diagnosing intestinal diseases, particularly in data-scarce scenarios. High-quality video generation demands temporal consistency and precise control over clinical attributes, but faces challenges from irregular intestinal structures, diverse disease representations, and various imaging modalities. To this end, we propose ColoDiff, a diffusion-based framework that generates dynamic-consistent and content-aware colonoscopy videos, aiming to alleviate data shortage and assist clinical analysis. At the inter-frame level, our TimeStream module decouples temporal dependency from video sequences through a cross-frame tokenization mechanism, enabling intricate dynamic modeling despite irregular intestinal structures. At the intra-frame level, our Content-Aware module incorporates noise-injected embeddings and learnable prototypes to realize precise control over clinical attributes, breaking through the coarse guidance of diffusion models. Additionally, ColoDiff employs a non-Markovian sampling strategy that cuts steps by over 90% for real-time generation. ColoDiff is evaluated across three public datasets and one hospital database, based on both generation metrics and downstream tasks including disease diagnosis, modality discrimination, bowel preparation scoring, and lesion segmentation. Extensive experiments show ColoDiff generates videos with smooth transitions and rich dynamics. ColoDiff presents an effort in controllable colonoscopy video generation, revealing the potential of synthetic videos in complementing authentic representation and mitigating data scarcity in clinical settings.
Figures & tables
Fig. 1: (a) - (b) Colonoscopy video analysis serves as an important diagnostic approach, integrating diverse imaging modalities and disease manifestations. (c) - (e) Existing challenges for colonoscopy video generation.
Fig. 2: The overall workflow of ColoDiff. E and D denote pre-trained encoder and decoder in VAE, respectively. zt represents the latent features that have been added noise. The light red area shows how the TimeStream module decouples temporal information, and the light blue area shows how the Content-Aware module controls generation process.
Type
Method
Colonoscopic [ 25 ]
HyperKvasir [ 2 ]
SUN-SEG [ 4 ]
Hospital Data
FVD ↓
FID ↓
IS ↑
FVD ↓
FID ↓
IS ↑
FVD ↓
FID ↓
IS ↑
FVD ↓
FID ↓
IS ↑
GAN-based
StyleGAN-V [ 20 ]
2111
226.1
2.12
729
88.5
1.98
682
46.5
3.91
703
51.7
3.42
MoStGAN-V [ 21 ]
469
53.2
3.37
607
29.4
2.74
412
28.6
4.04
522
46.6
3.59
Diffusion-based
LVDM [ 18 ]
1037
96.9
1.93
528
49.7
2.15
387
35.9
3.77
416
40.3
3.12
Endora [ 22 ]
461
13.4
3.90
496
19.6
3.29
375
17.8
4.59
389
23.2
3.65
FEAT-L [ 23 ]
351
12.3
4.01
511
21.2
3.27
356
13.6
4.61
402
18.9
3.88
TABLE I: Quantitative Comparison With SOTA Algorithms. Best Results Are in Bold .
Category
Colitis
Polyp
Adenoma
Metric
FVD ↓
FID ↓
IS ↑
FVD ↓
FID ↓
IS ↑
FVD ↓
FID ↓
IS ↑
Temporal Coherence
Transformer Encoder Layer
518
29.1
3.37
501
22.9
3.61
655
32.9
3.15
TimeStream Module
372
16.4
3.92
316
11.8
4.08
328
17.5
3.79
Content Controllability
One-hot Encoding
491
20.9
3.71
430
14.3
3.85
581
25.3
3.46
Random Encoding
486
21.6
3.64
446
15.1
3.79
567
25.8
3.50
Learnable Prototypes
390
17.7
3.81
341
13.6
3.85
396
20.8
3.55
TABLE II: Ablation Experiment Results for TimeStream And Content-Aware Modules. Best Results Are in Bold .
Fig. 3: Visual comparison of generated videos on Colonoscopic dataset [ 25 ] . (a) Videos generated by other compared methods. The blue boxes indicate regions that exhibit corresponding issues. Row 1: anatomical and textural visual distortion. Row 2: limited content variation. Rows 3-5: inter-frame incoherence. (b) Videos generated by ColoDiff.
Time
If
FID ↓
Inference
Frame
Steps
Markovian
Colitis
Polyp
Adenoma
Time (s)
Rate
T =250
Yes
16.4
11.8
17.5
13.46
1.19
T =100
No
16.7
12.3
17.9
5.17
3.09
T =50
No
17.1
12.5
18.4
2.61
6.13
T =10
No
17.5
13.1
18.7
0.49
32.65
T =5
No
19.3
14.2
21.6
0.24
66.67
TABLE III: Ablation Experiment Results for Non-Markovian Sampling Strategy. The Inference Time Column Indicates The Time Required to Generate A 16-frame Video.
Fig. 4: Examples of videos generated by ColoDiff under different resolution settings. For uniform display, slight stretching deformation is applied to the original images.
Fig. 5: The experimental results of Turing test and Consistency test.
Metric
Disease
Modality
BBPS
Colitis
Polyp
Adenoma
WLI
NBI
0-1
2-3
Original
+Synthetic
Original
+Synthetic
Original
+Synthetic
Original
+Synthetic
Original
+Synthetic
Original
+Synthetic
Original
+Synthetic
Precision ↑
79.2
85.9
85.3
92.8
73.5
81.0
88.4
93.7
93.1
96.8
86.7
90.3
87.5
93.8
Recall ↑
82.1
87.4
87.7
89.1
76.8
83.4
91.2
96.3
90.6
93.7
89.2
92.6
85.3
91.8
F1-score ↑
80.6
86.6
86.5
90.9
75.1
82.2
89.8
95.0
91.8
95.2
87.9
91.4
86.4
92.8
Accuracy ↑
79.8
85.8
85.4
91.7
74.3
83.1
90.2
94.4
91.8
95.1
88.5
92.1
86.6
92.5
TABLE IV: Quantitative Results for Downstream Classification Tasks (Unit: %). For Paired Metrics, Models Trained With Real Videos (Left) vs. Real + Synthetic Videos (Right) Are Compared. Better Results Are in Bold .
Fig. 6: UMAP visualization for downstream classification tasks on test sets, with the left column representing models trained with original training set, and the right column representing models trained with an equal number of synthetic videos added.
Fig. 7: Visualized examples of downstream classification performance.
Metric
Test-Easy (Seen)
Test-Easy (Unseen)
Test-Hard (Seen)
Test-Hard (Unseen)
Original
+Synthetic
Original
+Synthetic
Original
+Synthetic
Original
+Synthetic
Dice ↑
91.2
93.6
81.3
86.5
84.5
90.7
72.9
84.1
mIoU ↑
89.3
91.5
80.4
85.2
88.1
92.6
77.4
86.8
Precision ↑
89.9
93.3
80.8
83.1
84.9
96.1
78.0
86.3
Recall ↑
92.5
93.9
81.8
90.2
84.1
85.9
68.4
82.0
TABLE V: Quantitative Results for Downstream Segmentation Tasks (Unit: %). For Paired Metrics, Models Trained With Real Videos (Left) vs. Real + Synthetic Videos (Right) Are Compared. Better Results Are in Bold .
Fig. 8: Representative examples of downstream segmentation performance. Translucent blue regions indicate the ground truth, red contours denote results without generated data for training, and green contours show results after integrating generated data for training.
Controllable medical video generation has achieved remarkable progress, but it still lacks interpretability, which requires the alignment of generated contents with physical priors and faithful clinical manifestations. To push the boundaries from mere controllability to interpretability, we propose DepthPilot, the first interpretable framework for colonoscopy video generation. This work takes a step toward trustworthy generation through two synergistic paradigms. To achieve explicit geometric grounding, DepthPilot devises a prior distribution alignment strategy, injecting depth constraints into the diffusion backbone via parameter-efficient fine-tuning to ensure anatomical fidelity. To enhance intrinsic nonlinear modeling under these geometric constraints, DepthPilot employs an adaptive spline denoising module, replacing fixed linear weights with learnable spline functions to capture complex spatio-temporal dynamics. Extensive evaluations across three public datasets and in-house clinical data confirm DepthPilot's robust ability to produce physically consistent videos. It achieves FID scores below 15 across all benchmarks and ranks first in clinician assessments, bridging the gap between "visually realistic" and "clinically interpretable". Moreover, DepthPilot-generated videos are expected to enable reliable 3D reconstruction, facilitating surgical navigation and blind region identification, and serve as a foundation toward the colorectal world model.
Junhu Fu, Ke Chen, Weidong Guo +9
College of Biomedical Engineering, Fudan University, Shanghai 200433, China · Key Laboratory of Medical Imaging Computing and Computer Assisted Intervention of Shanghai, Shanghai 200032, China · Department of Endoscopy, Fudan University Shanghai Cancer Center, Shanghai 200032, China +4
Learning robust representations of polyp tracklets is key to enabling multiple AI-assisted colonoscopy applications, from polyp characterization to automated reporting and retrieval. Supervised contrastive learning is an effective approach for learning such representations, but it typically relies on correct positive and negative definitions. Collecting these labels requires linking tracklets that depict the same underlying polyp entity throughout the video, which is costly and demands specialized clinical expertise. In this work, we leverage the sequential workflow of colonoscopy procedures to derive self-supervised associations from temporal structure. Since temporally derived associations are not guaranteed to be correct, we introduce a noise-aware contrastive loss to account for noisy associations. We demonstrate the effectiveness of the learned representations across multiple downstream tasks, including polyp retrieval and re-identification, size estimation, and histology classification. Our method outperforms prior self-supervised and supervised baselines, and matches or exceeds recent foundation models across all tasks, using a lightweight encoder trained on only 27 videos. Code is available at https://github.com/lparolari/ntssl.
Luca Parolari, Pietro Gori, Lamberto Ballan +2
Department of Mathematics, University of Padova, Padova, Italy · LTCI, Telecom Paris, Institut Polytechnique de Paris, Palaiseau, France · Cosmo Intelligent Medical Devices, Dublin, Ireland
Accurate 3D reconstruction of colonoscopy data, accounting for complex peristaltic movements, is crucial for advanced surgical navigation and retrospective diagnostics. While recent novel view synthesis and 3D reconstruction methods have demonstrated remarkable success in general endoscopic scenarios, they struggle in the highly constrained environment of the colon. Due to the limited field of view of a camera moving through an actively deforming tubular structure, existing endoscopic methods reconstruct the colon appearance only for initial camera trajectory. However, the underlying anatomy remains largely static; instead of updating Gaussians' spatial coordinates (xyz), these methods encode deformation through either rotation, scale or opacity adjustments. In this paper, we first present a benchmark analysis of state-of-the-art dynamic endoscopic methods for realistic colonoscopic scenes, showing that they fail to model true anatomical motion. To enable rigorous evaluation of global reconstruction quality, we introduce DynamicColon, a synthetic dataset with ground-truth point clouds at every timestep. Building on these insights, we propose ColonSplat, a dynamic Gaussian Splatting framework that captures peristaltic-like motion while preserving global geometric consistency, achieving superior geometric fidelity on C3VDv2 and DynamicColon datasets. Project page: https://wmito.github.io/ColonSplat
Weronika Smolak-Dyżewska, Joanna Kaleta, Diego Dall'Alba +1
Jagiellonian University, Kraków, Poland · Warsaw University of Technology, Poland · Sano Centre for Computational Medicine, Kraków, Poland +2