ColoDiff: Integrating Dynamic Consistency With Content Awareness for Colonoscopy Video Generation

Junhu Fu; Shuyu Liang; Wutong Li; Chen Ma; Peng Huang; Kehao Wang; Ke Chen; Shengli Lin

cs.CVFeb 26, 2026

ColoDiff: Integrating Dynamic Consistency With Content Awareness for Colonoscopy Video Generation

Authors: Junhu Fu, Shuyu Liang, Wutong Li, Chen Ma, Peng Huang, Kehao Wang, Ke Chen, Shengli Lin,

Organizations: College of Biomedical Engineering, Fudan University, Shanghai 200433, China · Department of Endoscopy, Fudan University Shanghai Cancer Center, Shanghai 200032, China · Endoscopy Center and Endoscopy Research Institute, Zhongshan Hospital, Fudan University, Shanghai 200032, China · Shanghai Collaborative Innovation Center of Endoscopy, Shanghai 200032, China

Abstract

Colonoscopy video generation delivers dynamic, information-rich data critical for diagnosing intestinal diseases, particularly in data-scarce scenarios. High-quality video generation demands temporal consistency and precise control over clinical attributes, but faces challenges from irregular intestinal structures, diverse disease representations, and various imaging modalities. To this end, we propose ColoDiff, a diffusion-based framework that generates dynamic-consistent and content-aware colonoscopy videos, aiming to alleviate data shortage and assist clinical analysis. At the inter-frame level, our TimeStream module decouples temporal dependency from video sequences through a cross-frame tokenization mechanism, enabling intricate dynamic modeling despite irregular intestinal structures. At the intra-frame level, our Content-Aware module incorporates noise-injected embeddings and learnable prototypes to realize precise control over clinical attributes, breaking through the coarse guidance of diffusion models. Additionally, ColoDiff employs a non-Markovian sampling strategy that cuts steps by over 90% for real-time generation. ColoDiff is evaluated across three public datasets and one hospital database, based on both generation metrics and downstream tasks including disease diagnosis, modality discrimination, bowel preparation scoring, and lesion segmentation. Extensive experiments show ColoDiff generates videos with smooth transitions and rich dynamics. ColoDiff presents an effort in controllable colonoscopy video generation, revealing the potential of synthetic videos in complementing authentic representation and mitigating data scarcity in clinical settings.

Figures & tables

Fig. 1: (a) - (b) Colonoscopy video analysis serves as an important diagnostic approach, integrating diverse imaging modalities and disease manifestations. (c) - (e) Existing challenges for colonoscopy video generation.

Fig. 2: The overall workflow of ColoDiff. $\mathscr{E}$ and $\mathscr{D}$ denote pre-trained encoder and decoder in VAE, respectively. $\bm{z}_{t}$ represents the latent features that have been added noise. The light red area shows how the TimeStream module decouples temporal information, and the light blue area shows how the Content-Aware module controls generation process.

Type	Method	Colonoscopic [ 25 ]			HyperKvasir [ 2 ]			SUN-SEG [ 4 ]			Hospital Data
Type	Method	FVD $\downarrow$	FID $\downarrow$	IS $\uparrow$	FVD $\downarrow$	FID $\downarrow$	IS $\uparrow$	FVD $\downarrow$	FID $\downarrow$	IS $\uparrow$	FVD $\downarrow$	FID $\downarrow$	IS $\uparrow$
GAN-based	StyleGAN-V [ 20 ]	2111	226.1	2.12	729	88.5	1.98	682	46.5	3.91	703	51.7	3.42
GAN-based	MoStGAN-V [ 21 ]	469	53.2	3.37	607	29.4	2.74	412	28.6	4.04	522	46.6	3.59
Diffusion-based	LVDM [ 18 ]	1037	96.9	1.93	528	49.7	2.15	387	35.9	3.77	416	40.3	3.12
	Endora [ 22 ]	461	13.4	3.90	496	19.6	3.29	375	17.8	4.59	389	23.2	3.65
	FEAT-L [ 23 ]	351	12.3	4.01	511	21.2	3.27	356	13.6	4.61	402	18.9	3.88

TABLE I: Quantitative Comparison With SOTA Algorithms. Best Results Are in Bold .

Category		Colitis			Polyp			Adenoma
Metric		FVD $\downarrow$	FID $\downarrow$	IS $\uparrow$	FVD $\downarrow$	FID $\downarrow$	IS $\uparrow$	FVD $\downarrow$	FID $\downarrow$	IS $\uparrow$
Temporal Coherence	Transformer Encoder Layer	518	29.1	3.37	501	22.9	3.61	655	32.9	3.15
Temporal Coherence	TimeStream Module	372	16.4	3.92	316	11.8	4.08	328	17.5	3.79
Content Controllability	One-hot Encoding	491	20.9	3.71	430	14.3	3.85	581	25.3	3.46
	Random Encoding	486	21.6	3.64	446	15.1	3.79	567	25.8	3.50
	Learnable Prototypes	390	17.7	3.81	341	13.6	3.85	396	20.8	3.55

TABLE II: Ablation Experiment Results for TimeStream And Content-Aware Modules. Best Results Are in Bold .

Fig. 3: Visual comparison of generated videos on Colonoscopic dataset [ 25 ] . (a) Videos generated by other compared methods. The blue boxes indicate regions that exhibit corresponding issues. Row 1: anatomical and textural visual distortion. Row 2: limited content variation. Rows 3-5: inter-frame incoherence. (b) Videos generated by ColoDiff.

Time	If	FID $\downarrow$			Inference	Frame
Steps	Markovian	Colitis	Polyp	Adenoma	Time (s)	Rate
$T$ =250	Yes	16.4	11.8	17.5	13.46	1.19
$T$ =100	No	16.7	12.3	17.9	5.17	3.09
$T$ =50	No	17.1	12.5	18.4	2.61	6.13
$T$ =10	No	17.5	13.1	18.7	0.49	32.65
$T$ =5	No	19.3	14.2	21.6	0.24	66.67

TABLE III: Ablation Experiment Results for Non-Markovian Sampling Strategy. The Inference Time Column Indicates The Time Required to Generate A 16-frame Video.

Fig. 4: Examples of videos generated by ColoDiff under different resolution settings. For uniform display, slight stretching deformation is applied to the original images.

Fig. 5: The experimental results of Turing test and Consistency test.

Metric	Disease						Modality				BBPS
	Colitis		Polyp		Adenoma		WLI		NBI		0-1		2-3
	Original	+Synthetic	Original	+Synthetic	Original	+Synthetic	Original	+Synthetic	Original	+Synthetic	Original	+Synthetic	Original	+Synthetic
Precision $\uparrow$	79.2	85.9	85.3	92.8	73.5	81.0	88.4	93.7	93.1	96.8	86.7	90.3	87.5	93.8
Recall $\uparrow$	82.1	87.4	87.7	89.1	76.8	83.4	91.2	96.3	90.6	93.7	89.2	92.6	85.3	91.8
F1-score $\uparrow$	80.6	86.6	86.5	90.9	75.1	82.2	89.8	95.0	91.8	95.2	87.9	91.4	86.4	92.8
Accuracy $\uparrow$	79.8	85.8	85.4	91.7	74.3	83.1	90.2	94.4	91.8	95.1	88.5	92.1	86.6	92.5

TABLE IV: Quantitative Results for Downstream Classification Tasks (Unit: %). For Paired Metrics, Models Trained With Real Videos (Left) vs. Real + Synthetic Videos (Right) Are Compared. Better Results Are in Bold .

Fig. 6: UMAP visualization for downstream classification tasks on test sets, with the left column representing models trained with original training set, and the right column representing models trained with an equal number of synthetic videos added.

Fig. 7: Visualized examples of downstream classification performance.

Metric	Test-Easy (Seen)		Test-Easy (Unseen)		Test-Hard (Seen)		Test-Hard (Unseen)
Metric	Original	+Synthetic	Original	+Synthetic	Original	+Synthetic	Original	+Synthetic
Dice $\uparrow$	91.2	93.6	81.3	86.5	84.5	90.7	72.9	84.1
mIoU $\uparrow$	89.3	91.5	80.4	85.2	88.1	92.6	77.4	86.8
Precision $\uparrow$	89.9	93.3	80.8	83.1	84.9	96.1	78.0	86.3
Recall $\uparrow$	92.5	93.9	81.8	90.2	84.1	85.9	68.4	82.0

TABLE V: Quantitative Results for Downstream Segmentation Tasks (Unit: %). For Paired Metrics, Models Trained With Real Videos (Left) vs. Real + Synthetic Videos (Right) Are Compared. Better Results Are in Bold .

Fig. 8: Representative examples of downstream segmentation performance. Translucent blue regions indicate the ground truth, red contours denote results without generated data for training, and green contours show results after integrating generated data for training.

Video Generation

Video generation studies models that create and edit moving images from text, images, audio, and structured controls.

See all 364 papers

DepthPilot: From Controllability to Interpretability in Colonoscopy Video Generation
Apr 29, 2026Junhu Fu, Ke Chen, Weidong Guo +9Endoscopy Physical Consistency in Video Generation
College of Biomedical Engineering, Fudan University, Shanghai 200433, China · Key Laboratory of Medical Imaging Computing and Computer Assisted Intervention of Shanghai, Shanghai 200032, China · Department of Endoscopy, Fudan University Shanghai Cancer Center, Shanghai 200032, China +4
Contrastive Learning under Noisy Temporal Self-Supervision for Colonoscopy Videos
May 12, 2026Luca Parolari, Pietro Gori, Lamberto Ballan +2Contrastive Learning Endoscopy
Department of Mathematics, University of Padova, Padova, Italy · LTCI, Telecom Paris, Institut Polytechnique de Paris, Palaiseau, France · Cosmo Intelligent Medical Devices, Dublin, Ireland
ColonSplat: Reconstruction of Peristaltic Motion in Colonoscopy with Dynamic Gaussian Splatting
Mar 6, 2026Weronika Smolak-Dyżewska, Joanna Kaleta, Diego Dall'Alba +1Dynamic Gaussian Splatting Endoscopic 3D Reconstruction
Jagiellonian University, Kraków, Poland · Warsaw University of Technology, Poland · Sano Centre for Computational Medicine, Kraków, Poland +2

Apr 29, 2026J/K move · Enter open · S save

DepthPilot: From Controllability to Interpretability in Colonoscopy Video Generation

Junhu Fu, Ke Chen, Weidong Guo, Shuyu Liang+8

College of Biomedical Engineering, Fudan University, Shanghai 200433, China · Key Laboratory of Medical Imaging Computing and Computer Assisted Intervention of Shanghai, Shanghai 200032, China · Department of Endoscopy, Fudan University Shanghai Cancer Center, Shanghai 200032, China+4

Endoscopy Physical Consistency in Video Generation Image-to-Video Generation Video Diffusion Models Video Generation Controllable Video Generation

Open paper AskPDF arXiv

Controllable medical video generation has achieved remarkable progress, but it still lacks interpretability, which requires the alignment of generated contents with physical priors and faithful clinical manifestations. To push the boundaries from mere controllability to interpretability, we propose DepthPilot, the first interpretable framework for colonoscopy video generation. This work takes a step toward trustworthy generation through two synergistic paradigms. To achieve explicit geometric grounding, DepthPilot devises a prior distribution alignment strategy, injecting depth constraints into the diffusion backbone via parameter-efficient fine-tuning to ensure anatomical fidelity. To enhance intrinsic nonlinear modeling under these geometric constraints, DepthPilot employs an adaptive spline denoising module, replacing fixed linear weights with learnable spline functions to capture complex spatio-temporal dynamics. Extensive evaluations across three public datasets and in-house clinical data confirm DepthPilot's robust ability to produce physically consistent videos. It achieves FID scores below 15 across all benchmarks and ranks first in clinician assessments, bridging the gap between "visually realistic" and "clinically interpretable". Moreover, DepthPilot-generated videos are expected to enable reliable 3D reconstruction, facilitating surgical navigation and blind region identification, and serve as a foundation toward the colorectal world model.

Abstract

Video Generation

Explore similar work

DepthPilot: From Controllability to Interpretability in Colonoscopy Video Generation

Contrastive Learning under Noisy Temporal Self-Supervision for Colonoscopy Videos

ColonSplat: Reconstruction of Peristaltic Motion in Colonoscopy with Dynamic Gaussian Splatting