Structure-to-Image: Zero-Shot Depth Estimation in Colonoscopy via High-Fidelity Sim-to-Real Adaptation

Juan Yang; Yuyan Zhang; Han Jia; Bing Hu; Wanzhong Song

cs.CVFeb 25, 2026

Structure-to-Image: Zero-Shot Depth Estimation in Colonoscopy via High-Fidelity Sim-to-Real Adaptation

Authors: Juan Yang, Yuyan Zhang, Han Jia, Bing Hu, Wanzhong Song

Organizations: College of Computer Science, Sichuan University, Chengdu, China · Department of Gastroenterology and Hepatology, West China Hospital, Sichuan University, Chengdu, China

Abstract

Monocular depth estimation (MDE) for colonoscopy is hampered by the domain gap between simulated and real-world images. Existing image-to-image translation methods, which use depth as a posterior constraint, often produce structural distortions and specular highlights by failing to balance realism with structure consistency. To address this, we propose a Structure-to-Image paradigm that transforms the depth map from a passive constraint into an active generative foundation. We are the first to introduce phase congruency to colonoscopic domain adaptation and design a cross-level structure constraint to co-optimize geometric structures and fine-grained details like vascular textures. In zero-shot evaluations conducted on a publicly available phantom dataset, the MDE model that was fine-tuned on our generated data achieved a maximum reduction of 44.18% in RMSE compared to competing methods. Our code is available at https://github.com/YyangJJuan/PC-S2I.git.

Figures & tables

Figure 1: Results of existing CycleGAN-based methods for Sim-to-Real colonoscopy show structural distortions and specular artifacts.

Figure 2: Structure-to-Image training pipeline with a cross-level structure constraint for geometric and micro-structural alignment

Figure 3: Inference process of the proposed method.

Figure 4: Stair-step depth map causes contour-like artifacts in the generated image. (a) Depth map of the SimCol dataset, (b) depth profile along the red dashed line of (a), (c) generated realistic image from (a) with depth-to-image translation.

Figure 5: Structures extracted with various methods. (a) real colonoscopy image; (b) Y-Channel of (a); (c) phase consistency map of (a); (d)-(h) edges extracted using Roberts, Prewitt, Sobel, Canny, and Laplacian operators, respectively. Phase congruency map shows more and accurate structures.

Figure 6: Comparison of generated realistic images. Boxes denote structural errors; circles denote specular artifacts. Methods: #1-CycleGAN, #2-Struct-Preserve, #3-Sim2Real, #4-Ours-w/o Normal, #5-Ours-w/o Phase, #6-Ours.

Figure 7: Qualitative comparison of zero-shot depth estimation on C3VD (phantom) and Colon10K (real) datasets. An pre-trained MDE model was fine-tuned on datasets from each method listed. Green boxes highlight key prediction differences. Methods: #1-Baseline, #2-XDCycleGAN, #3-Struct-Preserve, #4-Sim2Real, #5-Ours-w/o Normal, #6-Ours-w/o Phase, #7-Ours.

Methods	PSNR $\uparrow$	SSIM $\uparrow$	IS $\uparrow$
XDCycleGAN [ 18 ]	17.70	0.61	3.38 $\pm$ 0.15
Struct-Preserve [ 30 ]	16.68	0.50	3.20 $\pm$ 0.11
Sim2Real [ 27 ]	17.76	0.66	2.92 $\pm$ 0.08
Ours-w/o Phase	19.89	0.72	3.10 $\pm$ 0.09
Ours-w/o Normal	19.84	0.66	3.21 $\pm$ 0.09
Ours	20.65	0.74	3.47 $\pm$ 0.14

Table 1: Quantitative image generation results. Bold : Best result; Underline : Second-best.

Model	RMSE $\downarrow$ (mm)	MAE $\downarrow$ (mm)	SqRel $\downarrow$ (mm)
XDCycleGAN [ 18 ]	7.74	5.51	1.40
XDCycleGAN-C [ 18 ]	15.00	8.89	8.66
NormDepth [ 31 ]	7.41	-	2.88
Ours-w/o Phase	7.60	5.46	1.38
Ours-w/o Normal	7.60	5.32	1.44
Ours-C	9.25	6.04	2.98

Table 2: Zero-shot depth generation errors on C3VD dataset of image-to-depth generative models. Bold : Best result; Underline : Second-best. Models trained on Colon-Ours are marked with “-C”.

Model	RMSE $\downarrow$ (mm)	MAE $\downarrow$ (mm)	SqRel $\downarrow$ (mm)
Baseline	6.28	4.27	0.97
XDCycleGAN [ 18 ]	7.27	4.81	1.19
Struct-Preserve [ 30 ]	6.90	4.70	1.13
Sim2Real [ 27 ]	8.33	5.58	1.84
Ours-w/o Phase	4.94	3.34	0.67
Ours-w/o Normal	5.15	3.42	0.71

Table 3: Zero-shot depth estimation errors on C3VD dataset of fine-tuned MDE models. Bold : Best result; Underline : Second-best.

Monocular Depth Estimation

Monocular depth estimation infers scene geometry and distance from a single camera image, enabling 3D perception without stereo or depth sensors.

See all 132 papers

Spectral Rectification for Parameter-Efficient Adaptation of Foundation Models in Colonoscopy Depth Estimation
Mar 16, 2026Xiaoxian Zhang, Minghai Shi, Lei LiMonocular Depth Estimation Vision Foundation Model Adaptation
Department of Biomedical Engineering, National University of Singapore, Singapore · Department of Radiotherapy, The First Affiliated Hospital of Xi’an Jiaotong University, Xi’an, China
DepthPolyp: Pseudo-Depth Guided Lightweight Segmentation for Real-Time Colonoscopy
May 15, 2026Zhuoyu Wu, Wenhui Ou, Lexi Zhang +5Endoscopy Image Segmentation
CyPhi AI Lab, Monash University, Malaysia Campus, Malaysia · Department of Electronic & Computer Engineering, Hong Kong University of Science & Technology, Hong Kong, P.R. China · Harbin Institute of Technology, Harbin, P.R. China +1
Colon3R: Cross-Domain 3D Reconstruction from Monocular Colonoscopic Video
Sep 21, 2026Zhihao Xing, Yingyu Wang, Liang Zhao +1Domain Adaptation 3D Reconstruction

Mar 16, 2026J/K move · Enter open · S save

Spectral Rectification for Parameter-Efficient Adaptation of Foundation Models in Colonoscopy Depth Estimation

Xiaoxian Zhang, Minghai Shi, Lei Li

Department of Biomedical Engineering, National University of Singapore, Singapore · Department of Radiotherapy, The First Affiliated Hospital of Xi’an Jiaotong University, Xi’an, China

Monocular Depth Estimation Vision Foundation Model Adaptation Domain Adaptation Endoscopy Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Accurate monocular depth estimation is critical in colonoscopy for lesion localization and navigation. Foundation models trained on natural images fail to generalize directly to colonoscopy. We identify the core issue not as a semantic gap, but as a statistical shift in the frequency domain: colonoscopy images lack the strong high-frequency edge and texture gradients that these models rely on for geometric reasoning. To address this, we propose SpecDepth, a parameter-efficient adaptation framework that preserves the robust geometric representations of the pre-trained models while adapting to the colonoscopy domain. Its key innovation is an adaptive spectral rectification module, which uses a learnable wavelet decomposition to explicitly model and amplify the attenuated high-frequency components in feature maps. Different from conventional fine-tuning that risks distorting high-level semantic features, this targeted, low-level adjustment realigns the input signal with the original inductive bias of the foundational model. On the public C3VD and SimCol3D datasets, SpecDepth achieved state-of-the-art performance with an absolute relative error of 0.022 and 0.027, respectively. Our work demonstrates that directly addressing spectral mismatches is a highly effective strategy for adapting vision foundation models to specialized medical imaging tasks. The code will be released publicly after the manuscript is accepted for publication.

Abstract

Monocular Depth Estimation

Explore similar work

Spectral Rectification for Parameter-Efficient Adaptation of Foundation Models in Colonoscopy Depth Estimation

DepthPolyp: Pseudo-Depth Guided Lightweight Segmentation for Real-Time Colonoscopy

Colon3R: Cross-Domain 3D Reconstruction from Monocular Colonoscopic Video