Spectral Rectification for Parameter-Efficient Adaptation of Foundation Models in Colonoscopy Depth Estimation
Authors: Xiaoxian Zhang, Minghai Shi, Lei Li
Organizations: Department of Biomedical Engineering, National University of Singapore, Singapore · Department of Radiotherapy, The First Affiliated Hospital of Xi’an Jiaotong University, Xi’an, China
Accurate monocular depth estimation is critical in colonoscopy for lesion localization and navigation. Foundation models trained on natural images fail to generalize directly to colonoscopy. We identify the core issue not as a semantic gap, but as a statistical shift in the frequency domain: colonoscopy images lack the strong high-frequency edge and texture gradients that these models rely on for geometric reasoning. To address this, we propose SpecDepth, a parameter-efficient adaptation framework that preserves the robust geometric representations of the pre-trained models while adapting to the colonoscopy domain. Its key innovation is an adaptive spectral rectification module, which uses a learnable wavelet decomposition to explicitly model and amplify the attenuated high-frequency components in feature maps. Different from conventional fine-tuning that risks distorting high-level semantic features, this targeted, low-level adjustment realigns the input signal with the original inductive bias of the foundational model. On the public C3VD and SimCol3D datasets, SpecDepth achieved state-of-the-art performance with an absolute relative error of 0.022 and 0.027, respectively. Our work demonstrates that directly addressing spectral mismatches is a highly effective strategy for adapting vision foundation models to specialized medical imaging tasks. The code will be released publicly after the manuscript is accepted for publication.
Figures & tables
Figure 1: Statistical shift in the frequency domain between natural and colonoscopy images. (a) Visual comparison of Fourier magnitude spectra. Natural images (left) display prominent star-like anisotropic radiation patterns, reflecting the abundance of straight lines and sharp edges in real-world scenes. Colonoscopy images (right) show centrally concentrated, more isotropic diffusion patterns, indicating a lack of directional geometric structures, with high-frequency energy decaying rapidly but influenced by specular highlights and unstructured noise. (b) Scatter plot of power-law spectral slope ( α ) versus linearity of the fit ( R2 ). Natural street scenes (orange clusters) exhibit spectral slopes and linearity that strictly adhere to the characteristic power-law distribution of natural scenes. Conversely, colonoscopy images (blue clusters) significantly deviate from this norm, demonstrating a distinct statistical shift in the frequency domain. This shift is characterized by impulsive high-frequency components derived from mucosal artifacts, which elevate the energy in the spectral tail and result in a slower decay rate. The consistently lower R2 values further indicate that biological tissue lacks the strict scale invariance typical of macroscopic natural scenes.
Figure 2: Overview of the proposed SpecDepth framework for the depth estimation of colonoscopy images. It employs a partially frozen DINOv2 encoder for stable feature extraction, followed by the gated wavelet transform to decouple feature representations in the frequency domain and selectively amplify critical edge signals. LN: layer normalization. MHSA: multi-head self-attention. MLP: multi-layer perceptron. BN: batch normalization.
Figure 3: Illustration of wavelet decomposition process used in wavelet transform convolution. Note that here 2× upsampling applied to frequency-domain components for visual alignment with spatial domain.
Table 1: Depth estimation metrics used in this study.
Method
Params (M)
AbsRel↓
SqRel↓
log10↓
RMSE↓
RMSElog↓
SILog↓
δ1↑
δ2↑
δ3↑
NeWCRFs ( Yuan et al., 2022 )
270.4
0.133
0.557
0.048
3.058
0.151
11.817
0.854
0.973
0.997
NDDepth ( Shao et al., 2024 )
348.4
0.316
2.651
0.111
7.043
0.322
24.255
0.498
0.873
0.956
BTS ( Lee et al., 2019 )
50.3
0.127
0.622
0.055
3.823
0.150
11.400
0.812
0.979
0.998
Col3D-MTL+SSL ( Solano et al., 2025 )
50.3
0.107
0.346
0.042
2.729
0.128
10.072
0.899
0.989
0.999
EcoDepth ( Patni et al., 2024 )
993.48
0.103
0.367
0.042
2.721
0.116
8.539
0.901
0.994
0.998
Marigold ( Ke et al., 2024 )
865.92
0.075
0.228
0.032
2.353
0.115
11.424
0.933
0.988
0.995
Table 2: Summary of quantitative results of different method for depth estimation on the C3VD dataset.
Method
AbsRel↓
SqRel↓
RMSE↓
RMSElog↓
δ1↑
Monodepth2 ( Godard et al., 2019 )
0.076
0.061
0.402
0.106
0.950
FeatDepth ( Shu et al., 2020 )
0.077
0.069
0.374
0.098
0.957
HR-Depth ( Lyu et al., 2021 )
0.072
0.044
0.378
0.100
0.961
DIFFNet ( Zhou et al., 2021 )
0.074
0.053
0.401
0.105
0.957
Endo-SfMLearner ( Ozyoruk et al., 2021 )
0.072
0.042
0.407
0.103
0.950
AF-SfMLearner ( Shao et al., 2022 )
0.071
0.045
0.372
0.099
0.961
Table 3: Summary of quantitative results of different method for depth estimation on the SimCol3D dataset.
Figure 5: Visualization of monocular depth estimation on the SimCol3D dataset of different methods. EcoDepth was heavily corrupted by structured noise, while Marigold, though smoother, tended to produce blob-shaped depth maps that failed to follow the elongated recession of the lumen. Depth Anything V2 captured the overall scene geometry more faithfully but often folded boundaries and transitional regions. SpecDepth produced the most coherent results, accurately rendering the tubular depth gradient along the lumen axis and maintaining crisp fold structures throughout.
Encoder
GWT
MC
AbsRel ↓
SqRel ↓
RMSE ↓
δ1↑
Full Fine-Tuning
×
×
0.125
0.571
5.205
0.866
Full Fine-Tuning
✓
×
0.104
0.578
4.461
0.939
Full Fine-Tuning
×
✓
0.092
0.562
5.517
0.881
Full Fine-Tuning
✓
✓
0.077
0.553
3.703
0.960
LoRA
×
×
0.133
0.688
5.862
0.859
LoRA
✓
×
0.086
0.567
3.866
0.953
Table 4: Ablation study of the proposed model on the C3VD validation set. GWT: gated-wavelet transform. MC: multi-constraint.
λgrad
λsmooth
AbsRel (↓)
SqRel (↓)
RMSE (↓)
δ1↑
0
0
0.125
0.571
5.205
0.866
0
0.1
0.115
0.571
5.310
0.876
0
0.2
0.112
0.570
5.308
0.881
0.1
0
0.099
0.566
5.694
0.879
0.1
0.1
0.092
0.562
5.517
0.881
0.1
0.2
0.122
0.572
5.213
0.863
Table 5: Sensitivity analysis of the gradient and smoothness loss weights on the C3VD validation set.
Figure 6: Qualitative results of 3D point cloud reconstruction on the C3VD dataset. The top row displays the original colonoscopy image. The bottom row presents reconstructed point cloud generated by Open3D library.
Figure 7: Visualization of monocular depth estimation on colorectal polyp scenes of different methods, evaluated in a zero-shot setting without any fine-tuning. Images are frames extracted from the Polyp-Size dataset, a high-resolution colonoscopy video collection featuring colorectal polyps of diverse morphologies. While EcoDepth struggled with visible grid-like artifacts and Marigold tended to dissolve polyp boundaries into the surrounding mucosa, Depth Anything V2 recovered a more coherent global layout yet still lost the three-dimensional relief of lesions. SpecDepth preserved the convex profile of each polyp with well-defined boundary transitions and smoothly varying surface gradients, demonstrating more robust generalization across diverse polyp morphologies.
Figure 8: Visualization of monocular depth estimation on the SERV-CT dataset of different methods, evaluated in a zero-shot setting without any fine-tuning. SERV-CT is a laparoscopic stereo-endoscopic dataset capturing ex vivo porcine abdominal tissue. Ground-truth depth maps are derived by aligning cone-beam CT scans with the corresponding stereo endoscope images through constrained manual registration. Each predicted depth map is paired with an absolute "Prediction-GT" difference map, where deep amber regions highlight large errors, and white areas indicate minimal error. SpecDepth produced difference maps with significantly more white areas compared to the other baselines, demonstrating superior zero-shot generalization and accuracy in complex endoscopic scenes.
Monocular depth estimation (MDE) for colonoscopy is hampered by the domain gap between simulated and real-world images. Existing image-to-image translation methods, which use depth as a posterior constraint, often produce structural distortions and specular highlights by failing to balance realism with structure consistency. To address this, we propose a Structure-to-Image paradigm that transforms the depth map from a passive constraint into an active generative foundation. We are the first to introduce phase congruency to colonoscopic domain adaptation and design a cross-level structure constraint to co-optimize geometric structures and fine-grained details like vascular textures. In zero-shot evaluations conducted on a publicly available phantom dataset, the MDE model that was fine-tuned on our generated data achieved a maximum reduction of 44.18% in RMSE compared to competing methods. Our code is available at https://github.com/YyangJJuan/PC-S2I.git.
Juan Yang, Yuyan Zhang, Han Jia +2
College of Computer Science, Sichuan University, Chengdu, China · Department of Gastroenterology and Hepatology, West China Hospital, Sichuan University, Chengdu, China
Accurate polyp segmentation in colonoscopy is essential for early colorectal cancer detection, yet real-world clinical environments pose persistent challenges such as motion blur, specular reflections, and illumination instability. Most existing methods are optimized on clean benchmark images and suffer noticeable performance degradation when deployed in authentic surgical scenarios. We propose DepthPolyp, a lightweight and robust segmentation framework based on pseudo-depth-guided multi-task learning and efficient feature modulation. The architecture combines hierarchical Ghost factorization for compact feature generation, Interleaved Shuffle Fusion for low-cost cross-scale interaction, and Dynamic Group Gating for adaptive group-wise feature weighting. Extensive experiments demonstrate that DepthPolyp achieves strong cross-dataset generalization when trained on degraded data and evaluated on both clean and noisy target domains, consistently outperforming lightweight baselines and remaining competitive with substantially larger models. In real surgical video evaluation on PolypGen, DepthPolyp achieves better segmentation performance than models up to 20× larger while preserving real-time inference speed. With only 3.57M parameters and 0.86 GMACs, the proposed method runs at over 180 FPS on mobile devices, making it well suited for real-time deployment in resource-constrained clinical environments. Code and pretrained weights are available at: https://github.com/ReaganWu/DepthPolyp/
Zhuoyu Wu, Wenhui Ou, Lexi Zhang +5
CyPhi AI Lab, Monash University, Malaysia Campus, Malaysia · Department of Electronic & Computer Engineering, Hong Kong University of Science & Technology, Hong Kong, P.R. China · Harbin Institute of Technology, Harbin, P.R. China +1
Depth estimation is a foundational component for 3D reconstruction in minimally invasive endoscopic surgeries. However, existing monocular depth estimation techniques often exhibit limited performance to the varying illumination and complex textures of the surgical environment. While applying foundation models offers a promising approach to enhance the depth estimation performance, the domain gap between the natural images used for pre-training and the target endoscopic images leads to significant semantic perception deficiencies. In this study, EndoUFM is introduced as an unsupervised monocular depth estimation framework that innovatively Utilizes dual Foundation Models for Endoscopic images, thereby enhancing the depth estimation performance by leveraging the powerful pre-learned priors. The framework features a novel adaptive fine-tuning strategy that incorporates Random Vector Low-Rank Adaptation (RVLoRA) to enhance model adaptability, and a Residual block based on Depthwise Separable Convolution (Res-DSC) to improve the capture of fine-grained local features. A mask-guided smoothness loss is also introduced to enforce depth consistency within anatomical structures. Extensive experiments on the SCARED, Hamlyn, SERV-CT, and EndoNeRF datasets confirm that our method achieves state-of-the-art performance while maintaining an efficient model size. This work contributes to augmenting surgeons' spatial perception during minimally invasive procedures, thereby enhancing surgical precision and safety, with crucial implications for augmented reality and navigation systems. Our code is available at https://github.com/RealMindyY/EndoUFM.
Xinning Yao, Bo Liu, Bojian Li +3
Image Processing Center, Beihang University, Beijing, 100191, China. · State Key Laboratory of High-Efficiency Reusable Aerospace Transportation Technology, Beijing 102206, China.