Lightweight Prompt-Guided CLIP Adaptation for Monocular Depth Estimation
Authors: Reyhaneh Ahani Manghotay, Jie Liang
Organizations: Simon Fraser University, Burnaby, Canada · School of Engineering Science, Simon Fraser University, Burnaby, BC, Canada · Eastern Institute of Technology, Ningbo, China · School of Information Science and Technology, Eastern Institute of Technology, Ningbo, China
Leveraging the rich semantic features of vision-language models (VLMs) like CLIP for monocular depth estimation tasks is a promising direction, yet often requires extensive fine-tuning or lacks geometric precision. We present a parameter-efficient framework, named MoA-DepthCLIP, that adapts pretrained CLIP representations for monocular depth estimation with minimal supervision. Our method integrates a lightweight Mixture-of-Adapters (MoA) module into the pretrained Vision Transformer (ViT-B/32) backbone combined with selective fine-tuning of the final layers. This design enables spatially-aware adaptation, guided by a global semantic context vector and a hybrid prediction architecture that synergizes depth bin classification with direct regression. To enhance structural accuracy, we employ a composite loss function that enforces geometric constraints. On the NYU Depth V2 benchmark, MoA-DepthCLIP achieves competitive results, significantly outperforming the DepthCLIP baseline by improving the δ1 accuracy from 0.390 to 0.745 and reducing the RMSE from 1.176 to 0.520. These results are achieved while requiring substantially few trainable parameters, demonstrating that lightweight, prompt-guided MoA is a highly effective strategy for transferring VLM knowledge to fine-grained monocular depth estimation tasks.
Figures & tables
Figure 1: Overall architecture of MoA-DepthCLIP. Scene prompts are encoded using a frozen CLIP text encoder to form a global scene context vector. The image is encoded with a frozen ViT-B/32 backbone augmented with MoAs. Fused features are then passed to a dual-head prediction module: one head performs depth bin classification and produces a binned depth map via weighted summation, while the other head performs direct regression. The final output depth map is a fusion of both predictions.
Figure 2: Overview of our lightweight adaptation strategy. (a) Illustrates the selective placement of MoA modules within the ViT backbone. (b) Details the internal architecture of a single MoA module.
Method
Backbone
Comp. Loss
MoA
#Bins
δ1↑
δ2↑
δ3↑
AbsRel ↓
log10 ↓
RMSE ↓
DepthCLIP
ResNet-50
10
0.390
0.680
0.848
0.393
0.158
1.176
Our Baseline
ViT-B/32
10
0.417
0.701
0.890
0.377
0.147
1.096
+ Composite Loss
ViT-B/32
✓
10
0.503
0.791
0.925
0.310
0.121
0.843
+ MoA
ViT-B/32
✓
✓
10
0.508
0.797
0.931
0.308
0.120
0.821
MoA-DepthCLIP
ViT-B/32
✓
✓
128
0.745
0.841
0.895
0.321
0.098
0.520
Table 1: Quantitative results on NYU Depth V2. We evaluate different configurations by progressively introducing composite loss, MoA, and increasing the number of bins. The final configuration, MoA-DepthCLIP (with ViT-B/32, MoA, composite loss, and 128 bins), achieves the best performance across all metrics.
#Experts
AbsRel ↓
RMSE ↓
δ1↑
1
0.394
0.560
0.660
2
0.396
0.559
0.661
4
0.396
0.561
0.661
8
0.395
0.557
0.665
16
0.395
0.558
0.661
Table 2: Ablation on the number of experts, with 10 bins. K=4 is chosen as a balanced trade-off.
#Bins
AbsRel ↓
RMSE ↓
δ1↑
40
0.327
0.522
0.737
64
0.329
0.534
0.730
128
0.321
0.521
0.745
180
0.323
0.532
0.735
200
0.329
0.541
0.717
Table 3: Ablation on the number of depth bins, with 4 experts. N=128 provides the best performance.
We propose PureCLIP-Depth, a completely prompt-free, decoder-free Monocular Depth Estimation (MDE) model that operates entirely within the Contrastive Language-Image Pre-training (CLIP) embedding space. Unlike recent models that rely heavily on geometric features, we explore a novel approach to MDE driven by conceptual information, performing computations directly within the conceptual CLIP space. The core of our method lies in learning a direct mapping from the RGB domain to the depth domain strictly inside this embedding space. Our approach achieves state-of-the-art performance among CLIP embedding-based models on both indoor and outdoor datasets. The code used in this research is available at: https://github.com/ryutaroLF/PureCLIP-Depth
Robotic and autonomous systems need dense spatial cues, yet adding a dedicated depth estimator can duplicate visual processing already performed by a multimodal model. CLIP-based depth methods offer an alternative, but commonly rely on text-derived conditioning or backbone adaptation. We present SPACE-CLIP, a decoder-only framework for supervised monocular depth estimation with a frozen CLIP vision backbone and no text encoder at inference. A FiLM-conditioned semantic pathway combines global image context with multilevel patch features, while a structural pathway supplies separately processed spatial features to a hierarchical fusion decoder. Indoor and outdoor evaluations demonstrate depth reconstruction with this architecture, and controlled component comparisons support the contribution of the structural pathway. Layer-selection experiments and frequency interventions further characterize the structural pathway's contribution to depth reconstruction. A shared-backbone microbenchmark further illustrates the reduction in duplicated computation. SPACE-CLIP provides a modular approach to adding dense depth prediction to compatible visual perception stacks. Code is available at https://github.com/taewan2002/SPACE-CLIP.
Taewan Cho, Taeryang Kim, Andrew Jaeyong Choi
School of Computing, Gachon University, Republic of Korea
Vision-language foundation models such as CLIP provide strong semantic representations, but their patch tokens are not directly optimized for dense metric geometry. SPACE-CLIP showed that frozen CLIP features can support monocular depth estimation through layer-group feature fusion, yet it leaves open how neighboring CLIP tokens should be combined to recover fine local structure. We present SPACE-CLIPv2, a frozen-backbone depth decoder that aggregates fixed local neighborhoods in CLIP token space. At selected decoder stages, the model samples a fixed token stencil, predicts aggregation weights, and injects the resulting response through a gated residual update. A token-space high-pass branch further preserves shallow local contrast. On NYU Depth V2, SPACE-CLIPv2 improves over a matched SPACE-CLIP baseline, while five-seed experiments consistently favor fixed over learned-offset sampling. Zero-shot iBims-1 evaluation further improves boundary and planar-geometry measures. These results support constrained local token aggregation as a practical mechanism for decoding geometry from frozen CLIP representations.
Hyun Song, Taewan Cho, Kangmin Kim +1
School of Computing, iRASC Lab., Gachon University