Lightweight Prompt-Guided CLIP Adaptation for Monocular Depth Estimation
Authors: Reyhaneh Ahani Manghotay, Jie Liang
Organizations: Simon Fraser University, Burnaby, Canada · School of Engineering Science, Simon Fraser University, Burnaby, BC, Canada · Eastern Institute of Technology, Ningbo, China · School of Information Science and Technology, Eastern Institute of Technology, Ningbo, China
Leveraging the rich semantic features of vision-language models (VLMs) like CLIP for monocular depth estimation tasks is a promising direction, yet often requires extensive fine-tuning or lacks geometric precision. We present a parameter-efficient framework, named MoA-DepthCLIP, that adapts pretrained CLIP representations for monocular depth estimation with minimal supervision. Our method integrates a lightweight Mixture-of-Adapters (MoA) module into the pretrained Vision Transformer (ViT-B/32) backbone combined with selective fine-tuning of the final layers. This design enables spatially-aware adaptation, guided by a global semantic context vector and a hybrid prediction architecture that synergizes depth bin classification with direct regression. To enhance structural accuracy, we employ a composite loss function that enforces geometric constraints. On the NYU Depth V2 benchmark, MoA-DepthCLIP achieves competitive results, significantly outperforming the DepthCLIP baseline by improving the δ1 accuracy from 0.390 to 0.745 and reducing the RMSE from 1.176 to 0.520. These results are achieved while requiring substantially few trainable parameters, demonstrating that lightweight, prompt-guided MoA is a highly effective strategy for transferring VLM knowledge to fine-grained monocular depth estimation tasks.
Figures & tables
Figure 1: Overall architecture of MoA-DepthCLIP. Scene prompts are encoded using a frozen CLIP text encoder to form a global scene context vector. The image is encoded with a frozen ViT-B/32 backbone augmented with MoAs. Fused features are then passed to a dual-head prediction module: one head performs depth bin classification and produces a binned depth map via weighted summation, while the other head performs direct regression. The final output depth map is a fusion of both predictions.
Figure 2: Overview of our lightweight adaptation strategy. (a) Illustrates the selective placement of MoA modules within the ViT backbone. (b) Details the internal architecture of a single MoA module.
Method
Backbone
Comp. Loss
MoA
#Bins
δ1↑
δ2↑
δ3↑
AbsRel ↓
log10 ↓
RMSE ↓
DepthCLIP
ResNet-50
10
0.390
0.680
0.848
0.393
0.158
1.176
Our Baseline
ViT-B/32
10
0.417
0.701
0.890
0.377
0.147
1.096
+ Composite Loss
ViT-B/32
✓
10
0.503
0.791
0.925
0.310
0.121
0.843
+ MoA
ViT-B/32
✓
✓
10
0.508
0.797
0.931
0.308
0.120
0.821
MoA-DepthCLIP
ViT-B/32
✓
✓
128
0.745
0.841
0.895
0.321
0.098
0.520
Table 1: Quantitative results on NYU Depth V2. We evaluate different configurations by progressively introducing composite loss, MoA, and increasing the number of bins. The final configuration, MoA-DepthCLIP (with ViT-B/32, MoA, composite loss, and 128 bins), achieves the best performance across all metrics.
#Experts
AbsRel ↓
RMSE ↓
δ1↑
1
0.394
0.560
0.660
2
0.396
0.559
0.661
4
0.396
0.561
0.661
8
0.395
0.557
0.665
16
0.395
0.558
0.661
Table 2: Ablation on the number of experts, with 10 bins. K=4 is chosen as a balanced trade-off.
#Bins
AbsRel ↓
RMSE ↓
δ1↑
40
0.327
0.522
0.737
64
0.329
0.534
0.730
128
0.321
0.521
0.745
180
0.323
0.532
0.735
200
0.329
0.541
0.717
Table 3: Ablation on the number of depth bins, with 4 experts. N=128 provides the best performance.