Language Prompt vs. Image Enhancement: Boosting Object Detection With CLIP in Hazy Environments
Authors: Jian Pang, Bingfeng Zhang, Jin Wang, Baodi Liu, Dapeng Tao, Weifeng Liu
Organizations: College of Control Science and Engineering, China University of Petroleum (East China), Qingdao 266580, Shandong, P.R. China · School of Information Science and Engineering, Yunnan University, Kunming 650091, Yunnan, P.R. China · Yunnan United Vision Technology Co., Ltd., Kunming 650504, Yunnan, P.R. China
Object detection in hazy environments is challenging because degraded objects are nearly invisible and their semantics are weakened by environmental noise, making it difficult for detectors to identify. Common approaches involve image enhancement to boost weakened semantics, but these methods are limited by the instability of enhanced modules. This paper proposes a novel solution by employing language prompts to enhance weakened semantics without image enhancement. Specifically, we design Approximation of Mutual Exclusion (AME) to provide credible weights for Cross-Entropy Loss, resulting in CLIP-guided Cross-Entropy Loss (CLIP-CE). The provided weights assess the semantic weakening of objects. Through the backpropagation of CLIP-CE, weakened semantics are enhanced, making degraded objects easier to detect. In addition, we present Fine-tuned AME (FAME) which adaptively fine-tunes the weight of AME based on the predicted confidence. The proposed FAME compensates for the imbalanced optimization in AME. Furthermore, we present HazyCOCO, a large-scale synthetic hazy dataset comprising 61258 images. Experimental results demonstrate that our method achieves state-of-the-art performance. The code and dataset will be released.
Figures & tables
Fig. 1: Pipeline comparison. Our method can detect overlooked objects using only language prompts.
Fig. 2: Failure cases of IA-YOLO. GT: Ground Truth. Involving image enhancement may lead to missed detections or misclassifications.
Fig. 3: Overall of our pipeline. In detection pipeline: The image is processed to obtain ROI features. The predicted logits of objects are combined with estimated weights to form CLIP-CE. In AME: We crop objects within the image and extract their visual embeddings. Meanwhile, we get text embeddings from mutually exclusive prompts. We calculate the negative similarity between these embeddings to determine the weight of AME. In FAME: We concatenate the ROI features with the visual embeddings to create adapted visual embeddings. We use negative similarities from adapted visual embeddings as weight offset. The final weight for FAME is averaged from the AME weight and the weight offset.
Dataset
Train
Test
Total
Environment
HazyCOCO
58800
2458
61258
Synthetic Haze
RTTS
2161
2161
4,322
Real Haze
Exdark
4411
2952
7363
Low light
TrashCan
5936
1147
7083
Underwater
TABLE I: Image statistics of the datasets.
Fig. 4: Left part: Using the clear image and its depth map to generate HazyCOCO image. Right part: Hazy images from VOC FOG [ 10 ] .
Fig. 5: Visualizations of the depth map with and without clamping. The depth map without clamping is directly generated by Depth-Anything. The HazyCOCO images are synthesized using the depth maps described above. In the depth map with clamping, the area within the orange box exhibits more hierarchical depth information.
Fig. 6: Visualizations for the HazyCOCO images with different hazy densities.
Strategy
Method
Detector
HazyCOCO
RTTS
Separation
Advancing [ 13 ]
Faster-RCNN
36.77
70.29
DCP [ 41 ]
42.82
74.26
Fusion
IA-YOLO [ 10 ]
YOLOV3
29.27
59.20
TogetherNet [ 12 ]
YOLOX
32.31
54.00
Direct
Focal Loss [ 16 ]
Faster-RCNN
39.67
70.20
Ours
44.92
76.76
TABLE II: Performance comparison in hazy environment. Bold is the best. Results are reported by mAP.
Strategy
Method
Detector
Exdark
TrashCan
Separation
RUSA [ 44 ]
Faster-RCNN
61.12
-
Zero-DCE [ 45 ]
68.77
-
UHDFour [ 46 ]
58.85
-
HCLR [ 47 ]
-
48.23
UFormer [ 48 ]
-
46.57
Fusion
IA-YOLO [ 10 ]
YOLOV3
51.50
-
TABLE III: Performance comparison in low-light and underwater environments. Bold is the best. Results are reported by mAP. “-” means that the method is not suitable for the environment.
Method
Cross-Entropy Loss (CE)
CLIP-CE
Deformable-DETR [ 32 ]
64.42
65.36
RT-DETR [ 31 ]
74.61
76.48
Faster-RCNN [ 30 ]
74.12
76.76
TABLE IV: Performance comparison of different detectors with and without the proposed CLIP-CE loss on the RTTS dataset.
Methods
HazyCOCO (synthetic hazy)
RTTS (real hazy)
ExDark (real dark)
TrashCan (real underwater)
CE (baseline)
42.40
74.12
68.59
52.03
CE w/ AME
44.58
75.91
69.77
58.52
CE w/ FAME
44.92
76.76
70.42
59.63
TABLE V: Ablation study. CE: Cross-Entropy loss. AME: Approximation of Mutual Exclusion. FAME: Fine-tuned AME. “CE w/ AME” means CE with the weight from AME, while “CE w/ FAME” means CE weighted by FAME. Bold is the best. Results are reported by mAP.
Fig. 7: Visualization for the AME weights and Focal weights. The AME weights are the similarity between object and negative prompt (“a photo without { cls }”). The focal weight is computed from the predicted logit of object.
Fig. 8: Visualization of detection results on HazyCOCO and RTTS datasets. We recommend examining the zoomed-in areas (highlighted with red dashed boxes) for a clearer comparison. GT: ground truth.
Fig. 9: Visualization of detection results. We recommend examining the zoomed-in areas for a clearer comparison. GT: ground truth. IA-YOLO, UHDFour and UFormer are image enhancement-based methods. Focal Loss refers to using Faster-RCNN with Focal loss. Ours is using Faster-RCNN with CLIP-CE.
Prompt
RTTS
AME
FAME
“a photo of a { cls }”
75.91
76.76
“a photo without { cls }”
“a foggy photo of a { cls }”
75.22
76.30
“a foggy photo without { cls }”
“it is a { cls }”
75.39
75.48
TABLE VI: Prompt comparison on the real hazy dataset RTTS. The first prompt is used in our paper. The second prompt additionally has environmental factors. The third prompt is similar to ours. Bold is the best.