A Contrastive Learning Framework Empowered by Attention-based Feature Adaptation for Street-View Image Classification

Qi You; Yitai Cheng; Zichao Zeng; James Haworth

cs.CVFeb 18, 2026

A Contrastive Learning Framework Empowered by Attention-based Feature Adaptation for Street-View Image Classification

Authors: Qi You, Yitai Cheng, Zichao Zeng, James Haworth

Organizations: 3DIMPact & SpaceTimeLab, University College London

Abstract

Street-view image attribute classification is a vital downstream task of image classification, enabling applications such as autonomous driving, urban analytics, and high-definition map construction. It remains computationally demanding whether training from scratch, initialising from pre-trained weights, or fine-tuning large models. Although pre-trained vision-language models such as CLIP offer rich image representations, existing adaptation or fine-tuning methods often rely on their global image embeddings, limiting their ability to capture fine-grained, localised attributes essential in complex, cluttered street scenes. To address this, we propose CLIP-MHAdapter, a variant of the current lightweight CLIP adaptation paradigm that appends a bottleneck MLP equipped with multi-head self-attention operating on patch tokens to model inter-patch dependencies. With approximately 1.4 million trainable parameters, CLIP-MHAdapter achieves superior or competitive accuracy across eight attribute classification tasks on the Global StreetScapes dataset, attaining new state-of-the-art results while maintaining low computational cost. The code is available at https://github.com/SpaceTimeLab/CLIP-MHAdapter.

Figures & tables

Figure 1: The CLIP-MHAdapter framework. A visual MHAdapter module is integrated downstream of the pretrained image encoder, enabling task-specific adaptation while preserving the representational strengths of the pretrained CLIP backbone [ 20 ] .

Figure 2: Details of the Multi-Head Feature Adaptation Module. Note that the input images are partitioned into standard 16 × 16 patches before being fed into the Vision Transformer encoder [ 8 ] . A larger patch size is shown here solely for clarity of illustration.

Attribute	No. of classes	Number of images		Possible categories
		train	test
Platform	6	25,407	7,311	driving/walking/cycling surface, railway, fields, tunnel
Weather	5	27,771	8,068	clear, cloudy, rainy, snowy, foggy
View direction	2	7,632	2,012	front/back, side
Lighting condition	3	9,380	3,079	day, night, dusk/dawn
Panoramic status	2	8,372	2,172	true, false

Table 1: Overview of the manually labelled subset of GSS dataset [ 14 ] .

Attribute	Class Weights	Blend Ratio	No. of Attention Heads
Platform	Uniform	0.8	4
Weather	Uniform	0.2	8
View direction	Inverse	0.8	4
Lighting condition	Inverse	0.8	8
Panoramic status	Uniform	0.2	4
Quality	Inverse	0.2	16

Table 2: Optimal hyperparameters combinations for CLIP-MHAdapter per Attribute.

Symbol	Definition
$K$	Total number of classes
$P_{i}$	Precision for class $i$
$R_{i}$	Recall for class $i$
$n_{i}$	Number of true samples in class $i$
$N_{s}$	Total number of samples across all classes

Table 3: Symbol glossary for evaluation metric formulas.

Contextual Attribute	Paradigm	Model	# T. Params	Acc.	Macro F1	Weighted F1	Bal. Acc.
Glare	Zero-shot Transfer	ZeroR-Trainer	–	97.21	49.29	95.84	0.00
	Zero-shot Transfer	Zero-shot CLIP	–	3.03	2.96	0.62	0.24
	Vision Transformer	MaxViT	30.9M	94.09	63.15	95.03	39.59
	Parameter-Efficient Adaptation	CLIP-Linear Probe	3K	95.51	53.61	95.24	6.48
		CoOp	8K	96.60	57.27	95.98	10.89
		CLIP-Adapter	0.52M	84.16	53.65	89.16	39.26

Table 4: Performance across contextual attributes organised by model paradigm described in 5.1 . # T. Params denotes the total number of trainable parameters. All metrics are reported as percentages.

Panel of Figure 3 — Figure 3: Qualitative results of CLIP-MHAdapter. The attention maps from the MHSA layer in the Visual MHAdapter are overlaid on the original input images for visualization.

Figure 4: Confusion matrices of CLIP-MHAdapter on the GSS test set across eight attributes classification tasks. Each matrix corresponds to one attribute.

Figure 5: The class distribution for each SVI attribute of labelled GSS dataset [ 14 ] .

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 545 papers

LAS-CLIP: A Lightweight Adapter Steering Approach for CLIP's Visual Encoder
Oct 2, 2026Anh-Khoa Dinh-Duc, Duc-Tai Dinh, Tam V. Nguyen +1Referring Expression Comprehension VLM Adaptation
University of Science, Ho Chi Minh City, Vietnam · Viet Nam National University, Ho Chi Minh City, Vietnam · University of Dayton, USA
Local-Global Prompt Learning via Sparse Optimal Transport
Mar 9, 2026Deniz Kizaroğlu, Ülku Tuncer Küçüktas, Emre Çakmakyurdu +1VLM Adaptation Visual Prompt Learning
Graduate School of Informatics, Middle East Technical University, Ankara, Turkey · Department of Electrical and Electronics Engineering, Gazi University, Ankara, Turkey
What Does CLIP Learn for Regional Geolocalization? Probing Visual Cues and Scene Configuration After Adaptation
Aug 22, 2026Changyu Lee, Yeonsoo Park, Abdullah Alfarrarjeh +1Geospatial Representation Learning Visual Place Recognition
University of Southern California Los Angeles, USA · USC IMSC San Francisco, USA · German Jordanian University Amman, Jordan +1

Oct 2, 2026J/K move · Enter open · S save

LAS-CLIP: A Lightweight Adapter Steering Approach for CLIP's Visual Encoder

Anh-Khoa Dinh-Duc, Duc-Tai Dinh, Tam V. Nguyen, Minh-Triet Tran

University of Science, Ho Chi Minh City, Vietnam · Viet Nam National University, Ho Chi Minh City, Vietnam · University of Dayton, USA

Referring Expression Comprehension VLM Adaptation Zero-Shot Image Classification Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

CLIP's visual encoder produces only global image representations, limiting its use in region-level tasks. Existing adaptations rely on visual prompting, input masking, or encoder fine-tuning, each compromising pre-trained representations. We propose LAS-CLIP, a Lightweight Adapter Steering approach that keeps every CLIP parameter frozen. A compact MaskAdapter generates per-head, per-layer attention biases from an input mask and injects them into the frozen self-attention layers, steering attention toward the target region. Crucially, because the backbone remains strictly untouched, LAS-CLIP seamlessly reverts to vanilla CLIP when no mask is provided, preserving its foundational zero-shot capabilities. With approximately 116K to 145K trainable parameters and 100K training samples on two T4 GPUs, LAS-CLIP achieves competitive or superior results compared to Alpha-CLIP on ImageNet-S zero-shot classification and RefCOCO referring expression comprehension, despite the latter fine-tuning its entire encoder on millions of samples. Qualitative analysis further confirms stronger representational fidelity under incorrect masks and in downstream generation.

Abstract

VLM Adaptation

Explore similar work

LAS-CLIP: A Lightweight Adapter Steering Approach for CLIP's Visual Encoder

Local-Global Prompt Learning via Sparse Optimal Transport

What Does CLIP Learn for Regional Geolocalization? Probing Visual Cues and Scene Configuration After Adaptation