A Multihead Continual Learning Framework for Fine-Grained Fashion Image Retrieval with Contrastive Learning and Exponential Moving Average Distillation

Ling Xiao; Toshihiko Yamasaki

cs.CVMar 21, 2026

A Multihead Continual Learning Framework for Fine-Grained Fashion Image Retrieval with Contrastive Learning and Exponential Moving Average Distillation

Authors: Ling Xiao, Toshihiko Yamasaki

Organizations: Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan

Abstract

Most fine-grained fashion image retrieval (FIR) methods assume a static setting, requiring full retraining when new attributes appear, which is costly and impractical for dynamic scenarios. Although pretrained models support zero-shot inference, their accuracy drops without supervision, and no prior work explores class-incremental learning (CIL) for fine-grained FIR. We propose a multihead continual learning framework for fine-grained fashion image retrieval with contrastive learning and exponential moving average (EMA) distillation (MCL-FIR). MCL-FIR adopts a multi-head design to accommodate evolving classes across increments, reformulates triplet inputs into doublets with InfoNCE for simpler and more effective training, and employs EMA distillation for efficient knowledge transfer. Experiments across four datasets demonstrate that, beyond its scalability, MCL-FIR achieves a strong balance between efficiency and accuracy. It significantly outperforms CIL baselines under similar training cost, and compared with static methods, it delivers comparable performance while using only about 30% of the training cost. The source code is publicly available in https://github.com/Dr-LingXiao/MCL-FIR.

Figures & tables

Fig. 1: The difference between MCL-FIR and SOTA static learning methods.

Fig. 2: Overall pipeline of the proposed MCL-FIR model.

Attn_{\{0\sim\tilde{A}-1\}}

denote the respective attention module for each task (attribute). In this paper,

\widetilde{A}

is set to

22

to accommodate the eight attributes in FashionAI, five in DeepFashion, and nine in DARN. If the input attribute is “skirt-length,” it can be split into two words, with

a_{i}^{1}

denoting “skirt,” and

a_{i}^{2}

denoting “length.”

t(a_{i}^{1})

and

t(a_{i}^{2})

represent the corresponding text embeddings, extracted using a pre-trained text encoder.

Fig. 3: The detailed structure of

Attn_{\{0\sim\tilde{A}-1\}}

. Note that

Attn_{0}

Attn_{1}

, …, and

Attn_{\tilde{A}-1}

share the same structure, as shown in this figure.

Fig. 4: We implemented two CIL baselines for comparison: an ER method and a multi-head method.

	Methods	MAP for each attribute (Fashion AI) $\uparrow$								MAP $\uparrow$
	Methods	Skirt -length	Sleeve -length	Coat -length	Pant -length	Collar -design	Lapel -design	Neckline -design	Neck -design	MAP $\uparrow$
Static methods	CSN [ 18 ]	59.59	43.99	42.82	62.80	67.57	48.10	41.73	55.29	50.80
	ASENet_V2 [ 19 ]	64.57	54.96	51.76	64.50	71.93	66.72	60.29	60.83	60.76
	ASEN ${g}$ [ 20 ]	59.95	50.72	47.58	64.32	68.66	59.39	55.06	54.73	56.47
	ASEN [ 20 ]	63.04	53.05	52.29	65.26	71.81	61.39	63.87	65.25	61.00
	ASENet_V2+PT [ 23 ]	67.50	60.52	55.20	70.58	77.35	72.31	68.31	67.28	66.29

TABLE I: Comparisons with the ER and multi-head CIL baselines and SOTA static methods. Results are reported as

A,(B)

, where

A

is the final accuracy after all sequential updates and

B

is the accuracy immediately after learning each attribute.

Panel of Figure 6 — Fig. 5: The t-SNE visualization on the FashionAI dataset shows that MCL-FIR effectively captures subtle differences between sub-classes within each attribute.

Methods	MAP for each attribute (Fashion AI)								MAP $\uparrow$
Methods	Skirt -length	Sleeve -length	Coat -length	Pant -length	Collar -design	Lapel -design	Neckline -design	Neck -design	MAP $\uparrow$
Multi-head-based ( $L_{\rm triplet}$ )	29.02	17.28	22.03	30.99	30.93	27.11	17.31	26.28	23.72
Multi-head-based + ( $L_{\rm triplet}$ + $L_{\text{kd}}$ )	60.80	51.05	49.56	64.51	67.69	65.56	60.99	60.44	58.92
MCL-FIR( $L_{\text{ins}}$ )	31.16	19.92	21.21	31.89	29.80	28.37	16.54	26.52	24.27
MCL-FIR( $L_{\text{ins}}$ + $L_{\text{kd}}$ )	64.28	58.68	53.28	68.56	70.75	68.17	67.65	61.02	63.52

TABLE III: Ablation studies conducted with 10 training epochs and 20,000 training doublets.

Methods	#Batch size	MAP for each attribute (Fashion AI)								MAP $\uparrow$
Methods	#Batch size	Skirt -length	Sleeve -length	Coat -length	Pant -length	Collar -design	Lapel -design	Neckline -design	Neck -design	MAP $\uparrow$
MCL-FIR (Ours)	4	56.87	49.10	46.63	58.96	62.49	55.40	54.86	53.28	54.01
	8	62.37	58.13	52.87	64.94	67.46	64.85	63.65	58.41	61.16
	16	63.72	58.29	52.94	66.82	69.84	67.56	67.01	63.26	63.04
	32	64.28	58.68	53.28	68.56	70.75	68.17	67.65	61.02	63.52
	64	61.05	58.92	51.93	65.89	69.74	67.78	67.07	60.57	62.40

TABLE VI: Analysis on the impact of different batch sizes during training, conducted over 10 epochs with 20,000 training doublets.

Fine-Grained Image Retrieval

Fine-grained image retrieval finds visually similar images by matching subtle attributes, parts, instances, or user-specified changes.

See all 43 papers

Decoupled and Distilled: Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer for Few-Shot Class-Incremental Learning
Sep 30, 2026Hongwei Zhao, Rui Liu, Yansong Liu +2Class-Incremental Learning Low-Rank Adaptation
School of Computer Science and Engineering, Beihang University · School of Computer Science, Beijing University of Posts and Telecommunications
FashionLens: Toward Versatile Fashion Image Retrieval via Task-Adaptive Learning
May 21, 2026Haokun Wen, Xuemeng Song, Xinghao Xie +3Multi-Task Learning Image-Text Retrieval
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen 518055, China · Department of Data Science, City University of Hong Kong, Hong Kong, China · Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China +4
Exploring Multi-Modal Large Language Models and Two-Stage Fine-Tuning for Fashion Image Retrieval
Jun 18, 2026Nguyen Cao Hoang, Hoang Bui Le, Nam Vo Hoang +1Multimodal Large Language Models Image-Text Retrieval
University of Science, VNU-HCM, Ho Chi Minh, Vietnam · Vietnam National University, Ho Chi Minh, Vietnam

Sep 30, 2026J/K move · Enter open · S save

Decoupled and Distilled: Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer for Few-Shot Class-Incremental Learning

Hongwei Zhao, Rui Liu, Yansong Liu, Zhiyuan Zou+1

School of Computer Science and Engineering, Beihang University · School of Computer Science, Beijing University of Posts and Telecommunications

Class-Incremental Learning Low-Rank Adaptation Multi-Teacher Knowledge Distillation Few-Shot Image Classification Knowledge Distillation

Open paper AskPDF arXiv

Few-Shot Class-Incremental Learning (FSCIL) addresses the challenge of learning new classes from very limited samples while retaining knowledge of previously learned ones. Although parameter-efficient fine-tuning methods with pre-trained models show promise for class-incremental learning, strict gradient-based constraints can be unreliable under severe data scarcity, while multi-expert approaches can impose substantial inference-time costs. We propose TALON (Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer), an inference-efficient FSCIL framework. TALON dynamically allocates an independent LoRA-Teacher to each incremental task for task-specific representation learning, then distills multiple frozen teachers into a unified LoRA-Student through Ensemble Knowledge Transfer, eliminating runtime module selection or generation. A semantic-guided distillation strategy weights teacher contributions by feature-space similarity to mitigate catastrophic forgetting and overfitting. Across three class-order runs, TALON achieves comparable or better mean average accuracy across four FSCIL benchmarks, obtaining 86.68 +/- 1.22% on CUB200, 90.39 +/- 0.27% on CIFAR100, 78.38 +/- 0.94% on ImageNet-R, and 96.34 +/- 0.33% on miniImageNet. TALON uses up to 33x fewer deployment parameters and reduces average inference time per task to 26.7 s, a 41.70% reduction relative to ASP.

Abstract

Fine-Grained Image Retrieval

Explore similar work

Decoupled and Distilled: Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer for Few-Shot Class-Incremental Learning

FashionLens: Toward Versatile Fashion Image Retrieval via Task-Adaptive Learning

Exploring Multi-Modal Large Language Models and Two-Stage Fine-Tuning for Fashion Image Retrieval