Semantic Modality Compensation for Unsupervised Visible-Infrared Person Re-identification under Unpaired Settings

Duanning Chen; Ke He; Bin Yang; Yongxiang Yao

cs.CVSep 28, 2026

Semantic Modality Compensation for Unsupervised Visible-Infrared Person Re-identification under Unpaired Settings

Authors: Duanning Chen, Ke He, Bin Yang, Yongxiang Yao

Organizations: Wuhan University

Abstract

Unsupervised visible-infrared person re-identification (USL-VI-ReID) learns person representations that can be compared across modalities without identity annotations. In the unpaired setting, however, identity correspondences between modalities are often incomplete, leaving many identities without an observed counterpart in the other modality. Existing unpaired methods bridge this gap by generating or mapping features for the other modality, mainly by exploiting the statistics of visual features without explicitly separating content that is discriminative for identity from style that is specific to modality. Consequently, the generated features may distort identity cues or inherit bias from the source modality, undermining the reliability of supervision across modalities. We formulate unpaired learning across modalities as a semantic compensation problem and propose Semantic Modality Compensation (SMC), a framework based on prompt composition that decouples identity semantics from modality style within a shared visual semantic space. SMC first constructs a discriminative ReID space through augmented dual contrastive learning, yielding pseudo labels, cluster prototypes, and memory banks for each modality. It then learns visible and infrared modality prompts in the CLIP semantic space and maps clusters obtained from pseudo labels to identity semantic tokens. For each cluster lacking a reliable match in the other modality, SMC combines its identity token with the prompt for the target modality to synthesize a semantic counterpart in the missing modality. The synthesized counterpart is then projected back into the ReID space and injected into a compensation memory through confidence gating. Extensive experiments under both paired and unpaired settings demonstrate that SMC consistently outperforms state-of-the-art methods, with particularly large gains when identity mismatch is severe.

Figures & tables

Figure 1: Illustration of the motivation for SMC. In unpaired settings, missing cross-modality counterparts make direct transformation unreliable. SMC decouples identity content and modality style, and recombines them in frozen CLIP semantic space to generate reliable semantic counterparts.

Figure 2: Overview of SMC. ADC constructs modality-specific clusters and memories. The training-only semantic branch composes cluster identity tokens with a target-modality prompt and caches reliable outputs as auxiliary positives.

		SYSU-MM01						RegDB
		All Search			Indoor Search			Visible $\rightarrow$ Infrared			Infrared $\rightarrow$ Visible
Method	Venue	R1	mAP	mINP	R1	mAP	mINP	R1	mAP	mINP	R1	mAP	mINP
AGW ( Ye et al. 2022 )	TPAMI’22	47.50	47.65	35.30	54.17	62.97	59.23	70.05	66.37	50.19	70.49	65.90	51.24
DEEN ( Zhang and Wang 2023 )	CVPR’23	74.70	71.80	–	80.30	83.30	–	91.10	85.10	–	89.50	83.40	–
PartMix ( Kim et al. 2023 )	CVPR’23	77.78	74.62	–	81.52	84.38	–	85.66	82.27	–	84.93	82.52	–
MUN ( Yu et al. 2023 )	ICCV’23	76.24	73.81	–	79.42	82.06	–	95.19	87.15	–	91.86	85.01	–

Table 1: Comparison with supervised and unsupervised methods under paired settings on SYSU-MM01 and RegDB. Rank-1 (R1), mAP, and mINP (%) are reported. Best results within each supervision group are boldfaced; “–” denotes results not reported by the original paper.

\dagger

denotes SAAI with its reported affinity-inference protocol, and

\ddagger

denotes the camera-information-free GUR variant.

Figure 3: Embedding distributions of 20 randomly sampled identities on SYSU-MM01 at $\alpha=0.5$ . Circles and triangles denote visible and infrared samples, respectively.

		SYSU-MM01 (All)		SYSU-MM01 (Indoor)		RegDB (V2T)		RegDB (T2V)
$\boldsymbol{\alpha}$	Method	R1	mAP	R1	mAP	R1	mAP	R1	mAP
0.25	MMM	30.94	33.58	35.49	44.21	62.27	55.09	66.69	59.92
	N-ULC	52.21	48.67	54.33	62.10	60.76	56.82	68.31	66.06
	DLM	54.61	54.07	53.66	62.30	–	–	–	–
	MCL	55.29	55.82	61.46	68.95	65.97	63.53	76.12	67.83
	SMC	57.21	55.92	63.54	70.26	76.21	71.23	82.31	73.18

Table 2: Comparison under unpaired settings on SYSU-MM01 in the all search and indoor search modes and on RegDB in the V2T and T2V directions as

\alpha

varies. Rank-1 accuracy and mAP (%) are reported. All competing results are obtained by retraining the corresponding methods on our newly constructed splits, using the same split for every method at each value of

\alpha

. Methods are listed in ascending order of accuracy within each ratio, and every entry is the mean of three runs with different random seeds.

			Visible $\rightarrow$ Infrared		Infrared $\rightarrow$ Visible
	Method	Venue	R1	mAP	R1	mAP
	MMN	MM’21	59.90	62.70	52.50	58.90
	DEEN	CVPR’23	62.50	65.80	54.90	62.90
Sup.	DSFAD	TIFS’25	66.20	68.90	57.50	64.10
	ADCA	MM’22	40.32	45.68	35.51	42.29
	SDCL	CVPR’24	46.90	52.40	43.40	48.20

Table 3: Comparison on LLCM under the standard protocol. Rank-1 (R1) and mAP (%) are reported. Best results within each supervision group are boldfaced.

Figure 4: Class activation maps from ADC, MCL, and SMC on representative queries. Black boxes mark salient regions, and red arrows trace changes in attention from ADC through MCL to SMC.

Figure 5: Top six retrieval results for three representative queries under the unpaired setting. Results are shown for ADC, MCL, and SMC. Green and red boxes mark correct and incorrect matches, respectively.

	Components				SYSU-MM01 (All Search)			SYSU-MM01 (Indoor Search)			RegDB (Visible to Infrared)
Index	ADC	DMP	ISM	CSC	R1	mAP	mINP	R1	mAP	mINP	R1	mAP	mINP
1	$\checkmark$				41.36	40.02	27.14	47.15	55.28	50.94	55.83	53.16	41.27
2	$\checkmark$	$\checkmark$			44.26	43.22	30.54	50.25	58.08	53.94	58.53	55.66	43.47
3	$\checkmark$	$\checkmark$	$\checkmark$		48.43	47.82	35.44	54.65	62.08	58.24	62.53	59.26	46.67
4	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	55.62	55.74	43.78	62.21	69.03	65.82	69.42	65.55	52.28

Table 4: Ablation studies on SYSU-MM01 and RegDB under unpaired settings (

\alpha=0.5

). Rank 1 accuracy, mAP, and mINP (%) are reported. Row 1 is the ADC baseline retrained on the mismatched split, and is therefore below the paired ADCA result of Table 1 .

Explore similar work

Dual-Edged Homogeneous-Modality Similarity: Towards Visible-Infrared Modality-Incomplete Person Re-Identification with Modality Adaptive Matching
Jul 21, 2026Xin Xu, Shuhao Zhan, Wei Liu +3Re-Identification Modalities
School of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan, 430065, Hubei, China. · Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan University of Science and Technology, Wuhan, 430081, Hubei, China. · School of Computer Science, Wuhan University, Wuhan, 430079, Hubei, China. +2
Prototype Matters: Modality-unified Prototype Self-distillation for Unsupervised Visible-infrared Person Re-identification
Sep 11, 2026Menglin Wang, Xiaojin GongRe-Identification Cross-Modality
Nanjing Normal University Nanjing, China · Zhejiang University Hangzhou, China
Learning Language-Driven Sequence-Level Modal-Invariant Representations for Video-Based Visible-Infrared Person Re-Identification
Jan 17, 2026Xiaomei Yang, Antai Liu, Xizhan Gao +3Re-Identification Cross-Modal Attention
Shandong Key Laboratory of Ubiquitous Intelligent Computing, School of Information Science and Engineering, University of Jinan, Jinan 250022, China · College of Information Science and Technology & Artificial Intelligence, Nanjing Forestry University, Nanjing 210037, China · Department of Informatics, Modeling, Electronics, and Systems, University of Calabria, Via P. Bucci, cubo 41C, 87036, Rende (CS), Italy

Jul 21, 2026, cs.CVJ/K move · Enter open · S save

Dual-Edged Homogeneous-Modality Similarity: Towards Visible-Infrared Modality-Incomplete Person Re-Identification with Modality Adaptive Matching

Xin Xu, Shuhao Zhan, Wei Liu, Zheng Wang+2

School of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan, 430065, Hubei, China. · Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan University of Science and Technology, Wuhan, 430081, Hubei, China. · School of Computer Science, Wuhan University, Wuhan, 430079, Hubei, China.+2

Re-Identification Modalities

Open paper AskPDF arXiv

Visible-Infrared Person Re-Identification (VI-ReID) operates under a closed-world assumption, where queries and galleries are from heterogeneous modalities. However, in open-world scenarios, both sets are likely to contain homogeneous and heterogeneous modality images. A query may consist of visible-only, infrared-only, or mixed-modality images, while galleries present multi-modal images over long-term collection. Under these conditions, VI-ReID methods, built on a heterogeneous-modality retrieval paradigm, suffer from three trustworthiness challenges: matching conflicts due to high homogeneous-modality similarity, interference from modality uncertainty, and robustness degradation induced by unknown modality combinations. They fail to meet the requirements of trustworthy visual recognition in reliability, consistency, and dynamic adaptability. To address these challenges, we formalize the Visible-Infrared Modality-Incomplete Re-Identification (VIMI-ReID) task. We reorganize existing datasets to construct the SYSU-VIMI and RegDB-VIMI benchmarks. The unpredictable modality combinations and inherent similarity of homogeneous-modality samples in VIMI-ReID cause a significant performance drop in existing VI-ReID methods. We propose the Modality Adaptive Matching Transformer (MAMT). It employs a Divergence Transformer Module (DTM) and a Shared Transformer Module (STM) to extract modality-specific and modality-shared features, respectively. Guided by a divergence loss, the DTM enriches modality-specific features with modality-style information to enhance discriminability within the same modality. A Modality Adaptive Matching Module (MAM) dynamically fuses features according to the query-gallery modality relationship, enabling stable matching under arbitrary and uncertain modality conditions. Extensive experiments on the VIMI benchmarks demonstrate the effectiveness and adaptability of MAMT.

Abstract

Explore similar work

Dual-Edged Homogeneous-Modality Similarity: Towards Visible-Infrared Modality-Incomplete Person Re-Identification with Modality Adaptive Matching

Prototype Matters: Modality-unified Prototype Self-distillation for Unsupervised Visible-infrared Person Re-identification

Learning Language-Driven Sequence-Level Modal-Invariant Representations for Video-Based Visible-Infrared Person Re-Identification