MODAL: Multi-Modal Object Re-ID via Model-Driven Sparse Decoupling and Text-Image Differential Filtering

Chengbo Huang; Jun-Jie Huang; Long Lan; Tianrui Liu; Xueqiong Li; Yuanxi Peng; Xinwang Liu; Meng Wang

cs.CVAug 15, 2026

MODAL: Multi-Modal Object Re-ID via Model-Driven Sparse Decoupling and Text-Image Differential Filtering

Authors: Chengbo Huang, Jun-Jie Huang, Long Lan, Tianrui Liu, Xueqiong Li, Yuanxi Peng, Xinwang Liu, Meng Wang

Organizations: College of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China · School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230002, China

Abstract

Multi-modal object re-identification (Re-ID) aims to facilitate cross-camera object retrieval in complex environments by leveraging complementary information from visual (e.g., RGB, NIR, TIR) and textual modalities. However, existing approaches often lack principled feature disentanglement and coherent multi-modal integration, leading to entangled representations that introduce cross-modal conflicts, obscure discriminative cues, and suffer distribution shift under modality-missing conditions. To tackle these challenges, we propose MODAL, a novel multi-modal object re-identification framework, grounded in coupled sparse coding theory and differential suppression principles. A core component of MODAL is a Multi-modal Feature Sparse Decoupling module, developed in a model-driven deep unrolling manner based on multi-modal coupled sparse coding. It explicitly decomposes multi-modal features into uni-modal specific, bi-modal and tri-modal shared representations, thereby achieving more transparent and effective feature disentanglement. Benefiting from the principled feature disentanglement, MODAL naturally mitigates performance degradation in incomplete-modality scenarios via a Modality-Aware Subspace Activation that selectively activates only the consistently shared subspaces. Moreover, we propose a Text-Image Differential Filtering module that leverages coarse-grained textual semantics to adaptively suppress task-irrelevant responses in the decoupled visual representations, thereby enhancing discriminative information. Extensive experiments on four datasets demonstrate that MODAL achieves state-of-the-art performance with superior transparency.

Figures & tables

Fig. 1: Motivation and overview of MODAL. Left: Existing methods directly fuse entangled multi-modal features in a unified space, which introduces inter-modal interference and leads to significant performance degradation under missing-modality scenarios. Right: In contrast, MODAL mitigates these issues through sparse feature decoupling, Text-Image Differential Filtering, and modality-aware subspace activation, effectively reducing cross-modal conflict, enhancing representation discriminability, and enabling more robust matching under incomplete-modality scenarios.

Fig. 2: Overview of the proposed MODAL framework. Given multi-modal text and image inputs, their global and local features are extracted through CLIP encoders. These features are then sparsely decomposed into modality-specific and shared features via Multi-modal Feature Sparse Decoupling. Next, a Text-Image Differential Filtering module fuses the decomposed textual and visual features with local visual features to suppress task-irrelevant features and improve discriminative capability. Finally, a Mixture-of-Experts model selects and connects features to form the final representation.

Fig. 3: Architecture of the Uni-modal (UFD), Bi-modal (BFD), and Tri-modal (TFD) feature decoupling blocks. All three blocks utilize a unified iterative structure. Each block refines its corresponding sparse codes through

K

repeated stages of learnable transformations and soft thresholding.

Fig. 4: Architecture of the Text-Image Differential Filtering block.

Fig. 5: Schematic illustration of the Modality-Aware Subspace Activation (MASA) for incomplete modality inference..

Methods	Venue	RGBNT201
Methods	Venue	mAP	Rank-1	Rank-5
HAMNet [ 5 ]	AAAI-20	27.7	26.3	41.5
PFNet [ 6 ]	AAAI-21	38.5	38.9	52.0
IEEE [ 7 ]	AAAI-22	46.4	47.1	58.5
LRMM [ 48 ]	ESWA-25	52.3	53.4	64.6
TIENet [ 49 ]	TNNLS-25	54.4	54.4	66.3

TABLE I: Comparison of different methods on pedestrian dataset RGBNT201. (

\dagger

denotes the methods that additionally utilize text annotations from IDEA [ 30 ] .)

Methods	Venue	RGBNT100			MSVR310			WMVEID863
Methods	Venue	mAP	Rank-1	Rank-5	mAP	Rank-1	Rank-5	mAP	Rank-1	Rank-5	Rank-10
IEEE [ 7 ]	AAAI-22	61.3	87.8	90.2	21.0	41.0	57.7	45.9	48.6	64.3	67.9
PFNet [ 6 ]	AAAI-21	68.1	94.1	95.3	23.5	37.4	57.0	50.1	55.9	68.7	75.1
HAMNet [ 5 ]	AAAI-20	74.5	93.3	94.3	27.1	42.3	61.6	45.6	48.5	63.1	68.8
CCNet [ 46 ]	INFFUS-23	77.2	96.3	97.2	36.4	55.2	72.4	50.3	52.7	69.6	75.1
TOP-ReID [ 25 ]	AAAI-24	81.2	96.4	96.9	35.9	44.6	–	67.7	75.3	80.8	83.5

TABLE II: Comparison of different methods on three vehicle datasets: RGBNT100, MSVR310, and WMVEID863. (

\dagger

denotes the methods that additionally utilize text annotations from IDEA [ 30 ] .)


Methods	Venue	M (RGB)		M (NIR)		M (TIR)		M (RGB+NIR)		M (RGB+TIR)		M (NIR+TIR)		Average
Methods	Venue	m AP	R-1	m AP	R-1	m AP	R-1	m AP	R-1	m AP	R-1	m AP	R-1	m AP	R-1
HACNN [ 53 ]	CVPR-18	12.5	11.1	20.5	19.4	16.7	13.3	9.2	6.2	6.3	2.2	14.8	12.0	13.3	10.7
MLFN [ 54 ]	CVPR-18	20.2	18.9	21.1	19.7	17.6	11.1	13.2	12.1	8.3	3.5	13.1	9.1	15.6	12.4
PCB [ 55 ]	ECCV-18	23.6	24.2	24.4	25.1	19.9	14.7	20.6	23.6	11.0	6.8	18.6	14.4	19.7	18.1
OSNet [ 56 ]	ICCV-19	19.8	17.3	21.0	19.0	18.7	14.6	12.3	10.9	9.4	5.4	13.0	10.2	15.7	12.9

TABLE III: Performance of missing-modality settings on RGBNT201.(

*

indicates that the MASA is not applied; instead, fully decoupled features are concatenated as the final representation.) “M (X)” means missing the X image modality.


Methods	Venue	RT-to-NT		RT-to-N		R-to-N		R-to-NT		Average
Methods	Venue	m AP	R-1	m AP	R-1	m AP	R-1	m AP	R-1	m AP	R-1
EDITOR [ 26 ]	CVPR-24	27.3	27.9	2.80	0.0	4.0	2.3	4.3	4.1	9.6	8.6
TOP-ReID [ 25 ]	AAAI-24	43.0	44.6	14.4	13.3	15.4	14.0	11.9	8.7	21.2	20.2
MODAL ∗	Ours	37.8	32.9	5.2	4.9	5.3	4.3	4.9	1.8	13.3	11.0
MODAL	Ours	54.9	52.9	17.5	13.2	13.7	8.5	12.9	6.9	24.8	20.4

TABLE IV: Performance of modality-mismatched Scenarios.(

*

indicates that the MASA is not applied; instead, fully decoupled features are concatenated as the final representation.)

Fig. 6: Impact of the MFSD unfolding depth and the number of heads in TIDF blocks on model performance.

Methods	Metric			Params
Methods	mAP	Rank-1	Rank-5	Params
No-decoupling	77.2	79.8	85.8	0
Linear Mapping	78.1	81.1	88.6	6.3M
HDM [ 29 ]	79.9	83.6	85.9	14.7M
$\text{MFSD}_{K=1}$ (ours)	82.2	84.7	91.1	12.5M
$\text{MFSD}_{K=4}$ (ours)	83.8	86.1	90.3	34.9M

TABLE V: Ablation study of feature decoupling methods on RGBNT201. (

K

denotes the number of unfolding blocks.)


Methods	Metric			Params
Methods	mAP	Rank-1	Rank-5	Params
Concatenation	75.1	76.8	84.7	0
Cross Attention	80.1	82.2	88.9	10.4M
InverseNet [ 30 ]	81.6	82.4	88.2	13.0M
TIDF (ours)	83.8	86.1	90.3	8.5M

TABLE VI: Ablation study on text-image fusion methods on RGBNT201.

Learnable Transform	RGBNT201
Learnable Transform	mAP	Rank1	Rank5
None (only bottleneck)	80.8	82.1	89.0
Single Linear Layer	82.4	84.5	90.1
Group convolution	82.7	85.0	89.2
Mlp (Expansion=2)	81.8	83.5	90.6
Linear layers w/ dropout	83.8	86.1	90.3

TABLE VII: Ablation study on different structural designs of MFSD, evaluated on RGBNT201 dataset.


Data Usage	Metric
Data Usage	mAP	Rank-1	Rank-5
Text Only	10.2	13.9	20.1
Image Only	81.4	82.9	87.6
Text & Image	83.8	86.1	90.3

TABLE VIII: Ablation study on text and image usage on RGBNT201.


Text Granularity	Metric
Text Granularity	mAP	Rank-1	Rank-5
w/o Text	81.4	82.9	87.6
w/ Learnable Prompt	81.9	85.8	90.1
w/ Fixed Caption	82.1	84.7	91.4
w/ Prompt & Caption	83.8	86.1	90.3

TABLE IX: Ablation study on text granularity on RGBNT201.

Fig. 7: Visualization comparison of the Euclidean distance distributions between MODAL and the baseline method.

Fig. 8: Visualization of attention maps for the decoupled uni-modal ( $\bm{h}_{r}$ , $\bm{h}_{n}$ , $\bm{h}_{t}$ ), bi-modal ( $\bm{h}_{rn}$ , $\bm{h}_{rt}$ , $\bm{h}_{nt}$ ), and tri-modal ( ${h}_{rnt}$ ) feature components produced by MFSD. Diffdifferent sparse components attend to distinct object regions and exhibit limited overlap, indicating effective disentanglement of modality-specific and shared representations.

Fig. 9: Visualization of attention maps for the proposed Text-Image Differential Filtering (TIDF) module. The left panel presents the RGB, NIR, and TIR textual descriptions of two representative instances, where salient textual attributes are highlighted and linked to their corresponding textual attention regions. The right panel shows, for four examples, the image attention, text attention, and differential attention maps, respectively.

Multimodal Disentangled Representation Learning

Multimodal disentangled representation learning separates information shared across data types from modality-specific, task-irrelevant, or complementary factors.

See all 54 papers

Multi-Modal Object Re-Identification with Dual Semantic Guidance and Global-Local Mutual Modulation
Jul 31, 2026Weixiang Zhou, Xingguo Xu, Yuhao Wang +4Cross-Modal Representation Learning Multimodal Robustness
School of Mathematical Sciences and the Key Laboratory for Computational Mathematics and Data Intelligence of Liaoning Province, Dalian University of Technology, Dalian, 116024, China · School of Future Technology, School of Artificial Intelligence, Dalian University of Technology, Dalian, 116024, China · Department of Radiology and Biomedical Imaging, University of California, San Francisco, 94107, USA +1
Multi-Modal Object Re-Identification with Prompt-S6 and Semantic-Aware Knowledge Guidance
Jul 26, 2026Weixiang Zhou, Jiabei Zuo, Yuhao Wang +3Cross-Modal Alignment Multimodal Fusion
School of Mathematical Sciences and the Key Laboratory for Computational Mathematics and Data Intelligence of Liaoning Province, Dalian University of Technology, Dalian, 116024, China · School of Computer Science and Technology, Dalian University of Technology, Dalian, 116024, China · School of Future Technology, School of Artificial Intelligence, Dalian University of Technology, Dalian, 116024, China +1
Incentive Noise and Structural Prior Infusion for Multi-modal Object Re-Identification
Sep 21, 2026Weixiang Zhou, Yuhao Wang, Xingguo Xu +4Multimodal Robustness Cross-Modal Alignment
Dalian University of Technology, China · New York University, USA · Nanjing University of Science and Technology, China +1

Jul 31, 2026J/K move · Enter open · S save

Multi-Modal Object Re-Identification with Dual Semantic Guidance and Global-Local Mutual Modulation

Weixiang Zhou, Xingguo Xu, Yuhao Wang, Cong Wang+3

School of Mathematical Sciences and the Key Laboratory for Computational Mathematics and Data Intelligence of Liaoning Province, Dalian University of Technology, Dalian, 116024, China · School of Future Technology, School of Artificial Intelligence, Dalian University of Technology, Dalian, 116024, China · Department of Radiology and Biomedical Imaging, University of California, San Francisco, 94107, USA+1

Cross-Modal Representation Learning Multimodal Robustness Cross-Modal Alignment Multimodal Fusion

Open paper AskPDF arXiv

Multi-modal object Re-Identification (ReID) aims to retrieve target instances by leveraging complementary information across modalities. However, existing methods suffer from two challenges. First, they often fail to exploit well-aligned and reliable semantic priors, making them vulnerable to background clutter and cross-modal misalignment. On the other hand, they typically rely on holistic feature modeling, overlooking the synergy between global and local representations. To overcome these limitations, we propose a robust multi-modal ReID framework with dual semantic guidance and global-local mutual modulation, which mainly consists of three key components, namely the Text-Semantic Injector (TSI), the Masked Global-Local Modulator (MGLM), and the Hierarchical MoE Fusion (HMF). The TSI enhances semantic awareness by integrating clean and coherent textual features into visual tokens. The MGLM enables part-aware cross-modal interaction through joint guidance from soft masks and global context, improving fine-grained feature alignment. Finally, the HMF adaptively aggregates multi-spectral features under local semantic supervision, yielding discriminative and robust representations. Extensive experiments on three multi-modal ReID benchmarks demonstrate the effectiveness of the proposed method. The code will be made publicly available at https://github.com/zw-absin/DSGM upon acceptance.

Abstract

Multimodal Disentangled Representation Learning

Explore similar work

Multi-Modal Object Re-Identification with Dual Semantic Guidance and Global-Local Mutual Modulation

Multi-Modal Object Re-Identification with Prompt-S6 and Semantic-Aware Knowledge Guidance

Incentive Noise and Structural Prior Infusion for Multi-modal Object Re-Identification