MODAL: Multi-Modal Object Re-ID via Model-Driven Sparse Decoupling and Text-Image Differential Filtering

Chengbo Huang; Jun-Jie Huang; Long Lan; Tianrui Liu; Xueqiong Li; Yuanxi Peng; Xinwang Liu; Meng Wang

cs.CVAug 15, 2026

MODAL: Multi-Modal Object Re-ID via Model-Driven Sparse Decoupling and Text-Image Differential Filtering

Authors: Chengbo Huang, Jun-Jie Huang, Long Lan, Tianrui Liu, Xueqiong Li, Yuanxi Peng, Xinwang Liu, Meng Wang

Organizations: College of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China · School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230002, China

Abstract

Multi-modal object re-identification (Re-ID) aims to facilitate cross-camera object retrieval in complex environments by leveraging complementary information from visual (e.g., RGB, NIR, TIR) and textual modalities. However, existing approaches often lack principled feature disentanglement and coherent multi-modal integration, leading to entangled representations that introduce cross-modal conflicts, obscure discriminative cues, and suffer distribution shift under modality-missing conditions. To tackle these challenges, we propose MODAL, a novel multi-modal object re-identification framework, grounded in coupled sparse coding theory and differential suppression principles. A core component of MODAL is a Multi-modal Feature Sparse Decoupling module, developed in a model-driven deep unrolling manner based on multi-modal coupled sparse coding. It explicitly decomposes multi-modal features into uni-modal specific, bi-modal and tri-modal shared representations, thereby achieving more transparent and effective feature disentanglement. Benefiting from the principled feature disentanglement, MODAL naturally mitigates performance degradation in incomplete-modality scenarios via a Modality-Aware Subspace Activation that selectively activates only the consistently shared subspaces. Moreover, we propose a Text-Image Differential Filtering module that leverages coarse-grained textual semantics to adaptively suppress task-irrelevant responses in the decoupled visual representations, thereby enhancing discriminative information. Extensive experiments on four datasets demonstrate that MODAL achieves state-of-the-art performance with superior transparency.

Figures & tables

Fig. 1: Motivation and overview of MODAL. Left: Existing methods directly fuse entangled multi-modal features in a unified space, which introduces inter-modal interference and leads to significant performance degradation under missing-modality scenarios. Right: In contrast, MODAL mitigates these issues through sparse feature decoupling, Text-Image Differential Filtering, and modality-aware subspace activation, effectively reducing cross-modal conflict, enhancing representation discriminability, and enabling more robust matching under incomplete-modality scenarios.

Fig. 2: Overview of the proposed MODAL framework. Given multi-modal text and image inputs, their global and local features are extracted through CLIP encoders. These features are then sparsely decomposed into modality-specific and shared features via Multi-modal Feature Sparse Decoupling. Next, a Text-Image Differential Filtering module fuses the decomposed textual and visual features with local visual features to suppress task-irrelevant features and improve discriminative capability. Finally, a Mixture-of-Experts model selects and connects features to form the final representation.

Fig. 3: Architecture of the Uni-modal (UFD), Bi-modal (BFD), and Tri-modal (TFD) feature decoupling blocks. All three blocks utilize a unified iterative structure. Each block refines its corresponding sparse codes through

K

repeated stages of learnable transformations and soft thresholding.

Fig. 4: Architecture of the Text-Image Differential Filtering block.

Fig. 5: Schematic illustration of the Modality-Aware Subspace Activation (MASA) for incomplete modality inference..

Methods	Venue	RGBNT201
Methods	Venue	mAP	Rank-1	Rank-5
HAMNet [ 5 ]	AAAI-20	27.7	26.3	41.5
PFNet [ 6 ]	AAAI-21	38.5	38.9	52.0
IEEE [ 7 ]	AAAI-22	46.4	47.1	58.5
LRMM [ 48 ]	ESWA-25	52.3	53.4	64.6
TIENet [ 49 ]	TNNLS-25	54.4	54.4	66.3

TABLE I: Comparison of different methods on pedestrian dataset RGBNT201. (

\dagger

denotes the methods that additionally utilize text annotations from IDEA [ 30 ] .)

Methods	Venue	RGBNT100			MSVR310			WMVEID863
Methods	Venue	mAP	Rank-1	Rank-5	mAP	Rank-1	Rank-5	mAP	Rank-1	Rank-5	Rank-10
IEEE [ 7 ]	AAAI-22	61.3	87.8	90.2	21.0	41.0	57.7	45.9	48.6	64.3	67.9
PFNet [ 6 ]	AAAI-21	68.1	94.1	95.3	23.5	37.4	57.0	50.1	55.9	68.7	75.1
HAMNet [ 5 ]	AAAI-20	74.5	93.3	94.3	27.1	42.3	61.6	45.6	48.5	63.1	68.8
CCNet [ 46 ]	INFFUS-23	77.2	96.3	97.2	36.4	55.2	72.4	50.3	52.7	69.6	75.1
TOP-ReID [ 25 ]	AAAI-24	81.2	96.4	96.9	35.9	44.6	–	67.7	75.3	80.8	83.5

TABLE II: Comparison of different methods on three vehicle datasets: RGBNT100, MSVR310, and WMVEID863. (

\dagger

denotes the methods that additionally utilize text annotations from IDEA [ 30 ] .)


Methods	Venue	M (RGB)		M (NIR)		M (TIR)		M (RGB+NIR)		M (RGB+TIR)		M (NIR+TIR)		Average
Methods	Venue	m AP	R-1	m AP	R-1	m AP	R-1	m AP	R-1	m AP	R-1	m AP	R-1	m AP	R-1
HACNN [ 53 ]	CVPR-18	12.5	11.1	20.5	19.4	16.7	13.3	9.2	6.2	6.3	2.2	14.8	12.0	13.3	10.7
MLFN [ 54 ]	CVPR-18	20.2	18.9	21.1	19.7	17.6	11.1	13.2	12.1	8.3	3.5	13.1	9.1	15.6	12.4
PCB [ 55 ]	ECCV-18	23.6	24.2	24.4	25.1	19.9	14.7	20.6	23.6	11.0	6.8	18.6	14.4	19.7	18.1
OSNet [ 56 ]	ICCV-19	19.8	17.3	21.0	19.0	18.7	14.6	12.3	10.9	9.4	5.4	13.0	10.2	15.7	12.9

TABLE III: Performance of missing-modality settings on RGBNT201.(

*

indicates that the MASA is not applied; instead, fully decoupled features are concatenated as the final representation.) “M (X)” means missing the X image modality.


Methods	Venue	RT-to-NT		RT-to-N		R-to-N		R-to-NT		Average
Methods	Venue	m AP	R-1	m AP	R-1	m AP	R-1	m AP	R-1	m AP	R-1
EDITOR [ 26 ]	CVPR-24	27.3	27.9	2.80	0.0	4.0	2.3	4.3	4.1	9.6	8.6
TOP-ReID [ 25 ]	AAAI-24	43.0	44.6	14.4	13.3	15.4	14.0	11.9	8.7	21.2	20.2
MODAL ∗	Ours	37.8	32.9	5.2	4.9	5.3	4.3	4.9	1.8	13.3	11.0
MODAL	Ours	54.9	52.9	17.5	13.2	13.7	8.5	12.9	6.9	24.8	20.4

TABLE IV: Performance of modality-mismatched Scenarios.(

*

indicates that the MASA is not applied; instead, fully decoupled features are concatenated as the final representation.)

Fig. 6: Impact of the MFSD unfolding depth and the number of heads in TIDF blocks on model performance.

Methods	Metric			Params
Methods	mAP	Rank-1	Rank-5	Params
No-decoupling	77.2	79.8	85.8	0
Linear Mapping	78.1	81.1	88.6	6.3M
HDM [ 29 ]	79.9	83.6	85.9	14.7M
$\text{MFSD}_{K=1}$ (ours)	82.2	84.7	91.1	12.5M
$\text{MFSD}_{K=4}$ (ours)	83.8	86.1	90.3	34.9M

TABLE V: Ablation study of feature decoupling methods on RGBNT201. (

K

denotes the number of unfolding blocks.)


Methods	Metric			Params
Methods	mAP	Rank-1	Rank-5	Params
Concatenation	75.1	76.8	84.7	0
Cross Attention	80.1	82.2	88.9	10.4M
InverseNet [ 30 ]	81.6	82.4	88.2	13.0M
TIDF (ours)	83.8	86.1	90.3	8.5M

TABLE VI: Ablation study on text-image fusion methods on RGBNT201.

Learnable Transform	RGBNT201
Learnable Transform	mAP	Rank1	Rank5
None (only bottleneck)	80.8	82.1	89.0
Single Linear Layer	82.4	84.5	90.1
Group convolution	82.7	85.0	89.2
Mlp (Expansion=2)	81.8	83.5	90.6
Linear layers w/ dropout	83.8	86.1	90.3

TABLE VII: Ablation study on different structural designs of MFSD, evaluated on RGBNT201 dataset.


Data Usage	Metric
Data Usage	mAP	Rank-1	Rank-5
Text Only	10.2	13.9	20.1
Image Only	81.4	82.9	87.6
Text & Image	83.8	86.1	90.3

TABLE VIII: Ablation study on text and image usage on RGBNT201.


Text Granularity	Metric
Text Granularity	mAP	Rank-1	Rank-5
w/o Text	81.4	82.9	87.6
w/ Learnable Prompt	81.9	85.8	90.1
w/ Fixed Caption	82.1	84.7	91.4
w/ Prompt & Caption	83.8	86.1	90.3

TABLE IX: Ablation study on text granularity on RGBNT201.

Fig. 7: Visualization comparison of the Euclidean distance distributions between MODAL and the baseline method.

Fig. 8: Visualization of attention maps for the decoupled uni-modal ( $\bm{h}_{r}$ , $\bm{h}_{n}$ , $\bm{h}_{t}$ ), bi-modal ( $\bm{h}_{rn}$ , $\bm{h}_{rt}$ , $\bm{h}_{nt}$ ), and tri-modal ( ${h}_{rnt}$ ) feature components produced by MFSD. Diffdifferent sparse components attend to distinct object regions and exhibit limited overlap, indicating effective disentanglement of modality-specific and shared representations.

Fig. 9: Visualization of attention maps for the proposed Text-Image Differential Filtering (TIDF) module. The left panel presents the RGB, NIR, and TIR textual descriptions of two representative instances, where salient textual attributes are highlighted and linked to their corresponding textual attention regions. The right panel shows, for four examples, the image attention, text attention, and differential attention maps, respectively.

Multimodal Disentangled Representation Learning

Multimodal disentangled representation learning separates information shared across data types from modality-specific, task-irrelevant, or complementary factors.

See all 54 papers

Jul 31, 2026

Multi-Modal Object Re-Identification with Dual Semantic Guidance and Global-Local Mutual Modulation

Multi-modal object Re-Identification (ReID) aims to retrieve target instances by leveraging complementary information across modalities. However, existing methods suffer from two challenges. First, they often fail to exploit well-aligned and reliable semantic priors, making them vulnerable to background clutter and cross-modal misalignment. On the other hand, they typically rely on holistic feature modeling, overlooking the synergy between global and local representations. To overcome these limitations, we propose a robust multi-modal ReID framework with dual semantic guidance and global-local mutual modulation, which mainly consists of three key components, namely the Text-Semantic Injector (TSI), the Masked Global-Local Modulator (MGLM), and the Hierarchical MoE Fusion (HMF). The TSI enhances semantic awareness by integrating clean and coherent textual features into visual tokens. The MGLM enables part-aware cross-modal interaction through joint guidance from soft masks and global context, improving fine-grained feature alignment. Finally, the HMF adaptively aggregates multi-spectral features under local semantic supervision, yielding discriminative and robust representations. Extensive experiments on three multi-modal ReID benchmarks demonstrate the effectiveness of the proposed method. The code will be made publicly available at https://github.com/zw-absin/DSGM upon acceptance.

Weixiang Zhou, Xingguo Xu, Yuhao Wang +4

School of Mathematical Sciences and the Key Laboratory for Computational Mathematics and Data Intelligence of Liaoning Province, Dalian University of Technology, Dalian, 116024, China · School of Future Technology, School of Artificial Intelligence, Dalian University of Technology, Dalian, 116024, China · Department of Radiology and Biomedical Imaging, University of California, San Francisco, 94107, USA +1

Cross-Modal Representation Learning Multimodal Robustness Cross-Modal Alignment Multimodal Fusion

Ask PDF

Jul 26, 2026

Multi-Modal Object Re-Identification with Prompt-S6 and Semantic-Aware Knowledge Guidance

Multi-modal object Re-Identification (ReID) aims to retrieve specific objects by integrating complementary information from multiple modalities. However, existing multi-modal ReID methods do not effectively address background interference suppression or achieve tri-modal alignment, instead focusing on pairwise feature fusion. Moreover, many current aggregation approaches suffer from high computational complexity. To address these limitations, we propose PRISM, a novel multi-modal ReID framework built upon Prompt-S6 (PS6) and semantic-aware knowledge guidance. PS6 maintains the linear complexity and strong sequence modeling capability of Mamba while enabling efficient cross-modal interaction. Leveraging these advantages, we design two key components: Semantic-Driven Token Pruning (SDTP) and Progressive Fusion Network (PFN). Parsing semantic priors from the segmentation foundation models, the SDTP then leverages these priors and applies dynamic token pruning to suppress background noise and refine feature representations. The PFN progressively aggregates multi-modal features to achieve tri-modal alignment and fully exploit modality complementarity. With the proposed modules, PRISM generates more robust multi-modal representations under complex scenarios. Extensive experiments on four multi-modal object ReID benchmarks demonstrate the effectiveness and efficiency of our approach. The source code is available at https://github.com/zw-absin/PRISM.

Weixiang Zhou, Jiabei Zuo, Yuhao Wang +3

School of Mathematical Sciences and the Key Laboratory for Computational Mathematics and Data Intelligence of Liaoning Province, Dalian University of Technology, Dalian, 116024, China · School of Computer Science and Technology, Dalian University of Technology, Dalian, 116024, China · School of Future Technology, School of Artificial Intelligence, Dalian University of Technology, Dalian, 116024, China +1

Cross-Modal Alignment Multimodal Fusion

Ask PDF

Sep 21, 2026

Incentive Noise and Structural Prior Infusion for Multi-modal Object Re-Identification

Multi-modal object Re-Identification (ReID) benefits from complementary information across heterogeneous imaging modalities. To further enrich semantic representation, text descriptions have recently been incorporated as an additional modality. However, recent vision-language approaches often treat text descriptions as clean, deterministic signals and overlook their inherent noise, including modality-mismatched phrases and semantically ambiguous expressions. Moreover, prevailing methods lack explicit mechanisms to reconcile fine-grained structural discrepancies between modalities, even after high-level semantic alignment. To address these challenges, we propose a novel framework centered on Positive-Incentive Noise (π-noise) and structured prompt modulation. First, the Semantic Cross-Modal Modulator harnesses task-aware π-noise, sampled from a distribution conditioned on both visual and text inputs, to perturb global tokens and enable semantics-guided cross-modal compensation. Second, the Structure-Aware Prompt Adapter injects learnable geometric priors via prompts to enhance spatial consistency. Third, the Context-Aware Sparse Fusion module distills structural context to guide adaptive fusion while shielding identity features from noisy local details. Experiments on three multi-modal ReID benchmarks demonstrate the effectiveness and robustness of our approach. The code is available at https://github.com/zw-absin/INSPI.

Weixiang Zhou, Yuhao Wang, Xingguo Xu +4

Dalian University of Technology, China · New York University, USA · Nanjing University of Science and Technology, China +1

Multimodal Robustness Cross-Modal Alignment Multimodal Fusion Multimodal Learning

Ask PDF

Abstract

Multimodal Disentangled Representation Learning

Explore similar work

Multi-Modal Object Re-Identification with Dual Semantic Guidance and Global-Local Mutual Modulation

Multi-Modal Object Re-Identification with Prompt-S6 and Semantic-Aware Knowledge Guidance

Incentive Noise and Structural Prior Infusion for Multi-modal Object Re-Identification