In recent years, video-based person Re-Identification (ReID) has gained attention for its ability to leverage spatiotemporal cues to match individuals across non-overlapping cameras. However, current methods struggle with high-difficulty scenarios, such as sports and dance performances, where multiple individuals wear similar clothing while performing dynamic movements. To overcome these challenges, we propose CG-CLIP, a novel caption-guided CLIP framework that leverages explicit textual descriptions and learnable tokens. Our method introduces two key components: Caption-guided Memory Refinement (CMR) and Token-based Feature Extraction (TFE). CMR utilizes captions generated by Multi-modal Large Language Models (MLLMs) to refine identity-specific features, capturing fine-grained details. TFE employs a cross-attention mechanism with fixed-length learnable tokens to efficiently aggregate spatiotemporal features, reducing computational overhead. We evaluate our approach on two standard datasets (MARS and iLIDS-VID) and two newly constructed high-difficulty datasets (SportsVReID and DanceVReID). Experimental results demonstrate that our method outperforms current state-of-the-art approaches, achieving significant improvements across all benchmarks.
Figures & tables
Figure 1: Two basketball players with similar appearances. They wear identical uniforms and shoes of similar colors but differ in their hairstyles and jersey numbers. These subtle differences can be explicitly represented in textual descriptions.
Figure 2: Comparison of existing approaches and our method. (a) Prompt learning and text-free learning approaches. (b) Our proposed CG-CLIP framework that leverages text captions.
Figure 3: Illustration of our proposed CG-CLIP framework. Caption-guided Memory Refinement (CMR) refines Image Memory based on synthesized captions to capture fine-grained identity-specific features. Token-based Feature Extraction (TFE) aggregates features along both temporal and spatial dimensions using a cross-attention mechanism with learnable tokens.
Figure 4: Illustration of the proposed Token-based Feature Extraction (TFE) module.
Datasets
SportsV
DanceV
MARS [ 62 ]
iLIDS [ 45 ]
# Identities
513
692
1,261
300
# Tracklets
13,061
12,330
20,715
600
# Images
599,332
574,078
1,067,516
42,460
Table 1: Overview of the datasets used in our experiments. SportsV and DanceV are the newly constructed datasets.
Methods
MARS
iLIDS-VID
SportsVReID
DanceVReID
mAP
Rank-1
Rank-1
Rank-5
mAP
Rank-1
mAP
Rank-1
AP3D [ 16 ]
85.1
90.1
88.7
-
65.4
84.2
37.9
62.0
TCLNet [ 23 ]
85.1
89.8
86.6
-
60.6
79.8
35.3
50.9
MGH [ 52 ]
85.8
90.0
85.6
97.1
72.6
90.1
42.9
70.1
GRL [ 36 ]
84.8
91.0
90.4
98.3
65.9
86.4
35.8
53.5
BiCnet-TKS [ 22 ]
86.0
90.2
-
-
62.2
80.2
34.7
47.6
Table 2: Comparison with state-of-the-art methods across datasets. The bold and underline denote the best and second results, respectively.
Model
Components
MARS
SportsVReID
CMR
TFE
mAP
Rank-1
mAP
Rank-1
1
×
×
88.5
91.2
75.5
88.6
2
✓
×
89.7
92.2
77.3
87.9
3
×
✓
88.5
91.5
76.6
89.0
4
✓
✓
89.8
92.5
77.7
90.4
Table 3: Comparison of different components.
Memory
MARS
SportsVReID
mAP
Rank-1
mAP
Rank-1
Image Mem. only
88.5
91.5
76.6
89.0
Text Mem. only
86.8
90.6
73.1
84.6
Image+Text Mem. (naive sum)
89.3
92.2
77.2
87.9
Refined Mem. (ours, w/ CMR)
89.8
92.5
77.7
90.4
Table 4: Comparison of different types of Memory.
# Token
MARS
SportsVReID
( NQ )
mAP
Rank-1
mAP
Rank-1
5
89.5
92.1
77.1
89.3
15
89.6
92.2
77.7
90.4
50
89.8
92.5
77.8
89.3
100
89.6
92.4
77.0
88.2
200
89.4
92.6
77.2
88.2
Table 5: Effect of the number of learnable tokens in TFE.
Methods
MARS
SportsVReID
20%
40%
60%
80%
100%
20%
40%
60%
80%
100%
FT-CLIP [ 59 ]
78.2
82.2
84.6
87.0
87.8
63.9
68.6
70.8
74.7
74.4
VSLA-CLIP [ 59 ]
78.1
83.2
86.1
87.7
88.2
57.8
68.6
70.2
73.7
74.1
TF-CLIP [ 55 ]
79.5
84.1
87.0
88.1
89.4
65.4
70.5
73.3
76.3
77.3
CG-CLIP (Ours)
81.1
85.6
87.7
88.7
89.8
66.6
72.8
74.2
76.3
77.7
Table 6: Comparison under varying training data sizes on MARS and SportsVReID (mAP).
Figure 5: Accuracy–latency trade-off on DanceVReID
Figure 6: Visualization of attention maps on SportsVReID. (a) Original images, (b) Vanilla CLIP, (c) TF-CLIP, (d) CG-CLIP. Our method effectively focuses on fine-grained, discriminative regions.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Sports
Gender
Uniform
Number
Others
1
basketball
woman
yellow
5
bun hair
2
basketball
woman
blue
10
red shoes
91
soccer
man
gray, blue
32
black socks
92
soccer
man
white
1
red shoes
Appendix
Table S.1 : Examples of manual annotations for SportsVReID.
Dataset
Images
Caption
A female basketball player is wearing a blue uniform with the number 3. She has a ponytail.
SportsVReID
A woman basketball player is seen in a blue uniform with the number 7, and she also wears black shoes.
A man soccer player, in a white uniform with the number 32, also has black shoes.
A female is dressed in a white cropped T-shirt with a black inner layer and white track pants. She wears white sneakers and her hair is styled in a half-updo.
DanceVReID
A male is wearing a shirt with red, white, and black stripes on the upper body and red, white, and black striped pants on the lower body. He also has on white socks and white sneakers with black accents, and his hair is short.
She is a woman dressed in a red, white, and black top and matching red, white, and black pants. She also wears white socks and white sneakers with black accents, and her hair is in a ponytail. She is wearing a short jacket.
Appendix
Table S.2 : Examples of images and captions from the SportsVReID and DanceVReID datasets. Each row corresponds to a different person.
Figure S.1 : Illustration of three types of fusion encoders in CMR.
Method
MARS
SportsVReID
mAP
Rank-1
mAP
Rank-1
(a) Self-attn (concat)
89.6
92.0
76.9
87.1
(b) Self-attn → Cross-attn
89.7
92.4
77.2
88.2
(c) Cross-attn → Self-attn
89.8
92.5
77.7
90.4
Appendix
Table S.3 : Comparison of different types of fusion encoders in CMR.
# Blocks
MARS
SportsVReID
mAP
Rank-1
mAP
Rank-1
1
89.5
91.9
77.5
90.1
2
89.8
92.5
77.7
90.4
3
89.6
91.9
76.8
87.1
4
89.6
92.4
77.8
88.2
Appendix
Table S.4 : Effect of varying the number of Transformer blocks in CMR.
Figure S.2 : Illustration of text variants used to construct the Text Memory on existing benchmarks (MARS and iLIDS-VID): caption, ID text , and ID emb .
Input text type
MARS
iLIDS-VID
Caption
ID text
ID emb
mAP
Rank-1
Rank-1
Rank-5
✓
×
×
89.7
92.3
96.0
99.9
×
✓
×
89.5
92.3
94.7
99.9
×
×
✓
89.6
92.3
95.3
99.3
✓
✓
×
89.8
92.5
96.7
99.9
✓
×
✓
89.8
92.4
96.0
99.9
Appendix
Table S.5 : Comparison of different types of input texts.
Method
SportsVReID
DanceVReID
mAP
Rank-1
mAP
Rank-1
TF-CLIP [ 55 ] (w/o caption)
77.3
89.7
51.7
70.8
Ours: MLLM only
77.5
89.3
53.5
74.2
Ours: Manual + MLLM
77.7
90.4
53.8
76.0
Appendix
Table S.6 : Effect of different caption sources.
Figure S.3 : Visualization of top-5 retrieval results. Green and red boxes represent correct and incorrect matches, respectively. Blue-outlined rows show SportsVReID results, and orange-outlined rows show DanceVReID results. Each image shows the first frame of a tracklet.
Figure S.4 : t-SNE visualization of TF-CLIP and our CG-CLIP on the DanceVReID val set. Different colored dots represent different identities. Best viewed in color.
Figure S.5 : Visualization of attention weights between the first learnable token and image features from each frame in TFE. Higher values indicate stronger attention to the corresponding frame.
Hyperparameter
Settings
MARS
iLIDS-VID
Others
Batch size
32
32
32
# Identities per batch
8
8
8
# Tracklets per identity
4
4
4
# Frames per tracklet
8
8
8
Patch size
16
16
16
Appendix
Table S.7 : Training settings. “Others” include SportsVReID and DanceVReID.
CLIP-based person re-identification (ReID) methods aggregate spatial features into a single global [CLS] token optimized for image-text alignment rather than spatial selectivity, making representations fragile under occlusion and cross-camera variation. We propose SAGA-ReID, which reconstructs identity representations by aligning intermediate patch tokens with anchor vectors parameterized in CLIP's text embedding space -- emphasizing spatially stable evidence while suppressing corrupted or absent regions, without requiring textual descriptions of individual images. Controlled experiments isolate the aggregation mechanism under two qualitatively distinct conditions -- synthetic masking, where identity signal is absent, and realistic human distractors, where an overlapping person introduces semantically confusing signal -- with SAGA's advantage over global pooling growing substantially as occlusion increases across both conditions. Benchmark evaluations confirm consistent gains over CLIP-ReID across standard and occluded settings, with the largest improvements where global pooling is most unreliable: up to +10.6 Rank-1 on occluded benchmarks. SAGA's aggregation outperforms dedicated sequential patch aggregation on a stronger backbone, confirming that structured reconstruction addresses a bottleneck that backbone quality and architectural complexity alone cannot resolve. Code available at https://github.com/ipl-uw/Structured-Anchor-Guided-Aggregation-for-ReID.
Aotian Zheng, Winston Sun, Bahaa Alattar +2
Department of Electrical and Computer Engineering, University of Washington, Seattle, WA 98195 · Applied Physics Laboratory, University of Washington, Seattle, WA 98195
As a fundamental image processing task, object Re-Identification (ReID) aims to retrieve objects across non-overlapping cameras. Recently, with the development of deep learning, significant advancements have been made in object ReID. However, most existing methods suffer from generalization due to the limited size and diversity of ReID datasets. Meanwhile, current models tend to focus on extracting semantic patterns rather than learning identity-aware feature distributions. To address these issues, we propose a novel feature learning framework named DiffReID for object ReID. It leverages a discriminative diffusion model to gradually learn identity-aware distributions and generate identity-invariant features. More specifically, with the Contrastive Language-Image Pre-training (CLIP) model, we first obtain identity-aware text features by prompt tuning. Then, we propose a Vision-guided Noise Generator (VNG) to initialize probabilistic noises and gradually corrupt identity-aware text features. Afterwards, we take visual features as conditions and propose a Light Weight Denoiser (LWD) to denoise the corrupted text features step-by-step for identity-aware distribution learning. To obtain discriminative features, we further generate identity-invariant guided features from randomly sampling visual-guided noises. Finally, we propose a Mutual Enhancement Constraint (MEC) to facilitate mutual learning between visual features and guided features to enhance the representation robustness and discrimination. Extensive experiments on five object ReID benchmarks demonstrate that our method shows better results than most state-of-the-art methods. The source code is available at https://github.com/AWangYQ/DiffReID.
Yingquan Wang, Pingping Zhang, Dong Wang +1
School of Information and Communication Engineering, Dalian University of Technology · School of Future Technology, Dalian University of Technology
RGB-Event Video Person Re-Identification (RE-VReID) aims to retrieve specific person across non-overlapping cameras with complementary RGB videos and event streams. However, existing methods often decouple spatial and temporal modeling, which limits their interaction. In addition, global-level RGB-Event fusion fails to fully exploit fine-grained discriminative cues. To address these issues, we propose Paths, a unified framework with spatio-temporal modeling and hierarchical multi-modal fusion for RE-VReID. Specifically, we first design a Memory-Augmented Backbone (MAB) to maintain modality-specific identity prototypes for stable intra-modal representation learning. Then, we propose a Prompt-aware Spatio-temporal Transformer (PST) to jointly model spatial and temporal cues within a unified Transformer. Finally, we introduce a Hierarchical Multi-modal Fusion (HMF) to integrate RGB and event features at global and local levels. With these modules, our framework can learn robust and discriminative representations for RE-VReID. Extensive experiments on three public RE-VReID benchmarks including EvReID, MARS and iLIDS-VID, demonstrate the effectiveness of our proposed method. The code is available at https://github.com/Reflection0427/Paths.