Beyond Pedestrians: Caption-Guided CLIP Framework for High-Difficulty Video-based Person Re-Identification

Shogo Hamano; Shunya Wakasugi; Tatsuhito Sato; Sayaka Nakamura

cs.CVApr 9, 2026

Beyond Pedestrians: Caption-Guided CLIP Framework for High-Difficulty Video-based Person Re-Identification

Authors: Shogo Hamano, Shunya Wakasugi, Tatsuhito Sato, Sayaka Nakamura

Organizations: Sony Group Corporation

Abstract

In recent years, video-based person Re-Identification (ReID) has gained attention for its ability to leverage spatiotemporal cues to match individuals across non-overlapping cameras. However, current methods struggle with high-difficulty scenarios, such as sports and dance performances, where multiple individuals wear similar clothing while performing dynamic movements. To overcome these challenges, we propose CG-CLIP, a novel caption-guided CLIP framework that leverages explicit textual descriptions and learnable tokens. Our method introduces two key components: Caption-guided Memory Refinement (CMR) and Token-based Feature Extraction (TFE). CMR utilizes captions generated by Multi-modal Large Language Models (MLLMs) to refine identity-specific features, capturing fine-grained details. TFE employs a cross-attention mechanism with fixed-length learnable tokens to efficiently aggregate spatiotemporal features, reducing computational overhead. We evaluate our approach on two standard datasets (MARS and iLIDS-VID) and two newly constructed high-difficulty datasets (SportsVReID and DanceVReID). Experimental results demonstrate that our method outperforms current state-of-the-art approaches, achieving significant improvements across all benchmarks.

Figures & tables

Figure 1: Two basketball players with similar appearances. They wear identical uniforms and shoes of similar colors but differ in their hairstyles and jersey numbers. These subtle differences can be explicitly represented in textual descriptions.

Figure 2: Comparison of existing approaches and our method. (a) Prompt learning and text-free learning approaches. (b) Our proposed CG-CLIP framework that leverages text captions.

Figure 3: Illustration of our proposed CG-CLIP framework. Caption-guided Memory Refinement (CMR) refines Image Memory based on synthesized captions to capture fine-grained identity-specific features. Token-based Feature Extraction (TFE) aggregates features along both temporal and spatial dimensions using a cross-attention mechanism with learnable tokens.

Figure 4: Illustration of the proposed Token-based Feature Extraction (TFE) module.

Datasets	SportsV	DanceV	MARS [ 62 ]	iLIDS [ 45 ]
# Identities	513	692	1,261	300
# Tracklets	13,061	12,330	20,715	600
# Images	599,332	574,078	1,067,516	42,460

Table 1: Overview of the datasets used in our experiments. SportsV and DanceV are the newly constructed datasets.

Methods	MARS		iLIDS-VID		SportsVReID		DanceVReID
Methods	mAP	Rank-1	Rank-1	Rank-5	mAP	Rank-1	mAP	Rank-1
AP3D [ 16 ]	85.1	90.1	88.7	-	65.4	84.2	37.9	62.0
TCLNet [ 23 ]	85.1	89.8	86.6	-	60.6	79.8	35.3	50.9
MGH [ 52 ]	85.8	90.0	85.6	97.1	72.6	90.1	42.9	70.1
GRL [ 36 ]	84.8	91.0	90.4	98.3	65.9	86.4	35.8	53.5
BiCnet-TKS [ 22 ]	86.0	90.2	-	-	62.2	80.2	34.7	47.6

Table 2: Comparison with state-of-the-art methods across datasets. The bold and underline denote the best and second results, respectively.

Model	Components		MARS		SportsVReID
Model	CMR	TFE	mAP	Rank-1	mAP	Rank-1
1	$\times$	$\times$	88.5	91.2	75.5	88.6
2	✓	$\times$	89.7	92.2	77.3	87.9
3	$\times$	✓	88.5	91.5	76.6	89.0
4	✓	✓	89.8	92.5	77.7	90.4

Table 3: Comparison of different components.

Memory	MARS		SportsVReID
Memory	mAP	Rank-1	mAP	Rank-1
Image Mem. only	88.5	91.5	76.6	89.0
Text Mem. only	86.8	90.6	73.1	84.6
Image+Text Mem. (naive sum)	89.3	92.2	77.2	87.9
Refined Mem. (ours, w/ CMR)	89.8	92.5	77.7	90.4

Table 4: Comparison of different types of Memory.

# Token	MARS		SportsVReID
( $N^{Q}$ )	mAP	Rank-1	mAP	Rank-1
5	89.5	92.1	77.1	89.3
15	89.6	92.2	77.7	90.4
50	89.8	92.5	77.8	89.3
100	89.6	92.4	77.0	88.2
200	89.4	92.6	77.2	88.2

Table 5: Effect of the number of learnable tokens in TFE.

Methods	MARS					SportsVReID
Methods	20%	40%	60%	80%	100%	20%	40%	60%	80%	100%
FT-CLIP [ 59 ]	78.2	82.2	84.6	87.0	87.8	63.9	68.6	70.8	74.7	74.4
VSLA-CLIP [ 59 ]	78.1	83.2	86.1	87.7	88.2	57.8	68.6	70.2	73.7	74.1
TF-CLIP [ 55 ]	79.5	84.1	87.0	88.1	89.4	65.4	70.5	73.3	76.3	77.3
CG-CLIP (Ours)	81.1	85.6	87.7	88.7	89.8	66.6	72.8	74.2	76.3	77.7

Table 6: Comparison under varying training data sizes on MARS and SportsVReID (mAP).

Panel of Figure 5 — Figure 5: Accuracy–latency trade-off on DanceVReID

Figure 6: Visualization of attention maps on SportsVReID. (a) Original images, (b) Vanilla CLIP, (c) TF-CLIP, (d) CG-CLIP. Our method effectively focuses on fine-grained, discriminative regions.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Sports	Gender	Uniform	Number	Others
1	basketball	woman	yellow	5	bun hair
2	basketball	woman	blue	10	red shoes
91	soccer	man	gray, blue	32	black socks
92	soccer	man	white	1	red shoes

Appendix

Table S.1 : Examples of manual annotations for SportsVReID.

Dataset	Images	Caption
		A female basketball player is wearing a blue uniform with the number 3. She has a ponytail.
SportsVReID		A woman basketball player is seen in a blue uniform with the number 7, and she also wears black shoes.
		A man soccer player, in a white uniform with the number 32, also has black shoes.
		A female is dressed in a white cropped T-shirt with a black inner layer and white track pants. She wears white sneakers and her hair is styled in a half-updo.
DanceVReID		A male is wearing a shirt with red, white, and black stripes on the upper body and red, white, and black striped pants on the lower body. He also has on white socks and white sneakers with black accents, and his hair is short.
		She is a woman dressed in a red, white, and black top and matching red, white, and black pants. She also wears white socks and white sneakers with black accents, and her hair is in a ponytail. She is wearing a short jacket.

Appendix

Table S.2 : Examples of images and captions from the SportsVReID and DanceVReID datasets. Each row corresponds to a different person.

Figure S.1 : Illustration of three types of fusion encoders in CMR.

Method	MARS		SportsVReID
Method	mAP	Rank-1	mAP	Rank-1
(a) Self-attn (concat)	89.6	92.0	76.9	87.1
(b) Self-attn $\rightarrow$ Cross-attn	89.7	92.4	77.2	88.2
(c) Cross-attn $\rightarrow$ Self-attn	89.8	92.5	77.7	90.4

Appendix

Table S.3 : Comparison of different types of fusion encoders in CMR.

# Blocks	MARS		SportsVReID
# Blocks	mAP	Rank-1	mAP	Rank-1
1	89.5	91.9	77.5	90.1
2	89.8	92.5	77.7	90.4
3	89.6	91.9	76.8	87.1
4	89.6	92.4	77.8	88.2

Appendix

Table S.4 : Effect of varying the number of Transformer blocks in CMR.

Figure S.2 : Illustration of text variants used to construct the Text Memory on existing benchmarks (MARS and iLIDS-VID): caption, ID text , and ID emb .

Input text type			MARS		iLIDS-VID
Caption	ID text	ID emb	mAP	Rank-1	Rank-1	Rank-5
✓	$\times$	$\times$	89.7	92.3	96.0	99.9
$\times$	✓	$\times$	89.5	92.3	94.7	99.9
$\times$	$\times$	✓	89.6	92.3	95.3	99.3
✓	✓	$\times$	89.8	92.5	96.7	99.9
✓	$\times$	✓	89.8	92.4	96.0	99.9

Appendix

Table S.5 : Comparison of different types of input texts.

Method	SportsVReID		DanceVReID
Method	mAP	Rank-1	mAP	Rank-1
TF-CLIP [ 55 ] (w/o caption)	77.3	89.7	51.7	70.8
Ours: MLLM only	77.5	89.3	53.5	74.2
Ours: Manual + MLLM	77.7	90.4	53.8	76.0

Appendix

Table S.6 : Effect of different caption sources.

Figure S.3 : Visualization of top-5 retrieval results. Green and red boxes represent correct and incorrect matches, respectively. Blue-outlined rows show SportsVReID results, and orange-outlined rows show DanceVReID results. Each image shows the first frame of a tracklet.

Figure S.4 : t-SNE visualization of TF-CLIP and our CG-CLIP on the DanceVReID val set. Different colored dots represent different identities. Best viewed in color.

Figure S.5 : Visualization of attention weights between the first learnable token and image features from each frame in TFE. Higher values indicate stronger attention to the corresponding frame.

Hyperparameter	Settings
Hyperparameter	MARS	iLIDS-VID	Others
Batch size	32	32	32
# Identities per batch	8	8	8
# Tracklets per identity	4	4	4
# Frames per tracklet	8	8	8
Patch size	16	16	16

Appendix

Table S.7 : Training settings. “Others” include SportsVReID and DanceVReID.

Person Re-Identification

Person re-identification matches the same individual across camera views, times, clothing, and sensing modalities despite changes in appearance.

See all 90 papers

From Global to Local: Rethinking CLIP Feature Aggregation for Person Re-Identification
Apr 24, 2026Aotian Zheng, Winston Sun, Bahaa Alattar +2VLM Robustness Person Re-Identification
Department of Electrical and Computer Engineering, University of Washington, Seattle, WA 98195 · Applied Physics Laboratory, University of Washington, Seattle, WA 98195
DiffReID: Discriminative Diffusion Model for Object Re-Identification
Sep 29, 2026Yingquan Wang, Pingping Zhang, Dong Wang +1Representation Learning Query Re-Identification
School of Information and Communication Engineering, Dalian University of Technology · School of Future Technology, Dalian University of Technology
Paths: Prompt-aware Spatio-temporal Transformer with Hierarchical Multi-modal Fusion for RGB-Event Video Person Re-Identification
Aug 13, 2026Yakun Huo, Yingquan Wang, Yangyang Liu +4Cross-Modal Representation Learning Transformer
Dalian University of Technology Dalian, China

Apr 24, 2026J/K move · Enter open · S save

From Global to Local: Rethinking CLIP Feature Aggregation for Person Re-Identification

Aotian Zheng, Winston Sun, Bahaa Alattar, Vitaly Ablavsky+1

Department of Electrical and Computer Engineering, University of Washington, Seattle, WA 98195 · Applied Physics Laboratory, University of Washington, Seattle, WA 98195

VLM Robustness Person Re-Identification

Open paper AskPDF arXiv

CLIP-based person re-identification (ReID) methods aggregate spatial features into a single global [CLS] token optimized for image-text alignment rather than spatial selectivity, making representations fragile under occlusion and cross-camera variation. We propose SAGA-ReID, which reconstructs identity representations by aligning intermediate patch tokens with anchor vectors parameterized in CLIP's text embedding space -- emphasizing spatially stable evidence while suppressing corrupted or absent regions, without requiring textual descriptions of individual images. Controlled experiments isolate the aggregation mechanism under two qualitatively distinct conditions -- synthetic masking, where identity signal is absent, and realistic human distractors, where an overlapping person introduces semantically confusing signal -- with SAGA's advantage over global pooling growing substantially as occlusion increases across both conditions. Benchmark evaluations confirm consistent gains over CLIP-ReID across standard and occluded settings, with the largest improvements where global pooling is most unreliable: up to +10.6 Rank-1 on occluded benchmarks. SAGA's aggregation outperforms dedicated sequential patch aggregation on a stronger backbone, confirming that structured reconstruction addresses a bottleneck that backbone quality and architectural complexity alone cannot resolve. Code available at https://github.com/ipl-uw/Structured-Anchor-Guided-Aggregation-for-ReID.

Abstract

Person Re-Identification

Explore similar work

From Global to Local: Rethinking CLIP Feature Aggregation for Person Re-Identification

DiffReID: Discriminative Diffusion Model for Object Re-Identification

Paths: Prompt-aware Spatio-temporal Transformer with Hierarchical Multi-modal Fusion for RGB-Event Video Person Re-Identification