Vision-Grounded Semantic Learning and Alignment for Scalable Multilingual Text-Based Person Retrieval
Authors: Xinyu Zhou, Fan Li, Shasha Chen, Min Cao, Ahtamjan Ahmat, Qiaoming Zhu, Mang Ye
Organizations: Soochow University, Suzhou, China · The Xinjiang Technical Institute of Physics & Chemistry, Chinese Academy of Sciences, Xinjiang, China · Wuhan University, Wuhan, China
Multilingual Text-Based Person Retrieval (TBPR) aims to retrieve individuals with texts in multiple languages. Despite its practical importance, it has been overlooked by existing research, which mainly focuses on English-only retrieval. The only prior attempt at multilingual TBPR suffers from (1) the absence of a large-scale, multilingual pretraining dataset and (2) a suboptimal model architecture for multilingual text encoding and cross-modal alignment, which employs a shared encoder that overlooks linguistic heterogeneity and relies on English as a pivot for cross-modal alignment, introducing English-centric bias. For this, we contribute MultiPedes, the first large-scale, pedestrian-centric multilingual pretraining dataset. It contains 10 million image-text pairs across four languages, constructed by a multilingual quality-aware data construction pipeline. We also propose ViSeLA (Vision-grounded Semantic Learning and Alignment), a novel framework featuring: (1) a semantic-aware multilingual learning module that extends the mixture-of-experts architecture by routing language-invariant semantics to specialized experts for improved multilingual feature learning; and (2) a vision-grounded cross-modal alignment module that employs a semantic probe to capture fine-grained semantics across modalities, enabling direct cross-modal alignment without relying on a pivot language. Extensive experiments on both multilingual and English-only TBPR datasets demonstrate the effectiveness of ViSeLA. The dataset and code are available at https://github.com/Flame-Chasers/ViSeLA
Figures & tables
Figure 1. (a) Multilingual TBPR, where person images are retrieved with texts in multiple languages. (b) Existing method adopts a shared text encoder for all languages and relies on an English-centered knowledge transfer paradigm for cross-lingual cross-modal alignment. (c) Our ViSeLA framework introduces Se-L module for multilingual learning and Vi-A module for vision-grounded cross-modal alignment.
Figure 2. Multilingual quality-aware data construction pipeline consisting of two stages: (1) data curation with consensus-guided filtering, and (2) multilingual text translation. In the first stage, consensus-guided filtering leverages both domain-specific and domain-agnostic evaluators to identify and remove noisy samples from large-scale image–text pairs collected from public datasets, resulting in higher-quality data. In the second stage, large language models are employed to perform machine translation at scale, generating multilingual descriptions and constructing the MultiPedes dataset.
Dataset
#Images
#Texts
Avg. Len
Language
TTR
CLIPScore
LUPerson-T ( Shao et al., 2023 )
1.30M
1.30M
27.39
En
0.88
0.22
LUPerson-M ( Tan et al., 2024 )
1.00M
4.00M
28.61
En
0.80
0.36
HAM-PEDES ( Jiang et al., 2025 )
1.00M
2.00M
26.30
En
0.80
0.40
SynTBPR ( Cao et al., 2025a )
1.63M
6.62M
34.10
En
0.81
0.43
WebPerson ( Zheng et al., 2025b )
5M
10M
23.34
En
0.81
0.40
Table 1. Comparison with other mainstream pretraining datasets for TBPR. # denotes the number of samples, and Avg. Len indicates the average text length.
Figure 3. Comparison of COMET-Wiki score distributions between MultiPedes and COCO-35L ( Thapliyal et al., 2022 ) . The scores are averaged over three target languages for each sample.
Figure 4. Overview of our proposed ViSeLA framework. It consists of two key submodules: (1) the Se-L module, which leverages semantic information to route input text features to specialized experts, thereby extracting language-invariant textual features; and (2) the Vi-A module, which employs the semantic probe to capture fine-grained cross-modal semantics and enables image-grounded cross-modal alignment.
English
Chinese
French
German
Avg
Dataset
Method
R1
R10
mAP
R1
R10
mAP
R1
R10
mAP
R1
R10
mAP
R1
R10
mAP
Without Pre-training on Large-scale Person Data:
CUHK-PEDE(M)
CCLM ( Zeng et al., 2022 )
74.68
93.42
65.78
70.83
92.85
63.14
71.17
92.30
62.89
70.19
92.24
62.47
71.72
92.70
63.57
X²-VLM ( Zeng et al., 2023 )
76.97
94.96
67.86
74.81
94.17
66.00
75.18
94.18
66.17
74.71
94.30
65.73
75.42
94.40
66.44
CCRK ( Nie et al., 2024 )
70.11
92.53
62.10
68.31
91.91
60.69
68.29
91.36
60.33
67.01
90.74
59.54
68.43
91.64
60.67
Table 2. Performance comparison with state-of-the-art methods on multilingual TBPR datasets. We categorize these methods into two groups based on whether they are pre-trained on large-scale person data (per their original papers): Bi-IRRA uses an English-only dataset (LUPerosn-M ( Tan et al., 2024 ) ), while ViSeLA leverages our proposed MultiPedes dataset. Results are reported for four languages and their average. The best results are highlighted in bold, and the second-best results are underlined.
CUHK-PEDES
ICFG-PEDES
UFineBench
Methods
R@1
R@5
R@10
mAP
R@1
R@5
R@10
mAP
R@1
R@5
R@10
mAP
Without large-scale person-centric pretraining:
IRRA ( Jiang and Ye, 2023 )
73.38
89.93
93.71
66.13
63.46
80.25
85.82
38.06
83.53
92.94
95.95
82.79
RaSa ( Bai et al., 2023 )
76.51
90.29
94.25
69.38
65.28
80.40
85.12
41.29
86.13
94.55
96.73
84.17
TBPS-CLIP ( Cao et al., 2024 )
73.54
88.19
92.35
65.38
65.05
80.34
85.47
39.83
76.21
88.92
93.09
75.07
Table 3. Performance comparison on English TBPR with state-of-the-art methods on CUHK-PEDES, ICFG-PEDES, and UFineBench. Methods are grouped by whether they use large-scale person-centric pretraining. The best results are highlighted in bold, and the second-best results are underlined.
English
Chinese
French
German
Avg
Method
Pretraining Dataset
R@1
R@10
mAP
R@1
R@10
mAP
R@1
R@10
mAP
R@1
R@10
mAP
R@1
R@10
mAP
Bi-IRRA ( Cao et al., 2025b )
-
78.82
95.47
69.68
76.43
94.79
67.79
76.46
94.35
67.26
75.57
94.31
67.07
76.82
94.73
67.95
LUPerson-M ( Tan et al., 2024 )
79.43
95.68
70.51
76.55
94.49
68.05
76.60
94.31
67.65
75.79
94.38
67.89
77.09
94.72
68.53
MultiPedes
79.64
95.55
70.60
76.86
94.57
68.56
76.77
94.54
68.38
76.68
94.57
68.05
77.49
94.81
68.90
ViSeLA
-
78.65
95.21
70.33
76.08
94.17
68.30
76.65
94.23
68.35
75.92
94.18
67.87
76.83
94.45
68.71
Table 4. Ablation study on different pretraining datasets. "-" indicates no pretraining. Results are reported on CUHK-PEDES.
Se-L
No.
Vi-A
S-MoE
V-MoE
L-MoE
R@1
R@5
R@10
mAP
1
✓
77.67
90.97
94.39
69.54
2
✓
77.21
90.73
94.20
69.56
3
✓
✓
77.49
91.00
94.36
69.78
4
✓
✓
77.37
90.91
94.42
69.61
Table 5. Ablation study on key components of the proposed ViSeLA framework. S-MoE, V-MoE and L-MoE denote our proposed semantic-aware MoE, vanilla MoE, and language-aware MoE, respectively. Results are averaged over the CUHK-PEDES dataset. Results are averaged on CUHK-PEDES.