Vision-Grounded Semantic Learning and Alignment for Scalable Multilingual Text-Based Person Retrieval
Authors: Xinyu Zhou, Fan Li, Shasha Chen, Min Cao, Ahtamjan Ahmat, Qiaoming Zhu, Mang Ye
Organizations: Soochow University, Suzhou, China · The Xinjiang Technical Institute of Physics & Chemistry, Chinese Academy of Sciences, Xinjiang, China · Wuhan University, Wuhan, China
Multilingual Text-Based Person Retrieval (TBPR) aims to retrieve individuals with texts in multiple languages. Despite its practical importance, it has been overlooked by existing research, which mainly focuses on English-only retrieval. The only prior attempt at multilingual TBPR suffers from (1) the absence of a large-scale, multilingual pretraining dataset and (2) a suboptimal model architecture for multilingual text encoding and cross-modal alignment, which employs a shared encoder that overlooks linguistic heterogeneity and relies on English as a pivot for cross-modal alignment, introducing English-centric bias. For this, we contribute MultiPedes, the first large-scale, pedestrian-centric multilingual pretraining dataset. It contains 10 million image-text pairs across four languages, constructed by a multilingual quality-aware data construction pipeline. We also propose ViSeLA (Vision-grounded Semantic Learning and Alignment), a novel framework featuring: (1) a semantic-aware multilingual learning module that extends the mixture-of-experts architecture by routing language-invariant semantics to specialized experts for improved multilingual feature learning; and (2) a vision-grounded cross-modal alignment module that employs a semantic probe to capture fine-grained semantics across modalities, enabling direct cross-modal alignment without relying on a pivot language. Extensive experiments on both multilingual and English-only TBPR datasets demonstrate the effectiveness of ViSeLA. The dataset and code are available at https://github.com/Flame-Chasers/ViSeLA
Figures & tables
Figure 1. (a) Multilingual TBPR, where person images are retrieved with texts in multiple languages. (b) Existing method adopts a shared text encoder for all languages and relies on an English-centered knowledge transfer paradigm for cross-lingual cross-modal alignment. (c) Our ViSeLA framework introduces Se-L module for multilingual learning and Vi-A module for vision-grounded cross-modal alignment.
Figure 2. Multilingual quality-aware data construction pipeline consisting of two stages: (1) data curation with consensus-guided filtering, and (2) multilingual text translation. In the first stage, consensus-guided filtering leverages both domain-specific and domain-agnostic evaluators to identify and remove noisy samples from large-scale image–text pairs collected from public datasets, resulting in higher-quality data. In the second stage, large language models are employed to perform machine translation at scale, generating multilingual descriptions and constructing the MultiPedes dataset.
Dataset
#Images
#Texts
Avg. Len
Language
TTR
CLIPScore
LUPerson-T ( Shao et al., 2023 )
1.30M
1.30M
27.39
En
0.88
0.22
LUPerson-M ( Tan et al., 2024 )
1.00M
4.00M
28.61
En
0.80
0.36
HAM-PEDES ( Jiang et al., 2025 )
1.00M
2.00M
26.30
En
0.80
0.40
SynTBPR ( Cao et al., 2025a )
1.63M
6.62M
34.10
En
0.81
0.43
WebPerson ( Zheng et al., 2025b )
5M
10M
23.34
En
0.81
0.40
Table 1. Comparison with other mainstream pretraining datasets for TBPR. # denotes the number of samples, and Avg. Len indicates the average text length.
Figure 3. Comparison of COMET-Wiki score distributions between MultiPedes and COCO-35L ( Thapliyal et al., 2022 ) . The scores are averaged over three target languages for each sample.
Figure 4. Overview of our proposed ViSeLA framework. It consists of two key submodules: (1) the Se-L module, which leverages semantic information to route input text features to specialized experts, thereby extracting language-invariant textual features; and (2) the Vi-A module, which employs the semantic probe to capture fine-grained cross-modal semantics and enables image-grounded cross-modal alignment.
English
Chinese
French
German
Avg
Dataset
Method
R1
R10
mAP
R1
R10
mAP
R1
R10
mAP
R1
R10
mAP
R1
R10
mAP
Without Pre-training on Large-scale Person Data:
CUHK-PEDE(M)
CCLM ( Zeng et al., 2022 )
74.68
93.42
65.78
70.83
92.85
63.14
71.17
92.30
62.89
70.19
92.24
62.47
71.72
92.70
63.57
X²-VLM ( Zeng et al., 2023 )
76.97
94.96
67.86
74.81
94.17
66.00
75.18
94.18
66.17
74.71
94.30
65.73
75.42
94.40
66.44
CCRK ( Nie et al., 2024 )
70.11
92.53
62.10
68.31
91.91
60.69
68.29
91.36
60.33
67.01
90.74
59.54
68.43
91.64
60.67
Table 2. Performance comparison with state-of-the-art methods on multilingual TBPR datasets. We categorize these methods into two groups based on whether they are pre-trained on large-scale person data (per their original papers): Bi-IRRA uses an English-only dataset (LUPerosn-M ( Tan et al., 2024 ) ), while ViSeLA leverages our proposed MultiPedes dataset. Results are reported for four languages and their average. The best results are highlighted in bold, and the second-best results are underlined.
CUHK-PEDES
ICFG-PEDES
UFineBench
Methods
R@1
R@5
R@10
mAP
R@1
R@5
R@10
mAP
R@1
R@5
R@10
mAP
Without large-scale person-centric pretraining:
IRRA ( Jiang and Ye, 2023 )
73.38
89.93
93.71
66.13
63.46
80.25
85.82
38.06
83.53
92.94
95.95
82.79
RaSa ( Bai et al., 2023 )
76.51
90.29
94.25
69.38
65.28
80.40
85.12
41.29
86.13
94.55
96.73
84.17
TBPS-CLIP ( Cao et al., 2024 )
73.54
88.19
92.35
65.38
65.05
80.34
85.47
39.83
76.21
88.92
93.09
75.07
Table 3. Performance comparison on English TBPR with state-of-the-art methods on CUHK-PEDES, ICFG-PEDES, and UFineBench. Methods are grouped by whether they use large-scale person-centric pretraining. The best results are highlighted in bold, and the second-best results are underlined.
English
Chinese
French
German
Avg
Method
Pretraining Dataset
R@1
R@10
mAP
R@1
R@10
mAP
R@1
R@10
mAP
R@1
R@10
mAP
R@1
R@10
mAP
Bi-IRRA ( Cao et al., 2025b )
-
78.82
95.47
69.68
76.43
94.79
67.79
76.46
94.35
67.26
75.57
94.31
67.07
76.82
94.73
67.95
LUPerson-M ( Tan et al., 2024 )
79.43
95.68
70.51
76.55
94.49
68.05
76.60
94.31
67.65
75.79
94.38
67.89
77.09
94.72
68.53
MultiPedes
79.64
95.55
70.60
76.86
94.57
68.56
76.77
94.54
68.38
76.68
94.57
68.05
77.49
94.81
68.90
ViSeLA
-
78.65
95.21
70.33
76.08
94.17
68.30
76.65
94.23
68.35
75.92
94.18
67.87
76.83
94.45
68.71
Table 4. Ablation study on different pretraining datasets. "-" indicates no pretraining. Results are reported on CUHK-PEDES.
Se-L
No.
Vi-A
S-MoE
V-MoE
L-MoE
R@1
R@5
R@10
mAP
1
✓
77.67
90.97
94.39
69.54
2
✓
77.21
90.73
94.20
69.56
3
✓
✓
77.49
91.00
94.36
69.78
4
✓
✓
77.37
90.91
94.42
69.61
Table 5. Ablation study on key components of the proposed ViSeLA framework. S-MoE, V-MoE and L-MoE denote our proposed semantic-aware MoE, vanilla MoE, and language-aware MoE, respectively. Results are averaged over the CUHK-PEDES dataset. Results are averaged on CUHK-PEDES.
Text-Based Person Search (TBPS) aims to retrieve pedestrian images using natural language queries. However, existing TBPS models, especially those based on CLIP, struggle with fine-grained understanding due to global representational bias and semantic sparsity inherited from training on short captions. This results in weak fine-grained alignment, exacerbated by the scarcity of region-level annotations. To address this, we propose ROGLE (Robust Global-Local Embedding), a unified framework that overcomes reliance on costly manual annotations through an automated Region-to-Sentence Matching (RSM) strategy. RSM automatically mines pseudo region-sentence pairs for scalable fine-grained supervision. Furthermore, ROGLE employs a multi-granular learning strategy that fuses global contrastive learning with region-level local alignment. We also introduce the P-VLG Benchmark, a large-scale dataset constructed by curating and enriching images from established public benchmarks. It features over 100,000 annotated regions and rich long-form captions, making it the first TBPS benchmark to support both global and local assessment protocols. Extensive experiments show that ROGLE significantly outperforms existing approaches, particularly on challenging long-form queries. Code and the P-VLG benchmark will be made publicly available.
Previous research shows that vision-language (VL) models struggle to understand fine-grained image details and tend to focus on the overall layout. Moreover, any weakness in English tends to be further amplified in the multilingual setting. To study that, we extend InpaintCOCO, an English dataset focused on fine-grained visual details, to 13 typologically diverse languages across five language families, creating MultiInpaintCOCO - a new multilingual benchmark of 953 minimal-pair examples. We further test several state-of-the-art models in multilingual multimodal retrieval, considering both light VL CLIP-like encoders and large language models. Our results show that performance degrades substantially for lower-resource languages, with the strongest predictor being the data resources available for the languages rather than typological similarity to English. In cross-lingual retrieval, most models exhibit a systematic bias toward English: when given a correct caption in a non-English language and an incorrect one in English, models preferentially select the English caption. This bias varies considerably across architectures. Among the tested models, Gemma 3 shows the strongest English preference while Qwen 3 is the least biased, despite both reaching similar monolingual accuracy across languages. The code is available at https://github.com/ufal/multi-inpaint-coco.
Andrei-Alexandru Manea, Jindřich Libovický, Philipp J. Rösch
Faculty of Mathematics and Physics, Charles University, Czech Republic · Bundeswehr University Munich, Germany
Text-based person anomaly retrieval aims to retrieve pedestrians exhibiting anomalous behaviors from a large image gallery using natural language descriptions. Compared with conventional text-based person retrieval, this task requires fine-grained reasoning over pedestrian appearance, behaviors, object interactions, and scene context, making robust cross-modal matching significantly more challenging. This paper presents the GENAI4E team's solution to AI City Challenge 2026 Track 4. Our framework builds upon a strong retrieval backbone and progressively integrates heterogeneous vision-language embedding models through score alignment and iterative ensemble fusion, followed by disagreement-aware VLM reranking for ambiguous queries. On the official Pedestrian Anomaly Behavior (PAB) benchmark, our approach achieves 90.92% mAP, 85.13% Recall@1, 97.72% Recall@5, and 98.68% Recall@10, demonstrating the effectiveness of combining complementary vision-language representations with selective multimodal reasoning for large-scale text-based person anomaly retrieval.
Huu-An Vu, Cam Tu Tran Thi, Thanh Toan Le Ngo +5
Hanoi University of Science and Technology, Hanoi, Vietnam · GenAI4E Lab · University of Information Technology, VNU-HCM, Ho Chi Minh City, Vietnam +4