Harnessing Weak Pair Uncertainty for Text-based Person Search

Jintao Sun; Zhedong Zheng; Gangyi Ding

cs.CVApr 10, 2026

Harnessing Weak Pair Uncertainty for Text-based Person Search

Authors: Jintao Sun, Zhedong Zheng, Gangyi Ding

Organizations: School of Computer Science and Technology, Beijing Institute of Technology, China · Faculty of Science and Technology, University of Macau, China

Abstract

In this paper, we study the text-based person search, which is to retrieve the person of interest via natural language description. Prevailing methods usually focus on the strict one-to-one correspondence pair matching between the visual and textual modality, such as contrastive learning. However, such a paradigm unintentionally disregards the weak positive image-text pairs, which are of the same person but the text descriptions are annotated from different views (cameras). To take full use of weak positives, we introduce an uncertainty-aware method to explicitly estimate image-text pair uncertainty, and incorporate the uncertainty into the optimization procedure in a smooth manner. Specifically, our method contains two modules: uncertainty estimation and uncertainty regularization. (1) Uncertainty estimation is to obtain the relative confidence on the given positive pairs; (2) Based on the predicted uncertainty, we propose the uncertainty regularization to adaptively adjust loss weight. Additionally, we introduce a group-wise image-text matching loss to further facilitate the representation space among the weak pairs. Compared with existing methods, the proposed method explicitly prevents the model from pushing away potentially weak positive candidates. Extensive experiments on three widely-used datasets, .e.g, CUHK-PEDES, RSTPReid and ICFG-PEDES, verify the mAP improvement of our method against existing competitive methods +3.06%, +3.55% and +6.94%, respectively.

Figures & tables

Figure 1: Here we show the discrepancy between weak positive pairs (dotted arrows) and positive pairs (solid arrows) of the same identity. For instance, there are four image-text pairs of one person. Green shows a shared description among four texts, blue denotes the description discrepancy, and red indicates the unique description. We can observe that the text description is strongly related to camera views instead of only depending on the identity. Such difference is mainly due to the text annotation process, where annotators can only see a single view of the person. Considering such matching difficulty, most previous methods, thus, take advantage of the positive pairs for strict one-to-one matching (solid arrows), while disregarding the weak positive pairs (dashed arrows). In contrast, we mine the relation among weak positive image-text pairs and explicitly harness such weak positive pairs to learn discriminative cross-modality matching.

Figure 2: An architecture overview of our approach. Firstly, weakly positive text and weakly positive image corresponding to the anchor image and text are randomly selected from the dataset according to the same ID. Then we send all sampled data to the image / text encoder to obtain the corresponding features. Secondly, we compute the ITC loss between images and texts. For the anchor image-text pairs, we calculate the original contrastive learning loss, i.e. , $\mathcal{L}_{itc}(I,T)$ via the aggregated embeddings of [CLS]. For the weak image-text pairs, we derive the contrastive objective, i.e. , $\mathcal{L}_{itc}(I,T_{w})$ , and further regularize this term with uncertainty estimation as $\mathcal{L}_{uitc}$ . The proposed uncertainty-aware contrastive loss $\mathcal{L}_{uitc}$ adaptively adjust the metric learning between weakly positive texts and images. Thirdly, ITM loss and MLM loss are calculated based on the features of image-text pairs. For weak positive counterparts, we introduce Group-wise Image-Text Matching (GITM) loss to facilitate the representation learning.

Figure 3: Intuitive illustration of our Group-wise Image-Text Matching (GITM) loss approach. (a) In the conventional ITM loss calculation, one pair of positive pairs and two negative pairs are used, resulting in limited negative pair diversity. This lack of diversity leads to a skewed representation space distribution, potentially reducing the accuracy of the model performance. Additionally, conventional ITM does not fully leverage all available image-text data, causing semantic deviations between images and texts captured from different perspectives. These deviations can further hinder the ability of the model to effectively learn positive and negative pair matching. (b) Our Group-wise Image-Text Matching (GITM) approach introduces weak positive pairs, allowing the model to learn a more robust latent space for positive pairs while accounting for more diverse scenarios. By utilizing a larger and more diverse set of negative pairs, GITM increases both the number and difficulty of these pairs, resulting in a more evenly distributed representation space and, consequently, enhanced learning accuracy.

Step	Summary
Anchor pair	Start from one strong matched image-text pair $(I_{i},T_{i})$ .
Weak positives	Sample one weak-positive image $I_{i}^{w}$ and one weak-positive text $T_{i}^{w}$ from the same identity, forming two weak-positive pairs $(I_{i},T_{i}^{w})$ and $(I_{i}^{w},T_{i})$ .
Hard negatives	Within the current mini-batch, only consider unpaired samples from different identities. The standard ITM term on the strong pair $(I_{i},T_{i})$ provides two directional hard negatives (one image $\rightarrow$ text and one text $\rightarrow$ image). In addition, for the two weak-positive branches, we mine the top- $K$ most similar negatives in each direction based on the current embeddings.
Per-anchor group	The final group contains three matched pairs, i.e., $(I_{i},T_{i})$ , $(I_{i},T_{i}^{w})$ , and $(I_{i}^{w},T_{i})$ , together with $2+2K$ mined negatives in total. Thus, neg3v4 uses $K=1$ , while neg3v6 uses $K=2$ .
Loss evaluation	Flatten all constructed pairs in the mini-batch, evaluate them using the same ITM classifier, and average the loss over anchor groups.

Table 1: Compact implementation summary of GITM for one anchor pair

(I_{i},T_{i})

Method	R@1	R@5	R@10	mAP
Dual Path [ 52 ]	44.40	66.26	75.07	-
MIA [ 32 ]	53.10	75.00	82.90	-
DSSL [ 54 ]	59.98	80.41	87.56	-
SSAN [ 10 ]	61.37	80.15	86.73	-
TIPCB [ 6 ]	63.63	82.82	89.01	-
LBUL [ 41 ]	64.04	82.66	87.22	-

Table 2: Performance comparison on CUHK-PEDES. Here we show the performance of the previous methods on the recall@1,5,10, mAP in %. For a fair comparison, we change the backbone of the baseline. Baseline: We re-implement APTM [ 48 ] with backbone Swinv2-B. ∗ indicates the use of additional information, e.g. , human parsing.

Table 6

Method	I $\rightarrow$ C			C $\rightarrow$ I
Method	R@1	R@5	R@10	R@1	R@5	R@10
Dual Path [ 52 ]	15.41	29.80	38.19	7.63	17.14	23.52
MIA [ 32 ]	19.35	36.78	46.42	10.93	23.77	32.39
SCAN [ 19 ]	21.27	39.26	48.83	13.63	28.61	37.05
SSAN [ 10 ]	24.72	43.43	53.01	16.68	33.84	43.00
LGUR [ 36 ]	34.25	52.58	60.85	25.44	44.48	54.39

Table 5: Comparison results (%) on the domain generalization tasks ( i.e. , CUHK-PEDES to ICFG-PEDES (C

\rightarrow

I) and ICFG-PEDES to CUHK-PEDES (I

\rightarrow

C)). The bold and underline texts denote the best and runner-up results, respectively.

Tasks	Method	R@1	R@5	R@10
C $\rightarrow$ R	Ours	56.35	77.30	85.30
R $\rightarrow$ C	Ours	39.49	61.88	70.63
R $\rightarrow$ I	Ours	45.04	60.71	67.28
I $\rightarrow$ R	Ours	55.70	74.55	82.45

Table 6: Our results (%) on the domain generalization tasks ( i.e. , CUHK-PEDES to RSTPReid (C

\rightarrow

R) and RSTPReid to CUHK-PEDES (R

\rightarrow

C), RSTPReid to ICFG-PEDES (R

\rightarrow

I) and ICFG-PEDES to RSTPReid (I

\rightarrow

R)).

Method	$\mathcal{L}_{uitc}$	$\mathcal{L}_{gitm}$ (neg3v4)	$\mathcal{L}_{gitm}$ (neg3v6)	R@1	R@5	R@10	mAP
Baseline				76.53	90.04	94.15	66.91
Baseline †				76.90	90.76	94.33	68.86
Baseline	✓			76.88	90.60	94.35	70.49
Baseline	✓	✓		76.85	90.77	94.54	71.89
Baseline	✓		✓	77.88	91.05	94.57	72.44

Table 7: Ablation study of our method with different settings on CUHK-PEDES. The difference between Baseline and Baseline † is that we re-implement APTM [ 48 ] with backbone Swinv2-B for a fair comparison.

\mathcal{L}_{uitc}

is the optimization objective that uses uncertainty-aware ITC to leverage information about the weak positive pairs fully.

\mathcal{L}_{gitm}

(neg3v4) is that we adopt the weak positive pairs to increase the number and difficulty of negative pairs by using Group-wise Image-Text Matching (GITM), comprising 1 positive pair, 2 weak positive pairs, and 4 negative pairs. Similarly,

\mathcal{L}_{gitm}

(neg3v6) is expanded to 1 positive pair, 2 weak positive pairs, and 6 negative pairs.

Method	I $\rightarrow$ C				C $\rightarrow$ I
Method	R@1	R@5	R@10	mAP	R@1	R@5	R@10	mAP
Baseline †	45.96	68.52	76.51	40.85	47.06	68.13	75.02	25.60
Baseline † + $\mathcal{L}_{uitc}$	45.87	68.67	76.43	41.81	47.66	68.35	75.15	26.39
Baseline † + $\mathcal{L}_{gitm}$ (neg3v6)	45.35	69.33	77.62	41.88	48.44	68.83	75.91	27.21
Ours	47.19	70.27	78.09	44.10	49.33	68.61	75.79	28.53

Table 8: Cross-domain ablation results (%) under the source-only transfer protocol on the domain generalization tasks, i.e., ICFG-PEDES

\rightarrow

CUHK-PEDES (I

\rightarrow

C) and CUHK-PEDES

\rightarrow

ICFG-PEDES (C

\rightarrow

I). “Baseline † +

\mathcal{L}_{uitc}

” uses only the uncertainty-aware ITC objective, “Baseline † +

\mathcal{L}_{gitm}

” uses only the GITM objective, and “Ours” combines both components. All models are trained on the source domain and directly evaluated on the target domain without target-domain fine-tuning. Bold denotes the best result in each column.

Table 11

Table 12

Figure 4: Visualization of the top 10 person search results on CUHK-PEDES, RSTPReid, and ICFG-PEDES. We present the results of two sets of text queries for each of the three datasets, arranged from top to bottom according to our method, in descending order based on match probability. Images in green boxes indicate correct matches, while images in red boxes represent incorrect matches.

Figure 5: Macro-averaged Precision–Recall (PR) curves on CUHK-PEDES for text-to-image retrieval. Our method consistently dominates the baseline across recall levels, indicating improved ranking quality beyond top-K recall. Positives are defined by the same-ID mapping (txt2img) used in mAP evaluation.

Panel of Figure 6 — Figure 6: Reliability analysis of the consistency-based uncertainty. We emphasize that $u_{w}=\exp(-s_{w})$ is a reliability/ambiguity proxy derived from weak-pair consistency, rather than a calibrated aleatoric/epistemic uncertainty. On CUHK-PEDES test set ( $N{=}6156$ ), incorrect top-1 matches show higher $u$ than correct ones (TP mean $0.509$ vs. FP mean $0.550$ ), and sorting by $u$ yields a risk–coverage behavior, indicating that $u$ is monotonic with retrieval risk.

Panel of Figure 7 — Figure 7: Embedding geometry and margin analysis before/after applying our uncertainty-aware learning. (a,b) Joint t-SNE visualization in a shared setting, where $\{T,I^{+},I_{w},I^{-}\}$ denote the text query, its matched image, its weak-view counterpart, and a negative image, respectively. Colored connectors indicate the associations from each query to its positives/weak-positives. (c,d) Distributions of ITC margins in the same embedding space. The vertical line at $0$ marks the decision boundary where a negative becomes as similar as (or more similar than) the positive/weak-positive. After training, both margin distributions shift right (larger mean margins), indicating more reliable separation against negatives.

Mapping of $u_{w}$	R@1	R@5	R@10	mAP
$1.5-s_{w}$	77.00	90.48	94.14	71.88
$(1.5-s_{w})^{2}$	77.73	90.60	93.96	72.35
$\exp(-s_{w})$	77.88	91.05	94.57	72.44

Table 13: Comparison of different uncertainty mappings in Eq. (3) on CUHK-PEDES. All settings are kept identical, and we only replace the mapping from the consistency score

s_{w}

to the uncertainty score

u_{w}

GITM supervision	R@1	R@5	R@10	mAP
Default (ours)	77.88	91.05	94.57	72.44
Uncertainty-weighted GITM	77.27	90.48	94.19	72.40

Table 14: Effect of introducing pair-level uncertainty into GITM on CUHK-PEDES. “Default” denotes our original design, where

u_{w}

is only used in the ITC branch. “Uncertainty-weighted GITM” additionally applies pair-level uncertainty weighting to the weak-positive GITM branches.

Method	R@1	R@5	R@10	mAP
Baseline w/o MALS pre-training	68.81	86.60	91.42	62.36
Ours w/o MALS pre-training	70.37	87.12	91.96	66.20

Table 15: Effect of removing MALS pre-training on CUHK-PEDES. Here “w/o MALS pre-training” means that we keep the same APTM architecture and standard backbone initialization, but directly train on the downstream dataset without loading the MALS pre-trained checkpoint.

Text-Based Person Search

See all 8 papers

Sep 11, 2026

Generative Retrieval for Unsupervised Text-Based Person Search

Text-based person search (TBPS) aims to retrieve images of a target person from a large image gallery based on a given natural language description. Most existing methods rely on supervised learning with manually annotated image-text pairs. In this paper, we explore unsupervised TBPS, with only unlabeled images. We propose GTR+, a two-stage generation-then-retrieval framework. In the generation stage, we introduce a tiered description generation framework designed to produce fine-grained and stylistically diverse textual descriptions through a three-tier sequential process. The base tier leverages an automated question-and-answer mechanism to generate basic visual attribute descriptions; the intermediate tier enhances fine-grained detail using an inter-sample contrastive mechanism; the advanced tier further enriches textual diversity via a stylized expansion mechanism. In the retrieval stage, to mitigate the impact of noisy pseudo texts, we develop an adaptive confidence-weighted retrieval learning framework. We model image-text pairs as clean or noisy using a Gaussian Mixture Model, calibrated by real-time image-text similarity and static text generation probability from the prior stage, yielding adaptive sample weights during training. Beyond that, we also contribute LargeFine-Person, a large-scale TBPS dataset with high-quality, fine-grained, and diverse textual annotations, enabling a practical and generalizable TBPS pre-training benchmark under unsupervised setting. Experiments on multiple TBPS benchmarks demonstrate the effectiveness and generalization of both GTR+ and LargeFine-Person. Code is available at: https://github.com/Flame-Chasers/GTR.

Mang Ye, Yucheng Ji, Yang Bai +4

School of Computer Science, Wuhan University, Wuhan, China · School of Computer Science and Technology, Soochow University, SuZhou, China · Zhipu AI, Beijing 100089, China +1

Text-Based Person Search Unsupervised Learning Image-Text Retrieval Generative Retrieval Image Retrieval

Ask PDF

Apr 7, 2026

Pretrain-then-Adapt: Uncertainty-Aware Test-Time Adaptation for Text-based Person Search

Text-based person search faces inherent limitations due to data scarcity, driven by stringent privacy constraints and the high cost of manual annotation. To mitigate this, existing methods usually rely on a Pretrain-then-Finetune paradigm, where models are first pretrained on synthetic person-caption data to establish cross-modal alignment, followed by fine-tuning on labeled real-world datasets. However, this paradigm lacks practicality in real-world deployment scenarios, where large-scale annotated target-domain data is typically inaccessible. In this work, we propose a new Pretrain-then-Adapt paradigm that eliminates reliance on extensive target-domain supervision through an offline test-time adaptation manner, enabling dynamic model adaptation using only unlabeled test data with minimal post-train time cost. To mitigate overconfidence with false positives of previous entropy-based test-time adaptation, we propose an Uncertainty-Aware Test-Time Adaptation (UATTA) framework, which introduces a bidirectional retrieval disagreement mechanism to estimate uncertainty, i.e., low uncertainty is assigned when an image-text pair ranks highly in both image-to-text and text-to-image retrieval, indicating high alignment; otherwise, high uncertainty is detected. This indicator drives offline test-time model recalibration without labels, effectively mitigating domain shift. We validate UATTA on four benchmarks, i.e., CUHK-PEDES, ICFG-PEDES, RSTPReid, and PAB, showing consistent improvements across both CLIP-based (one-stage) and XVLM-based (two-stage) frameworks. Ablation studies confirm that UATTA outperforms existing offline test-time adaptation strategies, establishing a new benchmark for label-efficient, deployable person search systems. Our code is available at https://github.com/nkuzjh/UATTA.

Jiahao Zhang, Shaofei Huang, Yaxiong Wang +1

University of Macau, Macau, China · Hefei University of Technology, Hefei, China

Text-Based Person Search Test-Time Adaptation Test-Time Adaptation of VLMs Image-Text Retrieval Person Re-Identification

Ask PDF

Jun 1, 2026

ROGLE: Robust Global-Local Alignment with Automated Region Supervision for Text-Based Person Search

Text-Based Person Search (TBPS) aims to retrieve pedestrian images using natural language queries. However, existing TBPS models, especially those based on CLIP, struggle with fine-grained understanding due to global representational bias and semantic sparsity inherited from training on short captions. This results in weak fine-grained alignment, exacerbated by the scarcity of region-level annotations. To address this, we propose ROGLE (Robust Global-Local Embedding), a unified framework that overcomes reliance on costly manual annotations through an automated Region-to-Sentence Matching (RSM) strategy. RSM automatically mines pseudo region-sentence pairs for scalable fine-grained supervision. Furthermore, ROGLE employs a multi-granular learning strategy that fuses global contrastive learning with region-level local alignment. We also introduce the P-VLG Benchmark, a large-scale dataset constructed by curating and enriching images from established public benchmarks. It features over 100,000 annotated regions and rich long-form captions, making it the first TBPS benchmark to support both global and local assessment protocols. Extensive experiments show that ROGLE significantly outperforms existing approaches, particularly on challenging long-form queries. Code and the P-VLG benchmark will be made publicly available.

Chaodong Jia, Zequn Xie, Xibei Jia +3

Text-Based Person Search Image-Text Retrieval Vision-Language Grounding Vision-Language Alignment

Ask PDF

Abstract

Text-Based Person Search

Explore similar work

Generative Retrieval for Unsupervised Text-Based Person Search

Pretrain-then-Adapt: Uncertainty-Aware Test-Time Adaptation for Text-based Person Search

ROGLE: Robust Global-Local Alignment with Automated Region Supervision for Text-Based Person Search