Harnessing Weak Pair Uncertainty for Text-based Person Search

Jintao Sun; Zhedong Zheng; Gangyi Ding

Figure 1: Here we show the discrepancy between weak positive pairs (dotted arrows) and positive pairs (solid arrows) of the same identity. For instance, there are four image-text pairs of one person. Green shows a shared description among four texts, blue denotes the description discrepancy, and red indicates the unique description. We can observe that the text description is strongly related to camera views instead of only depending on the identity. Such difference is mainly due to the text annotation process, where annotators can only see a single view of the person. Considering such matching difficulty, most previous methods, thus, take advantage of the positive pairs for strict one-to-one matching (solid arrows), while disregarding the weak positive pairs (dashed arrows). In contrast, we mine the relation among weak positive image-text pairs and explicitly harness such weak positive pairs to learn discriminative cross-modality matching.

Figure 2: An architecture overview of our approach. Firstly, weakly positive text and weakly positive image corresponding to the anchor image and text are randomly selected from the dataset according to the same ID. Then we send all sampled data to the image / text encoder to obtain the corresponding features. Secondly, we compute the ITC loss between images and texts. For the anchor image-text pairs, we calculate the original contrastive learning loss, i.e. ,

\mathcal{L}_{itc}(I,T)

via the aggregated embeddings of [CLS]. For the weak image-text pairs, we derive the contrastive objective, i.e. ,

\mathcal{L}_{itc}(I,T_{w})

, and further regularize this term with uncertainty estimation as

\mathcal{L}_{uitc}

. The proposed uncertainty-aware contrastive loss

\mathcal{L}_{uitc}

adaptively adjust the metric learning between weakly positive texts and images. Thirdly, ITM loss and MLM loss are calculated based on the features of image-text pairs. For weak positive counterparts, we introduce Group-wise Image-Text Matching (GITM) loss to facilitate the representation learning.

Figure 3: Intuitive illustration of our Group-wise Image-Text Matching (GITM) loss approach. (a) In the conventional ITM loss calculation, one pair of positive pairs and two negative pairs are used, resulting in limited negative pair diversity. This lack of diversity leads to a skewed representation space distribution, potentially reducing the accuracy of the model performance. Additionally, conventional ITM does not fully leverage all available image-text data, causing semantic deviations between images and texts captured from different perspectives. These deviations can further hinder the ability of the model to effectively learn positive and negative pair matching. (b) Our Group-wise Image-Text Matching (GITM) approach introduces weak positive pairs, allowing the model to learn a more robust latent space for positive pairs while accounting for more diverse scenarios. By utilizing a larger and more diverse set of negative pairs, GITM increases both the number and difficulty of these pairs, resulting in a more evenly distributed representation space and, consequently, enhanced learning accuracy.

Step	Summary
Anchor pair	Start from one strong matched image-text pair $(I_{i},T_{i})$ .
Weak positives	Sample one weak-positive image $I_{i}^{w}$ and one weak-positive text $T_{i}^{w}$ from the same identity, forming two weak-positive pairs $(I_{i},T_{i}^{w})$ and $(I_{i}^{w},T_{i})$ .
Hard negatives	Within the current mini-batch, only consider unpaired samples from different identities. The standard ITM term on the strong pair $(I_{i},T_{i})$ provides two directional hard negatives (one image $\rightarrow$ text and one text $\rightarrow$ image). In addition, for the two weak-positive branches, we mine the top- $K$ most similar negatives in each direction based on the current embeddings.
Per-anchor group	The final group contains three matched pairs, i.e., $(I_{i},T_{i})$ , $(I_{i},T_{i}^{w})$ , and $(I_{i}^{w},T_{i})$ , together with $2+2K$ mined negatives in total. Thus, neg3v4 uses $K=1$ , while neg3v6 uses $K=2$ .
Loss evaluation	Flatten all constructed pairs in the mini-batch, evaluate them using the same ITM classifier, and average the loss over anchor groups.

Table 1: Compact implementation summary of GITM for one anchor pair

(I_{i},T_{i})

.

Method	R@1	R@5	R@10	mAP
Dual Path [ 52 ]	44.40	66.26	75.07	-
MIA [ 32 ]	53.10	75.00	82.90	-
DSSL [ 54 ]	59.98	80.41	87.56	-
SSAN [ 10 ]	61.37	80.15	86.73	-
TIPCB [ 6 ]	63.63	82.82	89.01	-
LBUL [ 41 ]	64.04	82.66	87.22	-

Table 2: Performance comparison on CUHK-PEDES. Here we show the performance of the previous methods on the recall@1,5,10, mAP in %. For a fair comparison, we change the backbone of the baseline. Baseline: We re-implement APTM [ 48 ] with backbone Swinv2-B. ∗ indicates the use of additional information, e.g. , human parsing.

Table 6

Method	I $\rightarrow$ C			C $\rightarrow$ I
Method	R@1	R@5	R@10	R@1	R@5	R@10
Dual Path [ 52 ]	15.41	29.80	38.19	7.63	17.14	23.52
MIA [ 32 ]	19.35	36.78	46.42	10.93	23.77	32.39
SCAN [ 19 ]	21.27	39.26	48.83	13.63	28.61	37.05
SSAN [ 10 ]	24.72	43.43	53.01	16.68	33.84	43.00
LGUR [ 36 ]	34.25	52.58	60.85	25.44	44.48	54.39

Table 5: Comparison results (%) on the domain generalization tasks ( i.e. , CUHK-PEDES to ICFG-PEDES (C

\rightarrow

I) and ICFG-PEDES to CUHK-PEDES (I

\rightarrow

C)). The bold and underline texts denote the best and runner-up results, respectively.

Tasks	Method	R@1	R@5	R@10
C $\rightarrow$ R	Ours	56.35	77.30	85.30
R $\rightarrow$ C	Ours	39.49	61.88	70.63
R $\rightarrow$ I	Ours	45.04	60.71	67.28
I $\rightarrow$ R	Ours	55.70	74.55	82.45

Table 6: Our results (%) on the domain generalization tasks ( i.e. , CUHK-PEDES to RSTPReid (C

\rightarrow

R) and RSTPReid to CUHK-PEDES (R

\rightarrow

C), RSTPReid to ICFG-PEDES (R

\rightarrow

I) and ICFG-PEDES to RSTPReid (I

\rightarrow

R)).

Method	$\mathcal{L}_{uitc}$	$\mathcal{L}_{gitm}$ (neg3v4)	$\mathcal{L}_{gitm}$ (neg3v6)	R@1	R@5	R@10	mAP
Baseline				76.53	90.04	94.15	66.91
Baseline †				76.90	90.76	94.33	68.86
Baseline	✓			76.88	90.60	94.35	70.49
Baseline	✓	✓		76.85	90.77	94.54	71.89
Baseline	✓		✓	77.88	91.05	94.57	72.44

Table 7: Ablation study of our method with different settings on CUHK-PEDES. The difference between Baseline and Baseline † is that we re-implement APTM [ 48 ] with backbone Swinv2-B for a fair comparison.

\mathcal{L}_{uitc}

is the optimization objective that uses uncertainty-aware ITC to leverage information about the weak positive pairs fully.

\mathcal{L}_{gitm}

(neg3v4) is that we adopt the weak positive pairs to increase the number and difficulty of negative pairs by using Group-wise Image-Text Matching (GITM), comprising 1 positive pair, 2 weak positive pairs, and 4 negative pairs. Similarly,

\mathcal{L}_{gitm}

(neg3v6) is expanded to 1 positive pair, 2 weak positive pairs, and 6 negative pairs.

Method	I $\rightarrow$ C				C $\rightarrow$ I
Method	R@1	R@5	R@10	mAP	R@1	R@5	R@10	mAP
Baseline †	45.96	68.52	76.51	40.85	47.06	68.13	75.02	25.60
Baseline † + $\mathcal{L}_{uitc}$	45.87	68.67	76.43	41.81	47.66	68.35	75.15	26.39
Baseline † + $\mathcal{L}_{gitm}$ (neg3v6)	45.35	69.33	77.62	41.88	48.44	68.83	75.91	27.21
Ours	47.19	70.27	78.09	44.10	49.33	68.61	75.79	28.53

Table 8: Cross-domain ablation results (%) under the source-only transfer protocol on the domain generalization tasks, i.e., ICFG-PEDES

\rightarrow

CUHK-PEDES (I

\rightarrow

C) and CUHK-PEDES

\rightarrow

ICFG-PEDES (C

\rightarrow

I). “Baseline † +

\mathcal{L}_{uitc}

” uses only the uncertainty-aware ITC objective, “Baseline † +

\mathcal{L}_{gitm}

” uses only the GITM objective, and “Ours” combines both components. All models are trained on the source domain and directly evaluated on the target domain without target-domain fine-tuning. Bold denotes the best result in each column.

Table 11

Table 12

Figure 4: Visualization of the top 10 person search results on CUHK-PEDES, RSTPReid, and ICFG-PEDES. We present the results of two sets of text queries for each of the three datasets, arranged from top to bottom according to our method, in descending order based on match probability. Images in green boxes indicate correct matches, while images in red boxes represent incorrect matches.

Figure 5: Macro-averaged Precision–Recall (PR) curves on CUHK-PEDES for text-to-image retrieval. Our method consistently dominates the baseline across recall levels, indicating improved ranking quality beyond top-K recall. Positives are defined by the same-ID mapping (txt2img) used in mAP evaluation.

Figure 6: Reliability analysis of the consistency-based uncertainty. We emphasize that

u_{w}=\exp(-s_{w})

is a reliability/ambiguity proxy derived from weak-pair consistency, rather than a calibrated aleatoric/epistemic uncertainty. On CUHK-PEDES test set (

N{=}6156

), incorrect top-1 matches show higher

u

than correct ones (TP mean

0.509

vs. FP mean

0.550

), and sorting by

u

yields a risk–coverage behavior, indicating that

u

is monotonic with retrieval risk.

Figure 7: Embedding geometry and margin analysis before/after applying our uncertainty-aware learning. (a,b) Joint t-SNE visualization in a shared setting, where

\{T,I^{+},I_{w},I^{-}\}

denote the text query, its matched image, its weak-view counterpart, and a negative image, respectively. Colored connectors indicate the associations from each query to its positives/weak-positives. (c,d) Distributions of ITC margins in the same embedding space. The vertical line at

0

marks the decision boundary where a negative becomes as similar as (or more similar than) the positive/weak-positive. After training, both margin distributions shift right (larger mean margins), indicating more reliable separation against negatives.

Mapping of $u_{w}$	R@1	R@5	R@10	mAP
$1.5-s_{w}$	77.00	90.48	94.14	71.88
$(1.5-s_{w})^{2}$	77.73	90.60	93.96	72.35
$\exp(-s_{w})$	77.88	91.05	94.57	72.44

Table 13: Comparison of different uncertainty mappings in Eq. (3) on CUHK-PEDES. All settings are kept identical, and we only replace the mapping from the consistency score

s_{w}

to the uncertainty score

u_{w}

.

GITM supervision	R@1	R@5	R@10	mAP
Default (ours)	77.88	91.05	94.57	72.44
Uncertainty-weighted GITM	77.27	90.48	94.19	72.40

Table 14: Effect of introducing pair-level uncertainty into GITM on CUHK-PEDES. “Default” denotes our original design, where

u_{w}

is only used in the ITC branch. “Uncertainty-weighted GITM” additionally applies pair-level uncertainty weighting to the weak-positive GITM branches.

Method	R@1	R@5	R@10	mAP
Baseline w/o MALS pre-training	68.81	86.60	91.42	62.36
Ours w/o MALS pre-training	70.37	87.12	91.96	66.20

Table 15: Effect of removing MALS pre-training on CUHK-PEDES. Here “w/o MALS pre-training” means that we keep the same APTM architecture and standard backbone initialization, but directly train on the downstream dataset without loading the MALS pre-trained checkpoint.

Harnessing Weak Pair Uncertainty for Text-based Person Search

Abstract

Figures & tables

Text-Based Person Search

Generative Retrieval for Unsupervised Text-Based Person Search

Pretrain-then-Adapt: Uncertainty-Aware Test-Time Adaptation for Text-based Person Search

ROGLE: Robust Global-Local Alignment with Automated Region Supervision for Text-Based Person Search

Abstract

Text-Based Person Search

Explore similar work

Generative Retrieval for Unsupervised Text-Based Person Search

Pretrain-then-Adapt: Uncertainty-Aware Test-Time Adaptation for Text-based Person Search

ROGLE: Robust Global-Local Alignment with Automated Region Supervision for Text-Based Person Search