Pretrain-then-Adapt: Uncertainty-Aware Test-Time Adaptation for Text-based Person Search

Jiahao Zhang; Shaofei Huang; Yaxiong Wang; Zhedong Zheng

cs.IRApr 7, 2026

Pretrain-then-Adapt: Uncertainty-Aware Test-Time Adaptation for Text-based Person Search

Authors: Jiahao Zhang, Shaofei Huang, Yaxiong Wang, Zhedong Zheng

Organizations: University of Macau, Macau, China · Hefei University of Technology, Hefei, China

Abstract

Text-based person search faces inherent limitations due to data scarcity, driven by stringent privacy constraints and the high cost of manual annotation. To mitigate this, existing methods usually rely on a Pretrain-then-Finetune paradigm, where models are first pretrained on synthetic person-caption data to establish cross-modal alignment, followed by fine-tuning on labeled real-world datasets. However, this paradigm lacks practicality in real-world deployment scenarios, where large-scale annotated target-domain data is typically inaccessible. In this work, we propose a new Pretrain-then-Adapt paradigm that eliminates reliance on extensive target-domain supervision through an offline test-time adaptation manner, enabling dynamic model adaptation using only unlabeled test data with minimal post-train time cost. To mitigate overconfidence with false positives of previous entropy-based test-time adaptation, we propose an Uncertainty-Aware Test-Time Adaptation (UATTA) framework, which introduces a bidirectional retrieval disagreement mechanism to estimate uncertainty, i.e., low uncertainty is assigned when an image-text pair ranks highly in both image-to-text and text-to-image retrieval, indicating high alignment; otherwise, high uncertainty is detected. This indicator drives offline test-time model recalibration without labels, effectively mitigating domain shift. We validate UATTA on four benchmarks, i.e., CUHK-PEDES, ICFG-PEDES, RSTPReid, and PAB, showing consistent improvements across both CLIP-based (one-stage) and XVLM-based (two-stage) frameworks. Ablation studies confirm that UATTA outperforms existing offline test-time adaptation strategies, establishing a new benchmark for label-efficient, deployable person search systems. Our code is available at https://github.com/nkuzjh/UATTA.

Figures & tables

Panel of Figure 2 — Figure 2. Statistical Overview of our Uncertainty Indicator on (a) RSTPReid ( Zhu et al., 2021 ) and (b) CUHK-PEDES ( Li et al., 2017 ) . We count the True Positives (TP) and False Positives (FP) sample number in the initial ranking list before adaptation according to the proposed uncertainty score. TP samples consistently cluster in the low-uncertainty region, while FP samples concentrate in the high-uncertainty region across both benchmarks. Therefore, it could serve as the indicator for the test-time adaptation.

Figure 3. Uncertainty-aware Test-Time Adaptation Framework(UATTA) . Given the image gallery set $\mathcal{G}_{I}$ and text query set $\mathcal{Q}_{T}$ in the test set, we first select reliable samples by Cycle-Consistency Selection and obtain reliable text query set $\mathcal{Q}^{\prime}_{T}$ and reliable image gallery set $\mathcal{G}^{\prime}_{I}$ . Then we compute the similarity matrix $S(T,I)$ for every pairs to calculate uncertainty. Finally we re-weight the entropy minimization objective with calculated uncertainty. As shown in Cycle-Consistency Selection stage, we sort samples who are mutual top- $K$ neighbors, in which the initial given text $t^{1}_{q}$ is supposed to be inversely found in $\mathcal{G}^{1}_{K\cdot K}$ by images in retrieval result set $\mathcal{G}^{1}_{K}$ of $t^{1}_{q}$ . Conversely, text $t^{2}_{q}$ is unreliable as image set $\mathcal{G}^{2}_{K\cdot K}$ do not contain itself. After selection, based on the reliable images and texts, we further exploits the Bidirectional Retrieval Disagreement mechanism to estimate uncertainty with both text-to-image top- $1$ retrieval probability and inverse image-to-text retrieval probability, as detailed in Bidirectional Retrieval Disagreement Uncertainty stage. This uncertainty signal is calculated by $D(t,i)$ , as detailed in Eq. 9 and dynamically re-weights the entropy minimization objective to uncertainty-weighted gradient re-calibration loss $\mathcal{L}_{\text{UATTA}}$ , as detailed in Eq. 10 . Our UATTA framework mitigates domain gaps with minimal adaptation cost and zero extra architecture.

Pure pretraining (no adaptation / finetuning)
Method	Type	Post-train	R@1	R@5	R@10	mAP
MRA ( Yang et al., 2026 )	Pretrain	—	9.91	23.66	31.45	17.15
RaSa ( Bai et al., 2023a )	Pretrain	—	21.74	27.30	27.96	24.35
WoRA ( Sun et al., 2025 )	Pretrain	—	22.25	45.91	53.54	33.39
APTM ( Yang et al., 2023 )	Pretrain	—	22.90	45.80	52.38	33.56
CAMeL ( Yu et al., 2025 )	Pretrain	—	24.47	50.00	58.75	36.75

Table 1. Quantitative comparison of our proposed Pretrain-then-Adapt paradigm with state-of-the-art methods on Text-based Person Anomaly Search benchmark PAB ( Yang et al., 2025 ) . The gpu used for post-training is NVIDIA GeForce RTX 3090 GPU. Best results are bold . Second best results are underlined .

Method	RSTPReid				CUHK-PEDES				ICFG-PEDES
Method	R1	R5	R10	mAP	R1	R5	R10	mAP	R1	R5	R10	mAP
Pure pretraining (no adaptation / finetuning)
CLIP ( Radford et al., 2021 )	12.65	27.16	–	11.15	6.67	17.91	–	2.51	13.45	33.85	–	10.31
LuPerson-T ( Shao et al., 2023 )	22.40	–	–	17.08	21.88	–	–	19.96	11.46	–	–	4.56
SYNTH-PEDES ( Zuo et al., 2024 )	42.69	–	–	31.18	57.58	–	–	52.45	57.08	–	–	32.06
LuPerson-MLLM ( Tan et al., 2024 )	51.65	74.20	82.85	38.31	38.29	56.60	64.56	20.43	57.61	75.99	82.76	51.45

Table 2. Quantitative comparison of our proposed Pretrain-then-Adapt paradigm with state-of-the-art direct transfer models and other existing Test-Time-Adaptation , Semi-supervised and Unsupervised methods on real-world text-based person search benchmarks ( Li et al., 2017 ; Zhu et al., 2021 ; Ding et al., 2021 ) . Best results are bold . Second best results are underlined .

Uncertainty Formulation	Bidirectional	RSTPReid
	Retrieval Probability	R1	R5	R10	mAP
$\exp\left(1-\dfrac{p_{\text{T2I}}+p_{\text{I2T}}}{2}\right)$	$p_{\text{T2I}}=\text{softmax}(\mathbf{s}^{\text{top-$ K $}}_{\text{T2I}})$ $p_{\text{I2T}}=\text{softmax}(\mathbf{s}^{\text{top-$ K $}}_{\text{I2T}})$	$61.40$	$80.50$	$87.75$	$46.16$
$\left\|\log(p_{\text{T2I}}+\varepsilon)-\log(p_{\text{I2T}}+\varepsilon)\right\|$	same as above	$61.75$	$81.30$	$88.40$	$45.88$
$\exp\left(\dfrac{\|p_{\text{T2I}}-p_{\text{I2T}}\|}{\frac{p_{\text{T2I}}+p_{\text{I2T}}}{2}}\right)$	same as above	61.85	$81.40$	$88.40$	$46.37$
$\exp\left(\dfrac{\|p_{\text{T2I}}\cdot N_{\text{T2I}}-p_{\text{I2T}}\cdot N_{\text{I2T}}\|}{\frac{p_{\text{T2I}}\cdot N_{\text{T2I}}+p_{\text{I2T}}\cdot N_{\text{I2T}}}{2}}\right)$	same as above	$61.75$	$81.35$	$88.60$	46.58
$\exp\left(\dfrac{\|p_{\text{T2I}}-p_{\text{I2T}}\cdot\frac{N_{\text{I2T}}}{N_{\text{T2I}}}\|}{\frac{p_{\text{T2I}}+p_{\text{I2T}}\cdot\frac{N_{\text{I2T}}}{N_{\text{T2I}}}}{2}}\right)$	same as above	$61.70$	$81.45$	$88.60$	$46.57$

Table 3. Comparison of Uncertainty Formulations on RSTPReid ( Zhu et al., 2021 ) benchmark.

p_{\text{T2I}}

is the text-to-image retrieval probability,

p_{\text{I2T}}

is the inverse retrieval probability, which uses the gallery image from

p_{\text{T2I}}

to retrieve the corresponding query text.

N_{\text{T2I}}

is the size of image gallery per identity.

N_{\text{I2T}}

is the size of text query per identity.

\epsilon

is a small constant to prevent divided by zero.

\mathbf{s}^{\text{top-

}}_{\text{T2I}}

denotes the top-

K

similarity matrix of text-to-image retrieval. Equally,

\mathbf{s}^{\text{top-

}}_{\text{I2T}}

denotes the top-

K

similarity matrix of inverse directional image-to-text retrieval. The similarity matrix is transformed by a softmax function to obtain retrieval probabilities for the top-

K

results. Best results are bolded .

Figure 4. Ablation study of bidirectional Top- $K$ retrieval consistent sample selection on RSTPReid. $K$ denotes the mutual top range in bidirectional retrieval. Best performance is achieved at $K=3$ . Since each identity in RSTPReid contains 5 ground-truth images, we adopt $K=5$ as the default setting to represent the borderline of true and false positives.

Ratio	R@1	R@5	R@10	mAP
1 : 1	60.70	81.20	88.50	46.01
1 : 3	61.85	81.40	88.40	46.37
1 : 5	60.95	80.85	88.05	46.10
1 : 7	61.55	81.35	88.75	46.32
1 : 15	60.80	81.00	88.50	46.24

Table 4. Ablation of the ratio between positive and negatives on RSTPReid benchmarks.. We apply different ratio of positive and negatives to compute entropy. The ratio of 1 : 3 improves the stability in test-time adaptation. Our default setting is in gray .

Method	Tuning Layers	R@1	R@5	R@10	mAP
Baseline	–	60.64	77.50	83.26	35.54
CoOp* ( Zhou et al., 2022 )	Prompt Emb.	58.60	79.65	87.50	43.65
Prefix-Tuning* ( Li and Liang, 2021 )	Prefix Emb.	26.25	51.45	63.75	23.14
LoRA* ( Hu et al., 2022 )	LoRA Matrix	49.80	73.80	82.70	38.61
Ours	Norm. Layer	61.85	81.40	88.40	46.37

Table 5. Comparison of other prevailing lightweight tuning methods on RSTPReid ( Zhu et al., 2021 ) benchmark.

Figure 6. T-SNE Visualization of Feature Space Shifts on RSTPReid. 3 distinct point types represent: original query features (circles) before TTA, query features after TTA (diamonds), and gallery features (squares). Different colors distinguish individual person identities.

Text-Based Person Search

See all 8 papers

Sep 11, 2026

Generative Retrieval for Unsupervised Text-Based Person Search

Text-based person search (TBPS) aims to retrieve images of a target person from a large image gallery based on a given natural language description. Most existing methods rely on supervised learning with manually annotated image-text pairs. In this paper, we explore unsupervised TBPS, with only unlabeled images. We propose GTR+, a two-stage generation-then-retrieval framework. In the generation stage, we introduce a tiered description generation framework designed to produce fine-grained and stylistically diverse textual descriptions through a three-tier sequential process. The base tier leverages an automated question-and-answer mechanism to generate basic visual attribute descriptions; the intermediate tier enhances fine-grained detail using an inter-sample contrastive mechanism; the advanced tier further enriches textual diversity via a stylized expansion mechanism. In the retrieval stage, to mitigate the impact of noisy pseudo texts, we develop an adaptive confidence-weighted retrieval learning framework. We model image-text pairs as clean or noisy using a Gaussian Mixture Model, calibrated by real-time image-text similarity and static text generation probability from the prior stage, yielding adaptive sample weights during training. Beyond that, we also contribute LargeFine-Person, a large-scale TBPS dataset with high-quality, fine-grained, and diverse textual annotations, enabling a practical and generalizable TBPS pre-training benchmark under unsupervised setting. Experiments on multiple TBPS benchmarks demonstrate the effectiveness and generalization of both GTR+ and LargeFine-Person. Code is available at: https://github.com/Flame-Chasers/GTR.

Mang Ye, Yucheng Ji, Yang Bai +4

School of Computer Science, Wuhan University, Wuhan, China · School of Computer Science and Technology, Soochow University, SuZhou, China · Zhipu AI, Beijing 100089, China +1

Text-Based Person Search Unsupervised Learning Image-Text Retrieval Generative Retrieval Image Retrieval

Ask PDF

Jun 1, 2026

ROGLE: Robust Global-Local Alignment with Automated Region Supervision for Text-Based Person Search

Text-Based Person Search (TBPS) aims to retrieve pedestrian images using natural language queries. However, existing TBPS models, especially those based on CLIP, struggle with fine-grained understanding due to global representational bias and semantic sparsity inherited from training on short captions. This results in weak fine-grained alignment, exacerbated by the scarcity of region-level annotations. To address this, we propose ROGLE (Robust Global-Local Embedding), a unified framework that overcomes reliance on costly manual annotations through an automated Region-to-Sentence Matching (RSM) strategy. RSM automatically mines pseudo region-sentence pairs for scalable fine-grained supervision. Furthermore, ROGLE employs a multi-granular learning strategy that fuses global contrastive learning with region-level local alignment. We also introduce the P-VLG Benchmark, a large-scale dataset constructed by curating and enriching images from established public benchmarks. It features over 100,000 annotated regions and rich long-form captions, making it the first TBPS benchmark to support both global and local assessment protocols. Extensive experiments show that ROGLE significantly outperforms existing approaches, particularly on challenging long-form queries. Code and the P-VLG benchmark will be made publicly available.

Chaodong Jia, Zequn Xie, Xibei Jia +3

Text-Based Person Search Image-Text Retrieval Vision-Language Grounding Vision-Language Alignment

Ask PDF

Apr 10, 2026

Harnessing Weak Pair Uncertainty for Text-based Person Search

In this paper, we study the text-based person search, which is to retrieve the person of interest via natural language description. Prevailing methods usually focus on the strict one-to-one correspondence pair matching between the visual and textual modality, such as contrastive learning. However, such a paradigm unintentionally disregards the weak positive image-text pairs, which are of the same person but the text descriptions are annotated from different views (cameras). To take full use of weak positives, we introduce an uncertainty-aware method to explicitly estimate image-text pair uncertainty, and incorporate the uncertainty into the optimization procedure in a smooth manner. Specifically, our method contains two modules: uncertainty estimation and uncertainty regularization. (1) Uncertainty estimation is to obtain the relative confidence on the given positive pairs; (2) Based on the predicted uncertainty, we propose the uncertainty regularization to adaptively adjust loss weight. Additionally, we introduce a group-wise image-text matching loss to further facilitate the representation space among the weak pairs. Compared with existing methods, the proposed method explicitly prevents the model from pushing away potentially weak positive candidates. Extensive experiments on three widely-used datasets, .e.g, CUHK-PEDES, RSTPReid and ICFG-PEDES, verify the mAP improvement of our method against existing competitive methods +3.06%, +3.55% and +6.94%, respectively.

Jintao Sun, Zhedong Zheng, Gangyi Ding

School of Computer Science and Technology, Beijing Institute of Technology, China · Faculty of Science and Technology, University of Macau, China

Text-Based Person Search Image-Text Retrieval Cross-Modal Feature Matching Multimodal Contrastive Learning Person Re-Identification

Ask PDF

Abstract

Text-Based Person Search

Explore similar work

Generative Retrieval for Unsupervised Text-Based Person Search

ROGLE: Robust Global-Local Alignment with Automated Region Supervision for Text-Based Person Search

Harnessing Weak Pair Uncertainty for Text-based Person Search