Synthetic-to-Real Text-Based Person Anomaly Search via Multi-Model Retrieval Fusion and Cross-Encoder Reranking
Authors: Eugene Denteh, Blessing Agyei Kyem, Andrews Danyo, Joshua Kofi Asamoah, Richard Dzinyela, Solomon Owusu-Ansah, Dickson G. Asiedu, Armstrong Aboah
Organizations: North Dakota State University, Fargo, ND, USA · Sustainable Mobility and Advanced Research in Transportation (SMART) Lab · Texas A&M University, College Station, TX, USA · Ghana Institute of Management and Public Administration
Text-based person anomaly search requires a model to retrieve individuals from a large image gallery using natural-language descriptions of both appearance and behavior. This task is challenging because model training relies on synthetic image-text pairs, whereas evaluation is performed on real surveillance images containing domain differences, near-duplicate frames, and many visually similar distractors. We present a multi-model retrieval and reranking framework designed to improve synthetic-to-real transfer. The framework first adapts Qwen3-VL-Embedding-8B to synthetic person descriptions using bidirectional contrastive learning, explicit negative sampling, false-negative masking, and domain randomization. It then combines the normalized scores of six complementary Qwen3-VL checkpoints with those of a frozen SigLIP2 model using four-view gallery encoding. The fused retrieval scores identify the strongest candidates, which are jointly evaluated with the query using Qwen3-VL-Reranker-8B. The cross-encoder and retrieval scores are subsequently blended to produce the final ranking. We also construct a distractor-matched validation protocol entirely from synthetic data to select checkpoint weights, fusion coefficients, candidate depths, and reranking parameters without accessing the official test images. The final system achieves 94.7815% mAP and 90.8493% R@1 under the official AI City Challenge 2026 Track 4 test setting, with a reported eighth-place finish on the public leaderboard.
Figures & tables
Figure 1: Synthetic-to-real text-based person anomaly retrieval setting. Given synthetic image-text pairs for training and a natural-language query at inference, the goal is to rank images from a real-world gallery according to their agreement with the described person and behavior. The examples show running and walking queries where multi-model retrieval fusion and cross-encoder reranking promote the best-matching real image to Rank 1 among visually similar distractors.
Figure 2: Overview of the proposed synthetic-to-real text-based person anomaly retrieval framework. Synthetic image-text pairs are used to adapt Qwen3-VL-Embedding-8B through bidirectional contrastive learning, explicit negative sampling, false-negative masking, and domain randomization, producing six complementary checkpoints. During inference, their weighted similarity scores are fused with scores from a frozen SigLIP2 retrieval branch using four-view gallery encoding. The combined retrieval score retains the top-100 candidates, of which the top-50 are jointly processed by Qwen3-VL-Reranker-8B and blended with the retrieval score to produce the final ranking. Distractor-matched synthetic validation is used for model and parameter selection without accessing the official challenge test images.
Figure 3: Qualitative retrieval results under the official AI City Challenge 2026 Track 4 setting [ 8 ] . Each row shows the query, the initial top-five ranking from Qwen3-VL–SigLIP2 fusion, and the final ranking after cross-encoder refinement. Green checkmarks indicate the annotated match, while red crosses indicate distractors.
Text-based person anomaly search requires retrieving real-world pedestrian images from detailed natural-language descriptions using models trained primarily on synthetic data. This Sim2Real setting is particularly challenging because visually similar candidates may differ only in subtle actions, object interactions, or appearance attributes, while applying multimodal large language models to the entire gallery is computationally expensive. We propose an anchor-constrained coarse-to-fine retrieval framework that combines global semantic matching with fine-grained verification. First, each query is represented by its original caption, a structured concatenation, and several semantic facets. Heterogeneous vision-language retrievers are then integrated through robust per-query score calibration and soft claim-aware fusion. Full and concatenated captions serve as anchors to preserve candidate recall, whereas appearance, action, and object facets provide bounded corrective evidence. The resulting candidate pool is further refined by a discriminative Qwen3 reranker and two complementary semantic verification modules based on anomaly-aware cloze completion and multi-agent evidence reasoning. Finally, an uncertainty-gated consensus module adaptively reweights the three experts on ambiguous queries. Experiments on the PAB benchmark show that the proposed soft claim-aware retrieval achieves 86.44% mAP@10, substantially outperforming individual retrieval backbones. The complete framework further improves performance to 95.41% mAP@10, 94.44% R@1, and 99.09% R@5. These results demonstrate that preserving strong global retrieval while restricting expensive semantic reasoning to a small candidate pool is effective for fine-grained Sim2Real person anomaly search. Our code will be available on Github.
Hieu Dinh Trung Pham, Phuong Huu Vu Tran, Thuan Duc Mai +6
PAMI Lab, Vietnamese-German University, Vietnam · Ho Chi Minh City University of Technology, Vietnam · Ho Chi Minh city University of Science, Vietnam +2
Text-based person anomaly retrieval aims to retrieve pedestrians exhibiting anomalous behaviors from a large image gallery using natural language descriptions. Compared with conventional text-based person retrieval, this task requires fine-grained reasoning over pedestrian appearance, behaviors, object interactions, and scene context, making robust cross-modal matching significantly more challenging. This paper presents the GENAI4E team's solution to AI City Challenge 2026 Track 4. Our framework builds upon a strong retrieval backbone and progressively integrates heterogeneous vision-language embedding models through score alignment and iterative ensemble fusion, followed by disagreement-aware VLM reranking for ambiguous queries. On the official Pedestrian Anomaly Behavior (PAB) benchmark, our approach achieves 90.92% mAP, 85.13% Recall@1, 97.72% Recall@5, and 98.68% Recall@10, demonstrating the effectiveness of combining complementary vision-language representations with selective multimodal reasoning for large-scale text-based person anomaly retrieval.
Huu-An Vu, Cam Tu Tran Thi, Thanh Toan Le Ngo +5
Hanoi University of Science and Technology, Hanoi, Vietnam · GenAI4E Lab · University of Information Technology, VNU-HCM, Ho Chi Minh City, Vietnam +4
Text-based person anomaly search requires distinguishing individuals based on fine-grained, context-dependent behaviors rather than mere appearance. Existing methods struggle to capture these context-conditioned actions, frequently relying on isolated skeletal geometry, discarding raw query details during reformulation, or utilizing absolute pointwise scoring for multimodal verification. To address these limitations, we propose ActPair, a unified three-stage coarse-to-fine framework that combines action-aligned retrieval with pairwise multimodal reranking to bridge the pose-semantic gap. First, we fine-tune a vision-language model (VLM) with an action-aligned multi-task objective that encourages the representations to encode action-discriminative semantics. Second, we perform parallel late-fusion retrieval using the original query and a large language model (LLM)-generated context-grounded rewrite, retaining complementary details from both semantic views. Finally, we propose an efficient off-the-shelf reranking module that leverages a pivot-promote algorithm to perform direct pairwise visual comparisons, mitigating residual spatial and compositional ambiguities without the prohibitive inference costs of exhaustive evaluation. Extensive experiments demonstrate that our framework achieves the best results among the compared methods on the Pedestrian Anomaly Behavior (PAB) public test and transfers effectively to an unseen, non-anomaly-specific dataset.