Text-based person anomaly search requires distinguishing individuals based on fine-grained, context-dependent behaviors rather than mere appearance. Existing methods struggle to capture these context-conditioned actions, frequently relying on isolated skeletal geometry, discarding raw query details during reformulation, or utilizing absolute pointwise scoring for multimodal verification. To address these limitations, we propose ActPair, a unified three-stage coarse-to-fine framework that combines action-aligned retrieval with pairwise multimodal reranking to bridge the pose-semantic gap. First, we fine-tune a vision-language model (VLM) with an action-aligned multi-task objective that encourages the representations to encode action-discriminative semantics. Second, we perform parallel late-fusion retrieval using the original query and a large language model (LLM)-generated context-grounded rewrite, retaining complementary details from both semantic views. Finally, we propose an efficient off-the-shelf reranking module that leverages a pivot-promote algorithm to perform direct pairwise visual comparisons, mitigating residual spatial and compositional ambiguities without the prohibitive inference costs of exhaustive evaluation. Extensive experiments demonstrate that our framework achieves the best results among the compared methods on the Pedestrian Anomaly Behavior (PAB) public test and transfers effectively to an unseen, non-anomaly-specific dataset.
Figures & tables
Figure 1: Overview of ActPair . An action-aligned dual encoder retrieves candidates from the original query and a context-grounded rewrite. Their rankings are fused before pairwise multimodal reranking with pivot-promote.
Figure 2: Action-aligned representation learning. Image and text embeddings are optimized by the SigLIP2 objective, shared action supervision, and cross-modal action consistency.
Training-free
Fine-tuned on PAB
Method
Venue
R@1
R@5
R@10
mAP
R@1
R@5
R@10
mAP
MRA [ 22 ]
PR’26
9.91
23.66
31.45
17.15
70.53
94.69
97.47
81.59
RaSa [ 21 ]
IJCAI’23
21.74
27.30
27.96
24.35
80.79
98.89
99.65
89.20
WoRA [ 15 ]
WWW’25
22.25
45.91
53.54
33.39
74.47
96.82
98.48
84.60
APTM [ 24 ]
MM’23
22.90
45.80
52.38
33.56
72.14
95.30
97.17
82.78
CAMeL [ 25 ]
TIFS’25
24.47
50.00
58.75
36.75
74.30
96.79
98.84
84.20
Table 1: Quantitative comparison on the original PAB test protocol under training-free and PAB fine-tuning settings.
Method
Venue
Supervised
Zero-shot Transfer
R@1
R@5
R@10
R@1
R@5
R@10
DSSL [ 28 ]
MM’21
32.43
55.08
63.19
–
–
–
IVT [ 14 ]
ECCVW’22
46.70
70.00
78.80
43.70
65.10
75.55
CFine [ 20 ]
TIP’23
50.55
72.50
81.60
47.40
70.60
79.35
IRRA [ 4 ]
CVPR’23
60.20
81.30
88.20
45.10
69.20
78.75
RDE [ 11 ]
CVPR’24
65.35
83.95
89.90
–
–
–
Table 2: Comparison on RSTPReid under supervised fine-tuning and cross-dataset transfer. In the zero-shot setting, IVT, IRRA, CFine, APTM, and RaSa are trained on ICFG-PEDES, while CMP and Ours are trained on PAB. “–” denotes unavailable results.
Figure 3: Qualitative top-5 retrieval results across the three stages. The numbers indicate the ground-truth ranks after action-aligned retrieval, dual-view query fusion, and pairwise reranking. Queries 1–3 are successful cases, while Query 4 illustrates a failure on complex spatial and action-state constraints.
Method
Challenge Set
Public Test
R@1
R@5
R@10
mAP
R@1
R@5
R@10
mAP
CMP [ 23 ]
69.41
89.43
91.91
–
84.93
99.09
99.75
91.66
Action-Aligned Retriever
72.14
92.77
95.50
81.68
84.98
99.29
99.60
91.73
+ Dual-View Query
75.73
94.99
97.27
84.48
85.95
99.34
99.85
92.30
+ Pairwise Reranking
84.88
96.36
97.27
90.14
88.62
99.75
99.85
93.97
Table 3: Progressive evaluation on the AI City 2026 challenge test and the original PAB test.
Backbone
Supervision
R@1
R@5
R@10
Effect of pose information in CMP [ 23 ]
CMP
RGB and text
68.60
87.51
89.64
CMP
RGB, text, and pose
69.41
89.43
91.91
Controlled comparison under the same SigLIP2 backbone [ 17 ]
SigLIP2
Pose-based supervision
67.95
91.20
94.74
SigLIP2
Action-based supervision
72.14
92.77
95.50
Table 4: Controlled comparison of pose- and action-based supervision on the challenge set. The SigLIP2 [ 17 ] comparison keeps the backbone and training configuration fixed, isolating the effect of the supervision signal.
Reranking strategy
R@1 ↑
R@5 ↑
R@10 ↑
mAP ↑
Req./Q ↓
No VLM reranking
75.73
94.99
97.27
84.48
0.00
Qwen3.5 [ 16 ] pointwise
80.13
96.41
97.27
87.70
10.00
Qwen3.5 [ 16 ] pairwise
84.88
96.36
97.27
90.14
9.09
Pairwise − Pointwise
+4.75
−0.05
0.00
+2.44
− 0.91
Paired bootstrap 95% CI
[+ 3.08 , + 6.42 ]
[ −0.61 , + 0.51 ]
[0, 0]
[+ 1.45 , + 3.43 ]
–
Table 5: Comparison of pointwise and pairwise VLM reranking on the challenge set. Both variants use the same Qwen3.5-9B model [ 16 ] and candidate pool. Requests denote the average number of VLM calls per query.
Text-based person anomaly retrieval aims to retrieve pedestrians exhibiting anomalous behaviors from a large image gallery using natural language descriptions. Compared with conventional text-based person retrieval, this task requires fine-grained reasoning over pedestrian appearance, behaviors, object interactions, and scene context, making robust cross-modal matching significantly more challenging. This paper presents the GENAI4E team's solution to AI City Challenge 2026 Track 4. Our framework builds upon a strong retrieval backbone and progressively integrates heterogeneous vision-language embedding models through score alignment and iterative ensemble fusion, followed by disagreement-aware VLM reranking for ambiguous queries. On the official Pedestrian Anomaly Behavior (PAB) benchmark, our approach achieves 90.92% mAP, 85.13% Recall@1, 97.72% Recall@5, and 98.68% Recall@10, demonstrating the effectiveness of combining complementary vision-language representations with selective multimodal reasoning for large-scale text-based person anomaly retrieval.
Huu-An Vu, Cam Tu Tran Thi, Thanh Toan Le Ngo +5
Hanoi University of Science and Technology, Hanoi, Vietnam · GenAI4E Lab · University of Information Technology, VNU-HCM, Ho Chi Minh City, Vietnam +4
Text-based person anomaly search retrieves specific behavioral events from surveillance archives using natural-language queries. Although recent pose-aware methods align geometric structures well, they face a fundamental Pose-Semantic Gap: semantically different actions can share similar skeletal geometries. While Multimodal Large Language Models (MLLMs) can reduce this ambiguity, using them for large-scale retrieval is computationally prohibitive. We propose the Structure-Semantic Decoupled Cascade (SSDC) framework, which decouples retrieval into two stages: (1) Structure-Aware Coarse Retrieval, where a lightweight model quickly filters candidates by skeletal similarity ; and (2) Detective Squad Interaction, a multi-agent semantic verification module. The squad consists of a Detective for fast binary filtering, an Analyst for evidence extraction, and a Writer for semantic synthesis. Finally, we re-rank candidates by fusing the synthesized captions with structural priors. Experiments on the PAB benchmark show that SSDC achieves state-of-the-art performance by balancing efficiency and semantic reasoning.
Zequn Xie, Guijin Luo, Chuxin Wang +4
Zhejiang University · National University of Singapore
Text-based person anomaly search requires retrieving real-world pedestrian images from detailed natural-language descriptions using models trained primarily on synthetic data. This Sim2Real setting is particularly challenging because visually similar candidates may differ only in subtle actions, object interactions, or appearance attributes, while applying multimodal large language models to the entire gallery is computationally expensive. We propose an anchor-constrained coarse-to-fine retrieval framework that combines global semantic matching with fine-grained verification. First, each query is represented by its original caption, a structured concatenation, and several semantic facets. Heterogeneous vision-language retrievers are then integrated through robust per-query score calibration and soft claim-aware fusion. Full and concatenated captions serve as anchors to preserve candidate recall, whereas appearance, action, and object facets provide bounded corrective evidence. The resulting candidate pool is further refined by a discriminative Qwen3 reranker and two complementary semantic verification modules based on anomaly-aware cloze completion and multi-agent evidence reasoning. Finally, an uncertainty-gated consensus module adaptively reweights the three experts on ambiguous queries. Experiments on the PAB benchmark show that the proposed soft claim-aware retrieval achieves 86.44% mAP@10, substantially outperforming individual retrieval backbones. The complete framework further improves performance to 95.41% mAP@10, 94.44% R@1, and 99.09% R@5. These results demonstrate that preserving strong global retrieval while restricting expensive semantic reasoning to a small candidate pool is effective for fine-grained Sim2Real person anomaly search. Our code will be available on Github.
Hieu Dinh Trung Pham, Phuong Huu Vu Tran, Thuan Duc Mai +6
PAMI Lab, Vietnamese-German University, Vietnam · Ho Chi Minh City University of Technology, Vietnam · Ho Chi Minh city University of Science, Vietnam +2