IDSelect: A RL-Based Cost-Aware Selection Agent for Video-based Multi-Modal Person Recognition
Authors: Yuyang Ji, Yixuan Shen, Kien Nguyen, Lifeng Zhou, Feng Liu
Organizations: Department of Computer Science, Drexel University · School of Electrical Engineering and Robotics, Queensland University of Technology · Department of Electrical and Computer Engineerin, Drexel University
Video-based person recognition achieves robust identification by integrating face, body, and gait. However, current systems waste computational resources by processing all modalities with fixed heavyweight ensembles regardless of input complexity. To address these limitations, we propose IDSelect, a reinforcement learning-based cost-aware selector that chooses one pre-trained model per modality per-sequence to optimize the accuracy-efficiency trade-off. Our key insight is that an input-conditioned selector can discover complementary model choices that surpass fixed ensembles while using substantially fewer resources. IDSelect trains a lightweight agent end-to-end using actor-critic reinforcement learning with budget-aware optimization. The reward balances recognition accuracy with computational cost, while entropy regularization prevents premature convergence. At inference, the policy selects the most probable model per modality and fuses modality-specific similarities for the final score. Extensive experiments on challenging video-based datasets demonstrate IDSelect's superior efficiency: on CCVID, it achieves 95.9% Rank-1 accuracy with 92.4% less computation than strong baselines while improving accuracy by 1.8%; on MEVID, it reduces computation by 41.3% while maintaining competitive performance.
Figures & tables
Figure 1: Top: Prior multi-modal person recognition methods ( e.g. , QME [ 51 ] ) use fixed model combinations for all inputs, while our IDSelect employs an RL-based cost-aware agent to adaptively select complementary models from diverse pools based on input characteristics. Bottom: Accuracy vs. computational cost on CCVID dataset [ 11 ] . Our method achieves superior accuracy (95.9%, +1.8% ) with 92.4% fewer FLOPs than QME.
Figure 2: IDSelect framework architecture. The selection agent processes input video pairs through a feature encoder and attention pooling to generate modality-specific selection distributions. An actor-critic reinforcement learning policy optimizes model selection from pre-trained pools under a budget constraint. The framework minimizes a multi-objective loss combining classification reward, computational cost, and selection diversity to discover optimal model combinations for multi-modal fusion.
Table 1: Performance comparison on CCVID Configuration 1 . IDSelect achieves superior accuracy-efficiency trade-offs compared to single modality baselines, fixed fusion strategies, and quality-guided methods. ’Ada.’ denotes input-adaptive selection. Colored icons in “Comb.” represent modality and model capacity ( ∘ =face, □ =body, △ =gait; fill level increases with model size, from light to heavy). Models marked with an asterisk * are not finetuned on the CCVID dataset, while all other models are finetuned. Heavy/Light denotes the highest/lowest-cost model.
Table 2: Performance comparison on CCVID Configuration 2 . IDSelect demonstrates adaptability across model pool configurations, maintaining strong performance while achieving significant computational savings compared to exhaustive fusion.
Table 3: Performance comparison on MEVID dataset with face-body fusion configurations. Results highlight the benefits of cross-modal complementarity while revealing diminishing returns from intra-modal redundancy among body models.
Figure 3: (a) Selection frequency distribution for CCVID configurations. Config. 1 shows concentrated patterns with IDSelect predominantly selecting adaface_101 + gaitset + ap3d_34 (71.3%) and adaface_101 + gaitbase + ap3d_34 (24.2%). Config. 2 exhibits more diverse selection patterns, demonstrating adaptability to different model pool characteristics. (b) Selection probability evolution during training (Config. 2). Body converges to TF-CLIP; Gait learns GaitSet/GaitBase preference; Face maintains input-adaptive diversity.
Method
GFLOPs ↓
Rank1 ↑
mAP ↑
Face + Gait
46.1
95.5
91.2
Face + Body
31.4
94.2
90.3
Gait + Body
29.9
87.5
85.1
Face + Gait + Body
53.7
95.9
94.6
Table 4: Modality combination analysis on CCVID Configuration 1 . Results demonstrate the effectiveness of cross-modal complementarity, with the full three-modality approach achieving optimal performance while two-modality combinations provide competitive accuracy at reduced computational cost.
Person re-identification (ReID) is essential for multi-camera surveillance and tracking, yet remains difficult due to viewpoint and illumination changes, occlusion, background clutter, and low resolution imagery. We propose a generative AI integrated multimodal ReID framework designed explicitly for robustness under missing cues and low latency deployment. The key idea is a cost aware early-exit cascade that prioritizes inexpensive, high confidence evidence and only triggers expensive modalities for ambiguous cases. Our system integrates (i) global visual embeddings from segmented person regions, (ii) automatically generated fine grained semantic attribute descriptions generated by vision-language models (VLMs), and (iii) optional facial embeddings when face observations are reliable. To optimize the balance between accuracy and latency, we use a cost aware early-exit cascade instead of fusing all modalities. Specifically, we first inspect the top-k retrieval results to determine whether the query is unambiguous. If the best match is clearly separated from the remaining candidates, we stop early and return the result to minimize latency; in ambiguous cases, we keep multiple hypotheses and invoke additional modalities (face/semantic) with adaptive reliability weighting to refine the decision. We report person re-identification performance using mAP and Rank-1 accuracy on the Market-1501 and DukeMTMC-reID benchmarks. The proposed adaptive early-exit cascade resolves 60.7% of DukeMTMC-reID queries and 68.4% of Market-1501 queries without invoking semantic reasoning, reducing computational overhead while maintaining competitive retrieval performance.
Leon Fernando, C Dombawala, P. Hettigoda +3
University of Moratuwa · Zone24x7 (Pvt) Ltd · Chulalongkorn University +2
Transformers dominate video recognition. They split videos into tokens, and processing them has expensive superlinear computational cost. Yet videos are filled with redundancy, so we can question the need for this expense. We introduce LookWhen, a selector-extractor framework that factorizes video recognition into learning when, where, and what to compute. Our shallow selector gets a scaled-down video and quickly scores all tokens across space-time, while our deep extractor gets the top-K selected tokens to approximate full-video representations without actually processing all the tokens. A key challenge is defining effective supervision for selection and extraction. For selection pre-training, we introduce a score on representations that ranks tokens by uniqueness using a simple nearest-neighbor distance. For extraction pre-training, we distill both a video teacher and an image teacher, for which we normalize its frame-wise representations to learn what changes within videos. Through these strategies, our selector-extractor learns general and efficient representations for feature extraction or fine-tuning to a task. Through experiments on Kinetics-400, SSv2, Epic-Kitchens, Diving48, Jester, and Charades, we show that LookWhen achieves a better accuracy-computation trade-off than efficient models and upgraded baselines of similar size. LookWhen Pareto-dominates in accuracy-FLOPs on 9 of 12 cases (6 tasks x 2 settings) and roughly matches on 3. In accuracy-throughput, measuring time in practice, LookWhen is more efficient still at 6.7x faster than InternVideo2-B at equal accuracy.
Ali Salamatian, Anthony Fuller, Pritam Sarkar +3
University of British Columbia · Carleton University · Vector Institute
RGB-Event Video Person Re-Identification (RE-VReID) aims to retrieve specific person across non-overlapping cameras with complementary RGB videos and event streams. However, existing methods often decouple spatial and temporal modeling, which limits their interaction. In addition, global-level RGB-Event fusion fails to fully exploit fine-grained discriminative cues. To address these issues, we propose Paths, a unified framework with spatio-temporal modeling and hierarchical multi-modal fusion for RE-VReID. Specifically, we first design a Memory-Augmented Backbone (MAB) to maintain modality-specific identity prototypes for stable intra-modal representation learning. Then, we propose a Prompt-aware Spatio-temporal Transformer (PST) to jointly model spatial and temporal cues within a unified Transformer. Finally, we introduce a Hierarchical Multi-modal Fusion (HMF) to integrate RGB and event features at global and local levels. With these modules, our framework can learn robust and discriminative representations for RE-VReID. Extensive experiments on three public RE-VReID benchmarks including EvReID, MARS and iLIDS-VID, demonstrate the effectiveness of our proposed method. The code is available at https://github.com/Reflection0427/Paths.