IDSelect: A RL-Based Cost-Aware Selection Agent for Video-based Multi-Modal Person Recognition
Authors: Yuyang Ji, Yixuan Shen, Kien Nguyen, Lifeng Zhou, Feng Liu
Organizations: Department of Computer Science, Drexel University · School of Electrical Engineering and Robotics, Queensland University of Technology · Department of Electrical and Computer Engineerin, Drexel University
Video-based person recognition achieves robust identification by integrating face, body, and gait. However, current systems waste computational resources by processing all modalities with fixed heavyweight ensembles regardless of input complexity. To address these limitations, we propose IDSelect, a reinforcement learning-based cost-aware selector that chooses one pre-trained model per modality per-sequence to optimize the accuracy-efficiency trade-off. Our key insight is that an input-conditioned selector can discover complementary model choices that surpass fixed ensembles while using substantially fewer resources. IDSelect trains a lightweight agent end-to-end using actor-critic reinforcement learning with budget-aware optimization. The reward balances recognition accuracy with computational cost, while entropy regularization prevents premature convergence. At inference, the policy selects the most probable model per modality and fuses modality-specific similarities for the final score. Extensive experiments on challenging video-based datasets demonstrate IDSelect's superior efficiency: on CCVID, it achieves 95.9% Rank-1 accuracy with 92.4% less computation than strong baselines while improving accuracy by 1.8%; on MEVID, it reduces computation by 41.3% while maintaining competitive performance.
Figures & tables
Figure 1: Top: Prior multi-modal person recognition methods ( e.g. , QME [ 51 ] ) use fixed model combinations for all inputs, while our IDSelect employs an RL-based cost-aware agent to adaptively select complementary models from diverse pools based on input characteristics. Bottom: Accuracy vs. computational cost on CCVID dataset [ 11 ] . Our method achieves superior accuracy (95.9%, +1.8% ) with 92.4% fewer FLOPs than QME.
Figure 2: IDSelect framework architecture. The selection agent processes input video pairs through a feature encoder and attention pooling to generate modality-specific selection distributions. An actor-critic reinforcement learning policy optimizes model selection from pre-trained pools under a budget constraint. The framework minimizes a multi-objective loss combining classification reward, computational cost, and selection diversity to discover optimal model combinations for multi-modal fusion.
Table 1: Performance comparison on CCVID Configuration 1 . IDSelect achieves superior accuracy-efficiency trade-offs compared to single modality baselines, fixed fusion strategies, and quality-guided methods. ’Ada.’ denotes input-adaptive selection. Colored icons in “Comb.” represent modality and model capacity ( ∘ =face, □ =body, △ =gait; fill level increases with model size, from light to heavy). Models marked with an asterisk * are not finetuned on the CCVID dataset, while all other models are finetuned. Heavy/Light denotes the highest/lowest-cost model.
Table 2: Performance comparison on CCVID Configuration 2 . IDSelect demonstrates adaptability across model pool configurations, maintaining strong performance while achieving significant computational savings compared to exhaustive fusion.
Table 3: Performance comparison on MEVID dataset with face-body fusion configurations. Results highlight the benefits of cross-modal complementarity while revealing diminishing returns from intra-modal redundancy among body models.
Figure 3: (a) Selection frequency distribution for CCVID configurations. Config. 1 shows concentrated patterns with IDSelect predominantly selecting adaface_101 + gaitset + ap3d_34 (71.3%) and adaface_101 + gaitbase + ap3d_34 (24.2%). Config. 2 exhibits more diverse selection patterns, demonstrating adaptability to different model pool characteristics. (b) Selection probability evolution during training (Config. 2). Body converges to TF-CLIP; Gait learns GaitSet/GaitBase preference; Face maintains input-adaptive diversity.
Method
GFLOPs ↓
Rank1 ↑
mAP ↑
Face + Gait
46.1
95.5
91.2
Face + Body
31.4
94.2
90.3
Gait + Body
29.9
87.5
85.1
Face + Gait + Body
53.7
95.9
94.6
Table 4: Modality combination analysis on CCVID Configuration 1 . Results demonstrate the effectiveness of cross-modal complementarity, with the full three-modality approach achieving optimal performance while two-modality combinations provide competitive accuracy at reduced computational cost.