Service robots in domestic and office environments must identify users to provide personalized services based on their preferences and previous interactions. This is challenging because only a few observations may be available for learning, while observation conditions change owing to appearance, environmental noise, and utterance content. To address this challenge, this study proposes a multimodal person concept model that learns person concepts from a small number of observations, without person labels, and enables robust identification under changing conditions. The model employs a nonparametric Bayesian generative model based on the Chinese restaurant process and integrates visual, acoustic, and linguistic features extracted using Contrastive Language-Image Pre-training (CLIP) and Contrastive Language-Audio Pretraining (CLAP), followed by principal component analysis (PCA). We evaluated six participants under eight conditions in which appearance changes, additive noise, and transcription errors were introduced individually and in combination. Averaged over the eight conditions, the proposed model achieved an identification accuracy of 0.89 while inferring the number of person concepts from the observations, which is comparable to k-means clustering that was supplied with the true number of participants (0.88). We also examined PCA dimensionality, modality availability, and variation strength, and demonstrated the model on a Toyota Human Support Robot. These results show that multimodal person concept learning supports robust person identification under changing conditions, and that identification can succeed even when the number of concepts is not correctly recovered. The project page and demonstration video are available at https://okitamisan01.github.io/person_concept/.
Figures & tables
Fig. 1: Overview of the proposed method. Only the first author is shown.
Fig. 2: Graphical model of the proposed model.
Symbol
Definition
d,D
clip index, number of clips
p,P
concept index, number of concepts
z
concept assigned to a clip (latent)
γ
concentration parameter of the CRP
oc
observed feature of each modality
μc
mean of a concept in each modality
Table 1: Definition of symbols used in the graphical model ( c∈{v,l,a} denotes the visual, linguistic, and acoustic modalities).
Item
Value
Detail
Participants
6
four male, two female, in their twenties
Recording conditions
3
reference (training), similar, varied
Recording length
180 sec
per participant and per condition
Clips per recording
18
10 s each
Clips per condition
108
6 participants ×18 clips
Table 2: Experimental parameters.
Table 3: Multimodal information and the datasets used in the experiment. The recordings are in Japanese; the linguistic row shows an English translation of each transcript, and the features are computed from the original Japanese text.
Condition
Visual
Acoustic
Linguistic
Reference (training)
ref.
ref.
ref.
Stable
sim.
sim.
sim.
Visual
var.
sim.
sim.
Acoustic
sim.
var.
sim.
Linguistic
sim.
sim.
var.
Visual + acoustic
var.
var.
sim.
Table 4: Relation between the conditions and the datasets.
Condition
Proposed method
k -means
Qwen2.5-VL (3B-Instruct)
CRP ( P inferred) ↑
P=7 given ↑
P=6 given ↑
P=6 given ↑
image ↑
text ↑
image + text ↑
Stable
0.98(0.05)
0.91(0.09)
0.83(0.09)
1.00(0.00)
0.36
0.26
0.27
Visual variation
0.85(0.02)
0.69(0.03)
0.69(0.08)
0.94(0.00)
0.22
0.26
0.28
Acoustic variation
0.97(0.06)
0.70(0.07)
0.69(0.16)
0.85(0.00)
0.36
0.26
0.27
Linguistic variation
0.98(0.05)
0.90(0.09)
0.83(0.10)
1.00(0.00)
0.36
0.27
0.28
Visual + acoustic variation
0.75(0.02)
0.30(0.00)
0.40(0.18)
0.75(0.00)
0.22
0.26
0.28
Table 5: Identification accuracy for each condition, mean (standard deviation) over ten trials. The CRP model is not told how many people there are: it inferred P=6 in six trials and P=7 in four trials. All the other columns marked “given” are supplied with the number of people; the k -means baseline is deterministic across the trials. The three right-hand columns give the accuracy of Qwen2.5-VL from the image only, the text only and both; it uses greedy decoding, so its results do not vary over the trials. Higher is better ( ↑ ); in each row the best value is shown in bold and underlined, and the second best in bold.
Fig. 3: Number of concepts P over the iterations in ten trials. Iteration 0 is the initial state in which every clip forms its own concept ( P=D=108 ). The thick line represents the mean over the ten trials.
PCA dimensions
Trials with P=6↑
Training ARI ↑
P chg. / 199
All var. ↑
3
4/10
0.963
9.4
0.75
4
6/10
0.961
11.9
0.76
5
6/10
0.978
13.8
0.68
6
4/10
0.974
14.3
0.66
512 (no compression)
0/10
0.301
0.0
0.68
Table 6: Estimated number of concepts and identification performance for different PCA dimensions (ten trials). “Training ARI” is the adjusted Rand index (ARI) between the inferred partition and the participant labels on the training data. “ P chg.” is the number of the 199 comparisons between consecutive sweeps in which the number of concepts differed; the change from the singleton initialisation to the first sweep is excluded, so a value of 0.0 means that the chain did not move after the first sweep. It is a diagnostic of the sampler and has no preferred direction, whereas ↑ marks the columns for which a larger value is better; the best value in each such column is shown in bold and underlined. The four-dimensional setting is the one used in Section 4 .
Condition
Modalities used for identification ↑
Visual
Acoustic
Linguistic
Visual + Acoustic
Visual + Linguistic
Acoustic + Linguistic
Visual + Acoustic + Linguistic
Stable
0.98
0.89
0.18
0.98
0.98
0.89
0.98
Visual var.
0.79
0.89
0.18
0.86
0.79
0.89
0.85
Acoustic var.
0.98
0.53
0.18
0.97
0.98
0.59
0.97
Linguistic var.
0.98
0.89
0.17
0.98
0.98
0.89
0.98
All var.
0.77
0.53
0.18
0.73
0.77
0.58
0.76
Table 7: Identification accuracy when the available modalities are restricted (mean over ten trials). Five representative conditions are shown.
Modalities
P=6 trials ↑
Training ARI ↑
Visual
3/10
0.87
Acoustic
7/10
0.89
Linguistic
5/10
−0.01
Visual + Acoustic
10/10
1.00
Visual + Linguistic
4/10
0.80
Acoustic + Linguistic
0/10
0.23
Table 8: Effect of the modalities used for concept formation. ARI is computed on the training data. The P=6 column reports the number of trials in which the inferred number of concepts matched the six participants.
Variation
Strength
Varied modality alone ↑
All three modalities ↑
Acoustic variation (SNR)
10 dB
0.53(0.06)
0.97(0.06)
5 dB
0.43(0.06)
0.98(0.05)
0 dB
0.39(0.05)
0.98(0.05)
−5 dB
0.34(0.00)
0.98(0.05)
Linguistic variation (word corruption)
0.3
0.17(0.02)
0.98(0.05)
0.5
0.19(0.01)
0.98(0.05)
Table 9: Identification accuracy under different strengths of acoustic and linguistic variation, shown as mean (standard deviation) over ten trials. The strength is given as the SNR for the acoustic variation and as the synthetic word corruption probability, that is, the probability with which each word of the transcript is deleted or replaced, for the linguistic variation. “Varied modality alone” uses only the modality that is degraded, that is, the acoustic modality in the upper block and the linguistic modality in the lower block. The chance level for six-way classification is 0.17 . The best value in each row is shown in bold and underlined.
Acoustic condition
Relative CER
All three modalities ↑
Clean
0.00
0.98(0.05)
SNR =10 dB
0.15
0.98(0.05)
SNR =5 dB
0.20
0.98(0.05)
SNR =0 dB
0.30
0.98(0.05)
SNR =−5 dB
0.53
0.98(0.05)
Table 10: End-to-end degradation of the linguistic observation. The noisy waveforms are transcribed again using Whisper large-v3, and the resulting text is used as the linguistic observation; the visual and acoustic observations are those of the corresponding acoustic condition. Degradation is measured by the relative character error rate (CER) against the transcript that the same procedure produces from the clean waveform, so the clean condition is 0.00 by definition; the values are means over the 108 clips. Identification accuracy is computed using all three modalities and is shown as mean (standard deviation) over the ten models learned in Section 4 .
Fig. 4: Proof-of-concept demonstration on the HSR. Demo 1 (top) uses only a visual observation from the HSR camera, and Demo 2 (bottom) only an acoustic observation. In each case, the observation is passed to the person concept model, the concept that maximises the log-likelihood of the available modality is selected, and the linguistic information stored with the selected concept is recalled to generate the robot response. Both modalities accessed the same concept (Concept 1, highlighted), from which the stored utterance “I like Japanese food.” was retrieved. The person shown in the figure is the first author.
Video-based person recognition achieves robust identification by integrating face, body, and gait. However, current systems waste computational resources by processing all modalities with fixed heavyweight ensembles regardless of input complexity. To address these limitations, we propose IDSelect, a reinforcement learning-based cost-aware selector that chooses one pre-trained model per modality per-sequence to optimize the accuracy-efficiency trade-off. Our key insight is that an input-conditioned selector can discover complementary model choices that surpass fixed ensembles while using substantially fewer resources. IDSelect trains a lightweight agent end-to-end using actor-critic reinforcement learning with budget-aware optimization. The reward balances recognition accuracy with computational cost, while entropy regularization prevents premature convergence. At inference, the policy selects the most probable model per modality and fuses modality-specific similarities for the final score. Extensive experiments on challenging video-based datasets demonstrate IDSelect's superior efficiency: on CCVID, it achieves 95.9% Rank-1 accuracy with 92.4% less computation than strong baselines while improving accuracy by 1.8%; on MEVID, it reduces computation by 41.3% while maintaining competitive performance.
Yuyang Ji, Yixuan Shen, Kien Nguyen +2
Department of Computer Science, Drexel University · School of Electrical Engineering and Robotics, Queensland University of Technology · Department of Electrical and Computer Engineerin, Drexel University
Person re-identification (ReID) is essential for multi-camera surveillance and tracking, yet remains difficult due to viewpoint and illumination changes, occlusion, background clutter, and low resolution imagery. We propose a generative AI integrated multimodal ReID framework designed explicitly for robustness under missing cues and low latency deployment. The key idea is a cost aware early-exit cascade that prioritizes inexpensive, high confidence evidence and only triggers expensive modalities for ambiguous cases. Our system integrates (i) global visual embeddings from segmented person regions, (ii) automatically generated fine grained semantic attribute descriptions generated by vision-language models (VLMs), and (iii) optional facial embeddings when face observations are reliable. To optimize the balance between accuracy and latency, we use a cost aware early-exit cascade instead of fusing all modalities. Specifically, we first inspect the top-k retrieval results to determine whether the query is unambiguous. If the best match is clearly separated from the remaining candidates, we stop early and return the result to minimize latency; in ambiguous cases, we keep multiple hypotheses and invoke additional modalities (face/semantic) with adaptive reliability weighting to refine the decision. We report person re-identification performance using mAP and Rank-1 accuracy on the Market-1501 and DukeMTMC-reID benchmarks. The proposed adaptive early-exit cascade resolves 60.7% of DukeMTMC-reID queries and 68.4% of Market-1501 queries without invoking semantic reasoning, reducing computational overhead while maintaining competitive retrieval performance.
Leon Fernando, C Dombawala, P. Hettigoda +3
University of Moratuwa · Zone24x7 (Pvt) Ltd · Chulalongkorn University +2
Learning identity-discriminative representations with multi-scene generality has become a critical objective in person re-identification (ReID). However, mainstream perception-driven paradigms tend to identify fitting from massive annotated data rather than identity-causal cues understanding, which presents a fragile representation against multiple disruptions. In this work, ReID-R is proposed as a novel reasoning-driven paradigm that achieves explicit identity understanding and reasoning by incorporating chain-of-thought into the ReID pipeline. Specifically, ReID-R consists of a two-stage contribution: (i) Discriminative reasoning warm-up, where a model is trained in a CoT label-free manner to acquire identity-aware feature understanding; and (ii) Efficient reinforcement learning, which proposes a non-trivial sampling to construct scene-generalizable data. On this basis, ReID-R leverages high-quality reward signals to guide the model toward focusing on ID-related cues, achieving accurate reasoning and correct responses. Extensive experiments on multiple ReID benchmarks demonstrate that ReID-R achieves competitive identity discrimination as superior methods using only 14.3K non-trivial data (20.9% of the existing data scale). Furthermore, benefit from inherent reasoning, ReID-R can provide high-quality interpretation for results.