Service robots in domestic and office environments must identify users to provide personalized services based on their preferences and previous interactions. This is challenging because only a few observations may be available for learning, while observation conditions change owing to appearance, environmental noise, and utterance content. To address this challenge, this study proposes a multimodal person concept model that learns person concepts from a small number of observations, without person labels, and enables robust identification under changing conditions. The model employs a nonparametric Bayesian generative model based on the Chinese restaurant process and integrates visual, acoustic, and linguistic features extracted using Contrastive Language-Image Pre-training (CLIP) and Contrastive Language-Audio Pretraining (CLAP), followed by principal component analysis (PCA). We evaluated six participants under eight conditions in which appearance changes, additive noise, and transcription errors were introduced individually and in combination. Averaged over the eight conditions, the proposed model achieved an identification accuracy of 0.89 while inferring the number of person concepts from the observations, which is comparable to k-means clustering that was supplied with the true number of participants (0.88). We also examined PCA dimensionality, modality availability, and variation strength, and demonstrated the model on a Toyota Human Support Robot. These results show that multimodal person concept learning supports robust person identification under changing conditions, and that identification can succeed even when the number of concepts is not correctly recovered. The project page and demonstration video are available at https://okitamisan01.github.io/person_concept/.
Figures & tables
Fig. 1: Overview of the proposed method. Only the first author is shown.
Fig. 2: Graphical model of the proposed model.
Symbol
Definition
d,D
clip index, number of clips
p,P
concept index, number of concepts
z
concept assigned to a clip (latent)
γ
concentration parameter of the CRP
oc
observed feature of each modality
μc
mean of a concept in each modality
Table 1: Definition of symbols used in the graphical model ( c∈{v,l,a} denotes the visual, linguistic, and acoustic modalities).
Item
Value
Detail
Participants
6
four male, two female, in their twenties
Recording conditions
3
reference (training), similar, varied
Recording length
180 sec
per participant and per condition
Clips per recording
18
10 s each
Clips per condition
108
6 participants ×18 clips
Table 2: Experimental parameters.
Table 3: Multimodal information and the datasets used in the experiment. The recordings are in Japanese; the linguistic row shows an English translation of each transcript, and the features are computed from the original Japanese text.
Condition
Visual
Acoustic
Linguistic
Reference (training)
ref.
ref.
ref.
Stable
sim.
sim.
sim.
Visual
var.
sim.
sim.
Acoustic
sim.
var.
sim.
Linguistic
sim.
sim.
var.
Visual + acoustic
var.
var.
sim.
Table 4: Relation between the conditions and the datasets.
Condition
Proposed method
k -means
Qwen2.5-VL (3B-Instruct)
CRP ( P inferred) ↑
P=7 given ↑
P=6 given ↑
P=6 given ↑
image ↑
text ↑
image + text ↑
Stable
0.98(0.05)
0.91(0.09)
0.83(0.09)
1.00(0.00)
0.36
0.26
0.27
Visual variation
0.85(0.02)
0.69(0.03)
0.69(0.08)
0.94(0.00)
0.22
0.26
0.28
Acoustic variation
0.97(0.06)
0.70(0.07)
0.69(0.16)
0.85(0.00)
0.36
0.26
0.27
Linguistic variation
0.98(0.05)
0.90(0.09)
0.83(0.10)
1.00(0.00)
0.36
0.27
0.28
Visual + acoustic variation
0.75(0.02)
0.30(0.00)
0.40(0.18)
0.75(0.00)
0.22
0.26
0.28
Table 5: Identification accuracy for each condition, mean (standard deviation) over ten trials. The CRP model is not told how many people there are: it inferred P=6 in six trials and P=7 in four trials. All the other columns marked “given” are supplied with the number of people; the k -means baseline is deterministic across the trials. The three right-hand columns give the accuracy of Qwen2.5-VL from the image only, the text only and both; it uses greedy decoding, so its results do not vary over the trials. Higher is better ( ↑ ); in each row the best value is shown in bold and underlined, and the second best in bold.
Fig. 3: Number of concepts P over the iterations in ten trials. Iteration 0 is the initial state in which every clip forms its own concept ( P=D=108 ). The thick line represents the mean over the ten trials.
PCA dimensions
Trials with P=6↑
Training ARI ↑
P chg. / 199
All var. ↑
3
4/10
0.963
9.4
0.75
4
6/10
0.961
11.9
0.76
5
6/10
0.978
13.8
0.68
6
4/10
0.974
14.3
0.66
512 (no compression)
0/10
0.301
0.0
0.68
Table 6: Estimated number of concepts and identification performance for different PCA dimensions (ten trials). “Training ARI” is the adjusted Rand index (ARI) between the inferred partition and the participant labels on the training data. “ P chg.” is the number of the 199 comparisons between consecutive sweeps in which the number of concepts differed; the change from the singleton initialisation to the first sweep is excluded, so a value of 0.0 means that the chain did not move after the first sweep. It is a diagnostic of the sampler and has no preferred direction, whereas ↑ marks the columns for which a larger value is better; the best value in each such column is shown in bold and underlined. The four-dimensional setting is the one used in Section 4 .
Condition
Modalities used for identification ↑
Visual
Acoustic
Linguistic
Visual + Acoustic
Visual + Linguistic
Acoustic + Linguistic
Visual + Acoustic + Linguistic
Stable
0.98
0.89
0.18
0.98
0.98
0.89
0.98
Visual var.
0.79
0.89
0.18
0.86
0.79
0.89
0.85
Acoustic var.
0.98
0.53
0.18
0.97
0.98
0.59
0.97
Linguistic var.
0.98
0.89
0.17
0.98
0.98
0.89
0.98
All var.
0.77
0.53
0.18
0.73
0.77
0.58
0.76
Table 7: Identification accuracy when the available modalities are restricted (mean over ten trials). Five representative conditions are shown.
Modalities
P=6 trials ↑
Training ARI ↑
Visual
3/10
0.87
Acoustic
7/10
0.89
Linguistic
5/10
−0.01
Visual + Acoustic
10/10
1.00
Visual + Linguistic
4/10
0.80
Acoustic + Linguistic
0/10
0.23
Table 8: Effect of the modalities used for concept formation. ARI is computed on the training data. The P=6 column reports the number of trials in which the inferred number of concepts matched the six participants.
Variation
Strength
Varied modality alone ↑
All three modalities ↑
Acoustic variation (SNR)
10 dB
0.53(0.06)
0.97(0.06)
5 dB
0.43(0.06)
0.98(0.05)
0 dB
0.39(0.05)
0.98(0.05)
−5 dB
0.34(0.00)
0.98(0.05)
Linguistic variation (word corruption)
0.3
0.17(0.02)
0.98(0.05)
0.5
0.19(0.01)
0.98(0.05)
Table 9: Identification accuracy under different strengths of acoustic and linguistic variation, shown as mean (standard deviation) over ten trials. The strength is given as the SNR for the acoustic variation and as the synthetic word corruption probability, that is, the probability with which each word of the transcript is deleted or replaced, for the linguistic variation. “Varied modality alone” uses only the modality that is degraded, that is, the acoustic modality in the upper block and the linguistic modality in the lower block. The chance level for six-way classification is 0.17 . The best value in each row is shown in bold and underlined.
Acoustic condition
Relative CER
All three modalities ↑
Clean
0.00
0.98(0.05)
SNR =10 dB
0.15
0.98(0.05)
SNR =5 dB
0.20
0.98(0.05)
SNR =0 dB
0.30
0.98(0.05)
SNR =−5 dB
0.53
0.98(0.05)
Table 10: End-to-end degradation of the linguistic observation. The noisy waveforms are transcribed again using Whisper large-v3, and the resulting text is used as the linguistic observation; the visual and acoustic observations are those of the corresponding acoustic condition. Degradation is measured by the relative character error rate (CER) against the transcript that the same procedure produces from the clean waveform, so the clean condition is 0.00 by definition; the values are means over the 108 clips. Identification accuracy is computed using all three modalities and is shown as mean (standard deviation) over the ten models learned in Section 4 .
Fig. 4: Proof-of-concept demonstration on the HSR. Demo 1 (top) uses only a visual observation from the HSR camera, and Demo 2 (bottom) only an acoustic observation. In each case, the observation is passed to the person concept model, the concept that maximises the log-likelihood of the available modality is selected, and the linguistic information stored with the selected concept is recalled to generate the robot response. Both modalities accessed the same concept (Concept 1, highlighted), from which the stored utterance “I like Japanese food.” was retrieved. The person shown in the figure is the first author.
Department of Computer Science, Drexel University · School of Electrical Engineering and Robotics, Queensland University of Technology · Department of Electrical and Computer Engineerin, Drexel University