Memory is fundamental to social interaction, enabling humans to recall meaningful past experiences and adapt their behavior accordingly based on the context. However, most current social robots and embodied agents rely on non-selective, text-based memory, limiting their ability to support personalized, context-aware interactions. Drawing inspiration from cognitive neuroscience, we propose a context-selective, multimodal memory architecture for social robots that captures and retrieves both textual and visual episodic traces, prioritizing moments characterized by high emotional salience or scene novelty. By associating these memories with individual users, our system enables socially personalized recall and more natural, grounded dialogue. We evaluate the selective storage mechanism using a curated dataset of social scenarios, achieving a Spearman correlation of 0.506, surpassing human consistency (ρ=0.415) and outperforming existing image memorability models. In multimodal retrieval experiments, our fusion approach improves Recall@1 by up to 13% over unimodal text or image retrieval. Runtime evaluations confirm that the system maintains real-time performance. Qualitative analyses further demonstrate that the proposed framework produces richer and more socially relevant responses than baseline models. This work advances memory design for social robots by bridging human-inspired selectivity and multimodal retrieval to enhance long-term, personalized human-robot interaction.
Figures & tables
Figure 1. Overview of the SUMMER architecture for selective multimodal memory in social robots. The perception layer analyzes the user’s scene and face to estimate memorability; memorable scenes are stored along with generated scene descriptions. The interaction layer manages and retrieves contextually significant multimodal memories, enabling socially relevant and personalized responses. SUMMER architecture
Figure 2. Overview of the user identification process Overview of the user identification process
Category
Approach
Mean ρ
Std ρ
pFisher
Human
Consistency
0.415
—
—
Heuristic
Random Selection
-0.000
0.269
ns
Interval (n=5)
-0.052
0.226
ns
Interval (n=10)
0.013
0.230
ns
Image Mem.
ResMem ( Needell and Bainbridge, 2022 )
-0.055
0.249
ns
ViTMem ( Hagen and Espeseth, 2023 )
0.046
0.235
ns
Table 1. Spearman correlation ( ρ ), standard deviation, and significance level ( pFisher ) for human consistency, baseline models, and SUMMER variants. For SUMMER, the triplet denotes (wemotion,wnovelty,wcomplexity) in the memorability score. Significance: ** p<10−100 , * p<10−10 , ns = not significant. Bold indicates the best-performing configuration.
Image Encoder
Text Encoder
Mode
Flickr8k
Flickr30k
MS COCO
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
CLIP ViT-L/14 ( Radford et al., 2021 )
mE5-large ( Wang et al., 2024 )
Text
49.2
76.8
85.6
54.6
80.2
86.6
28.8
52.2
63.3
Image
61.6
86.3
92.8
64.7
87.1
92.1
36.5
61.1
71.1
Fusion (0.7)
68.4
90.5
95.4
72.2
90.8
95.2
42.1
66.9
76.5
mGTE-base ( Zhang et al., 2024 )
Text
51.9
78.8
87.2
55.6
79.8
86.1
30.9
54.7
65.2
Image
61.6
86.3
92.8
64.7
87.1
92.1
36.5
61.1
71.1
Table 2. Recall@K (%) for Image , Text , and Fusion (inline best α ) across datasets. Bold indicates the best per dataset.
Figure 3. Qualitative comparison of response generated by the baseline model (left) and the SUMMER-augmented system (right) across diverse user queries. Each sample shows the query, both responses, and the retrieved image (for SUMMER), along with the retrieved cue type. Qualitative results
System
Module
Time (s)
Total Time (s)
Baseline
Generation
0.43±0.21
0.44±0.00
SUMMER
Retrieval
0.03±0.00
Generation
0.69±0.16
0.86±0.16
Table 3. Runtime of modules in the interaction layer. ‘Generation’ refers to response generation, ‘Retrieval’ to multimodal retrieval, and ‘Total Time’ includes all modules in the interaction layer.
Module
Time (s)
Total Time (s)
Memorability Decision
0.08±0.01
Scene Description Generator
0.25±0.03
Multimodal Encoding
0.05±0.00
0.37±0.03
Table 4. Runtime of modules in the perception layer. ‘Total Time’ indicates the cumulative runtime for the entire perception pipeline.
Multimodal memory lets AI systems retain, organize, and retrieve information across text, images, audio, and video for long-term reasoning and personalization.