Memory is fundamental to social interaction, enabling humans to recall meaningful past experiences and adapt their behavior accordingly based on the context. However, most current social robots and embodied agents rely on non-selective, text-based memory, limiting their ability to support personalized, context-aware interactions. Drawing inspiration from cognitive neuroscience, we propose a context-selective, multimodal memory architecture for social robots that captures and retrieves both textual and visual episodic traces, prioritizing moments characterized by high emotional salience or scene novelty. By associating these memories with individual users, our system enables socially personalized recall and more natural, grounded dialogue. We evaluate the selective storage mechanism using a curated dataset of social scenarios, achieving a Spearman correlation of 0.506, surpassing human consistency (ρ=0.415) and outperforming existing image memorability models. In multimodal retrieval experiments, our fusion approach improves Recall@1 by up to 13% over unimodal text or image retrieval. Runtime evaluations confirm that the system maintains real-time performance. Qualitative analyses further demonstrate that the proposed framework produces richer and more socially relevant responses than baseline models. This work advances memory design for social robots by bridging human-inspired selectivity and multimodal retrieval to enhance long-term, personalized human-robot interaction.
Figures & tables
Figure 1. Overview of the SUMMER architecture for selective multimodal memory in social robots. The perception layer analyzes the user’s scene and face to estimate memorability; memorable scenes are stored along with generated scene descriptions. The interaction layer manages and retrieves contextually significant multimodal memories, enabling socially relevant and personalized responses. SUMMER architecture
Figure 2. Overview of the user identification process Overview of the user identification process
Category
Approach
Mean ρ
Std ρ
pFisher
Human
Consistency
0.415
—
—
Heuristic
Random Selection
-0.000
0.269
ns
Interval (n=5)
-0.052
0.226
ns
Interval (n=10)
0.013
0.230
ns
Image Mem.
ResMem ( Needell and Bainbridge, 2022 )
-0.055
0.249
ns
ViTMem ( Hagen and Espeseth, 2023 )
0.046
0.235
ns
Table 1. Spearman correlation ( ρ ), standard deviation, and significance level ( pFisher ) for human consistency, baseline models, and SUMMER variants. For SUMMER, the triplet denotes (wemotion,wnovelty,wcomplexity) in the memorability score. Significance: ** p<10−100 , * p<10−10 , ns = not significant. Bold indicates the best-performing configuration.
Image Encoder
Text Encoder
Mode
Flickr8k
Flickr30k
MS COCO
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
CLIP ViT-L/14 ( Radford et al., 2021 )
mE5-large ( Wang et al., 2024 )
Text
49.2
76.8
85.6
54.6
80.2
86.6
28.8
52.2
63.3
Image
61.6
86.3
92.8
64.7
87.1
92.1
36.5
61.1
71.1
Fusion (0.7)
68.4
90.5
95.4
72.2
90.8
95.2
42.1
66.9
76.5
mGTE-base ( Zhang et al., 2024 )
Text
51.9
78.8
87.2
55.6
79.8
86.1
30.9
54.7
65.2
Image
61.6
86.3
92.8
64.7
87.1
92.1
36.5
61.1
71.1
Table 2. Recall@K (%) for Image , Text , and Fusion (inline best α ) across datasets. Bold indicates the best per dataset.
Figure 3. Qualitative comparison of response generated by the baseline model (left) and the SUMMER-augmented system (right) across diverse user queries. Each sample shows the query, both responses, and the retrieved image (for SUMMER), along with the retrieved cue type. Qualitative results
System
Module
Time (s)
Total Time (s)
Baseline
Generation
0.43±0.21
0.44±0.00
SUMMER
Retrieval
0.03±0.00
Generation
0.69±0.16
0.86±0.16
Table 3. Runtime of modules in the interaction layer. ‘Generation’ refers to response generation, ‘Retrieval’ to multimodal retrieval, and ‘Total Time’ includes all modules in the interaction layer.
Module
Time (s)
Total Time (s)
Memorability Decision
0.08±0.01
Scene Description Generator
0.25±0.03
Multimodal Encoding
0.05±0.00
0.37±0.03
Table 4. Runtime of modules in the perception layer. ‘Total Time’ indicates the cumulative runtime for the entire perception pipeline.
Multimodal memory lets AI systems retain, organize, and retrieve information across text, images, audio, and video for long-term reasoning and personalization.
Social robots that rely on large language models for conversation are unable to retain information across sessions. This absence of memory violates social expectations, potentially preventing the formation of persistent relationships. This paper presents a lightweight episodic memory module that integrates vector-based semantic retrieval with an LLM-controlled dialog system, deployed on the humanoid robot head Kim. The module employs a hybrid scoring function combining cosine similarity with a memory strength metric to retrieve contextually relevant past interactions and inject them into the generation prompt. The system was evaluated in a within-subjects video-based online study (N = 43) using the Human-Robot Interaction Evaluation Scale (HRIES). Results show that episodic memory significantly increased perceived sociability (d = 0.60, p < .001), with the strongest effects on perceived trustworthiness (d = 0.62) and warmth (d = 0.56). Perceived disturbance remained unchanged (d = 0.00), indicating that the implemented approach to personalized recall did not trigger privacy-related discomfort or uncanny valley effects. These findings suggest that episodic memory serves as a social lubricant in embodied Human-Robot Interaction, enhancing relational quality without eliciting negative affective responses.
Steve Aschenbrenner, Marcel Heisler, Thomas Sievers +1
Stuttgart Media University, Germany · University of Lübeck
Conventionally, memory in end-to-end robotic learning involves inputting a sequence of past observations into the learned policy. However, in complex multi-stage real-world tasks, the robot's memory must represent past events at multiple levels of granularity: from long-term memory that captures abstracted semantic concepts (e.g., a robot cooking dinner should remember which stages of the recipe are already done) to short-term memory that captures recent events and compensates for occlusions (e.g., a robot remembering the object it wants to pick up once its arm occludes it). In this work, our main insight is that an effective memory architecture for long-horizon robotic control should combine multiple modalities to capture these different levels of abstraction. We introduce Multi-Scale Embodied Memory (MEM), an approach for mixed-modal long-horizon memory in robot policies. MEM combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory. Together, they enable robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich. Additionally, we find that memory enables MEM policies to intelligently adapt manipulation strategies in-context.
Marcel Torne, Karl Pertsch, Homer Walke +14
Physical Intelligence · Stanford University · UC Berkeley +1
Multimodal large language model (MLLM)-based embodied agents have shown strong potential for solving complex tasks in physical environments. However, personalized assistance requires more than following generic instruction or recognizing object categories. In real-world scenarios, the intended target is often specified only implicitly through prior interactions, requiring agents to leverage personalized context accumulated over time. In this work, we propose POLAR, a multiomodal memory-augmented framework for personalized embodied agents over long-term user interactions. POLAR organizes prior interactions into a multimodal knowledge graph that captures semantic memory for personalized context and visual concepts, and episodic memory for embodied experiences such as agent trajectories. To execute embodied tasks, POLAR retrieves relevant memories to interpret the current request and guide task execution. We evaluate POLAR across multiple MLLM backbones and diverse evaluation scenarios to study the role of memory in long-term personalization. Results show that the proposed memory mechanism consistently improves performance by enabling more effective use of information accumulated over prior interactions. The gains are especially pronounced when the agents are required to reason across multiple interactions, perform multi-hop inference, or tracking updates in user-specific context over time.