An Edge-Host-Cloud Architecture for Robot-Agnostic, Caregiver-in-the-Loop Personalized Cognitive Exercise: Multi-Site Deployment in Dementia Care

Wenzheng Zhao; Ruth Palan Lopez; Shu Fen Wung; Fengpei Yuan

cs.ROApr 1, 2026

An Edge-Host-Cloud Architecture for Robot-Agnostic, Caregiver-in-the-Loop Personalized Cognitive Exercise: Multi-Site Deployment in Dementia Care

Authors: Wenzheng Zhao, Ruth Palan Lopez, Shu Fen Wung, Fengpei Yuan

Organizations: Worcester Polytechnic Institute

Abstract

We present Speaking Memories, a distributed, stakeholder-in-the-loop robotic interaction platform for personalized cognitive exercise support. Rather than a single robot-centric system, Speaking Memories is designed as a generalizable robotics architecture that integrates caregiver-authored knowledge, local edge intelligence, and embodied robotic agents into a unified socio-technical loop. The platform fuses auditory, visual, and textual signals to enable emotion-aware, personalized dialogue, while decoupling multimodal perception and reasoning from robot-specific hardware through a local edge interaction server. This design achieves low-latency, privacy-preserving operation and supports scalable deployment across heterogeneous robotic embodiments. Caregivers and family members contribute structured biographical knowledge via a secure cloud portal, which conditions downstream dialogue policies and enables longitudinal personalization across interaction sessions. Beyond real-time interaction, the system incorporates an automated multimodal evaluation layer that continuously analyzes user responses, affective cues, and engagement patterns, producing structured interaction metrics at scale. These metrics support systematic assessment of interaction quality, enable data-driven model fine-tuning, and lay the foundation for future clinician- and caregiver-informed personalization and intervention planning. We evaluate the platform through real-world deployments, measuring end-to-end latency, dialogue coherence, interaction stability, and stakeholder-reported usability and engagement. Results demonstrate sub-6-second response latency, robust multimodal synchronization, and consistently positive feedback from both participants and caregivers. Furthermore, subsets of the dataset can be shared upon request, subject to participant consent and IRB constraints.

Figures & tables

Fig. 1: System architecture of the Speaking Memories framework, illustrating a layered host–edge–cloud design for emotion-aware reminiscence interaction. The Hardware Layer interfaces with the user through onboard sensors and actuators. The Edge Robot Layer performs real-time multimodal sensing, lightweight context aggregation, and robot-side interaction state management, enabling low-latency perception and feedback independent of robot embodiment. The Host Processing Layer handles multimodal context construction, emotion-aware dialogue reasoning, conversation policy execution, and interaction logging, serving as the core runtime intelligence of the system. The Cloud Coordination Layer supports caregiver-in-the-loop personalization and long-term operation by managing user profiles and memory databases, authentication and privacy control, and large-scale analytics and dataset aggregation. This separation of concerns decouples real-time interaction from long-term data management, enabling privacy-preserving deployment, cross-session personalization, and scalable operation across heterogeneous robotic platforms.

Fig. 2: Pipeline of the Speaking Memories platform, a distributed, stakeholder-in-the-loop architecture for adaptive reminiscence interaction. The system comprises three tightly coupled layers. (Left) A multimodal user input layer acquires visual, auditory, and textual cues from the participant during real-time interaction with an embodied robot agent. (Center) A local edge interaction server decouples multimodal perception and reasoning from robot-specific hardware, performing emotion-aware dialogue generation using vision–language and large language models conditioned on conversational context, affective cues, and structured caregiver-provided knowledge. (Right) A cloud–local data and analytics layer supports caregiver-in-the-loop personalization through a secure portal and automated multimodal evaluation, producing engagement metrics, longitudinal interaction summaries, and caregiver-oriented reports. This closed-loop design enables low-latency, privacy-preserving interaction, scalable deployment across heterogeneous robotic embodiments, and continuous data-driven personalization over repeated sessions.

Fig. 3: System software flow diagram of the Speaking Memories framework for adaptive reminiscence interaction. Colored boxes represent different system components, white boxes with borders denote data flow and information exchange. Italicized text without boxes indicates user actions, while non-italicized text without boxes represents Robot/User-executed actions.

Fig. 5: Flowchart of the experimental procedure.

Fig. 6: Real-world deployment of the Speaking Memories framework using two different hardware configurations at an elder care facility. (a) The left deployment used Robot I , relied soly on Pepper’s native hardware , using its onboard microphone, camera, and tablet for multimodal interaction. (b) The right deployment used Robot II , a Jetson-based edge client , where sensing and feedback were executed on an external Jetson Orin NX unit attached to the Pepper robot. Although both configurations delivered personalized reminiscence dialogues, the Jetson setup achieved substantially lower latency and greater modularity by offloading computation to the edge server.

$on\_topic(u)$ — On-topic relevance of the utterance: • 0: the utterance content is not semantically related to the current reminiscence prompt or displayed media; • 1: the utterance content is semantically related to the prompt or media, including reflective or affective responses.
$r\_depth(u)$ — Reminiscence depth reflecting autobiographical richness: • 0: no personal content (e.g., yes/no or filler responses); • 1: a single personal detail or value; • 2: multiple personal details or reflective elaboration; • 3: rich autobiographical narrative involving connected experiences.
$emo\_tone(u)$ — Emotional tone capturing dementia-relevant affective expression: • POSITIVE: clearly pleasant, appreciative, or joyful affect; • BITTERSWEET: mixed but meaningful affect, such as reminiscence-related sadness co-occurring with appreciation; this category captures non-harmful sadness common in dementia reminiscence; • NEGATIVE_DISTRESS: clear emotional distress, discomfort, or expressed desire to avoid the topic; • NEUTRAL/MIXED: primarily factual, ambiguous, or emotionally flat responses without a dominant tone.
$da(u)$ — Dialog act category describing the functional role of the utterance in interaction: • ANSWER: brief responses directly answering the robot’s question; • SELF_DISCLOSURE: autobiographical sharing beyond minimal yes/no responses; • QUESTION: utterances seeking information or clarification; • ACKNOWLEDGMENT/BACKCHANNEL: brief signals indicating attention or listening without adding new content; • PHATIC/SOCIAL: social remarks not directly related to reminiscence content; • OFF_TOPIC_COMMENT: content clearly unrelated to the prompt or displayed media; • REPAIR/MISUNDERSTANDING: utterances indicating hearing, comprehension, or interactional difficulties.

TABLE I: LLM-based semantic verbal engagement annotation scheme.

$gaze\_robot\_ratio(u)$ — Proportion of time the participant attends to the robot or tablet during the turn: • Computed as the fraction of sampled frames labeled with attention toward the robot or tablet interaction area. • Captures sustained visual or postural engagement rather than eye contact with the camera.
$valence\_mean(u)$ — Average facial valence during the turn: • Frame-level facial valence is labeled as negative ( $-1$ ), neutral ( $0$ ), or positive ( $+1$ ).
$valence\_pos\_ratio(u)$ — Proportion of positively valenced frames: • Measures the prevalence of positive facial expressions during the turn.

TABLE II: VLM-based nonverbal engagement annotation scheme (turn-level).

ID	Sex	Age Range	Group	Robot Setup	Duration (min)	Dialogue Rounds	Triggers Used
P04	F	75–84	Healthy	I	21	20	4
P06	F	75–84	PLWD	I	23	17	3
P07	M	75–84	PLWD	I	24	15	3
P08	F	85+	PLWD	I	17	15	2
P09	M	85+	PLWD	I	22	26	3
P10	F	85+	PLWD	I	5	4	1

TABLE III: Session Summary per Participant (N=12). Five additional older-adult sessions were conducted but excluded from analysis due to missing signed consent documentation required by the IRB-approved protocol; engineering/debug sessions are not included.

Metric	Pepper-native	Jetson-edge
End-to-end latency (s)	$9.25\pm 2.81$	$\textbf{5.89}\pm\textbf{1.11}$
Interaction completion rate	84.82%	100%
Dialogue coherence rate	88.30%	92.28%
Session-level statistics
Dialogue turns per session	$16.81\pm 9.27$
Session duration (min)	$17.94\pm 7.15$

TABLE IV: Summary of Quantitative System Evaluation Results

Component	Pepper-native (s)	Jetson-edge (s)
Speech recognition (ASR)	1.18	0.97
Multimodal reasoning	4.63	4.48
System orchestration	3.44	0.44
Total	9.25	5.89

TABLE V: Component-Level Latency Breakdown Across System Configurations

Dimension	Items	Mean $\pm$ SD
Attractiveness	Q1–Q2	4.7 $\pm$ 0.5
Interaction quality	Q3–Q5	4.5 $\pm$ 0.7
Usability	Q6-Q7	4.0 $\pm$ 0.9

TABLE VI: Participant (dementia) evaluation scores (Likert 1–5) aggregated by interaction dimension. Each dimension groups related survey items (Q1–Q7) corresponding to perceived attractiveness, interaction quality, and usability of the robot system. Values are reported as mean

\pm

standard deviation across all sessions.

Dimension	Items	Mean $\pm$ SD
Interface usability	Q1–Q2	4.6 $\pm$ 0.7
Emotional/positive engagement	Q3, Q8	3.7 $\pm$ 1.5
Linguistic/dialogue naturalness	Q4–Q6	3.5 $\pm$ 1.2
System stability	Q7	3.7 $\pm$ 1.0
Overall recommendation	Q9	4.5 $\pm$ 0.6

TABLE VII: Caregiver ratings by dimension (Likert 1–5). Each dimension groups related survey items (Q1–Q9) corresponding to interface usability, emotional engagement, dialogue naturalness, system stability, and overall recommendation.

Turn	$t_{\text{user}}$ (s)	$on\_topic$	$r\_d$	$emo$	$da$	$gaze\_r$	$val+$
1	10	1	1	POS	DIS	1	0.667
2	8	1	1	BIT	DIS	1	0.875
3	30	1	2	BIT	DIS	0.967	0.9
4	39	1	2	POS	DIS	1	0.842
5	54	1	2	POS	DIS	1	0.226
6	41	1	2	BIT	DIS	1	0.25

TABLE VIII: Illustrative turn-level multimodal engagement for Participant P08.

ID	Robot Setup	On-topic	$r_{d}$	SDis	Gaze	Val-mean	Val+
P06	I	0.92	1.58	0.92	0.94	$-$ 0.23	0.00
P07	I	0.83	1.33	0.83	0.45	$-$ 0.47	0.00
P08	I	0.93	1.57	0.93	1.00	0.30	0.50
P09	I	0.64	0.80	0.52	0.78	$-$ 0.12	0.00
P10	I	0.00	0.00	0.00	1.00	0.00	0.00
P11	I	0.50	0.61	0.44	1.00	0.64	0.61

TABLE IX: Participant-level multimodal engagement statistics aggregated across dialog turns.

Metric	Robot Setup I	Robot Setup II
On-topic rate	0.64 $\pm$ 0.35	0.55 $\pm$ 0.16
Mean reminiscence depth	0.98 $\pm$ 0.63	0.60 $\pm$ 0.25
Self-disclosure ratio	0.61 $\pm$ 0.36	0.37 $\pm$ 0.12
Gaze-to-robot rate	0.86 $\pm$ 0.22	0.98 $\pm$ 0.04
Valence mean	0.02 $\pm$ 0.39	0.25 $\pm$ 0.41
Positive valence ratio	0.18 $\pm$ 0.29	0.26 $\pm$ 0.37

TABLE X: Comparison of multimodal engagement metrics across robot setups.

Explore similar work

MEMOR-E: In-Context and Fine-Tuned LLM Personalization for Alzheimer's Assistive Robotics
Apr 28, 2026Maissa Abir Smaili, Eren Sadikoglu, Ransalu SenanayakeAlzheimer's Disease LLM Personalization
Istanbul Medipol University Istanbul, Türkiye · Arizona State University Tempe, USA
Factors Influencing Conversational Engagement in Robot-Delivered Individual Cognitive Stimulation Therapy (iCST) for Dementia in Home Settings
Jul 9, 2026Emmanuel Akinrintoyo, Nicole SalomonsHealthcare Human-AI Interaction
Imperial College London
TALK-Dem: Benchmarking Embodied Task Planning under Dementia-Associated Communication Patterns
Sep 29, 2026Guangxin Zhao, Yiran Hu, Yuan Cao +7Large Language Model-Based Robot Planning Robot Task Planning
University of Notre Dame · Nanyang Technological University · Tohoku University

Apr 28, 2026J/K move · Enter open · S save

MEMOR-E: In-Context and Fine-Tuned LLM Personalization for Alzheimer's Assistive Robotics

Maissa Abir Smaili, Eren Sadikoglu, Ransalu Senanayake

Istanbul Medipol University Istanbul, Türkiye · Arizona State University Tempe, USA

Alzheimer's Disease LLM Personalization Assistive Robotics Medical LLMs

Open paper AskPDF arXiv

Alzheimer's disease is a neurodegenerative disorder marked by progressive declines in memory and language that reduce independence in daily life, motivating socially assistive robotic support. This paper presents MEMOR-E, a mobile quadruped robot with an interactive tablet interface that assists patients and caregivers through medication reminders, routine guidance, memory oriented interactions, and companionship. We evaluated the feasibility of fine tuning large language models (LLMs) to emulate stage consistent cognitive behavior and interpret responses across standard neuropsychological language tasks, using audio transcriptions from 235 Alzheimer's patients and synthetically generated healthy controls. We also report findings on using in context learning (ICL) in LLMs, where a second LLM produced domain and severity level cognitive error summaries. Our results show that MEMOR-E can generate stage aware, non diagnostic cognitive summaries that support personalized assistive interactions, while explainable AI mechanisms translate model outputs into transparent, human readable evidence to enable caregiver oversight and trustworthy human robot interaction.

Abstract

Explore similar work

MEMOR-E: In-Context and Fine-Tuned LLM Personalization for Alzheimer's Assistive Robotics

Factors Influencing Conversational Engagement in Robot-Delivered Individual Cognitive Stimulation Therapy (iCST) for Dementia in Home Settings

TALK-Dem: Benchmarking Embodied Task Planning under Dementia-Associated Communication Patterns