An Edge-Host-Cloud Architecture for Robot-Agnostic, Caregiver-in-the-Loop Personalized Cognitive Exercise: Multi-Site Deployment in Dementia Care

Wenzheng Zhao; Ruth Palan Lopez; Shu Fen Wung; Fengpei Yuan

cs.ROApr 1, 2026

An Edge-Host-Cloud Architecture for Robot-Agnostic, Caregiver-in-the-Loop Personalized Cognitive Exercise: Multi-Site Deployment in Dementia Care

Authors: Wenzheng Zhao, Ruth Palan Lopez, Shu Fen Wung, Fengpei Yuan

Organizations: Worcester Polytechnic Institute

Abstract

We present Speaking Memories, a distributed, stakeholder-in-the-loop robotic interaction platform for personalized cognitive exercise support. Rather than a single robot-centric system, Speaking Memories is designed as a generalizable robotics architecture that integrates caregiver-authored knowledge, local edge intelligence, and embodied robotic agents into a unified socio-technical loop. The platform fuses auditory, visual, and textual signals to enable emotion-aware, personalized dialogue, while decoupling multimodal perception and reasoning from robot-specific hardware through a local edge interaction server. This design achieves low-latency, privacy-preserving operation and supports scalable deployment across heterogeneous robotic embodiments. Caregivers and family members contribute structured biographical knowledge via a secure cloud portal, which conditions downstream dialogue policies and enables longitudinal personalization across interaction sessions. Beyond real-time interaction, the system incorporates an automated multimodal evaluation layer that continuously analyzes user responses, affective cues, and engagement patterns, producing structured interaction metrics at scale. These metrics support systematic assessment of interaction quality, enable data-driven model fine-tuning, and lay the foundation for future clinician- and caregiver-informed personalization and intervention planning. We evaluate the platform through real-world deployments, measuring end-to-end latency, dialogue coherence, interaction stability, and stakeholder-reported usability and engagement. Results demonstrate sub-6-second response latency, robust multimodal synchronization, and consistently positive feedback from both participants and caregivers. Furthermore, subsets of the dataset can be shared upon request, subject to participant consent and IRB constraints.

Figures & tables

Fig. 1: System architecture of the Speaking Memories framework, illustrating a layered host–edge–cloud design for emotion-aware reminiscence interaction. The Hardware Layer interfaces with the user through onboard sensors and actuators. The Edge Robot Layer performs real-time multimodal sensing, lightweight context aggregation, and robot-side interaction state management, enabling low-latency perception and feedback independent of robot embodiment. The Host Processing Layer handles multimodal context construction, emotion-aware dialogue reasoning, conversation policy execution, and interaction logging, serving as the core runtime intelligence of the system. The Cloud Coordination Layer supports caregiver-in-the-loop personalization and long-term operation by managing user profiles and memory databases, authentication and privacy control, and large-scale analytics and dataset aggregation. This separation of concerns decouples real-time interaction from long-term data management, enabling privacy-preserving deployment, cross-session personalization, and scalable operation across heterogeneous robotic platforms.

Fig. 2: Pipeline of the Speaking Memories platform, a distributed, stakeholder-in-the-loop architecture for adaptive reminiscence interaction. The system comprises three tightly coupled layers. (Left) A multimodal user input layer acquires visual, auditory, and textual cues from the participant during real-time interaction with an embodied robot agent. (Center) A local edge interaction server decouples multimodal perception and reasoning from robot-specific hardware, performing emotion-aware dialogue generation using vision–language and large language models conditioned on conversational context, affective cues, and structured caregiver-provided knowledge. (Right) A cloud–local data and analytics layer supports caregiver-in-the-loop personalization through a secure portal and automated multimodal evaluation, producing engagement metrics, longitudinal interaction summaries, and caregiver-oriented reports. This closed-loop design enables low-latency, privacy-preserving interaction, scalable deployment across heterogeneous robotic embodiments, and continuous data-driven personalization over repeated sessions.

Fig. 3: System software flow diagram of the Speaking Memories framework for adaptive reminiscence interaction. Colored boxes represent different system components, white boxes with borders denote data flow and information exchange. Italicized text without boxes indicates user actions, while non-italicized text without boxes represents Robot/User-executed actions.

Fig. 5: Flowchart of the experimental procedure.

Fig. 6: Real-world deployment of the Speaking Memories framework using two different hardware configurations at an elder care facility. (a) The left deployment used Robot I , relied soly on Pepper’s native hardware , using its onboard microphone, camera, and tablet for multimodal interaction. (b) The right deployment used Robot II , a Jetson-based edge client , where sensing and feedback were executed on an external Jetson Orin NX unit attached to the Pepper robot. Although both configurations delivered personalized reminiscence dialogues, the Jetson setup achieved substantially lower latency and greater modularity by offloading computation to the edge server.

$on\_topic(u)$ — On-topic relevance of the utterance: • 0: the utterance content is not semantically related to the current reminiscence prompt or displayed media; • 1: the utterance content is semantically related to the prompt or media, including reflective or affective responses.
$r\_depth(u)$ — Reminiscence depth reflecting autobiographical richness: • 0: no personal content (e.g., yes/no or filler responses); • 1: a single personal detail or value; • 2: multiple personal details or reflective elaboration; • 3: rich autobiographical narrative involving connected experiences.
$emo\_tone(u)$ — Emotional tone capturing dementia-relevant affective expression: • POSITIVE: clearly pleasant, appreciative, or joyful affect; • BITTERSWEET: mixed but meaningful affect, such as reminiscence-related sadness co-occurring with appreciation; this category captures non-harmful sadness common in dementia reminiscence; • NEGATIVE_DISTRESS: clear emotional distress, discomfort, or expressed desire to avoid the topic; • NEUTRAL/MIXED: primarily factual, ambiguous, or emotionally flat responses without a dominant tone.
$da(u)$ — Dialog act category describing the functional role of the utterance in interaction: • ANSWER: brief responses directly answering the robot’s question; • SELF_DISCLOSURE: autobiographical sharing beyond minimal yes/no responses; • QUESTION: utterances seeking information or clarification; • ACKNOWLEDGMENT/BACKCHANNEL: brief signals indicating attention or listening without adding new content; • PHATIC/SOCIAL: social remarks not directly related to reminiscence content; • OFF_TOPIC_COMMENT: content clearly unrelated to the prompt or displayed media; • REPAIR/MISUNDERSTANDING: utterances indicating hearing, comprehension, or interactional difficulties.

TABLE I: LLM-based semantic verbal engagement annotation scheme.

$gaze\_robot\_ratio(u)$ — Proportion of time the participant attends to the robot or tablet during the turn: • Computed as the fraction of sampled frames labeled with attention toward the robot or tablet interaction area. • Captures sustained visual or postural engagement rather than eye contact with the camera.
$valence\_mean(u)$ — Average facial valence during the turn: • Frame-level facial valence is labeled as negative ( $-1$ ), neutral ( $0$ ), or positive ( $+1$ ).
$valence\_pos\_ratio(u)$ — Proportion of positively valenced frames: • Measures the prevalence of positive facial expressions during the turn.

TABLE II: VLM-based nonverbal engagement annotation scheme (turn-level).

ID	Sex	Age Range	Group	Robot Setup	Duration (min)	Dialogue Rounds	Triggers Used
P04	F	75–84	Healthy	I	21	20	4
P06	F	75–84	PLWD	I	23	17	3
P07	M	75–84	PLWD	I	24	15	3
P08	F	85+	PLWD	I	17	15	2
P09	M	85+	PLWD	I	22	26	3
P10	F	85+	PLWD	I	5	4	1

TABLE III: Session Summary per Participant (N=12). Five additional older-adult sessions were conducted but excluded from analysis due to missing signed consent documentation required by the IRB-approved protocol; engineering/debug sessions are not included.

Metric	Pepper-native	Jetson-edge
End-to-end latency (s)	$9.25\pm 2.81$	$\textbf{5.89}\pm\textbf{1.11}$
Interaction completion rate	84.82%	100%
Dialogue coherence rate	88.30%	92.28%
Session-level statistics
Dialogue turns per session	$16.81\pm 9.27$
Session duration (min)	$17.94\pm 7.15$

TABLE IV: Summary of Quantitative System Evaluation Results

Component	Pepper-native (s)	Jetson-edge (s)
Speech recognition (ASR)	1.18	0.97
Multimodal reasoning	4.63	4.48
System orchestration	3.44	0.44
Total	9.25	5.89

TABLE V: Component-Level Latency Breakdown Across System Configurations

Dimension	Items	Mean $\pm$ SD
Attractiveness	Q1–Q2	4.7 $\pm$ 0.5
Interaction quality	Q3–Q5	4.5 $\pm$ 0.7
Usability	Q6-Q7	4.0 $\pm$ 0.9

TABLE VI: Participant (dementia) evaluation scores (Likert 1–5) aggregated by interaction dimension. Each dimension groups related survey items (Q1–Q7) corresponding to perceived attractiveness, interaction quality, and usability of the robot system. Values are reported as mean

\pm

standard deviation across all sessions.

Dimension	Items	Mean $\pm$ SD
Interface usability	Q1–Q2	4.6 $\pm$ 0.7
Emotional/positive engagement	Q3, Q8	3.7 $\pm$ 1.5
Linguistic/dialogue naturalness	Q4–Q6	3.5 $\pm$ 1.2
System stability	Q7	3.7 $\pm$ 1.0
Overall recommendation	Q9	4.5 $\pm$ 0.6

TABLE VII: Caregiver ratings by dimension (Likert 1–5). Each dimension groups related survey items (Q1–Q9) corresponding to interface usability, emotional engagement, dialogue naturalness, system stability, and overall recommendation.

Turn	$t_{\text{user}}$ (s)	$on\_topic$	$r\_d$	$emo$	$da$	$gaze\_r$	$val+$
1	10	1	1	POS	DIS	1	0.667
2	8	1	1	BIT	DIS	1	0.875
3	30	1	2	BIT	DIS	0.967	0.9
4	39	1	2	POS	DIS	1	0.842
5	54	1	2	POS	DIS	1	0.226
6	41	1	2	BIT	DIS	1	0.25

TABLE VIII: Illustrative turn-level multimodal engagement for Participant P08.

ID	Robot Setup	On-topic	$r_{d}$	SDis	Gaze	Val-mean	Val+
P06	I	0.92	1.58	0.92	0.94	$-$ 0.23	0.00
P07	I	0.83	1.33	0.83	0.45	$-$ 0.47	0.00
P08	I	0.93	1.57	0.93	1.00	0.30	0.50
P09	I	0.64	0.80	0.52	0.78	$-$ 0.12	0.00
P10	I	0.00	0.00	0.00	1.00	0.00	0.00
P11	I	0.50	0.61	0.44	1.00	0.64	0.61

TABLE IX: Participant-level multimodal engagement statistics aggregated across dialog turns.

Metric	Robot Setup I	Robot Setup II
On-topic rate	0.64 $\pm$ 0.35	0.55 $\pm$ 0.16
Mean reminiscence depth	0.98 $\pm$ 0.63	0.60 $\pm$ 0.25
Self-disclosure ratio	0.61 $\pm$ 0.36	0.37 $\pm$ 0.12
Gaze-to-robot rate	0.86 $\pm$ 0.22	0.98 $\pm$ 0.04
Valence mean	0.02 $\pm$ 0.39	0.25 $\pm$ 0.41
Positive valence ratio	0.18 $\pm$ 0.29	0.26 $\pm$ 0.37

TABLE X: Comparison of multimodal engagement metrics across robot setups.

Explore similar work

Apr 28, 2026

MEMOR-E: In-Context and Fine-Tuned LLM Personalization for Alzheimer's Assistive Robotics

Alzheimer's disease is a neurodegenerative disorder marked by progressive declines in memory and language that reduce independence in daily life, motivating socially assistive robotic support. This paper presents MEMOR-E, a mobile quadruped robot with an interactive tablet interface that assists patients and caregivers through medication reminders, routine guidance, memory oriented interactions, and companionship. We evaluated the feasibility of fine tuning large language models (LLMs) to emulate stage consistent cognitive behavior and interpret responses across standard neuropsychological language tasks, using audio transcriptions from 235 Alzheimer's patients and synthetically generated healthy controls. We also report findings on using in context learning (ICL) in LLMs, where a second LLM produced domain and severity level cognitive error summaries. Our results show that MEMOR-E can generate stage aware, non diagnostic cognitive summaries that support personalized assistive interactions, while explainable AI mechanisms translate model outputs into transparent, human readable evidence to enable caregiver oversight and trustworthy human robot interaction.

Maissa Abir Smaili, Eren Sadikoglu, Ransalu Senanayake

Istanbul Medipol University Istanbul, Türkiye · Arizona State University Tempe, USA

Alzheimer's Disease LLM Personalization Assistive Robotics Medical LLMs

Ask PDF

Jul 9, 2026

Factors Influencing Conversational Engagement in Robot-Delivered Individual Cognitive Stimulation Therapy (iCST) for Dementia in Home Settings

Social robots offer a promising means of supporting cognitive therapies for dementia care by guiding structured conversation and therapeutic activities. However, little is known about the conversational dynamics that emerge during robot-delivered cognitive stimulation therapy (CST) sessions. This study analysed the interaction patterns from robot-delivered individual CST (iCST) sessions conducted with people living with dementia in home settings. Our Co-STAR (Cognitive Stimulation Therapy by an Autonomous Robot) system was deployed in the homes of eight PwDs for one week, who completed 30-minute sessions. Conversational metrics, including words per turn, speech production rate, response duration, response latency, and self-referential language, were analysed to examine how conversational engagement is shaped by prompt personalisation, interaction phase, and participant characteristics. The findings highlight three key interactional properties of robot-delivered iCST. First, personalised prompts significantly increase response duration, self-referential language, and overall engagement compared to generic prompts. Second, conversational behaviour changes within sessions, with a reduction in the verbal output and autobiographical engagement observed during later interaction phases, which suggests cognitive fatigue. Third, first-session conversational metrics were associated with long-term participation, while living situation influenced conversational engagement patterns. These findings provide empirical insights into the factors that shape conversational engagement in robot-delivered iCST. They inform the design of adaptive conversational robots for dementia therapy.

Emmanuel Akinrintoyo, Nicole Salomons

Imperial College London

Healthcare Human-AI Interaction Assistive Robotics Human-Robot Interaction

Ask PDF

Sep 29, 2026

TALK-Dem: Benchmarking Embodied Task Planning under Dementia-Associated Communication Patterns

Existing LLM-driven robot task planners rely on a taken-for-granted assumption of an ideal user whose instructions are clear, complete, and task-focused. However, when interacting with real-world users, especially those experiencing cognitive impairments, such as people living with dementia (PLWD), the planners often make mistakes and even pose physical safety risks. We proposed TALK-Dem (Talking Attributes and Linguistic Knowledge in Dementia), the first benchmark for evaluating LLM-driven robot task planning under dementia-associated verbal communication. TALK-Dem contains 4,800 instructions and covers five typical communication patterns, including Referential Imprecision, Object Substitution, Empty Speech, Topic Drift, and Intrusion, at three intensity levels. Experiments across six open-weight LLMs reveal a substantial robustness gap. Across communication patterns, open-weight models exhibited performance drops of up to 22.3 percentage points compared to ideal instructions. This revealed a critical gap and even danger for real-world applications, especially in assistive robotics, where locally deployable models are necessary due to privacy concerns and connectivity constraints. To mitigate this issue, we proposed the Context-Aware Retrieval from Experience (CARE) method, which retrieves relevant previously resolved tasks to provide task-specific interpretation and planning context. CARE generally outperformed standard prompting baselines across the six open-weight models, improving average task success by 18.1 percentage points over the vanilla prompt. These results highlighted the importance of both evaluating communication robustness and developing effective adaptation strategies for locally deployable assistive robots. The TALK-Dem dataset is publicly available at https://anonymous.4open.science/r/TALK-Dem-A6B3/.

Guangxin Zhao, Yiran Hu, Yuan Cao +7

University of Notre Dame · Nanyang Technological University · Tohoku University

Large Language Model-Based Robot Planning Robot Task Planning Assistive Robotics Language Model Robustness

Ask PDF