cs.ROMar 19, 2026

Empathetic Motion Generation for Humanoid Educational Robots via Reasoning-Guided Vision--Language--Motion Diffusion Architecture

Authors: Fuze Sun, Lingyu Li, Lekan Dai, Xinyu Fan

Organizations: Department of Computer Science and Engineering, University of Liverpool

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

This article suggests a reasoning-guided vision-language-motion diffusion framework (RG-VLMD) for generating instruction-aware co-speech gestures for humanoid robots in educational scenarios. The system integrates multi-modal affective estimation, pedagogical reasoning, and teaching-act-conditioned motion synthesis to enable adaptive and semantically consistent robot behavior. A gated mixture-of-experts model predicts Valence/Arousal from input text, visual, and acoustic features, which then mapped to discrete teaching-act categories through an affect-driven policy.These signals condition a diffusion-based motion generator using clip-level intent and frame-level instructional schedules via additive latent restriction with auxiliary action-group supervision. Compared to a baseline diffusion model, our proposed method produces more structured and distinctive motion patterns, as verified by motion statics and pairwise distance analysis. Generated motion sequences remain physically plausible and can be retargeted to a NAO robot for real-time execution. The results reveal that reasoning-guided instructional conditioning improves gesture controllability and pedagogical expressiveness in educational human-robot interaction.

Figures & tables

Figure 1: Training and inference flow of the proposed valence/arousal estimator. During training, modality-specific XGBoost experts are first optimized on CMU-MOSEI using Huber-style regression objectives, and their outputs are then used to train a softmax-based reliability gate with fused mean squared error loss. During inference, student text, visual, and acoustic features are processed by the trained modality experts, followed by reliability-aware gated fusion and calibration to produce final valence and arousal estimates for downstream teaching-act selection.
Figure 2: RAPID-Motion diffusion framework for pedagogical co-speech gesture generation. Student affect is estimated using a MOSEI-trained Valence–Arousal model and converted by an LLM reasoning module into a clip-level teaching-act vector uu and frame-level schedule u1:Tu_{1:T} . Audio and text embeddings are used as multimodal conditioning signals, together with instructional vectors and an optional motion prefix. A transformer-based diffusion denoiser generates motion tokens using local attention and self-attention. Training minimizes diffusion reconstruction loss with an auxiliary act-classification loss to enforce pedagogical consistency.
Teaching-ActAmplitudeVelocityJerkEnergyRangeInterpretation
explain1.001.001.001.001.00highly expressive explanatory motion
checkin0.500.500.140.070.80moderate monitoring gestures
challenge0.470.470.040.020.84controlled but directive motion
neutral0.450.450.140.040.85regular conversational motion
praise0.340.340.030.020.90smooth congratulatory gestures
unclear0.610.610.040.030.53irregular / uncertain motion
Table 1: Normalized motion statistics computed from BVH trajectories under different teaching-act conditions. Values are normalized by the maximum across all conditions.
Figure 3: Representative gesture poses generated by the proposed model under different teaching-act conditions in the simulation environment. The poses illustrate that the diffusion policy produces distinct motion styles corresponding to pedagogical intentions, including expressive gestures for explanation and praise, directive posture for challenge, and low-intensity motion for neutral interaction.
Figure 4: Comparison between ground-truth and predicted valence trajectories. The predicted curves closely follow the temporal trend of the real values. In Dataset 3, the negative dip and final increase are correctly reproduced. In Dataset 4, the model captures the overall positive tendency and major trend changes, although the predicted amplitudes are smoother than the ground-truth values. These results indicate that the model can track valence dynamics while slightly compressing extreme intensities.
Figure 5: Pairwise distance heatmaps of normalized motion statistics across teaching-act categories. The baseline model shows weak separation between several acts, while the proposed conditioning produces a more structured distribution, with clear separation for highly expressive acts such as explain .
More in this topic

Co-Speech Gesture Generation

Co-speech gesture generation creates body movements that express and synchronize with spoken language.

See all 50 papers

Explore similar work

CardsList

  1. EmoPose: Vision-Language Model Guided Emotion-Aware Gesture Generation for Humanoid Robots

    Sep 20, 2026Daojie Peng, Bingtao Wang, Fulong Ma +3Large Language Model-Based Robot PlanningHumanoid Robot Control

    The Hong Kong University of Science and Technology (Guangzhou) · The Shandong University · RoboScience

  2. EgoMotion: Hierarchical Reasoning and Diffusion for Egocentric Vision-Language Motion Generation

    Apr 21, 2026Ruibing Hou, Mingyue Zhou, Yuwei Gui +5Human Motion Generation

    Key Laboratory of Intelligent Information Processing, Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS), Beijing, 100190, China · Jilin University (JLU), Changchun, 130012, China · Beijing University of Posts and Telecommunications (BUPT), Beijing, 100876, China +1

  3. ECHO-G: Embodied Co-speech Humanoid mOtion Generation

    Sep 30, 2026Yizhao Li, Pusen Gao, Ming Wang +3Text-to-Motion GenerationHuman Motion Generation

    Beihang University · Mondo Robotics · The Hong Kong University of Science and Technology +1