Efficient Emotion-Aware Iconic Gesture Prediction for Robot Co-Speech
Authors: Edwin C. Montiel-Vazquez, Christian Arzate Cruz, Stefanos Gkikas, Thomas Kassiotis, Giorgos Giannakakis, Randy Gomez
Organizations: School of Engineering and Sciences Tecnologico de Monterrey State of Mexico, Mexico · Honda Research Institute Japan Wako City, Japan · Department of Electronic Engineering Hellenic Mediterranean University Chania, Greece
Co-speech gestures increase engagement and improve speech understanding. Most data-driven robot systems generate rhythmic beat-like motion, yet few integrate semantic emphasis. To address this, we propose a lightweight transformer that derives iconic gesture placement and intensity from text and emotion alone, requiring no audio input at inference time. The model outperforms GPT-4o in both semantic gesture placement classification and intensity regression on the BEAT2 dataset, while remaining computationally compact and suitable for real-time deployment on embodied agents.
Figures & tables
Fig. 1 : Task overview. An utterance is separated into words, and the semantic gesture placement and intensity is calculated per word.
Fig. 2 : High-level overview of the proposed model.
Depth
SA
Accuracy
Precision
F1
GFLOPs
Latency (ms) ↓
2
8
68.78
53.92
50.27
5.79
8.39
2
4
68.64
53.55
47.84
3.11
4.46
2
2
68.75
53.82
49.72
1.77
3.20
2
1
68.68
53.76
49.38
1.09
2.16
1
8
68.53
53.76
49.57
2.90
4.02
1
4
68.56
53.68
48.98
1.55
2.45
TABLE I : Performance and computational cost across model configurations.
Model
Accuracy
Precision
Recall
F1
LLM
53.36
52.63
53.36
52.92
Ours
68.64
53.55
68.64
47.84
TABLE II : Macro-averaged results for iconic placement per word.
Model
MAE
MSE
RMSE
R 2
PR
Spearman
LLM
0.08
0.05
0.22
-1.23
0.09
0.06
Ours
0.08
0.02
0.15
-0.07
0.20
0.16
TABLE III : Regression results for word intensity of iconic gestures.
Fig. 3 : Semantic co-speech implementation on the social robot Haru.