GestAdapt: Workspace-Conditioned Co-Speech Gesture Generation for Humanoid Robots

Bosong Ding; Xianglin Zhang; Miao Xin; Murat Kirtay; Giacomo Spigler

cs.ROSep 29, 2026

GestAdapt: Workspace-Conditioned Co-Speech Gesture Generation for Humanoid Robots

Authors: Bosong Ding, Xianglin Zhang, Miao Xin, Murat Kirtay, Giacomo Spigler

Organizations: Department of Intelligent Systems, Tilburg University, The Netherlands. · TU-Dresden, Dresden, Germany. · School of Artificial Intelligence, China University of Mining and Technology-Beijing (CUMTB), Beijing, China.

Abstract

Co-speech gestures for robots must adapt not only to speech and embodiment, but also to the workspace available for performing the motion. Since the same speech can be accompanied by different gestures, a robot can respond to workspace constraints, e.g., gestures for speech next to a wall. In these scenarios, the robot should gesture in a suitable motion rather than simply correcting an unconstrained one. To achieve this goal, we present GestAdapt, a workspace-conditioned framework that conditions co-speech gesture generation on a prescribed wrist workspace. The GestAdapt framework learns from six complementary co-speech corpora through a shared motion representation and supports retargeting to different robot embodiments. Quantitative evaluation shows that generated motions remain close to the real-motion distribution while respecting the workspace. In a user study, gestures generated under modified workspace constraints receive a mean quality score of 3.24/5, above our no-workspace variant (2.43/5) and below the reference motions (3.68/5). In a real robot evaluation, all compared motions are retargeted to the Reachy2 humanoid robot under identical workspace constraints. Motions generated with our framework rank first in 69.7% of comparisons, higher than our no-workspace variant baseline and retargeted ground-truth motions constrained afterward. Overall, the results support adapting gestures to the available workspace during generation, rather than modifying unconstrained trajectories afterward to satisfy workspace constraints, potentially compromising gesture naturalness.

Figures & tables

Fig. 1: Examples of workspace-adaptive co-speech gestures by a Reachy2 humanoid robot. The chosen gesture space changes across three settings: (a) free space, (b) a nearby wall, and (c) a tabletop workspace. The robot adapts its gesture motion to remain within the specified workspace.

Fig. 2: Complementary motion and spatial support across the six co-speech corpora. The Principal Component Analysis (PCA) views show the motion distributions, while the front, side, and top views visualize the corresponding wrist-motion envelopes. Heatmap intensity indicates the number of envelopes covering each location, and colored contours show the spatial support of individual corpora.

Fig. 3: Unified motion representation across the six source datasets. Motion is first evaluated on each native kinematic hierarchy, and 38 anatomically corresponding joints are then selected and aligned to a common skeleton. The unified representation contains 33 dimensions for the upper body and palm axes and 45 dimensions for each hand, resulting in 123 dimensions.

Fig. 4: Overview of the speech-to-robot framework. Speech, motion history, and a workspace box condition a diffusion model to generate future gesture motion. The workspace is encoded as six spatial tokens and injected through cross-attention, while analytic guidance during denoising further corrects wrist motion toward the requested workspace. The generated motion is then retargeted to the target robot and executed under embodiment-specific constraints.

Method	$\mathrm{FGD}\downarrow$	$\mathrm{BC}\uparrow$	Jerk $\downarrow$	Violation $\downarrow$
Trimodal (deterministic) [ 2 ]	$11.045\pm 0.000$	$0.5283\pm 0.0000$	$10.202\pm 0.000$	$6.064\pm 0.000$
StyleGestures [ 3 ]	$11.579\pm 0.274$	$\mathbf{0.6356\pm 0.0019}$	$6.526\pm 0.005$	$19.667\pm 0.189$
DiffSHEG [ 28 ]	$15.579\pm 0.957$	$0.4889\pm 0.0038$	$5.163\pm 0.117$	$16.481\pm 0.366$
Ours w/o Workspace	$6.933\pm 0.419$	$0.4989\pm 0.0076$	$4.695\pm 0.105$	$9.815\pm 0.262$
Ours	$\mathbf{5.019\pm 0.223}$	$0.4515\pm 0.0036$	$\mathbf{4.295\pm 0.104}$	$\mathbf{1.095\pm 0.017}$

TABLE I: Quantitative results over 5 runs.

	Study 1	Study 2
Method	Mean score $\uparrow$	Mean rank $\downarrow$	Top-1 (%) $\uparrow$
Ground truth	$3.68\pm 0.20$	$2.53\pm 0.06$	$10.6\pm 3.9$
Ours w/o Workspace	$2.43\pm 0.18$	$2.11\pm 0.06$	$19.7\pm 4.8$
Ours	$\mathbf{3.24\pm 0.20}$	$\mathbf{1.37\pm 0.09}$	$\mathbf{69.7\pm 7.1}$

TABLE II: User-study results with 95% confidence intervals

(a) Six-corpus model versus single-corpus experts
Model or setting	FGD $\downarrow$	Jerk $\downarrow$	Violation $\downarrow$
Ours	$\mathbf{5.019\pm 0.223}$	$4.295\pm 0.104$	$\mathbf{1.095\pm 0.017}$
BEAT expert	$11.211\pm 0.241$	$6.707\pm 0.044$	$2.950\pm 0.040$
Trinity expert	$11.918\pm 0.446$	$5.790\pm 0.086$	$5.142\pm 0.098$
SHOW expert	$12.453\pm 0.501$	$4.935\pm 0.078$	$3.472\pm 0.106$
ZEGGS expert	$12.862\pm 0.405$	$7.671\pm 0.114$	$7.507\pm 0.120$

TABLE III: Ablation results over 5 runs.

Explore similar work

WaveSync: Constrained Wavefront Optimization for Synchronized Co-Speech Gestures in Humanoid Robots
Jun 15, 2026Thang Tran Viet, Thanh Nguyen Canh, Gia Huy Uong +4Co-Speech Gesture Generation Importance
University of Engineering and Technology, Vietnam National University, 10000, Hanoi, Vietnam. · School of Information Science, Japan Advanced Institute of Science and Technology, Nomi, 923-1211, Ishikawa, Japan. · School of Electronics and Computer Science, University of Southampton, SO17 1BJ Southampton, United Kingdom. +1
ECHO-G: Embodied Co-speech Humanoid mOtion Generation
Sep 30, 2026Yizhao Li, Pusen Gao, Ming Wang +3Co-Speech Gesture Generation Human Motion Generation
Beihang University · Mondo Robotics · The Hong Kong University of Science and Technology +1
Generating Natural and Expressive Robot Gestures through Iterative Reinforcement Learning with Human Feedback using LLMs
Jun 17, 2026Chris Lee, Flora Salim, Benjamin Tag +1Co-Speech Gesture Generation Humanoid Robotics
School of Computer Science and Engineering, University of New South Wales, Sydney, Australia · Universidad Central de Chile, Santiago, Chile

Jun 15, 2026, cs.ROJ/K move · Enter open · S save

WaveSync: Constrained Wavefront Optimization for Synchronized Co-Speech Gestures in Humanoid Robots

Thang Tran Viet, Thanh Nguyen Canh, Gia Huy Uong, Phuc Van Dinh+3

University of Engineering and Technology, Vietnam National University, 10000, Hanoi, Vietnam. · School of Information Science, Japan Advanced Institute of Science and Technology, Nomi, 923-1211, Ishikawa, Japan. · School of Electronics and Computer Science, University of Southampton, SO17 1BJ Southampton, United Kingdom.+1

Co-Speech Gesture Generation Importance

Open paper AskPDF arXiv

Expressive co-speech gestures are crucial for natural human--robot interaction, yet generating them on physical humanoid robots remains challenging because, unlike virtual avatars, robots must synchronize gestures with speech under strict kinematic and actuator constraints. We present \textbf{WaveSync}, a hybrid framework in which a Large Language Model decomposes dialogue responses into structured semantic schemas and assigns per-word importance weights, forming a continuous Semantic Importance Wave. Gesture trajectories are shaped through Dynamic Movement Primitives to ensure kinematic feasibility while enhancing expressiveness. A Wavefront Optimization stage aligns gesture stroke peaks with speech emphasis peaks and resolves residual temporal conflicts through gesture-duration compression and forward propagation. Experimental evaluation across five dialogue scenarios demonstrates effective gesture--speech alignment and favorable performance in both objective and subjective evaluations. The results further show that the key components of WaveSync contribute to producing gestures that are expressive, semantically grounded, and kinematically feasible. The code, resources, and videos are available at \href{https://github.com/pairs-lab/WaveSync}{WaveSync}.

Abstract

Explore similar work

WaveSync: Constrained Wavefront Optimization for Synchronized Co-Speech Gestures in Humanoid Robots

ECHO-G: Embodied Co-speech Humanoid mOtion Generation

Generating Natural and Expressive Robot Gestures through Iterative Reinforcement Learning with Human Feedback using LLMs