Perceptually Grounded and Semantics-Aware Evaluation for Holistic Co-Speech Gesture Generation

Nick Milkin; Lanmiao Liu; Esam Ghaleb; Asli Ozyurek; Zerrin Yumak

cs.CVOct 8, 2026

Perceptually Grounded and Semantics-Aware Evaluation for Holistic Co-Speech Gesture Generation

Authors: Nick Milkin, Lanmiao Liu, Esam Ghaleb, Asli Ozyurek, Zerrin Yumak

Organizations: Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands · Utrecht University, Utrecht, The Netherlands · Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands

Abstract

Holistic and semantics-aware co-speech gesture generation has advanced rapidly, yet evaluation remains behind: objective metrics do not consistently reflect human perception, and semantic appropriateness remains difficult to quantify. We present a perceptually grounded and semantics-aware benchmark that combines standardized model comparison, human-centered metric validation, and fine-grained semantic evaluation. We first curate a list of 13 objective metrics covering different aspects, including distributional similarity, geometric fidelity, kinematic quality, cross-modal synchrony, and semantic appropriateness. For the semantic-appropriateness category, we propose a new metric, Semantic Gesture Preservation (SGP), which measures how far semantic gestures in the ground truth are preserved in the generated gestures. For this, we augment the BEAT2 dataset's annotations using a multi-modal LLM. We then conduct a perceptual study where 101 participants score generated gestures among five dimensions, including human-likeness, motion diversity, absence of animation errors, speech timing and content match. We systematically analyze objective metric--subjective score correlations. Unlike Semantic Score (SC), which shows no significant association with the evaluated perceptual dimensions, SGP is selectively aligned with speech-aware human judgments. We construct five target-specific composite metrics aligned with the subjective dimensions. These composites improve perceptual alignment across all five dimensions, with the largest gains for absence of animation errors and content match, indicating that complementary objective signals can better approximate human judgments than individual metrics alone. Overall, our results show that objective metrics require validation against subjective evaluations.

Figures & tables

Panel of Figure 1 — Figure 1: Perceptual study interfaces for the audio (left) and muted (right) conditions, evaluating speech-aware and visual-quality dimensions, respectively. Participants complete one condition using the same seven-point Likert scale and video interface.

Method	FGD $\downarrow$	BC $\uparrow$	Div. $\uparrow$	Dens. $\uparrow$	Cov. $\uparrow$	Dice $\uparrow$	LDLJ ${}_{\mathrm{REL}}\!\rightarrow\!0$	SRGR $\uparrow$	PCK $\uparrow$	MJD $\downarrow$	Chamfer $\downarrow$	Hausdorff $\downarrow$	Foot Contact $\downarrow$	SC $\uparrow$	SGP $\uparrow$
EMAGE	5.523	0.692	88	0.8972	0.07275	0.6407	0.0530	0.05494	0.3309	0.2410	0.01375	0.2908	0.5048	0.248	0.4372
SemConFlow	2.245	0.780	120	0.01559	0.06647	0.6793	-0.4294	0.05868	0.3552	0.2007	0.01112	0.2530	0.5932	0.314	0.4205
SemTalk	4.266	0.727	116	0.1123	0.06673	0.6241	-0.6020	0.05059	0.3037	0.2303	0.13976	0.4663	0.4040	0.268	0.4014
GestureLSM	4.268	0.525	112	0.009239	0.009817	0.6573	1.5386	0.04536	0.2715	0.2391	0.01797	0.2909	0.8883	0.248	0.2622

Table 1: Objective comparison of the benchmarked holistic co-speech gesture generation methods on BEAT2. All metrics are computed over the 265 shared test sequences, except SGP, which is evaluated on the 15 sequences used in the perceptual study with Gemini-based semantic annotations. Arrows indicate the preferred direction, and bold denotes the best result for each metric.

Figure 2: Subjective ratings across five perceptual dimensions. Points show mean ratings with standard-deviation error bars; BEAT2 serves as the reference, and the remaining methods are generated results.

Perceptual dimension pair	Generated ( $n=60$ )	Including GT ( $n=75$ )
Human-likeness – Motion diversity	0.728	0.626
Human-likeness – Absence of animation errors	0.928	0.960
Human-likeness – Speech timing	0.650	0.813
Human-likeness – Content match	0.624	0.801
Motion diversity – Absence of animation errors	0.770	0.647
Motion diversity – Speech timing	0.665	0.564

Table 2: Pairwise Spearman correlations among subjective perceptual dimensions.

Figure 3: Spearman correlations between thirteen objective metrics and five subjective evaluation dimensions ( $n=60$ ), with significance indicated by ${}^{*}p<0.05$ , ${}^{**}p<0.01$ , and ${}^{***}p<0.001$ .

Target Composite Metric	MSE $\downarrow$	MAE $\downarrow$	Composite $\rho\uparrow$	Best individual	$\Delta\|\rho\|\uparrow$
Human-likeness	0.2846	0.4140	0.6568	$\mathrm{LDLJ}_{\mathrm{REL}}$	$+0.0468$
Motion diversity	0.2035	0.3610	0.5375	Coverage	$+0.0117$
Absence of animation errors	0.1560	0.2984	0.7065	$\mathrm{LDLJ}_{\mathrm{REL}}$	$+0.0819$
Speech timing	0.2671	0.4349	0.7439	Coverage	$+0.0246$
Content match	0.3665	0.4844	0.7824	Coverage	$+0.0805$

Table 3: LOOCV performance of target-specific composite predictors over 60 generated model–sequence observations using 13 predictors. MSE and MAE report out-of-fold error on the seven-point subjective scale, while composite

\rho

measures Spearman correlation with subjective ratings. Best individual is the strongest single objective metric by absolute Spearman correlation, and

\Delta|\rho|

denotes the composite gain.

Panel of Figure 4 — Figure 4: Participant interface for the muted condition. Ratings become available only after the complete stimulus has been viewed.

Panel of Figure 5 — Figure 5: Participant interface for the audio condition. After viewing the audiovisual stimulus, participants rate speech timing and content match.

Panel of Figure 6 — Figure 6: Attention-check interfaces used in the two perceptual studies. The muted condition uses an instructed-response check, while the audio condition uses a content-based question about the preceding stimulus.

Panel of Figure 7 — Figure 7: Prompt used for LLM-based semantic gesture annotation. The model receives a short video segment together with its predefined semantic word and returns a structured description of gesture form and contextual meaning.

Figure 8: Representative event-level motion example used for semantic gesture annotation. The figure visualizes a short rendered motion clip of approximately 2–3 s centered on the annotated semantic interval. This event clip, together with the corresponding target word and structured annotation prompt, is provided to Gemini 2.5 Pro as input for fine-grained semantic annotation.

Panel of Figure 9 — Figure 9: Example of an LLM-generated semantic gesture annotation. The target semantic expression is paired with a fine-grained description of the visible motion and an interpretation of its communicative meaning.

Explore similar work

Mar 27, 2026cs.CV

SemConFlow: Semantic Grounding of Holistic Co-Speech Gesture Generation with Contrastive Flow-Matching

While the field of co-speech gesture generation has seen significant advances, producing holistic, semantically grounded gestures remains a challenge. Existing approaches rely on external semantic retrieval methods, which limit their generalisation capability due to dependency on predefined linguistic rules. Flow-matching-based methods produce promising results; however, the network is optimised using only semantically congruent samples without exposure to negative examples, leading to learning rhythmic gestures rather than sparse motion, such as iconic and metaphoric gestures. Furthermore, by modelling body parts in isolation, the majority of methods fail to maintain crossmodal consistency. We introduce a Contrastive Flow Matching-based co-speech gesture generation model that uses mismatched audio-text conditions as negatives, training the velocity field to follow the correct motion trajectory while repelling semantically incongruent trajectories. Our model ensures cross-modal coherence by embedding text, audio, and holistic motion into a composite latent space via cosine and contrastive objectives. Extensive experiments and a user study demonstrate that our proposed approach outperforms state-of-the-art methods on two datasets, BEAT2 and SHOW.

Lanmiao Liu, Esam Ghaleb, Aslı Özyürek +1

Max Planck Institute for Psycholinguistics · Donders Institute for Brain Cognition and Behaviour · Utrecht University

Multimodal Grounding Flow Matching Cross-Modal Alignment Human Motion Generation Co-Speech Gesture Generation

Ask PDF

Sep 29, 2026cs.CV

When Semantics Matter: Reliability-Aware Semantic-Rhythm Control for Co-Speech Gesture Generation

Co-speech gesture generation aims to synthesize natural gestures that are both temporally synchronized with speech and semantically consistent with the spoken content. Although recent methods can generate rhythmically plausible motions, they often rely heavily on acoustic prosody while underutilizing textual semantics, especially when semantic annotations are incomplete, noisy, or unavailable. Consequently, the generated gestures may follow speech rhythm while failing to express the intended semantics. To address this problem, we propose a reliability-aware semantic-rhythm control framework for co-speech gesture generation. We first learn a discrete motion prior that represents continuous gestures in a compact and structured motion-code space. We then introduce a dual-branch semantic contribution estimation mechanism consisting of a full multimodal branch and an audio-only branch. Their distributional discrepancy is formulated as conditional information gain to quantify how much textual semantics changes the predicted motion. Based on this estimate, a controllable semantic-rhythm objective selectively strengthens semantic guidance in content-relevant segments while limiting unnecessary semantic intervention in rhythm-dominant segments. Furthermore, we treat background noise as an acoustic reliability condition and introduce noise-conditioned feature modulation together with beneficial latent perturbation to improve generation robustness under realistic acoustic environments. Experiments on benchmark datasets demonstrate that the proposed framework achieves a favorable balance among semantic expressiveness, rhythmic synchronization, motion diversity, and robustness, enabling reliable and controllable co-speech gesture generation.

Zhirui Xing, Long Ye, Kaige Li +2

Hainan International College, Communication University of China, Lingshui, China · School of Data Science and Intelligent Media, Communication University of China, Beijing, China · School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen, China

Co-Speech Gesture Generation

Ask PDF

Aug 31, 2026cs.CV

Puppeteer: Object-Grounded Posture-Aware Co-Speech Gesture Generation

Generating co-speech gestures that are temporally coherent, semantically aligned with speech, and grounded with surrounding objects remains challenging. Prior speech-driven gesture models emphasize audio-gesture alignment but do not explicitly account for posture constraints or surrounding objects, failing to capture the inherent correlation between body gestures and the physical space. We present Puppeteer, a posture-aware, object-grounded co-speech gesture diffusion model operating in a causal latent space. We decompose long gestures into structured primitives and learn a causal variational autoencoder that encodes them into temporally ordered latent tokens, each depending only on the past. We then perform conditional diffusion directly in the causal latent space, conditioning on speech signals, motion history, an initial posture reference, and object geometry to synthesize physically consistent gestures. This temporally ordered latent formulation enables explicit temporal control and supports tasks such as gesture in-betweening and gesture completion. To better assess co-speech gesture synthesis beyond existing measures, we introduce new evaluation metrics tailored to this task. We also created SceneGes, the first curated synthetic 3D dataset of embodied co-speech gestures and corresponding 3D objects, enabling object-grounded gesture generation. Experiments show that Puppeteer generates more diverse and temporally synchronized gestures than prior methods, while enabling object-grounded gesture synthesis.

Vida Adeli, Soroush Mehraban, Jacob Rommann +3

Pickford AI · University of Toronto · Vector Institute

Motion Diffusion Models Human Motion Generation Co-Speech Gesture Generation Motion Representation Learning Diffusion Models

Ask PDF