Communication in shared space interweaves verbal and non-verbal signals, and pointing gestures anchor language to the environment: "put the cup on that one" is uninterpretable without the gesture that fixes the referent. Yet no common framework exists for evaluating whether generated gestures indicate their intended referent; distributional metrics reward a gesture aimed at the wrong object as long as it looks natural. We introduce a benchmark for spatially grounded gesture generation, comprising ~2K pointing-annotated clips from naturalistic VR dialogue with ground-truth 3D referents, a task in which systems must decide when, how and where to point within conversational speech, and a protocol that separates temporal alignment, spatial grounding and perceived naturalness. We also provide a flow-matching baseline, MM-Conv-Flow. Evaluating it alongside an independent retrieval-based system and captured human motion, we find that geometric grounding can exceed that of human pointing without any gain in perceived naturalness, showing that referential gesture quality must be measured along separate dimensions.
Figures & tables
Figure 2: The five AI2-THOR rooms of MM-Conv. Room 211 is held out for evaluation.
Figure 3: Geometry of the 100 held-out test targets. Each point represents one test clip. Targets span substantial variation in speaker–target distance and apparent angular extent.
Split
Rooms
MM-Conv
SGS-HSI
Total
Train
4 rooms
1,503
1,022
2,525
Test
1 room
339
116
455
Table 1: Scene-disjoint split statistics on MM-Conv and SGS-HSI. Final benchmark scores (Table 3 ) use a fixed 100-item evaluation set drawn from the held-out MM-Conv clips.
Figure 4: Training data samples across target configurations. The blue mesh shows the captured SMPL-X body, and the green volume marks the target object. Visualisation tool based on [ 25 ] , released with the benchmark.
Variant
Score a ↑
Score b ↑
STG ↑
θeff↓
GT MoCap
0.909
0.765
0.718
11.8°
Mode A (frozen)
0.527
0.315
0.173
36.2°
Mode B (unfrozen)
0.593
0.593
0.374
19.5°
Mode B + nudge ∗
0.581
0.743
0.463
11.0°
Table 2: Development-stage evaluation of MM-Conv-Flow variants on the full 455-clip held-out set (MM-Conv + SGS-HSI), using an earlier hold-based temporal scorer. Included to show the relative effect of the baseline design choices; temporal and STG values are not directly comparable to the final 100-item MM-Conv evaluation in Table 3 . ∗ Configuration used in the final evaluation.
Metric
GT
MM-Conv-Flow
RePointer
Dir.
Spatial
Score b
0.767±0.223
0.691±0.291
0.905±0.195
↑
Temporal
Score a
0.941±0.210
0.499±0.402
0.784±0.278
↑
Spatio-Temporal
STG
0.720±0.265
0.361±0.365
0.719±0.307
↑
Table 3: Objective evaluation on 100 held-out MM-Conv clips. Grounding scores are the mean ± SD across clips.
Figure 5: Temporal alignment in seconds. Cumulative distribution of absolute apex timing error over the 100 test clips. The shaded region marks the primary tolerance τ=0.33 s; markers indicate median errors of 0.40 s for RePointer and 0.92 s for MM-Conv-Flow. For GT, the same frozen apex detector is applied to the captured reference motion and is shown as a detector diagnostic rather than an estimate of human timing variability.
Condition
MOS
95% CI
n
vs. GT
GT MoCap
3.58
± 0.10
473
—
MM-Conv-Flow
3.11
± 0.10
470
RePointer
2.95
± 0.11
467
MM-Conv-Flow vs. RePointer
n.s.
Table 4: Mean opinion scores for motion naturalness (1–5). CIs are 95% CIs of the mean rating; n denotes individual ratings. Pairwise tests use Wilcoxon signed-rank tests on per-participant condition means with Holm correction ( N=62 ; one zero difference discarded in the MM-Conv-Flow vs. RePointer test, N=61 ; *** p<.001 ; n.s.: p≥.05 ).
Figure 6: Perceptual evaluation of motion naturalness. Left : mean opinion scores (MOS; 1–5) for each of the 23 evaluated clips under GT MoCap, MM-Conv-Flow, and RePointer. Right : overall means. Error bars show 95% CIs of the mean. Significance is computed using Wilcoxon signed-rank tests on per-participant condition means with Holm correction ( ∗∗∗p<.001 ; n.s.: p≥.05 ).
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Sensitivity of the temporal evaluation to τ and σt . (a) Mean Score a for GT, RePointer, and MM-Conv-Flow. Absolute score levels vary substantially with the temporal-score parameters. (b) Difference in Score a between RePointer and MM-Conv-Flow; RePointer remains higher throughout the tested parameter range. (c) Difference in STG between RePointer and GT; this ordering changes with the temporal-score parameters. The primary benchmark setting is τ=10 , σt=15 frames.
Figure 8: Sensitivity of the RePointer–GT combined-grounding comparison. Each cell shows the mean STG difference (RePointer minus GT) for one (τ,σt) parameterisation. The black contour marks zero difference, hatching indicates regions where the 95% confidence interval excludes zero, and the star marks the primary setting (τ,σt)=(10,15) . Its proximity to the zero contour illustrates why the RePointer–GT STG near-tie is parameter-sensitive.
Figure 9: Single MotionDiT transformer block (layers 4–11) showing P1 co-speech components (gray) and P2 spatial injection sites (yellow). The spatial pathway adds a gated residual bias on layers 4–11 (Mode A and B) and single-token cross-attention on layers 8–11 (Mode B only). Both paths use zero-initialized projections, so the initial P2 output matches P1 exactly.
Setting
P1
P2
Data
BEAT2
70% pointing, 30% BEAT2
Optimizer
AdamW
AdamW
Learning rate
2×10−4
3×10−5 backbone; 5×10−5 spatial
Weight decay
0.05
0.01
Effective batch size
192
64
Epochs
1100
200
Appendix
Table 5: Training configuration for the co-speech prior (P1) and spatial adaptation (P2). A dash denotes a loss not used in that phase.
Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands · Utrecht University, Utrecht, The Netherlands · Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands