Can Motion-Language Models Ground Structure? STRIDE for Evaluating the Evaluators

Lixing Tan; Qing Xia; Yuting Guo; Shuai Li; Aimin Hao

cs.CVSep 26, 2026

Can Motion-Language Models Ground Structure? STRIDE for Evaluating the Evaluators

Authors: Lixing Tan, Qing Xia, Yuting Guo, Shuai Li, Aimin Hao

Organizations: Beihang University · Beijing Information Science and Technology University

Abstract

Motion-language models are typically scored by motion-language evaluators, but how well these evaluators ground language structure remains unclear. Here, we introduce the Structure grounding via Temporal-order, Reflection, and Identity Diagnostic Evaluation (STRIDE) benchmark to systematically evaluate the ability of evaluators to track temporal order, mirror reflection, and action identity. STRIDE comprises $5{,}869$ triples, each consisting of a motion, its original caption, and a perturbed caption, spanning both short and long descriptions. We likelihood-balance caption pairs to reduce text-only bias and estimate each evaluator's caption preference under unrelated motions to measure the discrimination gain from matched motions relative to this baseline. Our experiments reveal weak structural grounding and severe deficits in mirror sensitivity among the audited evaluators, which commonly used evaluation protocols fail to expose. To understand why these limitations go undetected in standard tests, we examine the evaluators more closely. We find that text-only priors alone can solve naive perturbation tests on existing datasets, while common retrieval and distributional metrics barely respond to structural corruption introduced by mirroring ground-truth motions. These findings suggest a natural intervention: structural hard negatives. Our experiments show that a simple modification to contrastive learning substantially improves performance on temporal order and mirror reflection. The benchmark and code will be released.

Figures & tables

Figure 1: Structural grounding failures of the HumanML3D evaluator. Each panel holds a fixed real motion and perturbs its caption along one axis: temporal order, mirror reflection, or action identity. In all three examples, the correct $-$ wrong score is negative, indicating the evaluator ranks the incorrect caption above the correct one.

Evaluation Setting	Content Matching	Temporal Order	Mirror Reflection	Action Identity	Matched Pairs	Prior Controls	Short Captions	Long Captions
HumanML3D suite ( Guo et al., 2022a )	✓	–	–	–	–	–	✓	–
TMR protocols ( Petrovich et al., 2023 )	✓	–	–	–	–	–	✓	–
CLaM ( Chen et al., 2024b )	✓	–	–	–	–	–	✓	–
MoBERT ( Voas et al., 2023 )	–	–	–	–	–	–	✓	–
ChronRet (CAR) ( Fujiwara et al., 2024 )	✓	✓	–	–	✓	–	✓	–
SnapMoGen suite ( Guo et al., 2025 )	✓	–	–	–	–	–	–	✓

Table 1: Comparison with existing motion-language evaluation protocols. STRIDE combines controlled tests of temporal order, mirror reflection, and action identity with text-only bias checks and evaluator-specific caption-preference controls across both short and long descriptions.

Figure 2: STRIDE probes three structural distinctions and measures each evaluator’s caption preference under an unrelated-motion control. Left: each dark motion sequence is paired with its original and edited captions; highlighted phrases mark the structural change, and lighter-to-darker shading encodes relative time. Right: green motion sequences match the caption, whereas grey sequences are unrelated to it and serve to estimate model preference. Each evaluator assesses motion-caption matching under the matched- and unrelated-motion conditions to obtain acc and bias, respectively.

Figure 3: Comparison of text-only score gaps between rule-based perturbations and STRIDE. Colors distinguish

\Delta\ell\ell

, Vera, and Grammar; open and filled boxes represent rule-based perturbations and STRIDE, respectively. Gaps are original minus perturbed scores, scaled by each scorer’s interquartile range (IQR) on the rule-based structural pairs; zero indicates equal scores. The two pools are not matched before/after samples.

Figure 4: Vocabulary and motion coverage. (a) Frequent words by axis. (b) Shared-feature t-SNE projection. (c) Smoothed corpus footprints. (d) Occupied cells (Occ.), effective cells (Eff.), and long-tail share (Tail) on a $55\times 55$ grid; bars are normalized per metric, with unscaled labels.

Figure 5: Comparison of caption preferences across models under unrelated motions. Cells report the rate of preferring the original over the perturbed caption (bias, %; ties count as half). White marks

50\%

; purple and teal indicate preferences for the original and perturbed captions, respectively.

HumanML3D (native input)
	Temporal Order			Mirror Reflection			Action Identity			Random Content			Macro GPR (%) $\uparrow$
Published evaluator	acc	bias	$\Delta\uparrow$	acc	bias	$\Delta\uparrow$	acc	bias	$\Delta\uparrow$	acc	bias	$\Delta\uparrow$
HumanML3D evaluator ( Guo et al., 2022a )	0.624	0.392	0.381	0.463	0.484	-0.041	0.858	0.500	0.717	0.940	0.503	0.879	66.4
TMR ( Petrovich et al., 2023 )	0.704	0.494	0.416	0.728	0.527	0.424	0.887	0.557	0.745	0.955	0.506	0.910	70.7
CLaM ( Chen et al., 2024b )	0.726	0.441	0.510	0.737	0.519	0.454	0.925	0.538	0.837	0.978	0.490	0.956	73.9
MoBERT ( Voas et al., 2023 )	0.726	0.495	0.457	0.594	0.501	0.186	0.844	0.533	0.667	0.937	0.476	0.880	70.3

Table 2: Performance comparison of existing motion-language evaluators on STRIDE. We report matched-motion accuracy (acc), original-caption preference under unrelated motions (bias), and normalized grounding gain

\Delta

; acc and bias are reported as proportions. Macro GPR (%) averages GPR over the three structural axes, while Random Content serves as a content-matching reference. Bold and underlined values denote the best and second-best

\Delta

and GPR within each dataset block, respectively. Appendix C details model inputs and evaluation settings.

Figure 6: Performance comparison of existing evaluators across perturbation axes. Bars report normalized grounding

\Delta

(higher is better) from Table 2 for the three structural axes and the Random Content reference. In each panel, the first five evaluators are tested on HumanML3D; the SnapMoGen evaluator, to the right of the vertical separator, is tested on SnapMoGen. Comparisons across the separator are not same-dataset rankings. The horizontal dashed and dotted lines mark

\Delta=0

(no gain over unrelated motion) and

\Delta=1

(perfect matched-motion accuracy), respectively.

Motion Condition	R@1 $\uparrow$	R@2 $\uparrow$	R@3 $\uparrow$	FID $\downarrow$	MM-Dist $\downarrow$	Diversity $\rightarrow$
Original Motion	0.529	0.710	0.812	0.000	2.914	9.272
Mirror Reflection	0.523	0.713	0.815	0.020	2.904	9.777
Time Reversal	0.357	0.523	0.632	8.906	4.729	8.327
Shuffled Pairing	0.032	0.066	0.098	0.000	9.516	9.349

Table 3: Comparison of standard motion-generation metrics under motion-side interventions. We compare mirroring, time reversal, and shuffled pairing against the original ground-truth reference. R@

K

denotes R-Precision at

K

, reported as a proportion; FID and MM-Dist denote Fréchet Inception Distance and motion-text matching distance, respectively.

Figure 7: Finetuning with structural hard negatives. Left: correct motion–caption pairs, ordinary negatives, and structural hard negatives. The added motion-side supervision compares only an original pair and its perturbed versions. Right: normalized grounding $\Delta$ on the three structural axes and the Random Content control for HumanML3D, TMR, and SnapMoGen, comparing the original evaluators (Base) with their hard-negative-trained variants (Neg Suite).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Benchmark composition and likelihood balancing. (a) Pair counts by axis and corpus. (b) Smoothed

\Delta\ell\ell

densities before and after control. Positive values indicate preference for the original caption; the shaded region marks the acceptance interval.

Published evaluator	Temporal Order $\uparrow$	Mirror Reflection $\uparrow$	Action Identity $\uparrow$	Macro $\uparrow$
HumanML3D (native input)
HumanML3D evaluator ( Guo et al., 2022a )	66.9	48.4	84.0	66.4
TMR ( Petrovich et al., 2023 )	65.5	66.6	80.2	70.7
CLaM ( Chen et al., 2024b )	68.1	64.0	89.6	73.9
MoBERT ( Voas et al., 2023 )	69.7	58.5	82.5	70.3
ChronRet ( Fujiwara et al., 2024 )	67.4	66.9	83.0	72.4

Appendix

Table 4: Per-axis GPR (%; chance

50\%

). Bold and underline mark the best and second-best values within each corpus.

\dagger

denotes fixed-adapter cross-domain transfer.

	Temporal Order			Mirror Reflection			Action Identity			Random Content			Macro GPR (%) $\uparrow$
M2T likelihood adapter	acc	bias	$\Delta\uparrow$	acc	bias	$\Delta\uparrow$	acc	bias	$\Delta\uparrow$	acc	bias	$\Delta\uparrow$
MotionGPT ( Jiang et al., 2023 )	0.902	0.894	0.081	0.730	0.514	0.445	0.764	0.679	0.265	0.621	0.488	0.261	62.9
MotionGPT3 ( Zhu et al., 2025 )	0.935	0.873	0.492	0.704	0.526	0.374	0.783	0.557	0.511	0.767	0.518	0.516	68.9
MG-MotionLLM ( Wu et al., 2025 )	0.982	0.960	0.541	0.762	0.514	0.510	0.868	0.557	0.702	0.747	0.514	0.480	69.7
IRG-MotionLLM (Stage-1) ( Li et al., 2026 )	0.953	0.955	-0.048	0.475	0.475	0.000	0.726	0.726	0.000	0.541	0.522	0.041	50.5

Appendix

Table 5: Adapted non-evaluator interfaces. (a) M2T models use length-normalized

\log p(c\mid m)

; (b) Qwen-VL uses six-keyframe forced choice. Metrics follow Table 2 ; Macro GPR averages three structural axes (null

50\%

). Panel (b) reports only acc for Random Content. In (a), bold marks best acc,

\Delta

, and GPR, and underline marks second-best

\Delta

and GPR. In (b), bold marks best acc and

\Delta

per corpus, ranked before rounding.

Model	acc $\uparrow$	bias	$\Delta\uparrow$	acc $\uparrow$	bias	$\Delta\uparrow$	acc $\uparrow$	bias	$\Delta\uparrow$	acc $\uparrow$	bias	$\Delta\uparrow$	MedR $\downarrow$
	Temporal Order			Mirror Reflection			Action Identity			Random Content			Retrieval
HumanML3D evaluator	0.624	0.392	0.381	0.463	0.484	-0.041	0.858	0.500	0.717	0.940	0.503	0.879	77
+NegHML (ours)	0.732	0.418	0.540	0.600	0.542	0.127	0.849	0.538	0.673	0.958	0.524	0.912	66
TMR	0.704	0.494	0.416	0.728	0.527	0.424	0.887	0.557	0.745	0.955	0.506	0.910	25
+NegTMR suite (ours)	0.760	0.483	0.536	0.738	0.537	0.435	0.887	0.547	0.750	0.953	0.511	0.903	24
SnapMoGen evaluator	0.965	0.518	0.927	0.728	0.519	0.435	0.838	0.529	0.656	0.998	0.500	0.995	1

Appendix

Table 6: Structural-hard-negative interventions. Metrics follow Table 2 . Bold marks better acc and

\Delta

within each base–finetuned pair; bias is not ranked. MedR is median retrieval rank (lower is better). All variants use caption negatives, only NegHML adds paired motion supervision, and

\ddagger

marks separate Action Identity checkpoints (NegTMR-A and NegSnap-A), with other NegTMR/NegSnap results from pooled checkpoints.

Figure 9: Temporal Order failures of the evaluators.

Figure 10: Mirror Reflection failures of the evaluators.

Figure 11: Action Identity failures of the evaluators.

Temporal Reasoning

Temporal reasoning studies how events, facts, and actions relate across time, including order, duration, change, persistence, and prediction.

See all 43 papers

NextMotionQA: Benchmarking and Judging Human Motion Understanding with Vision-Language Models
Jun 3, 2026Yong Cao, Chuqiao Li, Xianghui Xie +2VLM Evaluation Human Motion Understanding
University of Tübingen, Tübingen AI Center, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany
PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding?
Sep 2, 2025Mennatullah SiamVision-Language Grounding Video-Language Model Evaluation
MotionBlind: Probing the Illusion of Motion Understanding in Video-LLMs
Sep 8, 2026Dhairya Bhatia, Bishoy Galoaa, Oliver Fritsche +7Video-Language Model Evaluation Video-Language Models
Electrical and Computer Engineering Department Northeastern University, Boston, MA, USA

Jun 3, 2026J/K move · Enter open · S save

NextMotionQA: Benchmarking and Judging Human Motion Understanding with Vision-Language Models

Yong Cao, Chuqiao Li, Xianghui Xie, Gerard Pons-Moll+1

University of Tübingen, Tübingen AI Center, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany

VLM Evaluation Human Motion Understanding Temporal Video Understanding

Open paper AskPDF arXiv

Reliable evaluation of human motion understanding is fundamental to advancing embodied AI, robotics, and animation. However, existing benchmarks suffer from coarse semantic granularity, undifferentiated difficulty, limited annotation quality, and pervasive answer ambiguity, leaving them unable to diagnose where current models fail. To bridge this gap, we introduce NextMotionQA, a comprehensive benchmark that leverages vision-language models (VLMs) for semi-automated, expert-verified dataset. NextMotionQA features three complementary tasks: multiple-choice question answering, video captioning, and fine-grained error correction. Each task is systematically structured across three core semantic axes and stratified into three task complexity levels. Our extensive evaluation of twelve representative VLMs uncovers critical capability gaps and weakness that remain invisible under conventional, single-task evaluations. In a complementary direction, recent work has begun using VLMs as judges for text-to-motion evaluation; we ask whether they show the same degradation under harder tasks. We find that VLMs align strongly with expert ratings on coarse criteria (Cohen's κ=0.70) but break down on fine-grained, part-level judgment (κ=0.10), validating the paradigm in its strong regime while clarifying its limits.

Abstract

Temporal Reasoning

Explore similar work

NextMotionQA: Benchmarking and Judging Human Motion Understanding with Vision-Language Models

PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding?

MotionBlind: Probing the Illusion of Motion Understanding in Video-LLMs