Motion Language Models (MoLMs) typically understand human motions by tokenizing 3D motion and processing the resulting tokens using a language model. However, obtaining accurate 3D motions from monocular videos is challenging, limiting their real-world applicability. To address this issue, we introduce a plug-and-play 2D Motion Interface that enables 3D-pretrained MoLMs to accept 2D motion inputs without modifying or fine-tuning the original models. Experiments on public datasets show that our method achieves performance comparable to 3D motion inputs across multiple MoLMs and outperforms training MoLMs from scratch on 2D motions. We further construct a monocular real-world video motion evaluation dataset and introduce a real-video adapter, demonstrating the usefulness of 2D motions over 3D motions under the evaluated monocular pose-estimation setting. These results suggest that 2D motion provides a practical interface for deploying MoLMs in real-world motion understanding settings. Code is available at https://github.com/irajisamurai/2D-Motion-Interface.
Figures & tables
Figure 1: While existing MoLMs typically take 3D motions as input, the proposed 2D Motion Interface (i.e., 2D motion encoder) enables existing MoLMs based on VQ-VAE motion tokenization to accept 2D motion inputs. Since 2D pose estimation is generally easier than 3D pose estimation, our approach improves the deployability of MoLMs in real-world applications.
Figure 2: Overview of the proposed pipeline. During training, the pretrained MoLM remains frozen and only the 2D motion encoder is trained. Specifically, the encoder is trained using an L1 feature-matching loss to align 2D motion features with the continuous latent space of the VQ-VAE encoder. During inference, the resulting 2D motion features are quantized into discrete motion tokens using the VQ-VAE codebook and fed into the pretrained language model.
Model
Method
R-Prec ↑
MM-Dist ↓
BLEU-1 ↑
BLEU-4 ↑
ROUGE-L ↑
CIDEr ↑
BERTScore ↑
Avg Drop vs 3D ↑
Top-1
Top-2
Top-3
TM2T
3D Input
0.488
0.689
0.786
3.137
0.615
0.233
0.492
0.691
0.368
-
2D Scratch
0.279
0.447
0.562
4.921
0.550
0.175
0.437
0.499
0.312
−25.8%
Ours
0.485 ± 0.002
0.683 ± 0.003
0.788 ± 0.004
3.129 ± 0.005
0.616 ± 0.001
0.232 ± 0.000
0.492 ± 0.000
0.688 ± 0.004
0.368 ± 0.001
−0.2%
MotionGPT
3D Input
0.516
0.707
0.803
2.994
0.429
0.062
0.345
0.079
0.316
-
Ours
0.523 ± 0.002
0.717 ± 0.003
0.802 ± 0.003
2.985 ± 0.015
0.431 ± 0.000
0.063 ± 0.001
0.346 ± 0.001
0.077 ± 0.001
0.319 ± 0.001
+0.4%
Table 1: Comparison of motion captioning results on HumanML3D test set. The best and second-best results within each model are highlighted in bold and underlined, respectively. Avg Drop vs 3D denotes the average percentage difference from the 3D Input across all metrics. The MotionGPT 2D Scratch result is omitted due to reproducibility issues; see the Appendix for details.
Level
Method
BLEU-1 ↑
BLEU-4 ↑
BLEU-7 ↑
ROUGE-L ↑
BERTScore ↑
Avg Drop vs 3D ↑
Sequence
3D Input
0.828
0.667
0.536
0.651
0.523
-
2D Scratch
0.814
0.637
0.497
0.629
0.483
−4.9%
Ours
0.828 ± 0.001
0.665 ± 0.001
0.533 ± 0.001
0.648 ± 0.001
0.517 ± 0.001
−0.5%
Snippet
3D Input
0.672
0.478
0.354
0.606
0.509
-
2D Scratch
0.653
0.456
0.329
0.588
0.487
−4.4%
Ours
0.666 ± 0.000
0.473 ± 0.001
0.349 ± 0.001
0.601 ± 0.001
0.502 ± 0.001
−1.1%
Table 2: Comparison of motion-to-detailed-text results on the FineMotion test set.
Figure 5
Figure 4: Training pipeline of the real-video adapter. We render 3D motions from random viewpoints and apply 2D pose estimation to the rendered videos to obtain estimated 2D motions XPE2D and joint confidence scores Conf . During training, the adapter Areal is inserted before the 2D motion encoder, and only Areal is optimized using the same alignment loss as in Sec. 3.3 .
Model
Method
BLEU-1 ↑
BLEU-4 ↑
ROUGE-L ↑
CIDEr ↑
BERTScore ↑
Avg Drop vs Ref 3D ↑
TM2T
Ref 3D Input
0.599
0.190
0.460
0.609
0.354
-
3D Input (TRACE)
0.453
0.102
0.360
0.252
0.222
−37.7%
3D Input (WHAM)
0.474
0.092
0.380
0.258
0.241
−35.9%
2D Input
0.554
0.147
0.419
0.403
0.285
−18.5%
3D Input (WHAM) w/ A3D
0.588
0.166
0.436
0.486
0.304
−10.8%
2D Input w/ Areal
0.619
0.218
0.461
0.645
0.344
+4.3%
Table 4: Comparison of motion captioning results on monocular real-world video motion dataset.
Figure 5: Qualitative results on monocular real-world video motion dataset. For reconstruction, we compare two inputs: the estimated 2D motion used as input (2D Input), and the clean 2D motion constructed from the corresponding Reference 3D motion following the same procedure as in Sec. 3.2 (Clean 2D Input).
Figure S1 : Qualitative results of motion captioning on HumanML3D test set.
Figure S2 : Qualitative results of motion-to-detailed text on FineMotion test set.
Method
MPJPE ↓
PAMPJPE ↓
FID ↓
Div ↑
ACCEL ↓
3DInput
57.5
41.4
0.083
9.32
8.19
Ours (2D Input)
74.0
43.1
0.110
9.84
8.32
Table S1 : Comparison of 3D reconstruction ability using MotionGPT’s VQ-VAE. The best and second-best results are highlighted in bold and underlined, respectively.
Figure S3 : Qualitative results of viewpoint robustness. The same 3D motion is observed from two viewpoints, while the generated captions remain consistent.
Figure S4 : Qualitative comparison with and without the adapter on monocular real videos.
Architecture
BLEU-1 ↑
BLEU-4 ↑
ROUGE-L ↑
CIDEr ↑
BERTScore ↑
Linear
0.413
0.057
0.362
0.105
0.294
Temporal Conv
0.399
0.053
0.378
0.110
0.299
Ours (MLP)
0.435
0.045
0.367
0.105
0.306
Table S2 : Comparison of adapter architectures using MotionGPT.