Motion language models (MoLMs) encode 3D human motion as discrete tokens for language-based understanding. Deploying them on monocular video raises two problems. Pose detection yields 2D joint sequences, leaving a dimensional gap from the 3D motion used in training. Moreover, the 2D observation changes with the camera viewpoint, while the depth lost in projection makes the 2D pose ambiguous across viewpoints. We propose MoPort, a view-aware 2D interface that enables MoLMs to take 2D observations as input. We introduce Paired View Alignment (PVA), which gives two views of the same motion a shared 3D teacher target and directly constrains the consistency of their latents, so that 2D observations from different viewpoints are mapped to a consistent semantic representation. Camera-Modulated Encoding (CME) estimates the camera extrinsics from the 2D sequence and uses them to modulate encoding, providing the 2D latents with geometric information about the viewpoint. MoPort achieves R@1 of 0.5674 on HumanML3D and 0.3286 on KIT-ML, outperforming all evaluated baselines. On HumanML3D, its R@1 decreases only from 0.5763 on in-distribution viewpoints to 0.5585 on out-of-distribution viewpoints. We validate portability on three downstream MoLMs. Furthermore, on the real-world RGB video dataset Fit3D, MoPort achieves 80.36% accuracy. Code will be released upon acceptance.
Figures & tables
Figure 2: MoPort framework. (a) A shared MoPort encodes different 2D projections of one 3D motion and aligns them with latent and token targets from a pretrained 3D encoder. The view loss pulls the two latents together, and sampled cameras supervise the camera head. (b) The camera head estimates the extrinsics as a , and Gram-Schmidt gives R , which rotates the rays into the 3D reference frame. An MLP maps the standardized R6D representation c of R to multi-scale FiLM parameters. (c) At inference, YOLOv8x [ 11 ] and ViTPose-H [ 37 ] extract 2D tracks, which MoPort maps to the native tokens of a pretrained MoLM. Dashed arrows indicate training supervision.
Method
Input
Overall
OOD
ID
PA-MPJPE (mm) ↓
R@1 ↑
R@2 ↑
R@3 ↑
MM-D ↓
BERTScore ↑
CIDEr ↑
R@1 ↑
R@2 ↑
R@3 ↑
MM-D ↓
R@1 ↑
R@2 ↑
R@3 ↑
MM-D ↓
GT3D
Privileged
0.5924
0.7812
0.8643
2.5513
0.4227
0.6457
-
-
-
-
-
-
-
-
-
MotionBERT [ 46 ]
2D → 3D
0.0845
0.1499
0.2056
7.8712
0.1867
0.0738
0.0705
0.1267
0.1762
8.3220
0.0985
0.1732
0.2351
7.4203
162
MotionAGFormer [ 21 ]
2D → 3D
0.0529
0.0981
0.1399
8.7388
0.1636
0.0540
0.0448
0.0857
0.1256
9.0162
0.0609
0.1105
0.1542
8.4615
194
D3DP-P-Agg [ 26 ]
2D → 3D
0.0343
0.0657
0.0972
9.6378
0.1585
0.0455
0.0330
0.0639
0.0962
9.6884
0.0356
0.0674
0.0981
9.5872
171
WHAM [ 28 ]
RGB → 3D
0.2559
0.3844
0.4696
5.4994
0.2698
0.2305
0.1936
0.3007
0.3780
6.3281
0.3182
0.4682
0.5611
4.6707
93
Table 1: Comparison on HumanML3D. All routes use pretrained MG-MotionLLM. Overall averages ID and OOD views. - marks a non-applicable cell. In all tables, bold marks the best per-column value among non-privileged methods, per dataset.
Figure 3: MoPort describes the stair descent from all four views. Other routes miss the descent or change with the camera. Captions use native variable-length sequence processing. Each column corresponds to one of two ID or two OOD cameras. The top row shows the motion in temporal order from light to dark. MoPort and 2D-Interface receive 2D joints, GVHMR reconstructs 3D motion from RGB video, and MotionBERT lifts 2D joints to 3D; all routes use MG-MotionLLM. Amber marks correct stair-descent descriptions, and red marks incorrect actions.
Method
Input
R@1 ↑
R@2 ↑
R@3 ↑
MM-D ↓
BERTScore ↑
CIDEr ↑
GT3D
Privileged
0.3771
0.5896
0.7197
3.0579
0.5046
1.2407
MotionBERT [ 46 ]
2D → 3D
0.0892
0.1625
0.2231
8.5425
0.2644
0.2154
WHAM [ 28 ]
RGB → 3D
0.1612
0.2773
0.3641
7.0163
0.3520
0.4893
GVHMR [ 27 ]
RGB → 3D
0.1524
0.2540
0.3294
7.7564
0.3118
0.3650
2D-Interface [ 39 ]
2D joints
0.3145
0.4933
0.6070
4.3737
0.4644
0.9973
MoPort (Ours)
2D joints
0.3286
0.5100
0.6257
4.1658
0.4744
1.0328
Table 2: Supervised adaptation on KIT-ML. All routes share one MG-MotionLLM fine-tuned on KIT-ML and follow native variable-length sequence processing.
Figure 4: MoPort estimates extrinsics in ID and OOD views. Left: ID case. Right: OOD case. Green and amber frustums are the ground-truth and estimated cameras, overlapping in khaki. Body color darkens over time.
Variant
Overall
R@1 ↑
R@2 ↑
R@3 ↑
MM-D ↓
BERTScore ↑
CIDEr ↑
HumanML3D
2D-Interface (Baseline)
0.4537
0.6218
0.7071
3.7025
0.3152
0.3673
PVA-only
0.5045
0.7017
0.8085
3.4399
0.3574
0.5043
CME-only
0.5137
0.7101
0.8061
3.3554
0.3558
0.5021
MoPort
0.5674
0.7525
0.8378
2.7274
0.4091
0.6119
Table 3: Ablation study on HumanML3D and KIT-ML. Baseline results are reused from the two main comparison tables, Tables 1 and 2 . Within each dataset, all variants share the MG-MotionLLM of the corresponding main table, pretrained for HumanML3D and fine-tuned for KIT-ML.
Camera error ( ∘ )
Alignment
Caption performance
Camera rep.
ID ↓
OOD ↓
Token agree. ↑
Latent ℓ1↓
R@1 ↑
MM-D ↓
CIDEr ↑
16D concat.
11.73 ± 0.11
14.37 ± 0.10
0.3615 ± 0.0006
1.5063 ± 0.0014
0.5660 ± 0.0012
2.7371 ± 0.0083
0.6116 ± 0.0029
Quaternion
11.61 ± 0.12
13.73 ± 0.12
0.3634 ± 0.0005
1.5083 ± 0.0017
0.5653 ± 0.0016
2.7475 ± 0.0070
0.6088 ± 0.0036
R6D
11.47 ± 0.12
13.72 ± 0.10
0.3643 ± 0.0006
1.4990 ± 0.0015
0.5671 ± 0.0019
2.7283 ± 0.0091
0.6118 ± 0.0028
GT-camera oracle
0.00
0.00
0.3692 ± 0.0005
1.4885 ± 0.0015
0.5683 ± 0.0013
2.7158 ± 0.0073
0.6124 ± 0.0026
Table 4: CME variants with pretrained MG-MotionLLM. Mean ± std. over 20 runs with resampled test cameras; alignment and caption metrics average four views. Oracle: true extrinsics.
Figure 5: R@1 varies by less than one percentage point across hyperparameters. Each panel varies one setting of MoPort with pretrained MG-MotionLLM. Open circles mark the default.
Pretrained MoLM
Input
R@1 ↑
R@2 ↑
R@3 ↑
MM-D ↓
BERTScore ↑
CIDEr ↑
MotionGPT [ 10 ]
GT3D
0.5229
0.7096
0.7995
2.9831
0.3396
0.4167
MoPort
0.5129
0.6964
0.7859
3.0607
0.3354
0.4035
TM2T-512 [ 7 ]
GT3D
0.5279
0.7244
0.8204
2.8880
0.3394
0.2319
MoPort
0.5120
0.7049
0.8011
3.0331
0.3330
0.2282
MG-MotionLLM [ 36 ]
GT3D
0.5924
0.7812
0.8643
2.5513
0.4227
0.6457
MoPort
0.5674
0.7525
0.8378
2.7274
0.4091
0.6119
Table 5: Portability across pretrained downstream MoLMs on HumanML3D. Metrics average two ID and two OOD views; GT3D is scored once per motion.
Input route
Class accuracy (%) ↑
Template EM (%) ↑
GT3D
78.57
75.00
2D-Interface
71.43
69.64
MoPort
80.36
78.57
Table 6: Real-video adaptation on Fit3D. All routes share the same fine-tuned MG-MotionLLM. EM denotes exact match.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: MoPort captions of two Fit3D repetitions: a squat on the left and a lateral raise on the right. Amber marks correct actions. Dark amber marks an incorrectly added repetition. Skeletons are the ViTPose-H input trajectories.
Route
Model (ms)
Bridge + tok. (ms)
Total (ms) ↓
R@1 ↑
2D-Interface
1.9
-
1.9
0.4537
MoPort
3.5
-
3.5
0.5674
MotionBERT
13.3
15.5
28.8
0.0845
MotionAGFormer
47.4
14.5
61.9
0.0529
D3DP-P-Agg
442.4
14.8
457.2
0.0343
GVHMR
1814.3
23.7
1838.0
0.2419
Appendix
Table 7: Runtime after 2D detection. Mean time per clip at batch one over 100 HumanML3D test clips, from detected 2D input to MG-MotionLLM native tokens, next to the Overall R@1 of Table 1 , where the 2D and lifting routes take projected 2D joints. Interface models include 2D feature construction and native quantization; 3D routes add the 263D bridge and native tokenization. GVHMR’s model includes HMR2 image features. Shared 2D detection, 6.91 s per clip, is excluded.
τinfer
ID NLL ↓
OOD NLL ↓
Argmax hash
128
15.4270
18.1157
identical
256
8.1663
9.5230
identical
512
4.8994
5.5885
identical
1024
3.7478
4.0971
identical
Appendix
Table 8: Inference-temperature sensitivity. Temperature changes the soft target NLL but not the argmax tokens, so token and view agreement stay at 0.3755 and 0.4871.
Figure 7: Captions of a clapping motion from four viewpoints. Layout and colors follow Figure 3 .
Figure 8: Captions of a jumping-jack motion from four viewpoints. Layout and colors follow Figure 3 .
Figure 9: Failure cases of MoPort. Arm circles fail from the two OOD cameras, boxing is identified only from ID view 1, and going down to the ground is missed from every camera. For all three motions the pretrained MoLM captions the ground-truth 3D motion correctly. Each case includes the ground-truth caption, labeled GT caption, and the output of the pretrained MoLM from GT3D, labeled GT3D oracle. Layout and colors follow Figure 3 .
Figure 10: 3D reconstructions of one motion from four input views. The top row shows rendered RGB frames and 2D joints. Blue denotes the raw 17-joint skeletons predicted by lifting methods. Orange denotes neutral SMPL bodies fitted to the joints predicted by RGB methods. Translucent gray denotes the ground-truth body. Each column uses its corresponding input camera. Predictions are aligned to the ground truth in root position and global scale, preserving their orientation and pose. Numbers are PA-MPJPE for each view.
Motion Language Models (MoLMs) typically understand human motions by tokenizing 3D motion and processing the resulting tokens using a language model. However, obtaining accurate 3D motions from monocular videos is challenging, limiting their real-world applicability. To address this issue, we introduce a plug-and-play 2D Motion Interface that enables 3D-pretrained MoLMs to accept 2D motion inputs without modifying or fine-tuning the original models. Experiments on public datasets show that our method achieves performance comparable to 3D motion inputs across multiple MoLMs and outperforms training MoLMs from scratch on 2D motions. We further construct a monocular real-world video motion evaluation dataset and introduce a real-video adapter, demonstrating the usefulness of 2D motions over 3D motions under the evaluated monocular pose-estimation setting. These results suggest that 2D motion provides a practical interface for deploying MoLMs in real-world motion understanding settings. Code is available at https://github.com/irajisamurai/2D-Motion-Interface.
Kaname Yokoyama, Norimichi Ukita
Toyota Technological Institute, Nagoya, Aichi, Japan
3D human pose estimation is a key enabling technology for applications such as healthcare monitoring, human-robot collaboration, and immersive gaming, but real-world deployment remains challenged by viewpoint variations. Existing methods struggle to generalize to unseen camera viewpoints, require large amounts of training data, and suffer from high inference latency. We propose MoViD, a viewpoint-invariant 3D human pose estimation framework that disentangles viewpoint information from motion features. The key idea is to extract viewpoint information from intermediate pose features and leverage it to enhance both the robustness and efficiency of pose estimation. MoViD introduces a view estimator that models key joint relationships to predict viewpoint information, and an orthogonal projection module to disentangle motion and view features, further enhanced through physics-grounded contrastive alignment across views. For real-time edge deployment, MoViD employs a frame-by-frame inference pipeline with a view-aware strategy that adaptively activates flip refinement based on the estimated viewpoint. Evaluations on nine public datasets and newly collected multiview UAV and gait analysis datasets show that MoViD reduces pose estimation error by over 24.2% compared to state-of-the-art methods, maintains robust performance under severe occlusions with 60% less training data, and achieves real-time inference at 15 FPS on NVIDIA edge devices.
Yejia Liu, Hengle Jiang, Haoxian Liu +2
Department of Computer Science and Engineering, Hong Kong University of Science and Technology
Fine-grained motion understanding requires knowing not only which poses occur, but also when they occur. Existing motion-language models either compress motion clips or concatenate pose features without retaining the timing of each pose, making action duration and temporal scale ambiguous under temporal subsampling. We introduce FiGMo, an LLM-based model that represents motion as timestamp-grounded skeletal poses, supporting reasoning over action order and duration. A unified pose encoder applies this representation to both 2D and 3D skeletons, including poses detected from real videos, while RGB can optionally provide contextual information. To reason over timestamped pose sequences, the model must understand both individual body configurations and how they change over time. As existing data provides limited pose-level grounding, we construct the required pose-to-motion supervision. Controlled experiments show that timestamps improve temporal localization and preserve physical timing under subsampling, while the constructed supervision drives the main performance gains. FiGMo achieves state-of-the-art results across motion question answering, dense motion captioning, and action recognition with detected poses. Our code is available at: https://github.com/Thomas-Markhorst/FiGMo
Thomas Markhorst, Zhi-Yi Lin, Jouh Yeong Chew +2
Delft University of Technology · Honda Research Institute Japan