Motion language models (MoLMs) encode 3D human motion as discrete tokens for language-based understanding. Deploying them on monocular video raises two problems. Pose detection yields 2D joint sequences, leaving a dimensional gap from the 3D motion used in training. Moreover, the 2D observation changes with the camera viewpoint, while the depth lost in projection makes the 2D pose ambiguous across viewpoints. We propose MoPort, a view-aware 2D interface that enables MoLMs to take 2D observations as input. We introduce Paired View Alignment (PVA), which gives two views of the same motion a shared 3D teacher target and directly constrains the consistency of their latents, so that 2D observations from different viewpoints are mapped to a consistent semantic representation. Camera-Modulated Encoding (CME) estimates the camera extrinsics from the 2D sequence and uses them to modulate encoding, providing the 2D latents with geometric information about the viewpoint. MoPort achieves R@1 of 0.5674 on HumanML3D and 0.3286 on KIT-ML, outperforming all evaluated baselines. On HumanML3D, its R@1 decreases only from 0.5763 on in-distribution viewpoints to 0.5585 on out-of-distribution viewpoints. We validate portability on three downstream MoLMs. Furthermore, on the real-world RGB video dataset Fit3D, MoPort achieves 80.36% accuracy. Code will be released upon acceptance.
Figures & tables
Figure 2: MoPort framework. (a) A shared MoPort encodes different 2D projections of one 3D motion and aligns them with latent and token targets from a pretrained 3D encoder. The view loss pulls the two latents together, and sampled cameras supervise the camera head. (b) The camera head estimates the extrinsics as a , and Gram-Schmidt gives R , which rotates the rays into the 3D reference frame. An MLP maps the standardized R6D representation c of R to multi-scale FiLM parameters. (c) At inference, YOLOv8x [ 11 ] and ViTPose-H [ 37 ] extract 2D tracks, which MoPort maps to the native tokens of a pretrained MoLM. Dashed arrows indicate training supervision.
Method
Input
Overall
OOD
ID
PA-MPJPE (mm) ↓
R@1 ↑
R@2 ↑
R@3 ↑
MM-D ↓
BERTScore ↑
CIDEr ↑
R@1 ↑
R@2 ↑
R@3 ↑
MM-D ↓
R@1 ↑
R@2 ↑
R@3 ↑
MM-D ↓
GT3D
Privileged
0.5924
0.7812
0.8643
2.5513
0.4227
0.6457
-
-
-
-
-
-
-
-
-
MotionBERT [ 46 ]
2D → 3D
0.0845
0.1499
0.2056
7.8712
0.1867
0.0738
0.0705
0.1267
0.1762
8.3220
0.0985
0.1732
0.2351
7.4203
162
MotionAGFormer [ 21 ]
2D → 3D
0.0529
0.0981
0.1399
8.7388
0.1636
0.0540
0.0448
0.0857
0.1256
9.0162
0.0609
0.1105
0.1542
8.4615
194
D3DP-P-Agg [ 26 ]
2D → 3D
0.0343
0.0657
0.0972
9.6378
0.1585
0.0455
0.0330
0.0639
0.0962
9.6884
0.0356
0.0674
0.0981
9.5872
171
WHAM [ 28 ]
RGB → 3D
0.2559
0.3844
0.4696
5.4994
0.2698
0.2305
0.1936
0.3007
0.3780
6.3281
0.3182
0.4682
0.5611
4.6707
93
Table 1: Comparison on HumanML3D. All routes use pretrained MG-MotionLLM. Overall averages ID and OOD views. - marks a non-applicable cell. In all tables, bold marks the best per-column value among non-privileged methods, per dataset.
Figure 3: MoPort describes the stair descent from all four views. Other routes miss the descent or change with the camera. Captions use native variable-length sequence processing. Each column corresponds to one of two ID or two OOD cameras. The top row shows the motion in temporal order from light to dark. MoPort and 2D-Interface receive 2D joints, GVHMR reconstructs 3D motion from RGB video, and MotionBERT lifts 2D joints to 3D; all routes use MG-MotionLLM. Amber marks correct stair-descent descriptions, and red marks incorrect actions.
Method
Input
R@1 ↑
R@2 ↑
R@3 ↑
MM-D ↓
BERTScore ↑
CIDEr ↑
GT3D
Privileged
0.3771
0.5896
0.7197
3.0579
0.5046
1.2407
MotionBERT [ 46 ]
2D → 3D
0.0892
0.1625
0.2231
8.5425
0.2644
0.2154
WHAM [ 28 ]
RGB → 3D
0.1612
0.2773
0.3641
7.0163
0.3520
0.4893
GVHMR [ 27 ]
RGB → 3D
0.1524
0.2540
0.3294
7.7564
0.3118
0.3650
2D-Interface [ 39 ]
2D joints
0.3145
0.4933
0.6070
4.3737
0.4644
0.9973
MoPort (Ours)
2D joints
0.3286
0.5100
0.6257
4.1658
0.4744
1.0328
Table 2: Supervised adaptation on KIT-ML. All routes share one MG-MotionLLM fine-tuned on KIT-ML and follow native variable-length sequence processing.
Figure 4: MoPort estimates extrinsics in ID and OOD views. Left: ID case. Right: OOD case. Green and amber frustums are the ground-truth and estimated cameras, overlapping in khaki. Body color darkens over time.
Variant
Overall
R@1 ↑
R@2 ↑
R@3 ↑
MM-D ↓
BERTScore ↑
CIDEr ↑
HumanML3D
2D-Interface (Baseline)
0.4537
0.6218
0.7071
3.7025
0.3152
0.3673
PVA-only
0.5045
0.7017
0.8085
3.4399
0.3574
0.5043
CME-only
0.5137
0.7101
0.8061
3.3554
0.3558
0.5021
MoPort
0.5674
0.7525
0.8378
2.7274
0.4091
0.6119
Table 3: Ablation study on HumanML3D and KIT-ML. Baseline results are reused from the two main comparison tables, Tables 1 and 2 . Within each dataset, all variants share the MG-MotionLLM of the corresponding main table, pretrained for HumanML3D and fine-tuned for KIT-ML.
Camera error ( ∘ )
Alignment
Caption performance
Camera rep.
ID ↓
OOD ↓
Token agree. ↑
Latent ℓ1↓
R@1 ↑
MM-D ↓
CIDEr ↑
16D concat.
11.73 ± 0.11
14.37 ± 0.10
0.3615 ± 0.0006
1.5063 ± 0.0014
0.5660 ± 0.0012
2.7371 ± 0.0083
0.6116 ± 0.0029
Quaternion
11.61 ± 0.12
13.73 ± 0.12
0.3634 ± 0.0005
1.5083 ± 0.0017
0.5653 ± 0.0016
2.7475 ± 0.0070
0.6088 ± 0.0036
R6D
11.47 ± 0.12
13.72 ± 0.10
0.3643 ± 0.0006
1.4990 ± 0.0015
0.5671 ± 0.0019
2.7283 ± 0.0091
0.6118 ± 0.0028
GT-camera oracle
0.00
0.00
0.3692 ± 0.0005
1.4885 ± 0.0015
0.5683 ± 0.0013
2.7158 ± 0.0073
0.6124 ± 0.0026
Table 4: CME variants with pretrained MG-MotionLLM. Mean ± std. over 20 runs with resampled test cameras; alignment and caption metrics average four views. Oracle: true extrinsics.
Figure 5: R@1 varies by less than one percentage point across hyperparameters. Each panel varies one setting of MoPort with pretrained MG-MotionLLM. Open circles mark the default.
Pretrained MoLM
Input
R@1 ↑
R@2 ↑
R@3 ↑
MM-D ↓
BERTScore ↑
CIDEr ↑
MotionGPT [ 10 ]
GT3D
0.5229
0.7096
0.7995
2.9831
0.3396
0.4167
MoPort
0.5129
0.6964
0.7859
3.0607
0.3354
0.4035
TM2T-512 [ 7 ]
GT3D
0.5279
0.7244
0.8204
2.8880
0.3394
0.2319
MoPort
0.5120
0.7049
0.8011
3.0331
0.3330
0.2282
MG-MotionLLM [ 36 ]
GT3D
0.5924
0.7812
0.8643
2.5513
0.4227
0.6457
MoPort
0.5674
0.7525
0.8378
2.7274
0.4091
0.6119
Table 5: Portability across pretrained downstream MoLMs on HumanML3D. Metrics average two ID and two OOD views; GT3D is scored once per motion.
Input route
Class accuracy (%) ↑
Template EM (%) ↑
GT3D
78.57
75.00
2D-Interface
71.43
69.64
MoPort
80.36
78.57
Table 6: Real-video adaptation on Fit3D. All routes share the same fine-tuned MG-MotionLLM. EM denotes exact match.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: MoPort captions of two Fit3D repetitions: a squat on the left and a lateral raise on the right. Amber marks correct actions. Dark amber marks an incorrectly added repetition. Skeletons are the ViTPose-H input trajectories.
Route
Model (ms)
Bridge + tok. (ms)
Total (ms) ↓
R@1 ↑
2D-Interface
1.9
-
1.9
0.4537
MoPort
3.5
-
3.5
0.5674
MotionBERT
13.3
15.5
28.8
0.0845
MotionAGFormer
47.4
14.5
61.9
0.0529
D3DP-P-Agg
442.4
14.8
457.2
0.0343
GVHMR
1814.3
23.7
1838.0
0.2419
Appendix
Table 7: Runtime after 2D detection. Mean time per clip at batch one over 100 HumanML3D test clips, from detected 2D input to MG-MotionLLM native tokens, next to the Overall R@1 of Table 1 , where the 2D and lifting routes take projected 2D joints. Interface models include 2D feature construction and native quantization; 3D routes add the 263D bridge and native tokenization. GVHMR’s model includes HMR2 image features. Shared 2D detection, 6.91 s per clip, is excluded.
τinfer
ID NLL ↓
OOD NLL ↓
Argmax hash
128
15.4270
18.1157
identical
256
8.1663
9.5230
identical
512
4.8994
5.5885
identical
1024
3.7478
4.0971
identical
Appendix
Table 8: Inference-temperature sensitivity. Temperature changes the soft target NLL but not the argmax tokens, so token and view agreement stay at 0.3755 and 0.4871.
Figure 7: Captions of a clapping motion from four viewpoints. Layout and colors follow Figure 3 .
Figure 8: Captions of a jumping-jack motion from four viewpoints. Layout and colors follow Figure 3 .
Figure 9: Failure cases of MoPort. Arm circles fail from the two OOD cameras, boxing is identified only from ID view 1, and going down to the ground is missed from every camera. For all three motions the pretrained MoLM captions the ground-truth 3D motion correctly. Each case includes the ground-truth caption, labeled GT caption, and the output of the pretrained MoLM from GT3D, labeled GT3D oracle. Layout and colors follow Figure 3 .
Figure 10: 3D reconstructions of one motion from four input views. The top row shows rendered RGB frames and 2D joints. Blue denotes the raw 17-joint skeletons predicted by lifting methods. Orange denotes neutral SMPL bodies fitted to the joints predicted by RGB methods. Translucent gray denotes the ground-truth body. Each column uses its corresponding input camera. Predictions are aligned to the ground truth in root position and global scale, preserving their orientation and pose. Numbers are PA-MPJPE for each view.