BAL: Bidirectional Autoregression in Latent Space for Learning Human Movement Representations
Organizations: Graduate School of Informatics, Kyoto University
Abstract
Learning semantically meaningful human movement representations from 3D pose sequences is essential for human behavior modeling. Existing self-supervised methods derive supervision from artificial assumptions, e.g., invariance to augmentation within a certain range or recoverability of masked regions at a certain temporal scale, introducing heuristic hyperparameters that make the learned semantics sensitive to their settings. We propose BAL, a self-supervised framework that instead exploits supervision intrinsic to human movement: the natural ordering structure of everyday behavior. BAL autoregressively predicts in a semantically abstract latent space, analogous to how humans anticipate others' actions rather than their precise joint configurations. Bidirectional autoregressive prediction, both forward and backward in time, fully leverages these ordering constraints while enabling representation extraction using both past and future context at inference. To suppress representation collapse, BAL decomposes each prediction target into a global component capturing sequence-level semantics and a local component capturing context-independent movement. We prove that the local targets retain nontrivial angular diversity unless the representations are completely collapsed, preserving a meaningful training signal. Experiments on large-scale datasets demonstrate that BAL's representations are highly effective for motion captioning, forecasting, and interpolation, outperforming baselines on most evaluation metrics.
Figures & tables
| Method | Objective | Captioning | Forecasting (mm) | Interpolation (mm) | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BLEU-1 Papineni et al. (2002) | R-L Lin (2004) | CIDEr Vedantam et al. (2015) | BERT Zhang et al. (2020) | ||||||||||||
| Nym | E3D | Nym | E3D | Nym | E3D | Nym | E3D | AMD | Nym | E3D | AMD | Nym | E3D | ||
| A4Mer Kinoshita et al. (2026) | JEPA | 15.7 | 32.9 | 20.5 | 22.1 | 4.3 | 4.3 | 8.7 | 10.2 | 641.6 | 359.0 | 381.3 | 163.7 | 145.1 | 86.3 |
| PUMPS Mo et al. (2025) | Reconstruct | 8.2 | 58.7 | 17.0 | 44.8 | 16.2 | 17.3 | 9.5 | 34.3 | 186.4 | 453.6 | 111.7 | 195.6 | 472.4 | 89.2 |
| USDRL Weng et al. (2025) | Contrastive | 27.1 | 58.0 | 34.8 | 45.3 | 32.5 | 18.4 | 17.7 | 35.0 | 136.1 | 373.8 | 90.2 | 117.7 | 424.6 | 59.2 |
| BehaveMAE Stoffl et al. (2024) | MAE | 33.9 | 58.8 | 38.3 | 44.6 | 49.4 | 17.4 | 24.6 | 33.6 | 195.5 | 394.8 | 152.7 | 135.6 | 103.5 | 77.8 |
| Key | Substitute | Captioning | Forecasting (mm) | Interpolation (mm) | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| BLEU-1 | R-L | CIDEr | BERT | AMD | Nymeria | E3D | AMD | Nymeria | E3D | ||
| Latent AR | Pose-space Bi-AR | 17.6 | 18.9 | 12.6 | 7.3 | 118.6 | 96.6 | 64.7 | 125.7 | 87.4 | 60.1 |
| JEPA, mask 15% | 34.5 | 39.0 | 46.2 | 25.2 | 109.3 | 107.9 | 91.6 | 145.8 | 112.4 | 83.4 | |
| JEPA, mask 50% | 32.0 | 38.4 | 41.7 | 24.9 | 102.7 | 99.0 | 86.7 | 133.5 | 96.8 | 82.1 | |
| Decompose targets | w/o decomp. | 33.1 | 37.9 | 37.3 | 22.2 | 95.5 | 91.7 | 75.0 | 121.6 | 87.3 | 64.1 |
| SIGReg Balestriero and LeCun (2025) ; Maes et al. (2026) | 22.7 | 33.0 | 13.0 | 17.9 | 297.9 | 267.7 | 148.9 | 155.3 | 117.2 | 82.5 | |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Module | Token Dim. | #Layers | Component |
| BAL | Encoder | 256 | 8 | Transformer blocks |
| LatentFormer | 256 | 8 | Transformer blocks | |
| Predictor | 256 | 3 | Transformer blocks | |
| Predictor head | 256 | 2 | LN + linear layers | |
| Decoder | Decoder model | 256 | 4 | Transformer blocks |
| Head | 256 | 1 | LN + linear layer |
| Normalization | Distance | Captioning | Forecasting (mm) | Interpolation (mm) | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BLEU-1 | ROUGE-L | CIDEr | BERTScore | ||||||||||||
| Nym | E3D | Nym | E3D | Nym | E3D | Nym | E3D | AMD | Nym | E3D | AMD | Nym | E3D | ||
| L2 norm | Cosine | 28.9 | 58.3 | 39.4 | 44.7 | 50.8 | 17.1 | 27.2 | 33.9 | 96.7 | 84.6 | 68.8 | 121.8 | 81.7 | 63.1 |
| Layer norm | Smooth L1 | 37.3 | 58.9 | 41.2 | 45.3 | 63.1 | 18.6 | 29.6 | 35.0 | 90.5 | 82.1 | 63.8 | 117.5 | 77.0 | 59.2 |
| Task | Batch size | Learning rate | Weight decay | Iterations | Warmup iter. | Training time (h) |
|---|---|---|---|---|---|---|
| BAL | 128 | 429K | 2000 | 39.5 | ||
| Decoder | 128 | 0.01 | 143K | 2000 | 4.9 | |
| Forecasting | 128 | 0.01 | 286K | 2000 | 9.6 | |
| Interpolation | 128 | 0.01 | 143K | 2000 | 21.5 | |
| Captioning | 64 | 0.0 | 2.6K | 1000 | 2.3 |
| Method | Captioning | Forecasting (mm) | Interpolation (mm) | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BLEU-1 | ROUGE-L | CIDEr | BERTScore | |||||||||||
| Nym | E3D | Nym | E3D | Nym | E3D | Nym | E3D | AMD | Nym | E3D | AMD | Nym | E3D | |
| XLNet | 35.2 | 57.9 | 38.6 | 43.5 | 42.9 | 16.3 | 26.7 | 32.3 | 85.8 | 87.1 | 65.6 | 116.5 | 79.3 | 58.1 |
| BAL | 37.3 | 58.9 | 41.2 | 45.3 | 63.1 | 18.6 | 29.6 | 35.0 | 90.5 | 82.1 | 63.8 | 117.5 | 77.0 | 59.2 |
| Method | Forecasting (mm) | Interpolation (mm) | ||||
|---|---|---|---|---|---|---|
| AMD | Nym | E3D | AMD | Nym | E3D | |
| A4Mer | 633.9 | 285.1 | 375.5 | 163.7 | 120.8 | 86.3 |
| PUMPS | 163.2 | 130.3 | 102.2 | 182.6 | 154.3 | 87.0 |
| USDRL | 133.3 | 115.9 | 87.9 | 117.7 | 106.7 | 59.2 |
| BehaveMAE | 170.2 | 169.1 | 126.1 | 135.6 | 103.4 | 77.8 |
| MotionBERT | 282.4 | 244.1 | 162.0 | 135.2 | 129.8 | 69.2 |
| Method | Chunk length | Captioning | Forecasting (mm) | Interpolation (mm) | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BLEU-1 | ROUGE-L | CIDEr | BERTScore | ||||||||||||
| Nym | E3D | Nym | E3D | Nym | E3D | Nym | E3D | AMD | Nym | E3D | AMD | Nym | E3D | ||
| BAL | 2 sec | 37.4 | 58.7 | 41.0 | 45.0 | 62.6 | 16.9 | 28.0 | 34.2 | 89.1 | 72.2 | 61.4 | 110.6 | 70.3 | 55.0 |
| BAL | 3 sec | 33.4 | 58.9 | 40.2 | 45.1 | 63.6 | 17.7 | 28.7 | 34.6 | 95.1 | 82.2 | 63.4 | 111.3 | 75.2 | 56.3 |
| BAL | 4 sec | 37.3 | 58.9 | 41.2 | 45.3 | 63.1 | 18.6 | 29.6 | 35.0 | 90.5 | 82.1 | 63.8 | 117.5 | 77.0 | 59.2 |
| BAL | 5 sec | 41.4 | 59.1 | 41.7 | 45.2 | 71.8 | 18.3 | 31.6 | 34.5 | 115.3 | 87.5 | 66.4 | 118.3 | 80.7 | 59.6 |
| Method | #Params [M] | Latency [ms] |
|---|---|---|
| A4Mer | 9.64 | 16.0 |
| PUMPS | 53.01 | 8.1 |
| USDRL | 95.15 | 9.2 |
| BehaveMAE | 5.81 | 4.3 |
| MotionBERT | 42.42 | 21.4 |
| MacDiff | 11.96 | 5.3 |
| Method | #Params [M] | Captioning | Forecasting (mm) | Interpolation (mm) | |||||||||||
| BLEU-1 | ROUGE-L | CIDEr | BERTScore | ||||||||||||
| Nym | E3D | Nym | E3D | Nym | E3D | Nym | E3D | AMD | Nym | E3D | AMD | Nym | E3D | ||
| A4Mer | 14.66 | 6.4 | 54.2 | 7.7 | 40.7 | 2.1 | 12.4 | 3.3 | 27.1 | 235.2 | 313.4 | 136.0 | 135.8 | 109.4 | 73.9 |
| BehaveMAE | 14.19 | 30.9 | 59.1 | 37.8 | 44.6 | 47.9 | 17.8 | 26.0 | 34.1 | 171.0 | 139.0 | 116.4 | 139.1 | 98.3 | 76.9 |
| MacDiff | 14.33 | 39.0 | 59.1 | 41.3 | 45.0 | 62.4 | 18.5 | 28.7 | 34.1 | 142.9 | 367.1 | 84.5 | 118.3 | 404.1 | 58.1 |
| BAL | 14.18 | 37.3 | 58.9 | 41.2 | 45.3 | 63.1 | 18.6 | 29.6 | 35.0 | 90.5 | 82.1 | 63.8 | 117.5 | 77.0 | 59.2 |
Motion Representation Learning
Motion representation learning develops structured encodings of movement for recognition, prediction, generation, and control.
See all 181 papers