CISyn: Human Mesh Recovery in Close Interactions with Physics-Simulated Synthetic Data
Organizations: VGG, University of Oxford · The Podium Institute for Sports Medicine and Technology, University of Oxford
Abstract
3D human mesh recovery (HMR) in close-interaction scenarios is hindered by the lack of large-scale, accurately annotated training data: real capture is expensive, and existing synthetic datasets rarely include physical contact. We introduce CISyn, a large-scale synthetic dataset purpose-built for close human-human interactions, generated via physics simulation and Blender-based rendering. By simulating collision dynamics between SMPL-X-rigged humanoids, CISyn naturally produces the contact, mutual occlusion, and pose entanglement that characterize close-interaction scenes, with no manual annotation effort. The released set provides 25,000 sequences with accurate 3D annotations. We further propose an uncertainty-aware loss that allows models to learn more effectively from the heavily occluded examples in our data. Augmenting state-of-the-art HMR training with CISyn and replacing the regression loss with our uncertainty-aware variant yields consistent improvements on close-interaction benchmarks, validating CISyn as a scalable complement to real-world data. The dataset, generation pipeline, and code are available at https://github.com/yfde/CISyn.
Figures & tables
| Dataset | #Seq | #Frm | Syn. | SMPL | CC | CP |
|---|---|---|---|---|---|---|
| CMU Panoptic [ 19 ] | 65 | 1.5M | ✓ | |||
| MuPoTS-3D [ 36 ] | 20 | 8K | ✓ | |||
| ExPI [ 16 ] | 115 | 30K | ✓ | ✓ | ||
| 3DPW [ 54 ] | 60 | 51K | ✓ | |||
| 4D-Humans [ 15 ] | - | 3.2M | ✓ | |||
| CameraHMR [ 39 ] | - | 3.2M | ✓ |
| Method | Hi4D | CHI3D | Harmony4D | ||||||
|---|---|---|---|---|---|---|---|---|---|
| MPJPE | PA-MPJPE | PVE | MPJPE | PA-MPJPE | PVE | MPJPE | PA-MPJPE | PVE | |
| Multi-HMR [ 3 ] | 86.7 | 52.8 | 106.2 | 63.0 | 35.7 | 76.2 | 117.9 | 65.2 | 135.3 |
| PromptHMR [ 56 ] | 63.9 | 39.2 | – | 48.0 | 27.2 | – | – | – | – |
| MA-HMR [ 55 ] | 61.2 | 45.9 | 77.2 | 47.2 | 31.4 | 55.7 | 77.5 | 51.6 | 91.9 |
| MA-HMR (Base) | 61.2 | 46.2 | 77.1 | 47.1 | 31.7 | 55.7 | 77.8 | 51.3 | 92.4 |
| CISyn -HMR (Base) | 59.4 | 45.7 | 75.1 | 46.4 | 31.2 | 54.9 | 77.0 | 51.2 | 91.7 |
| Training data | Hi4D | CHI3D | Harmony4D | ||||||
|---|---|---|---|---|---|---|---|---|---|
| MPJPE | PA-MPJPE | PVE | MPJPE | PA-MPJPE | PVE | MPJPE | PA-MPJPE | PVE | |
| CISyn -HMR (Base = AGORA + BEDLAM + DTO-Humans) | |||||||||
| Base+Hi4D | – | – | – | 45.1 | 30.2 | 53.4 | 75.4 | 50.4 | 90.0 |
| Base+CHI3D | 56.2 | 42.8 | 70.9 | – | – | – | 75.5 | 50.2 | 89.9 |
| Base+Harmony4D | 58.1 | 44.4 | 73.2 | 45.9 | 30.5 | 54.2 | – | – | – |
| Base+ CISyn | 52.3 | 39.3 | 65.6 | 44.9 | 29.9 | 53.4 | 74.4 | 50.4 | 88.5 |
| Phys. Sim. | Occ. | Unc. | Hi4D | CHI3D | Harmony4D | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| MPJPE | PA-MPJPE | PVE | MPJPE | PA-MPJPE | PVE | MPJPE | PA-MPJPE | PVE | |||
| ✓ | ✓ | 55.4 | 41.6 | 69.3 | 45.8 | 30.8 | 54.2 | 75.8 | 50.6 | 90.4 | |
| ✓ | ✓ | 52.6 | 39.4 | 65.8 | 46.2 | 30.5 | 54.5 | 74.6 | 50.5 | 88.7 | |
| ✓ | ✓ | 54.6 | 39.3 | 68.2 | 46.4 | 30.9 | 54.9 | 76.6 | 50.9 | 90.8 | |
| ✓ | ✓ | ✓ | 52.3 | 39.3 | 65.6 | 44.9 | 29.9 | 53.4 | 74.4 | 50.4 | 88.5 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Group | Joint | [°] | [°] | [°] |
|---|---|---|---|---|
| Body joints (left side shown; right mirrors / ) | ||||
| Root | Pelvis | |||
| Spine | Torso | |||
| Spine | ||||
| Chest | ||||
| Head | Neck | |||
| Parameter | Standard | High-speed |
|---|---|---|
| Body shape | sampled w/o replacement from 1100 assets | |
| Initial pose | sampled w/o replacement from AMASS | |
| Spawn radius / height | m, m | |
| Min. angular separation | ||
| Yaw (root orientation) | ||
| Pitch / roll noise | (clipped ) | (clipped ) |
| Energy Distance | RBF-MMD | |
| Real vs. real | ||
| Hi4D – CHI3D | 0.1918 | 0.0830 |
| Hi4D – Harmony4D | 0.1395 | 0.0630 |
| CHI3D – Harmony4D | 0.1677 | 0.0716 |
| Synthetic vs. pooled real | ||
| CISyn | 0.2350 (1.40 ) | 0.1101 (1.54 ) |
| Dataset | Mean depth (mm) | Max | Frames | |
|---|---|---|---|---|
| all frames | w/ pen. | (mm) | (%) | |
| Hi4D [ 62 ] | 3.99 | 8.06 | 81.64 | 49.56 |
| CHI3D [ 14 ] | 2.63 | 9.62 | 133.59 | 27.37 |
| Harmony4D [ 23 ] | 3.89 | 12.18 | 134.82 | 31.97 |
| CISyn (ours) | 1.36 | 5.41 | 78.05 | 25.12 |
| Head | Architecture |
|---|---|
| Encoder heads | |
| Scale Map | MLP( , , ; 4 layers), sigmoid |
| FOV | MLP( , , ; 4 layers) |
| Decoder heads (per layer) | |
| Pose (6D) | MLP( , , ; 6 layers) |
| Shape | MLP( , , ; 5 layers) |
| Symbol | Loss | Weight |
|---|---|---|
| 3D joint Laplacian NLL (root-aligned) | 8.0 | |
| 2D joint Laplacian NLL (projected) | 40.0 | |
| SMPL 6D pose L1 | 5.0 | |
| SMPL shape L1 (per-coeff. weighted) | 3.0 | |
| Bounding-box L1 + GIoU | 2.0 | |
| Detection confidence (focal loss) | 4.0 |
| Method | CISyn test set | ExPI † | |||
|---|---|---|---|---|---|
| MPJPE | PA-MPJPE | PVE | MPJPE | PA-MPJPE | |
| CISyn -HMR (Base = AGORA + BEDLAM + DTO-Humans) | |||||
| Base+Hi4D | 265.5 | 162.8 | 286.5 | 114.5 | 80.7 |
| Base+CHI3D | 260.0 | 160.3 | 280.6 | 111.6 | 80.9 |
| Base+Harmony4D | 298.5 | 171.2 | 322.4 | 113.6 | 82.0 |
| Base+ CISyn | 133.2 ‡ | 103.2 ‡ | 149.8 ‡ | 110.3 | 80.2 |
| Training data | Hi4D | CHI3D | ExPI † | Harmony4D | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Casual | Combat | Dance | Social | Sports | Casual | Combat | Social | Acrobatics | Combat | Dance | Social | |
| Base+Hi4D | – | – | – | – | – | 56.0 | 46.4 | 47.1 | 114.5 | 81.4 | 72.8 | 65.7 |
| Base+CHI3D | 61.7 | 55.8 | 52.7 | 58.7 | 46.9 | – | – | – | 111.6 | 81.5 | 73.4 | 65.9 |
| Base+Harmony4D | 64.2 | 56.7 | 53.4 | 60.3 | 47.8 | 56.5 | 46.6 | 47.9 | 113.6 | – | – | – |
| Base+ CISyn | 56.3 | 51.1 | 47.5 | 52.6 | 45.3 | 55.5 | 45.6 | 46.8 | 110.3 | 80.3 | 71.3 | 64.5 |
| Method | Hi4D | CHI3D | Harmony4D | |||
| Rec. | Miss | Rec. | Miss | Rec. | Miss | |
| Base | ||||||
| MA-HMR | 99.18 | 242 | 98.92 | 349 | 97.47 | 912 |
| CISyn -HMR | 99.35 | 191 | 98.98 | 330 | 98.11 | 680 |
| Base+ CISyn | ||||||
| MA-HMR | 99.98 | 7 | 99.29 | 231 | 98.00 | 718 |
| Method | Hi4D | CHI3D | Harmony4D | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Dist. | Pen. (mm) | Pen. (%) | Dist. | Pen. (mm) | Pen. (%) | Dist. | Pen. (mm) | Pen. (%) | |
| Ground truth | – | 4.9 | 56.6 | – | 2.6 | 27.3 | – | 3.6 | 33.1 |
| MA-HMR (Base) | 29.7 | 11.0 | 49.8 | 38.4 | 3.2 | 21.2 | 87.2 | 2.7 | 18.2 |
| CISyn -HMR (Base) | 27.5 | 10.1 | 45.9 | 38.8 | 2.8 | 18.9 | 78.9 | 2.4 | 16.2 |
| MA-HMR (Base+ CISyn ) | 24.9 | 11.6 | 53.3 | 37.4 | 3.3 | 22.0 | 86.7 | 3.1 | 21.4 |
| CISyn -HMR (Base+ CISyn ) | 24.0 | 10.9 | 50.3 | 37.7 | 2.7 | 18.5 | 71.8 | 2.6 | 17.6 |
| Training | MPJPE | PA-MPJPE | PVE |
|---|---|---|---|
| MA-HMR (Base) | 62.9 | 39.9 | 73.5 |
| MA-HMR (Base+ CISyn ) | 62.2 | 39.8 | 73.0 |
| CISyn -HMR (Base) | 62.6 | 39.6 | 73.0 |
| CISyn -HMR (Base+ CISyn ) | 62.6 | 39.5 | 73.0 |
Human Mesh Recovery
Human mesh recovery estimates articulated 3D body shape and pose from images, video, or other sensors for animation, tracking, and physical analysis.
See all 55 papers