cs.CVOct 10, 2026
SaveCAST: Real-Time Motion Capture for Any Skeleton Topology
Organizations: Xi’an Jiaotong University, China · Zhongguancun Academy, China · Tencent, China · Tsinghua University, China · Beijing Normal University, China
Abstract
This paper studies the problem of recovering motion from monocular video, even when the target skeleton topology differs from the subject observed in the video. Although previous methods have made efforts to resolve this problem, they remain vulnerable to joint-rotation errors that are coupled across the skeleton and can propagate along its kinematic hierarchy. To mitigate these errors, we present CAST, a topology-aware motion capture framework that combines learned local rotations with geometric correction guided by predicted global bone directions. Our observability analysis characterizes which rotational components can be constrained by outgoing bone directions. Guided by this analysis, GRACE uses reliability-gated alignment to correct composed orientations in parent-to-child order, leaving unconstrained components to the learned rotation predictor. The motion predictor and correction module are trained jointly with motion-derived supervision, using a frozen image encoder. Experiments under the TopoCap evaluation protocol show that the proposed framework improves motion recovery across different target topologies, including unseen structures. Reference-based cross-skeleton evaluation on a curated Mixamo benchmark and a user preference study further assess transfer to different target structures. A causal CAST-B variant achieves over 51 FPS on a single NVIDIA A100 GPU, with timing covering image encoding and motion prediction. Code repository: https://github.com/fankewen/CAST.
Figures & tables
Figure 1. CAST animates four target skeletons (left) from each of seven input clips (top). Seven input video clips drive four different target skeletons using CAST.
Figure 2. CAST overview. (a) CAST predicts initial local rotations and global bone directions from an RGB video and a target skeleton, then refines the rotations with GRACE. (b) Rotational observability: a leaf provides no directional constraint, a single child constrains swing but not axial twist, and two non-collinear child bones constrain the full rotation. (c) GRACE aligns FK directions (blue) with predicted directions (teal) through minimum-angle swing for a single direction or joint alignment of multiple directions.
| Task | Dataset | Model | MPJPE | PA-MPJPE | MPJVE | CD |
|---|---|---|---|---|---|---|
| Matched-rig | Zoo | Puppeteer | 454.70 | 236.38 | 19.74 | 0.245 |
| TopoCap | – | – | – | – | ||
| MCA V2 ∗ | 105.52 | 67.04 | 10.35 | 0.077 | ||
| CAST-B | 42.68 | 33.23 | 5.48 | 0.036 | ||
| CAST-L | 39.59 | 31.05 | 5.14 | 0.034 | ||
| MObj Seen | Puppeteer | 205.85 | 103.89 | 22.09 | 0.136 |
Table 1. Combined matched-rig and cross-skeleton results. Lower is better for all metrics, and a dash denotes an unreported result. MPJPE and PA-MPJPE are reported in millimeters, MPJVE in millimeters per frame, and CD in normalized coordinates.
Figure 3. Qualitative comparison with baselines. Left: matched-rig motion capture. Right: cross-skeleton animation, with retargeted target-character references in the final column (Retarget). Each example shows an input frame and the three methods’ outputs on the same target character.
Figure 4. Graph attention by kinematic relation on Zoo. (a) An illustrative query joint and its topological relations. (b) Final encoder-layer attention distributions pooled over valid frames and query joints across the evaluated Zoo sequences.
| Method | First choice (%) | Mean rank |
|---|---|---|
| TopoCap | 19.7 | 2.261 |
| MCA V2 ∗ | 16.9 | 2.208 |
| CAST-L | 63.3 | 1.531 |
Table 2. Overall user-study preferences.
| Zoo | MObj Seen | MObj Unseen | ||||
|---|---|---|---|---|---|---|
| Model | MPJPE | PA-MPJPE | MPJPE | PA-MPJPE | MPJPE | PA-MPJPE |
| CAST-B w/o GRACE (DINOv2) | 49.58 | 38.70 | 64.56 | 37.13 | 93.32 | 52.12 |
| CAST-B w/o GRACE (DINOv3) | 50.41 | 38.78 | 63.71 | 36.87 | 86.64 | 50.71 |
| CAST-B (DINOv2) | 42.41 | 33.03 | 57.34 | 33.27 | 83.15 | 48.19 |
| CAST-B (DINOv3) | 42.68 | 33.23 | 55.39 | 32.92 | 79.21 | 46.52 |
Table 3. GRACE and visual-encoder ablations on CAST-B. Bold indicates the best result.
| Zoo | MObj Seen | MObj Unseen | ||||
|---|---|---|---|---|---|---|
| Task setting | MPJPE | PA-MPJPE | MPJPE | PA-MPJPE | MPJPE | PA-MPJPE |
| Full sequence | 42.68 | 33.23 | 55.39 | 32.92 | 79.21 | 46.52 |
| Key frame ( ) | 45.61 | 35.15 | 58.52 | 35.84 | 85.93 | 49.65 |
| Key frame ( ) | 48.79 | 37.35 | 67.98 | 41.37 | 92.01 | 53.90 |
| Key frame ( ) | 58.53 | 43.68 | 82.73 | 50.15 | 104.45 | 61.69 |
| Key frame ( ) | 75.96 | 52.77 | 99.39 | 59.09 | 118.54 | 68.47 |
Table 4. Causal inference and motion completion with CAST-B. The Full sequence setting denotes the bidirectional model with dense input, and denotes the observation interval for key-frame completion. Each Real-time model is trained separately for different chunk sizes. Errors are reported in millimeters, with lower values indicating better performance. Bold and underline mark the best and second-best results.
| 30 joints | 52 joints | 89 joints | ||||||
|---|---|---|---|---|---|---|---|---|
| Model | Params | Frames | Time | FPS | Time | FPS | Time | FPS |
| TopoCap (10 steps) | 2913.8 | 8 | 321 | – | 342 | – | 321 | – |
| TopoCap (300 steps) | 2913.8 | 8 | 8550 | – | 9080 | – | 8510 | – |
| MCA V2 | 452.6 | 32 | 192 | – | 191 | – | 192 | – |
| CAST-B | 352.6 | 81 | 142 | – | 145 | – | 154 | – |
| CAST-L | 488.2 | 81 | 186 | – | 189 | – | 201 | – |
Table 5. Inference efficiency on a single NVIDIA A100. Params (millions) and Time (ms per forward pass) include the frozen image backbone. Frames denotes the input length per pass, and FPS measures streaming inference throughput.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5. The Blender add-on for live capture. (a) The add-on panel: runtime paths, service URL and API key, pose and control ports, transfer controls, and a status log. (b) A live capture session: the performer is recorded while the add-on drives the rigged character in real time.
| Collection/split | Rig records | Motion clips | Frame records |
|---|---|---|---|
| Zoo | 137 | 1,163 | 1,396,500 |
| MObj Train | 12,723 | 17,955 | 1,905,136 |
| MObj Seen | 440 | 468 | 39,261 |
| MObj Unseen | 468 | 468 | 64,279 |
Table 6. Statistics of the processed collections. Zoo statistics describe the collection before the 95%/5% training/test split. MObj statistics follow the official Train, Seen, and Unseen assignments after rendering-quality filtering.
Figure 6. The ranking interface used for the user study. (a) The landing page, which states the task, the estimated duration, and the anonymity of the study. (b) The background questions and the consent box. (c) A ranking item: one input or source animation above three anonymized results, each assigned a distinct rank.
Figure 7. Twist errors. The input is shown to the left of the vertical divider; CAST (Ours) outputs on five target characters are shown to the right. The overall body configuration is largely preserved, but twist errors cause the head orientation to differ from the input. With rank-two directional support, outgoing bone directions constrain swing but do not determine axial twist.
Figure 8. Matched-rig motion capture on Zoo. Columns within each panel show Input, MCA V2 ∗ , and Ours. All methods use the same rig, and successive rows show the motion advancing over time.
Figure 9. Matched-rig motion capture on Zoo (continued). Columns within each panel: Input, MCA V2 ∗ , and Ours. Rows show five successive sampled frames.
Figure 10. Matched-rig motion capture on MObj. Columns within each panel show Input, TopoCap, MCA V2 ∗ , and Ours, evaluated on the same rig. Time advances from top to bottom.
Figure 11. Matched-rig motion capture on MObj (continued). Columns within each panel: Input, TopoCap, MCA V2 ∗ , and Ours. Rows show five successive sampled frames.
Figure 12. Matched-rig motion capture on MObj (continued). Columns within each panel: Input, TopoCap, MCA V2 ∗ , and Ours. Rows show five successive sampled frames.
Figure 13. Matched-rig motion capture on MObj (continued). Columns within each panel: Input, TopoCap, MCA V2 ∗ , and Ours. Rows show five successive sampled frames.
Figure 14. Cross-skeleton motion capture on Mixamo. Each action group shows the input and CAST (Ours) on five target characters over five successive frames.
Figure 15. Cross-skeleton motion capture on Mixamo (continued). Each action group shows the input and CAST (Ours) on five target characters over five successive frames.
Figure 16. Cross-skeleton motion capture from Zoo to MObj. Qualitative examples of this transfer direction, each shown as five successive frames of the source clip (Input) and our result on the target skeleton (Ours).
Figure 17. Cross-skeleton motion capture from Zoo to MObj (continued). Further qualitative examples for the same transfer direction.
Figure 18. Cross-skeleton motion capture between Zoo and MObj. Top: remaining Zoo-to-MObj examples. Bottom: MObj-to-Zoo examples. Each pair shows five successive frames of the source clip (Input) and our result on the target skeleton (Ours).
Figure 19. Cross-skeleton motion capture from MObj to Zoo (continued). Further qualitative examples for the same transfer direction.
Figure 20. Cross-skeleton motion capture from MObj to Zoo (continued). Remaining qualitative examples for the same transfer direction.
Figure 21. Cross-skeleton motion capture within Zoo. Qualitative examples of this transfer direction, each shown as five successive frames of the source clip (Input) and our result on the target skeleton (Ours).
Figure 22. Cross-skeleton motion capture within Zoo (continued). Further qualitative examples for the same transfer direction.
Figure 23. Cross-skeleton motion capture within Zoo and MObj. Top: remaining Zoo-to-Zoo examples. Bottom: MObj-to-MObj examples. Each pair shows five successive frames of the source clip (Input) and our result on the target skeleton (Ours).
Figure 24. Cross-skeleton motion capture within MObj (continued). Further qualitative examples for the same transfer direction.
Figure 25. Cross-skeleton motion capture within MObj (continued). Remaining qualitative examples for the same transfer direction.
Figure 26. Masked-frame completion examples. The first twelve of thirty-five examples drawn from 22 animation sequences. Each example shows exactly two observed sketch frames in the upper row, aligned with the first and last frames of the completed five-frame interval shown below. At inference, one image is observed every four frames and the remaining frames are completed by CAST.
Figure 27. Masked-frame completion examples (continued). The next twelve of the thirty-five examples, using the same two-endpoint observation setting.
Figure 28. Masked-frame completion examples (continued). The remaining eleven of the thirty-five examples, using the same two-endpoint observation setting.