cs.GRMar 18, 2026
SaveDancingBox: A Lightweight MoCap System for Character Animation from Physical Proxies
Organizations: University of Edinburgh Edinburgh, United Kingdom · Inria, Université Côte d’Azur Valbonne, France · Tsinghua University Beijing, China
Abstract
Creating compelling 3D character animations typically requires either expert use of professional software or expensive motion capture systems operated by skilled actors. We present DancingBox, a lightweight, vision-based system that makes motion capture accessible to novices by reimagining the process as digital puppetry. Instead of tracking precise human motions, DancingBox captures the approximate movements of everyday objects manipulated by users with a single webcam. These coarse proxy motions are then refined into realistic character animations by conditioning a generative motion model on bounding-box representations, enriched with human motion priors learned from large-scale datasets. To overcome the lack of paired proxy-animation data, we synthesize training pairs by converting existing motion capture sequences into proxy representations. A user study demonstrates that DancingBox enables intuitive and creative character animation using diverse proxies, from plush toys to bananas, lowering the barrier to entry for novice animators.
Figures & tables
Figure 1. a) Overview of our DancingBox system. The interactive setup consists of a mounted web camera ( e.g. , a smartphone) facing the scene, where users manipulate a physical proxy to convey their intended character animation, and a planar object ( e.g. , a checkerboard) for calibrating the up direction of the ground. b) From the captured monocular video input of finger performance, DancingBox first estimates a sequence of bounding boxes using a lightweight vision-based motion capture module ( i.e. , the red arrow, MoCap). It then transforms the coarse box motion into a natural skeletal motion through a conditional motion generation module ( i.e. , the blue arrow, MoGen). c) Our system supports a wide range of physical proxies, such as plush toys, humanoid puppets, or even everyday objects, like a pen or a book. Notably, a banana with peeled skin, mimicking a jumping action with outstretched hands, is faithfully interpreted into the corresponding character motion. For improved visualization, skeletal motions are retargeted to the ‘Michelle’ character from Mixamo. Please see supplemental material for the animation clips of all results shown in the paper. A teaser figure with three blocks - a) the interactive system setup, b) the high-level algorithm pipeline, and c) an appealing example for creating a jumping action.
| Method | Capture device | Physical Input | Motion quality | Virtual Outpu | Data-driven |
| KinEtre ( Chen et al., 2012 ) | Depth camera | Human body | low | Objects | No |
| 3D puppetry ( Held et al., 2012 ) | Kinect | Rigid objects | low | Objects | No |
| Puppeteer ( Hung et al., 2022 ) | RGB camera | Human body | low | Human | Yes |
| Motionmontage ( Gupta et al., 2014 ) | Kinect | Rigid objects | low | Objects | No |
| Fingerpuppet ( Hung et al., 2024 ) | RGB camera | Hand | low | Human | Yes |
| AniCraft ( Li et al., 2024 ) | RGB cameras | Markers + skeleton | low | Animals | No |
Table 1. Comparison against existing physical motion authoring systems.
Figure 2. Our lightweight motion capture module produces noisy, partial point clouds, which we need to relate to the virtual skeletons used to represent character animation datasets. Extracting a clean skeleton from a point cloud, or synthesizing a defect-laden point cloud from a skeleton, are two difficult tasks. Our key observation is that abstract bounding boxes form a suitable middle-ground representation as they are easy to extract from both point cloud and skeleton data. A figure clarifying the design choice of bounding boxes.
Figure 3. Overview of MoCap. From left to right: given the recorded video and user clicks in the first frame, we exploit SAM2 ( Ravi et al., 2024 ) to segment the parts of the puppet in all frames. The user clicks indicate the desired part segmentation, with different colors denoting different parts. The order of the clicks does not influence the result. We also run ( Wang et al., 2025c ) to estimate the point cloud of each frame, and CoTracker3 ( Karaev et al., 2024 ) to produce dense pixel-wise correspondences between frames. Combining semantic segments, point clouds, and motion tracks allows us to recover 3D bounding boxes of proxy parts and their motion across the video clip. A diagram showing the technical steps of the first motion capture module in our method.
Figure 4. Left: overview of our conditional motion generator. We train a custom box motion encoder alongside a ControlNet module to condition a pre-trained Motion Diffusion Model. Right: for a given frame, our box motion encoder first encodes each vertex of a box ( e.g. , the pink one) using an MLP, and aggregates all vertices of the box into a single code using mean and max operations to obtain a latent code that is invariant to vertex ordering. A self-attention layer then exchanges information between the latent codes of all boxes of the character proxy. Finally, the resulting latent codes are aggregated into a single code for the entire character, again using mean and max to be invariant to box ordering. A diagram showing the overview of the motion generation module and the illustration of our novel box motion encoder.
Figure 5. Impact of spatial guidance and its design illustration. Left: without spatial guidance, the bounding boxes are able to provide rough motion control, but do not guarantee precise alignment between boxes and joints ( e.g. , the joints on the right leg vs. the red box). Middle: for an exemplar bounding box (in red), the guidance measurement loss is computed against every joint, then accumulated using distance-based combination weights. Five such distances are visualized, with colors indicating the corresponding weights. Right: With the designed spatial guidance, the generated motion aligns closely with bounding boxes, ensuring each box contains at least one joint. A figure showing the impact of with and without spatial guidance, and its design intuition.
Figure 6. Representative replication results from our user study. Given a target motion (right), participants were asked to reproduce it (middle) by manipulating a designated physical proxy (left). A figure showing the user replication results with three typical examples. From left to right, the recorded video frames of a user manipulating a given puppet, the produced and the reference motions are presented, respectively.
Figure 7. Creative results from participants manipulating diverse puppets (first three rows), and from an experienced user performing the action using paper cut (fourth row), compositing daily objects (fifth row), and a phone (last row) , demonstrate the effectiveness of our system in producing high-quality motion from varying physical proxy . The left column shows the puppets and input texts (if any), the middle column shows the input video frames, while the right column presents the corresponding generated motions. Note that the text input is provided only in the first and third examples. For better visualization, we display the motion frames with spatial displacements on the ground, although these displacements are not part of the motion itself ( e.g. , the motion in the second last example is basically moving in place). A figure demonstrating diverse and compelling motions produced by our system.
Figure 8. Questionnaire statistics (subset). A selected subset of questions with corresponding responses is shown. Gray, green, and magenta colors indicate question categories asked at different stages: overall questions were asked once after the study; replication questions were asked after each replication task, yielding responses; creative questions were asked after the creative task with responses. Some questions were asked after both the replication and creative tasks, with responses. Each question has five response options ( i.e. , from Strongly Disagree to Strongly Agree, scored 1 to 5). Numbers on the bars indicate the count for each response (colored accordingly). Notably, across all selected questions, no participant chose Disagree and Strongly Disagree . See the supplementary for the complete set of questions and responses. A figure showing the questionnaire statistics on a selected subset of user study questions. For each question, five scores from Strongly Disagree to Strongly Agree can be chosen.
Figure 9. Generation result from the same video input of Fig. 1 (b), but adding extra text description to control the arms. A figure showing a generated motion with an extra text as input. The text control complements the box guidance.
Figure 10. Top row: users specify five keyframes, indicating the motions of both arms, the torso, and one leg. Middle row: bounding boxes are estimated for the five keyframes using the same motion capture pipeline. Additional boxes are interpolated between keyframes and shown with transparent coloring. Bottom row: the complete bounding-box sequence serves as guidance to generate the final realistic character motion. A figure showing the process of key-framing application. Top row: 3 frames of a humanoid puppet. Middle row: estimated bounding boxes and interpolation. Down row: generated motion between key frames.
More in this topic
Human Avatar Animation
Human avatar animation builds digital people that can be reconstructed, driven by motion or speech, and rendered consistently across poses and views.
See all 91 papers