Video generation models can now depict remarkably realistic motion. Yet it is unclear how to use this motion beyond pixels: to animate a specific 3D asset and replay the animation in its 3D scene. For instance, a video model can show a monkey jumping over a crossbar, but it cannot return that jump as replayable 3D motion placed in the 3D scene. Here we present a model that directly co-generates the video and 3D motion for any supplied rig, along with a camera that registers this motion to the scene. Prior video-motion models either handle only human skeletons, often adding billions of parameters, or triangulate 3D motion from generated 2D pose maps. We instead keep 3D motion in a separate model that supports arbitrary skeletons, and connect it to a pretrained video model through a small set of cross-attention layers. This separation lets the motion model first learn how skeletons move from motion-only data, which is easier to obtain and augment than paired video. Additionally, our model learns to locate each joint in the video frames, which lets us recover the camera and place the 3D motion in the scene. To train across many skeletons, we use a generative video editor to place animated 3D assets in scenes suited to their actions, curating the edits so their known motion remains valid. Together with human and robot video, this yields nearly 200K training pairs. Our evaluations demonstrate that our model animates diverse rigs from text more faithfully than video-first pipelines, and its motion responds to the surrounding scene. It also turns video into rig motion, rig motion into video, and enables keyframe-based control of the 3D motion. Videos and additional results are available on our project page: https://research.nvidia.com/labs/par/projects/jca/.
Figures & tables
Figure 1: Joints, Camera, Action! (JCA) generates a video and a synchronized 3D motion trajectory for a user-supplied rig of any topology. One model covers joint generation (top two rows) and either conditional direction (bottom two rows). Mesh animation: a rigged monkey clears a scene’s high-jump bar, adapting its motion to the surroundings. Keyframe control: given a first and last frame, JCA fills in the rig motion and video between them. Video-to-motion: recovering 3D motion from an input video. Motion-to-video: video following prescribed 3D motion.
Figure 2: Architecture overview. Given a text prompt, a rig, and an optional reference image, the model jointly denoises video latents Zv and motion tokens Zm . A graph network encodes the rig into per-joint descriptors D , using the joints’ 2D locations in the reference image, and these condition the motion branch; a graph-attention layer inside each Motion DiT block propagates features along the rig edges. Zero-initialized cross-attention bridges exchange features in both directions at paired blocks, each branch querying the other’s intermediate features, so joint training begins from the two pretrained branches acting as they did in isolation. An auxiliary joint-localization head (right) attends from joint features to video patches and returns each joint’s image position as a weighted combination of patch coordinates, giving 2D–3D correspondences used to fit a camera.
Figure 3: Text-driven rig animation across diverse assets. Each row shows a text prompt, an input mesh, and sampled poses from animations generated by competing methods and JCA (ours). Our method more faithfully captures the requested actions while preserving the asset’s geometry.
Method
Prompt adherence ↑
Motion smoothness ↑
Dynamic degree
Mesh distortion ↓
AnimateAnyMesh
20.6
.993
.11
1.96
Puppeteer
41.1
.984
.80
0.36
ActionMesh
43.4
.988
.50
7.07
JCA (ours)
58.4
.983
.83
0.20
Table 1: Rig animation from text, averaged over 100 asset–prompt pairs. Prompt adherence is judged by GPT-6 Astra on a 0–100 scale. Mesh distortion is in percent.
Synchronization
Video quality
Dynamics
Method
PA- MPJPE ↓
SA- MPJPE ↓
SA- MPJVE ↓
IQ ↑
Aesth. ↑
Smooth. ↑
Anat. ↑
Dyn. degree
Non-static camera
EchoMotion
97.7
114.9
431.3
.665
.533
.985
.730
.828
74.0%
JCA (ours)
114.8
151.9
598.2
.664
.578
.992
.934
.600
45.6%
JCA SMPL-FT
72.3
91.0
272.6
.657
.573
.994
.938
.488
44.8%
Table 2: Joint video–human-motion generation. Synchronization compares generated and GVHMR-recovered motion; video and dynamics metrics follow VBench.
Figure 4: Additional Applications of our model. Top: Unlike the setting in fig. 1 , we generate a video from a rig motion sequence and a text prompt, without an input image. Middle: An additional video-to-motion example with a different skeletal topology. Bottom: Given the rig and mesh of a real Unitree G1 robot, a 3D scene, an initial pose, and text, we jointly generate video and motion; we show the video alongside the mesh driven by the generated motion and rendered in the scene.
Configuration
PA-MPJPE ↓
SA-MPJPE ↓
SA-MPJVE ↓
JCA (ours)
72.27
90.98
272.58
Without bridge guidance
84.78
103.18
293.06
Disconnected cross-attention
140.71
173.38
456.07
Table 3: Ablation of video–motion coupling using JCA SMPL-FT. Position errors are in mm; velocity errors are in mm/s.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Scene-aware rig animation and motion localization. Given a penguin rig, a rendered initial frame of a snowy 3D scene, and a text prompt, our model jointly generates video and 3D rig motion. Top: frames from the generated video, where the penguin moves forward and slides down the snowy slope. Middle: the raw 3D motion produced by the motion branch, rendered in the input scene before localization. The penguin’s motion is not aligned with the depicted scene geometry and, in later frames, appears beneath the snow surface. Bottom: the same generated motion after applying our joint-localization and camera-alignment scheme described in section 3.3 . The estimated alignment correctly places the motion on the visible penguin and registers it with the generated video and scene.
Figure 6: Prompt-adherence evaluation of one animation with GPT-6 Astra. In the first turn, the judge is given the asset’s short description and 48 frames of the animation, 16 from each of the front, right, and left views, each frame labeled with its view; the figure shows two frames per view. It is asked to describe the motion it observes in chronological order. In the second turn, it scores the motion against the action prompt. The complete prompts and responses are given in section B.3 .
Method
PA-MPJPE ↓
SA-MPJPE ↓
SA-MPJVE ↓
MoCapAnythingV2 †
41.95
46.28
163.97
TopoCap
51.29
87.71
1412.25
Ours
66.40
81.60
351.47
Ours-FT
39.27
47.68
174.95
Appendix
Table 4: Video-to-motion evaluation on the TrueBones Zoo unseen test split, with the original white-background renders. Position errors are in benchmark-normalized mm; velocity errors are in mm/s.
Recent 3D generative models can synthesize high-quality assets, but their outputs are typically static: they lack the skeletal rigs, joint hierarchies, and skinning weights required for animation. This limits their use in games, film, simulation, virtual agents, and embodied AI, where assets must not only look plausible but also move plausibly. We introduce Rigel3D, a generative method for animation-ready 3D assets represented as rigged meshes. Unlike post-hoc auto-rigging methods that attach rigs to completed shapes, our method jointly models geometry and rig structure through coupled surface and skeleton structured latent representations. A rig-aware autoencoder decodes these representations into mesh geometry, skeleton topology, joint coordinates, and skinning weights, while a two-stage latent generative model synthesizes both surface and skeleton representations for image-conditioned generation. To support downstream animation workflows, we further introduce an open-vocabulary joint labeling module that embeds generated joints into a shared vision-language space, enabling correspondence to arbitrary retargeting templates. Experiments on large-scale rigged asset datasets demonstrate that our method generates diverse, high-quality animation-ready assets and outperforms existing rigging baselines across multiple metrics.
Current controllable video generation systems often rely on 2D motion trajectories or sparse drag signals for object motion. These controls are ambiguous because the same 2D trajectory can correspond to different 3D motions, especially when the camera and objects move simultaneously. We present Generative Cinematographer (GenCine), a system that lifts a single image into an editable 3D scene scaffold where artists jointly author camera and foreground motion. Artists specify a camera path and move selected foreground regions using local 3D motion handles. Several handles can move different parts of a subject independently, providing a piecewise-rigid approximation to non-rigid motion without a physics simulator or category-specific prior. To communicate these controls to a pretrained video model, we project them into guidance maps. These maps record where the controlled regions appear in each frame, assign each handle a fixed color across frames and encode the current 3D positions of its controlled points in the same world coordinate system as the background. This lets us describe object motion relative to the scene even as the camera moves. For training, we recover controls from the motion observed in real videos and use ground-truth geometry and trajectories from synthetic videos. We train a lightweight guidance branch and LoRA adapters on a pretrained Wan model to follow these controls. Our experiments show consistent camera-relative motion, improved geometric consistency under viewpoint changes, and strong controllability across diverse real-world scenes.
Animatable 3D assets, defined as geometry equipped with an articulated skeleton and skinning weights, are fundamental to interactive graphics, embodied agents, and animation production. While recent 3D generative models can synthesize visually plausible shapes from images, the results are typically static. Obtaining usable rigs via post-hoc auto-rigging is brittle and often produces skeletons that are topologically inconsistent with the generated geometry. We present AniGen, a unified framework that directly generates animate-ready 3D assets conditioned on a single image. Our key insight is to represent shape, skeleton, and skinning as mutually consistent S3 Fields (Shape, Skeleton, Skin) defined over a shared spatial domain. To enable the robust learning of these fields, we introduce two technical innovations: (i) a confidence-decaying skeleton field that explicitly handles the geometric ambiguity of bone prediction at Voronoi boundaries, and (ii) a dual skin feature field that decouples skinning weights from specific joint counts, allowing a fixed-architecture network to predict rigs of arbitrary complexity. Built upon a two-stage flow-matching pipeline, AniGen first synthesizes a sparse structural scaffold and then generates dense geometry and articulation in a structured latent space. Extensive experiments demonstrate that AniGen substantially outperforms state-of-the-art sequential baselines in rig validity and animation quality, generalizing effectively to in-the-wild images across diverse categories including animals, humanoids, and machinery. Homepage: https://yihua7.github.io/AniGen-web/
Yi-Hua Huang, Zi-Xin Zou, Yuting He +6
The University of Hong Kong, China · VAST, China · The Chinese University of Hong Kong, China +1