cs.CVAug 20, 2026
Save4DAnyone: Create Anyone in 4D from a Casual Monocular Video
Organizations: State Key Lab of CAD&CG, Zhejiang University, Hangzhou, China · Robbyant, Hangzhou, China · Chinese University of Hong Kong, Hong Kong, China · Ant Group, Hangzhou, China ·
Abstract
We present 4DAnyone, a framework for reconstructing 4D humans from an uncalibrated monocular video by generating reconstruction-grade multiview-consistent videos and lifting them into 4D Gaussian Splatting (4DGS). Existing camera-controlled video diffusion models synthesize plausible novel-view videos but fail to maintain consistency when scaled to the tens of target views required for 4DGS reconstruction. We identify this failure as a bounded-attention-context problem: when target views exceed the capacity of a single DiT forward pass, they must be split into groups, exposing two coupled bottlenecks. On the reference-context side, conditioning on all previously generated views grows as , weakening cross-view appearance guidance. On the target-context side, disjoint groups cannot directly exchange information, causing global structural drift. 4DAnyone addresses both bottlenecks with two complementary designs: Reference Context Packing (RCP) compresses growing reference views into a fixed-length mixed-resolution context with reference-context complexity, while Target Context Routing (TCR) rotates target-view groupings during denoising to share context across groups at high-noise steps and stabilize details at low-noise steps. We further build the MVGameHuman dataset using our in-house game engine and combine it with light-stage and in-the-wild video datasets for training. Experiments on DNA-Rendering and DyMVHumans show that 4DAnyone outperforms prior methods in both novel-view video quality and downstream 4DGS reconstruction, with robust in-the-wild generalization. See our project page for video results and source code: https://4danyone.github.io.
Figures & tables
Figure 1. Given a casually captured monocular video, typically with mild camera motion and unknown camera intrinsics and poses, 4DAnyone generates multiview-consistent human videos, enabling 4DGS reconstruction rendered from free viewpoints. See the project page and Fig. 7 for more results. One continuous 3D scene: a smartphone at the lower left films a bass-playing subject, a ring of twenty-four generated camera views surrounds the stage, and the reconstructed subject stands at the center on a turntable, rendered from a free viewpoint.
Figure 2. Overview of 4DAnyone. Given a source video, an HMR model (GVHMR ( Shen et al., 2024 ) ) estimates a 3D skeleton sequence, which is rendered into depth-buffered skeleton videos for target views. A 3D-aware skeleton encoder produces residual skeleton tokens that are added to the noisy target latents. The source tokens, RCP reference tokens, and skeleton-conditioned target tokens are concatenated along the view dimension and processed by the DiT with video attention, multiview attention, and text cross-attention. Generated target videos are packed back into the RCP context for subsequent generation rounds, while TCR improves consistency during grouped target-view generation. The final multi-view videos are used to train a 4DGS model with FreeTimeGS ( Wang et al., 2025b ) . Pipeline diagram. A source video is converted into skeleton-conditioned target-view tokens. Reference Context Packing supplies compact reference tokens, Target Context Routing exchanges information among target-view groups, and the generated multiview videos are reconstructed as a 4D Gaussian Splatting model.
Figure 3. Progressive inference with Reference Context Packing and Target Context Routing. We first generate reference videos and pack them with the source video into a fixed RCP context. During target-view generation, this fixed RCP context is shared by all groups. TCR partitions target views into four-view groups, cyclically regroups them at high noise levels to propagate global structure, and fixes adjacent groups at low noise levels for stable detail refinement. Three-stage inference diagram. Initial views become a fixed multiscale reference context; target views are cyclically regrouped at high noise to share global structure and arranged into fixed adjacent groups at low noise to refine local details.
| Dataset | Videos | Cameras | Actors | Resolution | Type |
|---|---|---|---|---|---|
| MVGameHuman | 38k | 24 | 318 | 2560 1440 | Multi-view |
| SynCamVideo | 34k | 10 | 66 | 1280 1280 | Multi-view |
| DNA-Rendering | 51k | 48 | 548 | 2048 2048 | Multi-view |
| TedTalk | 42k | 1 | 413 | 2160 1620 | Monocular |
| Pexels | 20k | 1 | 1,411 | 3840 2160 | Monocular |
Table 1. Training dataset statistics.
| Method | DNA-Rendering | DyMVHumans | |||||
|---|---|---|---|---|---|---|---|
| PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | ||
| Gen. Video Consistency | MV-Performer ( Zhi et al., 2025 ) | 21.25 | 0.799 | 0.222 | 19.98 | 0.797 | 0.182 |
| TrajectoryCrafter ( Yu et al., 2025 ) | 13.56 | 0.641 | 0.331 | 15.19 | 0.769 | 0.204 | |
| ReCamMaster † ( Bai et al., 2025a ) | 21.47 | 0.806 | 0.210 | 21.94 | 0.833 | 0.142 | |
| Ours | 24.33 | 0.862 | 0.163 | 24.48 | 0.862 | 0.109 | |
| 4DGS Reconstruction | MV-Performer ( Zhi et al., 2025 ) | 20.38 | 0.826 | 0.191 | 18.69 | 0.787 | 0.158 |
Table 2. Quantitative comparison on DNA-Rendering and DyMVHumans. We evaluate 4DGS reconstruction, generated video consistency, and generated video reconstruction. Best results are in bold. denotes our fine-tuned version.
| Configuration | PSNR | SSIM | LPIPS |
|---|---|---|---|
| w/o TCR & RCP | 21.09 | 0.766 | 0.216 |
| w/o RCP | 22.03 | 0.780 | 0.203 |
| w/o TCR | 22.21 | 0.788 | 0.196 |
| Full (Random) | 22.20 | 0.788 | 0.197 |
| Full (Strided) | 22.06 | 0.786 | 0.198 |
| Full (Sliding) | 22.63 | 0.796 | 0.191 |
Table 3. Ablation study. Following the Gen. Video Consistency setting, we evaluate the consistency among generated videos.
Figure 4. Qualitative comparison with baselines. We show target-view generated videos (Gen.) and their corresponding 4DGS renderings (Rend.) across diverse human-centric videos. 4DAnyone produces geometrically accurate and visually detailed results across viewpoints, while the baselines suffer from inaccurate camera control (our fine-tuned ReCamMaster † ) or geometric distortions (MV-Performer). See the project page for dynamic results. A grid compares generated target views and reconstructed 4D renderings from several methods on diverse human videos. The proposed method preserves body geometry, clothing appearance, and camera viewpoint more consistently than ReCamMaster and MV-Performer.
Figure 5. Qualitative ablation results. Left: 3D-aware skeleton conditioning resolves the inherent ambiguity of 2D skeletons, guiding the model to generate geometrically correct content. Right: RCP and TCR maintain an effective cross-view context, leading to consistent appearance across generated views. Side-by-side ablation grids compare outputs with and without 3D-aware skeleton conditioning, Reference Context Packing, and Target Context Routing. The complete model has more accurate body geometry and more consistent appearance across views.
Figure 6. Challenging cases. 4DAnyone robustly handles back-view source videos (left), complex subject appearance (right), and complex human motions. Two challenging examples show source frames and generated viewpoints for a person initially seen from the back and a person with complex clothing and appearance.
Figure 7. Robust generalization to diverse in-the-wild human videos. For each example, we show the source video (left), generated target-view videos (middle, 4 of 16 views), and a 4DGS novel-view rendering (right). See the project page for dynamic results. Sixteen in-the-wild examples are arranged in two columns. Each example shows a source human video frame, four generated target viewpoints, and a novel-view rendering of the reconstructed 4D Gaussian avatar.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8. Representative MVGameHuman samples. Each row shows one frame from four evenly spaced cameras in a synchronized 24-camera sequence captured with our in-house game data engine. MVGameHuman provides diverse actors, clothing, motions, lighting conditions, virtual scenes, and backgrounds for training multi-view human video generation. Eight rows of synthetic human scenes, each viewed simultaneously from four evenly spaced virtual cameras. The samples vary in actor, clothing, pose, lighting, environment, and background.
| Dataset | No. Tgt Cam | No. Src Cam | No. Frame | Weight |
|---|---|---|---|---|
| MVGameHuman | 6 | 1 / 4 / 8 | 41 | 0.4 |
| MVGameHuman | 4 | 1 / 4 / 8 | 61 | 0.4 |
| MVGameHuman | 1 | 1 / 4 / 8 | 121 | 0.2 |
| SynCamVideo | 4 | 1 / 4 | 61 | 0.8 |
| SynCamVideo | 1 | 1 / 4 / 8 | 81 | 0.2 |
| DNA-Rendering | 6 | 1 / 4 / 8 | 41 | 0.4 |
Table 4. Training data sampling. Source cameras are uniformly sampled from the listed options.
| Stage | Dataset | Bg. Removal | Indep. Src Prob | Skeleton |
|---|---|---|---|---|
| 1 | DNA-Rendering | 0.2 | B+H+F+Fi | |
| 2 | + MVGameHuman, SynCam. | – | 0.0 | B+H+F+Fi |
| 3 | + Pexels, TedTalk | – | 0.0 | B+H+F |
Table 5. Stage-specific training settings. “Indep. Src Prob” denotes the probability of independently sampling the source frame range. Skeleton: B=body, H=hands, F=feet, Fi=fingers.
| No. Layers | No. Cam / Layer | No. GPUs | No. Tgt Cam / GPU |
| 1 | 16 | 4 | 4 |
| 2 | 16 | 8 | 4 |
| 3 | 16 | 8 | 6 |
Table 6. Multi-GPU inference configurations. Configurations are shown for different camera setups.
| Sliding steps | PSNR | SSIM | LPIPS | |
|---|---|---|---|---|
| 1.00 | 0 | 22.2079 | 0.7880 | 0.1964 |
| 0.75 | 5 | 22.3030 | 0.7898 | 0.1947 |
| 0.50 | 10 | 22.4575 | 0.7925 | 0.1933 |
| 0.25 | 15 | 22.6093 | 0.7955 | 0.1912 |
| 0.20 | 16 | 22.6294 | 0.7963 | 0.1906 |
| 0.15 | 17 | 22.6023 | 0.7961 | 0.1908 |
Table 7. TCR switching-time sweep. Gen. Video Consistency when varying the number of sliding denoising steps.
Figure 9. Single image to 4D avatar. Given a single input image, we first generate a source video via pose-driven video generation (Wan-Animate), then produce multi-view target videos with 4DAnyone, and finally reconstruct a 4DGS avatar via FreeTimeGS. A left-to-right pipeline turns one portrait into a pose-driven source video, generates synchronized target-view videos with 4DAnyone, and reconstructs an animatable 4D Gaussian avatar.
Figure 10. Failure cases. Left: skeleton guidance is uninformative for the large flowing fabric, which is generated inconsistently across views and yields a degraded reconstruction (red box). Right: HMR mis-estimates the en-pointe pose (green circle in the source view) as flat feet, and all generated views inherit the pose error (red box). Two failure examples. Red boxes highlight inconsistent large flowing fabric in one reconstruction and incorrect feet in generated views caused by a source-pose estimation error marked with a green circle.
More in this topic
Video Diffusion Models
Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.
See all 735 papers