Reconstructing human motion in 4D requires identifying the same surface points through articulation, deformation and changes in visibility. We introduce Dense4D, to our knowledge the first feed-forward model dedicated to dense 4D correspondence across the body, clothing and hair from a set of monocular RGB images, without a body template. Trained specifically for human geometry, Dense4D jointly processes person-centred crops with known image-to-crop transforms to predict Human Point Maps (HPMs) and depth for each image. An HPM assigns each person pixel the 3D position of its observed surface point at a common reference time. These maps support amodal correspondence: points hidden in the reference image can be localised from observations in other frames. Matching HPM coordinates across crops establishes correspondence without predicting camera poses or large scene displacements; per-image depth then lifts the matches into 3D tracks. Experiments on real people demonstrate substantially higher tracking accuracy and closer agreement with scanned surfaces than the evaluated baselines. Points from different frames assemble into coherent human geometry in the reference pose, providing visible evidence of consistent dense correspondence through articulation and clothing deformation. Additional qualitative results, videos, and an interactive viewer are available at https://alpguler.com/dense4d.
Figures & tables
Figure 2: Dense 4D human correspondence. An extra crop token conditions the aggregator on cropping geometry ( Eq. 3 ). The heads predict reference-time HPMs and per-image depth, visualised as HPM point clouds and unprojected depth maps. Nearest-neighbour lookup matches HPM coordinates within the target frame ( Eq. 6 ). A 3D chequerboard defined in reference space colours the union and is transferred to the images through HPM coordinates, visualising dense correspondence. Unprojecting the matched pixels with predicted depth gives 4D tracks ( Eqs. 7 and 3.4 ), shown as coloured trajectories.
Method
AJ ↑
APD ↑
OA ↑
Ours
38.4
50.3
89.6
TAPIP3D
17.0
27.1
85.2
V-DPM †
16.9
27.6
76.7
SpatialTrackerV2
15.5
24.4
78.7
4RC †
14.4
24.1
76.6
Shape of Motion
12.5
21.3
80.7
Table 4.2 and Figure 4.2 : Panoptic Studio Humans. Left: tracking on human-body queries (†: adapted to benchmark intrinsics; Appendix C.2 ). Centre: our tracks from 1.7 to 5.0 s. Far right: HPM unions combining person pixels from across the input sequence in each person’s reference pose. Despite the motion across frames, these points assemble into coherent bodies with aligned limbs and surface detail, making the quality of temporal alignment directly visible.
Figure 3: Reference-time reconstruction from sixteen observations on 4D-DRESS. Left: input frames, with the reference boxed. For each method, four viewpoints show the union coloured by source appearance (top) and distance to the scan from 0 to 5 cm (bottom). The measured scan is overlaid in blue-grey in the top row.
Table 4.3 and Figure 4.3 : Amodal 3D localisation on 4D-DRESS. Left: mean distance to scan and percentages of points more than 2 and 5 cm from the surface. Centre: error by time; right: error after dropping each method’s least confident points. Bars show standard errors; the table and temporal plot use a union-wide 5 % cut.
Figure 4: Dense correspondences and 4D tracks. Each row follows a person through three instants, with matching times shown in A and B. (A) The same reference-space 3D chequerboard colours every image, making it possible to follow surface regions through body rotation and garment deformation. (B) The reconstructed person and coloured 3D trajectories, integrated with VGGT- Ω ’s scene points. Together, the panels connect correspondence on the image with the motion of human surface points in 3D.
Figure 5: Human surface consistency on DAVIS. (A) Reference image and unions of person points from multiple frames, all expressed at frame zero. Notice the distinct body outline preserved by Dense4D, compared with the overlapping limbs and dispersed points in the baseline unions. (B) Frame-zero points transferred to the target image, with their original colours retained against a dimmed background. These colours allow direct inspection of alignment between corresponding body regions. Dense4D anchors transfers to pixels within the target person mask and uses their predicted depth, ensuring consistency with its reconstructed geometry by construction.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Training surfaces and samples. (A) BEDLAM2 images and the body, garment and hair surfaces used for supervision. (B) Training crops from BEDLAM2, THuman2.1 and MammaSyn, each from a different clip or scan.
Figure 7: Hair envelope meshes. Envelopes derived from BEDLAM2’s strand grooms, shown from a common viewpoint with consistent lighting. Labels identify the source grooms.
Figure 8: Surface supervision for 4D correspondence. A human sequence with body, garment and hair labels, reference-coordinate colours and texture transfer. The plots summarise track length and labelled-pixel composition; the histogram’s training pool includes the development tracks.
SAM 3
VGGT- Ω
Dense4D, 512×512 crops
Frames
1280×720
512×288
crops, read-out
forward
Total
Peak memory
100
10.6 s
1.8 s
0.8 s
4.7 s
17.9 s
19.5 GB
150
18.2 s
3.3 s
1.4 s
10.0 s
32.9 s
21.9 GB
300
33.0 s
11.0 s
2.6 s
36.6 s
83.2 s
29.1 GB
Appendix
Table 1: Inference time and memory of the full pipeline on one GH200, for a 1280×720 internet video, with each stage at its own input size. SAM 3 tracks the person on the full frames; VGGT- Ω gives the cameras and scene depth from the full frames at 512×288 ; Dense4D runs on 512×512 person crops, which carry 1.8 × the tokens of VGGT- Ω ’s frames, so its forward pass is the longer one although the two networks use the same backbone architecture. “Crops, read-out” is CPU time for building the crops and writing the outputs. All models stay loaded in one process (loading, 27 s, is not included); peak memory is the GPU allocator maximum, including 11.4 GB of model weights.
Variant
Panoptic Studio Humans
BEDLAM2
AJ ↑
APD ↑
OA ↑
(cm) ↓
Without crop conditioning
30.4
41.4
89.3
4.06
Centre ray fixed to (0,0,1)
38.4
50.3
89.6
2.99
Final model
38.4
50.3
89.6
2.99
Appendix
Table 2: Crop-geometry conditioning. The fixed-ray row modifies only the final model’s inference input. Evaluation follows Sec. 4.2 for Panoptic Studio Humans and Sec. C.4 for BEDLAM2.
Figure 9: Temporal extent and input density on BEDLAM2. Pointwise reconstruction error in centimetres, with means and standard errors over tracks. Left: windows sampled at 6 fps on tracks lasting at least ten seconds (155 tracks) or sixteen seconds (25 tracks). Middle: 4 to 96 frames over the same sixteen-second window. Right: error by temporal offset within that window; the star marks the reference.
We introduce Point4D, a feed-forward model for 4D reconstruction of long-range video sequences. Point4D is able to reliably infer dense per-point 3D trajectories across multi-hundred-frame videos, unlike existing 4D methods that are limited to short input windows of at most a few dozen frames. A key innovation that enables this is our flexible 3D query-based motion decoder that decouples trajectory prediction from image-plane visibility. The predicted 3D endpoints are then directly re-queried in the next chunk without re-projection or matching. Furthermore, we show that extracting and reusing a visual descriptor from an arbitrary frame where the point is visible leads to better performance than relying solely on the source patch. Overall, Point4D achieves state-of-the-art performance across diverse long-video tracking benchmarks spanning over 200 frames and largely outperforms previous feed-forward 4D method. Project page: https://point-4d.github.io
Accurate reconstruction and tracking of dynamic human faces from image sequences is challenging because non-rigid deformations, expression changes, and viewpoint variations occur simultaneously, creating significant ambiguity in geometry and correspondence estimation. We present a unified method for high-fidelity 4D facial reconstruction based on canonical facial point prediction, a representation that assigns each pixel a normalized facial coordinate in a shared canonical space. This formulation transforms dense tracking and dynamic reconstruction into a canonical reconstruction problem, enabling temporally consistent geometry and reliable correspondences within a single feed-forward model. By jointly predicting depth and canonical coordinates, our method enables accurate depth estimation, temporally stable reconstruction, dense 3D geometry, and robust facial point tracking within a single architecture. We implement this formulation using a transformer-based model that jointly predicts depth and canonical facial coordinates, trained using multi-view geometry data that non-rigidly warps into the canonical space. Extensive experiments on image and video benchmarks demonstrate state-of-the-art performance across reconstruction and tracking tasks, achieving approximately 3× lower correspondence error and faster inference than prior dynamic reconstruction methods, while improving depth accuracy by 16%. These results highlight canonical facial point prediction as an effective foundation for unified feed-forward 4D facial reconstruction.
Umut Kocasari, Simon Giebenhain, Richard Shaw +1
Technical University of Munich, Germany · Huawei Noah’s Ark Lab, London, UK
Dense 4D reconstruction from unposed images remains a critical challenge, with current methods relying on slow test-time optimization or fragmented, task-specific feedforward models. We introduce UFO-4D, a unified feedforward framework to reconstruct a dense, explicit 4D representation from just a pair of unposed images. UFO-4D directly estimates dynamic 3D Gaussian Splats, enabling the joint and consistent estimation of 3D geometry, 3D motion, and camera pose in a feedforward manner. Our core insight is that differentiably rendering multiple signals from a single Dynamic 3D Gaussian representation offers major training advantages. This approach enables a self-supervised image synthesis loss while tightly coupling appearance, depth, and motion. Since all modalities share the same geometric primitives, supervising one inherently regularizes and improves the others. This synergy overcomes data scarcity, allowing UFO-4D to outperform prior work by up to 3 times in joint geometry, motion, and camera pose estimation. Our representation also enables high-fidelity 4D interpolation across novel views and time. Please visit our project page for visual results: https://ufo-4d.github.io/
Junhwa Hur, Charles Herrmann, Songyou Peng +4
Google · Princeton University · Harvard University