Recovering a dense 3D body mesh from monocular video remains challenging under occlusion from draping and continuously moving camera viewpoints. This configuration arises in surgical augmented reality (AR), where an anesthetized patient lies under surgical draping while a surgeon's head-mounted camera continuously changes viewpoint. Existing human mesh recovery (HMR) methods are typically trained on upright, moving subjects captured from relatively stable cameras, leading to performance degradation under such conditions. To address this, we present Patient4D, a stationarity-constrained reconstruction pipeline that explicitly exploits the stationarity prior. The pipeline combines image-level foundation models for perception with lightweight geometric mechanisms that enforce temporal consistency across frames. Two key components enable robust reconstruction: Pose Locking, which anchors pose parameters using stable keyframes, and Rigid Fallback, which recovers meshes under severe occlusion through silhouette-guided rigid alignment. Together, these mechanisms stabilize predictions while remaining compatible with off-the-shelf HMR models. We evaluate Patient4D on 4,680 synthetic surgical sequences and three public HMR video benchmarks. Under surgical drape occlusion, Patient4D achieves a 0.75 mean IoU, reducing failure frames from 30.5% to 1.3% compared to the best baseline. Our findings demonstrate that exploiting stationarity priors can substantially improve monocular reconstruction in clinical AR scenarios.
Figures & tables
Figure 1: Overview of the Patient4D pipeline. Given a monocular surgical video and a single point prompt, SAM3 produces per-frame body masks while MoGe estimates camera intrinsics from the first frame. SAM3DBody then recovers initial per-frame SMPL meshes from the masked regions; body shape β and scale are locked to a single keyframe estimate and broadcast to all frames. To enforce temporal consistency, Pose Locking anchors body pose parameters to a median value calculated across the sequence. Frames whose mesh-mask IoU falls below a threshold ( τIoU=0.6 ) are corrected by Rigid Fallback, which selects the best reference mesh from a keyframe pool and optimizes a rigid transformation to align its projection with the current mask via a Dice objective.
Figure 2: Rigid Fallback on a challenging frame. Left: per-frame HMR produces a misaligned mesh (IoU = 0.21) when the camera moves to an oblique viewpoint with partial visibility. Right: Rigid Fallback takes the best reference mesh from the keyframe pool and optimizes rotation and camera translation to align its projection with the SAM3 mask via the Dice objective, recovering correct alignment (IoU = 0.93). Bottom row: mask overlap (green = projected mesh, blue = SAM3 mask, cyan = intersection).
Figure 3: Representative frames from the simulation datasets. Top: Sim-Geometry sequences rendered from SLP-3Dfits with ground truth SMPL meshes, generated patient texture and orbiting camera silhouettes. Bottom: Sim-Visual frames at three drape coverage levels (none, partial, heavy), depicting realistic surgical scenes with photorealistic skin textures and operating room environments. These two are complementary: Sim-Geometry shows clean silhouettes with ground truth SMPL meshes whereas Sim-Visual demonstrates increasing visual complexity from drapes.
Figure 4: Qualitative comparison of mesh recovery on a synthetic surgical sequence. The patient is positioned in a lateral decubitus pose under no draping. As the camera orbits from the initial viewpoint (Frames 10, 50) to extreme oblique angles (Frames 150, 200), the 2D visual cues become ambiguous. Standard per-frame and video-based baselines fail to recognize the stationarity prior of the patient; they attempt to fit the changing 2D contours by distorting the 3D anatomy (e.g., unnaturally twisting the spine or flattening the torso), resulting in plummeting Mesh-Mask IoU. In contrast, Patient4D (orange) enforces temporal consistency, preserving the correct 3D geometry and maintaining strong spatial alignment across the entire camera trajectory.
Parameter
Sim-Geometry
Sim-Visual
Source
SLP-3Dfits [ 22 ]
Seedance2.0 [ 3 ]
Sequences
4545
135
Frames/seq.
375 (25 fps)
∼ 192 (24 fps)
Resolution
760 × 428
1280 × 720
3D ground truth
Yes
No
Drape occlusion
None
3 levels
Table 1: Simulation dataset configurations.
Drape Level
IoU ↑
> 0.6 ↑
< 0.3 ↓
Δˉ↓
Seqs
None
0.84
99.5%
0.0%
0.010
30
Partial
0.73
94.5%
0.5%
0.051
60
Heavy
0.69
80.1%
3.3%
0.013
45
All
0.75
90.8%
1.3%
0.029
135
Table 2: Performance breakdown of Patient4D across varying levels of drape occlusion on the Sim-Visual dataset. While overall baseline comparisons are provided in \cref tab:sim_baselines, this analysis isolates how Patient4D degrades under increasing visual ambiguity. IoU: Mesh-Mask IoU ( ↑ ). Δˉ : mean vertex displacement ( ↓ ).
Sim-Geometry
Sim-Visual
Method
PA-PVE ↓
IoU ↑
> 0.6 ↑
< 0.3 ↓
Δˉ↓
IoU ↑
CV-IoU 20↑
< 0.3 ↓
Δpose↓
Δˉ↓
HMR 2.0 [ 11 ]
84.2
0.68
78.2
4.5
0.185
0.52
0.28
35.1
0.14
0.650
GVHMR [ 32 ]
95.6
0.58
62.4
8.2
0.014
0.45
0.31
30.5
0.05
0.025
CoMotion [ 27 ]
87.9
0.65
71.5
10.1
0.042
0.38
0.22
52.4
0.08
0.065
SMPLest-X [ 43 ]
112.4
0.48
35.2
32.5
1.850
0.22
0.11
70.1
0.45
2.450
PromptHMR [ 39 ]
89.5
0.62
68.3
9.4
0.135
0.35
0.19
45.2
0.11
0.140
Table 3: Comparison on simulation datasets. Sim-Geometry isolates pose complexity with 3D ground truth. Sim-Visual isolates occlusion with realistic rendering. Best results are in bold . The asterisk ( ∗ ) indicates a statistically significant improvement ( p<0.001 ).
3DPW
Method
PA-MPJPE ↓
MPJPE ↓
PVE ↓
HMR 2.0 [ 11 ]
44.4
69.8
82.2
GVHMR [ 32 ]
37.0
56.6
68.7
CoMotion [ 27 ]
37.3
60.0
71.5
SMPLest-X [ 43 ]
45.2
74.8
88.4
PromptHMR [ 39 ]
36.6
58.7
69.4
Table 4: Results on standard HMR video benchmarks. 3DPW and EMDB: camera-space metrics (PA-MPJPE / MPJPE / PVE, mm). RICH: PA-PVE (mm) via nearest-neighbor vertex matching. The best result in each column is in bold .
Figure 5: Per-frame Mesh-Mask IoU ( ↑ ) over time for one simulation video. At approximately frame 260, the sequence introduces a severe camera viewpoint transition coupled with heavy drape occlusion. While all baseline methods suffer failure frames (IoU dropping below 0.2), Patient4D successfully triggers Rigid Fallback, retrieving a high-quality historical reference to immediately recover and maintain high spatial alignment (IoU ≈ 0.9).
Figure 6: Per-frame mesh vertex displacement ( ↓ ) between consecutive frames. Detection-based baselines (HMR 2.0, SMPLest-X, PromptHMR) exhibit large inter-frame jumps when detections shift or fail, with SMPLest-X exceeding 1 m regularly (capped for visibility). Patient4D and GVHMR maintain low displacement ( < 0.1 m), reflecting temporally consistent reconstructions.
Figure 7: Qualitative results on 3DPW video sequences. Patient4D recovers accurate meshes for multiple people in outdoor scenes without any domain-specific tuning. Each detected person is shown in a distinct color. Top: two people arguing on a cobblestone street. Bottom: two people descending stairs.
Config.
IoU ↑
> 0.6 ↑
< 0.3 ↓
Δˉ↓
(A) SAM3 + SAM3DBody (raw)
0.70
84.0
2.1
0.024
(B) (A) + Outlier rejection
0.71
84.2
2.3
0.022
(C) (B) + Smoothing
0.58
60.5
8.5
0.022
(D) (C) + Pose Locking
0.71
84.5
1.8
0.022
(E) (D) + Rigid Fallback (full)
0.72
85.0
2.0
0.022
(F) Patient4D (Ours)
0.75
92.7
0.7
0.019
Table 5: Component ablation (macro-averaged across Sim-Geometry and Sim-Visual datasets). Rows A–E: sequential additive chain. Row F (Patient4D): skips smoothing and applies Pose Locking and selective Rigid Fallback directly to outlier-rejected raw predictions.
Parameter
Value
IoU ↑
Δˉ↓
λtemp
0 (none)
0.75
0.021
0.05
0.75
0.020
0.1 (ours)
0.75
0.019
0.5
0.73
0.018
Max iterations
50
0.74
0.020
150 (ours)
0.75
0.019
Table 6: Sensitivity of Rigid Fallback to temporal regularization λtemp and optimization iterations, averaged across all surgical sequences.
Stage
Time (s)
VRAM (GB)
Device
SAM3 segmentation
40
12.4
GPU
MoGe FOV
2
3.8
GPU
SAM3DBody HMR
550
18.2
GPU
Post-processing
< 1
–
CPU
Rigid fallback
45
1.2
GPU
Total
638
18.2
–
Table 7: Per-stage cost for 300 frames on a single NVIDIA RTX 3090 GPU.
Monocular video human mesh recovery is essential for digital humans, avatar animation, and embodied simulation, where both temporal stability and expressive whole-body motion are required. Existing video HMR methods produce coherent body motion but often overlook detailed hand articulation, while image-based whole-body methods recover SMPL-X meshes independently per frame, often leading to jittery and inaccurate hand motion. We present a temporally coherent whole-body HMR framework for challenging in-the-wild monocular videos. Our model unifies body context and part-specific hand observations through residual body-hand fusion, enabling stable body motion and detailed hand recovery within a single temporal architecture. We further introduce close-up-aware augmentation to improve robustness under upper-body framing. Experiments on whole-body and body-only benchmarks demonstrate improved hand reconstruction and competitive body accuracy. Our method also produces temporally stable and 2D-consistent SMPL-X motion in challenging real-world videos.
Reconstructing dynamic non-rigid objects from monocular video requires integrating visual cues from direct observations with data-driven priors over geometry and appearance. Prior approaches either learn to directly predict 4D representations from visual input or initialize a 3D representation that is subsequently deformed and refined based on video evidence. However, the former are constrained by the scarcity of 4D training data, while the latter leverage priors only for the initial reconstruction and rely solely on video supervision thereafter; neither handles complex in-the-wild scenarios with large deformations and occlusions well. We present Lift4D, a test-time optimization framework that addresses both limitations. First, we adapt an existing single-view 3D reconstruction model to yield temporally consistent per-frame predictions via causal latent conditioning, providing a coherent initialization for a deformable 3D Gaussian Splatting representation. We then ``sculpt'' this representation to match the input video through an occlusion-aware optimization that faithfully recovers visible surface details while completing unobserved regions using a view-conditioned diffusion prior. We demonstrate that Lift4D clearly improves over prior 4D reconstruction methods, particularly on challenging in-the-wild sequences with severe occlusions and non-rigid motion.
Existing volumetric capture of dynamic human performance achieves high fidelity with dense camera arrays. However, in real-world scenarios, only a handful of low-overlap cameras are available, which degrades the output quality and leaves large areas unobserved. Recent 4D reconstruction methods have focused on low-overlap settings, yet they still produce noticeable artifacts in under-observed regions. Video diffusion models have emerged as another option, but they show geometrically inconsistent results for humans. To address these limitations, we propose StudioRecon, a pipeline that reconstructs 4D human scenes from sparse, low-overlap cameras by decoupling background and humans. We densify background supervision by synthesizing hundreds of camera-controlled novel views with a video diffusion model. We also robustly initialize deformable Gaussian humans with cross-view identity association and triangulated multi-view keypoint fitting. Finally, our recursive enhancement module with motion-adaptive consistency injection harmonizes the composed output, thereby further avoiding remaining artifacts. We achieve state-of-the-art novel view synthesis across four real-world datasets and demonstrate applications such as novel trajectory rendering and human replacement.