Patient4D: Temporally Consistent Patient Body Mesh Recovery from Monocular Operating Room Video

Mingxiao Tu; Hoijoon Jung; Alireza Moghadam; Andre Kyme; Jinman Kim

cs.CVMar 17, 2026

Patient4D: Temporally Consistent Patient Body Mesh Recovery from Monocular Operating Room Video

Authors: Mingxiao Tu, Hoijoon Jung, Alireza Moghadam, Andre Kyme, Jinman Kim

Organizations: University of Sydney

Abstract

Recovering a dense 3D body mesh from monocular video remains challenging under occlusion from draping and continuously moving camera viewpoints. This configuration arises in surgical augmented reality (AR), where an anesthetized patient lies under surgical draping while a surgeon's head-mounted camera continuously changes viewpoint. Existing human mesh recovery (HMR) methods are typically trained on upright, moving subjects captured from relatively stable cameras, leading to performance degradation under such conditions. To address this, we present Patient4D, a stationarity-constrained reconstruction pipeline that explicitly exploits the stationarity prior. The pipeline combines image-level foundation models for perception with lightweight geometric mechanisms that enforce temporal consistency across frames. Two key components enable robust reconstruction: Pose Locking, which anchors pose parameters using stable keyframes, and Rigid Fallback, which recovers meshes under severe occlusion through silhouette-guided rigid alignment. Together, these mechanisms stabilize predictions while remaining compatible with off-the-shelf HMR models. We evaluate Patient4D on 4,680 synthetic surgical sequences and three public HMR video benchmarks. Under surgical drape occlusion, Patient4D achieves a 0.75 mean IoU, reducing failure frames from 30.5% to 1.3% compared to the best baseline. Our findings demonstrate that exploiting stationarity priors can substantially improve monocular reconstruction in clinical AR scenarios.

Figures & tables

Figure 1: Overview of the Patient4D pipeline. Given a monocular surgical video and a single point prompt, SAM3 produces per-frame body masks while MoGe estimates camera intrinsics from the first frame. SAM3DBody then recovers initial per-frame SMPL meshes from the masked regions; body shape $\boldsymbol{\beta}$ and scale are locked to a single keyframe estimate and broadcast to all frames. To enforce temporal consistency, Pose Locking anchors body pose parameters to a median value calculated across the sequence. Frames whose mesh-mask IoU falls below a threshold ( $\tau_{\text{IoU}}{=}0.6$ ) are corrected by Rigid Fallback, which selects the best reference mesh from a keyframe pool and optimizes a rigid transformation to align its projection with the current mask via a Dice objective.

Figure 2: Rigid Fallback on a challenging frame. Left: per-frame HMR produces a misaligned mesh (IoU $=$ 0.21) when the camera moves to an oblique viewpoint with partial visibility. Right: Rigid Fallback takes the best reference mesh from the keyframe pool and optimizes rotation and camera translation to align its projection with the SAM3 mask via the Dice objective, recovering correct alignment (IoU $=$ 0.93). Bottom row: mask overlap (green $=$ projected mesh, blue $=$ SAM3 mask, cyan $=$ intersection).

Figure 3: Representative frames from the simulation datasets. Top: Sim-Geometry sequences rendered from SLP-3Dfits with ground truth SMPL meshes, generated patient texture and orbiting camera silhouettes. Bottom: Sim-Visual frames at three drape coverage levels (none, partial, heavy), depicting realistic surgical scenes with photorealistic skin textures and operating room environments. These two are complementary: Sim-Geometry shows clean silhouettes with ground truth SMPL meshes whereas Sim-Visual demonstrates increasing visual complexity from drapes.

Figure 4: Qualitative comparison of mesh recovery on a synthetic surgical sequence. The patient is positioned in a lateral decubitus pose under no draping. As the camera orbits from the initial viewpoint (Frames 10, 50) to extreme oblique angles (Frames 150, 200), the 2D visual cues become ambiguous. Standard per-frame and video-based baselines fail to recognize the stationarity prior of the patient; they attempt to fit the changing 2D contours by distorting the 3D anatomy (e.g., unnaturally twisting the spine or flattening the torso), resulting in plummeting Mesh-Mask IoU. In contrast, Patient4D (orange) enforces temporal consistency, preserving the correct 3D geometry and maintaining strong spatial alignment across the entire camera trajectory.

Parameter	Sim-Geometry	Sim-Visual
Source	SLP-3Dfits [ 22 ]	Seedance2.0 [ 3 ]
Sequences	4545	135
Frames/seq.	375 (25 fps)	${\sim}$ 192 (24 fps)
Resolution	760 $\times$ 428	1280 $\times$ 720
3D ground truth	Yes	No
Drape occlusion	None	3 levels

Table 1: Simulation dataset configurations.

Drape Level	IoU $\uparrow$	$>$ 0.6 $\uparrow$	$<$ 0.3 $\downarrow$	$\bar{\Delta}$ $\downarrow$	Seqs
None	0.84	99.5%	0.0%	0.010	30
Partial	0.73	94.5%	0.5%	0.051	60
Heavy	0.69	80.1%	3.3%	0.013	45
All	0.75	90.8%	1.3%	0.029	135

Table 2: Performance breakdown of Patient4D across varying levels of drape occlusion on the Sim-Visual dataset. While overall baseline comparisons are provided in \cref tab:sim_baselines, this analysis isolates how Patient4D degrades under increasing visual ambiguity. IoU: Mesh-Mask IoU (

\uparrow

\bar{\Delta}

: mean vertex displacement (

\downarrow

	Sim-Geometry					Sim-Visual
Method	PA-PVE $\downarrow$	IoU $\uparrow$	$>$ 0.6 $\uparrow$	$<$ 0.3 $\downarrow$	$\bar{\Delta}$ $\downarrow$	IoU $\uparrow$	CV-IoU ${}_{20}\uparrow$	$<$ 0.3 $\downarrow$	$\Delta_{\text{pose}}$ $\downarrow$	$\bar{\Delta}$ $\downarrow$
HMR 2.0 [ 11 ]	84.2	0.68	78.2	4.5	0.185	0.52	0.28	35.1	0.14	0.650
GVHMR [ 32 ]	95.6	0.58	62.4	8.2	0.014	0.45	0.31	30.5	0.05	0.025
CoMotion [ 27 ]	87.9	0.65	71.5	10.1	0.042	0.38	0.22	52.4	0.08	0.065
SMPLest-X [ 43 ]	112.4	0.48	35.2	32.5	1.850	0.22	0.11	70.1	0.45	2.450
PromptHMR [ 39 ]	89.5	0.62	68.3	9.4	0.135	0.35	0.19	45.2	0.11	0.140

Table 3: Comparison on simulation datasets. Sim-Geometry isolates pose complexity with 3D ground truth. Sim-Visual isolates occlusion with realistic rendering. Best results are in bold . The asterisk ( ∗ ) indicates a statistically significant improvement (

p<0.001

3DPW
Method	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	PVE $\downarrow$
HMR 2.0 [ 11 ]	44.4	69.8	82.2
GVHMR [ 32 ]	37.0	56.6	68.7
CoMotion [ 27 ]	37.3	60.0	71.5
SMPLest-X [ 43 ]	45.2	74.8	88.4
PromptHMR [ 39 ]	36.6	58.7	69.4

Table 4: Results on standard HMR video benchmarks. 3DPW and EMDB: camera-space metrics (PA-MPJPE / MPJPE / PVE, mm). RICH: PA-PVE (mm) via nearest-neighbor vertex matching. The best result in each column is in bold .

Figure 5: Per-frame Mesh-Mask IoU ( $\uparrow$ ) over time for one simulation video. At approximately frame 260, the sequence introduces a severe camera viewpoint transition coupled with heavy drape occlusion. While all baseline methods suffer failure frames (IoU dropping below 0.2), Patient4D successfully triggers Rigid Fallback, retrieving a high-quality historical reference to immediately recover and maintain high spatial alignment (IoU $\approx$ 0.9).

Figure 6: Per-frame mesh vertex displacement ( $\downarrow$ ) between consecutive frames. Detection-based baselines (HMR 2.0, SMPLest-X, PromptHMR) exhibit large inter-frame jumps when detections shift or fail, with SMPLest-X exceeding 1 m regularly (capped for visibility). Patient4D and GVHMR maintain low displacement ( $<$ 0.1 m), reflecting temporally consistent reconstructions.

Figure 7: Qualitative results on 3DPW video sequences. Patient4D recovers accurate meshes for multiple people in outdoor scenes without any domain-specific tuning. Each detected person is shown in a distinct color. Top: two people arguing on a cobblestone street. Bottom: two people descending stairs.

Config.	IoU $\uparrow$	$>$ 0.6 $\uparrow$	$<$ 0.3 $\downarrow$	$\bar{\Delta}$ $\downarrow$
(A) SAM3 + SAM3DBody (raw)	0.70	84.0	2.1	0.024
(B) (A) + Outlier rejection	0.71	84.2	2.3	0.022
(C) (B) + Smoothing	0.58	60.5	8.5	0.022
(D) (C) + Pose Locking	0.71	84.5	1.8	0.022
(E) (D) + Rigid Fallback (full)	0.72	85.0	2.0	0.022
(F) Patient4D (Ours)	0.75	92.7	0.7	0.019

Table 5: Component ablation (macro-averaged across Sim-Geometry and Sim-Visual datasets). Rows A–E: sequential additive chain. Row F (Patient4D): skips smoothing and applies Pose Locking and selective Rigid Fallback directly to outlier-rejected raw predictions.

Parameter	Value	IoU $\uparrow$	$\bar{\Delta}$ $\downarrow$
$\lambda_{\text{temp}}$	0 (none)	0.75	0.021
	0.05	0.75	0.020
	0.1 (ours)	0.75	0.019
	0.5	0.73	0.018
Max iterations	50	0.74	0.020
	150 (ours)	0.75	0.019

Table 6: Sensitivity of Rigid Fallback to temporal regularization

\lambda_{\text{temp}}

and optimization iterations, averaged across all surgical sequences.

Stage	Time (s)	VRAM (GB)	Device
SAM3 segmentation	40	12.4	GPU
MoGe FOV	2	3.8	GPU
SAM3DBody HMR	550	18.2	GPU
Post-processing	$<$ 1	–	CPU
Rigid fallback	45	1.2	GPU
Total	638	18.2	–

Table 7: Per-stage cost for 300 frames on a single NVIDIA RTX 3090 GPU.

AR in Surgery

See all 14 papers

DanceHMR: Hand-Aware Whole-Body Human Mesh Recovery from Monocular Videos
May 18, 2026Wenhao Shen, Ming Zhou, Hengyuan Zhang +33D Hand Pose Estimation Human Mesh Recovery
ByteDance Intelligent Creation
Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild
Jun 22, 2026Yehonathan Litman, Xiaoxuan Ma, Manan Shah +44D Reconstruction Dynamic Scene Reconstruction
Carnegie Mellon University
4D Human-Scene Reconstruction from Low-Overlap Captures
Jul 10, 2026Minhyuk Hwang, Sangmin Kim, Seunguk Do +24D Reconstruction Dynamic Scene Reconstruction
Seoul National University, Republic of Korea

May 18, 2026J/K move · Enter open · S save

DanceHMR: Hand-Aware Whole-Body Human Mesh Recovery from Monocular Videos

Wenhao Shen, Ming Zhou, Hengyuan Zhang, Siyuan Bian+2

ByteDance Intelligent Creation

3D Hand Pose Estimation Human Mesh Recovery 3D Mesh Reconstruction 3D Human Reconstruction 3D Human Motion Reconstruction

Open paper AskPDF arXiv

Monocular video human mesh recovery is essential for digital humans, avatar animation, and embodied simulation, where both temporal stability and expressive whole-body motion are required. Existing video HMR methods produce coherent body motion but often overlook detailed hand articulation, while image-based whole-body methods recover SMPL-X meshes independently per frame, often leading to jittery and inaccurate hand motion. We present a temporally coherent whole-body HMR framework for challenging in-the-wild monocular videos. Our model unifies body context and part-specific hand observations through residual body-hand fusion, enabling stable body motion and detailed hand recovery within a single temporal architecture. We further introduce close-up-aware augmentation to improve robustness under upper-body framing. Experiments on whole-body and body-only benchmarks demonstrate improved hand reconstruction and competitive body accuracy. Our method also produces temporally stable and 2D-consistent SMPL-X motion in challenging real-world videos.

Abstract

AR in Surgery

Explore similar work

DanceHMR: Hand-Aware Whole-Body Human Mesh Recovery from Monocular Videos

Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild

4D Human-Scene Reconstruction from Low-Overlap Captures