MedBOE: Optimization-Based Estimation of Complete Patient Body Models from Whole-Body Medical Imaging
Authors: Hoijoon Jung, Yuhao Wei, Mingxiao Tu, Younhyun Jung, Michael Fulham, Jinman Kim
Organizations: School of Computer Science, The University of Sydney, Camperdown, NSW, 2050, Australia. · School of Computing, Gachon University, Seongnam-Si, Gyeonggi-Do, 13120, Republic of Korea. · Department of Molecular Imaging, Royal Prince Alfred Hospital, Camperdown, NSW, 2050, Australia.
True whole-body medical imaging captures a patient from head to toe and provides comprehensive anatomical and/or functional information. This coverage enables reconstruction of the external body surface, yielding patient body models. However, scanner geometry (e.g., bore diameter), patient characteristics, and positioning can leave body regions missing (e.g., lateral arms/torso, hands, forearms, elbows), producing incomplete patient body models. Such incompleteness limits downstream medical analyses, including derivation of body-habitus metrics (BHMs), such as body volume and surface area, for injected radiotracer activity regimens, cardiometabolic risk assessment, and computational anatomical modeling for medical imaging simulation and radiation dosimetry. We present the Medical Body model Optimization-based Estimator (MedBOE), an automated, training-free framework for estimating complete patient body models from whole-body images with missing body regions. Our innovation is to leverage the Skinned Multi-Person Linear (SMPL) template, a parametric body mesh representing a broad range of body shapes and poses, in medical imaging. MedBOE aligns the SMPL template to the incomplete patient body models via attribute-guided optimization and estimates missing body regions through geometrically and volumetrically constrained deformation. We evaluated MedBOE using two public datasets: AutoPET (91 whole-body PET-CT images) and Human Implicit Tissues (HIT; 40 whole-body segmentations from MRI images). MedBOE achieved the best overall volumetric and geometric performance among evaluated methods. BHMs derived from MedBOE were within 2% error, suggesting low percentage errors for downstream analyses. MedBOE offers an automated, training-free method for estimating complete patient body models from whole-body images with missing body regions.
Figures & tables
Figure 1: Overview of the proposed MedBOE. A whole-body image with missing forearms, elbows, and hands (a) is processed through MedBOE’s three modules: (i) Body model reconstruction ; (ii) Global parametric fitting ; (iii) Missing Body Region Estimation and Surface Deformation , yielding outputs (b)–(e). The patient’s face is blurred for anonymization
Iteration
ESparse
EDist
EPrior
EWeight
EContact
EDense
0
100
0.5
0.5
10
0
10
30
80
2
0.5
10
0
10
100
50
5
0.5
10
0
5
200
5
10
0.5
1
0
2
300
2
10
0.5
0.5
0
2
400
2
10
0.5
0
2
2
Table 1: Attribute weights used in the global parametric fitting module during optimization.
Figure 2: Complete body models estimated from the AutoPET dataset. (a) is GT body models, (b) is simulated incomplete models where darker grey indicates the internal surface exposed by the trimming procedure. Results of (c) MedBOE, (d) IF-Net, (e) ShapeFormer, and (f) PCN. Blue dotted boxes highlight differences in estimating the missing body regions (elbows) between MedBOE and IF-Net
Figure 3: Detailed comparison of the missing body regions estimated by (a) MedBOE and (b) IF-Net. Blue dotted boxes highlight the key difference in estimated regions. GT models are shown as red wireframes
Figure 4: Complete body models estimated from the HIT dataset. (a) is GT body models, (b) is simulated incomplete models where darker grey indicates the internal surface exposed by the trimming procedure. Results of (c) MedBOE, (d) IF-Net, (e) ShapeFormer, (f) ArtEq, and (g) PCN. Orange dashed lines mark the 10 cm above-head apex cutoff. Blue dotted boxes highlight differences in forearms and hands region estimation among MedBOE, ShapeFormer, and PCN
Metrics
Volume ↓
Surface area ↓
Chest cir. ↓
Waist cir. ↓
Hip cir. ↓
APE
0.85±0.46 (0.13–2.18)
2.04±1.75 (0.01–7.27)
2.05±1.61 (0.08–8.02)
1.64±1.22 (0.12–5.67)
0.77±0.42 (0.08–1.75)
MAE
0.001±0.000 (0.000–0.002)
0.042±0.031 (0.004–0.138)
0.021±0.017 (0.001–0.096)
0.016±0.012 (0.001–0.056)
0.008±0.005 (0.001–0.022)
Table 3: BHM errors derived from the estimated complete body models of the HIT dataset. 0 0 footnotetext: APE: absolute percentage error, MAE: mean absolute error, cir.: circumference.
Figure 5: Circumferential measurement locations (red dotted lines) for chest, waist, and hip, on (a) the GT and (b) the MedBOE-estimated complete body models
Dataset
RBE ↓
RVE ↓
IoU ↑
CD ↓
HD ↓
With the module
AutoPET Male
1.47±1.61 * (0.12–9.04)
0.90±0.80 * (0.03–4.22)
0.92±0.03 * (0.83–0.94)
0.02±0.02 * (0.02–0.08)
0.10±0.10 * (0.03–0.39)
AutoPET Female
2.06±1.81 * (0.05–9.62)
0.93±0.72 * (0.03–3.41)
0.92±0.01 * (0.85–0.94)
0.02±0.02 * (0.01–0.12)
0.09±0.10 * (0.02–0.55)
HIT Male
1.85±1.87 * (0.01–7.27)
0.86±0.40 * (0.16–1.52)
0.94±0.01 * (0.93–0.95)
0.04±0.01 * (0.02–0.06)
0.19±0.09 (0.08–0.38)
HIT Female
2.15±1.70 * (0.06–7.09)
0.85±0.50 * (0.13–2.18)
0.94±0.01 * (0.93–0.95)
0.03±0.01 * (0.02–0.07)
0.14±0.06 (0.06–0.30)
Without the module
AutoPET Male
3.83±2.39 (0.65–13.87)
5.93±2.45 (1.68–11.19)
0.78±0.03 (0.70–0.83)
0.03±0.02 (0.02–0.09)
0.10±0.09 (0.04–0.39)
AutoPET Female
4.23±2.93 (0.07–13.81)
7.32±2.78 (0.02–15.51)
0.77±0.03 (0.68–0.82)
0.03±0.02 (0.02–0.14)
0.10±0.09 (0.04–0.55)
Table 4: Performance of MedBOE with and without the missing body region estimation and surface deformation module. The better results are shown in bold. 0 0 footnotetext: An asterisk (*) indicates significance from the results without the module (paired t-test, p<0.05 ).
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Representative front, left, and right views (left-to-right) comparing (a) the simulated trimmed incomplete patient body model and (b) the corresponding CT-derived incomplete patient body model. External surfaces are rendered in blue, and internal cut surfaces exposed by truncation are rendered in red
Figure 7: Zoomed views of the AutoPET male complete body model estimates. The top row shows the region above the shoulders, and the bottom row shows the leg region. (a) GT body model, (b) MedBOE, (c) IF-Net, (d) ShapeFormer, and (e) PCN
Figure 8: Zoomed views of the AutoPET female complete body model estimates. The top row shows the region above the shoulders, and the bottom row shows the leg region. (a) GT body model, (b) MedBOE, (c) IF-Net, (d) ShapeFormer, and (e) PCN
Figure 9: Representative challenging case from the AutoPET dataset. (a) GT body model, (b) incomplete patient body model, (c) MedBOE, (d) IF-Net, (e) ShapeFormer, and (f) PCN
Gender
Metric
Comparison
Mean difference
p -value
Cohen’s d
95% CI
Male
RBE
MedBOE - IF-Net
−1.817
0.439
−0.400
[−5.334,1.699]
ShapeFormer - IF-Net
11.474
2.02×10−11
1.532
[7.957,14.991]
ShapeFormer - MedBOE
13.291
<1.00×10−12
2.154
[9.774,16.808]
RVE
MedBOE - IF-Net
−2.920
0.338
−0.487
[−7.833,1.993]
ShapeFormer - IF-Net
15.539
6.51×10−11
1.470
[10.626,20.452]
ShapeFormer - MedBOE
18.459
<1.00×10−12
2.112
[13.546,23.372]
Appendix
Table 5: Pairwise method-comparison statistics from Tukey HSD tests for the AutoPET dataset.
Method
Metric
Mean difference
p -value
Cohen’s d
95% CI
MedBOE
RBE
0.590
0.109
0.340
[−0.134,1.315]
RVE
0.032
0.849
0.042
[−0.300,0.363]
IoU
0.001
0.785
0.067
[−0.008,0.011]
CD
−0.001
0.807
−0.051
[−0.008,0.006]
HD
−0.006
0.783
−0.060
[−0.047,0.036]
IF-Net
RBE
−1.876
0.088
−0.468
[−4.041,0.290]
Appendix
Table 6: Male-female comparison statistics from Welch’s independent two-sample t -tests for the AutoPET dataset.
Figure 10: Zoomed views of the HIT male complete patient body model estimates. The top row shows the hand and forearm regions, and the bottom row shows the leg region. (a) GT body model, (b) MedBOE, (c) IF-Net, (d) ShapeFormer, and (e) PCN
Figure 11: Zoomed views of the HIT female complete patient body model estimates. The top row shows the hand and forearm regions, and the bottom row shows the leg region. (a) GT body model, (b) MedBOE, (c) IF-Net, (d) ShapeFormer, and (e) PCN. PCN is shown as point-cloud output
Figure 12: Representative challenging case from the HIT dataset. (a) GT body model, (b) incomplete patient body model, (c) MedBOE, (d) IF-Net, (e) ShapeFormer, (f) ArtEq, and (g) PCN
Method
Metric
Mean difference
p -value
Cohen’s d
95% CI
MedBOE
RBE
0.294
0.622
0.167
[−0.915,1.504]
RVE
−0.019
0.896
−0.041
[−0.310,0.272]
IoU
0.004
0.123
0.476
[−0.001,0.008]
CD
−0.006
0.112
−0.510
[−0.012,0.001]
HD
−0.048
0.074
−0.666
[−0.100,0.005]
IF-Net
RBE
−3.780
1.08×10−5
−2.013
[−5.169,−2.391]
Appendix
Table 8: Male-female comparison statistics from Welch’s independent two-sample t -tests for the HIT dataset.
Method
Metric
Mean difference
p -value
Cohen’s d
95% CI
MedBOE
RBE
0.202
0.543
0.116
[−0.458,0.862]
RVE
−0.063
0.557
−0.093
[−0.274,0.148]
IoU
0.021
<1.00×10−12
1.218
[0.016,0.025]
CD
0.011
5.30×10−5
0.680
[0.006,0.015]
HD
0.067
3.45×10−5
0.748
[0.037,0.098]
IF-Net
RBE
8.401
<1.00×10−12
2.266
[7.220,9.583]
Appendix
Table 9: Dataset-wise comparison statistics between AutoPET and HIT.
Dataset
Gender
Metric
Mean difference
p -value
Cohen’s d
95% CI
AutoPET
Male
RBE
−2.359
9.85×10−9
−1.271
[−2.996,−1.721]
RVE
−5.036
<1.00×10−12
−2.112
[−5.855,−4.217]
IoU
0.133
<1.00×10−12
6.648
[0.126,0.140]
CD
−0.009
<1.00×10−12
−3.779
[−0.010,−0.009]
HD
−0.009
5.31×10−6
−0.912
[−0.012,−0.005]
Female
RBE
−2.168
4.97×10−9
−0.926
[−2.795,−1.541]
Appendix
Table 10: Ablation study statistics between with and without the missing region estimation and surface deformation module.
Methods
RBE ↓
RVE ↓
IoU ↑
CD ↓
HD ↓
Male (35)
MedBOE
30.28±42.69 * (0.32–225.96)
46.48±78.26 * (3.06–386.43)
0.44±0.16 * (0.09–0.70)
0.06±0.06 (0.01–0.24)
0.17±0.19 (0.02–0.76)
IF-Net ^
91.38±7.46 (68.81–99.98)
97.67±3.52 (82.99–100.00)
0.02±0.03 (0.00–0.17)
0.16±0.15 (0.04–0.69)
0.33±0.24 (0.06–0.90)
Shape Former ^
77.74±71.95 (10.34–412.23)
94.45±116.24 (5.26–676.19)
0.05±0.07 (0.00–0.23)
0.28±0.22 (0.06–0.89)
0.57±0.34 (0.09–1.33)
PCN
-
-
-
0.62±0.32 (0.25–1.54)
1.18±0.43 (0.45–1.96)
Female (56)
MedBOE
39.29±57.76 * (0.28–322.29)
56.36±72.07 * (0.57–363.75)
0.39±0.16 * (0.00–0.66)
0.06±0.07 (0.01–0.35)
0.17±0.21 * (0.02–0.84)
IF-Net ^
95.06±10.92 (21.09–99.93)
95.06±10.92 (21.09–99.93)
0.04±0.05 (0.00–0.25)
0.12±0.12 (0.01–0.85)
0.31±0.26 (0.02–0.92)
Appendix
Table 11: Results of missing region estimation on the AutoPET dataset. 0 0 footnotetext: An Asterisk (*) indicates statistical differences from the second-best results. A circumflex (^) indicates methods with estimation failure cases excluded from the results.
Methods
RBE ↓
RVE ↓
IoU ↑
CD ↓
HD ↓
Male (15)
MedBOE
19.50±17.96 (1.82–68.95)
15.50±15.97 * (1.73–51.69)
0.29±0.12 * (0.10–0.51)
0.12±0.07 (0.03–0.28)
0.53±0.59 (0.08–1.84)
IF-Net
94.03±2.40 (88.87–97.98)
97.78±1.39 (93.85–99.25)
0.02±0.01 (0.01–0.05)
0.20±0.06 (0.12–0.32)
0.63±0.56 (0.29–2.03)
Shape Former
56.05±20.33 (10.61–82.13)
61.50±23.69 (6.27–88.70)
0.16±0.09 (0.05–0.32)
0.14±0.06 (0.05–0.30)
0.70±0.55 (0.17–1.75)
PCN
-
-
-
0.16±0.13 (0.03–0.44)
0.82±0.37 (0.30–1.60)
ArtEq
148.41±89.68 (11.85–293.73)
-
-
0.60±0.25 (0.29–1.22)
1.07±0.48 (0.46–1.82)
Female (25)
MedBOE
26.50±17.05 (0.17–55.70)
24.81±19.84 * (1.44–80.44)
0.26±0.09 (0.12–0.44)
0.10±0.05 (0.04–0.24)
0.28±0.37 (0.06–1.71)
Appendix
Table 12: Results of body region estimation on the HIT dataset. 0 0 footnotetext: An Asterisk (*) indicates statistical differences from the second-best results.
We introduce SAM 3D Body (3DB), a promptable model for single-image full-body 3D human mesh recovery (HMR) that demonstrates state-of-the-art performance, with strong generalization and consistent accuracy in diverse in-the-wild conditions. 3DB estimates the human pose of the body, feet, and hands. It is the first model to use a new parametric mesh representation, Momentum Human Rig (MHR), which decouples skeletal structure and surface shape. 3DB employs an encoder-decoder architecture and supports auxiliary prompts, including 2D keypoints and masks, enabling user-guided inference similar to the SAM family of models. We derive high-quality annotations from a multi-stage annotation pipeline that uses various combinations of manual keypoint annotation, differentiable optimization, multi-view geometry, and dense keypoint detection. Our data engine efficiently selects and processes data to ensure data diversity, collecting unusual poses and rare imaging conditions. We present a new evaluation dataset organized by pose and appearance categories, enabling nuanced analysis of model behavior. Our experiments demonstrate superior generalization and substantial improvements over prior methods in both qualitative user preference studies and traditional quantitative analysis. Both 3DB and MHR are open-source.
Fitting an underlying body model to 3D clothed human assets has been extensively studied, yet most approaches focus on either single-modal inputs such as point clouds or multi-view images alone, often requiring a known metric scale. This constraint is frequently impractical, especially for AI-generated assets where scale distortion is common. We propose OmniFit, a method that can seamlessly handle diverse multi-modal inputs, including full scans, partial depth observations, and image captures, while remaining scale-agnostic for both real and synthetic assets. Our key innovation is a simple yet effective conditional transformer decoder that directly maps surface points to dense body landmarks, which are then used for SMPL-X parameter fitting. In addition, an optional plug-and-play image adapter incorporates visual cues to compensate for missing geometric information. We further introduce a dedicated scale predictor that rescales subjects to canonical body proportions. OmniFit substantially outperforms state-of-the-art methods by 57.1 to 80.9 percent across daily and loose clothing scenarios. To the best of our knowledge, it is the first body fitting method to surpass multi-view optimization baselines and the first to achieve millimeter-level accuracy on the CAPE and 4D-DRESS benchmarks.
Zeyu Cai, Yuliang Xiu, Renke Wang +8
Nanjing University · Westlake University · Nanjing University of Science and Technology +2
Parametric human body models are foundational to human reconstruction, animation, and simulation, yet they remain mutually incompatible: SMPL, SMPL-X, MHR, Anny, and related models each diverge in mesh topology, skeletal structure, shape parameterization, and unit convention, making it impractical to exploit their complementary strengths within a single pipeline. We present SOMA, a unified body layer that bridges these heterogeneous representations through three abstraction layers. Mesh topology abstraction maps any source model's identity to a shared canonical mesh in constant time per vertex. Skeletal abstraction recovers a full set of identity-adapted joint transforms from any body shape, whether in rest pose or an arbitrary posed configuration, in a single closed-form pass, with no iterative optimization or per-model training. Pose abstraction inverts the skinning pipeline to recover unified skeleton rotations directly from posed vertices of any supported model, enabling heterogeneous motion datasets to be consumed without custom retargeting. Together, these layers reduce the O(M2) per-pair adapter problem to O(M) single-backend connectors, letting practitioners freely mix identity sources and pose data at inference time. The entire pipeline is fully differentiable end-to-end and GPU-accelerated via NVIDIA-Warp.